Skip to content
Permalink
chubing
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

iam-knowledge-bits/connectors.adoc

Go to file
 
 
Cannot retrieve contributors at this time
connectors.adoc CSV Connector Work Plan building a csv connector for sis source drawn from BennOs 500k sample user files
584 lines (402 sloc) 23.2 KB
Raw Blame

connectors.adoc

2023-02-08 11:38:56 database table connector configuration

- Use workbench instance, guest resource, dbTable connector for a full working example -

demo in browser:

` resource, guest db, configuration in UI, in XML
` import task, operation statistics

.

2023-02-08 11:37:46 references and links

2022-09-19 13:08 chad redman developing SCIM 2 server

part of the Grouper roadmap for 2.7 is to rewrite the SCIM server. The current implementation uses a 3rd party library written for J2EE, which is why Grouper runs under TomEE and not regular Tomcat. There are a few options for replacement libraries, so this should be a reachable goal.

If the Grouper SCIM server is rewritten, the endpoints should not change significantly, but the object data is likely to change. The current service expresses objects in ways that differ from the published SCIM RFC’s [1][2], and a different solution would adhere more closely to the standards. An example of some ways SCIM in Grouper is non-standard and would change:

  • extensions are wrapped in an "extensions" node (includes group name or subject id, so essential fields)

  • userName is not present in user objects and is required

  • unknown attribute baseUrn

  • inconsistent use of group and subject ids vs. uuids

  • /Schemas endpoint is broken (infinite loop that eventually aborts)

  • no PATCH or BulkRequest support

Changes would impact integrations already in production, so the Grouper team is looking to hear from current users of the SCIM server.

Starting a conversation with the current users, as well as users holding back because of current limitations, would also be a good opportunity to make improvements to the system. BulkRequest isn’t supported, so large change sets are inefficient. PATCH operations are not currently supported, which means memberships can’t be managed through the group object. Instead, multiple calls potentially need to be made to look up uuids for the group, subject, and membership. That illustrates how cumbersome it is to work with uuids for groups and subjects in general, and maybe there is some opportunity to switch to more friendly subject ids and group names as resource keys.

So, if you are using the SCIM service in Grouper, or want to use a more standard version, please comment or let the Grouper team know, so that the needs can be better known.

2022-09-14 17:37 schema mapping, csv connector template

2022-05-19 09:43 utility for prompted user input in CLI scripts (for use in soliciting configuration items and choices)

2022-05-15 17:10 continue work on csv connector

TBD: SoR person to mP user schema mapping utility

Next resource definition: develop, test, document SIS resource creation using the 100-student csv sample from BennO’s mock data sets /Users/khazelton/opt/non.adoc/sis.csv

sorid GivenName MiddleInitial Surname Birthday EmailAddress TelephoneCountryCode TelephoneNumber NationalID Occupation Company

2022-05-13 05:50 continue work on csv connector

working example: /Users/khazelton/opt/non.adoc/source-hr.csv

A user template may be applied globally by including the following snippet in xref:/midpoint/reference/concepts/system-configuration-object/just after the "logging" element:

 <defaultUserTemplateRef oid="10000000-0000-0000-0000-000000000222"/>

System configuration xml; after logging element:

 <defaultObjectPolicyConfiguration id="101">
    <type>UserType</type>
    <objectTemplateRef xmlns:tns="http://midpoint.evolveum.com/xml/ns/public/common/common-3" oid="8098b124-c20c-4965-8adf-e528abedf7a4" relation="org:default" type="tns:ObjectTemplateType"/>
 </defaultObjectPolicyConfiguration>

That template ref, oid="8098b124-c20c-4965-8adf-e528abedf7a4", points to ../objects/objectTemplates/UserTemplate.xml which assigns the unique name and uid

uid,firstname,lastname,department,mail,validFrom,validTo
E600001,John R,Smith,HR_SOR,xjsmith@example.com,2018-01-01,9999-12-31
E600002,Alice,Anderson,HR_SOR,xaanderson@example.com,2016-03-15,9999-12-31
E600003,Ellen,Johnson,HR_SOR,xejohnson@example.com,2019-10-01,2019-12-31
E600004,Ron,Vasquez,HR_SOR,xrvasquez@example.com,2019-01-01,2019-10-31

csv resource def template: ../non.adoc/extCsvResourceA.xml

Resource on Aktis: 'HR SOR Source'

next task develop, test, document SIS resource creation from 100-student sample from BennO’s mock data sets /Users/khazelton/opt/non.adoc/sis.csv

2022-05-12 15:59 continue work on csv connector

$MIDPOINT_HOME:

in the workbench repo: ../Workbench/midpoint_server/container_files/mp-home in the running comtainer: /opt/midpoint/var

schema extension xsd’s go in $MIDPOINT_HOME/schema

~/opt/InCommonTAP-Examples-current/Workbench/midpoint_server/container_files/mp-home$

tree . -L 2
.
├── config.xml
├── cs-portal.csv
├── faculty-portal.csv
├── icf-connectors
│   ├── connector-grouper-rest-0.7.jar
│   ├── connector-rest-wordpress-.23-SNAPSHOT.jar
│   ├── connector-sympa-1.0.2-connector.jar
│   └── net.tirasa.connid.bundles.db.scriptedsql-2.2.6-SNAPSHOT.jar
├── mailing-lists.csv
├── post-initial-objects
│   ├── archetypes
│   ├── bulkActions
│   ├── functionLibraries
│   ├── objectTemplates
│   ├── ordering.txt
│   ├── orgs
│   ├── resources
│   ├── roles
│   ├── securityPolicy
│   ├── systemConfigurations
│   ├── tasks
│   └── users
├── res
│   └── sis-persons
├── schema
│   └── internet2.xsd  <-  example schema extension file   ────────────────────────────────
├── source-external.csv
└── staff-portal.csv

internet2.xsd user schema extension defines identifiers for each System of Record: HR_ID, SIS_ID, GUEST_ID, UserID

] Define 2 attributes SOR_ID and user_ID; each user record should have values for both attributes Avoids having to extend the schema every time a new SoR is integrated.

2022-04-28 19:42 how-to outline

0) A CSV file with attribute names on the 1st line 1) Bash script that collects info from users and uses that info to populate a fresh instance of a CSV resource definition file template

-) map from attr1 to uid -) map from attr2 to givenName -) …​

upload and execute resource definition

2022-04-27 18:44 CSV connector how-to

from csv file to generated resource def file to import of source into midPoint

userAttr.ods ← midPoint User Attribute Priority Categorization

CSV Connector Work Plan

CSV Resource Definition Steps

Document schema of csv file (see userCsvSchemaMap.ods/sisSor.csv)

Arrange for periodic publishing of the latest data to a location midpoint processes can reach (scp, sftp,…​) Dump into a db table for validation rules (

The heavy task: Compile a table that shows which source attributes map to which midPoint User attributes (for all connectors, not just csv) (See userCsvSchemaMap.ods/csv resource schema handling)

Extend the schema extension file (internet2.xsd in the Workbench) to include attributes that don’t have an appropriate match in the midPoint User Type.

The full current midPoint schema for “User”. We have learned that “truth is in the schema docs”: https://www.evolveum.com/downloads/midpoint/4.1/midpoint-4.1-schemadoc/ (midPoint Schema Doc home page)

Then click “UserType”

Starting from a template xml file (TBD), map the information from the schema document into the matching XML elements in the sections on <connectorRef/>, <connectorConfiguration/>, <schema/>, <schemaHandling>, and <synchronization/> (See sisSorResourceDef.xml)

Next step is to debug Resource Tasks
Import (and Reconcile)
Recompute

TBD
Build and test Synchronization Task
Live sync: Work with Ethan

2022-05-13 05:48 references and links

2022-04-05 13:59 csv connector how-to

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright (c) 2010-2017 Evolveum
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->
<name>Test CSV: username</name>
<description>Simple CSV resource that is using single identifier (username)</description>
<connectorRef type="ConnectorType">
    <filter>
        <q:equal>
            <q:path>c:connectorType</q:path>
            <q:value>com.evolveum.polygon.connector.csv.CsvConnector</q:value>
        </q:equal>
    </filter>
</connectorRef>
<connectorConfiguration xmlns:icfi="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/bundle/com.evolveum.polygon.connector-csv/com.evolveum.polygon.connector.csv.CsvConnector">
<icfc:configurationProperties>
    <icfi:filePath>/opt/midpoint/var/midpoint-username.csv</icfi:filePath>
    <icfi:encoding>utf-8</icfi:encoding>
    <icfi:fieldDelimiter>,</icfi:fieldDelimiter>
    <icfi:multivalueDelimiter>;</icfi:multivalueDelimiter>
    <icfi:uniqueAttribute>username</icfi:uniqueAttribute>
    <icfi:passwordAttribute>password</icfi:passwordAttribute>
</icfc:configurationProperties>
</connectorConfiguration>
<!-- Schema is empty. Schema should be generated by provisioning on the first use of this resource. -->
<schemaHandling>
<objectType>
    <displayName>Default Account</displayName>
    <default>true</default>
    <objectClass>ri:AccountObjectClass</objectClass>
<attribute>
    <ref>ri:username</ref>
    <outbound>
        <source>
            <path>$user/name</path>
        </source>
    </outbound>
</attribute>
<attribute>
    <ref>ri:firstname</ref>
    <outbound>
        <source>
            <path>$user/givenName</path>
        </source>
    </outbound>
</attribute>
<attribute>
    <ref>ri:lastname</ref>
    <outbound>
        <source>
            <path>$user/familyName</path>
        </source>
    </outbound>
</attribute>
<activation>
    <administrativeStatus>
        <outbound />
    </administrativeStatus>
</activation>
<credentials>
    <password>
        <outbound />
    </password>
</credentials>
    </objectType>
</schemaHandling>
<capabilities>
    <configured>
        <cap:activation>
            <cap:status>
                <cap:attribute>ri:disabled</cap:attribute>
                <cap:enableValue>false</cap:enableValue>
                <cap:disableValue>true</cap:disableValue>
            </cap:status>
        </cap:activation>
    </configured>
</capabilities>
<synchronization>
    <objectSynchronization>
        <objectClass>AccountObjectClass</objectClass>
        <kind>account</kind>
        <intent>Default</intent>
        <focusType>c:UserType</focusType>
        <enabled>true</enabled>
        <reconcile>false</reconcile>
    </objectSynchronization>
</synchronization>

</resource>

building a csv connector for sis source drawn from BennOs 500k sample user files

~/opt/InCommonTAP-Examples-current/Workbench/midpoint_server/container_files/mp-home/res/sis-persons
total 24
drwxr-xr-x 2 kh kh 4096 Jan 31 17:54 .
drwxr-xr-x 3 kh kh 4096 Jan 31 17:54 ..
-rw-r--r-- 1 kh kh 2531 Jan 31 17:54 SchemaScript.groovy
-rw-r--r-- 1 kh kh 5379 Jan 31 17:54 SearchScript.groovy
-rw-r--r-- 1 kh kh 1372 Jan 31 17:54 TestScript.groovy

end up in the midpoint server container:

ls -la /opt/midpoint/var/res/sis-persons
total 24
drwxr-xr-x 2 root root 4096 Feb 17 14:32 .
drwxr-xr-x 3 root root 4096 Feb 17 14:32 ..
-rw-r--r-- 1 root root 2531 Jan 31 17:54 SchemaScript.groovy
-rw-r--r-- 1 root root 5379 Jan 31 17:54 SearchScript.groovy
-rw-r--r-- 1 root root 1372 Jan 31 17:54 TestScript.groovy

2021-07-31 09:31 grouper connector enhancements

2020-06-11 21:36 db table connector how-to slide deck

~/Documents/dbTableConnConfig.odp

2020-05-06 09:27 Jason Everling midPoint samples

2020-05-06 09:22 handling LDAP object classes in connector config

2020-05-05 09:40 ConnID 2.0 delayed at least to end of year

Note
 Evolveum and Apache Syncope are the big contributors

2020-02-12 09:29 LDAP Connector Config How-to

I’d like to start drafting a how-to guide to configuration of the ConnID LDAP connector. Do you have time to help with that? I imagine mainly I’d draft a section and then go over it with you to correct and/or add detail.
If so, I’ll try to bring a couple paragraphs to the SI meeting and we could review on the call

2020-02-03 19:30 ConnID Futures discussion

ConnId 2.0.0

This page contains a notes regarding our current thinking about ConnId 2.

ConnId 2 should be a “next generation” of ConnId platform. It should support operations and use-cases that are not possible with ConnId 1 - and it is not feasible to implement them and keep connector compatibility at the same time. Therefore ConnId 2 can break the compatibility (in a reasonable way, see below). Evolution

ConnId2 should be an evolution of ConnId1. We are not all the rewriting the code.

Rewrite would be attractive. We can get rid of CDDL, we can clean up a lot of things, modernize from the ground up. But that would be a huge task. We do not have human resources (and funding) to do that.

Therefore we rather go for evolution. Rough plan:

Analyse and design. Prepare the list of all incompatible changes that we need to do in ConnId2.
Modify the interfaces by applying all the incompatible changes. Modify the implementation as well (if possible). Do not add new functionality yet. Just change the interfaces in such a way that any changes that follows can be made in a compatible manner.
Stabilize the functionality.
Release ConnId 2.0.0.
Add more functionality. In small steps. With compatible interface changes.
Goto step 5.

Schema

There are two issues here. Complex attributes

We want to support complex attributes. E.g. attribute foo contains a map with keys bar1 and bar2, string values, int values and so on.

We have two options:

Adopt some kind of schema language (JSON schema, XSD, …)
Extend current schema capabilities of ConnId (e.g. AttributeInfo class)

Use of schema language is quite attractive. But there are potential obstacles. Which language we would use? Can we implement all the features of that language? Schema languages are usually tightly bound to representation format, e.g. JSON schema is bound to JSON. But our data are not JSON, they are Java primitive types spiced with collections. Will JSON schema fit? What features of JSON schema we won’t be able to support? If we can support only a fraction of JSON schema capabilities, we might end up looking like a mouse in a elephant’s skin.

It looks like extending current ConnId schema seems to be much easier. This will naturally limit the capabilities to the set that we need. This will also mean that porting of ConnId1 connectors should be easy. In fact, pretty much the same schema code could be used and only some minor adjustments should be needed.

Therefore the decision for now is to extend native ConnId schema. But that decision may change if we run into unforeseen difficulties.

Attribute values should be easy to do. We will just use Map/List and primitive data types. Identifiers

We want cleaner handling of identifiers, especially better handling of NAME and UID atttibutes.

Firstly NAME and UID usually stand for some native attributes, such as DN and entryUUID. Masking those as NAME and UID makes troubleshooting difficult. We want to use native names instead.

Secondly, there are resources that only have one of the (e.g. only have mutable username). There are resources that have both, but NAME is not unique. And so on.

Thirdly, there are resources that will benefit if both identifiers are passed to operations. Such as AD resource in case of AD forests. In that case GUID (UID) is primary identifier. But GUID is not reveal the server where an object is stored. Passing both GUID and DN can make operations much more efficient.

The decision for ConnId2 is to make the use of NAME and UID optional. They will still be there, they may even be present in the schema by default, but they will no longer be required.

The schema will have new capability to define identifiers. E.g. The AttributeInfo class for the dn attribute may specify that this is a secondary identifier.

Operations should be modified to allow passign more than one identifier. E.g. the Uid parameter in the methods should be replaced with something like Identification that will be a container for identifiers. Or there may be additional parameters for operation. Specific implementation is still TBD. Remote Connector Servers

Connector servers are currently quite under-maintained. What we will do about that?

Keep Java connector server. This one is quite useful. But try to modernize it. Try to improve logging, error handling, packaging, etc.
Drop .NET connector server and all other .NET parts. We do not really need them and we do not have the manpower to maintain them. Deprecate .NET support in ConnId1, remove it completely in ConnId2.

API/SPI Operations

There is a need for some updates:

Create GetApiOp and GetOp. Hence split search and get operations.
Introduce CountApiOp and CountOp. Useful for GUI.
We have too many update operations. Let’s reduce that to just one set. That might be updateDelta operation.

Result Handlers

Nobody loves result handlers. We do not need them.

Deprecate them in ConnId1, remove them in ConnId2. Asynchronous Operations

This is a difficult problem. So far we have the questions only:

How to support operations that do not return immediately?
E.g. operations that implement "manual provisioning"?
Should it be integral part of operations? e.g. operation that started as sync can end up as async?
Should this be based on polling? Donating thread to connector? Shared queue?
What about REST service endpoints and message queues?

We need to get back to drawing board and think about it. Let’s discuss that later (approx. summer 2020?) Misc

Misc improvements:

Clarify definition of runAsUser, maybe rework the parameters to properly use identifiers
Improve the documentation

Testing

How the testing framework really works? We have some idea, but nobody has a complete knowledge. The testing should probably be improved as we plan to do more changes now.

How to include connector server in the tests?

This has to be discussed later. Low Priority (out of scope)

Those things are out of scope. We will not handle those in our initial attempts to create ConnId2. Of course, them may be added later during ConnId2 lifetime.

Capabilities
Versioning
Error handling
Synchronization improvements
Service accounts
Transactions
Entitlements

Compatibility and Migration

ConnId2 will not be compatible with ConnId1. We are deliberately dropping compatibility to avoid accumulating more technical debt.

However, we still need two things:

Reasonable way to port ConnId1 connectors to ConnId2 framework. The porting must not require a complete rewrite of the connector. The porting should be a matter of a couple of text replace operations, some minor adjustments and so on. Really old connectors may need some of their methods reworked. But overall, we want to keep porting overhead quite small, mostly a matter of few mandays or work.
We absolutely need a way how to run ConnId1 and ConnId2 connectors together. ConnId1 connectors will be there for a very long time. Even though ConnId1 will not evolve any more, we should be able to run those legacy connectors.
There a simple and elegant solution: change the package name of ConnId2 framework. We can use connid.net. Therefore the ConnId1 and ConnId2 platforms should be independent from Java runtime point of view.

Plan

There is no specific plan yet. There is no commitment yet. We are just exploring possibilities. We are making sure that we are aiming for the same goal, that we can agree on the approach and that our development efforts will converge.

So far, nobody is making any specific commitment about dates/resources and nobody is expecting any commitment. Origin

This text originated from on-line discussion (call) in February 2020.

See connid-dev mailing list archives.

2020-02-03 19:32 references and links