From 1598fbd7b34bb1e6aa407068c77bff1f634400c5 Mon Sep 17 00:00:00 2001 From: Keith Hazelton Date: Fri, 24 Mar 2023 10:01:19 -0500 Subject: [PATCH] Update connectors.adoc --- connectors.adoc | 571 ------------------------------------------------ 1 file changed, 571 deletions(-) diff --git a/connectors.adoc b/connectors.adoc index ffc108e..fca22fb 100644 --- a/connectors.adoc +++ b/connectors.adoc @@ -115,7 +115,6 @@ COPY ../Workbench/midpoint_server/container_files/mp-home/ /opt/midpoint/var/ *- in the container -* ``` - ├── container_files │   ├── csv │   │   ├── source-hr.csv @@ -245,573 +244,3 @@ demo in browser: ` import task, operation statistics . - -- - - -_2023-02-08 11:37:46 references and links_ - -https://docs.evolveum.com/connectors/connectors/org.identityconnectors.databasetable.DatabaseTableConnector/ + -https://docs.evolveum.com/connectors/resources/databasetable/ + -https://docs.evolveum.com/midpoint/reference/resources/connector-setup/ + - -https://evolveum.com/blog/ + - -- - - -_2022-09-19 13:08 chad redman developing SCIM 2 server_ - -part of the Grouper roadmap for 2.7 is to rewrite the SCIM server. The current implementation uses a 3rd party library written for J2EE, which is why Grouper runs under TomEE and not regular Tomcat. There are a few options for replacement libraries, so this should be a reachable goal. - -If the Grouper SCIM server is rewritten, the endpoints should not change significantly, but the object data is likely to change. The current service expresses objects in ways that differ from the published SCIM RFC's [1][2], and a different solution would adhere more closely to the standards. An example of some ways SCIM in Grouper is non-standard and would change: - -- extensions are wrapped in an "extensions" node (includes group name or subject id, so essential fields) - -- userName is not present in user objects and is required - -- unknown attribute baseUrn - -- inconsistent use of group and subject ids vs. uuids - -- /Schemas endpoint is broken (infinite loop that eventually aborts) - -- no PATCH or BulkRequest support - -Changes would impact integrations already in production, so the Grouper team is looking to hear from current users of the SCIM server. - -Starting a conversation with the current users, as well as users holding back because of current limitations, would also be a good opportunity to make improvements to the system. BulkRequest isn't supported, so large change sets are inefficient. PATCH operations are not currently supported, which means memberships can't be managed through the group object. Instead, multiple calls potentially need to be made to look up uuids for the group, subject, and membership. That illustrates how cumbersome it is to work with uuids for groups and subjects in general, and maybe there is some opportunity to switch to more friendly subject ids and group names as resource keys. - -So, if you are using the SCIM service in Grouper, or want to use a more standard version, please comment or let the Grouper team know, so that the needs can be better known. - -- - - -_2022-09-14 17:37 schema mapping, csv connector template_ - -- - - -_2022-05-19 09:43 utility for prompted user input in CLI scripts (for use in soliciting configuration items and choices)_ - -https://github.com/SBoudrias/Inquirer.js - <- + -https://github.com/mokkabonna/inquirer-autocomplete-prompt - <- + - -- - - -_2022-05-15 17:10 continue work on csv connector_ - -TBD: SoR person to mP user schema mapping utility - -Next resource definition: develop, test, document SIS resource creation using the 100-student csv sample from BennO's mock data sets -/Users/khazelton/opt/non.adoc/sis.csv - -sorid -GivenName -MiddleInitial -Surname -Birthday -EmailAddress -TelephoneCountryCode -TelephoneNumber -NationalID -Occupation -Company - - - -- - - -_2022-05-13 05:50 continue work on csv connector_ - -working example: -/Users/khazelton/opt/non.adoc/source-hr.csv - -- - - - -https://github.com/Evolveum/midpoint-samples/blob/master/samples/evolveum/object-template-user.xml - <- user template + - -A user template may be applied globally by including the following snippet in xref:/midpoint/reference/concepts/system-configuration-object/just after the "logging" element: - -``` - -``` - -System configuration xml; after logging element: -``` - - UserType - - -``` - -That template ref, oid="8098b124-c20c-4965-8adf-e528abedf7a4", points to ../objects/objectTemplates/UserTemplate.xml which assigns the unique name and uid - -``` -uid,firstname,lastname,department,mail,validFrom,validTo -E600001,John R,Smith,HR_SOR,xjsmith@example.com,2018-01-01,9999-12-31 -E600002,Alice,Anderson,HR_SOR,xaanderson@example.com,2016-03-15,9999-12-31 -E600003,Ellen,Johnson,HR_SOR,xejohnson@example.com,2019-10-01,2019-12-31 -E600004,Ron,Vasquez,HR_SOR,xrvasquez@example.com,2019-01-01,2019-10-31 -``` -csv resource def template: ../non.adoc/extCsvResourceA.xml - -Resource on Aktis: 'HR SOR Source' - -next task develop, test, document SIS resource creation from 100-student sample from BennO's mock data sets -/Users/khazelton/opt/non.adoc/sis.csv - -- - - -_2022-05-12 15:59 continue work on csv connector_ - -$MIDPOINT_HOME: - -in the workbench repo: ../Workbench/midpoint_server/container_files/mp-home -in the running comtainer: /opt/midpoint/var - -schema extension xsd's go in $MIDPOINT_HOME/schema - -~/opt/InCommonTAP-Examples-current/Workbench/midpoint_server/container_files/mp-home$ - -``` -tree . -L 2 -. -├── config.xml -├── cs-portal.csv -├── faculty-portal.csv -├── icf-connectors -│ ├── connector-grouper-rest-0.7.jar -│ ├── connector-rest-wordpress-.23-SNAPSHOT.jar -│ ├── connector-sympa-1.0.2-connector.jar -│ └── net.tirasa.connid.bundles.db.scriptedsql-2.2. -..-SNAPSHOT.jar -├── mailing-lists.csv -├── post-initial-objects -│ ├── archetypes -│ ├── bulkActions -│ ├── functionLibraries -│ ├── objectTemplates -│ ├── ordering.txt -│ ├── orgs -│ ├── resources -│ ├── roles -│ ├── securityPolicy -│ ├── systemConfigurations -│ ├── tasks -│ └── users -├── res -│ └── sis-persons -├── schema -│ └── internet2.xsd <- example schema extension file ──────────────────────────────── -├── source-external.csv -└── staff-portal.csv -``` - -internet2.xsd user schema extension defines identifiers for each System of Record: HR_ID, SIS_ID, GUEST_ID, UserID - -] Define 2 attributes SOR_ID and user_ID; each user record should have values for both attributes - Avoids having to extend the schema every time a new SoR is integrated. - -- - - -_2022-04-28 19:42 how-to outline_ - -0) A CSV file with attribute names on the 1st line -1) Bash script that collects info from users and uses that info to populate a fresh instance of a CSV resource definition file template - - --) map from attr1 to uid --) map from attr2 to givenName --) ... - -upload and execute resource definition - -- - - -_2022-04-27 18:44 CSV connector how-to_ - -*from csv file to generated resource def file to import of source into midPoint* - -https://www.evolveum.com/downloads/midpoint/4.1/midpoint-4.1-schemadoc/http---midpoint-evolveum-com-xml-ns-public-common-common-3/object/UserType.html - -userAttr.ods <- midPoint User Attribute Priority Categorization + - -==== CSV Connector Work Plan - -CSV Resource Definition Steps - -Document schema of csv file (see userCsvSchemaMap.ods/sisSor.csv) - -Arrange for periodic publishing of the latest data to a location midpoint processes can reach (scp, sftp,...) -Dump into a db table for validation rules ( - -The heavy task: Compile a table that shows which source attributes map to which midPoint User attributes (for all connectors, not just csv) (See userCsvSchemaMap.ods/csv resource schema handling) - -Extend the schema extension file (internet2.xsd in the Workbench) to include attributes that don’t have an appropriate match in the midPoint User Type. - -The full current midPoint schema for “User”. We have learned that “truth is in the schema docs”: -https://www.evolveum.com/downloads/midpoint/4.1/midpoint-4.1-schemadoc/ (midPoint Schema Doc home page) - -Then click http://midpoint.evolveum.com/xml/ns/public/common/common-3 - -Then click “UserType” - -Starting from a template xml file (TBD), map the information from the schema document into the matching XML elements in the sections on , , , , and (See sisSorResourceDef.xml) - -Next step is to debug Resource Tasks + -Import (and Reconcile) + -Recompute - -TBD + -Build and test Synchronization Task + -Live sync: Work with Ethan - -- - - -_2022-05-13 05:48 references and links_ - -https://github.com/Evolveum/midpoint-samples - <- + -https://github.com/Evolveum/midpoint-samples/tree/master/samples/contrib/bshp - <- Jason Everling, Bishop examples + - -- - - -_2022-04-05 13:59 csv connector how-to_ - -[source,xml] - - - - - - Test CSV: username - - Simple CSV resource that is using single identifier (username) - - - - - c:connectorType - com.evolveum.polygon.connector.csv.CsvConnector - - - - - - - - /opt/midpoint/var/midpoint-username.csv - utf-8 - , - ; - username - password - - - - - - - - - - Default Account - true - ri:AccountObjectClass - - - ri:username - - - $user/name - - - - - ri:firstname - - - $user/givenName - - - - - ri:lastname - - - $user/familyName - - - - - - - - - - - - - - - - - - - - - - - - ri:disabled - false - true - - - - - - - - AccountObjectClass - account - Default - c:UserType - true - false - - - - - - -==== building a csv connector for sis source drawn from BennOs 500k sample user files - -``` -~/opt/InCommonTAP-Examples-current/Workbench/midpoint_server/container_files/mp-home/res/sis-persons -total 24 -drwxr-xr-x 2 kh kh 4096 Jan 31 17:54 . -drwxr-xr-x 3 kh kh 4096 Jan 31 17:54 .. --rw-r--r-- 1 kh kh 2531 Jan 31 17:54 SchemaScript.groovy --rw-r--r-- 1 kh kh 5379 Jan 31 17:54 SearchScript.groovy --rw-r--r-- 1 kh kh 1372 Jan 31 17:54 TestScript.groovy -``` -end up in the midpoint server container: - -``` -ls -la /opt/midpoint/var/res/sis-persons -total 24 -drwxr-xr-x 2 root root 4096 Feb 17 14:32 . -drwxr-xr-x 3 root root 4096 Feb 17 14:32 .. --rw-r--r-- 1 root root 2531 Jan 31 17:54 SchemaScript.groovy --rw-r--r-- 1 root root 5379 Jan 31 17:54 SearchScript.groovy --rw-r--r-- 1 root root 1372 Jan 31 17:54 TestScript.groovy -``` -- - - -_2021-07-31 09:31 grouper connector enhancements_ - -https://docs.google.com/document/d/1-NxAlgFHaA30j0PZEqP98qq9ScY-A93fDGIDdYokJWc/edit - <- requirements + - -- - - -_2020-06-11 21:36 db table connector how-to slide deck_ - -~/Documents/dbTableConnConfig.odp - -- - - -_2020-05-06 09:27 Jason Everling midPoint samples_ - -https://github.com/JasonEverling/midpoint-samples/tree/master/samples/contrib/bshp + -https://github.com/JasonEverling/midpoint-samples - -- - - -_2020-05-06 09:22 handling LDAP object classes in connector config_ - -https://lists.evolveum.com/pipermail/midpoint/2017-December/004269.html <- Jason Everling on course group config + - -- - - -_2020-05-05 09:40 ConnID 2.0 delayed at least to end of year_ - -NOTE: Evolveum and Apache Syncope are the big contributors - -- - - -_2020-02-12 09:29 LDAP Connector Config How-to_ - -https://wiki.evolveum.com/display/midPoint/LDAP+Connector - - I’d like to start drafting a how-to guide to configuration of the ConnID LDAP connector. Do you have time to help with that? I imagine mainly I’d draft a section and then go over it with you to correct and/or add detail. - - If so, I’ll try to bring a couple paragraphs to the SI meeting and we could review on the call - -- - - -_2020-02-03 19:30 ConnID Futures discussion_ - -ConnId 2.0.0 - -This page contains a notes regarding our current thinking about ConnId 2. - -ConnId 2 should be a “next generation” of ConnId platform. It should support operations and use-cases that are not possible with ConnId 1 - and it is not feasible to implement them and keep connector compatibility at the same time. Therefore ConnId 2 can break the compatibility (in a reasonable way, see below). -Evolution - -ConnId2 should be an evolution of ConnId1. We are not all the rewriting the code. - -Rewrite would be attractive. We can get rid of CDDL, we can clean up a lot of things, modernize from the ground up. But that would be a huge task. We do not have human resources (and funding) to do that. - -Therefore we rather go for evolution. Rough plan: - - Analyse and design. Prepare the list of all incompatible changes that we need to do in ConnId2. - - Modify the interfaces by applying all the incompatible changes. Modify the implementation as well (if possible). Do not add new functionality yet. Just change the interfaces in such a way that any changes that follows can be made in a compatible manner. - - Stabilize the functionality. - - Release ConnId 2.0.0. - - Add more functionality. In small steps. With compatible interface changes. - - Goto step 5. - -Schema - -There are two issues here. -Complex attributes - -We want to support complex attributes. E.g. attribute foo contains a map with keys bar1 and bar2, string values, int values and so on. - -We have two options: - - Adopt some kind of schema language (JSON schema, XSD, …) - - Extend current schema capabilities of ConnId (e.g. AttributeInfo class) - -Use of schema language is quite attractive. But there are potential obstacles. Which language we would use? Can we implement all the features of that language? Schema languages are usually tightly bound to representation format, e.g. JSON schema is bound to JSON. But our data are not JSON, they are Java primitive types spiced with collections. Will JSON schema fit? What features of JSON schema we won’t be able to support? If we can support only a fraction of JSON schema capabilities, we might end up looking like a mouse in a elephant’s skin. - -It looks like extending current ConnId schema seems to be much easier. This will naturally limit the capabilities to the set that we need. This will also mean that porting of ConnId1 connectors should be easy. In fact, pretty much the same schema code could be used and only some minor adjustments should be needed. - -Therefore the decision for now is to extend native ConnId schema. But that decision may change if we run into unforeseen difficulties. - -Attribute values should be easy to do. We will just use Map/List and primitive data types. -Identifiers - -We want cleaner handling of identifiers, especially better handling of __NAME__ and __UID__ atttibutes. - -Firstly __NAME__ and __UID__ usually stand for some native attributes, such as DN and entryUUID. Masking those as __NAME__ and __UID__ makes troubleshooting difficult. We want to use native names instead. - -Secondly, there are resources that only have one of the (e.g. only have mutable username). There are resources that have both, but __NAME__ is not unique. And so on. - -Thirdly, there are resources that will benefit if both identifiers are passed to operations. Such as AD resource in case of AD forests. In that case GUID (__UID__) is primary identifier. But GUID is not reveal the server where an object is stored. Passing both GUID and DN can make operations much more efficient. - -The decision for ConnId2 is to make the use of __NAME__ and __UID__ optional. They will still be there, they may even be present in the schema by default, but they will no longer be required. - -The schema will have new capability to define identifiers. E.g. The AttributeInfo class for the dn attribute may specify that this is a secondary identifier. - -Operations should be modified to allow passign more than one identifier. E.g. the Uid parameter in the methods should be replaced with something like Identification that will be a container for identifiers. Or there may be additional parameters for operation. Specific implementation is still TBD. -Remote Connector Servers - -Connector servers are currently quite under-maintained. What we will do about that? - - Keep Java connector server. This one is quite useful. But try to modernize it. Try to improve logging, error handling, packaging, etc. - - Drop .NET connector server and all other .NET parts. We do not really need them and we do not have the manpower to maintain them. Deprecate .NET support in ConnId1, remove it completely in ConnId2. - -API/SPI Operations - -There is a need for some updates: - - Create GetApiOp and GetOp. Hence split search and get operations. - - Introduce CountApiOp and CountOp. Useful for GUI. - - We have too many update operations. Let’s reduce that to just one set. That might be updateDelta operation. - -Result Handlers - -Nobody loves result handlers. We do not need them. - -Deprecate them in ConnId1, remove them in ConnId2. -Asynchronous Operations - -This is a difficult problem. So far we have the questions only: - - How to support operations that do not return immediately? - - E.g. operations that implement "manual provisioning"? - - Should it be integral part of operations? e.g. operation that started as sync can end up as async? - - Should this be based on polling? Donating thread to connector? Shared queue? - - What about REST service endpoints and message queues? - -We need to get back to drawing board and think about it. Let’s discuss that later (approx. summer 2020?) -Misc - -Misc improvements: - - Clarify definition of runAsUser, maybe rework the parameters to properly use identifiers - - Improve the documentation - -Testing - -How the testing framework really works? We have some idea, but nobody has a complete knowledge. The testing should probably be improved as we plan to do more changes now. - -How to include connector server in the tests? - -This has to be discussed later. -Low Priority (out of scope) - -Those things are out of scope. We will not handle those in our initial attempts to create ConnId2. Of course, them may be added later during ConnId2 lifetime. - - Capabilities - - Versioning - - Error handling - - Synchronization improvements - - Service accounts - - Transactions - - Entitlements - -Compatibility and Migration - -ConnId2 will not be compatible with ConnId1. We are deliberately dropping compatibility to avoid accumulating more technical debt. - -However, we still need two things: - - Reasonable way to port ConnId1 connectors to ConnId2 framework. The porting must not require a complete rewrite of the connector. The porting should be a matter of a couple of text replace operations, some minor adjustments and so on. Really old connectors may need some of their methods reworked. But overall, we want to keep porting overhead quite small, mostly a matter of few mandays or work. - - We absolutely need a way how to run ConnId1 and ConnId2 connectors together. ConnId1 connectors will be there for a very long time. Even though ConnId1 will not evolve any more, we should be able to run those legacy connectors. - There a simple and elegant solution: change the package name of ConnId2 framework. We can use connid.net. Therefore the ConnId1 and ConnId2 platforms should be independent from Java runtime point of view. - -Plan - -There is no specific plan yet. There is no commitment yet. We are just exploring possibilities. We are making sure that we are aiming for the same goal, that we can agree on the approach and that our development efforts will converge. - -So far, nobody is making any specific commitment about dates/resources and nobody is expecting any commitment. -Origin - -This text originated from on-line discussion (call) in February 2020. - -See connid-dev mailing list archives. - -- - - -_2020-02-03 19:32 references and links_ - -https://evolveum.com/blog/ - <- +