Pipelines (Org Identity -> CO Person)

About Pipelines

Pipelines connect Organizational Identities, typically created from Organizational Identity Sources, to CO Person Records. Pipelines can be used to automatically enroll, update, and expire CO Person records linked to external sources. Pipelines are available in COmanage Registry v2.0.0 and later.

Pipeline Flow

  • The External Data Source holds person related records. This is typically a SQL or LDAP database, a flat file, an API, or another similar repository.
  • The Organizational identity Source is a configured Organizational Identity Source plugin, typically with a Sync Mode configured. It obtains information from the External Data Source and converts it to Organizational Identity Format.
  • The Organizational Identity Source Group Mapping is a configuration, attached to the Organizational Identity Source configuration, that maps attributes from the External Data Source into candidate CO Group Memberships.
  • The Organizational Identity Source Record is an artifact created when an Organizational Identity is instantiated from an Organizational Identity Source. It is a copy of the record of the External Data Source, linked to the Organizational Identity that was created from it.
  • The Pipeline takes the Organizational Identity record (including any candidate CO Group Memberships), and syncs them to the operational CO Person, CO Person Role, and CO Group Membership records. As part of this process, the Pipeline may attempt to instantiate a match process to determine if a new Organizational Identity matches an existing CO Person record in some way.

The use of Organizational Identity Sources is not required in order to use a Pipeline, but other usage scenarios may not be fully implemented yet.

Configuring Pipelines

Match Strategies

Match Strategies are used to determine if an Organizational Identity should be connected to an existing CO Person. The following Match Strategies are supported:

  • Email Address: The Pipeline looks for an existing CO Person record with an Email Address (of a specified type) that matches one attached to the Organizational Identity. The Email Address need not be verified, so be careful about matching on self-asserted email addresses.
  • Identifier: The Pipeline looks for an existing CO Person record with an Identifier (of a specified type) that matches one attached to the Organizational Identity.
  • External: Call out to an external matching service. For more information, see Integrating With ID Match.

Remember, while the source of attributes for searching is the Organizational Identity record provided to the Pipeline, the search target of Match Strategies are existing CO Person attributes, not Organizational Identity attributes.

If no existing CO Person is matched, then the Pipeline will create a new CO Person record.

If more than one candidate CO Person is found, an error is thrown.

Pipeline Match Strategies are unrelated to Enrollment Flow Identity Matching.

Sync Strategies

Sync Strategies are used to determine when a CO Person record should be created or updated by a Pipeline, and whether an associated CO Person Role record should also be created/updated.

  • Sync on Add/Update/Delete: These setting control when an Organizational Identity is processed using a Pipeline.
  • Create CO Person Role Record: If checked, when the Pipeline executes it will create a CO Person Role record, not just a CO Person record. This is useful to (eg) automatically add someone to a COU based on their Organizational Identity Source.
  • Sync to COU: If Create CO Person Role Record is set, this setting defines which COU the new Role Record will be placed into.
  • CO Person Role Affiliation: If Create CO Person Role Record is set, this setting defines the affiliation given to the new Role Record. If not set, the affiliation of the source Org Identity will be used.
  • Replace Record in COU: If the CO Person has an existing CO Person Role record in the specified COU, that Role Record will be expired. This is useful (eg) to expire a provisional record created before an authoritative record is received. Only executed on Add operations, not on Updates.
  • Role Status on Delete: If Create CO Person Role Record is set and the Organizational Identity Source record is deleted (no longer valid), the corresponding CO Person Role will be set to the specified status.

When a Sync Strategy executes, it copies all data provided by the Organizational Identity Source and any defined Group Mappings.

Sync Attributes

The following attributes are supported for sync from an Org Identity:

  • Address
  • EmailAddress (verified status is honored)
  • Identifier
  • Name (primary name flags are cleared)
  • TelephoneNumber

When a CO Person Role is created, the following attributes are supported for sync from the Org Identity:

  • Affiliation
  • O (organization)
  • OU (department)
  • Title
  • Valid From
  • Valid Through

Connecting Pipelines

Pipelines can be connected to various contexts:

  1. Enrollment Flows
  2. Organizational Identity Sources
  3. To the CO itself (for Default Registry Enrollment)

This is also the order of preference. That is, if an Organizational Identity is created from an Enrollment Flow, and that Enrollment Flow also queries an Organizational Identity Source, and both the Enrollment Flow and Organizational Identity Source are connected to Pipelines, the Organizational Identity will be processed via the Pipeline connected to the Enrollment Flow.

Except when connected to an Enrollment Flow, when a Pipeline creates a new CO Person record, Identifier Assignment is triggered, and when a Pipeline creates or updates a CO Person or CO Person Role record, provisioning is triggered.

Pipelines are executed according to the current configuration, so it is possible for an Organizational Identity to be processed by a different Pipeline than the one it was originally attached to.

Enrollment Flows

Pipelines attached to Enrollment Flows are not currently supported (CO-1380).

Note that Pipelines can be implicitly attached to Enrollment Flows via a second mechanism, Enrollment Sources. If an Organizational Identity is created from an Enrollment Source attached to an Enrollment Flow, and if a Pipeline is attached to the Organizational Identity Source that the Enrollment Source is configured to use, then that Pipeline will be executed (not any Pipeline directly attached to the Enrollment Flow).

Organizational Identity Source Sync

Pipelines can be executed during Organizational Identity Source sync processes, resulting in add, update or deletes being processed.

Default Enrollment

Pipeline execution during Default Registry Enrollment is not currently supported (CO-1381).

Manually Rerunning a Pipeline

An Organizational Identity can be reprocessed through a Pipeline by viewing the appropriate Org Identity (People » Organizational Identities) and clicking Rerun Pipeline. The Pipeline run will be selected in the same order of preference as defined above.

When used with Organizational Identity Sources, rerunning a Pipeline manually will not correctly recalculate Group Memberships, since the source record is not available. Resyncing the Org Identity from Source will work correctly.

Considerations When Sources Are Connected to Pipelines

If an Enrollment Source is connected to a Pipeline, Registry will attempt to determine the correct behavior with regard to linking CO Person records.

If the Enrollment Flow is configured in such a way that a CO Person is identified prior to the Enrollment Source being executed (for example if Identity Matching is set to Select), then the Pipeline will use that CO Person instead of creating a new one.

Otherwise, after the Pipeline runs (but before any additional attributes are collected), if the Pipeline created a CO Person that person will be attached to the Petition. In such a configuration, the Enrollment Flow should not collect any Organizational Identity or CO Person attributes, otherwise disconnected identities may be created. (In particular, do not request an Official CO Person Name, as the record will end up with two Primary Names.)

Linking Organizational Identities and COPerson objects

Automatic linking

Through a pipeline

needs to be described…

Creating CO Group Mappings through a Pipeline

Organizational Identity Sources can generate CO Group Memberships via Group Mappings, when the relevant OIS Plugin implements the appropriate interfaces. However, since group memberships attach to a CO Person and not an Organizational Identity, for this to be useful the OIS must typically be attached to a Pipeline, which will then create CO Group Memberships attached to the relevant CO Person record. For OIS Plugins that support this feature, the steps to enable it are:

  1. Make sure the group(s) you want to add memberships to already exists. Automatic groups (such as members groups) cannot be used here.
  2. View the OIS configuration (ie: use the Edit button, not the Configure button), and click Configure Group Mapping. a. NOTE: In Registry versions prior to v3.1.0, the Configure Group Mapping button is available on the index page of Organizational Identity Sources. b. The OIS must be connected to a Pipeline for CO Group Memberships to be assigned.
  3. Add one or more mappings. a. Attribute: The attribute found in the Organizational Identity Source. b. Target Group: The group for which a membership will be created, when the Org Identity has an Attribute matching the specified Comparison and Pattern.
  4. At this point, if you search the Organizational Identity Source, any records matching the defined mappings will also show what CO Group memberships would be assigned if a CO Person record were created from or attached to this source record. However, an action triggering the Pipeline (as described above) must take place. (warning) This means group memberships for existing records will not be assigned until there is a manual sync or a change in the source record.

If a CO Person already has a CO Group Membership (either manually created or from another Organizational Identity Source), a new membership will not be created.

Manually rerunning a Pipeline will not correctly recalculate group memberships, as the original source record is not available for processing. Completely resyncing the Org Identity from Source (which will in turn rerun the Pipeline) will work correctly.


Copyright (C) 2018-2020 University Corporation for Advanced Internet Development (Internet2) - All Rights Reserved