Skip to content

Commit

Permalink
Add files via upload
Browse files Browse the repository at this point in the history
  • Loading branch information
@khazelton
khazelton authored Aug 18, 2023
1 parent 77ce8d0 commit a007bf0
Showing 1 changed file with 298 additions and 0 deletions.
298 changes: 298 additions & 0 deletions test-data-howto1.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,298 @@
=== test-data-how-to.adoc
- - -
_2023-08-16 18:10:25 success importing a small sample of the csv SIS file_

*- CSV connector How-to: Bring a new resource into a Worbench instance -* +
*- (example modified from The Book, Chapter 8, HR section) -*

- Start with the small csv resource definition from Chapter 8 of the Book

- hr.csv contents:
```
"empno","firstname","lastname"
"001","Alice","Anderson"
"002","Bob","Brown"
```

- Place the new comma-delimited hr.csv file in the proper place on the Workbench host

- Replace {Workbench Directory} with the actual path to your Workbench folder

```
{Workbench Directory}/midpoint_server/container_files/mp-home/csv/hr.csv
```

- Add the new .csv data file to the volume bind section of the midpoint_server segment of the docker-compose.yml file

```
...
midpoint_server:
...
volumes:
- midpoint_home:/opt/midpoint/var
...
- type: bind
source: ./midpoint_server/container_files/mp-home/csv/hr.csv
target: /opt/midpoint/csv/hr.csv
...
```

---

*- The following template from Ch. 8 of THE BOOK can be used to create your own simple inbound csv resource definition -*

```
<?xml version="1.0" encoding="UTF-8"?>
<!-- https://docs.evolveum.com/book/practical-identity-management-with-midpoint.html#08-obhject-templates
~
~ Copyright (c) 2010-2019 Evolveum
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<resource oid="03c3ceea-78e2-11e6-954d-dfdfa9ace0cf"
xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"
xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
xmlns:ext="http://midpoint.evolveum.com/xml/ns/story/orgsync/ext"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:icfc="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/connector-schema-3">

<name>HR System, Book ch 8</name>

<description>
HR resource using CSV connector. This is the HR feed (source) resource.
This is a sample used in the "Practical Identity Management with MidPoint"
book, chapter 8.
</description>

<connectorRef type="ConnectorType">
<filter>
<q:equal>
<q:path>c:connectorType</q:path>
<q:value>com.evolveum.polygon.connector.csv.CsvConnector</q:value>
</q:equal>
</filter>
</connectorRef>
```
* In the connectorConfiguration section
** filepath should match the path defined in the docker-compose.yml for the target location of the bind element
** {filename.csv} should be your .csv file name
** uniqueAttribute should identify the field in the csv file that carries the unique identifier for each row

```
<connectorConfiguration>
<!-- Configuration specific for the CSV connector -->
<icfc:configurationProperties
xmlns:icfccsvfile="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/bundle/com.evolveum.polygon.connector-csv/com.evolveum.polygon.connector.csv.CsvConnector">
<icfccsvfile:filePath>/opt/midpoint/csv/{filename.csv}</icfccsvfile:filePath>
<icfccsvfile:encoding>utf-8</icfccsvfile:encoding>
<icfccsvfile:fieldDelimiter>,</icfccsvfile:fieldDelimiter>
<icfccsvfile:multivalueDelimiter>;</icfccsvfile:multivalueDelimiter>
<icfccsvfile:uniqueAttribute>empno</icfccsvfile:uniqueAttribute>
</icfc:configurationProperties>
</connectorConfiguration>
```
- NOTE: The <schema> section that goes here will be added automatically
when the first connection to the resource takes place

- The schema handling section should use the attribute names that match the "element" definitions in the auto-loaded schema section of the resource definition

- Near the top of the schema handling section, the element for the unique attribute should look like this

```
<schemaHandling>

<objectType>
<displayName>Default Account</displayName>
<default>true</default>
<objectClass>ri:AccountObjectClass</objectClass>
<attribute>
<ref>ri:empno</ref>
<displayName>Name (book hr empno)</displayName>
<limitations>
<minOccurs>0</minOccurs>
<access>
<read>true</read>
<add>true</add>
<modify>true</modify>
</access>
</limitations>
<inbound>
<target>
<path>$focus/employeeNumber</path>
</target>
</inbound>
</attribute>
```
- Now the rest of the attribute mappings

```
<attribute>
<ref>ri:firstname</ref>
<displayName>First name</displayName>
<inbound>
<target>
<path>$focus/givenName</path>
</target>
</inbound>
</attribute>
<attribute>
<ref>ri:lastname</ref>
<displayName>Last name</displayName>
<inbound>
<target>
<path>$focus/familyName</path>
</target>
</inbound>
</attribute>
<activation>
<administrativeStatus>
<inbound/>
</administrativeStatus>
</activation>

<credentials>
<password>
<inbound>
<strength>weak</strength>
<expression>
<generate/>
</expression>
</inbound>
</password>
</credentials>
</objectType>

</schemaHandling>

<projection>
<assignmentPolicyEnforcement>none</assignmentPolicyEnforcement>
</projection>


<synchronization>
<objectSynchronization>
<enabled>true</enabled>
```
- The correlation element should look like this:

```
<correlation>
<q:equal>
<q:path>employeeNumber</q:path>
<expression>
<path>$projection/attributes/empno</path>
</expression>
</q:equal>
</correlation>
```

- The rest of the synchronization element follows

```
<reaction>
<situation>linked</situation>
<synchronize>true</synchronize>
</reaction>
<reaction>
<situation>deleted</situation>
<synchronize>true</synchronize>
<action>
<handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/action-3#deleteFocus</handlerUri>
</action>
</reaction>
<reaction>
<situation>unlinked</situation>
<synchronize>true</synchronize>
<action>
<handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/action-3#link</handlerUri>
</action>
</reaction>
<reaction>
<situation>unmatched</situation>
<synchronize>true</synchronize>
<action>
<handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/action-3#addFocus</handlerUri>
</action>
</reaction>
</objectSynchronization>
</synchronization>
</resource>
```
- - -
_2023-08-17 08:44:37 Import 50,000 users from source-sis-full.csv_

*- process for importing large csv files into midPoint -*

Bringing in the full test SIS population (50k) to midPoint Workbench

Save the edited docker-compose.yml with data file binds: +
/csp-tap/InCommonTAP-Examples-0816/Workbench/docker-compose.yml

Save the contents of the ../csv directory

```
$ pwd
/csp-tap/InCommonTAP-Examples/Workbench/midpoint_server/container_files/mp-home/csv

ls -la
-rw-r--r-- 1 csprootuser csprootuser 6826528 Aug 5 22:54 source-hrms.csv
-rw-r--r-- 1 csprootuser csprootuser 8386727 Aug 5 22:54 source-sis.csv
```

*- Wipe all existing Workbench images and files -*

```
docker stop $(docker ps -a -q); docker rm $(docker ps -a -q); docker rmi $(docker images -q) --force; docker volume rm $(docker volume ls -q)
```

Replace downloaded ../csv with saved version

Replace repo version of docker-compose.yml with saved version

*- Build and bring up the fresh Workbench -*
```
docker-compose up --build -d
```

- Create a new (midpoint) project in midPoint Studio and edit the default configuration to point to the new Workbench host

*- import source-sis-full.csv -*

```
*Schema mapping and extension attributes*

CSV MIDPOINT USER
___ _____________
sorid org
given givenName
surname familyName
email emailAddress
ph telephoneNumber
cntry-code cntryCode
unid uid
occup title
dept orgUnit
refid rid

59614 (closed) Last object processed: 641-64-5552 +
8/17/23, 3:22:57 PM - +
8/17/23, 5:05:57 PM (01:42:59.164)
```

- The import task incorporates the synchronization process
---

0 comments on commit a007bf0

Please sign in to comment.