Perform upstream merge 2020-06

[iay]

It's time we brought this repository up to date with the UK Federation upstream tooling, to bring in new functionality that we want to use.

[iay]

Updated the upstream branch from the current GitHub ukf-meta/master branch.

[iay]

Complete list of upstream changes since last merge.

Key:

• ?: don't know yet
• -: change does not affect InCommon tooling
• M: does not affect InCommon tooling, but changes a common file (needs Merge)
• *: Affects (or probably affects) InCommon tooling

M 41514e4 Add support for development signing using SoftHSMv2 on macOS
M bcd743e Add support for development signing using Yubikey 4 on macOS
M 00c9607 Allow for OpenSC variants on macOS
M af6ab08 Add proof-of-concept discovery feed generation
* af68a77 Add Jakarta JSON runtime
* 0c68e3b Update ukf-mda to 0.9.10 to include DiscoFeedCollectionSerializer
- a7c700d Remove prod-old config for ACOS5 card on Windows
- 779512c Add cacheDuration of 6 hours to export preview aggregate
* 041a185 Warn and remove long imported mdui:Logo elements; initial threshold 50000
* 876b708 Add a check for long mdui:Logo elements; initial threshold of 40000
- f7ae6ed Increase validity of published metadata aggregates to 21 days - ukf/ukf-meta#218
- bcc9e30 Rollback change to validity to 14 days due to issue with MDQ
- 706a1b8 Normalise attributes in remd namespace for fallback aggregate
* 1bcb70e Increase validity of published metadata to 21 days - ukf/ukf-meta#217
- 601784a Add CN (CARSI) to monthly chart registrar names
* e8583f0 Stop using Mail.dodgyAddress from sdss-xalan-md project
* a88a38f Stop using TextUtils from sdss-xalan-md project
- df25447 Check that entities don't have both DisableFlow and EnableFlow constraints
- dfe97c9 Apply flow constraints in the main tooling flows
M 2927258 Update to ukf-mda 0.9.9
- c07818c Add FlowEnable and FlowDisable to ukfedlabel schema
- 9ce275f Update charting statistics to track Shibboleth 4 IdPs
- 86b6f9e Update statistics page to track Shibboleth 4 now that the IdP beta is out
* 85407ee Move check_hasreginfo to eduGAIN import policy
* 42bf733 Update list of registrar IDs from eduGAIN technical page
* 56b7dd7 Correct registrar ID for RoEduNetID (RO)
- 6bfeaa9 Normalise attributes in remd namespace for production aggregates
- 68087c1 Normalise attributes in remd namespace for test aggregate
- 00ed781 Use same timstamp for statistics as for aggregates
- 416112e Correct typo
* bbe7bbc Spell validUntil consistently
* 8ca2449 Add the ability to simulate running at 00:00:00 today
* 830c030 Use canonical eduGAIN aggregate endpoints
* aa2b28f Enable use of different eduGAIN aggregates
* f82b969 Remove handling of old-style eduGAIN test aggregate
* 40e2162 Stabilise order of eduGAIN ingress errors
- 57c8417 Add cacheDuration of 6 hours to test aggregate
- 8a1a6df Permit appropriate regular expression scopes from eduGAIN
* 99b2d49 Correct registrar ID for Tuakiri (NZ)
* d5f4917 Stop using sdss-xalan-md project
* f357f0b Stop using sdss-xalan-md project
* cec199d Correct registration authority URI for RASH (Albania)
? a1a2be6 Apply scope policy to eduGAIN ingress flows
M f35762f Import inc-mda artifact
* 948fd71 Allow SAML subject identifier attributes as RequestedAttribute elements
M d041e4f Add ability to block a set of entities from the fallback aggregate.
- dcd6955 Stop reporting certificate expiry for self-signed trust fabric certificates.
- 6a7c457 Simplify rules for export aggregate
- 3582ceb Correct a comment
- 7b01144 Inline importCommonTail (only used once)
* 31b324a Update amount of memory available for Java heap
* 8469213 Correct RA string for LIAF
* 666d6bc Fix reference to RO registrar key
* 883319f Update eduGAIN Registration Authorities
M 254b995 Correct git clean invocation
M da65773 Add git clean ant target for all repositories
- 0efa365 Add ukf-test-meta-config to preprod config
* 425da23 Edit common-beans.xml to update Registration Authority URLs from eduGAIN member federations
* 7ac2bc7 Remove discussion of repository variants
M 9767efb Include new tooling config repo in git operations
M 66273e1 Remove slacktee configuration; update build config to reference new location
* 23c38c1 Migrate to new eduGAIN aggregate location and signing certificate
* d87a481 Add new eduGAIN signing certificate and README
- 79da53c Remove check_future_9 which warns when SP sets WantAssertionsSigned
- 809817d Remove content in check_future_2 pending decision in ukf-meta#174
- 49741c6 Check children of Extensions element against registration policy
M e12b700 Remove the uiinfo.list target and associated XSLT file
M 05096a6 Add compare-members scripts and update build.xml to include
- 0fdd30a Fix stats generation for Wugen

[iay]

@sroddy reports that the current aggregate generation process is not running exactly the incommon-v9 tag, so we will also need to reconcile some commits from another repository before this merge can happen.

[iay]

Interested parties: @nroy @ij @sroddy @dshafer @awu

Feel free to unsubscribe from the issue if I've added you in error. List taken from the latest e-mail thread.

It appears that the per-entity metadata process in production is using the incommon-v9 tag directly. The aggregate production process in production is using something with a couple of new files and an additional blob in build.xml. Proposal is to pull those changes back into this repository before doing the upstream merge.

[iay]

Here's a more human-readable summary of the changes I think this merge will introduce to the InCommon tooling.

• Migrates to a new location for the imported eduGAIN aggregate, and its corresponding public key for validation. This should have no immediate effect, as the old aggregate and certificate are still operational and publish the same contents. Migrating will prevent issues when that stops being the case.
• Several revisions update the table of correspondences between registration authority URIs and the corresponding short codes. This is only likely to be visible in logging output.
• Allows SAML subject identifier attributes as RequestedAttribute elements.
• Adjusts the default amount of heap used by Java invocations (you're almost certainly overriding this in properties files anyway).
• Includes the inc-mda artifact to allow inbound scope checking. Obviously, this is already included in the InCommon tooling.
• The way inbound scope checking is applied (as opposed to the actual rules applied) will need to be carefully checked against the equivalent InCommon coding to make sure they can coexist without interference.
• To allow better debugging of new configurations, a facility has been introduced to fix the timestamps in UKf output. We should make sure this also works with the InCommon tooling.
• Logging output is changed (for the UKf) to be ordered by entityID. We should make sure this works for the InCommon tooling as well.
• Removed handling of an old style of eduGAIN test aggregate. Added the ability to handle different eduGAIN aggregates.
• Further movement towards reducing dependence on Java 8 and Xalan. We should make sure these changes are incorporated into the InCommon tooling.
• Default validUntil interval increased to 21 days for the UKf aggregates. This may need to be overridden to make sure the InCommon output is not affected.
• Added checks for long Logo elements. We should check that this is having the effect we want in the InCommon tooling.
• Added the ability to generate an EDS-format discovery feed. This is something we could potentially configure for the InCommon tooling if having such a thing was desirable (the UKf is in the process of generating this for experimental use, not at present as part of the defined service).

@nroy hopefully none of this sounds scary. If not, I will move to actually doing the merge and checking that nothing appears to have broken as a result.

[iay]

I have made a trial merge from upstream. This went very easily with just two conflicts which were easily resolved. I then had to make a small functional change to preserve behaviour which has moved from Xalan to a Java stage in the upstream.

The result is that:

• In inc.edugain.report output, as expected we see the following removed from the output:

ERROR - Item https://proxy.aai.lifescience-ri.eu/metadata/backend.xml (CZ) was marked with the following Error status messages
     [java] ERROR -     check_reqattr: RequestedAttribute uses unknown name urn:oasis:names:tc:SAML:attribute:subject-id with SAML 2.0 NameFormat (subject-id) 

• In output from inc.generate.import, inc.generate.export and inc.generate.mdq, everything is identical except for:
  • Timestamps (which are the current time and expected to be different... there's now a facility to pin those values, which we can implement if we want)
  • Document IDs (which are random)
  • Signatures (which are dependent on the document IDs and the timestamps)
  • In the imported.xml aggregate, the entity https://proxy.aai.lifescience-ri.eu/metadata/backend.xml is no longer being blocked, and appears as expected.

Everything else seems to be fine. The validity interval is hard-wired in the InCommon configuration, so the upstream changes don't affect that.

We aren't seeing the changes related to long Logo values, but that's expected. It probably needs another issue raised to incorporate that processing so that we remember to include it in the external import policy documentation.

[iay]

Created #13 to carry the Logo issue.

[nroy]

This all looks good to me, thanks Ian

[nroy]

A question: Kevin Morooney, our Vice President for Trust and Identity Services would like us to remove terms like master from code/etc, due to negative social connotations. We're working to switch the Federation Manager CRM application to use the main branch in GitHub instead of master. I don't want to unduly burden your work on this, but if you think something like that might be possible for this project, please let me know.

[iay]

A question: Kevin Morooney, our Vice President for Trust and Identity Services would like us to remove terms like master from code/etc, due to negative social connotations. We're working to switch the Federation Manager CRM application to use the main branch in GitHub instead of master. I don't want to unduly burden your work on this, but if you think something like that might be possible for this project, please let me know.

I've had a general to-do about this in my list for a little while now, as I intend to make this change in all my own repositories. The other places I do client work are also moving in that direction, it's just a matter of "when" and not "whether". To some extent we're also waiting to see whether the community consensus is for main as a replacement (that's my own preference, though).

I haven't actually done this myself yet, but my understanding is that it's not particularly hard. There may be some knock-on effects, but this seems like a good time to start finding out what they are. (The UKf part of the tooling is highly dependent on branch names as it uses git live but that won't affect your use of this repository as far as I can see).

I will write up an issue under the incommon-v10 milestone and look into doing this as part of the release.

[iay]

Done,

• Merge commit is 1f826ba
• Needed to add in explicit certificate-wrapping because upstream has moved that to a Java stage, commit aa621fb

One change I needed to make for this to work was to set something like this in my build.properties:

edugain.dir = ${basedir}/mdx/int_edugain

This is because, in the upstream, two files listing entities have been removed from the repository and placed in another location. This property change points that back to the original location for them.