Upstream merge for incommon-v13

[iay]

For the v13 release we need to incorporate the upstream changes from the UKf repository, including the switch to MDA 0.10.0.

This may need to be done in a couple of tranches as the upstream repository is still in the process of landing a series of changes I think are potentially relevant and worth incorporating.

[iay]

Here's a summary of my current understanding of the complete list of upstream changes since last merge. There are more upstream changes to come, related to removing Xalan from the toolchain.

Key:

• ?: don't know yet
• -: change does not affect InCommon tooling
• M: does not affect InCommon tooling, but changes a common file (needs Merge)
• *: Affects (or probably affects) InCommon tooling

Commits in reverse order:

• * 443cfa3 Adopt AsURLStringValidator
• M 2f01d30 Remove Xalan-specific library
• M 31adc88 Adopt new parent beans
• M 5fc3bf7 Accept changes proposed by deprecation warnings
• - 72c660c Replace GenerateIdStage constructor with property
• M 212653a Track changed location for keylist resources
• M fe66653 Use DuplicateEntityInAggregateCheckingStage instead of ESXLT
• - 1c043f6 Use Java 17 for MDA 0.10.0 CI
• M a8b00df Timeout properties on HTTP client
• M c817f02 Adopt MDA 0.10 and its beans resource
• * 32f7683 Import ukf-mda 0.10.0
• M 9ee9b45 Import ukf-members 2.0.0
• * 84f9b3a Update JAXB implementation to Jakarta EE 9
• * a48c820 Import MDA 0.10.0 GA
• * 45bd937 We no longer need the inc-mda components
  • This is probably going to generate a clash; in the InCommon deployment we'll need a copy of the inc-mda 0.10.0-SNAPSHOT artifact instead.
• * 17a4052 Remove dependencies for MDA 0.9
• * afe4e47 Remove MDA 0.9.2
• M fff282e Remove code for JSON discofeed generation and publication
• M 8c0014e Exbed selected inner beans for testability
• - 87780af Remove dead EXSLT dyn code from statistics stylesheet
• M 2eea3bf Use "curl" instead of <get> in order to use the system proxy settings
• M c9ec827 Use "rsync" instead of <scp> to copy files from Jeeves into Wooster
• M 3211217 Remove statistics-charting.xsl and related code
• M 5b938c0 Fix XML entity character references
  • This can change output in error messages; it's related to the JDK transition
• M 9cf7382 Fix multi predicate negation in check_saml2
• M d3f22ed Fix multi predicate negation in check_mdrpi
• M ca3ad20 Allow configuring the SSH port in "fs.scp.unsigned.files.to.orchestrator"
• - cf6c1c1 Remove charting directory
• M b8ee8f3 Resolve "Update repo.hostname as Jenkins has moved"
• M b0187a4 Fix multi-predicate support in check_saml2int ruleset
• M ff6950e Fix sam2int and shibboleth predicates
• M 28ab883 Fix check_saml2.xsl
• M f2b6adb Fix multi-predicate support in check_mdiop ruleset
• M 18aae5c Fix multi-predicate support in check_hoksso ruleset for binding
• M 33a1a25 Fix negation in check_algsupport ruleset for alg
• M 9bef3e5 Fix multi-predicate support in check_saml1 ruleset
• M b23632a Fix multi-predicate support in the hoksso ruleset
• M 79be026 Fix predicate in InCommon metadata ruleset
• M 5d6c11b Fix contactType value template priority
• M b171fdf Fix EntityAttributes rule predicate
• M 4ded8a6 Fix multi predicate negation in check mdui
• - 954699a Removed rsync of logfiles from Test SP.
• - fd2a178 Remove machinery for generating stats for wugen
• - 0e0376d Delete rsync commands attempting to pull from from non-existent CDS VMs
• - 6ab207a Remove checks from check_future_1
• M a365d86 Fix typo in PublicationInfo namespace prefix
  • This check was wrong, but we didn't see it firing anything after fixing it.
• M 1d87fe6 Remove members.xml from publish.otherfiles target
• M 7746350 Remove unused references to mdxURL
• - a822e4b Back out check_future_5 testing that errorURL is a URL
• - feeb7f8 Entity-level scope refactoring, phase 1
• M a401e70 Move conversionService to common-beans
• - 1fafb5c Remove extraneous XML attributes in import XSLT
• - 4bee46c Change eduGAIN metadata URL in by_registrar.py
• - b45569e Add support for IdPs asserting CoCo v2 entity category support
  • InCommon deployment doesn't use the standard composite check, so this might want to be considered for independent inclusion.
• M ae2b5df Update Xalan to 2.7.3
  • Not sure which if any parts of the InCommon deployment end up using Xalan, but this particular change is probably of no consequence.
• - db36e51 Implement basic continuous integration test
• ? 56b4ec3 Drop signature check on already checked file
  • Not sure how InCommon deployment acquires the eduGAIN aggregate
• - c8a0c44 Resolve "Fix stats page software category for SimpleSAMLphp"
• ? c1b3b26 Make check_entityid_prefix portable
  • Don't know if this check is included
• M c923ea3 Remove Xerces-J; use JDK's XML processor
• M 542785f Improve build.xml samlmd.mdq.sign.test target
• - e93ee79 Add target to select the appropriate signed products during parallel running
• - cbbb256 Update orchestration.xml to mirror changes in build.xml - ukf/ukf-meta#367
• M d01d683 Refactor Steps 4.2 and 5 to facilitate parallel running - ukf/ukf-meta#367
• - e43264e Add targets to transfer files from new signing node
• - 3ecf61c Add metadata checks for conformance with Sirtfi version 2.0 specification
• M 8d05662 Resolve "Remove int_reep channel"
• M 2675301 Purge Eclipse files from the repository
• - 22bc889 Add orchestration for signing nodes in development
• M 07e0ef9 Remove COBWEB channel
• - 9d4b5b6 Remove stages that add default encryption algorithms to output pipelines
• - 559d3b6 Add default encryption algorithms to input pipelines
• - 168ce08 Add default encryption algorithms to SPs in the fallback aggregate
• M 4783d5c Add ukf-mda 0.9.12
• - 6598f82 Use AssuranceCertificationMatcher to simplify SIRTFI handling
• M c6e3611 Remove ukf-mda 0.9.11 jar
• M 80a8da3 Add NG and TH sortcodes for Nigerian and Thai federations
• M 8035900 Update registrationAuthority for CSTCloudFederation to current value
• M 6bb2f38 Allow eduGAIN verify jobs to use new eduGAIN aggregate download mechanism
• M 8d92a8f Separate eduGAIN download and processing
  • Something similar is already present in the InCommon deployment; potential consolidation here.
• - 8bad5fa Remove xmlsectool v2.0.0
• M ad3e4f4 Enable setting HeapDumpOnOutOfMemoryError on MDA executions
• M f8ffb04 Implement reproducible signatures for per-entity metadata
• - 380feb3 Update deployment environment for new HSM
• - e28a201 Add new deployment environment for testing new HSM
• * 7ccb364 Progress "Migrate to xmlsectool v3.0.0"
  • Assuming that InCommon uses xmlsectool at all these days.

[iay]

The flip side of this is our expectations of the InCommon fork. Probably most easily represented by the changes between the previous upstream merge point (the current location of the upstream branch) and the incommon-v12 tag. Ignoring the mdx/incommon directory for clarity (as it's not part of upstream at all) then we're thinking about:

git diff -b upstream..incommon-v12 -- . ':!mdx/incommon' ':!deploy' ':!entities'

The -b is in there because of some variations in indentation in one file (mdx/default_regauth.xsl) which seems to have some other changes mixed in there. The three named directories are exclusive to the InCommon deployment.

Visible changes are:

• A .project file change which will become moot as upstream has since removed that file.
• An added prefix in README.md which will merge just fine.
• In build.xml:
  • A new edugain.dir property that I am a little surprised upstream doesn't have
  • A per-target directory facility in the MDA macro.
  • A bunch of InCommon-specific stuff at the end.
• common-beans.xml has a couple of changes to the standard identifier strategy.
• mdx/int_edugain has a couple of entity list files which are specific to this fork.
• tools/inc-mda has a couple of new files in it. I note that there seems to be some duplication of inc-mda artifacts in my working copy which I need to rationalise.

[iay]

The edugain.dir thing seems to have been an attempt to redirect some code that's looking for a data file into a local directory. It (probably) doesn't work because there is already a definition for that property and it's first-definition-wins. I think (if I recall correctly) that in deployment we needed to put that into the external properties in order to get it to override both.

It may be worth considering removing the redundant and ineffective definition of edugain.dir just so that we don't keep wondering about this.

[iay]

I have pushed an initial trial merge to the v14-trial-merge-1 branch. There was only one conflict to be resolved, and although I'm not certain that I have resolved it correctly it looks plausible; I need to re-check it.

This branch is guaranteed to be non-functional at this stage, as it has a mixture of stuff that's dependent on each version of the MDA: the inc-mda artifact, in particular, needs to be re-released for 0.10 but I believe all of the real work for that was completed months ago as part of the MDA 0.10 release, as one of the main aspects of 0.10 was upstreaming some of the components from there.

There are also going to be a fair number of deprecations that will pop up from the edugain-policy configuration and possibly elsewhere. Although they might work in the current case, they should be addressed before putting anything into production.

[iay]

A second tranche of commits in the upstream repository entirely removes Xalan, leaving the deployment reliant only on the JDK's XSL processor. This requires a small number of additional tweaks but is a fairly simple adaptation and I feel it's a must-have for this merge:

• ffd7ec7 (upstream/master) Extend expression size for check_reqattr
• 5cee5f8 Remove Xalan from the classpath
• a5a56f5 Pre-sort statistics aggregate
• 39a56a4 Hoist xenc namespace
• f5e4cfb Pass the members file location to the statistics transform
• 56bd01c JDK compatibility
• bdfed43 Simplify outboard statistics generation
• 8eb7d06 JDK compatibility
• fa54165 Update xslt which generates website content
• 769899e Do not verify the public copies of cdsall and wayf aggregates as we've stopped publishing them
• b036aaf Remove attic directory for real
• cf7c466 Remove attic directory
• 5bb4036 Remove obsolete sp_mdui_test bean and XSLT