Upstream merge for v11

[iay]

For v11, we should perform another merge to bring in changes from the upstream UK federation tooling.

[iay]

Complete list of upstream changes since last merge to 2021-03-31.

Key:

• ?: don't know yet
• -: change does not affect InCommon tooling
• M: does not affect InCommon tooling, but changes a common file (needs Merge)
• *: Affects (or probably affects) InCommon tooling

Commits in reverse order:

• - 3baa959 Update mdx/uk/README.md with correct date of status change
• - 7e763ea Add EncryptionMethod elements with AES128-CBC algorithm to production
• - d1eeb8b Permit mil.no as a non-regex scope on import from eduGAIN
  • A similar change has already been made to the InCommon tooling; this specific commit
    only affects the UKf tooling.
• M 2de4dcf Add ant target to generate list of IdPs asserting User Accountability
• - 65827e6 Define and use WugenSelector in the WugenPipeline
• - 0f18ef0 Show PrivacyStatementURL in list of SPs asserting R&S entity category
• M 548978b Generate and publish list of SPs which assert R&S entity category
• - f8bc440 Check for explicit CRs in import.metadata
• M 4d297e1 Remove interaction with CDI
• M d35bc08 Experiment with checking file checksum of remote files
• - b199617 Add UK-specific check for IdPs suppporting REFEDS R&S entity category
• - 6efc0a4 Add UK-specific check for SPs asserting REFEDS R&S entity category
• - b1f468a Update utilities/contacts-from-sf.sh due to changes in the salesforce report
• M 91c978a Check for duplicate xml:lang attributes in saml metadata
  • This adds a new rule checker and incorporates it into the CHECK_std group. I don't believe this is used in the InCommon tooling at present as we pulled out a separate enumeration of checking policy and decomposed that group. However, we may want to consider incorporating it in the InCommon policy; it's probably worth discussing. The check was added to avoid a bug in the Shibboleth 4.0.0/4.0.1 IdP. That bug was addressed in 4.1.0.
• M fdebe00 Change É into HTML character entity reference in generate.html.orgnamescope
• - b30d65d Generate statistics for SPs with RequestedAttribute elements
• M d8d058d Add registrationAuthority URL for CN-CSTCLOUD federation
• M 97b634e Re-apply "Generate a new aggregate for Wugen"
• M 0aa216a Fix typo in log message
• - c401f4c Update wugen log locations
• M 7249757 Revert "Generate a new aggregate for Wugen"
• M eff6589 Generate a new aggregate for Wugen
• - 9832a83 Discard inappropriate metadata elements during entity registration
• - c98bafa Update log locations to sync wugen logs from
• M d3e9625 Add ant target to push generated HTML to website
• M a0d5a1b Add ant target to generate HTML table of IdPs and scopes
• * fef2b65 Update Guava to bring public suffix list up to date
• - 0f01954 Add XSL for identifying SPs which don't list AES128-CBC in metadata
• - 8cf1007 Add EncryptionMethod elements with AES128-CBC algorithm where appropriate
• * 92077b4 Make compatible with Java 11
  • One of the ongoing themes on the UKf side is to move forward from Java 8 to Java 11; this change enables most things to run under either version.
• - 6f03868 Add cacheDuration to fallback aggregate
• - d9e6189 Indent check_embedded.pl consistently
• - 59e3403 Update check_embedded.pl to deal with output from OpenSSL 1.1.1g
• - 4c3110f Strip comments during metadata import
• - 53c119d move json files from /tmp to production directory
• - fdbc76b gzip json discofeed files in temporary directory
• M 9f0b848 scp JSON files to temporary directory in metadata servers
• M 3a5e838 Remove trailing slash from specification of githook directory
• M 9724323 Move post-receive githook to production location
• - 51bb5a0 Hoist "alg" namespace in preview aggregates
• M 0470d58 Fix typo when calling sshexec
• M 4a44c93 Change permissions on post-receive githook
• M 48fbe76 Add githook and an ant target to push it to metadata servers
• M bf312b2 Update registrationAuthority URL for Oman KID
• - eaaa2d0 Add cacheDuration to production aggregates
• - f8ed8f9 Parameterise cacheDuration in generated metadata

[iay]

New commits in upstream since last check, for review:

• * 47d6766 Migrate to new (2022-08-03) eduGAIN signing key
• - 170af39 remove UKFederationMember label from fallback and wayf aggregates
• - 2fff433 Remove CDS stats from stats-generate.sh
• - 7d1e24e Remove entity-level copy of scopes from wayf aggregate
• - c797e00 Remove entity-level copy of scopes from fallback aggregate
• - e38cc3b Change é and ü into HTML character entity references in generated HTML
• - 69cd56f Update tests of XSL that generates UserAccountability webpage
• - b5afad3 Remove check for UKFederationMember element when generating UserAccountability webpage.
• - 74c79cb Remove entity-level copy of scope from production aggregate
• - 1aaca6f Remove UKFederationMember element from production aggregate
• - 97049ff Impose stable ordering on eduGAIN verify target
• - 2654bde Add WARN conditions to the output of the eduGAIN verify target
• - e5ab456 Check for locally derived known compromised RSA keys
• - 6f2a315 Add find command to explicitly remove old Test SP logfiles from the stats server
• - 35aad9c Add LC_COLLATE env variable to contacts-from-sf.sh ukf-meta#321
• M ed88e56 Improve description of stsats.sync target
• - 3a944a6 Delete logfiles on stats server to align with Test SP privacy policy
• - a533196 Set includingLegacyDisplayNames in discofeeds generator
• * ecdd334 Bring MDQ validUntil in-line with that of metadata aggregate.
• M 53f31d4 Update federation URI for Yetkim federation (Turkey).
• - 107bba7 Amend comment in contacts-from-sf.sh script.
• M e3022d3 Fix target names to include correct samlmd prefix
• M 21b5f58 Add comma between ant targets to fix the build
• M 3d973d3 Adds new targets for generating signed files for MDQ.
• M 14f1819 Update federation URI for Bulgarian federation.
• M 6bc5e50 Add new eduGAIN registrar URIs
• * dbadbb4 Log INFO metadata on entities
• M f5dc916 Deconflict IdP discovery names using IdPDisplayNameDuplicateAvoidingStage
• - e27f98a Update ukf-mda to 0.9.10 to include IdPDisplayNameDuplicateAvoidingStage
• - b508259 Add validation for SAML subject identifier entity attribute
• - 842892f Hoist "alg" namespace in fallback aggregate
• - f8069bb Remove status dates in UKf maturity documentation
• - aa82f6c Hoist "alg" namespace in production aggregates
• * eeb8dff Switch to new eduGAIN signing certificate
• M 68956dc Add ant targets to generate and push member list for website
• - 926dd55 Add build/locations_noports.txt to .gitignore
• - ac84eb0 Calculate percentage of SPs which state support for GCM

[iay]

Combined complete list of upstream changes since last merge to 2022-04-07.

Key:

• ?: don't know yet
• -: change does not affect InCommon tooling
• M: does not affect InCommon tooling, but changes a common file (needs Merge)
• *: Affects (or probably affects) InCommon tooling

Commits in reverse order:

• * 47d6766 Migrate to new (2022-08-03) eduGAIN signing key
• - 170af39 remove UKFederationMember label from fallback and wayf aggregates
• - 2fff433 Remove CDS stats from stats-generate.sh
• - 7d1e24e Remove entity-level copy of scopes from wayf aggregate
• - c797e00 Remove entity-level copy of scopes from fallback aggregate
• - e38cc3b Change é and ü into HTML character entity references in generated HTML
• - 69cd56f Update tests of XSL that generates UserAccountability webpage
• - b5afad3 Remove check for UKFederationMember element when generating UserAccountability webpage.
• - 74c79cb Remove entity-level copy of scope from production aggregate
• - 1aaca6f Remove UKFederationMember element from production aggregate
• - 97049ff Impose stable ordering on eduGAIN verify target
• - 2654bde Add WARN conditions to the output of the eduGAIN verify target
• - e5ab456 Check for locally derived known compromised RSA keys
• - 6f2a315 Add find command to explicitly remove old Test SP logfiles from the stats server
• - 35aad9c Add LC_COLLATE env variable to contacts-from-sf.sh ukf-meta#321
• M ed88e56 Improve description of stsats.sync target
• - 3a944a6 Delete logfiles on stats server to align with Test SP privacy policy
• - a533196 Set includingLegacyDisplayNames in discofeeds generator
• * ecdd334 Bring MDQ validUntil in-line with that of metadata aggregate.
• M 53f31d4 Update federation URI for Yetkim federation (Turkey).
• - 107bba7 Amend comment in contacts-from-sf.sh script.
• M e3022d3 Fix target names to include correct samlmd prefix
• M 21b5f58 Add comma between ant targets to fix the build
• M 3d973d3 Adds new targets for generating signed files for MDQ.
• M 14f1819 Update federation URI for Bulgarian federation.
• M 6bc5e50 Add new eduGAIN registrar URIs
• * dbadbb4 Log INFO metadata on entities
• M f5dc916 Deconflict IdP discovery names using IdPDisplayNameDuplicateAvoidingStage
• - e27f98a Update ukf-mda to 0.9.10 to include IdPDisplayNameDuplicateAvoidingStage
• - b508259 Add validation for SAML subject identifier entity attribute
• - 842892f Hoist "alg" namespace in fallback aggregate
• - f8069bb Remove status dates in UKf maturity documentation
• - aa82f6c Hoist "alg" namespace in production aggregates
• * eeb8dff Switch to new eduGAIN signing certificate
• M 68956dc Add ant targets to generate and push member list for website
• - 926dd55 Add build/locations_noports.txt to .gitignore
• - ac84eb0 Calculate percentage of SPs which state support for GCM
• - 3baa959 Update mdx/uk/README.md with correct date of status change
• - 7e763ea Add EncryptionMethod elements with AES128-CBC algorithm to production
• - d1eeb8b Permit mil.no as a non-regex scope on import from eduGAIN
  • A similar change has already been made to the InCommon tooling; this specific commit
    only affects the UKf tooling.
• M 2de4dcf Add ant target to generate list of IdPs asserting User Accountability
• - 65827e6 Define and use WugenSelector in the WugenPipeline
• - 0f18ef0 Show PrivacyStatementURL in list of SPs asserting R&S entity category
• M 548978b Generate and publish list of SPs which assert R&S entity category
• - f8bc440 Check for explicit CRs in import.metadata
• M 4d297e1 Remove interaction with CDI
• M d35bc08 Experiment with checking file checksum of remote files
• - b199617 Add UK-specific check for IdPs suppporting REFEDS R&S entity category
• - 6efc0a4 Add UK-specific check for SPs asserting REFEDS R&S entity category
• - b1f468a Update utilities/contacts-from-sf.sh due to changes in the salesforce report
• M 91c978a Check for duplicate xml:lang attributes in saml metadata
  • This adds a new rule checker and incorporates it into the CHECK_std group. I don't believe this is used in the InCommon tooling at present as we pulled out a separate enumeration of checking policy and decomposed that group. However, we may want to consider incorporating it in the InCommon policy; it's probably worth discussing. The check was added to avoid a bug in the Shibboleth 4.0.0/4.0.1 IdP. That bug was addressed in 4.1.0.
• M fdebe00 Change É into HTML character entity reference in generate.html.orgnamescope
• - b30d65d Generate statistics for SPs with RequestedAttribute elements
• M d8d058d Add registrationAuthority URL for CN-CSTCLOUD federation
• M 97b634e Re-apply "Generate a new aggregate for Wugen"
• M 0aa216a Fix typo in log message
• - c401f4c Update wugen log locations
• M 7249757 Revert "Generate a new aggregate for Wugen"
• M eff6589 Generate a new aggregate for Wugen
• - 9832a83 Discard inappropriate metadata elements during entity registration
• - c98bafa Update log locations to sync wugen logs from
• M d3e9625 Add ant target to push generated HTML to website
• M a0d5a1b Add ant target to generate HTML table of IdPs and scopes
• * fef2b65 Update Guava to bring public suffix list up to date
  • This is required for the Feide IdP, which now has a .cloud scope as well as the mil.no one.
• - 0f01954 Add XSL for identifying SPs which don't list AES128-CBC in metadata
• - 8cf1007 Add EncryptionMethod elements with AES128-CBC algorithm where appropriate
• * 92077b4 Make compatible with Java 11
  • One of the ongoing themes on the UKf side is to move forward from Java 8 to Java 11; this change enables most things to run under either version.
• - 6f03868 Add cacheDuration to fallback aggregate
• - d9e6189 Indent check_embedded.pl consistently
• - 59e3403 Update check_embedded.pl to deal with output from OpenSSL 1.1.1g
• - 4c3110f Strip comments during metadata import
• - 53c119d move json files from /tmp to production directory
• - fdbc76b gzip json discofeed files in temporary directory
• M 9f0b848 scp JSON files to temporary directory in metadata servers
• M 3a5e838 Remove trailing slash from specification of githook directory
• M 9724323 Move post-receive githook to production location
• - 51bb5a0 Hoist "alg" namespace in preview aggregates
• M 0470d58 Fix typo when calling sshexec
• M 4a44c93 Change permissions on post-receive githook
• M 48fbe76 Add githook and an ant target to push it to metadata servers
• M bf312b2 Update registrationAuthority URL for Oman KID
• - eaaa2d0 Add cacheDuration to production aggregates
• - f8ed8f9 Parameterise cacheDuration in generated metadata

[iay]

Completed as commit b56e8ee.