Skip to content

Commit

Permalink
Merge branch 'upstream' for incommon-v10
Browse files Browse the repository at this point in the history
  • Loading branch information
@iay
iay committed Jul 15, 2020
2 parents 793d15d + 41514e4 commit 1f826ba
Show file tree
Hide file tree
Showing 54 changed files with 1,773 additions and 381 deletions.
16 changes: 0 additions & 16 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,22 +25,6 @@ repository.

This is the metadata repository and main toolset for the [UK Access Management Federation for Education and Research](http://ukfederation.org.uk) ("the UK federation").

## Private and Public Repository Variants

There are two variants of the repository:

* The private variant of the repository is used as part of UK federation operations.

* The public, read-only variant of the repository is made available on [GitHub](https://github.com/ukf/ukf-meta). If you're reading this, you're probably accessing this more restricted version. We provide it for the benefit of other federation operators who may be wondering how the UK federation systems operate, perhaps with a view to implementing something similar.

The public repository is kept roughly in synchronisation with the private one using [our `ukf-meta-meta` tools](https://github.com/ukf/ukf-meta-meta). We do this when we have made significant changes to the toolset and we think other federation operators may find the changes of interest.

We exclude a significant amount of material from the public repository. One reason for this is to save space: for example, the private repository retains a copy of every signed metadata aggregate we produced between 2005 and 2016.

The second main category excluded from the public repository is the historic registration database: version-controlled XML documents describing registered entities and the federation membership. We don't include this material because it may include personal information.

Since 2016, we have separated the entity database and aggregate record from the main toolchain repository, but the nature of Git is to never discard anything. We will therefore continue to make this repository publicly available only in redacted form.

## Copyright and License

The contents of this repository are Copyright (C) the named contributors or their
Expand Down
208 changes: 154 additions & 54 deletions build.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@
<property name="validUntil.default.days" value="14"/>

<!-- Specific validUntil duration for aggregates, in days. -->
<property name="validUntil.aggregate.days" value="${validUntil.default.days}"/>
<property name="validUntil.aggregate.days" value="21"/>
<!-- Same value, as an ISO 8601 duration. -->
<property name="validUntil.aggregate.duration" value="P${validUntil.aggregate.days}D"/>

Expand Down Expand Up @@ -146,13 +146,15 @@
<property name="git.repo.project.data" value="ukf-data"/>
<property name="git.repo.project.products" value="ukf-products"/>
<property name="git.repo.project.tooling" value="ukf-meta"/>
<property name="git.repo.project.tooling.config" value="ukf-meta-config"/>

<!--
Build access URLs for each repository.
-->
<property name="git.repo.data.url" value="${git.repo.user}@${git.repo.host}:${git.repo.group}/${git.repo.project.data}"/>
<property name="git.repo.products.url" value="${git.repo.user}@${git.repo.host}:${git.repo.group}/${git.repo.project.products}"/>
<property name="git.repo.tooling.url" value="${git.repo.user}@${git.repo.host}:${git.repo.group}/${git.repo.project.tooling}"/>
<property name="git.repo.tooling.config.url" value="${git.repo.user}@${git.repo.host}:${git.repo.group}/${git.repo.project.tooling.config}"/>

<!--
*********************************************
Expand Down Expand Up @@ -247,6 +249,8 @@
External specialised directories.
-->
<property name="aggregates.dir" value="${shared.ws.dir}/${git.repo.project.products}/aggregates"/>
<property name="blocklists.dir" value="${shared.ws.dir}/${git.repo.project.data}/blocklists"/>
<property name="edugain.dir" value="${shared.ws.dir}/${git.repo.project.data}/edugain"/>
<property name="entities.dir" value="${shared.ws.dir}/${git.repo.project.data}/entities"/>
<property name="members.dir" value="${shared.ws.dir}/${git.repo.project.data}/members"/>
<property name="output.dir" value="${shared.ws.dir}/build"/>
Expand All @@ -272,6 +276,12 @@
<property name="tools.slacktee" value="${tools.dir}/slacktee"/>
<property name="tools.xmlsectool" value="${tools.dir}/xmlsectool-2.0.0"/>

<!--
Location of configuration for externally supplied tool bundles.
-->
<property name="tools.config.dir" value="${shared.ws.dir}/${git.repo.project.tooling.config}/tools"/>
<property name="tools.slacktee.config" value="${tools.config.dir}/slacktee"/>

<!--
Full path to a commonly used temporary file.
-->
Expand Down Expand Up @@ -339,8 +349,16 @@
This is used as the max heap setting for all Java invocations. This amount will
be more than some invocations require, but there's no harm in having a higher
limit for all of them.
Note that this value is often overridden in production contexts (see, for example,
prod.properties) to give the production environment extra headroom. The value is
kept lower here so that we'll get warnings in development before the production
environment is affected.
The overrides for production environments should be updated in step with the
value here whenever this is necessary.
-->
<property name="java.max.memory" value="1280m"/>
<property name="java.max.memory" value="1536m"/>


<!--
Expand Down Expand Up @@ -693,66 +711,129 @@
*********************************
-->

<!--
Full hard reset of all repositories
-->
<target name="git.hardreset.all">
<!--
Full hard reset of all repositories
-->
<target name="git.hardreset.all">
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="checkout"/>
<arg value="master"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="origin/master"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="checkout"/>
<arg value="immediate"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="origin/immediate"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="checkout"/>
<arg value="deferred"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="deferred"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.products}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.products}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling.config}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling.config}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
</exec>
<echo>All branches on all repositories, reset HARD.</echo>
</target>

<!--
Full clean of all repositories
-->
<target name="git.clean.all">
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="checkout"/>
<arg value="master"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="origin/master"/>
<arg value="clean"/>
<arg value="-xdf"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="checkout"/>
<arg value="immediate"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="origin/immediate"/>
<arg value="clean"/>
<arg value="-xdf"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="checkout"/>
<arg value="deferred"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.data}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="deferred"/>
<arg value="clean"/>
<arg value="-xdf"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.products}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.products}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="clean"/>
<arg value="-xdf"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling}" failonerror="true">
<arg value="fetch"/>
<arg value="origin"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling}" failonerror="true">
<arg value="reset"/>
<arg value="--hard"/>
<arg value="clean"/>
<arg value="-xdf"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling.config}" failonerror="true">
<arg value="clean"/>
<arg value="-xdf"/>
</exec>
<echo>All branches on all repositories, reset HARD.</echo>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling.config}" failonerror="true">
<arg value="clean"/>
<arg value="-xdf"/>
</exec>
<echo>All branches on all repositories, cleaned.</echo>
</target>

<!--
Expand Down Expand Up @@ -787,6 +868,9 @@
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling}" failonerror="true">
<arg value="pull"/>
</exec>
<exec executable="${git.executable}" dir="${shared.ws.dir}/${git.repo.project.tooling.config}" failonerror="true">
<arg value="pull"/>
</exec>
</target>

<!--
Expand Down Expand Up @@ -1158,7 +1242,7 @@
<arg value="${utilities.dir}/stats-generate.sh"/>
<arg value="day"/>
</exec>
<SLACK.send conf="${tools.slacktee}/conf/repo-jenkins.conf" colour="good" channel="ukf-stats"
<SLACK.send conf="${tools.slacktee.config}/repo-jenkins.conf" colour="good" channel="ukf-stats"
message="${stats.daily.output}"/>
</target>

Expand Down Expand Up @@ -1451,6 +1535,11 @@
-->
<CHANNEL.do channel="uk" verb="generate"/>

<!--
Generate discovery feeds.
-->
<CHANNEL.do channel="uk" verb="discofeeds"/>

<!--
Post-process mda-generated output files.
-->
Expand Down Expand Up @@ -1487,6 +1576,7 @@
by properties defined in the external properties files, e.g., build.properties.
-->
<property name="mda.java.endorsed.dirs" value="${tools.dir}/endorsed"/>
<property name="mda.blocklists.dir" value="${blocklists.dir}"/>
<property name="mda.edugain.dir" value="${edugain.dir}"/>
<property name="mda.entities.dir" value="${entities.dir}"/>
<property name="mda.mdx.dir" value="${mdx.dir}"/>
Expand Down Expand Up @@ -1651,7 +1741,7 @@
<arg value="${git.repo.project.data}"/>
<arg value="${git.repo.project.products}"/>
</exec>
<SLACK.send conf="${tools.slacktee}/conf/aggr-ant.conf" colour="good" channel="ukf-events-publish"
<SLACK.send conf="${tools.slacktee.config}/aggr-ant.conf" colour="good" channel="ukf-events-publish"
message="${diff.between.publications}"/>
</then>
</if>
Expand Down Expand Up @@ -2252,10 +2342,6 @@
<CHANNEL.import channel="${channel}"/>
</target>

<target name="flow.int_edugain.testImport">
<CHANNEL.do channel="int_edugain" verb="importTest"/>
</target>

<!--
flow.verify.cobweb
Expand Down Expand Up @@ -2300,22 +2386,6 @@
<CHANNEL.do verb="verify.recovered" channel="int_edugain"/>
</target>

<!--
*********************************
*** ***
*** U I I N F O L I S T ***
*** ***
*********************************
-->

<target name="uiinfo.list">
<CHANNEL.do channel="uk" verb="collect"/>
<exec executable="xsltproc" failonerror="true">
<arg value="${build.dir}/list_uiinfo.xsl"/>
<arg value="${mdx.dir}/uk/collected.xml"/>
</exec>
</target>

<!--
*******************************
*** ***
Expand All @@ -2339,6 +2409,7 @@
<fixcrlf file="${output.dir}/${mdaggr.stats}" eol="lf" encoding="UTF-8"/>
</target>


<!--
This variant generates a much simpler file, intended for use when building the
monthly chart pack.
Expand Down Expand Up @@ -2536,6 +2607,35 @@
<echo>Check complete.</echo>
</target>

<!--
compare.members
Runs the compare-members.sh script, comparing Salesforce and UKf member data
-->
<target name="compare.members" depends="extract.members.to.csv">
<echo>Running compare.members</echo>
<exec executable="bash" failonerror="true">
<arg value="${utilities.dir}/compare-members.sh"/>
<arg value="${temp.dir}/ukf-members.csv"/>
</exec>
<delete file="${temp.dir}/ukf-members.csv" quiet="true" verbose="false"/>
</target>

<!--
extract.members.to.csv
Extract the ID and Name elements in UKf members to a CSV file for compare-members.sh to use
-->
<target name="extract.members.to.csv">
<echo>Running extract.members.to.csv</echo>
<exec executable="xsltproc" failonerror="true">
<arg value="--output"/>
<arg value="${temp.dir}/ukf-members.csv"/>
<arg value="${utilities.dir}/members-to-csv.xsl"/>
<arg value="../ukf-data/members/members.xml"/>
</exec>
</target>

<!--
echoproperties
Expand Down
1 change: 1 addition & 0 deletions charting/by_registrar.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"http://cafe.rnp.br": "BR",
"http://www.canarie.ca": "CA",
"http://cofre.reuna.cl/": "CL",
"https://www.carsi.edu.cn": "CN",
"http://www.srce.hr": "HR",
"http://www.eduid.cz/": "CZ",
"https://www.wayf.dk": "DK",
Expand Down
Loading

0 comments on commit 1f826ba

Please sign in to comment.