From 99da1b5185643afea27cbbac8b7692004d33cb35 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Wed, 8 Feb 2017 10:50:58 +0000
Subject: [PATCH 01/80] Hoist mdattr and saml prefixes in production aggregate
See ukf/ukf-meta#103 and ukf/ukf-meta#105.
---
mdx/uk/README.md | 13 ++++++-------
mdx/uk/ns_norm_uk.xsl | 2 +-
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/mdx/uk/README.md b/mdx/uk/README.md
index fdff0d19..64b27758 100644
--- a/mdx/uk/README.md
+++ b/mdx/uk/README.md
@@ -53,21 +53,20 @@ when it appeared in the fallback aggregate, which would be too late to take corr
### Test Aggregate vs. Production Aggregate
-Status (2017-01-27):
+Status (2017-02-08):
* the test aggregate implements a _blacklisting_ approach to entity attributes imported from eduGAIN,
while the production aggregate implements the traditional entity attribute _whitelist_.
* the test aggregate no longer implements the "key use" fixup required for pre-1.3.1 Shibboleth SPs.
This adds the `use="signing"` XML attribute to `` elements present in IdP metadata
without a `use` attribute. It is not needed for later releases of the Shibboleth SP.
-* The test aggregate defines the `saml` namespace prefix (used by entity attributes) on the document element
-instead of in each SAML ``.
-* The test aggregate defines the `mdattr` namespace prefix (used by entity attributes) on the document element
-instead of in each `` element.
* The test aggregate normalises the `xenc` namespace to not use a prefix, as it is not very commonly used.
### Fallback Aggregate vs. Production Aggregate
-Status (2017-01-27):
+Status (2017-02-08):
-* these two aggregates are currently identical
+* The production aggregate defines the `saml` namespace prefix (used by entity attributes) on the document element
+instead of in each SAML ``. (2017-02-08)
+* The production aggregate defines the `mdattr` namespace prefix (used by entity attributes) on the document element
+instead of in each `` element. (2017-02-08)
diff --git a/mdx/uk/ns_norm_uk.xsl b/mdx/uk/ns_norm_uk.xsl
index e42943f7..fb3d7896 100644
--- a/mdx/uk/ns_norm_uk.xsl
+++ b/mdx/uk/ns_norm_uk.xsl
@@ -39,7 +39,7 @@
xmlns:ukfedlabel="http://ukfederation.org.uk/2006/11/label"
xmlns:wayf="http://sdss.ac.uk/2006/06/WAYF"
- exclude-result-prefixes="alg md mdattr saml wayf"
+ exclude-result-prefixes="alg md wayf"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
From 658ce0094fc0f91b328c7ff60f327398dd05221e Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Wed, 8 Feb 2017 11:25:10 +0000
Subject: [PATCH 02/80] Separate normalisation transforms for export and export
preview
---
mdx/uk/generate.xml | 15 ++--
mdx/uk/ns_norm_export.xsl | 8 +--
mdx/uk/ns_norm_export_preview.xsl | 111 ++++++++++++++++++++++++++++++
3 files changed, 120 insertions(+), 14 deletions(-)
create mode 100644 mdx/uk/ns_norm_export_preview.xsl
diff --git a/mdx/uk/generate.xml b/mdx/uk/generate.xml
index d4fcbaaa..d8be9833 100644
--- a/mdx/uk/generate.xml
+++ b/mdx/uk/generate.xml
@@ -661,9 +661,6 @@
-
-
@@ -756,8 +753,10 @@
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
From a31c704e232439a6980bc7319399a80d1f9467b6 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Wed, 8 Feb 2017 14:00:39 +0000
Subject: [PATCH 03/80] Don't include usesAthensIdP member attribute in
statistics
See ukf/ukf-meta#116.
---
mdx/uk/statistics.xsl | 90 +++++--------------------------------------
1 file changed, 10 insertions(+), 80 deletions(-)
diff --git a/mdx/uk/statistics.xsl b/mdx/uk/statistics.xsl
index c31d1721..f6dad0d5 100644
--- a/mdx/uk/statistics.xsl
+++ b/mdx/uk/statistics.xsl
@@ -146,9 +146,10 @@
This may be because they have not yet registered any; perhaps because they have only recently joined.
Alternatively, they may have outsourced their identity provision.
Outsourcing of IdP provision is indicated by an asterisk in the OSrc column in the table.
- This indicates either outsourcing to an Eduserv virtual IdP or a member who "pushes" scopes
+ This indicates a member who "pushes" scopes
to an aggregate IdP.
- Other IdP outsourcing, and any SP outsourcing, is not recorded in the table.
+ Other IdP outsourcing (such as use of an OpenAthens virtual IdP),
+ and any SP outsourcing, is not recorded in the table.
@@ -180,12 +181,6 @@
select="set:difference($membersWithSps, $membersWithIdPs)"/>
-
-
-
@@ -193,8 +188,6 @@
-
-
Breakdown of members by entity registration status:
-
@@ -239,26 +232,13 @@
()
- -
-
- Without entities, but with Athens IdP access:
- ()
-
-
- -
-
- Without entities, and with no Athens IdP access:
- ()
-
-
-
Chart:
,
,
,
- ,
- .
+ .
@@ -271,34 +251,6 @@
*********************************
-->
-
-
-
-
-
-
-
-
-
-
-
-
-
+
-
-
+
+
The following
members of the UK federation have no deployed entities,
- either in their own name or deployed on their behalf by other members.
+ either in their own name or deployed on their behalf by other members
+ and to which they have "pushed" scopes.
+ Use of OpenAthens virtual IdPs is not considered here.
The list is ordered by date of joining the UK federation.
@@ -1256,11 +1191,6 @@
-
-
- *
-
-
-
@@ -407,12 +405,10 @@
-
-
From 1461ae8b5b0c0ac2b80f914ffa5d5abc91198913 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Mon, 13 Feb 2017 14:56:43 +0000
Subject: [PATCH 07/80] Normalise xenc namespace without prefix in production,
wayf and export preview aggregates
See ukf/ukf-meta#110.
---
mdx/uk/README.md | 10 +++++-----
mdx/uk/ns_norm_export_preview.xsl | 24 +++++++++++++++++++++++-
mdx/uk/ns_norm_uk.xsl | 24 +++++++++++++++++++++++-
3 files changed, 51 insertions(+), 7 deletions(-)
diff --git a/mdx/uk/README.md b/mdx/uk/README.md
index 18455dee..ac44d347 100644
--- a/mdx/uk/README.md
+++ b/mdx/uk/README.md
@@ -28,9 +28,9 @@ before being included in the `export` version consumed by interfederation partne
### Export Preview Aggregate vs. Export Aggregate
-Status (2017-02-10):
+Status (2017-02-13):
-* these aggregates are currently identical
+* The export preview aggregate normalises the `xenc` namespace to not use a prefix, as it is not very commonly used. (2017-02-13)
## Production Maturity Pipeline
@@ -59,15 +59,14 @@ when it appeared in the fallback aggregate, which would be too late to take corr
### Test Aggregate vs. Production Aggregate
-Status (2017-02-08):
+Status (2017-02-13):
* the test aggregate implements a _blacklisting_ approach to entity attributes imported from eduGAIN,
while the production aggregate implements the traditional entity attribute _whitelist_.
-* The test aggregate normalises the `xenc` namespace to not use a prefix, as it is not very commonly used.
### Fallback Aggregate vs. Production Aggregate
-Status (2017-02-08):
+Status (2017-02-13):
* The production aggregate defines the `saml` namespace prefix (used by entity attributes) on the document element
instead of in each SAML ``. (2017-02-08)
@@ -76,3 +75,4 @@ instead of in each `` element. (2017-02-08)
* the production aggregate no longer implements the "key use" fixup required for pre-1.3.1 Shibboleth SPs.
This adds the `use="signing"` XML attribute to `` elements present in IdP metadata
without a `use` attribute. It is not needed for later releases of the Shibboleth SP. (2017-02-10)
+* The production aggregate normalises the `xenc` namespace to not use a prefix, as it is not very commonly used. (2017-02-13)
diff --git a/mdx/uk/ns_norm_export_preview.xsl b/mdx/uk/ns_norm_export_preview.xsl
index 59e402e6..cb79a27c 100644
--- a/mdx/uk/ns_norm_export_preview.xsl
+++ b/mdx/uk/ns_norm_export_preview.xsl
@@ -32,8 +32,9 @@
xmlns:shibmd="urn:mace:shibboleth:metadata:1.0"
xmlns:ukfedlabel="http://ukfederation.org.uk/2006/11/label"
xmlns:wayf="http://sdss.ac.uk/2006/06/WAYF"
+ xmlns:xenc="http://www.w3.org/2001/04/xmlenc#"
- exclude-result-prefixes="alg md ukfedlabel wayf"
+ exclude-result-prefixes="alg md ukfedlabel wayf xenc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
@@ -107,5 +108,26 @@
+
+
+
+
+
+
+
+
+
+
+
diff --git a/mdx/uk/ns_norm_uk.xsl b/mdx/uk/ns_norm_uk.xsl
index fb3d7896..20930f67 100644
--- a/mdx/uk/ns_norm_uk.xsl
+++ b/mdx/uk/ns_norm_uk.xsl
@@ -38,8 +38,9 @@
xmlns:shibmd="urn:mace:shibboleth:metadata:1.0"
xmlns:ukfedlabel="http://ukfederation.org.uk/2006/11/label"
xmlns:wayf="http://sdss.ac.uk/2006/06/WAYF"
+ xmlns:xenc="http://www.w3.org/2001/04/xmlenc#"
- exclude-result-prefixes="alg md wayf"
+ exclude-result-prefixes="alg md wayf xenc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
@@ -113,5 +114,26 @@
+
+
+
+
+
+
+
+
+
+
+
From 7ed0dcbc16e4111c9c951ed4de8c3f19dacb12a4 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 14 Feb 2017 10:54:04 +0000
Subject: [PATCH 08/80] Apply R+S checks to eduGAIN as well as local entities
Resolves ukf/ukf-meta#20.
---
mdx/int_edugain/verbs.xml | 1 -
mdx/uk/beans.xml | 1 -
mdx/validation-beans.xml | 1 +
3 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/mdx/int_edugain/verbs.xml b/mdx/int_edugain/verbs.xml
index ed4a6400..e6cd01ce 100644
--- a/mdx/int_edugain/verbs.xml
+++ b/mdx/int_edugain/verbs.xml
@@ -96,7 +96,6 @@
p:whitelistingEntities="false"
p:designatedEntities-ref="int_edugain_verify_blacklist"/>
-
diff --git a/mdx/uk/beans.xml b/mdx/uk/beans.xml
index 6fb81d6a..eb185e63 100644
--- a/mdx/uk/beans.xml
+++ b/mdx/uk/beans.xml
@@ -334,7 +334,6 @@
-
diff --git a/mdx/validation-beans.xml b/mdx/validation-beans.xml
index da3237f7..d59f8ca9 100644
--- a/mdx/validation-beans.xml
+++ b/mdx/validation-beans.xml
@@ -660,6 +660,7 @@
+
From f8e9458d7c80164089b1c204014429400d20920b Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 14 Feb 2017 11:25:28 +0000
Subject: [PATCH 09/80] Normalise xenc namespace without prefix in export
aggregate
See ukf/ukf-meta#110.
---
mdx/uk/README.md | 4 ++--
mdx/uk/ns_norm_export.xsl | 24 +++++++++++++++++++++++-
2 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/mdx/uk/README.md b/mdx/uk/README.md
index ac44d347..3b19450c 100644
--- a/mdx/uk/README.md
+++ b/mdx/uk/README.md
@@ -28,9 +28,9 @@ before being included in the `export` version consumed by interfederation partne
### Export Preview Aggregate vs. Export Aggregate
-Status (2017-02-13):
+Status (2017-02-14):
-* The export preview aggregate normalises the `xenc` namespace to not use a prefix, as it is not very commonly used. (2017-02-13)
+* These aggregates are currently identical.
## Production Maturity Pipeline
diff --git a/mdx/uk/ns_norm_export.xsl b/mdx/uk/ns_norm_export.xsl
index d1448ab2..0d2b762f 100644
--- a/mdx/uk/ns_norm_export.xsl
+++ b/mdx/uk/ns_norm_export.xsl
@@ -32,8 +32,9 @@
xmlns:shibmd="urn:mace:shibboleth:metadata:1.0"
xmlns:ukfedlabel="http://ukfederation.org.uk/2006/11/label"
xmlns:wayf="http://sdss.ac.uk/2006/06/WAYF"
+ xmlns:xenc="http://www.w3.org/2001/04/xmlenc#"
- exclude-result-prefixes="alg md ukfedlabel wayf"
+ exclude-result-prefixes="alg md ukfedlabel wayf xenc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
@@ -107,5 +108,26 @@
+
+
+
+
+
+
+
+
+
+
+
From dd9abaefac6d5a71ff4e5239e78db3c8e6d14bc7 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 14 Feb 2017 14:34:56 +0000
Subject: [PATCH 10/80] Gather statistics about distinct RSA moduli used
See ukf/ukf-meta#113.
---
build.xml | 23 +++++++++++++++++++++++
utilities/check_embedded.pl | 13 +++++++++++++
2 files changed, 36 insertions(+)
diff --git a/build.xml b/build.xml
index ac61494e..05b47510 100644
--- a/build.xml
+++ b/build.xml
@@ -2434,6 +2434,29 @@
+
+
+ Extracting embedded certificates
+
+ Checking embedded certificates
+ Note: ignore expiry on eduGAIN entities
+
+
+
+
+
+
+
diff --git a/utilities/check_embedded.pl b/utilities/check_embedded.pl
index 8272fccd..6919014b 100755
--- a/utilities/check_embedded.pl
+++ b/utilities/check_embedded.pl
@@ -311,6 +311,14 @@ sub comment {
next;
}
+ #
+ # Track distinct RSA moduli
+ #
+ if (/^Modulus=(.*)$/) {
+ $modulus = $1;
+ # print " modulus: '$modulus'\n";
+ $rsa_modulus{$modulus} = 1;
+ }
}
close SSL;
#print " text lines: $#lines\n";
@@ -405,6 +413,11 @@ sub comment {
}
print "\n";
+ $distinct_moduli = scalar keys %rsa_modulus;
+ if ($distinct_moduli > 1) {
+ print "Distinct RSA moduli: $distinct_moduli\n";
+ }
+
my $first = 1;
foreach $fingerprint (sort keys %expiry_whitelist) {
if ($expiry_whitelist{$fingerprint} eq 'unused') {
From 60f355ef2a3811745a9cc8f47249da5f60493b6f Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 14 Feb 2017 15:53:09 +0000
Subject: [PATCH 11/80] Use portable enabling of warnings for perl scripts
Resolves ukf/ukf-meta#111.
---
attic/keynames.pl | 3 ++-
attic/keynames_inner.pl | 4 +++-
charting/fetch.pl | 3 ++-
charting/mdui.pl | 3 ++-
charting/saml2.pl | 3 ++-
charting/scopes.pl | 3 ++-
charting/sizes.pl | 3 ++-
utilities/2016-09-16/doall.pl | 4 +++-
utilities/2016-09-16/patch.pl | 4 +++-
utilities/2016-10-06/patch.pl | 4 +++-
10 files changed, 24 insertions(+), 10 deletions(-)
diff --git a/attic/keynames.pl b/attic/keynames.pl
index 23216794..a917025d 100755
--- a/attic/keynames.pl
+++ b/attic/keynames.pl
@@ -1,10 +1,11 @@
-#!/usr/bin/env perl -w
+#!/usr/bin/env perl
#
# keynames.pl
#
# Extracts statistics about KeyName elements from the published metadata.
#
+use warnings;
use lib "../build";
use Xalan;
use Months;
diff --git a/attic/keynames_inner.pl b/attic/keynames_inner.pl
index bed6059f..3778f942 100755
--- a/attic/keynames_inner.pl
+++ b/attic/keynames_inner.pl
@@ -1,4 +1,6 @@
-#!/usr/bin/env perl -w
+#!/usr/bin/env perl
+
+use warnings;
use POSIX qw(floor);
use File::Temp qw(tempfile);
use Date::Format;
diff --git a/charting/fetch.pl b/charting/fetch.pl
index 9b01e351..47d3e51e 100755
--- a/charting/fetch.pl
+++ b/charting/fetch.pl
@@ -1,8 +1,9 @@
-#!/usr/bin/env perl -w
+#!/usr/bin/env perl
#
# fetch.pl
#
+use warnings;
use File::stat;
use Months;
diff --git a/charting/mdui.pl b/charting/mdui.pl
index b6e14bd0..fb2a6bfd 100755
--- a/charting/mdui.pl
+++ b/charting/mdui.pl
@@ -1,8 +1,9 @@
-#!/usr/bin/env perl -w
+#!/usr/bin/env perl
#
# mdui.pl
#
+use warnings;
use lib "../build";
use Xalan;
use Months;
diff --git a/charting/saml2.pl b/charting/saml2.pl
index bc69fd04..fd790429 100755
--- a/charting/saml2.pl
+++ b/charting/saml2.pl
@@ -1,10 +1,11 @@
-#!/usr/bin/env perl -w
+#!/usr/bin/env perl
#
# saml2.pl
#
# Extracts statistics about SAML 2 adoption from the published metadata.
#
+use warnings;
use lib "../build";
use Xalan;
use Months;
diff --git a/charting/scopes.pl b/charting/scopes.pl
index 2e38809d..3927471b 100755
--- a/charting/scopes.pl
+++ b/charting/scopes.pl
@@ -1,10 +1,11 @@
-#!/usr/bin/env perl -w
+#!/usr/bin/env perl
#
# scopes.pl
#
# Extracts statistics about number of scopes from the published metadata.
#
+use warnings;
use lib "../build";
use Xalan;
use Months;
diff --git a/charting/sizes.pl b/charting/sizes.pl
index 4d36b33d..004c6e3f 100755
--- a/charting/sizes.pl
+++ b/charting/sizes.pl
@@ -1,8 +1,9 @@
-#!/usr/bin/env perl -w
+#!/usr/bin/env perl
#
# sizes.pl
#
+use warnings;
use lib "../build";
use File::stat;
use Xalan;
diff --git a/utilities/2016-09-16/doall.pl b/utilities/2016-09-16/doall.pl
index 7056b5d1..d2b8255b 100755
--- a/utilities/2016-09-16/doall.pl
+++ b/utilities/2016-09-16/doall.pl
@@ -1,4 +1,6 @@
-#!/usr/bin/env perl -W
+#!/usr/bin/env perl
+
+use warnings;
open(F, "id-to-name.txt") || die "could not open id-to-name map";
while () {
diff --git a/utilities/2016-09-16/patch.pl b/utilities/2016-09-16/patch.pl
index c47766c3..543e0a0c 100755
--- a/utilities/2016-09-16/patch.pl
+++ b/utilities/2016-09-16/patch.pl
@@ -1,4 +1,6 @@
-#!/usr/bin/env perl -W
+#!/usr/bin/env perl
+
+use warnings;
my $orgID = shift @ARGV;
diff --git a/utilities/2016-10-06/patch.pl b/utilities/2016-10-06/patch.pl
index 638f5d7d..86744119 100755
--- a/utilities/2016-10-06/patch.pl
+++ b/utilities/2016-10-06/patch.pl
@@ -1,4 +1,6 @@
-#!/usr/bin/env perl -W
+#!/usr/bin/env perl
+
+use warnings;
open(F, "id-to-name.txt") || die "could not open id-to-name map";
while () {
From 547057db6c39c2fae7a3ca4d7ad0f1b065712cde Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 14 Feb 2017 17:37:55 +0000
Subject: [PATCH 12/80] Automate the collection of some more charting
statistics
See ukf/ukf-meta#106.
---
mdx/uk/statistics.xsl | 88 ++++++++++++++++++++++++++++++++++++-------
1 file changed, 74 insertions(+), 14 deletions(-)
diff --git a/mdx/uk/statistics.xsl b/mdx/uk/statistics.xsl
index f6dad0d5..ba84c359 100644
--- a/mdx/uk/statistics.xsl
+++ b/mdx/uk/statistics.xsl
@@ -128,6 +128,7 @@
Shibboleth 1.3 Remnants
Export Aggregate: Entities Opted Out
Export Aggregate: Entities Explicitly Opted In
+ Charting Statistics
Entities Without SAML 2.0 Support
@@ -305,16 +306,6 @@
()
-
-
- Chart:
- ,
- ,
- ,
- ,
- .
-
-
@@ -1066,6 +1057,65 @@
+
+
+
+
+ - Members:
+ -
+ Outsourcing chart:
+ ,
+ ,
+ ,
+ ,
+
+
+ - Entities:
+ - IdPs:
+ - SPs:
+ - Entities per member:
+
+
+ -
+ Algorithm support:
+
+ of all entities
+
+
+
+ -
+ GCM support:
+
+ of all entities
+
+
+
+ -
+ Algorithm support:
+
+ of SP entities
+
+
+
+ -
+ Shibboleth IdP v3:
+
+ ( of IdPs)
+
+
+
+
+
+ ResponseLocation contains space character
+
+
+
+
+
+
+
+
+ ResponseLocation is not a valid URL:
+
+
+
+
+
+
diff --git a/mdx/_rules/check_sp_tls.xsl b/mdx/_rules/check_sp_tls.xsl
index b68c9663..401740ad 100644
--- a/mdx/_rules/check_sp_tls.xsl
+++ b/mdx/_rules/check_sp_tls.xsl
@@ -27,5 +27,10 @@
Location does not start with https://
-
+
+
+ ResponseLocation does not start with https://
+
+
+
From 122d13baa775de2bc74494ba19e8dbd7439275c6 Mon Sep 17 00:00:00 2001
From: Alex Stuart
Date: Fri, 24 Feb 2017 08:27:36 +0000
Subject: [PATCH 14/80] Add SAFIRE (ZA) to list of Federation
registrationAuthority URIs
---
mdx/common-beans.xml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/mdx/common-beans.xml b/mdx/common-beans.xml
index 7fcb3488..9bc882c7 100644
--- a/mdx/common-beans.xml
+++ b/mdx/common-beans.xml
@@ -525,6 +525,7 @@
+
@@ -580,6 +581,7 @@
+
From a8e99a784bdbef9e63a90d7d493c5a13cb0696a0 Mon Sep 17 00:00:00 2001
From: Alex Stuart
Date: Tue, 28 Feb 2017 16:11:03 +0000
Subject: [PATCH 15/80] Add utilities to replace HideFromWAYF element with
hide-from-disco Entity Category
---
utilities/2017-02-27/README.md | 38 +++++++++++++++++++
.../2017-02-27/listHideFromWAYFandEA.xsl | 21 ++++++++++
utilities/2017-02-27/replaceHideFromWAYF.pl | 18 +++++++++
3 files changed, 77 insertions(+)
create mode 100644 utilities/2017-02-27/README.md
create mode 100644 utilities/2017-02-27/listHideFromWAYFandEA.xsl
create mode 100755 utilities/2017-02-27/replaceHideFromWAYF.pl
diff --git a/utilities/2017-02-27/README.md b/utilities/2017-02-27/README.md
new file mode 100644
index 00000000..c56e45ef
--- /dev/null
+++ b/utilities/2017-02-27/README.md
@@ -0,0 +1,38 @@
+# `utilities/2017-02-27`
+
+Scripts to replace the HideFromWAYF element in entity fragment files
+with the REFEDS Hide from Discovery Entity Category.
+
+## 1. Check that no hidden IdPs have Entity Attributes already
+
+Since there can only be a single Entity Attribute element in an entity fragment file,
+we first check that there are no hidden IdPs that already have an Entity Attributes
+element. If there are (and there are not too many) we edit these files manually.
+
+Run the script on the entity fragment files: `xsltproc listHideFromWAYFandEA.xsl uk*.xml`
+
+## 2. Replace HideFromWAYF element with hide-from-disco Entity Category
+
+This command replaces the HideFromWAYF element with an Entity Attributes element
+containing the REFEDS hide-from-disco entity category:
+
+`replaceHideFromWAYF.pl uk*.xml`
+
+It presumes that the `saml` and `mdattr` namespace prefixes are already defined in the
+entity fragment files.
+
+The perl regex matches the string HideFromWAYF rather than an XML element, so check
+that transform has only modified the HideFromWAYF element by generating unsigned
+aggregates before and after the transform and and looking at the differences.
+The only changes should be the timestamp and quantities derived from the timestamp.
+There is a small possibility that the generate target imports different entities from
+eduGAIN -- these differences can be ignored.
+
+```
+ant samlmd.aggregates.generate
+cp ukfederation-metadata.xml /tmp/
+replaceHideFromWAYF.pl uk*.xml
+ant samlmd.aggregates.generate
+diff ukfederation-metadata.xml /tmp/
+```
+
diff --git a/utilities/2017-02-27/listHideFromWAYFandEA.xsl b/utilities/2017-02-27/listHideFromWAYFandEA.xsl
new file mode 100644
index 00000000..fcd27029
--- /dev/null
+++ b/utilities/2017-02-27/listHideFromWAYFandEA.xsl
@@ -0,0 +1,21 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/utilities/2017-02-27/replaceHideFromWAYF.pl b/utilities/2017-02-27/replaceHideFromWAYF.pl
new file mode 100755
index 00000000..32356fed
--- /dev/null
+++ b/utilities/2017-02-27/replaceHideFromWAYF.pl
@@ -0,0 +1,18 @@
+#!/usr/bin/perl -wni
+
+# If line contains HideFromWAYF, replace it with the Entity Category
+if (/HideFromWAYF/) {
+ print <
+
+ http://refeds.org/category/hide-from-discovery
+
+
+EOF
+# and don't print the line containing HideFromWAYF
+ next;
+}
+
+# If the line didn't have HideFromWAYF, print it unchanged
+print;
+
From 042e7744bd34dbd6bb97e492626b76d3808a923c Mon Sep 17 00:00:00 2001
From: Alex Stuart
Date: Wed, 1 Mar 2017 21:26:19 +0000
Subject: [PATCH 16/80] Add a test to the utility for replacing HideFromWAYF
element with hide-from-disco Entity Category
---
utilities/2017-02-27/README.md | 6 +++++-
utilities/2017-02-27/test.xml | 20 ++++++++++++++++++++
2 files changed, 25 insertions(+), 1 deletion(-)
create mode 100644 utilities/2017-02-27/test.xml
diff --git a/utilities/2017-02-27/README.md b/utilities/2017-02-27/README.md
index c56e45ef..acb2556c 100644
--- a/utilities/2017-02-27/README.md
+++ b/utilities/2017-02-27/README.md
@@ -9,7 +9,11 @@ Since there can only be a single Entity Attribute element in an entity fragment
we first check that there are no hidden IdPs that already have an Entity Attributes
element. If there are (and there are not too many) we edit these files manually.
-Run the script on the entity fragment files: `xsltproc listHideFromWAYFandEA.xsl uk*.xml`
+First, check that the XSLT will flag an entity fragment file that has an Entity
+Attribute and the HideFromWAYF element. Run `xsltproc listHideFromWAYFandEA.xsl ./test.xml`.
+This should report `https://idp.example.ac.uk/idp/shibboleth`.
+
+Then run the script on all entity fragment files: `xsltproc listHideFromWAYFandEA.xsl uk*.xml`
## 2. Replace HideFromWAYF element with hide-from-disco Entity Category
diff --git a/utilities/2017-02-27/test.xml b/utilities/2017-02-27/test.xml
new file mode 100644
index 00000000..8474b788
--- /dev/null
+++ b/utilities/2017-02-27/test.xml
@@ -0,0 +1,20 @@
+
+
+
+
+
+ http://refeds.org/category/research-and-scholarship
+
+
+
+
+
From a1ea68f7fce8d7b9aaa99f8a0f94c75e1d23af2b Mon Sep 17 00:00:00 2001
From: Alex Stuart
Date: Wed, 1 Mar 2017 21:30:06 +0000
Subject: [PATCH 17/80] Correct test script in utilities/2017-02-27/README.md
to refer to unsigned aggregate
---
utilities/2017-02-27/README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/utilities/2017-02-27/README.md b/utilities/2017-02-27/README.md
index acb2556c..525155d2 100644
--- a/utilities/2017-02-27/README.md
+++ b/utilities/2017-02-27/README.md
@@ -34,9 +34,9 @@ eduGAIN -- these differences can be ignored.
```
ant samlmd.aggregates.generate
-cp ukfederation-metadata.xml /tmp/
+cp ukfederation-metadata-unsigned.xml /tmp/
replaceHideFromWAYF.pl uk*.xml
ant samlmd.aggregates.generate
-diff ukfederation-metadata.xml /tmp/
+diff ukfederation-metadata-unsigned.xml /tmp/
```
From fee4287d6ba5b9831958bb0a3979018b80bac246 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Thu, 2 Mar 2017 15:03:15 +0000
Subject: [PATCH 18/80] Switch to entity attribute blacklist for production
aggregate
Second phase of ukf/ukf-meta#10.
---
mdx/uk/README.md | 9 +++++----
mdx/uk/generate.xml | 1 -
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/mdx/uk/README.md b/mdx/uk/README.md
index 3b19450c..bbb83c83 100644
--- a/mdx/uk/README.md
+++ b/mdx/uk/README.md
@@ -59,15 +59,16 @@ when it appeared in the fallback aggregate, which would be too late to take corr
### Test Aggregate vs. Production Aggregate
-Status (2017-02-13):
+Status (2017-03-02):
-* the test aggregate implements a _blacklisting_ approach to entity attributes imported from eduGAIN,
-while the production aggregate implements the traditional entity attribute _whitelist_.
+* These aggregates are currently identical.
### Fallback Aggregate vs. Production Aggregate
-Status (2017-02-13):
+Status (2017-03-02):
+* the production aggregate implements a _blacklisting_ approach to entity attributes imported from eduGAIN,
+while the production aggregate implements the traditional entity attribute _whitelist_. (2017-03-02)
* The production aggregate defines the `saml` namespace prefix (used by entity attributes) on the document element
instead of in each SAML ``. (2017-02-08)
* The production aggregate defines the `mdattr` namespace prefix (used by entity attributes) on the document element
diff --git a/mdx/uk/generate.xml b/mdx/uk/generate.xml
index d3382fea..e90a33c6 100644
--- a/mdx/uk/generate.xml
+++ b/mdx/uk/generate.xml
@@ -334,7 +334,6 @@
-
From 34ccd7185a437314d1326f00395ae26afb4523ab Mon Sep 17 00:00:00 2001
From: Alex Stuart
Date: Mon, 6 Mar 2017 16:33:20 +0000
Subject: [PATCH 19/80] Update federation URI and eduGAIN status for new
eduGAIN members
---
mdx/common-beans.xml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mdx/common-beans.xml b/mdx/common-beans.xml
index 9bc882c7..0f2d3602 100644
--- a/mdx/common-beans.xml
+++ b/mdx/common-beans.xml
@@ -505,7 +505,7 @@
-
+
@@ -568,6 +568,7 @@
+
@@ -576,6 +577,7 @@
+
@@ -597,8 +599,6 @@
-
-
From a1629a29a913bc484dc24ae09cb1dbc2a9ce4bb5 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Fri, 10 Mar 2017 10:12:03 +0000
Subject: [PATCH 20/80] Add new md dist servers into build scripting
---
build.xml | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 108 insertions(+)
diff --git a/build.xml b/build.xml
index 05b47510..0d248f6b 100644
--- a/build.xml
+++ b/build.xml
@@ -148,6 +148,10 @@
+
+
+
+
@@ -1317,6 +1321,70 @@
+ Verifying metadata held at ${md.dist.host-ne-01.name}
+
+
+
+
+
+
+
+
+ Verifying metadata held at ${md.dist.host-ne-02.name}
+
+
+
+
+
+
+
+
+ Verifying metadata held at ${md.dist.host-we-01.name}
+
+
+
+
+
+
+
+
+ Verifying metadata held at ${md.dist.host-we-02.name}
+
+
+
+
+
+
+
+
Verification completed.
@@ -2058,6 +2126,26 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@@ -2083,6 +2171,26 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
From 09cad3b4786eb1cf6484d7cfb1b9165e58e11bb1 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Fri, 10 Mar 2017 11:29:14 +0000
Subject: [PATCH 21/80] Add new MD test servers into build scripting
---
preprod.properties | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/preprod.properties b/preprod.properties
index 5d6b3cb4..23489f03 100644
--- a/preprod.properties
+++ b/preprod.properties
@@ -38,6 +38,10 @@ git.repo.project.tooling=ukf-test-meta
md.dist.host1.name=md1-test.infr.ukfederation.org.uk
md.dist.host2.name=md2-test.infr.ukfederation.org.uk
md.dist.host3.name=md3-test.infr.ukfederation.org.uk
+md.dist.host-ne-01.name=md-ne-01-test.infr.ukfederation.org.uk
+md.dist.host-ne-02.name=md-ne-02-test.infr.ukfederation.org.uk
+md.dist.host-we-01.name=md-we-01-test.infr.ukfederation.org.uk
+md.dist.host-we-02.name=md-we-02-test.infr.ukfederation.org.uk
md.dist.path.name=/
#
From 6ccec07d83f85e912f62a55b918f14116bed2509 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Fri, 10 Mar 2017 11:36:29 +0000
Subject: [PATCH 22/80] Add missing echo statements to publication output for
new MD servers
---
build.xml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/build.xml b/build.xml
index 0d248f6b..428b613e 100644
--- a/build.xml
+++ b/build.xml
@@ -2126,21 +2126,25 @@
+ -> MD-NE-01
+ -> MD-NE-02
+ -> MD-WE-01
+ -> MD-WE-01
@@ -2171,21 +2175,25 @@
+ -> MD-NE-01
+ -> MD-NE-02
+ -> MD-WE-01
+ -> MD-WE-02
From b55e1a34785ea33ada799ee638dd7169767057dd Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Fri, 10 Mar 2017 12:09:41 +0000
Subject: [PATCH 23/80] Hoist mdattr and saml prefix definitions in fallback
aggregate
See ukf/ukf-meta#103 and ukf/ukf-meta#105.
---
mdx/uk/README.md | 6 +-----
mdx/uk/ns_norm_back.xsl | 2 +-
2 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/mdx/uk/README.md b/mdx/uk/README.md
index bbb83c83..e10c6d8f 100644
--- a/mdx/uk/README.md
+++ b/mdx/uk/README.md
@@ -65,14 +65,10 @@ Status (2017-03-02):
### Fallback Aggregate vs. Production Aggregate
-Status (2017-03-02):
+Status (2017-03-10):
* the production aggregate implements a _blacklisting_ approach to entity attributes imported from eduGAIN,
while the production aggregate implements the traditional entity attribute _whitelist_. (2017-03-02)
-* The production aggregate defines the `saml` namespace prefix (used by entity attributes) on the document element
-instead of in each SAML ``. (2017-02-08)
-* The production aggregate defines the `mdattr` namespace prefix (used by entity attributes) on the document element
-instead of in each `` element. (2017-02-08)
* the production aggregate no longer implements the "key use" fixup required for pre-1.3.1 Shibboleth SPs.
This adds the `use="signing"` XML attribute to `` elements present in IdP metadata
without a `use` attribute. It is not needed for later releases of the Shibboleth SP. (2017-02-10)
diff --git a/mdx/uk/ns_norm_back.xsl b/mdx/uk/ns_norm_back.xsl
index f485059c..64967aac 100644
--- a/mdx/uk/ns_norm_back.xsl
+++ b/mdx/uk/ns_norm_back.xsl
@@ -39,7 +39,7 @@
xmlns:ukfedlabel="http://ukfederation.org.uk/2006/11/label"
xmlns:wayf="http://sdss.ac.uk/2006/06/WAYF"
- exclude-result-prefixes="alg md mdattr saml wayf"
+ exclude-result-prefixes="alg md wayf"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
From fdabcbc481d995676d74d7449834e3a38de7fb13 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Fri, 10 Mar 2017 15:34:53 +0000
Subject: [PATCH 24/80] Retire key use fixup in fallback aggregate
See ukf/ukf-meta#109.
---
mdx/uk/README.md | 3 --
mdx/uk/check_fixup_keyuse.xsl | 44 -----------------------------
mdx/uk/fixup_keyuse.xsl | 52 -----------------------------------
mdx/uk/generate.xml | 43 +----------------------------
4 files changed, 1 insertion(+), 141 deletions(-)
delete mode 100644 mdx/uk/check_fixup_keyuse.xsl
delete mode 100644 mdx/uk/fixup_keyuse.xsl
diff --git a/mdx/uk/README.md b/mdx/uk/README.md
index e10c6d8f..d0bd2d04 100644
--- a/mdx/uk/README.md
+++ b/mdx/uk/README.md
@@ -69,7 +69,4 @@ Status (2017-03-10):
* the production aggregate implements a _blacklisting_ approach to entity attributes imported from eduGAIN,
while the production aggregate implements the traditional entity attribute _whitelist_. (2017-03-02)
-* the production aggregate no longer implements the "key use" fixup required for pre-1.3.1 Shibboleth SPs.
-This adds the `use="signing"` XML attribute to `` elements present in IdP metadata
-without a `use` attribute. It is not needed for later releases of the Shibboleth SP. (2017-02-10)
* The production aggregate normalises the `xenc` namespace to not use a prefix, as it is not very commonly used. (2017-02-13)
diff --git a/mdx/uk/check_fixup_keyuse.xsl b/mdx/uk/check_fixup_keyuse.xsl
deleted file mode 100644
index cb91fe5d..00000000
--- a/mdx/uk/check_fixup_keyuse.xsl
+++ /dev/null
@@ -1,44 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
- IdP SSO KeyDescriptor lacking @use
-
-
-
-
-
- IdP AA KeyDescriptor lacking @use
-
-
-
-
diff --git a/mdx/uk/fixup_keyuse.xsl b/mdx/uk/fixup_keyuse.xsl
deleted file mode 100644
index 126069b6..00000000
--- a/mdx/uk/fixup_keyuse.xsl
+++ /dev/null
@@ -1,52 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
- signing
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/mdx/uk/generate.xml b/mdx/uk/generate.xml
index e90a33c6..9a5bcb41 100644
--- a/mdx/uk/generate.xml
+++ b/mdx/uk/generate.xml
@@ -61,46 +61,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
From 2a889b44733933b362d98fbd89c466998d0afc04 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 14 Mar 2017 11:14:12 +0000
Subject: [PATCH 25/80] Normalise xenc namespace without prefix in fallback
aggregate
See ukf/ukf-meta#110.
---
mdx/uk/README.md | 3 +--
mdx/uk/ns_norm_back.xsl | 24 +++++++++++++++++++++++-
2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/mdx/uk/README.md b/mdx/uk/README.md
index d0bd2d04..88b4703b 100644
--- a/mdx/uk/README.md
+++ b/mdx/uk/README.md
@@ -65,8 +65,7 @@ Status (2017-03-02):
### Fallback Aggregate vs. Production Aggregate
-Status (2017-03-10):
+Status (2017-03-14):
* the production aggregate implements a _blacklisting_ approach to entity attributes imported from eduGAIN,
while the production aggregate implements the traditional entity attribute _whitelist_. (2017-03-02)
-* The production aggregate normalises the `xenc` namespace to not use a prefix, as it is not very commonly used. (2017-02-13)
diff --git a/mdx/uk/ns_norm_back.xsl b/mdx/uk/ns_norm_back.xsl
index 64967aac..0d2402c2 100644
--- a/mdx/uk/ns_norm_back.xsl
+++ b/mdx/uk/ns_norm_back.xsl
@@ -38,8 +38,9 @@
xmlns:shibmd="urn:mace:shibboleth:metadata:1.0"
xmlns:ukfedlabel="http://ukfederation.org.uk/2006/11/label"
xmlns:wayf="http://sdss.ac.uk/2006/06/WAYF"
+ xmlns:xenc="http://www.w3.org/2001/04/xmlenc#"
- exclude-result-prefixes="alg md wayf"
+ exclude-result-prefixes="alg md wayf xenc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns="urn:oasis:names:tc:SAML:2.0:metadata">
@@ -113,5 +114,26 @@
+
+
+
+
+
+
+
+
+
+
+
From 0253352c05d7eb922f9890f21f427706309639ce Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Wed, 15 Mar 2017 14:25:34 +0000
Subject: [PATCH 26/80] Add separate textual charting statistics
This is a much cut-down version of the current stats page, converted to
output text rather than HTML.
See ukf/ukf-meta#106.
---
build.xml | 9 +
mdx/uk/statistics-charting.xsl | 597 +++++++++++++++++++++++++++++++++
mdx/uk/verbs.xml | 29 ++
3 files changed, 635 insertions(+)
create mode 100644 mdx/uk/statistics-charting.xsl
diff --git a/build.xml b/build.xml
index 428b613e..3530a35a 100644
--- a/build.xml
+++ b/build.xml
@@ -2409,6 +2409,15 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Members:
+
+
+
+ Outsourcing chart:
+
+ ,
+
+ ,
+
+ ,
+
+ ,
+
+
+
+ Entities:
+
+
+
+ IdPs:
+
+
+
+ SPs:
+
+
+
+ Entities per member:
+
+
+
+
+
+ Algorithm support:
+
+ of all entities
+
+
+
+
+ GCM support:
+
+ of all entities
+
+
+
+
+ Algorithm support:
+
+ of SP entities
+
+
+
+
+ Shibboleth IdP v3:
+
+ (
+
+ of IdPs)
+
+
+
+
+
+ SPs without SAML 2.0 support:
+
+
+
+
+
+
+
+ :
+
+ :
+
+
+
+
+
+ (
+
+ )
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ IdPs without SAML 2.0 support:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Breakdown by software used:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Shibboleth 3.x
+
+
+
+
+
+ Shibboleth 2.x
+
+
+
+
+
+ Shibboleth 1.3
+
+
+
+
+
+
+
+ Shibboleth combined
+
+
+
+
+
+
+ Other than Shibboleth
+
+
+
+
+
+ EZproxy
+
+
+
+
+
+ simpleSAMLphp
+
+
+
+
+
+ Atypon SAML SP
+
+
+
+
+
+ AthensIM
+
+
+
+
+
+ Guanxi
+
+
+
+
+
+ Athens/Shibboleth gateway
+
+
+
+
+
+ OpenAthens Virtual IdP
+
+
+
+
+
+ OpenAthens
+
+
+
+
+
+ Unknown or other
+
+
+
+
+
+
+
+
+
+
+
+ 0
+ 0
+ 8
+
+
+
+
+ :
+
+ (
+
+ )
+
+
+
+
+
+
+ :
+
+
+
+
+ ()
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/mdx/uk/verbs.xml b/mdx/uk/verbs.xml
index e613951b..7941a6ad 100644
--- a/mdx/uk/verbs.xml
+++ b/mdx/uk/verbs.xml
@@ -46,6 +46,35 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+
From 9d3b3d05005ad8e61760d2fe6b48ac9ecae26919 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Tue, 28 Mar 2017 17:37:46 +0100
Subject: [PATCH 30/80] Add new servers into stats gen and update ipv6 parsing
accordingly
---
utilities/stats-generate.sh | 127 ++++++++++++++++++++----------------
1 file changed, 70 insertions(+), 57 deletions(-)
diff --git a/utilities/stats-generate.sh b/utilities/stats-generate.sh
index fff9c495..ac8ca846 100755
--- a/utilities/stats-generate.sh
+++ b/utilities/stats-generate.sh
@@ -179,18 +179,18 @@ fi
# Get the filesize of the latest uncompressed main aggregate.
# Since this is just used for estimation, we'll just take the biggest
# unique filesize for the relevant periods
-aggrfilesizebytes=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | grep -v "GZIP" | cut -f 10 -d " " | sort -r | uniq | head -1)
+aggrfilesizebytes=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | grep -v "GZIP" | cut -f 10 -d " " | sort -r | uniq | head -1)
#
# Download counts
#
# Aggregate requests. Everything for .xml (HEAD/GET, 200 and 304)
-mdaggrcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | wc -l)
+mdaggrcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | wc -l)
mdaggrcountfriendly=$(echo $mdaggrcount | awk '{ printf ("%'"'"'d\n", $0) }')
# Main Aggregate requests. Everything for ukfederation-metadata.xml (HEAD/GET, 200 and 304)
-mdaggrmaincount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | wc -l)
+mdaggrmaincount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | wc -l)
mdaggrmaincountfriendly=$(echo $mdaggrmaincount | awk '{ printf ("%'"'"'d\n", $0) }')
if [[ "$mdaggrmaincount" -ne "0" ]]; then
mdaggrmainpc=$(echo "scale=4;($mdaggrmaincount/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
@@ -200,42 +200,42 @@ fi
# Other aggregate requests (don't calculate these if doing daily stats)
if [[ "$timeperiod" != "day" ]]; then
- mdaggrbackcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-back.xml" | wc -l)
+ mdaggrbackcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-back.xml" | wc -l)
mdaggrbackcountfriendly=$(echo $mdaggrbackcount | awk '{ printf ("%'"'"'d\n", $0) }')
if [[ "$mdaggrbackcount" -ne "0" ]]; then
mdaggrbackpc=$(echo "scale=4;($mdaggrbackcount/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
else
mdaggrbackpc="0.0"
fi
- mdaggrcdsallcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-cdsall.xml" | wc -l)
+ mdaggrcdsallcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-cdsall.xml" | wc -l)
mdaggrcdsallcountfriendly=$(echo $mdaggrcdsallcount | awk '{ printf ("%'"'"'d\n", $0) }')
if [[ "$mdaggrcdsallcount" -ne "0" ]]; then
mdaggrcdsallpc=$(echo "scale=4;($mdaggrcdsallcount/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
else
mdaggrcdsallpc="0.0"
fi
- mdaggrexportpreviewcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-export-preview.xml" | wc -l)
+ mdaggrexportpreviewcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-export-preview.xml" | wc -l)
mdaggrexportpreviewcountfriendly=$(echo $mdaggrexportpreviewcount | awk '{ printf ("%'"'"'d\n", $0) }')
if [[ "$mdaggrexportpreviewkcount" -ne "0" ]]; then
mdaggrexportpreviewpc=$(echo "scale=4;($mdaggrexportpreviewcount/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
else
mdaggrexportpreviewpc="0.0"
fi
- mdaggrexportcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-export.xml" | wc -l)
+ mdaggrexportcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-export.xml" | wc -l)
mdaggrexportcountfriendly=$(echo $mdaggrexportcount | awk '{ printf ("%'"'"'d\n", $0) }')
if [[ "$mdaggrexportcount" -ne "0" ]]; then
mdaggrexportpc=$(echo "scale=4;($mdaggrexportcount/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
else
mdaggrexportpc="0.0"
fi
- mdaggrtestcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-test.xml" | wc -l)
+ mdaggrtestcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-test.xml" | wc -l)
mdaggrtestcountfriendly=$(echo $mdaggrtestcount | awk '{ printf ("%'"'"'d\n", $0) }')
if [[ "$mdaggrtestcount" -ne "0" ]]; then
mdaggrtestpc=$(echo "scale=4;($mdaggrtestcount/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
else
mdaggrtestpc="0.0"
fi
- mdaggrwayfcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-wayf.xml" | wc -l)
+ mdaggrwayfcount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-wayf.xml" | wc -l)
mdaggrwayfcountfriendly=$(echo $mdaggrwayfcount | awk '{ printf ("%'"'"'d\n", $0) }')
if [[ "$mdaggrwayfcount" -ne "0" ]]; then
mdaggrwayfpc=$(echo "scale=4;($mdaggrwayfcount/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
@@ -245,11 +245,11 @@ if [[ "$timeperiod" != "day" ]]; then
fi
# Aggregate downloads (i.e. GETs with HTTP 200 responses only)
-mdaggrcountfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404| grep "\" 200" | grep "GET" | wc -l)
+mdaggrcountfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404| grep "\" 200" | grep "GET" | wc -l)
mdaggrcountfullfriendly=$(echo $mdaggrcountfull | awk '{ printf ("%'"'"'d\n", $0) }')
# Main Aggregate downloads (i.e. GETs with HTTP 200 responses only)
-mdaggrmaincountfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | wc -l)
+mdaggrmaincountfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | wc -l)
mdaggrmaincountfullfriendly=$(echo $mdaggrmaincountfull | awk '{ printf ("%'"'"'d\n", $0) }')
# Percentage of GETs with HTTP 200 responses compared to total requests
@@ -260,11 +260,11 @@ else
fi
# Compressed downloads for all
-mdaggrcountfullcompr=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | grep "\"GZIP\"" | wc -l)
+mdaggrcountfullcompr=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | grep "\"GZIP\"" | wc -l)
mdaggrcountfullcomprfriendly=$(echo $mdaggrcountfullcompr | awk '{ printf ("%'"'"'d\n", $0) }')
# Compressed downloads for main aggregate
-mdaggrmaincountfullcompr=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | grep "\"GZIP\"" | wc -l)
+mdaggrmaincountfullcompr=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | grep "\"GZIP\"" | wc -l)
# Percentage of GZIPPED HTTP 200 responses compared to total full downloads
if [[ "$mdaggrcountfull" -ne "0" ]]; then
@@ -274,18 +274,18 @@ else
fi
# Unique IP addresses requesting aggregates
-mdaggruniqueip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq | wc -l)
+mdaggruniqueip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l)
mdaggruniqueipfriendly=$(echo $mdaggruniqueip | awk '{ printf ("%'"'"'d\n", $0) }')
# Unique IP addresses requesting aggregates, full D/Ls only
-mdaggruniqueipfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq | wc -l)
+mdaggruniqueipfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l)
#
# Data shipped
#
# Total data shipped, all .xml files
-mdaggrtotalbytes=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 10 -d " " | awk '{sum+=$1} END {print sum}')
+mdaggrtotalbytes=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 10 -d " " | awk '{sum+=$1} END {print sum}')
if [[ "$mdaggrtotalbytes" -gt "0" ]]; then
mdaggrtotalhr=$(bytestohr $mdaggrtotalbytes)
else
@@ -293,7 +293,7 @@ else
fi
# Total data shipped, ukfederation-metadata.xml file
-mdaggrmaintotalbytes=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | cut -f 10 -d " " | awk '{sum+=$1} END {print sum}')
+mdaggrmaintotalbytes=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "ukfederation-metadata.xml" | grep "\" 200" | grep "GET" | cut -f 10 -d " " | awk '{sum+=$1} END {print sum}')
if [[ "$mdaggrtotalbytes" -gt "0" ]]; then
mdaggrmaintotalhr=$(bytestohr $mdaggrmaintotalbytes)
else
@@ -321,27 +321,34 @@ fi
#
# IPv4 vs IPv6 traffic (don't calculate these if doing daily stats)
-# Note, while all v6 traffic passes through v6v4proxy1/2, we're counting accesses from the IPv4 addresses of those servers vs all others.
-# When we add "real" v6 support to the servers, this needs changing to count IPv4 addresses vs IPv6 addresses.
+# Some v6 traffic has traditionally passed through v6v4proxy1/2, so to count v4 we're counting all accesses, minus those from the v4 proxy IP addresses, minus actual v6 addresses
if [[ "$timeperiod" != "day" ]]; then
- mdaggrv4count=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep -v 193.63.72.83 | grep -v 194.83.7.211 | wc -l)
+ mdaggrv4count=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 1 -d " " | cut -f 2-9 -d ":" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep -v ":" | wc -l)
mdaggrv4pc=$(echo "scale=4;($mdaggrv4count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
mdaggrv6count=$(( mdaggrcount - mdaggrv4count ))
mdaggrv6pc=$(echo "scale=4;($mdaggrv6count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
# Per-server request count
- mdaggrmd1count=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | grep md1 | wc -l)
+ mdaggrmd1count=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | wc -l)
mdaggrmd1pc=$(echo "scale=4;($mdaggrmd1count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
- mdaggrmd2count=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | grep md2 | wc -l)
+ mdaggrmd2count=$(grep $apachesearchterm $logslocation/md/md2/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | wc -l)
mdaggrmd2pc=$(echo "scale=4;($mdaggrmd2count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
- mdaggrmd3count=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | grep md3 | wc -l)
+ mdaggrmd3count=$(grep $apachesearchterm $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | wc -l)
mdaggrmd3pc=$(echo "scale=4;($mdaggrmd3count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ mdaggrmdne01count=$(grep $apachesearchterm $logslocation/md/md-ne-01/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | wc -l)
+ mdaggrmdne01pc=$(echo "scale=4;($mdaggrmdne01count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ mdaggrmdne02count=$(grep $apachesearchterm $logslocation/md/md-ne-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | wc -l)
+ mdaggrmdne02pc=$(echo "scale=4;($mdaggrmdne02count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ mdaggrmdwe01count=$(grep $apachesearchterm $logslocation/md/md-we-01/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | wc -l)
+ mdaggrmdwe01pc=$(echo "scale=4;($mdaggrmdwe01count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ mdaggrmdwe02count=$(grep $apachesearchterm $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 5 -d "/" | wc -l)
+ mdaggrmdwe02pc=$(echo "scale=4;($mdaggrmdwe02count/$mdaggrcount)*100" | bc | awk '{printf "%.1f\n", $0}')
fi
# Min queries per IP
if [[ $mdaggrcount -gt "0" ]]; then
- mdaggrminqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | tail -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
+ mdaggrminqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | tail -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
else
mdaggrinqueriesperip="0"
fi
@@ -355,14 +362,14 @@ fi
# Max queries per IP
if [[ $mdaggrcount -gt "0" ]]; then
- mdaggrmaxqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | head -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
+ mdaggrmaxqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | head -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
else
mdaggrmaxqueriesperip="0"
fi
# Min queries per IP, full D/L only
if [[ $mdaggrcountfull -gt "0" ]]; then
- mdaggrminqueriesperipfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | tail -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
+ mdaggrminqueriesperipfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | tail -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
else
mdaggrinqueriesperipfull="0"
fi
@@ -376,7 +383,7 @@ fi
# Max queries per IP, full D/L only
if [[ $mdaggrcountfull -gt "0" ]]; then
- mdaggrmaxqueriesperipfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | head -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
+ mdaggrmaxqueriesperipfull=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | head -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
else
mdaggrmaxqueriesperipfull="0"
fi
@@ -386,7 +393,7 @@ if [[ "$timeperiod" != "day" ]]; then
# Top 10 downloaders and how many downloads / total data shipped (full downloads only)
if [[ "$timeperiod" != "day" ]]; then
- mdaggrtoptenipsbycount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | head -10)
+ mdaggrtoptenipsbycount=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | head -10)
fi
#
@@ -408,7 +415,7 @@ if [[ "$timeperiod" != "day" ]]; then
countfriendly=$(echo $count | awk '{ printf ("%'"'"'d\n", $0) }')
# Figure out total traffic shipped to this IP
- totaldataforthisip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | grep $ipaddr | cut -f 10 -d " " | grep -v - | awk '{sum+=$1} END {print sum}')
+ totaldataforthisip=$(grep $apachesearchterm $logslocation/md/md1/metadata.uou-access_log* $logslocation/md/md2/metadata.uou-access_log* $logslocation/md/md3/metadata.uou-access_log* $logslocation/md/md-ne-01/metadata.uou-access_log* $logslocation/md/md-ne-02/metadata.uou-access_log* $logslocation/md/md-we-01/metadata.uou-access_log* $logslocation/md/md-we-02/metadata.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep ".xml" | grep -v 404 | grep "\" 200" | grep "GET" | grep $ipaddr | cut -f 10 -d " " | grep -v - | awk '{sum+=$1} END {print sum}')
if [[ "$totaldataforthisip" -gt "0" ]]; then
totaldataforthisiphr=$(bytestohr $totaldataforthisip)
else
@@ -436,11 +443,11 @@ fi
# =====
# MDQ requests
-mdqcount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep -v 404 | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | wc -l)
+mdqcount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep -v 404 | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | wc -l)
mdqcountfriendly=$(echo $mdqcount | awk '{ printf ("%'"'"'d\n", $0) }')
# MDQ downloads (i.e. HTTP 200 responses only)
-mdqcountfull=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | grep "GET" | wc -l)
+mdqcountfull=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | grep "GET" | wc -l)
mdqcountfullfriendly=$(echo $mdqcountfull | awk '{ printf ("%'"'"'d\n", $0) }')
# Percentage of HTTP 200 responses compared to total requests
@@ -451,7 +458,7 @@ else
fi
# Compressed downloads
-mdqfullcomprcount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | grep "GET" | grep "\"GZIP\"" | wc -l)
+mdqfullcomprcount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | grep "GET" | grep "\"GZIP\"" | wc -l)
mdqfullcomprcountfriendly=$(echo $mdqfullcomprcount | awk '{ printf ("%'"'"'d\n", $0) }')
# Percentage of GZIPPED HTTP 200 responses compared to total full downloads
@@ -465,10 +472,9 @@ fi
# IPv4 vs IPv6 traffic (don't calculate this for daily stats)
if [[ "$timeperiod" != "day" ]]; then
- # Note, while all v6 traffic passes through v6v4proxy1/2, we're counting accesses from the IPv4 addresses of those servers vs all others.
- # When we add "real" v6 support to the servers, this needs changing to count IPv4 addresses vs IPv6 addresses.
+ # Some v6 traffic has traditionally passed through v6v4proxy1/2, so to count v4 we're counting all accesses, minus those from the v4 proxy IP addresses, minus actual v6 addresses
if [[ "$mdqcount" -ne "0" ]]; then
- mdqv4count=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep -v 193.63.72.83 | grep -v 194.83.7.211 | wc -l)
+ mdqv4count=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep -v ":" | wc -l)
mdqv4pc=$(echo "scale=4;($mdqv4count/$mdqcount)*100" | bc | awk '{printf "%.1f\n", $0}')
mdqv6count=$(( mdqcount - mdqv4count ))
mdqv6pc=$(echo "scale=4;($mdqv6count/$mdqcount)*100" | bc | awk '{printf "%.1f\n", $0}')
@@ -479,8 +485,8 @@ if [[ "$timeperiod" != "day" ]]; then
fi
# MDQ requests for entityId based names
-mdqcountentityidhttp=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "/entities/http" | wc -l)
-mdqcountentityidurn=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "/entities/urn" | wc -l)
+mdqcountentityidhttp=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "/entities/http" | wc -l)
+mdqcountentityidurn=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep "/entities/urn" | wc -l)
mdqcountentityid=$((mdqcountentityidhttp+mdqcountentityidurn))
if [[ "$mdqcount" -ne "0" ]]; then
mdqcountentityidpc=$(echo "scale=3;($mdqcountentityid/$mdqcount)*100" | bc | awk '{printf "%.1f\n", $0}')
@@ -490,7 +496,7 @@ fi
mdqcountentityidfriendly=$(echo $mdqcountentityid | awk '{ printf ("%'"'"'d\n", $0) }')
# MDQ requests for hash based names
-mdqcountsha1=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep sha1 | wc -l)
+mdqcountsha1=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep sha1 | wc -l)
if [[ "$mdqcount" -ne "0" ]]; then
mdqcountsha1pc=$(echo "scale=3;($mdqcountsha1/$mdqcount)*100" | bc | awk '{printf "%.1f\n", $0}')
else
@@ -500,14 +506,14 @@ mdqcountsha1friendly=$(echo $mdqcountsha1 | awk '{ printf ("%'"'"'d\n", $0) }')
# MDQ requests for all entities
-mdqcountallentities=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities " | grep -v 404 | wc -l)
+mdqcountallentities=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities " | grep -v 404 | wc -l)
# Unique IP addresses requesting MDQ
-mdquniqueip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities/" | grep -v "/entities/ " | grep -v 404 | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq | wc -l)
+mdquniqueip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities/" | grep -v "/entities/ " | grep -v 404 | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l)
mdquniqueipfriendly=$(echo $mdquniqueip | awk '{ printf ("%'"'"'d\n", $0) }')
# Total data shipped
-mdqtotalbytes=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities/" | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | cut -f 10 -d " " | grep -v - | awk '{sum+=$1} END {print sum}')
+mdqtotalbytes=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities/" | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | cut -f 10 -d " " | grep -v - | awk '{sum+=$1} END {print sum}')
if [[ "$mdqtotalbytes" -gt "0" ]]; then
mdqtotalhr=$(bytestohr $mdqtotalbytes)
else
@@ -516,7 +522,7 @@ fi
# Min queries per IP
if [[ $mdqcount -gt "0" ]]; then
- mdqminqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | tail -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
+ mdqminqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | tail -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
else
mdqminqueriesperip="0"
fi
@@ -530,14 +536,14 @@ fi
# Max queries per IP
if [[ $mdqcount -gt "0" ]]; then
- mdqmaxqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | head -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
+ mdqmaxqueriesperip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | head -1 | awk '{print $1}' | awk '{ printf ("%'"'"'d\n", $0) }')
else
mdqmaxqueriesperip="0"
fi
if [[ "$timeperiod" != "day" ]]; then
# Top 10 downloaders and how many downloads / total data shipped
- mdqtoptenipsbycount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep "/entities" | grep -v "/entities/ " | grep -v 404 | grep -v "/entities/ " | cut -f 2 -d ":" | cut -f 1 -d " " | sort | uniq -c | sort -nr | head -10)
+ mdqtoptenipsbycount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep "/entities" | grep -v "/entities/ " | grep -v 404 | grep -v "/entities/ " | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq -c | sort -nr | head -10)
#
# Manipute results of the top 10
@@ -558,7 +564,7 @@ if [[ "$timeperiod" != "day" ]]; then
countfriendly=$(echo $count | awk '{ printf ("%'"'"'d\n", $0) }')
# Figure out total traffic shipped to this IP
- totaldataforthisip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities/" | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | grep $ipaddr | cut -f 10 -d " " | grep -v - | awk '{sum+=$1} END {print sum}')
+ totaldataforthisip=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities/" | grep -v "/entities/ " | grep -v 404 | grep "\" 200" | grep $ipaddr | cut -f 10 -d " " | grep -v - | awk '{sum+=$1} END {print sum}')
if [[ "$totaldataforthisip" -gt "0" ]]; then
totaldataforthisiphr=$(bytestohr $totaldataforthisip)
else
@@ -581,7 +587,7 @@ if [[ "$timeperiod" != "day" ]]; then
# Top 10 queries and how many downloads / total data shipped
- mdqtoptenqueriesbycount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep /entities/ | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | awk '{print $7}' | cut -f 3 -d "/" | sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b" | sort | uniq -c | sort -nr | head -10)
+ mdqtoptenqueriesbycount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep /entities/ | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | awk '{print $7}' | cut -f 3 -d "/" | sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b" | sort | uniq -c | sort -nr | head -10)
fi
# =====
@@ -589,32 +595,39 @@ fi
# =====
# How many accesses to .ds.
-cdscount=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | wc -l)
+cdscount=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* $logslocation/cds/shibcds-ne-01/ssl_access_log* $logslocation/cds/shibcds-ne-02/ssl_access_log* $logslocation/cds/shibcds-we-01/ssl_access_log* $logslocation/cds/shibcds-we-02/ssl_access_log* | grep .ds? | wc -l)
cdscountfriendly=$(echo $cdscount | awk '{ printf ("%'"'"'d\n", $0) }')
# IPv4 vs IPv6 traffic (don't count these for daily stats)
if [[ "$timeperiod" != "day" ]]; then
- # Note, while all v6 traffic passes through v6v4proxy1/2, we're counting accesses from the IPv4 addresses of those servers vs all others.
- # When we add "real" v6 support to the servers, this needs changing to count IPv4 addresses vs IPv6 addresses.
- cdsv4count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | grep -v 193.63.72.83 | grep -v 194.83.7.211 | wc -l)
+ # Some v6 traffic has traditionally passed through v6v4proxy1/2, so to count v4 we're counting all accesses, minus those from the v4 proxy IP addresses, minus actual v6 addresses
+ cdsv4count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* $logslocation/cds/shibcds-ne-01/ssl_access_log* $logslocation/cds/shibcds-ne-02/ssl_access_log* $logslocation/cds/shibcds-we-01/ssl_access_log* $logslocation/cds/shibcds-we-02/ssl_access_log* | grep .ds? | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep -v ":" | wc -l)
cdsv4pc=$(echo "scale=4;($cdsv4count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
cdsv6count=$(( cdscount - cdsv4count ))
cdsv6pc=$(echo "scale=4;($cdsv6count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
# Per-server request count
- cds1count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | grep shib-cds1 | wc -l)
+ cds1count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* | grep .ds? | wc -l)
cds1pc=$(echo "scale=4;($cds1count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
- cds2count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | grep shib-cds2 | wc -l)
+ cds2count=$(grep $apachesearchterm $logslocation/cds/shib-cds2/ssl_access_log* | grep .ds? | wc -l)
cds2pc=$(echo "scale=4;($cds2count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
- cds3count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | grep shib-cds3 | wc -l)
+ cds2count=$(grep $apachesearchterm $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | wc -l)
cds3pc=$(echo "scale=4;($cds3count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ cdsne01count=$(grep $apachesearchterm $logslocation/cds/shibcds-ne-01/ssl_access_log* | grep .ds? | wc -l)
+ cdsne01pc=$(echo "scale=4;($cdsne01count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ cdsne02count=$(grep $apachesearchterm $logslocation/cds/shibcds-ne-02/ssl_access_log* | grep .ds? | wc -l)
+ cdsne02pc=$(echo "scale=4;($cdsne02count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ cdswe01count=$(grep $apachesearchterm $logslocation/cds/shibcds-we-01/ssl_access_log* | grep .ds? | wc -l)
+ cdswe01pc=$(echo "scale=4;($cdswe01count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ cdswe02count=$(grep $apachesearchterm $logslocation/cds/shibcds-we-02/ssl_access_log* | grep .ds? | wc -l)
+ cdswe02pc=$(echo "scale=4;($cdswe02count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
fi
# How many of these were to the DS (has entityId in the parameters)
-cdsdscount=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | grep entityID | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
+cdsdscount=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* $logslocation/cds/shibcds-ne-01/ssl_access_log* $logslocation/cds/shibcds-ne-02/ssl_access_log* $logslocation/cds/shibcds-we-01/ssl_access_log* $logslocation/cds/shibcds-we-02/ssl_access_log* | grep .ds? | grep entityID | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
# How many of these were to the WAYF (has shire in the parameters)
-cdswayfcount=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | grep shire | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
+cdswayfcount=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* $logslocation/cds/shibcds-ne-01/ssl_access_log* $logslocation/cds/shibcds-ne-02/ssl_access_log* $logslocation/cds/shibcds-we-01/ssl_access_log* $logslocation/cds/shibcds-we-02/ssl_access_log* | grep .ds? | grep shire | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
# =====
@@ -706,7 +719,7 @@ else
msg+="--> * $mdaggrcountfullfriendly ($mdaggrfullpc%) were full downloads, of which $mdaggrcountfullcomprfriendly ($mdaggrfullcomprpc%) were compressed.\n"
msg+="--> ukfederation-metadata.xml: $mdaggrmaintotalhr of data actually shipped; would have been an estimated $mdaggrmaintotalestnocompresshr without compression, and $mdaggrmaintotalestnocompressnocgethr without compression or conditional gets.\n"
msg+="-> IPv4: $mdaggrv4pc% vs IPv6: $mdaggrv6pc%\n"
- msg+="-> Server distribution: md1: $mdaggrmd1pc% md2: $mdaggrmd2pc% md3: $mdaggrmd3pc%\n"
+ msg+="-> Server distribution: md-ne-01: $mdaggrmdne01pc% md-ne-02: $mdaggrmdne02pc% md-we-01: $mdaggrmdwe01pc% md-we-02: $mdaggrmdwe02pc% / md1: $mdaggrmd1pc% md2: $mdaggrmd2pc% md3: $mdaggrmd3pc%\n"
msg+="-> $mdaggrminqueriesperip/$mdaggravgqueriesperip/$mdaggrmaxqueriesperip min/avg/max queries per querying IP (all reqs)\n"
msg+="-> $mdaggrminqueriesperipfull/$mdaggravgqueriesperipfull/$mdaggrmaxqueriesperipfull min/avg/max queries per querying IP (full D/Ls only)\n"
msg+="\nRequests per published aggregate\n"
@@ -735,7 +748,7 @@ else
msg+="Central Discovery Service:\n"
msg+="-> $cdscountfriendly total requests serviced\n"
msg+="-> IPv4: $cdsv4pc% vs IPv6: $cdsv6pc%\n"
- msg+="-> Server distribution: shib-cds1: $cds1pc% shib-cds2: $cds2pc% shib-cds3: $cds3pc%\n"
+ msg+="-> Server distribution: shibcds-ne-01: $cdsne01pc% shibcds-ne-02: $cdsne021pc% shibcds-we-01: $cdswe01pc% shibcds-we-02: $cdswe02pc% / shib-cds1: $cds1pc% shib-cds2: $cds2pc% shib-cds3: $cds3pc%\n"
msg+="-> DS: $cdsdscount / WAYF: $cdswayfcount\n"
msg+="\n-----\n"
msg+="Wugen:\n"
From 2ab88a8a365772daf7279bf0c89d7ca477222aea Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Tue, 28 Mar 2017 17:55:17 +0100
Subject: [PATCH 31/80] Add website stats into stats generation
---
utilities/stats-generate.sh | 33 ++++++++++++++++++++++++++++++++-
1 file changed, 32 insertions(+), 1 deletion(-)
diff --git a/utilities/stats-generate.sh b/utilities/stats-generate.sh
index ac8ca846..bb79dcac 100755
--- a/utilities/stats-generate.sh
+++ b/utilities/stats-generate.sh
@@ -678,6 +678,31 @@ if [[ "$timeperiod" != "day" ]]; then
fi
+# =====
+# Website stats
+# =====
+
+# How many requests were there for the main content files?
+wwwaccesscount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
+
+# And from how many unique IdPs?
+wwwaccessipcount=$($apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
+
+# Don't count these when doing daily stats
+if [[ "$timeperiod" != "day" ]]; then
+
+ # Per-server request count
+ wwwaccessweb1count=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccessweb1pc=$(echo "scale=4;($wwwaccessweb1count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ wwwaccessweb2count=$(grep $apachesearchterm $logslocation/www/web2/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccessweb2pc=$(echo "scale=4;($wwwaccessweb2count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ wwwaccessne01count=$(grep $apachesearchterm $logslocation/www/www-ne-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccessne01pc=$(echo "scale=4;($wwwaccessne01count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+ wwwaccesswe01count=$(grep $apachesearchterm $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccesswe01pc=$(echo "scale=4;($wwwaccesswe01count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
+fi
+
+
# =====
# = Now we're ready to build the message. Different message for daily vs month/year
# =====
@@ -700,7 +725,9 @@ if [[ "$timeperiod" == "day" ]]; then
msg+=">*CDS:* $cdscountfriendly requests serviced (DS: $cdsdscount / WAYF: $cdswayfcount).\n"
msg+=">*Wugen:* $wugencount WAYFless URLs generated, $wugennewsubs new subscriptions.\n"
msg+=">*Test IdP:* $testidplogincount logins to $testidpspcount SPs.\n"
- msg+=">*Test SP:* $testsplogincount logins from $testspidpcount IdPs."
+ msg+=">*Test SP:* $testsplogincount logins from $testspidpcount IdPs.\n"
+ msg+=">*Website:* $wwwaccesscount hits from $wwwaccessipcount unique IPs."
+
else
#
@@ -766,6 +793,10 @@ else
msg+="-> $testsplogincount logins from $testspidpcount IdPs.\n"
msg+="\n-> Top 10 IdPs logged in from:\n"
msg+="$testsptoptenidpsbycount\n"
+ msg+="\n-----\n"
+ msg+="Website usage:\n"
+ msg+="-> $wwwaccesscount hits from $wwwaccessipcount unique IPs."
+ msg+="-> Server distribution: www-ne-01: $wwwaccessne01pc% www-we-01: $wwwaccesswe01pc% / web1: $wwwaccessweb1pc% web2: $wwwaccessweb2pc% \n"
msg+="\n-----"
fi
From dcc82b14562be30e3263c003893773e08e66dbbd Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Tue, 28 Mar 2017 19:02:02 +0100
Subject: [PATCH 32/80] Fix bugs in recent stats generation updates
---
utilities/stats-generate.sh | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/utilities/stats-generate.sh b/utilities/stats-generate.sh
index bb79dcac..afcfc1af 100755
--- a/utilities/stats-generate.sh
+++ b/utilities/stats-generate.sh
@@ -474,7 +474,7 @@ fi
if [[ "$timeperiod" != "day" ]]; then
# Some v6 traffic has traditionally passed through v6v4proxy1/2, so to count v4 we're counting all accesses, minus those from the v4 proxy IP addresses, minus actual v6 addresses
if [[ "$mdqcount" -ne "0" ]]; then
- mdqv4count=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep -v ":" | wc -l)
+ mdqv4count=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep "/entities" | grep -v "/entities " | grep -v "/entities/ " | grep -v 404 | cut -f 1 -d " " | cut -f 2-9 -d ":" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep -v ":" | wc -l)
mdqv4pc=$(echo "scale=4;($mdqv4count/$mdqcount)*100" | bc | awk '{printf "%.1f\n", $0}')
mdqv6count=$(( mdqcount - mdqv4count ))
mdqv6pc=$(echo "scale=4;($mdqv6count/$mdqcount)*100" | bc | awk '{printf "%.1f\n", $0}')
@@ -601,7 +601,7 @@ cdscountfriendly=$(echo $cdscount | awk '{ printf ("%'"'"'d\n", $0) }')
# IPv4 vs IPv6 traffic (don't count these for daily stats)
if [[ "$timeperiod" != "day" ]]; then
# Some v6 traffic has traditionally passed through v6v4proxy1/2, so to count v4 we're counting all accesses, minus those from the v4 proxy IP addresses, minus actual v6 addresses
- cdsv4count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* $logslocation/cds/shibcds-ne-01/ssl_access_log* $logslocation/cds/shibcds-ne-02/ssl_access_log* $logslocation/cds/shibcds-we-01/ssl_access_log* $logslocation/cds/shibcds-we-02/ssl_access_log* | grep .ds? | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep -v ":" | wc -l)
+ cdsv4count=$(grep $apachesearchterm $logslocation/cds/shib-cds1/ssl_access_log* $logslocation/cds/shib-cds2/ssl_access_log* $logslocation/cds/shib-cds3/ssl_access_log* $logslocation/cds/shibcds-ne-01/ssl_access_log* $logslocation/cds/shibcds-ne-02/ssl_access_log* $logslocation/cds/shibcds-we-01/ssl_access_log* $logslocation/cds/shibcds-we-02/ssl_access_log* | grep .ds? | cut -f 1 -d " " | cut -f 2-9 -d ":" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep -v ":" | wc -l)
cdsv4pc=$(echo "scale=4;($cdsv4count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
cdsv6count=$(( cdscount - cdsv4count ))
cdsv6pc=$(echo "scale=4;($cdsv6count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
@@ -611,7 +611,7 @@ if [[ "$timeperiod" != "day" ]]; then
cds1pc=$(echo "scale=4;($cds1count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
cds2count=$(grep $apachesearchterm $logslocation/cds/shib-cds2/ssl_access_log* | grep .ds? | wc -l)
cds2pc=$(echo "scale=4;($cds2count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
- cds2count=$(grep $apachesearchterm $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | wc -l)
+ cds3count=$(grep $apachesearchterm $logslocation/cds/shib-cds3/ssl_access_log* | grep .ds? | wc -l)
cds3pc=$(echo "scale=4;($cds3count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
cdsne01count=$(grep $apachesearchterm $logslocation/cds/shibcds-ne-01/ssl_access_log* | grep .ds? | wc -l)
cdsne01pc=$(echo "scale=4;($cdsne01count/$cdscount)*100" | bc | awk '{printf "%.1f\n", $0}')
@@ -686,7 +686,7 @@ fi
wwwaccesscount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
# And from how many unique IdPs?
-wwwaccessipcount=$($apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
+wwwaccessipcount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
# Don't count these when doing daily stats
if [[ "$timeperiod" != "day" ]]; then
@@ -775,7 +775,7 @@ else
msg+="Central Discovery Service:\n"
msg+="-> $cdscountfriendly total requests serviced\n"
msg+="-> IPv4: $cdsv4pc% vs IPv6: $cdsv6pc%\n"
- msg+="-> Server distribution: shibcds-ne-01: $cdsne01pc% shibcds-ne-02: $cdsne021pc% shibcds-we-01: $cdswe01pc% shibcds-we-02: $cdswe02pc% / shib-cds1: $cds1pc% shib-cds2: $cds2pc% shib-cds3: $cds3pc%\n"
+ msg+="-> Server distribution: shibcds-ne-01: $cdsne01pc% shibcds-ne-02: $cdsne02pc% shibcds-we-01: $cdswe01pc% shibcds-we-02: $cdswe02pc% / shib-cds1: $cds1pc% shib-cds2: $cds2pc% shib-cds3: $cds3pc%\n"
msg+="-> DS: $cdsdscount / WAYF: $cdswayfcount\n"
msg+="\n-----\n"
msg+="Wugen:\n"
@@ -795,7 +795,7 @@ else
msg+="$testsptoptenidpsbycount\n"
msg+="\n-----\n"
msg+="Website usage:\n"
- msg+="-> $wwwaccesscount hits from $wwwaccessipcount unique IPs."
+ msg+="-> $wwwaccesscount hits from $wwwaccessipcount unique IPs.\n"
msg+="-> Server distribution: www-ne-01: $wwwaccessne01pc% www-we-01: $wwwaccesswe01pc% / web1: $wwwaccessweb1pc% web2: $wwwaccessweb2pc% \n"
msg+="\n-----"
fi
From 0c1c2d0d90165ff762dd61f5d87b4f4016fe4991 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Tue, 28 Mar 2017 19:09:16 +0100
Subject: [PATCH 33/80] Fix another bug in stats generation updates
---
utilities/stats-generate.sh | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/utilities/stats-generate.sh b/utilities/stats-generate.sh
index afcfc1af..7295fb9c 100755
--- a/utilities/stats-generate.sh
+++ b/utilities/stats-generate.sh
@@ -683,7 +683,8 @@ fi
# =====
# How many requests were there for the main content files?
-wwwaccesscount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
+wwwaccesscount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+wwwaccesscountfriendly=$(echo $wwwaccesscount | awk '{ printf ("%'"'"'d\n", $0) }')
# And from how many unique IdPs?
wwwaccessipcount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
@@ -726,7 +727,7 @@ if [[ "$timeperiod" == "day" ]]; then
msg+=">*Wugen:* $wugencount WAYFless URLs generated, $wugennewsubs new subscriptions.\n"
msg+=">*Test IdP:* $testidplogincount logins to $testidpspcount SPs.\n"
msg+=">*Test SP:* $testsplogincount logins from $testspidpcount IdPs.\n"
- msg+=">*Website:* $wwwaccesscount hits from $wwwaccessipcount unique IPs."
+ msg+=">*Website:* $wwwaccesscountfriendly hits from $wwwaccessipcount unique IPs."
else
@@ -795,7 +796,7 @@ else
msg+="$testsptoptenidpsbycount\n"
msg+="\n-----\n"
msg+="Website usage:\n"
- msg+="-> $wwwaccesscount hits from $wwwaccessipcount unique IPs.\n"
+ msg+="-> $wwwaccesscountfriendly hits from $wwwaccessipcount unique IPs.\n"
msg+="-> Server distribution: www-ne-01: $wwwaccessne01pc% www-we-01: $wwwaccesswe01pc% / web1: $wwwaccessweb1pc% web2: $wwwaccessweb2pc% \n"
msg+="\n-----"
fi
From debb806fabdfba9f75778b99cdf68d7147dc9189 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Tue, 28 Mar 2017 21:38:10 +0100
Subject: [PATCH 34/80] Update stats generation to fix monthly mdq top 10 issue
---
utilities/stats-generate.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/utilities/stats-generate.sh b/utilities/stats-generate.sh
index 7295fb9c..8e09134b 100755
--- a/utilities/stats-generate.sh
+++ b/utilities/stats-generate.sh
@@ -587,7 +587,7 @@ if [[ "$timeperiod" != "day" ]]; then
# Top 10 queries and how many downloads / total data shipped
- mdqtoptenqueriesbycount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep -v 193.63.72.83 | grep -v 194.83.7.211 | grep /entities/ | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | awk '{print $7}' | cut -f 3 -d "/" | sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b" | sort | uniq -c | sort -nr | head -10)
+ mdqtoptenqueriesbycount=$(grep $apachesearchterm $logslocation/md/md1/mdq.uou-access_log* $logslocation/md/md2/mdq.uou-access_log* $logslocation/md/md3/mdq.uou-access_log* $logslocation/md/md-ne-01/mdq.uou-access_log* $logslocation/md/md-ne-02/mdq.uou-access_log* $logslocation/md/md-we-01/mdq.uou-access_log* $logslocation/md/md-we-02/mdq.uou-access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep /entities/ | grep -v 404 | grep -v "/entities/ " | grep -v "/entities/ " | awk '{print $7}' | cut -f 3 -d "/" | sed "s@+@ @g;s@%@\\\\x@g" | sort | uniq -c | xargs -0 printf "%b" | sort -nr | head -10)
fi
# =====
From 5fd9f02e178a728db6127b7a58840eca022ccc64 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Wed, 29 Mar 2017 11:39:04 +0100
Subject: [PATCH 35/80] Clarify past and ongoing intellectual property regimes
---
README.md | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 63fba15b..20dae1a5 100644
--- a/README.md
+++ b/README.md
@@ -18,9 +18,13 @@ The second main category excluded from the public repository is the historic reg
Since 2016, we have separated the entity database and aggregate record from the main toolchain repository, but the nature of Git is to never discard anything. We will therefore continue to make this repository publicly available only in redacted form.
-## Licensing
+## Copyright and License
-Everything in the public repository is Copyright (C) 2004–2015, University of Edinburgh. Each file is made available to you under the following terms:
+The contents of this repository are Copyright (C) the named contributors or their
+employers, as appropriate.
+
+In particular, all content authored prior to the 1st of August 2016 is
+Copyright (C) 2011—2016, University of Edinburgh.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
From fb323093c72ef7037dd7f49557af3126638f4e99 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Mon, 3 Apr 2017 10:46:38 +0100
Subject: [PATCH 36/80] Ignore common bots in website stats
---
utilities/stats-generate.sh | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/utilities/stats-generate.sh b/utilities/stats-generate.sh
index 8e09134b..a021ce27 100755
--- a/utilities/stats-generate.sh
+++ b/utilities/stats-generate.sh
@@ -682,24 +682,27 @@ fi
# Website stats
# =====
+# Set up grepping out bots
+botstringlist="(Googlebot|Bingbo|DuckDuckBot|Baiduspider|Yandexbot|Sogou|Exabot|AhrefsBot|seoscanners)"
+
# How many requests were there for the main content files?
-wwwaccesscount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+wwwaccesscount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Eiv "$botstringlist" | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
wwwaccesscountfriendly=$(echo $wwwaccesscount | awk '{ printf ("%'"'"'d\n", $0) }')
# And from how many unique IdPs?
-wwwaccessipcount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
+wwwaccessipcount=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* $logslocation/www/web2/ssl_access_log* $logslocation/www/www-ne-01/ssl_access_log* $logslocation/www/www-we-01/ssl_access_log* | grep -Eiv "$botstringlist" | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | cut -f 1 -d " " | cut -f 2-9 -d ":" | sort | uniq | wc -l | awk '{ printf ("%'"'"'d\n", $0) }')
# Don't count these when doing daily stats
if [[ "$timeperiod" != "day" ]]; then
# Per-server request count
- wwwaccessweb1count=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccessweb1count=$(grep $apachesearchterm $logslocation/www/web1/ssl_access_log* | grep -Eiv "$botstringlist" | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
wwwaccessweb1pc=$(echo "scale=4;($wwwaccessweb1count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
- wwwaccessweb2count=$(grep $apachesearchterm $logslocation/www/web2/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccessweb2count=$(grep $apachesearchterm $logslocation/www/web2/ssl_access_log* | grep -Eiv "$botstringlist" | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
wwwaccessweb2pc=$(echo "scale=4;($wwwaccessweb2count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
- wwwaccessne01count=$(grep $apachesearchterm $logslocation/www/www-ne-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccessne01count=$(grep $apachesearchterm $logslocation/www/www-ne-01/ssl_access_log* | grep -Eiv "$botstringlist" | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
wwwaccessne01pc=$(echo "scale=4;($wwwaccessne01count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
- wwwaccesswe01count=$(grep $apachesearchterm $logslocation/www/www-we-01/ssl_access_log* | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
+ wwwaccesswe01count=$(grep $apachesearchterm $logslocation/www/www-we-01/ssl_access_log* | grep -Eiv "$botstringlist" | grep -Ev "(Sensu-HTTP-Check|dummy|check_http|Balancer)" | grep 200 | grep "/content/" | wc -l)
wwwaccesswe01pc=$(echo "scale=4;($wwwaccesswe01count/$wwwaccesscount)*100" | bc | awk '{printf "%.1f\n", $0}')
fi
From caaa066bbdc81558423338cc57a34edef48285b3 Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Tue, 4 Apr 2017 17:02:41 +0100
Subject: [PATCH 37/80] Remove old MD servers from publishing pipeline
---
build.xml | 39 ---------------------------------------
preprod.properties | 3 ---
2 files changed, 42 deletions(-)
diff --git a/build.xml b/build.xml
index 7d58db56..dbbd1156 100644
--- a/build.xml
+++ b/build.xml
@@ -145,9 +145,6 @@
Metadata Distribution Service server properties.
-->
-
-
-
@@ -2108,24 +2105,6 @@
Push metadata files for the UK Federation to the MD dist servers
-->
Pushing UK Federation metadata files to MD dist.
- -> MD1
-
-
-
-
-
- -> MD2
-
-
-
-
-
- -> MD3
-
-
-
-
-
-> MD-NE-01
@@ -2157,24 +2136,6 @@
Push mdq cache tar to the MD dist servers
-->
Pushing UK Federation mdq cache to MD dist.
- -> MD1
-
-
-
-
-
- -> MD2
-
-
-
-
-
- -> MD3
-
-
-
-
-
-> MD-NE-01
diff --git a/preprod.properties b/preprod.properties
index 23489f03..26a0d200 100644
--- a/preprod.properties
+++ b/preprod.properties
@@ -35,9 +35,6 @@ git.repo.project.tooling=ukf-test-meta
#
# Preprod publishes its aggregates to / but accessible at a different hostname
#
-md.dist.host1.name=md1-test.infr.ukfederation.org.uk
-md.dist.host2.name=md2-test.infr.ukfederation.org.uk
-md.dist.host3.name=md3-test.infr.ukfederation.org.uk
md.dist.host-ne-01.name=md-ne-01-test.infr.ukfederation.org.uk
md.dist.host-ne-02.name=md-ne-02-test.infr.ukfederation.org.uk
md.dist.host-we-01.name=md-we-01-test.infr.ukfederation.org.uk
From d60507ddfcc89719b1ba776797eab43e6d18810b Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Wed, 5 Apr 2017 17:59:50 +0100
Subject: [PATCH 38/80] Remove old MD servers from Verify phase of build
process
---
build.xml | 48 ------------------------------------------------
1 file changed, 48 deletions(-)
diff --git a/build.xml b/build.xml
index dbbd1156..c277c79b 100644
--- a/build.xml
+++ b/build.xml
@@ -1270,54 +1270,6 @@
- Verifying metadata held at ${md.dist.host1.name}
-
-
-
-
-
-
-
-
- Verifying metadata held at ${md.dist.host2.name}
-
-
-
-
-
-
-
-
- Verifying metadata held at ${md.dist.host3.name}
-
-
-
-
-
-
-
-
Verifying metadata held at ${md.dist.host-ne-01.name}
From 723c01ff822fd4eed2a52df3bc95539c74d24516 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Mon, 10 Apr 2017 12:03:25 +0100
Subject: [PATCH 39/80] Switch to entity attribute blacklist for fallback
aggregate
Final phase of ukf/ukf-meta#10.
---
mdx/uk/generate.xml | 28 ----------------------------
1 file changed, 28 deletions(-)
diff --git a/mdx/uk/generate.xml b/mdx/uk/generate.xml
index 9a5bcb41..1d5af74d 100644
--- a/mdx/uk/generate.xml
+++ b/mdx/uk/generate.xml
@@ -219,33 +219,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
@@ -531,7 +504,6 @@
-
From 7bfc57b456340a1275fe669854d59f7b6500958b Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 11 Apr 2017 10:26:17 +0100
Subject: [PATCH 40/80] Check that ukfedlabel elements appear at most once per
entity
Resolves ukf/ukf-meta#42.
---
mdx/uk/beans.xml | 2 +
mdx/uk/check_ukfedlabel.xsl | 74 +++++++++++++++++++++++++++++++++++++
mdx/uk/check_ukreg.xsl | 12 ------
3 files changed, 76 insertions(+), 12 deletions(-)
create mode 100644 mdx/uk/check_ukfedlabel.xsl
diff --git a/mdx/uk/beans.xml b/mdx/uk/beans.xml
index eb185e63..e0f9e151 100644
--- a/mdx/uk/beans.xml
+++ b/mdx/uk/beans.xml
@@ -328,6 +328,8 @@
+
diff --git a/mdx/uk/check_ukfedlabel.xsl b/mdx/uk/check_ukfedlabel.xsl
new file mode 100644
index 00000000..8d236355
--- /dev/null
+++ b/mdx/uk/check_ukfedlabel.xsl
@@ -0,0 +1,74 @@
+
+
+
+
+
+
+
+
+
+
+
+
+ entity must not have more than one ukfedlabel:AccountableUsers element
+
+
+
+
+
+
+ entity must not have more than one ukfedlabel:ExportOptIn element
+
+
+
+
+
+
+ entity must not have more than one ukfedlabel:ExportOptOut element
+
+
+
+
+
+
+ entity must not have more than one ukfedlabel:Software element
+
+
+
+
+
+
+ entity must not have more than one ukfedlabel:UKFederationMember element
+
+
+
+
+
+
+
+
+
+ entity cannot be both opted in to and opted out from export
+
+
+
+
+
diff --git a/mdx/uk/check_ukreg.xsl b/mdx/uk/check_ukreg.xsl
index 728c0306..28434419 100644
--- a/mdx/uk/check_ukreg.xsl
+++ b/mdx/uk/check_ukreg.xsl
@@ -50,16 +50,4 @@
-
-
-
-
-
- entity cannot be both opted in to and opted out from export
-
-
-
-
From 9998168a0888a7bc75d538afa5a939b553487061 Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 11 Apr 2017 11:09:12 +0100
Subject: [PATCH 41/80] Remove statistics for Shibboleth 1.3 entities now there
are none
---
mdx/uk/statistics.xsl | 110 +-----------------------------------------
1 file changed, 2 insertions(+), 108 deletions(-)
diff --git a/mdx/uk/statistics.xsl b/mdx/uk/statistics.xsl
index ba84c359..7361c689 100644
--- a/mdx/uk/statistics.xsl
+++ b/mdx/uk/statistics.xsl
@@ -125,7 +125,6 @@
Entities by Owner
Identity Provider Accountability
Members Lacking Deployment
- Shibboleth 1.3 Remnants
Export Aggregate: Entities Opted Out
Export Aggregate: Entities Explicitly Opted In
Charting Statistics
@@ -922,33 +921,6 @@
-
-
-
- The following lists show entities that are believed to be running the
- Shibboleth 1.3 software, which reached its official end of life
- date on 30-June-2010.
- As heuristics have been used to create these lists, they may
- not be completely accurate.
-
-
- Shibboleth 1.3 Identity Provider Entities
-
-
-
-
- Shibboleth 1.3 Service Provider Entities
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- entities:
-
-
-
- -
- :
-
-
-
- [HasEncKey]
-
- (
-
- )
-
-
-
-
-
@@ -1707,25 +1623,10 @@
-
-
-
-
-
-
+
Shibboleth 2.x
-
-
-
- Shibboleth 1.3
-
-
-
-
+
Shibboleth combined
From 63f37ecd2aedd2dc1265a838d7b21214485fa04c Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Tue, 11 Apr 2017 11:27:57 +0100
Subject: [PATCH 42/80] Remove charting statistics from main stats page
See ukf/ukf-meta#106.
---
mdx/uk/statistics.xsl | 61 +------------------------------------------
1 file changed, 1 insertion(+), 60 deletions(-)
diff --git a/mdx/uk/statistics.xsl b/mdx/uk/statistics.xsl
index 7361c689..b1e51d68 100644
--- a/mdx/uk/statistics.xsl
+++ b/mdx/uk/statistics.xsl
@@ -127,7 +127,6 @@
Members Lacking Deployment
Export Aggregate: Entities Opted Out
Export Aggregate: Entities Explicitly Opted In
- Charting Statistics
Entities Without SAML 2.0 Support
@@ -1028,65 +1027,7 @@
-
-
-
-
-
- - Members:
- -
- Outsourcing chart:
- ,
- ,
- ,
- ,
-
-
- - Entities:
- - IdPs:
- - SPs:
- - Entities per member:
-
-
- -
- Algorithm support:
-
- of all entities
-
-
-
- -
- GCM support:
-
- of all entities
-
-
-
- -
- Algorithm support:
-
- of SP entities
-
-
-
- -
- Shibboleth IdP v3:
-
- ( of IdPs)
-
-
+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
Shibboleth 2.x
-
-
-
- Shibboleth 1.3
-
-
-
-
+
Shibboleth combined
From d0785b451e5503e8efce1737447310fd228e6e9a Mon Sep 17 00:00:00 2001
From: Rhys Smith
Date: Thu, 13 Apr 2017 09:20:36 +0100
Subject: [PATCH 45/80] Remove old servers from Stats syncing
---
utilities/stats-sync.sh | 8 --------
1 file changed, 8 deletions(-)
diff --git a/utilities/stats-sync.sh b/utilities/stats-sync.sh
index 81172ff9..774f5ddd 100755
--- a/utilities/stats-sync.sh
+++ b/utilities/stats-sync.sh
@@ -10,26 +10,18 @@ logslocation="/var/stats"
# Logs from API
# Logs from MD servers
-rsync -at --exclude modsec* stats@md1:/var/log/httpd/* $logslocation/md/md1/
-rsync -at --exclude modsec* stats@md2:/var/log/httpd/* $logslocation/md/md2/
-rsync -at --exclude modsec* stats@md3:/var/log/httpd/* $logslocation/md/md3/
rsync -at --exclude modsec* stats@md-ne-01:/var/log/httpd/* $logslocation/md/md-ne-01/
rsync -at --exclude modsec* stats@md-ne-02:/var/log/httpd/* $logslocation/md/md-ne-02/
rsync -at --exclude modsec* stats@md-we-01:/var/log/httpd/* $logslocation/md/md-we-01/
rsync -at --exclude modsec* stats@md-we-02:/var/log/httpd/* $logslocation/md/md-we-02/
# Logs from CDS servers
-rsync -at --exclude modsec* stats@shib-cds1:/var/log/httpd/* $logslocation/cds/shib-cds1/
-rsync -at --exclude modsec* stats@shib-cds2:/var/log/httpd/* $logslocation/cds/shib-cds2/
-rsync -at --exclude modsec* stats@shib-cds3:/var/log/httpd/* $logslocation/cds/shib-cds3/
rsync -at --exclude modsec* stats@shibcds-ne-01:/var/log/httpd/* $logslocation/cds/shibcds-ne-01/
rsync -at --exclude modsec* stats@shibcds-ne-02:/var/log/httpd/* $logslocation/cds/shibcds-ne-02/
rsync -at --exclude modsec* stats@shibcds-we-01:/var/log/httpd/* $logslocation/cds/shibcds-we-01/
rsync -at --exclude modsec* stats@shibcds-we-02:/var/log/httpd/* $logslocation/cds/shibcds-we-02/
# Logs from websites
-rsync -at --exclude modsec* stats@web1:/var/log/httpd/* $logslocation/www/web1/
-rsync -at --exclude modsec* stats@web2:/var/log/httpd/* $logslocation/www/web2/
rsync -at --exclude modsec* stats@www-ne-01:/var/log/httpd/* $logslocation/www/www-ne-01/
rsync -at --exclude modsec* stats@www-we-01:/var/log/httpd/* $logslocation/www/www-we-01/
From 13502945c2230e72b015a2b159b82b228fa5438a Mon Sep 17 00:00:00 2001
From: Ian Young
Date: Mon, 1 May 2017 10:22:01 +0100
Subject: [PATCH 46/80] Separate out definitions of MDA beans
All Shibboleth MDA abstract beans are now defined in mda-beans.xml,
which is a first cut of a resource planned for inclusion in MDA 0.10.0.
---
mdx/at_aconet/beans.xml | 4 +-
mdx/at_aconet/verbs.xml | 17 ++-
mdx/common-beans.xml | 270 +++++++++----------------------------
mdx/int_cobweb/beans.xml | 2 +-
mdx/int_cobweb/verbs.xml | 6 +-
mdx/int_edugain/beans.xml | 6 +-
mdx/int_edugain/verbs.xml | 22 +--
mdx/mda-beans.xml | 272 ++++++++++++++++++++++++++++++++++++++
mdx/test/beans.xml | 2 +-
mdx/test/verbs.xml | 4 +-
mdx/uk/beans.xml | 57 ++++----
mdx/uk/collect.xml | 2 +-
mdx/uk/generate.xml | 83 ++++++------
mdx/uk/mdq-multisign.xml | 16 +--
mdx/uk/verbs.xml | 34 ++---
mdx/us_incommon/beans.xml | 12 +-
mdx/us_incommon/verbs.xml | 8 +-
mdx/validation-beans.xml | 123 +++++++++--------
18 files changed, 529 insertions(+), 411 deletions(-)
create mode 100644 mdx/mda-beans.xml
diff --git a/mdx/at_aconet/beans.xml b/mdx/at_aconet/beans.xml
index 15d9c658..4d822ba2 100644
--- a/mdx/at_aconet/beans.xml
+++ b/mdx/at_aconet/beans.xml
@@ -95,7 +95,7 @@
-
+
@@ -117,7 +117,7 @@
-
+
diff --git a/mdx/at_aconet/verbs.xml b/mdx/at_aconet/verbs.xml
index 4b5a334a..9e77f6de 100644
--- a/mdx/at_aconet/verbs.xml
+++ b/mdx/at_aconet/verbs.xml
@@ -30,7 +30,7 @@
-
+
@@ -41,7 +41,7 @@
-
+
@@ -50,7 +50,7 @@
-
+
@@ -61,7 +61,7 @@
-
+
@@ -70,17 +70,17 @@
-
+
-
+
-
@@ -90,8 +90,7 @@
Remove a specific entity we know has a problem that it will take
a while to resolve.
-->
-
diff --git a/mdx/common-beans.xml b/mdx/common-beans.xml
index 0f2d3602..9a541410 100644
--- a/mdx/common-beans.xml
+++ b/mdx/common-beans.xml
@@ -19,6 +19,11 @@
-->
+
+
+
-
-
-
+
http://www.w3.org/2001/04/xmldsig-more#md5
@@ -117,57 +118,15 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
@@ -176,19 +135,10 @@
Parent for DOM resource source stages.
-->
-
+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
-
-
-
-
-
+
-
-
-
-
-
-
-
-
-
-
-
@@ -403,7 +268,7 @@
Remove the algorithm support namespace.
-->
-
-
-
-
-
-
-
-
-
+
@@ -626,8 +490,7 @@
A pipeline stage that logs any errors and warnings present,
but takes no action on them.
-->
-
+
@@ -642,8 +505,7 @@
This pipeline stage removes any items marked with an error status.
-->
-
+
@@ -657,8 +519,7 @@
This pipeline stage causes CLI termination if any item is marked with an error status.
-->
-
+
@@ -673,7 +534,7 @@
Announce any errors or warnings encountered, then remove
any items that had errors. Items with just warnings are retained.
-->
-
+
@@ -689,7 +550,7 @@
Warnings are not announced, and do not cause termination.
-->
-
+
@@ -726,25 +587,22 @@
-
+
-
+
-
+
-
@@ -752,43 +610,43 @@
Beans to strip out selected SAML metadata elements.
-->
-
-
-
-
-
-
-
-
-
-
@@ -805,8 +663,7 @@
-
+
-
@@ -838,7 +695,7 @@
-
@@ -847,7 +704,7 @@
Remove all MDUI metadata from attribute authority roles.
-->
-
-
-
-
+
@@ -886,7 +743,7 @@
-
@@ -905,7 +762,7 @@
Remove all ds:KeyName elements.
-->
-
@@ -1081,8 +938,7 @@
A pipeline stage that checks against all the common schemas, as above.
-->
-
+
@@ -1092,7 +948,7 @@
A pipeline stage that removes all XML comments from items.
-->
-
-
+
-
+
@@ -1130,7 +986,7 @@
presented, for example by removing redundant attributes or elements which only have
meaning when added by the UK federation registrar.
-->
-
-
+
@@ -1165,7 +1021,7 @@
errors. No announcement or removal of those entities is performed here;
that is left to the caller.
-->
-
+
@@ -1175,7 +1031,7 @@
Strip all elements and attributes that are in namespaces
other than the ones we accept from partners.
-->
-
@@ -1209,14 +1065,14 @@
-
+
-
-
+
-
+
diff --git a/mdx/int_cobweb/beans.xml b/mdx/int_cobweb/beans.xml
index 2922f2fc..3d8e2f4d 100644
--- a/mdx/int_cobweb/beans.xml
+++ b/mdx/int_cobweb/beans.xml
@@ -47,7 +47,7 @@
-
+
diff --git a/mdx/int_cobweb/verbs.xml b/mdx/int_cobweb/verbs.xml
index dde7a217..2cadfed2 100644
--- a/mdx/int_cobweb/verbs.xml
+++ b/mdx/int_cobweb/verbs.xml
@@ -30,7 +30,7 @@
-
+
@@ -41,7 +41,7 @@
-
+
@@ -51,7 +51,7 @@
-
+
diff --git a/mdx/int_edugain/beans.xml b/mdx/int_edugain/beans.xml
index 05b7c9b3..976fef1d 100644
--- a/mdx/int_edugain/beans.xml
+++ b/mdx/int_edugain/beans.xml
@@ -70,14 +70,14 @@
-
-
+
@@ -107,7 +107,7 @@
-
+
diff --git a/mdx/int_edugain/verbs.xml b/mdx/int_edugain/verbs.xml
index e6cd01ce..36d62cd6 100644
--- a/mdx/int_edugain/verbs.xml
+++ b/mdx/int_edugain/verbs.xml
@@ -30,7 +30,7 @@
-
+
@@ -41,7 +41,7 @@
-
+
@@ -50,7 +50,7 @@
-
+
@@ -69,11 +69,11 @@
blacklist are ignored, so that we only need to deal with each entity
entering an error state once.
-->
-
+
-
@@ -88,11 +88,11 @@
Live verify, but adding "new" checks that we don't impose on the live import.
-->
-
+
-
@@ -108,7 +108,7 @@
Looks for eduGAIN entities which *were* in an error state, as shown by
their inclusion in our verification blacklist, but have now recovered.
-->
-
+
@@ -118,12 +118,12 @@
-
-
@@ -140,7 +140,7 @@
Output also includes any warnings attached to entities, although
these do not result in an error termination.
-->
-
+
diff --git a/mdx/mda-beans.xml b/mdx/mda-beans.xml
new file mode 100644
index 00000000..ea592084
--- /dev/null
+++ b/mdx/mda-beans.xml
@@ -0,0 +1,272 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/mdx/test/beans.xml b/mdx/test/beans.xml
index 2795c95b..93c69d81 100644
--- a/mdx/test/beans.xml
+++ b/mdx/test/beans.xml
@@ -21,7 +21,7 @@
-
+
diff --git a/mdx/test/verbs.xml b/mdx/test/verbs.xml
index f5f692df..6f332c87 100644
--- a/mdx/test/verbs.xml
+++ b/mdx/test/verbs.xml
@@ -30,7 +30,7 @@
-
+
@@ -41,7 +41,7 @@
-
+
diff --git a/mdx/uk/beans.xml b/mdx/uk/beans.xml
index e0f9e151..1ff7171b 100644
--- a/mdx/uk/beans.xml
+++ b/mdx/uk/beans.xml
@@ -87,8 +87,7 @@
yet expired. Sets a bound of 30 days on the validity interval; 14 days is the
expected value.
-->
-
+
@@ -108,8 +107,7 @@
Each fragment file contains a single EntityDescriptor. The name of each
eligible file matches a particular regular expression.
-->
-
+
@@ -165,7 +163,7 @@
Adds "mailto:" to md:EmailAddress elements which don't already have it.
-->
-
@@ -174,7 +172,7 @@
Checks specific to the UK registrar function.
-->
-
-
-
-
-
@@ -219,7 +217,7 @@
This stage performs any standard cleanup required for UK federation fragment files.
-->
-
@@ -228,8 +226,7 @@
Remove any md:ContactPerson elements with contactType of "administrative".
-->
-
+
administrative
@@ -251,8 +248,7 @@
Name attribute is set to the federation URI. UK ordering is applied.
-->
-
+
@@ -265,8 +261,7 @@
Name attribute is not set. UK ordering is applied.
-->
-
+
@@ -276,7 +271,7 @@
-
+
@@ -286,7 +281,7 @@
Make all three potential scope lists equivalent (on the entity, on
the IDPSSODescriptor and on the AttributeAuthority).
-->
-
-
+
-
+
-
+
-
@@ -340,14 +335,14 @@
-
+
-
-
+
@@ -381,7 +376,7 @@
Strip those UK federation extensions which we never publish.
-->
-
@@ -400,7 +395,7 @@
used in an XML document. This one is UK-specific, as it makes specific choices
in order to limit the number of prefixes used.
-->
-
-