From f8a73c41ca450fd7ac24969b9533f06647360e9f Mon Sep 17 00:00:00 2001
From: Tom Scavo <trscavo@internet2.edu>
Date: Sat, 10 Jun 2017 21:45:23 -0400
Subject: [PATCH 1/7] Update list of installed files
---
README.md | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/README.md b/README.md
index eb7a1be..4117de8 100644
--- a/README.md
+++ b/README.md
@@ -25,11 +25,15 @@ A given target directory will be created if one doesn't already exist. In any ca
```Shell
$ ls -1 $BIN_DIR
cget.sh
+compute_md_vital_stats.sh
+request_http_resources.sh
+test_compressed_resources.sh
$ ls -1 $LIB_DIR
compatible_date.sh
config_tools.sh
core_lib.sh
+entities_timestamps_txt.xsl
entity_endpoints_txt.xsl
entity_identifiers_txt.xsl
entity_idp_names_txt.xsl
From e945268eeec434872487783703fbff6ccd3c86dd Mon Sep 17 00:00:00 2001
From: Tom Scavo <trscavo@internet2.edu>
Date: Sat, 10 Jun 2017 22:02:33 -0400
Subject: [PATCH 2/7] Add Environment section
---
README.md | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/README.md b/README.md
index 4117de8..cfef3c4 100644
--- a/README.md
+++ b/README.md
@@ -43,6 +43,18 @@ md_tools.sh
saml_tools.sh
```
+## Environment
+
+Besides `BIN_DIR` and `LIB_DIR` above, the following environment variables are required:
+
+| Variable | |
+| --- | --- |
+| `CACHE_DIR` | A persistent HTTP cache |
+| `TMPDIR` | A temporary directory |
+| `LOG_FILE` | A persistent log file |
+
+Some OSes define `TMPDIR` and some do not. In any case, a temporary directory is required to use these scripts.
+
## Overview
### `cget.sh`
From 51738e178c7ce821ede7ffcf853a179ac222843f Mon Sep 17 00:00:00 2001
From: Tom Scavo <trscavo@internet2.edu>
Date: Sat, 10 Jun 2017 22:11:53 -0400
Subject: [PATCH 3/7] Add Logging section
---
README.md | 23 ++++++++++++++---------
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git a/README.md b/README.md
index cfef3c4..5e1fdfd 100644
--- a/README.md
+++ b/README.md
@@ -45,23 +45,22 @@ saml_tools.sh
## Environment
-Besides `BIN_DIR` and `LIB_DIR` above, the following environment variables are required:
+Besides `BIN_DIR` and `LIB_DIR` above, the scripts leverage the following environment variables:
| Variable | |
| --- | --- |
| `CACHE_DIR` | A persistent HTTP cache |
| `TMPDIR` | A temporary directory |
| `LOG_FILE` | A persistent log file |
+| `LOG_LEVEL` | The global log level [0..5] |
-Some OSes define `TMPDIR` and some do not. In any case, a temporary directory is required to use these scripts.
+All but `LOG_LEVEL` are REQUIRED. See the following section for more info about logging.
-## Overview
+Note: Some OSes define `TMPDIR` and some do not. In any case, a temporary directory is required to use these scripts.
-### `cget.sh`
+## Logging
-Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead.
-
-First let's configure logging. For convenience, we will log directly to the terminal:
+For convenience, we will log directly to the terminal:
```Shell
$ export LOG_FILE=/dev/tty
@@ -79,9 +78,15 @@ Various log levels are supported:
| ERROR | 1 |
| FATAL | 0 |
-The default logging level is INFO (i.e., if you do not explicitly set the `LOG_LEVEL` environment variable, the value `LOG_LEVEL=3` is assumed by default).
+The default logging level is INFO (i.e., if you do not explicitly set the optional `LOG_LEVEL` environment variable, the value `LOG_LEVEL=3` is assumed by default).
+
+## Overview
+
+### `cget.sh`
+
+Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead.
-Next we define a couple of HTTP resources and a cache:
+First define a couple of HTTP resources and a cache:
```Shell
$ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
From f4de339dafd0cdce1c9b84555e590abf0ef68253 Mon Sep 17 00:00:00 2001
From: Tom Scavo <trscavo@internet2.edu>
Date: Sat, 10 Jun 2017 22:31:08 -0400
Subject: [PATCH 4/7] Add example to Overview section
---
README.md | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 53 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 5e1fdfd..2832297 100644
--- a/README.md
+++ b/README.md
@@ -60,7 +60,7 @@ Note: Some OSes define `TMPDIR` and some do not. In any case, a temporary direct
## Logging
-For convenience, we will log directly to the terminal:
+For convenience, we will log directly to the terminal in the examples below:
```Shell
$ export LOG_FILE=/dev/tty
@@ -82,6 +82,50 @@ The default logging level is INFO (i.e., if you do not explicitly set the option
## Overview
+First define a couple of HTTP resources and a cache:
+
+```Shell
+$ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
+$ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
+$ export CACHE_DIR=/tmp/http_cache
+```
+
+### `request_http_resources.sh`
+
+Bash script `request_http_resources.sh` issues a HEAD request for one or more HTTP resources:
+
+```Shell
+$ $BIN_DIR/request_http_resources.sh -d /tmp/out $url1 $url2
+2017-06-11T02:26:26Z INFO request_http_resources.sh BEGIN
+2017-06-11T02:26:26Z INFO request_http_resources.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
+2017-06-11T02:26:26Z INFO request_http_resources.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
+2017-06-11T02:26:26Z INFO request_http_resources.sh writing output file: http_response_headers.json
+2017-06-11T02:26:26Z INFO request_http_resources.sh moving output file to dir: /tmp/out
+2017-06-11T02:26:26Z INFO request_http_resources.sh END
+```
+
+The script produces a JSON array containing objects such as the following:
+
+```JavaScript
+ {
+ "successFlag": true,
+ "message": "Resource request successful",
+ "location": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml",
+ "ResponseCode": "200",
+ "Date": "Sun, 11 Jun 2017 02:16:44 GMT",
+ "LastModified": "Fri, 09 Jun 2017 19:05:16 GMT",
+ "ETag": "\"2a96bf6-5518ba65a0c50\"",
+ "ContentLength": "44657654",
+ "ContentType": "application/samlmetadata+xml"
+ }
+```
+
+See the inline help file for details:
+
+```Shell
+$ $BIN_DIR/request_http_resources.sh -h
+```
+
### `cget.sh`
Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead.
@@ -182,6 +226,14 @@ See the inline help file for details:
$ $BIN_DIR/cget.sh -h
```
+### `compute_md_vital_stats.sh`
+
+TBD
+
+### `test_compressed_resources.sh`
+
+TBD
+
## Compatibility
Shell scripts are compatible with both GNU/Linux and Mac OS. XSLT scripts are written in XSLT 1.0.
From e4eb7ef536fac7aec0b15a9f034d3f4dda5c2c9e Mon Sep 17 00:00:00 2001
From: Tom Scavo <trscavo@internet2.edu>
Date: Sun, 11 Jun 2017 09:17:22 -0400
Subject: [PATCH 5/7] Added a new example to the Overview section
---
README.md | 47 +++++++++++++++++++++++++++++++++++++++--------
1 file changed, 39 insertions(+), 8 deletions(-)
diff --git a/README.md b/README.md
index 2832297..0b87310 100644
--- a/README.md
+++ b/README.md
@@ -49,14 +49,14 @@ Besides `BIN_DIR` and `LIB_DIR` above, the scripts leverage the following enviro
| Variable | |
| --- | --- |
-| `CACHE_DIR` | A persistent HTTP cache |
+| `CACHE_DIR` | A persistent HTTP cache directory |
| `TMPDIR` | A temporary directory |
| `LOG_FILE` | A persistent log file |
| `LOG_LEVEL` | The global log level [0..5] |
All but `LOG_LEVEL` are REQUIRED. See the following section for more info about logging.
-Note: Some OSes define `TMPDIR` and some do not. In any case, a temporary directory is required to use these scripts.
+Note: Some OSes define `TMPDIR` and some do not. In any case, a temporary directory by that name is required to use these scripts.
## Logging
@@ -82,12 +82,11 @@ The default logging level is INFO (i.e., if you do not explicitly set the option
## Overview
-First define a couple of HTTP resources and a cache:
+The following HTTP resources are used repeatedly in the examples below:
```Shell
$ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
$ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
-$ export CACHE_DIR=/tmp/http_cache
```
### `request_http_resources.sh`
@@ -130,11 +129,9 @@ $ $BIN_DIR/request_http_resources.sh -h
Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead.
-First define a couple of HTTP resources and a cache:
+First define a cache:
```Shell
-$ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
-$ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
$ export CACHE_DIR=/tmp/http_cache
```
@@ -228,7 +225,41 @@ $ $BIN_DIR/cget.sh -h
### `compute_md_vital_stats.sh`
-TBD
+Bash script `compute_md_vital_stats.sh` parses one or more previously cached SAML metadata files:
+
+```Shell
+$ $BIN_DIR/compute_md_vital_stats.sh -d /tmp/out $url1 $url2
+2017-06-11T13:09:53Z INFO compute_md_vital_stats.sh BEGIN
+2017-06-11T13:09:53Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
+2017-06-11T13:09:53Z INFO compute_md_vital_stats.sh parsing cached metadata file: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
+2017-06-11T13:09:54Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
+2017-06-11T13:09:54Z INFO compute_md_vital_stats.sh parsing cached metadata file: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
+2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh writing output file: md_vital_statistics.json
+2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh moving output file to dir: /tmp/out
+2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh END
+```
+
+The script produces a JSON array containing objects such as the following:
+
+```JavaScript
+ {
+ "successFlag": true,
+ "message": "Metadata successfully parsed",
+ "metadataLocation": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml",
+ "currentTime": "2017-06-11T13:09:54Z",
+ "validUntil": "2017-06-23T18:56:31Z",
+ "creationInstant": "2017-06-09T18:56:31Z",
+ "validityInterval": "P14DT0H0M0S",
+ "untilInvalid": "P12DT5H46M37S",
+ "sinceCreation": "P1DT18H13M23S"
+ }
+```
+
+Note that this script depends on cached metadata. It will not fetch a metadata file from the server. See the inline help file for details:
+
+```Shell
+$ $BIN_DIR/compute_md_vital_stats.sh -h
+```
### `test_compressed_resources.sh`
From 6c2101aad479152f783f755740b1258d3eb9aa8a Mon Sep 17 00:00:00 2001
From: Tom Scavo <trscavo@internet2.edu>
Date: Sun, 11 Jun 2017 09:30:11 -0400
Subject: [PATCH 6/7] Added a new example to the Overview section
---
README.md | 37 ++++++++++++++++++++++++++++++++++++-
1 file changed, 36 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 0b87310..8c8f30a 100644
--- a/README.md
+++ b/README.md
@@ -263,7 +263,42 @@ $ $BIN_DIR/compute_md_vital_stats.sh -h
### `test_compressed_resources.sh`
-TBD
+Bash script `test_compressed_resources.sh` tests one or more resources for HTTP Compression:
+
+```Shell
+$ $BIN_DIR/test_compressed_resources.sh -d /tmp/out $url1 $url2
+2017-06-11T13:20:39Z INFO test_compressed_resources.sh BEGIN
+2017-06-11T13:20:39Z INFO test_compressed_resources.sh requesting (compressed) resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
+2017-06-11T13:20:43Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
+2017-06-11T13:20:43Z INFO test_compressed_resources.sh requesting (compressed) resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
+2017-06-11T13:20:45Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
+2017-06-11T13:20:46Z INFO test_compressed_resources.sh writing output file: compressed_response_headers.json
+2017-06-11T13:20:46Z INFO test_compressed_resources.sh moving output file to dir: /tmp/out
+2017-06-11T13:20:46Z INFO test_compressed_resources.sh END
+```
+
+The script produces a JSON array containing objects such as the following:
+
+```JavaScript
+ {
+ "successFlag": true,
+ "message": "Integrity of compressed metadata confirmed",
+ "location": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml",
+ "ResponseCode": "200",
+ "Date": "Sun, 11 Jun 2017 13:10:37 GMT",
+ "LastModified": "Fri, 09 Jun 2017 19:05:16 GMT",
+ "ETag": "\"80bc05-5518ba65a1fd5\"",
+ "ContentLength": "8436741",
+ "ContentType": "application/samlmetadata+xml",
+ "ContentEncoding": "gzip"
+ }
+```
+
+Like the previous script, this script also depends on cached metadata. In addition, this script requests each (compressed) resource from the server just-in-time. See the inline help file for details:
+
+```Shell
+$ $BIN_DIR/test_compressed_resources.sh -h
+```
## Compatibility
From 32e8fa5dff4fbb392386582fd520aa043b3078dc Mon Sep 17 00:00:00 2001
From: Tom Scavo <trscavo@internet2.edu>
Date: Sun, 11 Jun 2017 09:38:34 -0400
Subject: [PATCH 7/7] Minor edits
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 8c8f30a..0609502 100644
--- a/README.md
+++ b/README.md
@@ -119,7 +119,7 @@ The script produces a JSON array containing objects such as the following:
}
```
-See the inline help file for details:
+The JSON file is written to the output directory specified on the command line. See the inline help file for details:
```Shell
$ $BIN_DIR/request_http_resources.sh -h
@@ -294,7 +294,7 @@ The script produces a JSON array containing objects such as the following:
}
```
-Like the previous script, this script also depends on cached metadata. In addition, this script requests each (compressed) resource from the server just-in-time. See the inline help file for details:
+Like the previous script, this script also depends on cached metadata. In addition, this script requests (compressed) resources from the server just-in-time. See the inline help file for details:
```Shell
$ $BIN_DIR/test_compressed_resources.sh -h