Bash Library

A library of re-usable bash scripts

Installation

Download the source, change directory to the source directory, and install the source into /tmp as follows:

$ export BIN_DIR=/tmp/bin
$ export LIB_DIR=/tmp/lib
$ ./install.sh $BIN_DIR $LIB_DIR

or install into your home directory:

$ export BIN_DIR=$HOME/bin
$ export LIB_DIR=$HOME/lib
$ ./install.sh $BIN_DIR $LIB_DIR

A given target directory will be created if one doesn't already exist. In any case, the following files will be installed:

$ ls -1 $BIN_DIR
cget.sh
compute_md_vital_stats.sh
request_http_resources.sh
test_compressed_resources.sh

$ ls -1 $LIB_DIR 
compatible_date.sh
config_tools.sh
core_lib.sh
entities_timestamps_txt.xsl
entity_endpoints_txt.xsl
entity_identifiers_txt.xsl
entity_idp_names_txt.xsl
extract_entity.xsl
http_tools.sh
md_tools.sh
saml_tools.sh

Environment

Besides BIN_DIR and LIB_DIR above, the scripts leverage the following environment variables:

Variable
CACHE_DIR	A persistent HTTP cache directory
TMPDIR	A temporary directory
LOG_FILE	A persistent log file
LOG_LEVEL	The global log level [0..5]

All but LOG_LEVEL are REQUIRED. See the following section for more info about logging.

Note: Some OSes define TMPDIR and some do not. In any case, a temporary directory by that name is required to use these scripts.

Logging

For convenience, we will log directly to the terminal in the examples below:

$ export LOG_FILE=/dev/tty
$ export LOG_LEVEL=3

Various log levels are supported:

	LOG_LEVEL
TRACE	5
DEBUG	4
INFO	3
WARN	2
ERROR	1
FATAL	0

The default logging level is INFO (i.e., if you do not explicitly set the optional LOG_LEVEL environment variable, the value LOG_LEVEL=3 is assumed by default).

Overview

The following HTTP resources are used repeatedly in the examples below:

$ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
$ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml

request_http_resources.sh

Bash script request_http_resources.sh issues a HEAD request for one or more HTTP resources:

$ $BIN_DIR/request_http_resources.sh -d /tmp/out $url1 $url2
2017-06-11T02:26:26Z INFO request_http_resources.sh BEGIN
2017-06-11T02:26:26Z INFO request_http_resources.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-06-11T02:26:26Z INFO request_http_resources.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-06-11T02:26:26Z INFO request_http_resources.sh writing output file: http_response_headers.json
2017-06-11T02:26:26Z INFO request_http_resources.sh moving output file to dir: /tmp/out
2017-06-11T02:26:26Z INFO request_http_resources.sh END

The script produces a JSON array containing objects such as the following:

  {
    "successFlag": true,
    "message": "Resource request successful",
    "location": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml",
    "ResponseCode": "200",
    "Date": "Sun, 11 Jun 2017 02:16:44 GMT",
    "LastModified": "Fri, 09 Jun 2017 19:05:16 GMT",
    "ETag": "\"2a96bf6-5518ba65a0c50\"",
    "ContentLength": "44657654",
    "ContentType": "application/samlmetadata+xml"
  }

The JSON file is written to the output directory specified on the command line. See the inline help file for details:

$ $BIN_DIR/request_http_resources.sh -h

cget.sh

Bash script cget.sh retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead.

First define a cache:

$ export CACHE_DIR=/tmp/http_cache

Now GET the first resource:

$ $BIN_DIR/cget.sh $url1 > /dev/null
2017-05-07T20:33:25Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-05-07T20:33:30Z INFO conditional_get received response code: 200
2017-05-07T20:33:30Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
2017-05-07T20:33:30Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
$ echo $?
0
$ ls -1 $CACHE_DIR
1e6b844a49d1850b82feded72cf83ed7_content
1e6b844a49d1850b82feded72cf83ed7_headers
$ cat $CACHE_DIR/1e6b844a49d1850b82feded72cf83ed7_headers
HTTP/1.1 200 OK
Date: Sun, 07 May 2017 20:30:37 GMT
Server: Apache
Last-Modified: Fri, 05 May 2017 19:21:06 GMT
ETag: "29d99bc-54ecbca81c111"
Accept-Ranges: bytes
Content-Length: 43882940
Content-Type: application/samlmetadata+xml

Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the -C option:

$ $BIN_DIR/cget.sh -C $url1 | wc -c
2017-05-07T20:35:30Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-05-07T20:35:30Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
 43882940

Of course the -C option will fail if the resource is not cached:

# illustrate "quiet failure mode"
$ $BIN_DIR/cget.sh -C $url2
2017-05-07T20:36:07Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-05-07T20:36:07Z WARN conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
$ echo $?
1

OTOH, the -F option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however:

# further illustrate "quiet failure mode"
$ $BIN_DIR/cget.sh -F $url1
2017-05-07T20:36:58Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-05-07T20:36:58Z INFO conditional_get received response code: 304
2017-05-07T20:36:58Z WARN conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
$ echo $?
1

The -F option will work on the other URL, however:

$ $BIN_DIR/cget.sh -F $url2 > /dev/null
2017-05-07T20:37:50Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-05-07T20:37:54Z INFO conditional_get received response code: 200
2017-05-07T20:37:54Z INFO conditional_get writing cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
2017-05-07T20:37:54Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
$ echo $?
0
$ ls -1 $CACHE_DIR
1727196e5b7593f3b7528c539e7169d2_content
1727196e5b7593f3b7528c539e7169d2_headers
1e6b844a49d1850b82feded72cf83ed7_content
1e6b844a49d1850b82feded72cf83ed7_headers
$ cat $CACHE_DIR/1727196e5b7593f3b7528c539e7169d2_headers
HTTP/1.1 200 OK
Date: Sun, 07 May 2017 20:35:01 GMT
Server: Apache
Last-Modified: Fri, 05 May 2017 19:21:06 GMT
ETag: "29d99bc-54ecbca8059e8"
Accept-Ranges: bytes
Content-Length: 43882940
Content-Type: application/samlmetadata+xml

See the inline help file for details:

$ $BIN_DIR/cget.sh -h

compute_md_vital_stats.sh

Bash script compute_md_vital_stats.sh parses one or more previously cached SAML metadata files:

$ $BIN_DIR/compute_md_vital_stats.sh -d /tmp/out $url1 $url2
2017-06-11T13:09:53Z INFO compute_md_vital_stats.sh BEGIN
2017-06-11T13:09:53Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
2017-06-11T13:09:53Z INFO compute_md_vital_stats.sh parsing cached metadata file: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-06-11T13:09:54Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
2017-06-11T13:09:54Z INFO compute_md_vital_stats.sh parsing cached metadata file: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh writing output file: md_vital_statistics.json
2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh moving output file to dir: /tmp/out
2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh END

The script produces a JSON array containing objects such as the following:

  {
    "successFlag": true,
    "message": "Metadata successfully parsed",
    "metadataLocation": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml",
    "currentTime": "2017-06-11T13:09:54Z",
    "validUntil": "2017-06-23T18:56:31Z",
    "creationInstant": "2017-06-09T18:56:31Z",
    "validityInterval": "P14DT0H0M0S",
    "untilInvalid": "P12DT5H46M37S",
    "sinceCreation": "P1DT18H13M23S"
  }

Note that this script depends on cached metadata. It will not fetch a metadata file from the server. See the inline help file for details:

$ $BIN_DIR/compute_md_vital_stats.sh -h

test_compressed_resources.sh

Bash script test_compressed_resources.sh tests one or more resources for HTTP Compression:

$ $BIN_DIR/test_compressed_resources.sh -d /tmp/out $url1 $url2
2017-06-11T13:20:39Z INFO test_compressed_resources.sh BEGIN
2017-06-11T13:20:39Z INFO test_compressed_resources.sh requesting (compressed) resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-06-11T13:20:43Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
2017-06-11T13:20:43Z INFO test_compressed_resources.sh requesting (compressed) resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-06-11T13:20:45Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
2017-06-11T13:20:46Z INFO test_compressed_resources.sh writing output file: compressed_response_headers.json
2017-06-11T13:20:46Z INFO test_compressed_resources.sh moving output file to dir: /tmp/out
2017-06-11T13:20:46Z INFO test_compressed_resources.sh END

The script produces a JSON array containing objects such as the following:

  {
    "successFlag": true,
    "message": "Integrity of compressed metadata confirmed",
    "location": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml",
    "ResponseCode": "200",
    "Date": "Sun, 11 Jun 2017 13:10:37 GMT",
    "LastModified": "Fri, 09 Jun 2017 19:05:16 GMT",
    "ETag": "\"80bc05-5518ba65a1fd5\"",
    "ContentLength": "8436741",
    "ContentType": "application/samlmetadata+xml",
    "ContentEncoding": "gzip"
  }

Like the previous script, this script also depends on cached metadata. In addition, this script requests (compressed) resources from the server just-in-time. See the inline help file for details:

$ $BIN_DIR/test_compressed_resources.sh -h

Compatibility

Shell scripts are compatible with both GNU/Linux and Mac OS. XSLT scripts are written in XSLT 1.0.

Dependencies

None