Permalink
Cannot retrieve contributors at this time
Name already in use
A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
bash-library/README.md
Go to fileThis commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
309 lines (245 sloc)
11 KB
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Bash Library | |
| A library of re-usable bash scripts | |
| ## Installation | |
| Download the source, change directory to the source directory, and install the source into `/tmp` as follows: | |
| ```Shell | |
| $ export BIN_DIR=/tmp/bin | |
| $ export LIB_DIR=/tmp/lib | |
| $ ./install.sh $BIN_DIR $LIB_DIR | |
| ``` | |
| or install into your home directory: | |
| ```Shell | |
| $ export BIN_DIR=$HOME/bin | |
| $ export LIB_DIR=$HOME/lib | |
| $ ./install.sh $BIN_DIR $LIB_DIR | |
| ``` | |
| A given target directory will be created if one doesn't already exist. In any case, the following files will be installed: | |
| ```Shell | |
| $ ls -1 $BIN_DIR | |
| cget.sh | |
| compute_md_vital_stats.sh | |
| request_http_resources.sh | |
| test_compressed_resources.sh | |
| $ ls -1 $LIB_DIR | |
| compatible_date.sh | |
| config_tools.sh | |
| core_lib.sh | |
| entities_timestamps_txt.xsl | |
| entity_endpoints_txt.xsl | |
| entity_identifiers_txt.xsl | |
| entity_idp_names_txt.xsl | |
| extract_entity.xsl | |
| http_tools.sh | |
| md_tools.sh | |
| saml_tools.sh | |
| ``` | |
| ## Environment | |
| Besides `BIN_DIR` and `LIB_DIR` above, the scripts leverage the following environment variables: | |
| | Variable | | | |
| | --- | --- | | |
| | `CACHE_DIR` | A persistent HTTP cache directory | | |
| | `TMPDIR` | A temporary directory | | |
| | `LOG_FILE` | A persistent log file | | |
| | `LOG_LEVEL` | The global log level [0..5] | | |
| All but `LOG_LEVEL` are REQUIRED. See the following section for more info about logging. | |
| Note: Some OSes define `TMPDIR` and some do not. In any case, a temporary directory by that name is required to use these scripts. | |
| ## Logging | |
| For convenience, we will log directly to the terminal in the examples below: | |
| ```Shell | |
| $ export LOG_FILE=/dev/tty | |
| $ export LOG_LEVEL=3 | |
| ``` | |
| Various log levels are supported: | |
| | | `LOG_LEVEL` | | |
| | --- | :---: | | |
| | TRACE | 5 | | |
| | DEBUG | 4 | | |
| | INFO | 3 | | |
| | WARN | 2 | | |
| | ERROR | 1 | | |
| | FATAL | 0 | | |
| The default logging level is INFO (i.e., if you do not explicitly set the optional `LOG_LEVEL` environment variable, the value `LOG_LEVEL=3` is assumed by default). | |
| ## Overview | |
| The following HTTP resources are used repeatedly in the examples below: | |
| ```Shell | |
| $ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| $ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml | |
| ``` | |
| ### `request_http_resources.sh` | |
| Bash script `request_http_resources.sh` issues a HEAD request for one or more HTTP resources: | |
| ```Shell | |
| $ $BIN_DIR/request_http_resources.sh -d /tmp/out $url1 $url2 | |
| 2017-06-11T02:26:26Z INFO request_http_resources.sh BEGIN | |
| 2017-06-11T02:26:26Z INFO request_http_resources.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| 2017-06-11T02:26:26Z INFO request_http_resources.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml | |
| 2017-06-11T02:26:26Z INFO request_http_resources.sh writing output file: http_response_headers.json | |
| 2017-06-11T02:26:26Z INFO request_http_resources.sh moving output file to dir: /tmp/out | |
| 2017-06-11T02:26:26Z INFO request_http_resources.sh END | |
| ``` | |
| The script produces a JSON array containing objects such as the following: | |
| ```JavaScript | |
| { | |
| "successFlag": true, | |
| "message": "Resource request successful", | |
| "location": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml", | |
| "ResponseCode": "200", | |
| "Date": "Sun, 11 Jun 2017 02:16:44 GMT", | |
| "LastModified": "Fri, 09 Jun 2017 19:05:16 GMT", | |
| "ETag": "\"2a96bf6-5518ba65a0c50\"", | |
| "ContentLength": "44657654", | |
| "ContentType": "application/samlmetadata+xml" | |
| } | |
| ``` | |
| The JSON file is written to the output directory specified on the command line. See the inline help file for details: | |
| ```Shell | |
| $ $BIN_DIR/request_http_resources.sh -h | |
| ``` | |
| ### `cget.sh` | |
| Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead. | |
| First define a cache: | |
| ```Shell | |
| $ export CACHE_DIR=/tmp/http_cache | |
| ``` | |
| Now GET the first resource: | |
| ```Shell | |
| $ $BIN_DIR/cget.sh $url1 > /dev/null | |
| 2017-05-07T20:33:25Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| 2017-05-07T20:33:30Z INFO conditional_get received response code: 200 | |
| 2017-05-07T20:33:30Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content | |
| 2017-05-07T20:33:30Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content | |
| $ echo $? | |
| 0 | |
| $ ls -1 $CACHE_DIR | |
| 1e6b844a49d1850b82feded72cf83ed7_content | |
| 1e6b844a49d1850b82feded72cf83ed7_headers | |
| $ cat $CACHE_DIR/1e6b844a49d1850b82feded72cf83ed7_headers | |
| HTTP/1.1 200 OK | |
| Date: Sun, 07 May 2017 20:30:37 GMT | |
| Server: Apache | |
| Last-Modified: Fri, 05 May 2017 19:21:06 GMT | |
| ETag: "29d99bc-54ecbca81c111" | |
| Accept-Ranges: bytes | |
| Content-Length: 43882940 | |
| Content-Type: application/samlmetadata+xml | |
| ``` | |
| Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the `-C` option: | |
| ```Shell | |
| $ $BIN_DIR/cget.sh -C $url1 | wc -c | |
| 2017-05-07T20:35:30Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| 2017-05-07T20:35:30Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content | |
| 43882940 | |
| ``` | |
| Of course the `-C` option will fail if the resource is not cached: | |
| ```Shell | |
| # illustrate "quiet failure mode" | |
| $ $BIN_DIR/cget.sh -C $url2 | |
| 2017-05-07T20:36:07Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml | |
| 2017-05-07T20:36:07Z WARN conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml | |
| $ echo $? | |
| 1 | |
| ``` | |
| OTOH, the `-F` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however: | |
| ```Shell | |
| # further illustrate "quiet failure mode" | |
| $ $BIN_DIR/cget.sh -F $url1 | |
| 2017-05-07T20:36:58Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| 2017-05-07T20:36:58Z INFO conditional_get received response code: 304 | |
| 2017-05-07T20:36:58Z WARN conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| $ echo $? | |
| 1 | |
| ``` | |
| The `-F` option will work on the other URL, however: | |
| ```Shell | |
| $ $BIN_DIR/cget.sh -F $url2 > /dev/null | |
| 2017-05-07T20:37:50Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml | |
| 2017-05-07T20:37:54Z INFO conditional_get received response code: 200 | |
| 2017-05-07T20:37:54Z INFO conditional_get writing cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content | |
| 2017-05-07T20:37:54Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content | |
| $ echo $? | |
| 0 | |
| $ ls -1 $CACHE_DIR | |
| 1727196e5b7593f3b7528c539e7169d2_content | |
| 1727196e5b7593f3b7528c539e7169d2_headers | |
| 1e6b844a49d1850b82feded72cf83ed7_content | |
| 1e6b844a49d1850b82feded72cf83ed7_headers | |
| $ cat $CACHE_DIR/1727196e5b7593f3b7528c539e7169d2_headers | |
| HTTP/1.1 200 OK | |
| Date: Sun, 07 May 2017 20:35:01 GMT | |
| Server: Apache | |
| Last-Modified: Fri, 05 May 2017 19:21:06 GMT | |
| ETag: "29d99bc-54ecbca8059e8" | |
| Accept-Ranges: bytes | |
| Content-Length: 43882940 | |
| Content-Type: application/samlmetadata+xml | |
| ``` | |
| See the inline help file for details: | |
| ```Shell | |
| $ $BIN_DIR/cget.sh -h | |
| ``` | |
| ### `compute_md_vital_stats.sh` | |
| Bash script `compute_md_vital_stats.sh` parses one or more previously cached SAML metadata files: | |
| ```Shell | |
| $ $BIN_DIR/compute_md_vital_stats.sh -d /tmp/out $url1 $url2 | |
| 2017-06-11T13:09:53Z INFO compute_md_vital_stats.sh BEGIN | |
| 2017-06-11T13:09:53Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content | |
| 2017-06-11T13:09:53Z INFO compute_md_vital_stats.sh parsing cached metadata file: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| 2017-06-11T13:09:54Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content | |
| 2017-06-11T13:09:54Z INFO compute_md_vital_stats.sh parsing cached metadata file: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml | |
| 2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh writing output file: md_vital_statistics.json | |
| 2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh moving output file to dir: /tmp/out | |
| 2017-06-11T13:09:55Z INFO compute_md_vital_stats.sh END | |
| ``` | |
| The script produces a JSON array containing objects such as the following: | |
| ```JavaScript | |
| { | |
| "successFlag": true, | |
| "message": "Metadata successfully parsed", | |
| "metadataLocation": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml", | |
| "currentTime": "2017-06-11T13:09:54Z", | |
| "validUntil": "2017-06-23T18:56:31Z", | |
| "creationInstant": "2017-06-09T18:56:31Z", | |
| "validityInterval": "P14DT0H0M0S", | |
| "untilInvalid": "P12DT5H46M37S", | |
| "sinceCreation": "P1DT18H13M23S" | |
| } | |
| ``` | |
| Note that this script depends on cached metadata. It will not fetch a metadata file from the server. See the inline help file for details: | |
| ```Shell | |
| $ $BIN_DIR/compute_md_vital_stats.sh -h | |
| ``` | |
| ### `test_compressed_resources.sh` | |
| Bash script `test_compressed_resources.sh` tests one or more resources for HTTP Compression: | |
| ```Shell | |
| $ $BIN_DIR/test_compressed_resources.sh -d /tmp/out $url1 $url2 | |
| 2017-06-11T13:20:39Z INFO test_compressed_resources.sh BEGIN | |
| 2017-06-11T13:20:39Z INFO test_compressed_resources.sh requesting (compressed) resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml | |
| 2017-06-11T13:20:43Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content | |
| 2017-06-11T13:20:43Z INFO test_compressed_resources.sh requesting (compressed) resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml | |
| 2017-06-11T13:20:45Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content | |
| 2017-06-11T13:20:46Z INFO test_compressed_resources.sh writing output file: compressed_response_headers.json | |
| 2017-06-11T13:20:46Z INFO test_compressed_resources.sh moving output file to dir: /tmp/out | |
| 2017-06-11T13:20:46Z INFO test_compressed_resources.sh END | |
| ``` | |
| The script produces a JSON array containing objects such as the following: | |
| ```JavaScript | |
| { | |
| "successFlag": true, | |
| "message": "Integrity of compressed metadata confirmed", | |
| "location": "http://md.incommon.org/InCommon/InCommon-metadata-preview.xml", | |
| "ResponseCode": "200", | |
| "Date": "Sun, 11 Jun 2017 13:10:37 GMT", | |
| "LastModified": "Fri, 09 Jun 2017 19:05:16 GMT", | |
| "ETag": "\"80bc05-5518ba65a1fd5\"", | |
| "ContentLength": "8436741", | |
| "ContentType": "application/samlmetadata+xml", | |
| "ContentEncoding": "gzip" | |
| } | |
| ``` | |
| Like the previous script, this script also depends on cached metadata. In addition, this script requests (compressed) resources from the server just-in-time. See the inline help file for details: | |
| ```Shell | |
| $ $BIN_DIR/test_compressed_resources.sh -h | |
| ``` | |
| ## Compatibility | |
| Shell scripts are compatible with both GNU/Linux and Mac OS. XSLT scripts are written in XSLT 1.0. | |
| ## Dependencies | |
| None |