diff --git a/README.md b/README.md index 35ec0a2..40e850c 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A library of re-usable bash scripts ## Installation -Download the source, change directory to the source directory, and install the source into ``/tmp`` as follows: +Download the source, change directory to the source directory, and install the source into `/tmp` as follows: ```Shell $ export BIN_DIR=/tmp/bin @@ -41,82 +41,102 @@ saml_tools.sh ## Overview -### ``cget.sh`` +### `cget.sh` -Bash script ``cget.sh`` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead. +Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead. -To illustrate, define a couple of HTTP resources, a cache, and a log file: +First let's configure logging. For convenience, we will log directly to the terminal: + +```Shell +$ export LOG_FILE=/dev/tty +$ export LOG_LEVEL=3 +``` + +Various log levels are supported: + +| | `LOG_LEVEL` | +| --- | :---: | +| TRACE | 5 | +| DEBUG | 4 | +| INFO | 3 | +| WARN | 2 | +| ERROR | 1 | +| FATAL | 0 | + +The default logging level is INFO (i.e., if you do not explicitly set the `LOG_LEVEL` environment variable, the value `LOG_LEVEL=3` is assumed by default). + +Next we define a couple of HTTP resources and a cache: ```Shell $ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml $ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml $ export CACHE_DIR=/tmp/http_cache -$ export LOG_FILE=$CACHE_DIR/cget_log.txt -$ touch $LOG_FILE ``` -GET the first resource: +Now GET the first resource: ```Shell $ $BIN_DIR/cget.sh $url1 > /dev/null +2017-05-02T15:29:35Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T15:29:41Z INFO conditional_get received response code: 200 +2017-05-02T15:29:41Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content +2017-05-02T15:29:41Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content $ echo $? 0 $ ls -1 $CACHE_DIR 1e6b844a49d1850b82feded72cf83ed7_content 1e6b844a49d1850b82feded72cf83ed7_headers -cget_log.txt $ cat $CACHE_DIR/1e6b844a49d1850b82feded72cf83ed7_headers HTTP/1.1 200 OK -Date: Wed, 26 Apr 2017 16:29:44 GMT +Date: Tue, 02 May 2017 15:29:34 GMT Server: Apache -Last-Modified: Tue, 25 Apr 2017 19:14:05 GMT -ETag: "299866f-54e0286fb789a" +Last-Modified: Mon, 01 May 2017 19:02:13 GMT +ETag: "29bcdb4-54e7b0fa340c5" Accept-Ranges: bytes -Content-Length: 43615855 +Content-Length: 43765172 Content-Type: application/samlmetadata+xml - -$ cat $CACHE_DIR/cget_log.txt -2017-04-26T16:29:44Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml -2017-04-26T16:30:09Z INFO conditional_get received response code: 200 -2017-04-26T16:30:10Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content -2017-04-26T16:30:10Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content ``` -Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the ``-C`` option: +Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the `-C` option: ```Shell $ $BIN_DIR/cget.sh -C $url1 | wc -c - 43615855 +2017-05-02T15:30:39Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T15:30:39Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content + 43765172 ``` -Of course the ``-C`` option will fail if the resource is not cached: +Of course the `-C` option will fail if the resource is not cached: ```Shell # illustrate "quiet failure mode" $ $BIN_DIR/cget.sh -C $url2 +2017-05-02T15:31:07Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml +2017-05-02T15:31:07Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml $ echo $? 1 -$ cat $CACHE_DIR/cget_log.txt | tail -n 2 -2017-04-26T16:39:38Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml -2017-04-26T16:39:38Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml ``` -OTOH, the ``-F`` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however: +OTOH, the `-F` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however: ```Shell -# again illustrate "quiet failure mode" +# further illustrate "quiet failure mode" $ $BIN_DIR/cget.sh -F $url1 +2017-05-02T15:31:42Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T15:31:48Z INFO conditional_get received response code: 304 +2017-05-02T15:31:48Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml $ echo $? 1 -$ cat $CACHE_DIR/cget_log.txt | tail -n 2 -2017-04-26T16:41:55Z INFO conditional_get received response code: 304 -2017-04-26T16:41:55Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml ``` -The ``-F`` option will work on the other URL, however: +The `-F` option will work on the other URL, however: ```Shell $ $BIN_DIR/cget.sh -F $url2 > /dev/null +2017-05-02T15:32:27Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml +2017-05-02T15:32:32Z INFO conditional_get received response code: 200 +2017-05-02T15:32:32Z INFO conditional_get writing cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content +2017-05-02T15:32:32Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content $ echo $? 0 $ ls -1 $CACHE_DIR @@ -124,6 +144,15 @@ $ ls -1 $CACHE_DIR 1727196e5b7593f3b7528c539e7169d2_headers 1e6b844a49d1850b82feded72cf83ed7_content 1e6b844a49d1850b82feded72cf83ed7_headers +$ cat $CACHE_DIR/1727196e5b7593f3b7528c539e7169d2_headers +HTTP/1.1 200 OK +Date: Tue, 02 May 2017 15:32:26 GMT +Server: Apache +Last-Modified: Mon, 01 May 2017 19:02:13 GMT +ETag: "29bcdb4-54e7b0fa246c3" +Accept-Ranges: bytes +Content-Length: 43765172 +Content-Type: application/samlmetadata+xml ``` See the inline help file for details: