From 64aa1e47a6ddab5a8b5f22ebda1944ab8bd60696 Mon Sep 17 00:00:00 2001 From: Tom Scavo Date: Tue, 2 May 2017 09:27:41 -0400 Subject: [PATCH 1/7] Update example in overview section --- README.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 35ec0a2..f4bd9c5 100644 --- a/README.md +++ b/README.md @@ -51,14 +51,17 @@ To illustrate, define a couple of HTTP resources, a cache, and a log file: $ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml $ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml $ export CACHE_DIR=/tmp/http_cache -$ export LOG_FILE=$CACHE_DIR/cget_log.txt -$ touch $LOG_FILE +$ export LOG_FILE=/dev/tty ``` GET the first resource: ```Shell $ $BIN_DIR/cget.sh $url1 > /dev/null +2017-05-02T13:24:57Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T13:26:11Z INFO conditional_get received response code: 200 +2017-05-02T13:26:11Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content +2017-05-02T13:26:12Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content $ echo $? 0 $ ls -1 $CACHE_DIR @@ -74,19 +77,15 @@ ETag: "299866f-54e0286fb789a" Accept-Ranges: bytes Content-Length: 43615855 Content-Type: application/samlmetadata+xml - -$ cat $CACHE_DIR/cget_log.txt -2017-04-26T16:29:44Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml -2017-04-26T16:30:09Z INFO conditional_get received response code: 200 -2017-04-26T16:30:10Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content -2017-04-26T16:30:10Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content ``` Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the ``-C`` option: ```Shell $ $BIN_DIR/cget.sh -C $url1 | wc -c - 43615855 +2017-05-02T13:27:07Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T13:27:07Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content + 43765172 ``` Of course the ``-C`` option will fail if the resource is not cached: From 0663d118b606d68c3da3af9bec5d3593d33c3670 Mon Sep 17 00:00:00 2001 From: Tom Scavo Date: Tue, 2 May 2017 10:59:06 -0400 Subject: [PATCH 2/7] Update example in overview section --- README.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index f4bd9c5..cf4920f 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,6 @@ $ echo $? $ ls -1 $CACHE_DIR 1e6b844a49d1850b82feded72cf83ed7_content 1e6b844a49d1850b82feded72cf83ed7_headers -cget_log.txt $ cat $CACHE_DIR/1e6b844a49d1850b82feded72cf83ed7_headers HTTP/1.1 200 OK Date: Wed, 26 Apr 2017 16:29:44 GMT @@ -93,29 +92,32 @@ Of course the ``-C`` option will fail if the resource is not cached: ```Shell # illustrate "quiet failure mode" $ $BIN_DIR/cget.sh -C $url2 +2017-05-02T14:53:38Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml +2017-05-02T14:53:38Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml $ echo $? 1 -$ cat $CACHE_DIR/cget_log.txt | tail -n 2 -2017-04-26T16:39:38Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml -2017-04-26T16:39:38Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml ``` OTOH, the ``-F`` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however: ```Shell -# again illustrate "quiet failure mode" +# further illustrate "quiet failure mode" $ $BIN_DIR/cget.sh -F $url1 +2017-05-02T14:54:37Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T14:54:38Z INFO conditional_get received response code: 304 +2017-05-02T14:54:38Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml $ echo $? 1 -$ cat $CACHE_DIR/cget_log.txt | tail -n 2 -2017-04-26T16:41:55Z INFO conditional_get received response code: 304 -2017-04-26T16:41:55Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml ``` The ``-F`` option will work on the other URL, however: ```Shell $ $BIN_DIR/cget.sh -F $url2 > /dev/null +2017-05-02T14:55:24Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml +2017-05-02T14:55:30Z INFO conditional_get received response code: 200 +2017-05-02T14:55:30Z INFO conditional_get writing cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content +2017-05-02T14:55:30Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content $ echo $? 0 $ ls -1 $CACHE_DIR From 53f31a8be367514b92531d9f3e5b9d21dff21848 Mon Sep 17 00:00:00 2001 From: Tom Scavo Date: Tue, 2 May 2017 11:24:08 -0400 Subject: [PATCH 3/7] Document how to configure logging --- README.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index cf4920f..ebb5ae6 100644 --- a/README.md +++ b/README.md @@ -45,16 +45,27 @@ saml_tools.sh Bash script ``cget.sh`` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead. -To illustrate, define a couple of HTTP resources, a cache, and a log file: +First let's configure logging. For convenience, we will log directly to the terminal: + +```Shell +$ export LOG_FILE=/dev/tty +``` + +The system defaults to INFO logging, which corresponds to ``LOG_LEVEL=3``. We make that explicit as follows: + +```Shell +$ export LOG_LEVEL=3 +``` + +Next we define a couple of HTTP resources and a cache: ```Shell $ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml $ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml $ export CACHE_DIR=/tmp/http_cache -$ export LOG_FILE=/dev/tty ``` -GET the first resource: +Now GET the first resource: ```Shell $ $BIN_DIR/cget.sh $url1 > /dev/null From 385a4d8adfaa786e4b9f4cc51b34035fd1a3262d Mon Sep 17 00:00:00 2001 From: Tom Scavo Date: Tue, 2 May 2017 11:33:38 -0400 Subject: [PATCH 4/7] Capture the output of a fresh run --- README.md | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index ebb5ae6..aeee1f2 100644 --- a/README.md +++ b/README.md @@ -69,10 +69,10 @@ Now GET the first resource: ```Shell $ $BIN_DIR/cget.sh $url1 > /dev/null -2017-05-02T13:24:57Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml -2017-05-02T13:26:11Z INFO conditional_get received response code: 200 -2017-05-02T13:26:11Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content -2017-05-02T13:26:12Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content +2017-05-02T15:29:35Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T15:29:41Z INFO conditional_get received response code: 200 +2017-05-02T15:29:41Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content +2017-05-02T15:29:41Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content $ echo $? 0 $ ls -1 $CACHE_DIR @@ -80,12 +80,12 @@ $ ls -1 $CACHE_DIR 1e6b844a49d1850b82feded72cf83ed7_headers $ cat $CACHE_DIR/1e6b844a49d1850b82feded72cf83ed7_headers HTTP/1.1 200 OK -Date: Wed, 26 Apr 2017 16:29:44 GMT +Date: Tue, 02 May 2017 15:29:34 GMT Server: Apache -Last-Modified: Tue, 25 Apr 2017 19:14:05 GMT -ETag: "299866f-54e0286fb789a" +Last-Modified: Mon, 01 May 2017 19:02:13 GMT +ETag: "29bcdb4-54e7b0fa340c5" Accept-Ranges: bytes -Content-Length: 43615855 +Content-Length: 43765172 Content-Type: application/samlmetadata+xml ``` @@ -93,8 +93,8 @@ Assuming the resource doesn't change on the server, subsequent requests will ret ```Shell $ $BIN_DIR/cget.sh -C $url1 | wc -c -2017-05-02T13:27:07Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml -2017-05-02T13:27:07Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content +2017-05-02T15:30:39Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T15:30:39Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content 43765172 ``` @@ -103,8 +103,8 @@ Of course the ``-C`` option will fail if the resource is not cached: ```Shell # illustrate "quiet failure mode" $ $BIN_DIR/cget.sh -C $url2 -2017-05-02T14:53:38Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml -2017-05-02T14:53:38Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml +2017-05-02T15:31:07Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml +2017-05-02T15:31:07Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml $ echo $? 1 ``` @@ -114,9 +114,9 @@ OTOH, the ``-F`` option forces the return of a fresh resource from the server. I ```Shell # further illustrate "quiet failure mode" $ $BIN_DIR/cget.sh -F $url1 -2017-05-02T14:54:37Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml -2017-05-02T14:54:38Z INFO conditional_get received response code: 304 -2017-05-02T14:54:38Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T15:31:42Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml +2017-05-02T15:31:48Z INFO conditional_get received response code: 304 +2017-05-02T15:31:48Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml $ echo $? 1 ``` @@ -125,10 +125,10 @@ The ``-F`` option will work on the other URL, however: ```Shell $ $BIN_DIR/cget.sh -F $url2 > /dev/null -2017-05-02T14:55:24Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml -2017-05-02T14:55:30Z INFO conditional_get received response code: 200 -2017-05-02T14:55:30Z INFO conditional_get writing cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content -2017-05-02T14:55:30Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content +2017-05-02T15:32:27Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml +2017-05-02T15:32:32Z INFO conditional_get received response code: 200 +2017-05-02T15:32:32Z INFO conditional_get writing cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content +2017-05-02T15:32:32Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content $ echo $? 0 $ ls -1 $CACHE_DIR @@ -136,6 +136,15 @@ $ ls -1 $CACHE_DIR 1727196e5b7593f3b7528c539e7169d2_headers 1e6b844a49d1850b82feded72cf83ed7_content 1e6b844a49d1850b82feded72cf83ed7_headers +$ cat $CACHE_DIR/1727196e5b7593f3b7528c539e7169d2_headers +HTTP/1.1 200 OK +Date: Tue, 02 May 2017 15:32:26 GMT +Server: Apache +Last-Modified: Mon, 01 May 2017 19:02:13 GMT +ETag: "29bcdb4-54e7b0fa246c3" +Accept-Ranges: bytes +Content-Length: 43765172 +Content-Type: application/samlmetadata+xml ``` See the inline help file for details: From fe16e2953af1a826b443a44a2344304abec0c2bc Mon Sep 17 00:00:00 2001 From: Tom Scavo Date: Tue, 2 May 2017 11:42:16 -0400 Subject: [PATCH 5/7] Fix inline code formatting --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index aeee1f2..defc694 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A library of re-usable bash scripts ## Installation -Download the source, change directory to the source directory, and install the source into ``/tmp`` as follows: +Download the source, change directory to the source directory, and install the source into `/tmp` as follows: ```Shell $ export BIN_DIR=/tmp/bin @@ -41,9 +41,9 @@ saml_tools.sh ## Overview -### ``cget.sh`` +### `cget.sh` -Bash script ``cget.sh`` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead. +Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead. First let's configure logging. For convenience, we will log directly to the terminal: @@ -51,7 +51,7 @@ First let's configure logging. For convenience, we will log directly to the term $ export LOG_FILE=/dev/tty ``` -The system defaults to INFO logging, which corresponds to ``LOG_LEVEL=3``. We make that explicit as follows: +The system defaults to INFO logging, which corresponds to `LOG_LEVEL=3`. We make that explicit as follows: ```Shell $ export LOG_LEVEL=3 @@ -89,7 +89,7 @@ Content-Length: 43765172 Content-Type: application/samlmetadata+xml ``` -Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the ``-C`` option: +Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the `-C` option: ```Shell $ $BIN_DIR/cget.sh -C $url1 | wc -c @@ -98,7 +98,7 @@ $ $BIN_DIR/cget.sh -C $url1 | wc -c 43765172 ``` -Of course the ``-C`` option will fail if the resource is not cached: +Of course the `-C` option will fail if the resource is not cached: ```Shell # illustrate "quiet failure mode" @@ -109,7 +109,7 @@ $ echo $? 1 ``` -OTOH, the ``-F`` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however: +OTOH, the `-F` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however: ```Shell # further illustrate "quiet failure mode" @@ -121,7 +121,7 @@ $ echo $? 1 ``` -The ``-F`` option will work on the other URL, however: +The `-F` option will work on the other URL, however: ```Shell $ $BIN_DIR/cget.sh -F $url2 > /dev/null From 394209cb336ebef71b912cc82ad37f017e8fbf26 Mon Sep 17 00:00:00 2001 From: Tom Scavo Date: Tue, 2 May 2017 11:56:41 -0400 Subject: [PATCH 6/7] Document log levels --- README.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index defc694..10b97f9 100644 --- a/README.md +++ b/README.md @@ -49,13 +49,21 @@ First let's configure logging. For convenience, we will log directly to the term ```Shell $ export LOG_FILE=/dev/tty +$ export LOG_LEVEL=3 ``` -The system defaults to INFO logging, which corresponds to `LOG_LEVEL=3`. We make that explicit as follows: +Various log levels are supported: -```Shell -$ export LOG_LEVEL=3 -``` +| | `LOG_LEVEL` | +| --- | :---: | +| TRACE | 5 | +| DEBUG | 4 | +| INFO | 3 | +| WARN | 2 | +| ERROR | 1 | +| FATAL | 0 | + +The default logging level is INFO. Next we define a couple of HTTP resources and a cache: From 645ab1dc5c0f481fc5ef4c0c8d627b58f33e8e6c Mon Sep 17 00:00:00 2001 From: Tom Scavo Date: Tue, 2 May 2017 16:13:03 -0400 Subject: [PATCH 7/7] Clarify remark about default LOG_LEVEL --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 10b97f9..40e850c 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ Various log levels are supported: | ERROR | 1 | | FATAL | 0 | -The default logging level is INFO. +The default logging level is INFO (i.e., if you do not explicitly set the `LOG_LEVEL` environment variable, the value `LOG_LEVEL=3` is assumed by default). Next we define a couple of HTTP resources and a cache: