Skip to content

Commit

Permalink
Merge branch 'master' of https://github.internet2.edu/InCommon/bash-l…
Browse files Browse the repository at this point in the history 
…ibrary
  • Loading branch information
Tom Scavo committed May 2, 2017
2 parents 212e5e3 + 645ab1d commit d180325
Showing 1 changed file with 59 additions and 30 deletions.
89 changes: 59 additions & 30 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A library of re-usable bash scripts

## Installation

Download the source, change directory to the source directory, and install the source into ``/tmp`` as follows:
Download the source, change directory to the source directory, and install the source into `/tmp` as follows:

```Shell
$ export BIN_DIR=/tmp/bin
Expand Down Expand Up @@ -41,89 +41,118 @@ saml_tools.sh

## Overview

### ``cget.sh``
### `cget.sh`

Bash script ``cget.sh`` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead.
Bash script `cget.sh` retrieves and caches HTTP resources on disk. A previously cached resource is retrieved via HTTP Conditional GET [RFC 7232]. If the web server responds with HTTP 200 OK, the resource is cached and written to stdout. If the web server responds with 304 Not Modified, the cached resource is output instead.

To illustrate, define a couple of HTTP resources, a cache, and a log file:
First let's configure logging. For convenience, we will log directly to the terminal:

```Shell
$ export LOG_FILE=/dev/tty
$ export LOG_LEVEL=3
```

Various log levels are supported:

| | `LOG_LEVEL` |
| --- | :---: |
| TRACE | 5 |
| DEBUG | 4 |
| INFO | 3 |
| WARN | 2 |
| ERROR | 1 |
| FATAL | 0 |

The default logging level is INFO (i.e., if you do not explicitly set the `LOG_LEVEL` environment variable, the value `LOG_LEVEL=3` is assumed by default).

Next we define a couple of HTTP resources and a cache:

```Shell
$ url1=http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
$ url2=http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
$ export CACHE_DIR=/tmp/http_cache
$ export LOG_FILE=$CACHE_DIR/cget_log.txt
$ touch $LOG_FILE
```

GET the first resource:
Now GET the first resource:

```Shell
$ $BIN_DIR/cget.sh $url1 > /dev/null
2017-05-02T15:29:35Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-05-02T15:29:41Z INFO conditional_get received response code: 200
2017-05-02T15:29:41Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
2017-05-02T15:29:41Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
$ echo $?
0
$ ls -1 $CACHE_DIR
1e6b844a49d1850b82feded72cf83ed7_content
1e6b844a49d1850b82feded72cf83ed7_headers
cget_log.txt
$ cat $CACHE_DIR/1e6b844a49d1850b82feded72cf83ed7_headers
HTTP/1.1 200 OK
Date: Wed, 26 Apr 2017 16:29:44 GMT
Date: Tue, 02 May 2017 15:29:34 GMT
Server: Apache
Last-Modified: Tue, 25 Apr 2017 19:14:05 GMT
ETag: "299866f-54e0286fb789a"
Last-Modified: Mon, 01 May 2017 19:02:13 GMT
ETag: "29bcdb4-54e7b0fa340c5"
Accept-Ranges: bytes
Content-Length: 43615855
Content-Length: 43765172
Content-Type: application/samlmetadata+xml

$ cat $CACHE_DIR/cget_log.txt
2017-04-26T16:29:44Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-04-26T16:30:09Z INFO conditional_get received response code: 200
2017-04-26T16:30:10Z INFO conditional_get writing cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
2017-04-26T16:30:10Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
```

Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the ``-C`` option:
Assuming the resource doesn't change on the server, subsequent requests will return the cached resource. To bypass the network altogether, use the `-C` option:

```Shell
$ $BIN_DIR/cget.sh -C $url1 | wc -c
43615855
2017-05-02T15:30:39Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-05-02T15:30:39Z INFO conditional_get reading cached content file: /tmp/http_cache/1e6b844a49d1850b82feded72cf83ed7_content
43765172
```

Of course the ``-C`` option will fail if the resource is not cached:
Of course the `-C` option will fail if the resource is not cached:

```Shell
# illustrate "quiet failure mode"
$ $BIN_DIR/cget.sh -C $url2
2017-05-02T15:31:07Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-05-02T15:31:07Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
$ echo $?
1
$ cat $CACHE_DIR/cget_log.txt | tail -n 2
2017-04-26T16:39:38Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-04-26T16:39:38Z ERROR conditional_get: resource not cached: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
```

OTOH, the ``-F`` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however:
OTOH, the `-F` option forces the return of a fresh resource from the server. If the resource is cached and unchanged on the server (304), such a request will fail, however:

```Shell
# again illustrate "quiet failure mode"
# further illustrate "quiet failure mode"
$ $BIN_DIR/cget.sh -F $url1
2017-05-02T15:31:42Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
2017-05-02T15:31:48Z INFO conditional_get received response code: 304
2017-05-02T15:31:48Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
$ echo $?
1
$ cat $CACHE_DIR/cget_log.txt | tail -n 2
2017-04-26T16:41:55Z INFO conditional_get received response code: 304
2017-04-26T16:41:55Z ERROR conditional_get: resource not modified: http://md.incommon.org/InCommon/InCommon-metadata-preview.xml
```

The ``-F`` option will work on the other URL, however:
The `-F` option will work on the other URL, however:

```Shell
$ $BIN_DIR/cget.sh -F $url2 > /dev/null
2017-05-02T15:32:27Z INFO cget.sh requesting resource: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
2017-05-02T15:32:32Z INFO conditional_get received response code: 200
2017-05-02T15:32:32Z INFO conditional_get writing cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
2017-05-02T15:32:32Z INFO conditional_get reading cached content file: /tmp/http_cache/1727196e5b7593f3b7528c539e7169d2_content
$ echo $?
0
$ ls -1 $CACHE_DIR
1727196e5b7593f3b7528c539e7169d2_content
1727196e5b7593f3b7528c539e7169d2_headers
1e6b844a49d1850b82feded72cf83ed7_content
1e6b844a49d1850b82feded72cf83ed7_headers
$ cat $CACHE_DIR/1727196e5b7593f3b7528c539e7169d2_headers
HTTP/1.1 200 OK
Date: Tue, 02 May 2017 15:32:26 GMT
Server: Apache
Last-Modified: Mon, 01 May 2017 19:02:13 GMT
ETag: "29bcdb4-54e7b0fa246c3"
Accept-Ranges: bytes
Content-Length: 43765172
Content-Type: application/samlmetadata+xml
```

See the inline help file for details:
Expand Down

0 comments on commit d180325

Please sign in to comment.