diff --git a/content/GCP/06_running_analysis.ipynb b/content/GCP/06_running_analysis.ipynb index c563cb7..05428b8 100644 --- a/content/GCP/06_running_analysis.ipynb +++ b/content/GCP/06_running_analysis.ipynb @@ -22,6 +22,265 @@ "\n", "```" ] + }, + { + "cell_type": "markdown", + "id": "c87e200c-1d18-44f5-b4e6-28b697b04b3c", + "metadata": {}, + "source": [ + "## Connect to the VM\n", + "\n", + "First login to the instance from the Cloud Shell\n", + "```\n", + "gcloud compute ssh instance-1\n", + "```\n", + "If prompted for a zone select `n` to find it automatically. You can see an example session below." + ] + }, + { + "cell_type": "markdown", + "id": "44f32772-0abb-4fa7-a013-202fc646df5e", + "metadata": {}, + "source": [ + "```\n", + "student31@cloudshell:~ (just-armor-301114)$ gcloud compute ssh instance-1\n", + "Did you mean zone [us-central1-b] for instance: [instance-1] (Y/n)? n\n", + "\n", + "No zone specified. Using zone [us-west2-c] for instance: [instance-1].\n", + "Linux instance-1 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64\n", + "\n", + "The programs included with the Debian GNU/Linux system are free software;\n", + "the exact distribution terms for each program are described in the\n", + "individual files in /usr/share/doc/*/copyright.\n", + "\n", + "Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent\n", + "permitted by applicable law.\n", + "Last login: Tue Nov 9 20:12:49 2021 from 34.133.99.196\n", + "student31@instance-1:~$\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "b7c4db9e-f098-41bc-80d2-b524444eec7f", + "metadata": {}, + "source": [ + "## Get Example Code\n", + "\n", + "We will now install `git` and use it to download the example code and change into the directory." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "05fadd30-4f35-4a93-b713-cc132131b948", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "git is already the newest version (1:2.20.1-2+deb10u3).\n", + "0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.\n" + ] + } + ], + "source": [ + "sudo apt install --yes git" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cloning into 'CLASS-Examples'...\n", + "remote: Enumerating objects: 16, done.\u001b[K\n", + "remote: Counting objects: 100% (16/16), done.\u001b[K\n", + "remote: Compressing objects: 100% (13/13), done.\u001b[K\n", + "remote: Total 41 (delta 4), reused 15 (delta 3), pack-reused 25\u001b[K\n", + "Unpacking objects: 100% (41/41), done.\n" + ] + } + ], + "source": [ + "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", + "metadata": {}, + "outputs": [], + "source": [ + "cd CLASS-Examples/landsat/" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 24\n", + "-rw-r--r-- 1 student31 student31 72 Nov 9 20:18 clean.sh\n", + "-rw-r--r-- 1 student31 student31 256 Nov 9 20:18 download.sh\n", + "-rw-r--r-- 1 student31 student31 314 Nov 9 20:18 get-index.sh\n", + "-rw-r--r-- 1 student31 student31 841 Nov 9 20:18 ReadMe.md\n", + "-rw-r--r-- 1 student31 student31 110 Nov 9 20:18 search.json\n", + "-rw-r--r-- 1 student31 student31 1447 Nov 9 20:18 search.py\n" + ] + } + ], + "source": [ + "ls -l" + ] + }, + { + "cell_type": "markdown", + "id": "2b8b3144-7453-4350-a1cf-7fa74af2bcbf", + "metadata": {}, + "source": [ + "## Access the bucket\n", + "\n", + "First test bucket access by listing the bucket" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://gcp-public-data-landsat/index.csv.gz\n", + "gs://gcp-public-data-landsat/LC08/\n", + "gs://gcp-public-data-landsat/LE07/\n", + "gs://gcp-public-data-landsat/LM01/\n", + "gs://gcp-public-data-landsat/LM02/\n", + "gs://gcp-public-data-landsat/LM03/\n", + "gs://gcp-public-data-landsat/LM04/\n", + "gs://gcp-public-data-landsat/LM05/\n", + "gs://gcp-public-data-landsat/LO08/\n", + "gs://gcp-public-data-landsat/LT04/\n", + "gs://gcp-public-data-landsat/LT05/\n", + "gs://gcp-public-data-landsat/LT08/\n" + ] + } + ], + "source": [ + "gsutil ls gs://gcp-public-data-landsat/" + ] + }, + { + "cell_type": "markdown", + "id": "89acffba-cbce-436c-98dd-05467b6675a6", + "metadata": {}, + "source": [ + "Get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mkdir: created directory 'data'\n" + ] + } + ], + "source": [ + "mkdir -v data" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying gs://gcp-public-data-landsat/index.csv.gz...\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "| [1 files][757.2 MiB/757.2 MiB] 737.1 KiB/s \n", + "Operation completed over 1 objects/757.2 MiB. \n" + ] + } + ], + "source": [ + "gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "gzip: data/index.csv: No space left on device\n" + ] + }, + { + "ename": "", + "evalue": "1", + "output_type": "error", + "traceback": [] + } + ], + "source": [ + "gzip -d data/index.csv.gz" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b005876c-f9af-43d6-80c6-f04295413b9b", + "metadata": {}, + "outputs": [], + "source": [ + "ls -lh data" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bbb26cfb-4b0d-4458-bc24-13a069556c3a", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": {