diff --git a/content/GCP/06_running_analysis.ipynb b/content/GCP/06_running_analysis.ipynb index 72780cd..f5bfdd1 100644 --- a/content/GCP/06_running_analysis.ipynb +++ b/content/GCP/06_running_analysis.ipynb @@ -12,13 +12,22 @@ "\n", "**Teaching:** 80 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** 8 min\n", "\n", "**Questions:**\n", - "* Can you show me an example?\n", + " * Can you show me an example?\n", "\n", "**Objectives:**\n", - "* Create a simple workflow using a cloud VM and cloud object storage.\n", + " * Create a simple workflow using a cloud VM and cloud object storage.\n", + " * Update a VM Instance software for important security updates.\n", + " * Create a VM Instance with the appropriate storage scope.\n", + " * Create a private regional storage bucket with appropriate security settings.\n", + " * Using the CLI to install software.\n", + " * Download source code using git\n", + " * Retrieve data from a bucket\n", + " * Run the python analysis code\n", + " * Store results in a bucket\n", + " * View the results in the Cloud Storage browser.\n", "\n", "```" ] @@ -30,30 +39,55 @@ "source": [ "## A Research Computational and Data Workflow - Drew's story\n", "\n", - "Drew needs to do some analysis on the data. They need data (satellite images stored in the cloud), computational resources (a virtual machine), some software (we will supply this), and a place to store the results (Cloud Storage). We will assemble all these parts in the cloud \n" + "Drew needs to do some analysis on the data. They need data (satellite images stored in the cloud), computational resources (a virtual machine), some software (we will supply this), and a place to store the results (Cloud Storage). We will assemble and process all these parts in the cloud with a simple example.\n" ] }, { "cell_type": "markdown", - "id": "6291edee-c2df-4941-9b8e-de42649640f9", + "id": "245ffbcb-bf77-4ad1-8bde-1c7d2717ab46", "metadata": {}, "source": [ "## Create a VM\n", "\n", - "Since we only create resources as we need them in the cloud, we will now create a new virtual machine (VM) for Drew to use for their analysis.\n", + "Since we only create resources as we need them in the cloud, we will now create a new virtual machine (VM)instance for Drew to use for their analysis.\n", "\n", - "We will do this as an exercise to give you practice in creating resources. Since the virtual machine will need access to storage on your behalf, you will need to change the **access scope** to give **Full** access to the **Storage** API to the virtual machine. \n", - "\n", - "### Exercise\n", + "We will do this as an exercise to give you practice in creating resources. Since the virtual machine will need access to storage on your behalf, you will need to change the **access scope** to give **Full** access to the **Storage** API to the virtual machine." + ] + }, + { + "cell_type": "markdown", + "id": "45cb5d1d-c28a-4e0e-a0f3-bf68add065be", + "metadata": {}, + "source": [ + "````{admonition} Exercise\n", "\n", "Using the console navigate to the \"Compute Engine\" service and create a new VM with the following properties.\n", " * Call the VM \"essentials\"\n", " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" on the \"create an instance\" page and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", - " * Feel free to select a bit larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + " * Select a bit larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + "````" + ] + }, + { + "cell_type": "markdown", + "id": "009fbfee-f1ce-4046-8223-0aa11b21bd61", + "metadata": {}, + "source": [ + "*Instructor: place the above exercise instructions on the screen*\n", + "\n", + "*When you are done feel free to connect to the virtual machine on your own for additional practice. Once everyone has created their VM we will connect to the machine as described below.*\n", + "\n", + "Please verify that the virtual machine was created as above. If you are unsure delete the virtual machine instance and create it again.\n", "\n", - "*Instructor: place these instructions on the screen*\n", + "Verify that the **Compute Engine default service account** is being used.\n", + "![compute-iam-service-account](img/compute-iam-service-account.png)\n", "\n", - "*When you are done feel free to connect to the virtual machine on your own for additional practice. Once everyone has created their VM we will connect to the machine as described below.*" + "Change **Access scopes** to **Set access for each API**\n", + "![compute-iam-scope-top](img/compute-iam-scope-top.png)\n", + "\n", + "And set **Storage** to **Full**.\n", + "![compute-iam-scope-storage-full](img/compute-iam-scope-storage-full.png)\n", + "\n" ] }, { @@ -100,7 +134,7 @@ "source": [ "## Secure the VM\n", "\n", - "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs security patches." + "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs important security packages and does not upgrade all packages." ] }, { @@ -113,20 +147,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\n", - "Hit:2 http://deb.debian.org/debian buster InRelease \u001b[0m\u001b[33m\n", - "Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]\n", - "Get:4 http://deb.debian.org/debian buster-backports InRelease [46.7 kB]\n", - "Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease\n", + "Hit:1 http://security.debian.org/debian-security buster/updates InRelease\n", + "Hit:2 http://deb.debian.org/debian buster InRelease\n", + "Hit:3 http://deb.debian.org/debian buster-updates InRelease\n", + "Hit:4 http://deb.debian.org/debian buster-backports InRelease\n", + "Get:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease [6780 B]\n", "Hit:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease\n", "Hit:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease\n", - "Get:8 http://deb.debian.org/debian buster-backports/main amd64 Packages.diff/Index [27.8 kB]\n", - "Get:9 http://deb.debian.org/debian buster-backports/main Translation-en.diff/Index [27.8 kB]\n", - "Get:10 http://deb.debian.org/debian buster-backports/main amd64 Packages 2022-01-21-2002.00.pdiff [2515 B]\n", - "Get:11 http://deb.debian.org/debian buster-backports/main Translation-en 2022-01-21-2002.00.pdiff [3068 B]\n", - "Get:10 http://deb.debian.org/debian buster-backports/main amd64 Packages 2022-01-21-2002.00.pdiff [2515 B]\n", - "Get:11 http://deb.debian.org/debian buster-backports/main Translation-en 2022-01-21-2002.00.pdiff [3068 B]\n", - "Fetched 225 kB in 1s (278 kB/s)0m\u001b[33m \u001b[0m\u001b[33m\n", + "Fetched 6780 B in 1s (8978 B/s)\n", "Reading package lists... Done\n", "Building dependency tree \n", "Reading state information... Done\n", @@ -172,7 +200,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "bucket: essentials-learner-2022-01-21 region: us-west2\n" + "bucket: essentials-learner-2022-02-07 region: us-west2\n" ] } ], @@ -192,7 +220,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Creating gs://essentials-learner-2022-01-21/...\n" + "Creating gs://essentials-learner-2022-02-07/...\n" ] } ], @@ -218,7 +246,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-07/\n" ] } ], @@ -260,7 +288,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 11, "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", "metadata": {}, "outputs": [], @@ -270,7 +298,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 12, "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", "metadata": {}, "outputs": [ @@ -279,9 +307,11 @@ "output_type": "stream", "text": [ "Cloning into 'CLASS-Examples'...\n", - "remote: Enumerating objects: 66, done.\u001b[K\n", - "remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66\u001b[K\n", - "Unpacking objects: 100% (66/66), done.\n" + "remote: Enumerating objects: 4, done.\u001b[K\n", + "remote: Counting objects: 100% (4/4), done.\u001b[K\n", + "remote: Compressing objects: 100% (4/4), done.\u001b[K\n", + "remote: Total 70 (delta 0), reused 1 (delta 0), pack-reused 66\u001b[K\n", + "Unpacking objects: 100% (70/70), done.\n" ] } ], @@ -299,7 +329,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 13, "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", "metadata": {}, "outputs": [], @@ -320,7 +350,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 14, "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", "metadata": {}, "outputs": [ @@ -329,13 +359,13 @@ "output_type": "stream", "text": [ "total 28\n", - "-rw-r--r-- 1 learner learner 960 Jan 21 21:32 ReadMe.md\n", - "-rw-r--r-- 1 learner learner 72 Jan 21 21:32 clean.sh\n", - "-rw-r--r-- 1 learner learner 613 Jan 21 21:32 combine.py\n", - "-rw-r--r-- 1 learner learner 280 Jan 21 21:32 download.sh\n", - "-rw-r--r-- 1 learner learner 314 Jan 21 21:32 get-index.sh\n", - "-rw-r--r-- 1 learner learner 76 Jan 21 21:32 search.json\n", - "-rw-r--r-- 1 learner learner 783 Jan 21 21:32 search.py\n" + "-rw-r--r-- 1 learner learner 964 Feb 7 16:04 ReadMe.md\n", + "-rw-r--r-- 1 learner learner 72 Feb 7 16:04 clean.sh\n", + "-rw-r--r-- 1 learner learner 280 Feb 7 16:04 download.sh\n", + "-rw-r--r-- 1 learner learner 314 Feb 7 16:04 get-index.sh\n", + "-rw-r--r-- 1 learner learner 613 Feb 7 16:04 process_sat.py\n", + "-rw-r--r-- 1 learner learner 76 Feb 7 16:04 search.json\n", + "-rw-r--r-- 1 learner learner 783 Feb 7 16:04 search.py\n" ] } ], @@ -357,7 +387,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 15, "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", "metadata": {}, "outputs": [ @@ -406,7 +436,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 16, "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", "metadata": {}, "outputs": [ @@ -424,7 +454,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 17, "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", "metadata": {}, "outputs": [ @@ -438,7 +468,7 @@ "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", "\n", - "| [1 files][731.9 MiB/731.9 MiB] \n", + "/ [1 files][731.9 MiB/731.9 MiB] 63.0 MiB/s \n", "Operation completed over 1 objects/731.9 MiB. \n" ] } @@ -457,7 +487,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 18, "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", "metadata": {}, "outputs": [], @@ -475,7 +505,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 19, "id": "b005876c-f9af-43d6-80c6-f04295413b9b", "metadata": {}, "outputs": [ @@ -484,7 +514,7 @@ "output_type": "stream", "text": [ "total 2.5G\n", - "-rw-r--r-- 1 learner learner 2.5G Jan 21 21:32 index.csv\n" + "-rw-r--r-- 1 learner learner 2.5G Feb 7 16:04 index.csv\n" ] } ], @@ -502,7 +532,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 20, "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", "metadata": {}, "outputs": [ @@ -521,6 +551,18 @@ "head --lines=4 data/index.csv" ] }, + { + "cell_type": "markdown", + "id": "f98c38de-87fa-4e66-9d4b-186fbf81b3b2", + "metadata": {}, + "source": [ + "````{admonition} Tip\n", + ":class: Tip\n", + "\n", + "Now our virtual machine instance is ready and we can access the code and data. Now is a great time to take a short break.\n", + "````" + ] + }, { "cell_type": "markdown", "id": "532e6da3-302a-4e8a-8570-752995f30f1d", @@ -533,7 +575,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 21, "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", "metadata": {}, "outputs": [ @@ -577,7 +619,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 22, "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", "metadata": {}, "outputs": [ @@ -607,7 +649,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 23, "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", "metadata": {}, "outputs": [ @@ -638,7 +680,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 24, "id": "3572c518-df83-4906-bfa6-a37bde2a5063", "metadata": {}, "outputs": [ @@ -673,7 +715,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 25, "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", "metadata": { "scrolled": true, @@ -687,14 +729,14 @@ "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_ANG.txt...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B3.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B2.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B6.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B5.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B11.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B7.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B2.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B8.TIF...\n", "==> NOTE: You are downloading one or more large file(s), which would\n", "run significantly faster if you enabled sliced object downloads. This\n", @@ -704,21 +746,21 @@ "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B9.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_BQA.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_MTL.txt...\n", - "- [14/14 files][952.9 MiB/952.9 MiB] 100% Done \n", + "| [14/14 files][952.9 MiB/952.9 MiB] 100% Done 18.6 MiB/s ETA 00:00:00 \n", "Operation completed over 14 objects/952.9 MiB. \n", "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_ANG.txt...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B3.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B10.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B5.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B6.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B10.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B5.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B7.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B6.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B8.TIF...\n", - "==> NOTE: You are downloading one or more large file(s), which would\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", "run significantly faster if you enabled sliced object downloads. This\n", "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", @@ -726,7 +768,7 @@ "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B9.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_BQA.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_MTL.txt...\n", - "/ [14/14 files][ 1.0 GiB/ 1.0 GiB] 100% Done 30.3 MiB/s ETA 00:00:00 \n", + "- [14/14 files][ 1.0 GiB/ 1.0 GiB] 100% Done 19.1 MiB/s ETA 00:00:00 \n", "Operation completed over 14 objects/1.0 GiB. \n" ] } @@ -745,7 +787,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 26, "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", "metadata": {}, "outputs": [ @@ -754,9 +796,9 @@ "output_type": "stream", "text": [ "total 2564796\n", - "drwxr-xr-x 2 learner learner 4096 Jan 21 21:33 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20201007_20201016_01_T1\u001b[0m\n", - "drwxr-xr-x 2 learner learner 4096 Jan 21 21:33 \u001b[01;34mLC08_L1TP_025033_20210519_20210528_01_T1\u001b[0m\n", - "-rw-r--r-- 1 learner learner 2626336574 Jan 21 21:32 index.csv\n" + "drwxr-xr-x 2 learner learner 4096 Feb 7 16:07 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20201007_20201016_01_T1\u001b[0m\n", + "drwxr-xr-x 2 learner learner 4096 Feb 7 16:08 \u001b[01;34mLC08_L1TP_025033_20210519_20210528_01_T1\u001b[0m\n", + "-rw-r--r-- 1 learner learner 2626336574 Feb 7 16:04 index.csv\n" ] } ], @@ -776,7 +818,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 27, "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", "metadata": {}, "outputs": [ @@ -807,12 +849,12 @@ } ], "source": [ - "cat combine.py" + "cat process_sat.py" ] }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 28, "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", "metadata": {}, "outputs": [ @@ -821,14 +863,14 @@ "output_type": "stream", "text": [ "Traceback (most recent call last):\n", - " File \"combine.py\", line 3, in \n", + " File \"process_sat.py\", line 3, in \n", " import rasterio\n", "ModuleNotFoundError: No module named 'rasterio'\n" ] } ], "source": [ - "python3 combine.py\n", + "python3 process_sat.py\n", "/bin/true # ignore this line used for jupyter" ] }, @@ -843,7 +885,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 29, "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", "metadata": { "scrolled": true, @@ -900,8 +942,8 @@ "0 upgraded, 79 newly installed, 0 to remove and 2 not upgraded.\n", "Need to get 46.8 MB of archives.\n", "After this operation, 172 MB of additional disk space will be used.\n", - "Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", - "Get:2 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", + "Get:1 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", + "Get:2 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", "Get:3 http://deb.debian.org/debian buster/main amd64 fonts-dejavu-core all 2.37-1 [1068 kB]\n", "Get:4 http://deb.debian.org/debian buster/main amd64 fontconfig-config all 2.13.1-2 [280 kB]\n", "Get:5 http://deb.debian.org/debian buster/main amd64 gdal-data all 2.4.0+dfsg-1 [744 kB]\n", @@ -909,36 +951,36 @@ "Get:7 http://deb.debian.org/debian buster/main amd64 libgfortran5 amd64 8.3.0-6 [581 kB]\n", "Get:8 http://deb.debian.org/debian buster/main amd64 libblas3 amd64 3.8.0-2 [148 kB]\n", "Get:9 http://deb.debian.org/debian buster/main amd64 liblapack3 amd64 3.8.0-2 [2110 kB]\n", - "Get:10 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", - "Get:11 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", - "Get:12 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", - "Get:13 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", - "Get:14 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", - "Get:15 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", - "Get:16 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", - "Get:17 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", - "Get:18 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", - "Get:19 http://security.debian.org/debian-security buster/updates/main amd64 python3-lxml amd64 4.3.2-1+deb10u4 [1163 kB]\n", - "Get:20 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", - "Get:21 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", - "Get:22 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", - "Get:23 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", - "Get:24 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", - "Get:25 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", - "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", - "Get:27 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", - "Get:28 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", - "Get:29 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", - "Get:30 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", - "Get:31 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", - "Get:32 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", - "Get:33 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", - "Get:34 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", - "Get:35 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", - "Get:36 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", - "Get:37 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", - "Get:38 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", - "Get:39 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", + "Get:10 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", + "Get:11 http://security.debian.org/debian-security buster/updates/main amd64 liburiparser1 amd64 0.9.1-1+deb10u1 [48.1 kB]\n", + "Get:12 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u5 [1160 kB]\n", + "Get:13 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", + "Get:14 http://security.debian.org/debian-security buster/updates/main amd64 python3-lxml amd64 4.3.2-1+deb10u4 [1163 kB]\n", + "Get:15 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", + "Get:16 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", + "Get:17 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", + "Get:18 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", + "Get:19 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", + "Get:20 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", + "Get:21 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", + "Get:22 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", + "Get:23 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", + "Get:24 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", + "Get:25 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", + "Get:26 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", + "Get:27 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", + "Get:28 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", + "Get:29 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", + "Get:30 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", + "Get:31 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", + "Get:32 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", + "Get:33 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", + "Get:34 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", + "Get:35 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", + "Get:36 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", + "Get:37 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", + "Get:38 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", + "Get:39 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", "Get:40 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", "Get:41 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", @@ -979,11 +1021,11 @@ "Get:77 http://deb.debian.org/debian buster/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [89.6 kB]\n", "Get:78 http://deb.debian.org/debian buster/main amd64 python3-snuggs all 1.4.3-1 [7228 B]\n", "Get:79 http://deb.debian.org/debian buster/main amd64 python3-rasterio amd64 1.0.21-1 [818 kB]\n", - "Fetched 46.8 MB in 2s (28.9 MB/s) \n", + "Fetched 46.8 MB in 5s (8830 kB/s) \n", "Extracting templates from packages: 100%\n", "Preconfiguring packages ...\n", "Selecting previously unselected package poppler-data.\n", - "(Reading database ... 57529 files and directories currently installed.)\n", + "(Reading database ... 57683 files and directories currently installed.)\n", "Preparing to unpack .../00-poppler-data_0.4.9-2_all.deb ...\n", "Unpacking poppler-data (0.4.9-2) ...\n", "Selecting previously unselected package fonts-dejavu-core.\n", @@ -1089,8 +1131,8 @@ "Preparing to unpack .../34-libminizip1_1.1-8+b1_amd64.deb ...\n", "Unpacking libminizip1:amd64 (1.1-8+b1) ...\n", "Selecting previously unselected package liburiparser1:amd64.\n", - "Preparing to unpack .../35-liburiparser1_0.9.1-1_amd64.deb ...\n", - "Unpacking liburiparser1:amd64 (0.9.1-1) ...\n", + "Preparing to unpack .../35-liburiparser1_0.9.1-1+deb10u1_amd64.deb ...\n", + "Unpacking liburiparser1:amd64 (0.9.1-1+deb10u1) ...\n", "Selecting previously unselected package libkmlbase1:amd64.\n", "Preparing to unpack .../36-libkmlbase1_1.3.0-7_amd64.deb ...\n", "Unpacking libkmlbase1:amd64 (1.3.0-7) ...\n", @@ -1140,8 +1182,8 @@ "Preparing to unpack .../51-libnspr4_2%3a4.20-1_amd64.deb ...\n", "Unpacking libnspr4:amd64 (2:4.20-1) ...\n", "Selecting previously unselected package libnss3:amd64.\n", - "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u4_amd64.deb ...\n", - "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u5_amd64.deb ...\n", + "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u5) ...\n", "Selecting previously unselected package libpoppler82:amd64.\n", "Preparing to unpack .../53-libpoppler82_0.71.0-5_amd64.deb ...\n", "Unpacking libpoppler82:amd64 (0.71.0-5) ...\n", @@ -1257,7 +1299,7 @@ "Setting up libgfortran5:amd64 (8.3.0-6) ...\n", "Setting up libhdf4-0-alt (4.2.13-4) ...\n", "Setting up libgif7:amd64 (5.1.4-3) ...\n", - "Setting up liburiparser1:amd64 (0.9.1-1) ...\n", + "Setting up liburiparser1:amd64 (0.9.1-1+deb10u1) ...\n", "Setting up libfreexl1:amd64 (1.0.5-3) ...\n", "Setting up libfyba0:amd64 (4.1.1-6) ...\n", "Setting up libkmlbase1:amd64 (1.3.0-7) ...\n", @@ -1277,7 +1319,7 @@ "Setting up fontconfig-config (2.13.1-2) ...\n", "Setting up python3-cligj (0.5.0-1) ...\n", "Setting up libdap25:amd64 (3.20.3-1) ...\n", - "Setting up libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Setting up libnss3:amd64 (2:3.42.1-1+deb10u5) ...\n", "Setting up libdapserver7v5:amd64 (3.20.3-1) ...\n", "Setting up python3-bs4 (4.7.1-1) ...\n", "Setting up libgeotiff2:amd64 (1.4.3-1) ...\n", @@ -1315,7 +1357,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 31, "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", "metadata": {}, "outputs": [ @@ -1331,7 +1373,7 @@ } ], "source": [ - "/usr/bin/python3 combine.py" + "/usr/bin/python3 process_sat.py" ] }, { @@ -1352,7 +1394,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 32, "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", "metadata": {}, "outputs": [ @@ -1361,8 +1403,8 @@ "output_type": "stream", "text": [ "total 192M\n", - "-rw-r--r-- 1 learner learner 192M Jan 21 21:34 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20201007_20201016_01_T1.png\u001b[0m\n", - "-rw-r--r-- 1 learner learner 910 Jan 21 21:34 result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n" + "-rw-r--r-- 1 learner learner 192M Feb 7 16:09 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20201007_20201016_01_T1.png\u001b[0m\n", + "-rw-r--r-- 1 learner learner 910 Feb 7 16:09 result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n" ] } ], @@ -1382,7 +1424,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 33, "id": "9345472f-4ef3-490b-a80e-2462cd534c89", "metadata": {}, "outputs": [ @@ -1390,7 +1432,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "essentials-learner-2022-01-21\n" + "essentials-learner-2022-02-07\n" ] } ], @@ -1400,7 +1442,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 34, "id": "27dfae96-faf2-4d5d-8a78-97781841f172", "metadata": {}, "outputs": [ @@ -1408,7 +1450,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-07/\n" ] } ], @@ -1428,7 +1470,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 35, "id": "681e6b1d-98bb-448a-a57e-f5674214effd", "metadata": {}, "outputs": [ @@ -1469,7 +1511,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 36, "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", "metadata": {}, "outputs": [ @@ -1477,7 +1519,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/output/\n" + "gs://essentials-learner-2022-02-07/output/\n" ] } ], @@ -1487,7 +1529,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 37, "id": "b1ea18e9-5861-4479-9948-3303952dee8a", "metadata": {}, "outputs": [ @@ -1495,8 +1537,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "191.58 MiB 2022-01-21T21:34:56Z gs://essentials-learner-2022-01-21/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png\n", - " 910 B 2022-01-21T21:34:54Z gs://essentials-learner-2022-01-21/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n", + "191.58 MiB 2022-02-07T16:09:41Z gs://essentials-learner-2022-02-07/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png\n", + " 910 B 2022-02-07T16:09:39Z gs://essentials-learner-2022-02-07/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n", "TOTAL: 2 objects, 200890195 bytes (191.58 MiB)\n" ] } @@ -1507,7 +1549,7 @@ }, { "cell_type": "markdown", - "id": "66a41f97-3834-4d09-bfe5-c3f224628842", + "id": "36530927-2205-4586-bca7-10b8698f5a15", "metadata": {}, "source": [ "## Viewing the Results\n", @@ -1515,6 +1557,30 @@ "You now can view the results by using the Google Cloud Platform Web Console and navigating to \"Cloud Storage\", selecting the bucket, and then the result object you wish to view (select the `.png` file). You will need to click the \"Preview\" button given the large size of the image." ] }, + { + "cell_type": "markdown", + "id": "4de32b26-f4cb-4f33-b7d5-e3852769840a", + "metadata": {}, + "source": [ + "````{admonition} Exercise\n", + "\n", + " * Try to find and view the results on your own\n", + "\n", + "````" + ] + }, + { + "cell_type": "markdown", + "id": "9f7a4893-104a-4467-84ac-871d21db6050", + "metadata": {}, + "source": [ + "Navigate to **Cloud Storage** -> **Browser** -> **Bucket** -> **output** folder and then click on the **result** object.\n", + "![example-object](img/example-object.png)\n", + "\n", + "And press the **Preview** button below the object details (check the details) and you should see something similar to the following:\n", + "![example-object-preview](img/example-object-preview.png)" + ] + }, { "cell_type": "markdown", "id": "40d9ab41-8920-45f2-8218-550baac5b069", diff --git a/content/GCP/img/compute-iam-scope-storage-full.png b/content/GCP/img/compute-iam-scope-storage-full.png new file mode 100644 index 0000000..670ce48 Binary files /dev/null and b/content/GCP/img/compute-iam-scope-storage-full.png differ diff --git a/content/GCP/img/compute-iam-scope-top.png b/content/GCP/img/compute-iam-scope-top.png new file mode 100644 index 0000000..1400754 Binary files /dev/null and b/content/GCP/img/compute-iam-scope-top.png differ diff --git a/content/GCP/img/compute-iam-service-account.png b/content/GCP/img/compute-iam-service-account.png new file mode 100644 index 0000000..6e08ea2 Binary files /dev/null and b/content/GCP/img/compute-iam-service-account.png differ diff --git a/content/GCP/img/example-object-preview.png b/content/GCP/img/example-object-preview.png new file mode 100644 index 0000000..efcc6e3 Binary files /dev/null and b/content/GCP/img/example-object-preview.png differ diff --git a/content/GCP/img/example-object.png b/content/GCP/img/example-object.png new file mode 100644 index 0000000..7bcb81f Binary files /dev/null and b/content/GCP/img/example-object.png differ