From 554f2daba1294fc973c32733a67d21501de59895 Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 16 Dec 2021 16:54:34 +0000 Subject: [PATCH] Create a VM at the start of the analysis Episode. --- content/GCP/02_intro_to_compute.ipynb | 14 +- content/GCP/06_running_analysis.ipynb | 352 +++++++++++++++----------- 2 files changed, 218 insertions(+), 148 deletions(-) diff --git a/content/GCP/02_intro_to_compute.ipynb b/content/GCP/02_intro_to_compute.ipynb index 3fd69d4..ba6811e 100644 --- a/content/GCP/02_intro_to_compute.ipynb +++ b/content/GCP/02_intro_to_compute.ipynb @@ -57,16 +57,16 @@ "We will first create an *VM instance* with defaults to explore the Google Compute Engine service.\n", "\n", "To create a bucket we do the following:\n", - " * Click **Navigation Menu** -> **Compute Engine** (under Compute) -> **VM Instances** -> **+Create Bucket** (just under the blue bar) to open the *Create an instance* page.\n", + " * Click **Navigation Menu** -> **Compute Engine** (under Compute) -> **VM Instances** -> **+Create Instance** (just under the blue bar) to open the *Create an instance* page.\n", " * In the **New VM instance** tab on the left (selected by default) configure the *VM instance* as follows:\n", " * For **Name**, enter a unique name for the instances (example: \"essentials-instance-1\")\n", - " * For **Region** leave the default or select your \"home\" region. The region is the physical location where your data will reside. Your \"home\" region should be close to your work and the region you use most often.\n", - " * For **Zone** leave the default (note how the name is constructed and that it is a separate data center).\n", + " * For **Region** leave the default or select your \"home\" region. The region is the physical location where your data will reside. Your \"home\" region should be close to your work and should be the region you use most of the time.\n", + " * For **Zone** leave the default (note how the name is constructed and that it is a separate data center) some zones have different capabilities.\n", " * In the **Machine configuration** section:\n", " * For **Series** select *E2* (default). The other options are for types of machines.\n", " * For **Machine type** select *e2-medium* (default)\n", + " * In the **Identity and API access** section note that for the **Service Accounts** -> **Service account** option that the **Compute Engine default service account** is selected. This allows the *VM Instance* to access resources in the project. Note: the *\"Allow full access to all Cloud APIs\"* allows *everyone* on the VM to *access and edit all resources in the project*.\n", " * Leave the remaining options as default.\n", - " * In the **Identity and API access** section note that for the **Service Accounts** -> **Service account** option that the **Compute Engine default service account** is selected. This allows the *VM Instance* to ***access and edit any resources in the project***.\n", " * Note the **Monthly estimate** in the right side panel. This is only for running the *VM instance* and does not include other storage, networking, compute, and other charges that may occur by running software. Click **Details** show a breakdown of the *VM instance* only costs.\n", " * Finally, click **Create** to create the *VM Instance* (allocate the resource).\n", " " @@ -239,13 +239,15 @@ "tags": [] }, "source": [ - "## Delete the Storage - Exercise \n", + "## Delete the VM - Exercise \n", "\n", "We will now complete the resource life-cycle by deleting the *VM Instance* we just created. Think about the process we did for creating the bucket and follow a similar one. \n", "\n", "Did you \"Follow\" the bucket by looking at the **activity** page?\n", "\n", - "Since we care about paying for resources we are not using we **review** by visiting the **dashboard** and reviewing that we no longer have any buckets, if this was our only bucket. " + "Since we care about paying for resources we are not using we **review** our project by visiting the compute engine service and reviewing that we no longer have any virtual machines (VM) running. \n", + "\n", + "In later episodes we will show how to easily list all the resources in a project." ] }, { diff --git a/content/GCP/06_running_analysis.ipynb b/content/GCP/06_running_analysis.ipynb index cec1189..307266e 100644 --- a/content/GCP/06_running_analysis.ipynb +++ b/content/GCP/06_running_analysis.ipynb @@ -23,6 +23,23 @@ "```" ] }, + { + "cell_type": "markdown", + "id": "6291edee-c2df-4941-9b8e-de42649640f9", + "metadata": {}, + "source": [ + "## Create a VM\n", + "\n", + "Since we only create resources as we need them in the cloud. As an exercise you will now create a VM for our analysis. In this case will give the VM **Full** access to **Storage**. \n", + "\n", + "Using the console navigate to the \"Compute Engine\" service and create a new VM with the following properties.\n", + " * Call the VM \"essentials\"\n", + " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", + " * Feel free to select a larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + " \n", + "When you are done connect to the machine as described below." + ] + }, { "cell_type": "markdown", "id": "c87e200c-1d18-44f5-b4e6-28b697b04b3c", @@ -67,7 +84,7 @@ "source": [ "## Secure the VM\n", "\n", - "We first make sure that the VM is up to date with the latest security patches by running the following commands" + "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs security patches." ] }, { @@ -106,12 +123,12 @@ "source": [ "## Get Example Code\n", "\n", - "We will now install `git` and use it to download the example code and change into the directory." + "We will now install `git` and use it to download the example code into your home directory. For those of you who are unfamiliar with git, it is a way to collaboratively manage files and we will only be using it to download the example that we will be using. " ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 1, "id": "05fadd30-4f35-4a93-b713-cc132131b948", "metadata": {}, "outputs": [ @@ -133,7 +150,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 2, "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", "metadata": {}, "outputs": [], @@ -143,7 +160,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 3, "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", "metadata": {}, "outputs": [ @@ -151,23 +168,28 @@ "name": "stdout", "output_type": "stream", "text": [ - "fatal: destination path 'CLASS-Examples' already exists and is not an empty directory.\n" + "Cloning into 'CLASS-Examples'...\n", + "remote: Enumerating objects: 66, done.\u001b[K\n", + "remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66\u001b[K\n", + "Unpacking objects: 100% (66/66), done.\n" ] - }, - { - "ename": "", - "evalue": "128", - "output_type": "error", - "traceback": [] } ], "source": [ "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" ] }, + { + "cell_type": "markdown", + "id": "3d2eb9fc-bec9-449d-a7ab-f3127f59d512", + "metadata": {}, + "source": [ + "We now change the current directory to the `landsat` directory in the `CLASS-Examples` directory that was just created by the previous git command." + ] + }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 4, "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", "metadata": {}, "outputs": [], @@ -177,7 +199,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 5, "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", "metadata": {}, "outputs": [ @@ -185,15 +207,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "total 32\n", - "-rw-r--r-- 1 learner learner 960 Dec 13 15:11 ReadMe.md\n", - "-rw-r--r-- 1 learner learner 72 Dec 13 15:11 clean.sh\n", - "-rw-r--r-- 1 learner learner 514 Dec 13 15:11 combine.py\n", - "drwxr-xr-x 4 learner learner 4096 Dec 13 15:12 \u001b[0m\u001b[01;34mdata\u001b[0m\n", - "-rw-r--r-- 1 learner learner 280 Dec 13 15:11 download.sh\n", - "-rw-r--r-- 1 learner learner 314 Dec 13 15:11 get-index.sh\n", - "-rw-r--r-- 1 learner learner 76 Dec 13 15:11 search.json\n", - "-rw-r--r-- 1 learner learner 783 Dec 13 15:11 search.py\n" + "total 28\n", + "-rw-r--r-- 1 learner learner 960 Dec 16 16:27 ReadMe.md\n", + "-rw-r--r-- 1 learner learner 72 Dec 16 16:27 clean.sh\n", + "-rw-r--r-- 1 learner learner 613 Dec 16 16:27 combine.py\n", + "-rw-r--r-- 1 learner learner 280 Dec 16 16:27 download.sh\n", + "-rw-r--r-- 1 learner learner 314 Dec 16 16:27 get-index.sh\n", + "-rw-r--r-- 1 learner learner 76 Dec 16 16:27 search.json\n", + "-rw-r--r-- 1 learner learner 783 Dec 16 16:27 search.py\n" ] } ], @@ -208,12 +229,12 @@ "source": [ "## Access the bucket\n", "\n", - "First test bucket access by listing the bucket" + "First test that our tools are working and that we can access the public bucket that we will be using." ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 6, "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", "metadata": {}, "outputs": [ @@ -245,12 +266,14 @@ "id": "89acffba-cbce-436c-98dd-05467b6675a6", "metadata": {}, "source": [ - "Get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region" + "The index file is a list of all the files in the bucket and we can use it to search and filter files.\n", + "\n", + "We will get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region." ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 7, "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", "metadata": {}, "outputs": [ @@ -258,14 +281,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "mkdir: cannot create directory ‘data’: File exists\n" + "mkdir: created directory 'data'\n" ] - }, - { - "ename": "", - "evalue": "1", - "output_type": "error", - "traceback": [] } ], "source": [ @@ -274,7 +291,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 8, "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", "metadata": {}, "outputs": [ @@ -288,8 +305,8 @@ "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", "\n", - "| [1 files][760.7 MiB/760.7 MiB] \n", - "Operation completed over 1 objects/760.7 MiB. \n" + "\\ [1 files][760.9 MiB/760.9 MiB] 50.8 MiB/s \n", + "Operation completed over 1 objects/760.9 MiB. \n" ] } ], @@ -307,7 +324,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 9, "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", "metadata": {}, "outputs": [], @@ -315,9 +332,17 @@ "gzip -df data/index.csv.gz" ] }, + { + "cell_type": "markdown", + "id": "75faa74a-b458-472b-9018-034c89e75315", + "metadata": {}, + "source": [ + "Again, check what happened." + ] + }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 10, "id": "b005876c-f9af-43d6-80c6-f04295413b9b", "metadata": {}, "outputs": [ @@ -326,9 +351,7 @@ "output_type": "stream", "text": [ "total 2.6G\n", - "drwxr-xr-x 2 learner learner 4.0K Dec 13 15:13 \u001b[0m\u001b[01;34mLC08_L1GT_025033_20160215_20170224_01_T2\u001b[0m\n", - "drwxr-xr-x 2 learner learner 4.0K Dec 13 15:12 \u001b[01;34mLC08_L1GT_025033_20210127_20210305_01_T2\u001b[0m\n", - "-rw-r--r-- 1 learner learner 2.6G Dec 13 15:49 index.csv\n" + "-rw-r--r-- 1 learner learner 2.6G Dec 16 16:32 index.csv\n" ] } ], @@ -341,12 +364,12 @@ "id": "fcde8334-f58d-4c3d-995a-2491be0f95ea", "metadata": {}, "source": [ - "We will now explore the data" + "We will now explore the data. The `head` command simply displays the first few lines of the file." ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 11, "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", "metadata": {}, "outputs": [ @@ -355,9 +378,9 @@ "output_type": "stream", "text": [ "SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL\n", - "LE71730322004290ASN01,LE07_L1TP_173032_20041016_20170118_01_T1,LANDSAT_7,ETM,2004-10-16,01,T1,2004-10-16T07:51:09.5270348Z,L1TP,173,32,31.0,41.30611,39.3474,38.02284,40.86797,207913942,gs://gcp-public-data-landsat/LE07/01/173/032/LE07_L1TP_173032_20041016_20170118_01_T1\n", - "LT51320261997215BJC00,LT05_L1TP_132026_19970803_20161230_01_T1,LANDSAT_5,TM,1997-08-03,01,T1,1997-08-03T03:17:51.5040130Z,L1TP,132,26,63.0,49.83572,47.82939,104.30052,107.6158,136383230,gs://gcp-public-data-landsat/LT05/01/132/026/LT05_L1TP_132026_19970803_20161230_01_T1\n", - "LT51510262008107BJC01,LT05_L1GS_151026_20080416_20180611_01_T2,LANDSAT_5,TM,2008-04-16,01,T2,2008-04-16T05:33:27.4160940Z,L1GS,151,26,87.0,49.86189,47.86585,74.77182,78.13782,141793569,gs://gcp-public-data-landsat/LT05/01/151/026/LT05_L1GS_151026_20080416_20180611_01_T2\n" + "LC81030672020252LGN00,LC08_L1GT_103067_20200908_20200918_01_T2,LANDSAT_8,OLI_TIRS,2020-09-08,01,T2,2020-09-08T01:04:03.7134200Z,L1GT,103,67,44.99,-9.07156,-11.17811,134.61622,136.70979,1009724166,gs://gcp-public-data-landsat/LC08/01/103/067/LC08_L1GT_103067_20200908_20200918_01_T2\n", + "LC81590702021230LGN00,LC08_L1TP_159070_20210818_20210826_01_T1,LANDSAT_8,OLI_TIRS,2021-08-18,01,T1,2021-08-18T06:51:18.8060269Z,L1TP,159,70,15.72,-13.40348,-15.51165,47.1186,49.24972,989147313,gs://gcp-public-data-landsat/LC08/01/159/070/LC08_L1TP_159070_20210818_20210826_01_T1\n", + "LE70620122002162PAC00,LE07_L1TP_062012_20020611_20170129_01_T1,LANDSAT_7,ETM,2002-06-11,01,T1,2002-06-11T20:17:05.6581076Z,L1TP,62,12,41.0,69.38475,67.14636,-136.16562,-129.66669,224906243,gs://gcp-public-data-landsat/LE07/01/062/012/LE07_L1TP_062012_20020611_20170129_01_T1\n" ] } ], @@ -377,7 +400,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 12, "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", "metadata": {}, "outputs": [ @@ -421,7 +444,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 13, "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", "metadata": {}, "outputs": [ @@ -451,7 +474,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 14, "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", "metadata": {}, "outputs": [ @@ -459,8 +482,11 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1GT_025033_20210127_20210305_01_T2\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1GT_025033_20160215_20170224_01_T2\n" + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1\n", + "gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1GT_025033_20200109_20200114_01_T2\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20150706_20180131_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170711_20170726_01_T1\n" ] } ], @@ -480,7 +506,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 15, "id": "3572c518-df83-4906-bfa6-a37bde2a5063", "metadata": {}, "outputs": [ @@ -515,7 +541,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 16, "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", "metadata": { "scrolled": true, @@ -526,36 +552,48 @@ "name": "stdout", "output_type": "stream", "text": [ - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1GT_025033_20210127_20210305_01_T2\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_ANG.txt\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B1.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B10.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B11.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B2.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B3.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B5.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B4.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B8.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B7.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B6.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_BQA.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_B9.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20210127_20210305_01_T2/LC08_L1GT_025033_20210127_20210305_01_T2_MTL.txt\n", - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1GT_025033_20160215_20170224_01_T2\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B10.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_ANG.txt\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B1.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B11.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B2.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B3.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B7.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B4.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B8.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B5.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B6.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_B9.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_BQA.TIF\n", - "Skipping existing item: file://data/LC08_L1GT_025033_20160215_20170224_01_T2/LC08_L1GT_025033_20160215_20170224_01_T2_MTL.txt\n" + "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B5.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B7.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B8.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_B9.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20140124_20170306_01_T1/LC08_L1TP_025033_20140124_20170306_01_T1_MTL.txt...\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "| [14/14 files][890.9 MiB/890.9 MiB] 100% Done \n", + "Operation completed over 14 objects/890.9 MiB. \n", + "+++ gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B5.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_MTL.txt...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B7.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B8.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LO08/01/025/033/LO08_L1TP_025033_20201108_20201120_01_T1/LO08_L1TP_025033_20201108_20201120_01_T1_B9.TIF...\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "\\ [12/12 files][935.8 MiB/935.8 MiB] 100% Done 38.1 MiB/s ETA 00:00:00 \n", + "Operation completed over 12 objects/935.8 MiB. \n" ] } ], @@ -573,7 +611,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 17, "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", "metadata": {}, "outputs": [ @@ -581,10 +619,10 @@ "name": "stdout", "output_type": "stream", "text": [ - "total 2664744\n", - "drwxr-xr-x 2 learner learner 4096 Dec 13 15:13 \u001b[0m\u001b[01;34mLC08_L1GT_025033_20160215_20170224_01_T2\u001b[0m\n", - "drwxr-xr-x 2 learner learner 4096 Dec 13 15:12 \u001b[01;34mLC08_L1GT_025033_20210127_20210305_01_T2\u001b[0m\n", - "-rw-r--r-- 1 learner learner 2728684265 Dec 13 15:49 index.csv\n" + "total 2665520\n", + "drwxr-xr-x 2 learner learner 4096 Dec 16 16:37 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20140124_20170306_01_T1\u001b[0m\n", + "drwxr-xr-x 2 learner learner 4096 Dec 16 16:37 \u001b[01;34mLO08_L1TP_025033_20201108_20201120_01_T1\u001b[0m\n", + "-rw-r--r-- 1 learner learner 2729479404 Dec 16 16:32 index.csv\n" ] } ], @@ -604,7 +642,7 @@ }, { "cell_type": "code", - "execution_count": 36, + "execution_count": 18, "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", "metadata": {}, "outputs": [ @@ -629,7 +667,7 @@ "meta.update(count = 3, driver='PNG')\n", "with rasterio.open(\"output/result-%s.png\" % source, 'w+', **meta) as output:\n", " for i in range(1, 4):\n", - " print(i)\n", + " print(source, i)\n", " output.write_band(i,rasterio.open(\"data/%s/%s_B%d.TIF\" % (source, source, i+3)).read(1))\n" ] } @@ -640,7 +678,7 @@ }, { "cell_type": "code", - "execution_count": 37, + "execution_count": 27, "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", "metadata": {}, "outputs": [ @@ -653,16 +691,11 @@ " import rasterio\n", "ModuleNotFoundError: No module named 'rasterio'\n" ] - }, - { - "ename": "", - "evalue": "1", - "output_type": "error", - "traceback": [] } ], "source": [ - "python3 combine.py" + "python3 combine.py\n", + "/bin/true # ignore this line used for jupyter" ] }, { @@ -671,12 +704,12 @@ "metadata": {}, "source": [ "Oops, let's install the library (note: the output will be slightly different due to how this Lesson is built)\n", - "and while we are at it we also need to create the output directory" + "and while we are at it we also need to create the output directory." ] }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 28, "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", "metadata": { "scrolled": true, @@ -688,31 +721,22 @@ "output_type": "stream", "text": [ "install: creating directory 'output/'\n", - "Requirement already satisfied: rasterio in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (1.2.10)\n", - "Requirement already satisfied: affine in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (2.3.0)\n", - "Requirement already satisfied: snuggs>=1.4.1 in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (1.4.7)\n", - "Requirement already satisfied: click>=4.0 in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (8.0.3)\n", - "Requirement already satisfied: numpy in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (1.21.4)\n", - "Requirement already satisfied: attrs in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (21.2.0)\n", - "Requirement already satisfied: cligj>=0.5 in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (0.7.2)\n", - "Requirement already satisfied: certifi in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (2021.10.8)\n", - "Requirement already satisfied: setuptools in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (40.8.0)\n", - "Requirement already satisfied: click-plugins in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from rasterio) (1.1.1)\n", - "Requirement already satisfied: importlib-metadata in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from click>=4.0->rasterio) (4.8.2)\n", - "Requirement already satisfied: pyparsing>=2.1.6 in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from snuggs>=1.4.1->rasterio) (3.0.6)\n", - "Requirement already satisfied: typing-extensions>=3.6.4 in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from importlib-metadata->click>=4.0->rasterio) (4.0.1)\n", - "Requirement already satisfied: zipp>=0.5 in /home/learner/CLASS-Essentials/.venv/lib/python3.7/site-packages (from importlib-metadata->click>=4.0->rasterio) (3.6.0)\n" + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "python3-rasterio is already the newest version (1.0.21-1).\n", + "0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.\n" ] } ], "source": [ "install -dv output/\n", - "python3 -m pip install rasterio" + "sudo apt-get install python3-rasterio --yes" ] }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 29, "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", "metadata": {}, "outputs": [ @@ -720,19 +744,28 @@ "name": "stdout", "output_type": "stream", "text": [ - "1\n", - "2\n", - "3\n" + "LO08_L1TP_025033_20201108_20201120_01_T1 1\n", + "LO08_L1TP_025033_20201108_20201120_01_T1 2\n", + "LO08_L1TP_025033_20201108_20201120_01_T1 3\n", + "ERROR 4: output/result-LO08_L1TP_025033_20201108_20201120_01_T1.png: No such file or directory\n" ] } ], "source": [ - "python3 combine.py" + "/usr/bin/python3 combine.py" ] }, { "cell_type": "markdown", - "id": "91d62964-d0ff-48d1-b215-a74ac233b3ef", + "id": "4199910a-1056-455c-ae9d-669296441aed", + "metadata": {}, + "source": [ + "Note, the \"ERROR 4\" is expected." + ] + }, + { + "cell_type": "markdown", + "id": "a3a89af5-993e-45d6-9e7c-2ca0a61dff7a", "metadata": {}, "source": [ "Verify the results were created." @@ -740,7 +773,7 @@ }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 30, "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", "metadata": {}, "outputs": [ @@ -748,9 +781,9 @@ "name": "stdout", "output_type": "stream", "text": [ - "total 165M\n", - "-rw-r--r-- 1 learner learner 165M Dec 13 16:09 \u001b[0m\u001b[01;35mresult-LC08_L1GT_025033_20210127_20210305_01_T2.png\u001b[0m\n", - "-rw-r--r-- 1 learner learner 941 Dec 13 16:09 result-LC08_L1GT_025033_20210127_20210305_01_T2.png.aux.xml\n" + "total 196M\n", + "-rw-r--r-- 1 learner learner 196M Dec 16 16:43 \u001b[0m\u001b[01;35mresult-LO08_L1TP_025033_20201108_20201120_01_T1.png\u001b[0m\n", + "-rw-r--r-- 1 learner learner 910 Dec 16 16:43 result-LO08_L1TP_025033_20201108_20201120_01_T1.png.aux.xml\n" ] } ], @@ -781,7 +814,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 31, "id": "f36cb8c5-f305-4cb2-a5cc-0c9fd8592fb4", "metadata": {}, "outputs": [ @@ -789,7 +822,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "bucket: essentials-learner-2021-12-13 region: us-west2\n" + "bucket: essentials-learner-2021-12-16 region: us-west2\n" ] } ], @@ -801,7 +834,7 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 32, "id": "c2ae2b74-5e93-4c55-8bd7-63337f7dcbb8", "metadata": {}, "outputs": [ @@ -809,7 +842,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Creating gs://essentials-learner-2021-12-13/...\n" + "Creating gs://essentials-learner-2021-12-16/...\n" ] } ], @@ -827,7 +860,7 @@ }, { "cell_type": "code", - "execution_count": 63, + "execution_count": 35, "id": "681e6b1d-98bb-448a-a57e-f5674214effd", "metadata": {}, "outputs": [ @@ -835,8 +868,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "Copying file://output/result-LC08_L1GT_025033_20210127_20210305_01_T2.png.aux.xml [Content-Type=application/xml]...\n", - "Copying file://output/result-LC08_L1GT_025033_20210127_20210305_01_T2.png [Content-Type=image/png]...\n", + "Copying file://output/result-LO08_L1TP_025033_20201108_20201120_01_T1.png [Content-Type=image/png]...\n", + "Copying file://output/result-LO08_L1TP_025033_20201108_20201120_01_T1.png.aux.xml [Content-Type=application/xml]...\n", "==> NOTE: You are uploading one or more large file(s), which would run \n", "significantly faster if you enable parallel composite uploads. This\n", "feature can be enabled by editing the\n", @@ -849,13 +882,13 @@ "without a compiled crcmod, computing checksums on composite objects is\n", "so slow that gsutil disables downloads of composite objects.\n", "\n", - "\\ [2/2 files][164.5 MiB/164.5 MiB] 100% Done \n", - "Operation completed over 2 objects/164.5 MiB. \n" + "| [2/2 files][195.7 MiB/195.7 MiB] 100% Done \n", + "Operation completed over 2 objects/195.7 MiB. \n" ] } ], "source": [ - "gsutil -m cp -r \"output/\" \"gs://$BUCKET\"" + "gsutil -m cp -r \"output\" \"gs://$BUCKET\"" ] }, { @@ -868,7 +901,7 @@ }, { "cell_type": "code", - "execution_count": 66, + "execution_count": 37, "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", "metadata": {}, "outputs": [ @@ -876,15 +909,32 @@ "name": "stdout", "output_type": "stream", "text": [ - "164.53 MiB 2021-12-13T16:19:49Z gs://essentials-learner-2021-12-13/result-LC08_L1GT_025033_20210127_20210305_01_T2.png\n", - " 941 B 2021-12-13T16:19:47Z gs://essentials-learner-2021-12-13/result-LC08_L1GT_025033_20210127_20210305_01_T2.png.aux.xml\n", - " gs://essentials-learner-2021-12-13/output/\n", - "TOTAL: 2 objects, 172521251 bytes (164.53 MiB)\n" + "gs://essentials-learner-2021-12-16/output/\n" ] } ], "source": [ - "gsutil ls -lh \"gs://$BUCKET\"" + "gsutil ls \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "b1ea18e9-5861-4479-9948-3303952dee8a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "195.71 MiB 2021-12-16T16:45:52Z gs://essentials-learner-2021-12-16/output/result-LO08_L1TP_025033_20201108_20201120_01_T1.png\n", + " 910 B 2021-12-16T16:45:50Z gs://essentials-learner-2021-12-16/output/result-LO08_L1TP_025033_20201108_20201120_01_T1.png.aux.xml\n", + "TOTAL: 2 objects, 205222402 bytes (195.72 MiB)\n" + ] + } + ], + "source": [ + "gsutil ls -lh \"gs://$BUCKET/output\"" ] }, { @@ -896,6 +946,24 @@ "\n", "You now can view the results by using the Google Cloud Platform Web Console and navigating to \"Cloud Storage\", selecting the bucket, and then the result object you wish to view (select the `.png` file). You will need to click the \"Preview\" button given the large size of the image." ] + }, + { + "cell_type": "markdown", + "id": "40d9ab41-8920-45f2-8218-550baac5b069", + "metadata": {}, + "source": [ + "## Cleanup\n", + "\n", + "We will now leave the resources running in order to learn more about monitoring costs and will clean up all the resources as the end of Lesson. **Don't forget to do this!**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "691daf6d-89b8-4f32-ac1e-e234222c0efd", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": {