From 19a2cf2abb21ba05b38cd7be56da39ee09d953a1 Mon Sep 17 00:00:00 2001 From: jhickszen Date: Mon, 10 Jan 2022 15:19:55 -0500 Subject: [PATCH] rearanged Azure TOC to match GCP --- ...ompute.ipynb => 02_intro_to_compute.ipynb} | 0 ....ipynb => 03_intro_to_cloud_storage.ipynb} | 0 ...tro_to_cli.ipynb => 04_intro_to_cli.ipynb} | 4 +- ...nb => 05_intro_to_cloud_storage_cli.ipynb} | 2 +- content/Azure/06_running_analysis.ipynb | 1488 +++++++++++++++++ ...s.ipynb => 08_cleaning_up_resources.ipynb} | 0 content/Azure/08_running_analysis.ipynb | 47 - content/Azure/10_basic_networking_extra.ipynb | 469 ------ content/Azure/file | 1 - ...e_cli.ipynb => intro_to_compute_cli.ipynb} | 0 content/_toc.yml | 14 +- 11 files changed, 1497 insertions(+), 528 deletions(-) rename content/Azure/{03_intro_to_compute.ipynb => 02_intro_to_compute.ipynb} (100%) rename content/Azure/{05_intro_to_cloud_storage.ipynb => 03_intro_to_cloud_storage.ipynb} (100%) rename content/Azure/{02_intro_to_cli.ipynb => 04_intro_to_cli.ipynb} (99%) rename content/Azure/{06_intro_to_cloud_storage_cli.ipynb => 05_intro_to_cloud_storage_cli.ipynb} (98%) create mode 100644 content/Azure/06_running_analysis.ipynb rename content/Azure/{09_cleaning_up_resources.ipynb => 08_cleaning_up_resources.ipynb} (100%) delete mode 100644 content/Azure/08_running_analysis.ipynb delete mode 100644 content/Azure/10_basic_networking_extra.ipynb delete mode 100644 content/Azure/file rename content/Azure/{04_intro_to_compute_cli.ipynb => intro_to_compute_cli.ipynb} (100%) diff --git a/content/Azure/03_intro_to_compute.ipynb b/content/Azure/02_intro_to_compute.ipynb similarity index 100% rename from content/Azure/03_intro_to_compute.ipynb rename to content/Azure/02_intro_to_compute.ipynb diff --git a/content/Azure/05_intro_to_cloud_storage.ipynb b/content/Azure/03_intro_to_cloud_storage.ipynb similarity index 100% rename from content/Azure/05_intro_to_cloud_storage.ipynb rename to content/Azure/03_intro_to_cloud_storage.ipynb diff --git a/content/Azure/02_intro_to_cli.ipynb b/content/Azure/04_intro_to_cli.ipynb similarity index 99% rename from content/Azure/02_intro_to_cli.ipynb rename to content/Azure/04_intro_to_cli.ipynb index b1e6158..4ee4e92 100644 --- a/content/Azure/02_intro_to_cli.ipynb +++ b/content/Azure/04_intro_to_cli.ipynb @@ -2,10 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "3ba4b111", + "id": "d448d2c1", "metadata": {}, "source": [ - "# Navigating the Command Line Interface (CLI)\n", + "# Introduction to the Azure CLI\n", "\n", "Learner Questions\n", " * How do I use the Cloud CLI?\n", diff --git a/content/Azure/06_intro_to_cloud_storage_cli.ipynb b/content/Azure/05_intro_to_cloud_storage_cli.ipynb similarity index 98% rename from content/Azure/06_intro_to_cloud_storage_cli.ipynb rename to content/Azure/05_intro_to_cloud_storage_cli.ipynb index 652b66f..5451fe8 100644 --- a/content/Azure/06_intro_to_cloud_storage_cli.ipynb +++ b/content/Azure/05_intro_to_cloud_storage_cli.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Intro to Cloud Storage CLI\n", + "# Managing Cloud Storage from the Command Line\n", "\n", "**Overview**\n", "\n", diff --git a/content/Azure/06_running_analysis.ipynb b/content/Azure/06_running_analysis.ipynb new file mode 100644 index 0000000..052ffe8 --- /dev/null +++ b/content/Azure/06_running_analysis.ipynb @@ -0,0 +1,1488 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "dc57021c", + "metadata": {}, + "source": [ + "# Putting It All Together\n", + "\n", + "```{admonition} Overview\n", + ":class: tip\n", + "\n", + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + "* Can you show me an example?\n", + "\n", + "**Objectives:**\n", + "* Create a simple workflow using a cloud VM and cloud object storage.\n", + "* Understand the basics of Identity and Access Management (IAM)\n", + "* Add collaborators to a Bucket with appropriate permissions.\n", + "\n", + "\n", + "\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "6291edee-c2df-4941-9b8e-de42649640f9", + "metadata": {}, + "source": [ + "## Create a VM\n", + "\n", + "Since we only create resources as we need them in the cloud. As an exercise you will now create a VM for our analysis. In this case will give the VM **Full** access to **Storage**. \n", + "\n", + "Using the console navigate to the \"Compute Engine\" service and create a new VM with the following properties.\n", + " * Call the VM \"essentials\"\n", + " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", + " * Feel free to select a larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + " \n", + "When you are done connect to the machine as described below." + ] + }, + { + "cell_type": "markdown", + "id": "c87e200c-1d18-44f5-b4e6-28b697b04b3c", + "metadata": {}, + "source": [ + "## Connect to the VM\n", + "\n", + "First login to the instance from the Cloud Shell by running the following command:\n", + "```\n", + "gcloud compute ssh essentials\n", + "```\n", + "If prompted for a zone select `n` to find it automatically. You can see an example session below." + ] + }, + { + "cell_type": "markdown", + "id": "44f32772-0abb-4fa7-a013-202fc646df5e", + "metadata": {}, + "source": [ + "```\n", + "learner@cloudshell:~ (just-armor-301114)$ gcloud compute ssh essentials\n", + "Did you mean zone [us-central1-b] for instance: [essentials] (Y/n)? n\n", + "\n", + "No zone specified. Using zone [us-west2-c] for instance: [essentials].\n", + "Linux essentials 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64\n", + "\n", + "The programs included with the Debian GNU/Linux system are free software;\n", + "the exact distribution terms for each program are described in the\n", + "individual files in /usr/share/doc/*/copyright.\n", + "\n", + "Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent\n", + "permitted by applicable law.\n", + "Last login: Tue Nov 9 20:12:49 2021 from 34.133.99.196\n", + "learner@essentials:~$\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "0ab6ad28-24bf-4c0f-b4dd-3be2ec25aaa7", + "metadata": {}, + "source": [ + "## Secure the VM\n", + "\n", + "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs security patches." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "0f2df994-202e-4e3e-96c2-1536d8adb036", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\n", + "Hit:2 http://deb.debian.org/debian buster InRelease\n", + "Hit:3 http://deb.debian.org/debian buster-updates InRelease\n", + "Hit:4 http://deb.debian.org/debian buster-backports InRelease\n", + "Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease\n", + "Hit:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease\n", + "Get:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease [5526 B]\n", + "Fetched 70.9 kB in 1s (91.3 kB/s) \u001b[0m\u001b[33m\n", + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "2 packages can be upgraded. Run 'apt list --upgradable' to see them.\n" + ] + } + ], + "source": [ + "sudo apt update\n", + "sudo unattended-upgrades" + ] + }, + { + "cell_type": "markdown", + "id": "b7c4db9e-f098-41bc-80d2-b524444eec7f", + "metadata": {}, + "source": [ + "## Get Example Code\n", + "\n", + "We will now install `git` and use it to download the example code into your home directory. For those of you who are unfamiliar with git, it is a way to collaboratively manage files and we will only be using it to download the example that we will be using. " + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "05fadd30-4f35-4a93-b713-cc132131b948", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "git is already the newest version (1:2.20.1-2+deb10u3).\n", + "0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.\n" + ] + } + ], + "source": [ + "sudo apt-get install --yes git" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", + "metadata": {}, + "outputs": [], + "source": [ + "cd ~" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cloning into 'CLASS-Examples'...\n", + "remote: Enumerating objects: 66, done.\u001b[K\n", + "remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66\u001b[K\n", + "Unpacking objects: 100% (66/66), done.\n" + ] + } + ], + "source": [ + "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" + ] + }, + { + "cell_type": "markdown", + "id": "3d2eb9fc-bec9-449d-a7ab-f3127f59d512", + "metadata": {}, + "source": [ + "We now change the current directory to the `landsat` directory in the `CLASS-Examples` directory that was just created by the previous git command." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", + "metadata": {}, + "outputs": [], + "source": [ + "cd ~/CLASS-Examples/landsat/" + ] + }, + { + "cell_type": "markdown", + "id": "6ea81d23-e1b0-4cac-a3ac-fed7a9723eca", + "metadata": {}, + "source": [ + "Your prompt should now change showing the current directory as follows.\n", + "```\n", + "learner@essentials:~/CLASS-Examples/landsat$\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 28\n", + "-rw-r--r-- 1 learner learner 960 Dec 17 19:00 ReadMe.md\n", + "-rw-r--r-- 1 learner learner 72 Dec 17 19:00 clean.sh\n", + "-rw-r--r-- 1 learner learner 613 Dec 17 19:00 combine.py\n", + "-rw-r--r-- 1 learner learner 280 Dec 17 19:00 download.sh\n", + "-rw-r--r-- 1 learner learner 314 Dec 17 19:00 get-index.sh\n", + "-rw-r--r-- 1 learner learner 76 Dec 17 19:00 search.json\n", + "-rw-r--r-- 1 learner learner 783 Dec 17 19:00 search.py\n" + ] + } + ], + "source": [ + "ls -l" + ] + }, + { + "cell_type": "markdown", + "id": "2b8b3144-7453-4350-a1cf-7fa74af2bcbf", + "metadata": {}, + "source": [ + "## Access the bucket\n", + "\n", + "First test that our tools are working and that we can access the public bucket that we will be using." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://gcp-public-data-landsat/index.csv.gz\n", + "gs://gcp-public-data-landsat/LC08/\n", + "gs://gcp-public-data-landsat/LE07/\n", + "gs://gcp-public-data-landsat/LM01/\n", + "gs://gcp-public-data-landsat/LM02/\n", + "gs://gcp-public-data-landsat/LM03/\n", + "gs://gcp-public-data-landsat/LM04/\n", + "gs://gcp-public-data-landsat/LM05/\n", + "gs://gcp-public-data-landsat/LO08/\n", + "gs://gcp-public-data-landsat/LT04/\n", + "gs://gcp-public-data-landsat/LT05/\n", + "gs://gcp-public-data-landsat/LT08/\n" + ] + } + ], + "source": [ + "gsutil ls gs://gcp-public-data-landsat/" + ] + }, + { + "cell_type": "markdown", + "id": "89acffba-cbce-436c-98dd-05467b6675a6", + "metadata": {}, + "source": [ + "The index file is a list of all the files in the bucket and we can use it to search and filter files.\n", + "\n", + "We will get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mkdir: created directory 'data'\n" + ] + } + ], + "source": [ + "mkdir -v data" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying gs://gcp-public-data-landsat/index.csv.gz...\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "\\ [1 files][761.0 MiB/761.0 MiB] 42.0 MiB/s \n", + "Operation completed over 1 objects/761.0 MiB. \n" + ] + } + ], + "source": [ + "gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/" + ] + }, + { + "cell_type": "markdown", + "id": "76b905b4-1c2a-4960-a14f-974b77f671cd", + "metadata": {}, + "source": [ + "We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", + "metadata": {}, + "outputs": [], + "source": [ + "gzip -df data/index.csv.gz" + ] + }, + { + "cell_type": "markdown", + "id": "75faa74a-b458-472b-9018-034c89e75315", + "metadata": {}, + "source": [ + "Again, check what happened." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "b005876c-f9af-43d6-80c6-f04295413b9b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 2.6G\n", + "-rw-r--r-- 1 learner learner 2.6G Dec 17 19:01 index.csv\n" + ] + } + ], + "source": [ + "ls -lh data" + ] + }, + { + "cell_type": "markdown", + "id": "fcde8334-f58d-4c3d-995a-2491be0f95ea", + "metadata": {}, + "source": [ + "We will now explore the data. The `head` command simply displays the first few lines of the file." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL\n", + "LM51940171984262KIS00,LM05_L1GS_194017_19840918_20210913_01_T2,LANDSAT_5,MSS,1984-09-18,01,T2,1984-09-18T09:36:27.0970000Z,L1GS,194,17,51.0,62.58023,60.38359,14.45307,19.13901,34758212,gs://gcp-public-data-landsat/LM05/01/194/017/LM05_L1GS_194017_19840918_20210913_01_T2\n", + "LM10420331972311AAA04,LM01_L1TP_042033_19721106_20180429_01_T2,LANDSAT_1,MSS,1972-11-06,01,T2,1972-11-06T17:49:00.5000000Z,L1TP,42,33,3.0,39.68891,37.707,-115.59896,-112.90344,26932395,gs://gcp-public-data-landsat/LM01/01/042/033/LM01_L1TP_042033_19721106_20180429_01_T2\n", + "LT50150151992246PAC00,LT05_L1TP_015015_19920902_20170121_01_T1,LANDSAT_5,TM,1992-09-02,01,T1,1992-09-02T15:01:34.1390130Z,L1TP,15,15,90.0,65.32118,63.11726,-67.3438,-61.89671,118785510,gs://gcp-public-data-landsat/LT05/01/015/015/LT05_L1TP_015015_19920902_20170121_01_T1\n" + ] + } + ], + "source": [ + "head --lines=4 data/index.csv" + ] + }, + { + "cell_type": "markdown", + "id": "532e6da3-302a-4e8a-8570-752995f30f1d", + "metadata": {}, + "source": [ + "## Search for Data\n", + "\n", + "We can see the data is well formed and what we expect. We will now use this data to download data related to a specific point and for the Landsat 8. The following script does a simple filter." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "#!/usr/bin/python3\n", + "import json\n", + "import csv\n", + "import sys\n", + "\n", + "# Example: Burr Oak Tree\n", + "# 38.899313,-92.464562 (Lat north+, Long west-) ; Landsat Path 025, Row 033\n", + "config=json.load(open(\"search.json\"))\n", + "lat,lon=config['lat'],config['lon']\n", + "landsat=config['landsat']\n", + "\n", + "reader=csv.reader(sys.stdin)\n", + "header=next(reader) # skip header\n", + "for l in reader:\n", + " SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL=l\n", + " west,east=float(WEST_LON),float(EAST_LON)\n", + " north,south=float(NORTH_LAT),float(SOUTH_LAT)\n", + " if SPACECRAFT_ID==landsat and north >= lat and south <= lat and west <= lon and east >= lon:\n", + " print(BASE_URL) # output BASE_URL\n" + ] + } + ], + "source": [ + "cat search.py" + ] + }, + { + "cell_type": "markdown", + "id": "4aa3de47-3dd4-4a0f-9f07-f2f004de7054", + "metadata": {}, + "source": [ + "We can see that the actual search data comes from the file `search.json`. The program reads the data from the standard input and iterates over all rows in the CSV file. It filters the results for which the image contains the pint and prints out the bucket URL for them. We are interested in all products that contain the Burr Oak Tree." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"lat\": 38.899313,\n", + " \"lon\": -92.464562,\n", + " \"landsat\": \"LANDSAT_8\"\n", + "}\n" + ] + } + ], + "source": [ + "cat search.json" + ] + }, + { + "cell_type": "markdown", + "id": "cbb27235-6bc4-4eb6-b668-5c30427a28b8", + "metadata": {}, + "source": [ + "Now lets test this on a subset of the data." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20150127_20170302_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20161028_20170219_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20191224_20200110_01_T1\n" + ] + } + ], + "source": [ + "head --lines=200000 data/index.csv | python3 search.py" + ] + }, + { + "cell_type": "markdown", + "id": "a76f24f8-3b2d-4c0d-880b-f2911b9d9b84", + "metadata": {}, + "source": [ + "## Download the Data\n", + "\n", + "Now that we have a list of folders we are interested, we will now download them with a simple script that takes bucket addresses (URL's) and downloads them with the `gsutil` program." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "3572c518-df83-4906-bfa6-a37bde2a5063", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "#!/bin/bash\n", + "\n", + "# Read space separated URL from STDIN and download \n", + "while read -r URL ; do\n", + " echo \"+++ $URL\"\n", + " # -m parallel\n", + " # -n no-clobber (do not re-download data)\n", + " # -r recursive (download all the data in the specified URL)\n", + " gsutil -m cp -n -r \"${URL}/\" data/\n", + "done\n" + ] + } + ], + "source": [ + "cat download.sh" + ] + }, + { + "cell_type": "markdown", + "id": "a02100bd-f8c5-42dd-975e-a9eb5369bc81", + "metadata": {}, + "source": [ + "Get the first 2 datasets" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B9.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B8.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B11.TIF...\n", + "==> NOTE: You are downloading one or more large file(s), which would\n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_MTL.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B5.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B7.TIF...\n", + "| [14/14 files][997.2 MiB/997.2 MiB] 100% Done 65.3 MiB/s ETA 00:00:00 \n", + "Operation completed over 14 objects/997.2 MiB. \n", + "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B8.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B9.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B7.TIF...\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_MTL.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B5.TIF...\n", + "- [14/14 files][979.2 MiB/979.2 MiB] 100% Done 14.9 MiB/s ETA 00:00:00 \n", + "Operation completed over 14 objects/979.2 MiB. \n" + ] + } + ], + "source": [ + "head --lines=200000 data/index.csv | python3 search.py | head --lines=2 | bash download.sh" + ] + }, + { + "cell_type": "markdown", + "id": "14c23aab-a6e8-439d-8e55-44b7148f74d5", + "metadata": {}, + "source": [ + "Check that the data was downloaded" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 2665792\n", + "drwxr-xr-x 2 learner learner 4096 Dec 17 19:02 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20170913_20170928_01_T1\u001b[0m\n", + "drwxr-xr-x 2 learner learner 4096 Dec 17 19:01 \u001b[01;34mLC08_L1TP_025033_20211010_20211018_01_T1\u001b[0m\n", + "-rw-r--r-- 1 learner learner 2729754843 Dec 17 19:01 index.csv\n" + ] + } + ], + "source": [ + "ls -l data" + ] + }, + { + "cell_type": "markdown", + "id": "6073d5f6-68ef-41df-8044-73f221ce8780", + "metadata": {}, + "source": [ + "## Processing the Data\n", + "\n", + "We will now use simple script to combine the color bands and export it as a PNG" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "#!/usr/bin/python3\n", + "import os\n", + "import rasterio\n", + "\n", + "# Open the first directory in data, could walk entire tree\n", + "for dirname, dirs, files in os.walk('data'):\n", + " source = dirs[0]\n", + " break\n", + "\n", + "# Open band (B2/Blue) and copy metadata for result.\n", + "with rasterio.open(\"data/%s/%s_B2.TIF\" % (source, source)) as band2:\n", + " meta = band2.meta\n", + "\n", + "# Combine bands into PNG\n", + "meta.update(count = 3, driver='PNG')\n", + "with rasterio.open(\"output/result-%s.png\" % source, 'w+', **meta) as output:\n", + " for i in range(1, 4):\n", + " print(source, i)\n", + " output.write_band(i,rasterio.open(\"data/%s/%s_B%d.TIF\" % (source, source, i+3)).read(1))\n" + ] + } + ], + "source": [ + "cat combine.py" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Traceback (most recent call last):\n", + " File \"combine.py\", line 3, in \n", + " import rasterio\n", + "ModuleNotFoundError: No module named 'rasterio'\n" + ] + } + ], + "source": [ + "python3 combine.py\n", + "/bin/true # ignore this line used for jupyter" + ] + }, + { + "cell_type": "markdown", + "id": "52ada3ec-26cd-4e8b-94c2-8ed33e98be6e", + "metadata": {}, + "source": [ + "Oops, let's install the library (note: the output will be slightly different due to how this Lesson is built)\n", + "and while we are at it we also need to create the output directory." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mkdir: created directory 'output/'\n", + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "The following additional packages will be installed:\n", + " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", + " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", + " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", + " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", + " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", + " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", + " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", + " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", + " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", + " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", + " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", + " python3-chardet python3-click python3-click-plugins python3-cligj\n", + " python3-colorama python3-html5lib python3-lxml python3-numpy\n", + " python3-pyparsing python3-snuggs python3-soupsieve python3-webencodings\n", + "Suggested packages:\n", + " geotiff-bin gdal-bin libgeotiff-epsg libhdf4-doc libhdf4-alt-dev hdf4-tools\n", + " liblcms2-utils libmyodbc odbc-postgresql tdsodbc unixodbc-bin ogdi-bin\n", + " poppler-utils ghostscript fonts-japanese-mincho | fonts-ipafont-mincho\n", + " fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai\n", + " fonts-arphic-uming fonts-nanum python-attr-doc python3-genshi\n", + " python3-lxml-dbg python-lxml-doc gfortran python-numpy-doc python3-pytest\n", + " python3-numpy-dbg python-pyparsing-doc\n", + "The following NEW packages will be installed:\n", + " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", + " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", + " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", + " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", + " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", + " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", + " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", + " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", + " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", + " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", + " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", + " python3-chardet python3-click python3-click-plugins python3-cligj\n", + " python3-colorama python3-html5lib python3-lxml python3-numpy\n", + " python3-pyparsing python3-rasterio python3-snuggs python3-soupsieve\n", + " python3-webencodings\n", + "0 upgraded, 79 newly installed, 0 to remove and 2 not upgraded.\n", + "Need to get 46.8 MB of archives.\n", + "After this operation, 172 MB of additional disk space will be used.\n", + "Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", + "Get:2 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", + "Get:3 http://deb.debian.org/debian buster/main amd64 fonts-dejavu-core all 2.37-1 [1068 kB]\n", + "Get:4 http://deb.debian.org/debian buster/main amd64 fontconfig-config all 2.13.1-2 [280 kB]\n", + "Get:5 http://deb.debian.org/debian buster/main amd64 gdal-data all 2.4.0+dfsg-1 [744 kB]\n", + "Get:6 http://deb.debian.org/debian buster/main amd64 libaec0 amd64 1.0.2-1 [20.1 kB]\n", + "Get:7 http://deb.debian.org/debian buster/main amd64 libgfortran5 amd64 8.3.0-6 [581 kB]\n", + "Get:8 http://deb.debian.org/debian buster/main amd64 libblas3 amd64 3.8.0-2 [148 kB]\n", + "Get:9 http://deb.debian.org/debian buster/main amd64 liblapack3 amd64 3.8.0-2 [2110 kB]\n", + "Get:10 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", + "Get:11 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", + "Get:12 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", + "Get:13 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", + "Get:14 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", + "Get:15 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", + "Get:16 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", + "Get:17 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", + "Get:18 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", + "Get:19 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", + "Get:20 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", + "Get:21 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", + "Get:22 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", + "Get:23 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", + "Get:24 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", + "Get:25 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", + "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", + "Get:27 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", + "Get:28 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", + "Get:29 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", + "Get:30 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", + "Get:31 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", + "Get:32 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", + "Get:33 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", + "Get:34 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", + "Get:35 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", + "Get:36 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", + "Get:37 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", + "Get:38 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", + "Get:39 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", + "Get:40 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", + "Get:41 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", + "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlconvenience1 amd64 1.3.0-7 [50.1 kB]\n", + "Get:43 http://deb.debian.org/debian buster/main amd64 libkmlregionator1 amd64 1.3.0-7 [25.1 kB]\n", + "Get:44 http://deb.debian.org/debian buster/main amd64 libkmlxsd1 amd64 1.3.0-7 [34.6 kB]\n", + "Get:45 http://deb.debian.org/debian buster/main amd64 mysql-common all 5.8+1.0.5 [7324 B]\n", + "Get:46 http://deb.debian.org/debian buster/main amd64 mariadb-common all 1:10.3.31-0+deb10u1 [32.7 kB]\n", + "Get:47 http://deb.debian.org/debian buster/main amd64 libmariadb3 amd64 1:10.3.31-0+deb10u1 [173 kB]\n", + "Get:48 http://deb.debian.org/debian buster/main amd64 libnetcdf13 amd64 1:4.6.2-1 [403 kB]\n", + "Get:49 http://deb.debian.org/debian buster/main amd64 libltdl7 amd64 2.4.6-9 [390 kB]\n", + "Get:50 http://deb.debian.org/debian buster/main amd64 libodbc1 amd64 2.3.6-0.1 [223 kB]\n", + "Get:51 http://deb.debian.org/debian buster/main amd64 libogdi3.2 amd64 3.2.1+ds-4 [241 kB]\n", + "Get:52 http://deb.debian.org/debian buster/main amd64 libopenjp2-7 amd64 2.3.0-2+deb10u2 [158 kB]\n", + "Get:53 http://deb.debian.org/debian buster/main amd64 liblcms2-2 amd64 2.9-3 [145 kB]\n", + "Get:54 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]\n", + "Get:55 http://deb.debian.org/debian buster/main amd64 libpoppler82 amd64 0.71.0-5 [1507 kB]\n", + "Get:56 http://deb.debian.org/debian buster/main amd64 libqhull7 amd64 2015.2-4 [196 kB]\n", + "Get:57 http://deb.debian.org/debian buster/main amd64 libspatialite7 amd64 4.3.0a-5+b2 [1256 kB]\n", + "Get:58 http://deb.debian.org/debian buster/main amd64 libxerces-c3.2 amd64 3.2.2+debian-1+deb10u1 [886 kB]\n", + "Get:59 http://deb.debian.org/debian buster/main amd64 odbcinst amd64 2.3.6-0.1 [48.4 kB]\n", + "Get:60 http://deb.debian.org/debian buster/main amd64 odbcinst1debian2 amd64 2.3.6-0.1 [78.1 kB]\n", + "Get:61 http://deb.debian.org/debian buster/main amd64 libgdal20 amd64 2.4.0+dfsg-1+b1 [6171 kB]\n", + "Get:62 http://deb.debian.org/debian buster/main amd64 libxslt1.1 amd64 1.1.32-2.2~deb10u1 [237 kB]\n", + "Get:63 http://deb.debian.org/debian buster/main amd64 proj-bin amd64 5.2.0-1 [105 kB]\n", + "Get:64 http://deb.debian.org/debian buster/main amd64 python3-affine all 2.2.2-1 [14.5 kB]\n", + "Get:65 http://deb.debian.org/debian buster/main amd64 python3-attr all 18.2.0-1 [37.3 kB]\n", + "Get:66 http://deb.debian.org/debian buster/main amd64 python3-soupsieve all 1.8+dfsg-1 [27.6 kB]\n", + "Get:67 http://deb.debian.org/debian buster/main amd64 python3-bs4 all 4.7.1-1 [94.1 kB]\n", + "Get:68 http://deb.debian.org/debian buster/main amd64 python3-chardet all 3.0.4-3 [80.5 kB]\n", + "Get:69 http://deb.debian.org/debian buster/main amd64 python3-colorama all 0.3.7-1 [18.1 kB]\n", + "Get:70 http://deb.debian.org/debian buster/main amd64 python3-click all 7.0-1 [73.6 kB]\n", + "Get:71 http://deb.debian.org/debian buster/main amd64 python3-click-plugins all 1.0.4-1 [7540 B]\n", + "Get:72 http://deb.debian.org/debian buster/main amd64 python3-cligj all 0.5.0-1 [8212 B]\n", + "Get:73 http://deb.debian.org/debian buster/main amd64 python3-webencodings all 0.5.1-1 [10.9 kB]\n", + "Get:74 http://deb.debian.org/debian buster/main amd64 python3-html5lib all 1.0.1-1 [89.5 kB]\n", + "Get:75 http://deb.debian.org/debian buster/main amd64 python3-lxml amd64 4.3.2-1+deb10u3 [1162 kB]\n", + "Get:76 http://deb.debian.org/debian buster/main amd64 python3-numpy amd64 1:1.16.2-1 [2119 kB]\n", + "Get:77 http://deb.debian.org/debian buster/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [89.6 kB]\n", + "Get:78 http://deb.debian.org/debian buster/main amd64 python3-snuggs all 1.4.3-1 [7228 B]\n", + "Get:79 http://deb.debian.org/debian buster/main amd64 python3-rasterio amd64 1.0.21-1 [818 kB]\n", + "Fetched 46.8 MB in 1s (85.4 MB/s) \n", + "Extracting templates from packages: 100%\n", + "Preconfiguring packages ...\n", + "Selecting previously unselected package poppler-data.\n", + "(Reading database ... 57529 files and directories currently installed.)\n", + "Preparing to unpack .../00-poppler-data_0.4.9-2_all.deb ...\n", + "Unpacking poppler-data (0.4.9-2) ...\n", + "Selecting previously unselected package fonts-dejavu-core.\n", + "Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ...\n", + "Unpacking fonts-dejavu-core (2.37-1) ...\n", + "Selecting previously unselected package fontconfig-config.\n", + "Preparing to unpack .../02-fontconfig-config_2.13.1-2_all.deb ...\n", + "Unpacking fontconfig-config (2.13.1-2) ...\n", + "Selecting previously unselected package gdal-data.\n", + "Preparing to unpack .../03-gdal-data_2.4.0+dfsg-1_all.deb ...\n", + "Unpacking gdal-data (2.4.0+dfsg-1) ...\n", + "Selecting previously unselected package libaec0:amd64.\n", + "Preparing to unpack .../04-libaec0_1.0.2-1_amd64.deb ...\n", + "Unpacking libaec0:amd64 (1.0.2-1) ...\n", + "Selecting previously unselected package libgfortran5:amd64.\n", + "Preparing to unpack .../05-libgfortran5_8.3.0-6_amd64.deb ...\n", + "Unpacking libgfortran5:amd64 (8.3.0-6) ...\n", + "Selecting previously unselected package libblas3:amd64.\n", + "Preparing to unpack .../06-libblas3_3.8.0-2_amd64.deb ...\n", + "Unpacking libblas3:amd64 (3.8.0-2) ...\n", + "Selecting previously unselected package liblapack3:amd64.\n", + "Preparing to unpack .../07-liblapack3_3.8.0-2_amd64.deb ...\n", + "Unpacking liblapack3:amd64 (3.8.0-2) ...\n", + "Selecting previously unselected package libarpack2:amd64.\n", + "Preparing to unpack .../08-libarpack2_3.7.0-2_amd64.deb ...\n", + "Unpacking libarpack2:amd64 (3.7.0-2) ...\n", + "Selecting previously unselected package libsuperlu5:amd64.\n", + "Preparing to unpack .../09-libsuperlu5_5.2.1+dfsg1-4_amd64.deb ...\n", + "Unpacking libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", + "Selecting previously unselected package libarmadillo9.\n", + "Preparing to unpack .../10-libarmadillo9_1%3a9.200.7+dfsg-1_amd64.deb ...\n", + "Unpacking libarmadillo9 (1:9.200.7+dfsg-1) ...\n", + "Selecting previously unselected package libcharls2:amd64.\n", + "Preparing to unpack .../11-libcharls2_2.0.0+dfsg-1_amd64.deb ...\n", + "Unpacking libcharls2:amd64 (2.0.0+dfsg-1) ...\n", + "Selecting previously unselected package libicu63:amd64.\n", + "Preparing to unpack .../12-libicu63_63.1-6+deb10u2_amd64.deb ...\n", + "Unpacking libicu63:amd64 (63.1-6+deb10u2) ...\n", + "Selecting previously unselected package libxml2:amd64.\n", + "Preparing to unpack .../13-libxml2_2.9.4+dfsg1-7+deb10u2_amd64.deb ...\n", + "Unpacking libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", + "Selecting previously unselected package libdap25:amd64.\n", + "Preparing to unpack .../14-libdap25_3.20.3-1_amd64.deb ...\n", + "Unpacking libdap25:amd64 (3.20.3-1) ...\n", + "Selecting previously unselected package libdapclient6v5:amd64.\n", + "Preparing to unpack .../15-libdapclient6v5_3.20.3-1_amd64.deb ...\n", + "Unpacking libdapclient6v5:amd64 (3.20.3-1) ...\n", + "Selecting previously unselected package libdapserver7v5:amd64.\n", + "Preparing to unpack .../16-libdapserver7v5_3.20.3-1_amd64.deb ...\n", + "Unpacking libdapserver7v5:amd64 (3.20.3-1) ...\n", + "Selecting previously unselected package libepsilon1:amd64.\n", + "Preparing to unpack .../17-libepsilon1_0.9.2+dfsg-4_amd64.deb ...\n", + "Unpacking libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", + "Selecting previously unselected package libfontconfig1:amd64.\n", + "Preparing to unpack .../18-libfontconfig1_2.13.1-2_amd64.deb ...\n", + "Unpacking libfontconfig1:amd64 (2.13.1-2) ...\n", + "Selecting previously unselected package libfreexl1:amd64.\n", + "Preparing to unpack .../19-libfreexl1_1.0.5-3_amd64.deb ...\n", + "Unpacking libfreexl1:amd64 (1.0.5-3) ...\n", + "Selecting previously unselected package libfyba0:amd64.\n", + "Preparing to unpack .../20-libfyba0_4.1.1-6_amd64.deb ...\n", + "Unpacking libfyba0:amd64 (4.1.1-6) ...\n", + "Selecting previously unselected package libgeos-3.7.1:amd64.\n", + "Preparing to unpack .../21-libgeos-3.7.1_3.7.1-1_amd64.deb ...\n", + "Unpacking libgeos-3.7.1:amd64 (3.7.1-1) ...\n", + "Selecting previously unselected package libgeos-c1v5:amd64.\n", + "Preparing to unpack .../22-libgeos-c1v5_3.7.1-1_amd64.deb ...\n", + "Unpacking libgeos-c1v5:amd64 (3.7.1-1) ...\n", + "Selecting previously unselected package proj-data.\n", + "Preparing to unpack .../23-proj-data_5.2.0-1_all.deb ...\n", + "Unpacking proj-data (5.2.0-1) ...\n", + "Selecting previously unselected package libproj13:amd64.\n", + "Preparing to unpack .../24-libproj13_5.2.0-1_amd64.deb ...\n", + "Unpacking libproj13:amd64 (5.2.0-1) ...\n", + "Selecting previously unselected package libjbig0:amd64.\n", + "Preparing to unpack .../25-libjbig0_2.1-3.1+b2_amd64.deb ...\n", + "Unpacking libjbig0:amd64 (2.1-3.1+b2) ...\n", + "Selecting previously unselected package libjpeg62-turbo:amd64.\n", + "Preparing to unpack .../26-libjpeg62-turbo_1%3a1.5.2-2+deb10u1_amd64.deb ...\n", + "Unpacking libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", + "Selecting previously unselected package libwebp6:amd64.\n", + "Preparing to unpack .../27-libwebp6_0.6.1-2+deb10u1_amd64.deb ...\n", + "Unpacking libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", + "Selecting previously unselected package libtiff5:amd64.\n", + "Preparing to unpack .../28-libtiff5_4.1.0+git191117-2~deb10u3_amd64.deb ...\n", + "Unpacking libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", + "Selecting previously unselected package libgeotiff2:amd64.\n", + "Preparing to unpack .../29-libgeotiff2_1.4.3-1_amd64.deb ...\n", + "Unpacking libgeotiff2:amd64 (1.4.3-1) ...\n", + "Selecting previously unselected package libgif7:amd64.\n", + "Preparing to unpack .../30-libgif7_5.1.4-3_amd64.deb ...\n", + "Unpacking libgif7:amd64 (5.1.4-3) ...\n", + "Selecting previously unselected package libhdf4-0-alt.\n", + "Preparing to unpack .../31-libhdf4-0-alt_4.2.13-4_amd64.deb ...\n", + "Unpacking libhdf4-0-alt (4.2.13-4) ...\n", + "Selecting previously unselected package libsz2:amd64.\n", + "Preparing to unpack .../32-libsz2_1.0.2-1_amd64.deb ...\n", + "Unpacking libsz2:amd64 (1.0.2-1) ...\n", + "Selecting previously unselected package libhdf5-103:amd64.\n", + "Preparing to unpack .../33-libhdf5-103_1.10.4+repack-10_amd64.deb ...\n", + "Unpacking libhdf5-103:amd64 (1.10.4+repack-10) ...\n", + "Selecting previously unselected package libminizip1:amd64.\n", + "Preparing to unpack .../34-libminizip1_1.1-8+b1_amd64.deb ...\n", + "Unpacking libminizip1:amd64 (1.1-8+b1) ...\n", + "Selecting previously unselected package liburiparser1:amd64.\n", + "Preparing to unpack .../35-liburiparser1_0.9.1-1_amd64.deb ...\n", + "Unpacking liburiparser1:amd64 (0.9.1-1) ...\n", + "Selecting previously unselected package libkmlbase1:amd64.\n", + "Preparing to unpack .../36-libkmlbase1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlbase1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmldom1:amd64.\n", + "Preparing to unpack .../37-libkmldom1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmldom1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlengine1:amd64.\n", + "Preparing to unpack .../38-libkmlengine1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlengine1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlconvenience1:amd64.\n", + "Preparing to unpack .../39-libkmlconvenience1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlconvenience1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlregionator1:amd64.\n", + "Preparing to unpack .../40-libkmlregionator1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlregionator1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlxsd1:amd64.\n", + "Preparing to unpack .../41-libkmlxsd1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlxsd1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package mysql-common.\n", + "Preparing to unpack .../42-mysql-common_5.8+1.0.5_all.deb ...\n", + "Unpacking mysql-common (5.8+1.0.5) ...\n", + "Selecting previously unselected package mariadb-common.\n", + "Preparing to unpack .../43-mariadb-common_1%3a10.3.31-0+deb10u1_all.deb ...\n", + "Unpacking mariadb-common (1:10.3.31-0+deb10u1) ...\n", + "Selecting previously unselected package libmariadb3:amd64.\n", + "Preparing to unpack .../44-libmariadb3_1%3a10.3.31-0+deb10u1_amd64.deb ...\n", + "Unpacking libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", + "Selecting previously unselected package libnetcdf13:amd64.\n", + "Preparing to unpack .../45-libnetcdf13_1%3a4.6.2-1_amd64.deb ...\n", + "Unpacking libnetcdf13:amd64 (1:4.6.2-1) ...\n", + "Selecting previously unselected package libltdl7:amd64.\n", + "Preparing to unpack .../46-libltdl7_2.4.6-9_amd64.deb ...\n", + "Unpacking libltdl7:amd64 (2.4.6-9) ...\n", + "Selecting previously unselected package libodbc1:amd64.\n", + "Preparing to unpack .../47-libodbc1_2.3.6-0.1_amd64.deb ...\n", + "Unpacking libodbc1:amd64 (2.3.6-0.1) ...\n", + "Selecting previously unselected package libogdi3.2.\n", + "Preparing to unpack .../48-libogdi3.2_3.2.1+ds-4_amd64.deb ...\n", + "Unpacking libogdi3.2 (3.2.1+ds-4) ...\n", + "Selecting previously unselected package libopenjp2-7:amd64.\n", + "Preparing to unpack .../49-libopenjp2-7_2.3.0-2+deb10u2_amd64.deb ...\n", + "Unpacking libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", + "Selecting previously unselected package liblcms2-2:amd64.\n", + "Preparing to unpack .../50-liblcms2-2_2.9-3_amd64.deb ...\n", + "Unpacking liblcms2-2:amd64 (2.9-3) ...\n", + "Selecting previously unselected package libnspr4:amd64.\n", + "Preparing to unpack .../51-libnspr4_2%3a4.20-1_amd64.deb ...\n", + "Unpacking libnspr4:amd64 (2:4.20-1) ...\n", + "Selecting previously unselected package libnss3:amd64.\n", + "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u4_amd64.deb ...\n", + "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Selecting previously unselected package libpoppler82:amd64.\n", + "Preparing to unpack .../53-libpoppler82_0.71.0-5_amd64.deb ...\n", + "Unpacking libpoppler82:amd64 (0.71.0-5) ...\n", + "Selecting previously unselected package libpq5:amd64.\n", + "Preparing to unpack .../54-libpq5_11.14-0+deb10u1_amd64.deb ...\n", + "Unpacking libpq5:amd64 (11.14-0+deb10u1) ...\n", + "Selecting previously unselected package libqhull7:amd64.\n", + "Preparing to unpack .../55-libqhull7_2015.2-4_amd64.deb ...\n", + "Unpacking libqhull7:amd64 (2015.2-4) ...\n", + "Selecting previously unselected package libspatialite7:amd64.\n", + "Preparing to unpack .../56-libspatialite7_4.3.0a-5+b2_amd64.deb ...\n", + "Unpacking libspatialite7:amd64 (4.3.0a-5+b2) ...\n", + "Selecting previously unselected package libxerces-c3.2:amd64.\n", + "Preparing to unpack .../57-libxerces-c3.2_3.2.2+debian-1+deb10u1_amd64.deb ...\n", + "Unpacking libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", + "Selecting previously unselected package odbcinst.\n", + "Preparing to unpack .../58-odbcinst_2.3.6-0.1_amd64.deb ...\n", + "Unpacking odbcinst (2.3.6-0.1) ...\n", + "Selecting previously unselected package odbcinst1debian2:amd64.\n", + "Preparing to unpack .../59-odbcinst1debian2_2.3.6-0.1_amd64.deb ...\n", + "Unpacking odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", + "Selecting previously unselected package libgdal20.\n", + "Preparing to unpack .../60-libgdal20_2.4.0+dfsg-1+b1_amd64.deb ...\n", + "Unpacking libgdal20 (2.4.0+dfsg-1+b1) ...\n", + "Selecting previously unselected package libxslt1.1:amd64.\n", + "Preparing to unpack .../61-libxslt1.1_1.1.32-2.2~deb10u1_amd64.deb ...\n", + "Unpacking libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", + "Selecting previously unselected package proj-bin.\n", + "Preparing to unpack .../62-proj-bin_5.2.0-1_amd64.deb ...\n", + "Unpacking proj-bin (5.2.0-1) ...\n", + "Selecting previously unselected package python3-affine.\n", + "Preparing to unpack .../63-python3-affine_2.2.2-1_all.deb ...\n", + "Unpacking python3-affine (2.2.2-1) ...\n", + "Selecting previously unselected package python3-attr.\n", + "Preparing to unpack .../64-python3-attr_18.2.0-1_all.deb ...\n", + "Unpacking python3-attr (18.2.0-1) ...\n", + "Selecting previously unselected package python3-soupsieve.\n", + "Preparing to unpack .../65-python3-soupsieve_1.8+dfsg-1_all.deb ...\n", + "Unpacking python3-soupsieve (1.8+dfsg-1) ...\n", + "Selecting previously unselected package python3-bs4.\n", + "Preparing to unpack .../66-python3-bs4_4.7.1-1_all.deb ...\n", + "Unpacking python3-bs4 (4.7.1-1) ...\n", + "Selecting previously unselected package python3-chardet.\n", + "Preparing to unpack .../67-python3-chardet_3.0.4-3_all.deb ...\n", + "Unpacking python3-chardet (3.0.4-3) ...\n", + "Selecting previously unselected package python3-colorama.\n", + "Preparing to unpack .../68-python3-colorama_0.3.7-1_all.deb ...\n", + "Unpacking python3-colorama (0.3.7-1) ...\n", + "Selecting previously unselected package python3-click.\n", + "Preparing to unpack .../69-python3-click_7.0-1_all.deb ...\n", + "Unpacking python3-click (7.0-1) ...\n", + "Selecting previously unselected package python3-click-plugins.\n", + "Preparing to unpack .../70-python3-click-plugins_1.0.4-1_all.deb ...\n", + "Unpacking python3-click-plugins (1.0.4-1) ...\n", + "Selecting previously unselected package python3-cligj.\n", + "Preparing to unpack .../71-python3-cligj_0.5.0-1_all.deb ...\n", + "Unpacking python3-cligj (0.5.0-1) ...\n", + "Selecting previously unselected package python3-webencodings.\n", + "Preparing to unpack .../72-python3-webencodings_0.5.1-1_all.deb ...\n", + "Unpacking python3-webencodings (0.5.1-1) ...\n", + "Selecting previously unselected package python3-html5lib.\n", + "Preparing to unpack .../73-python3-html5lib_1.0.1-1_all.deb ...\n", + "Unpacking python3-html5lib (1.0.1-1) ...\n", + "Selecting previously unselected package python3-lxml:amd64.\n", + "Preparing to unpack .../74-python3-lxml_4.3.2-1+deb10u3_amd64.deb ...\n", + "Unpacking python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", + "Selecting previously unselected package python3-numpy.\n", + "Preparing to unpack .../75-python3-numpy_1%3a1.16.2-1_amd64.deb ...\n", + "Unpacking python3-numpy (1:1.16.2-1) ...\n", + "Selecting previously unselected package python3-pyparsing.\n", + "Preparing to unpack .../76-python3-pyparsing_2.2.0+dfsg1-2_all.deb ...\n", + "Unpacking python3-pyparsing (2.2.0+dfsg1-2) ...\n", + "Selecting previously unselected package python3-snuggs.\n", + "Preparing to unpack .../77-python3-snuggs_1.4.3-1_all.deb ...\n", + "Unpacking python3-snuggs (1.4.3-1) ...\n", + "Selecting previously unselected package python3-rasterio.\n", + "Preparing to unpack .../78-python3-rasterio_1.0.21-1_amd64.deb ...\n", + "Unpacking python3-rasterio (1.0.21-1) ...\n", + "Setting up liblcms2-2:amd64 (2.9-3) ...\n", + "Setting up python3-attr (18.2.0-1) ...\n", + "Setting up mysql-common (5.8+1.0.5) ...\n", + "update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", + "Setting up proj-data (5.2.0-1) ...\n", + "Setting up python3-colorama (0.3.7-1) ...\n", + "Setting up libcharls2:amd64 (2.0.0+dfsg-1) ...\n", + "Setting up libminizip1:amd64 (1.1-8+b1) ...\n", + "Setting up libproj13:amd64 (5.2.0-1) ...\n", + "Setting up libpq5:amd64 (11.14-0+deb10u1) ...\n", + "Setting up proj-bin (5.2.0-1) ...\n", + "Setting up libqhull7:amd64 (2015.2-4) ...\n", + "Setting up python3-click (7.0-1) ...\n", + "Setting up libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", + "Setting up libjbig0:amd64 (2.1-3.1+b2) ...\n", + "Setting up python3-webencodings (0.5.1-1) ...\n", + "Setting up libgeos-3.7.1:amd64 (3.7.1-1) ...\n", + "Setting up libaec0:amd64 (1.0.2-1) ...\n", + "Setting up libicu63:amd64 (63.1-6+deb10u2) ...\n", + "Setting up gdal-data (2.4.0+dfsg-1) ...\n", + "Setting up poppler-data (0.4.9-2) ...\n", + "Setting up mariadb-common (1:10.3.31-0+deb10u1) ...\n", + "update-alternatives: using /etc/mysql/mariadb.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", + "Setting up python3-chardet (3.0.4-3) ...\n", + "Setting up libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", + "Setting up python3-pyparsing (2.2.0+dfsg1-2) ...\n", + "Setting up libnspr4:amd64 (2:4.20-1) ...\n", + "Setting up libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", + "Setting up libgeos-c1v5:amd64 (3.7.1-1) ...\n", + "Setting up python3-click-plugins (1.0.4-1) ...\n", + "Setting up libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", + "Setting up fonts-dejavu-core (2.37-1) ...\n", + "Setting up python3-html5lib (1.0.1-1) ...\n", + "Setting up libltdl7:amd64 (2.4.6-9) ...\n", + "Setting up libgfortran5:amd64 (8.3.0-6) ...\n", + "Setting up libhdf4-0-alt (4.2.13-4) ...\n", + "Setting up libgif7:amd64 (5.1.4-3) ...\n", + "Setting up liburiparser1:amd64 (0.9.1-1) ...\n", + "Setting up libfreexl1:amd64 (1.0.5-3) ...\n", + "Setting up libfyba0:amd64 (4.1.1-6) ...\n", + "Setting up libkmlbase1:amd64 (1.3.0-7) ...\n", + "Setting up libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", + "Setting up libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", + "Setting up python3-affine (2.2.2-1) ...\n", + "Setting up libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", + "Setting up python3-soupsieve (1.8+dfsg-1) ...\n", + "Setting up libsz2:amd64 (1.0.2-1) ...\n", + "Setting up libkmlxsd1:amd64 (1.3.0-7) ...\n", + "Setting up libkmldom1:amd64 (1.3.0-7) ...\n", + "Setting up libspatialite7:amd64 (4.3.0a-5+b2) ...\n", + "Setting up libogdi3.2 (3.2.1+ds-4) ...\n", + "Setting up libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", + "Setting up libkmlengine1:amd64 (1.3.0-7) ...\n", + "Setting up libkmlconvenience1:amd64 (1.3.0-7) ...\n", + "Setting up fontconfig-config (2.13.1-2) ...\n", + "Setting up python3-cligj (0.5.0-1) ...\n", + "Setting up libdap25:amd64 (3.20.3-1) ...\n", + "Setting up libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Setting up libdapserver7v5:amd64 (3.20.3-1) ...\n", + "Setting up python3-bs4 (4.7.1-1) ...\n", + "Setting up libgeotiff2:amd64 (1.4.3-1) ...\n", + "Setting up libblas3:amd64 (3.8.0-2) ...\n", + "update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode\n", + "Setting up libkmlregionator1:amd64 (1.3.0-7) ...\n", + "Setting up libhdf5-103:amd64 (1.10.4+repack-10) ...\n", + "Setting up libodbc1:amd64 (2.3.6-0.1) ...\n", + "Setting up libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", + "Setting up libdapclient6v5:amd64 (3.20.3-1) ...\n", + "Setting up libfontconfig1:amd64 (2.13.1-2) ...\n", + "Setting up liblapack3:amd64 (3.8.0-2) ...\n", + "update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3 to provide /usr/lib/x86_64-linux-gnu/liblapack.so.3 (liblapack.so.3-x86_64-linux-gnu) in auto mode\n", + "Setting up libarpack2:amd64 (3.7.0-2) ...\n", + "Setting up libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", + "Setting up libpoppler82:amd64 (0.71.0-5) ...\n", + "Setting up libnetcdf13:amd64 (1:4.6.2-1) ...\n", + "Setting up python3-numpy (1:1.16.2-1) ...\n", + "Setting up python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", + "Setting up libarmadillo9 (1:9.200.7+dfsg-1) ...\n", + "Setting up python3-snuggs (1.4.3-1) ...\n", + "Setting up odbcinst (2.3.6-0.1) ...\n", + "Setting up odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", + "Setting up libgdal20 (2.4.0+dfsg-1+b1) ...\n", + "Setting up python3-rasterio (1.0.21-1) ...\n", + "Processing triggers for libc-bin (2.28-10) ...\n", + "Processing triggers for man-db (2.8.5-2) ...\n" + ] + } + ], + "source": [ + "mkdir -v output/\n", + "sudo apt-get install python3-rasterio --yes" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "LC08_L1TP_025033_20211010_20211018_01_T1 1\n", + "LC08_L1TP_025033_20211010_20211018_01_T1 2\n", + "LC08_L1TP_025033_20211010_20211018_01_T1 3\n", + "ERROR 4: output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png: No such file or directory\n" + ] + } + ], + "source": [ + "/usr/bin/python3 combine.py" + ] + }, + { + "cell_type": "markdown", + "id": "4199910a-1056-455c-ae9d-669296441aed", + "metadata": {}, + "source": [ + "Note, the \"ERROR 4\" is expected." + ] + }, + { + "cell_type": "markdown", + "id": "a3a89af5-993e-45d6-9e7c-2ca0a61dff7a", + "metadata": {}, + "source": [ + "Verify the results were created." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 195M\n", + "-rw-r--r-- 1 learner learner 195M Dec 17 19:03 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20211010_20211018_01_T1.png\u001b[0m\n", + "-rw-r--r-- 1 learner learner 910 Dec 17 19:03 result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n" + ] + } + ], + "source": [ + "ls -lh output" + ] + }, + { + "cell_type": "markdown", + "id": "c0257075-537c-4510-bafd-72e9756db17b", + "metadata": {}, + "source": [ + "## Exporting the Results\n", + "\n", + "Now that we have the output data we will create a bucket to place the results. We will first create a bucket with a reasonable set of options.\n", + "\n", + "We fisrt store the bucket name in the `BUCKET` environment variable for future use. This time we will specify a realistic set of options for a private bucket used for computation.\n", + "\n", + "Options (run `gsutil mb --help` for more information):\n", + " * `-b on` specifies uniform bucket-level access.\n", + " * `-l $REGION` puts the data in a specific region for lower cost and lower latency.\n", + " * `--pap enforced` turns on public access prevention to help keep data private. \n", + " \n", + "The uniform bucket level access (Bucket Policy Only enabled: true) puts the data access permissions (ACL) on the entire bucket, not on each object in the bucket. This makes the permissions obvious and makes security much more predictable.\n", + " \n", + "As usual, we must set our environment. In this case we also set a `REGION` environment variable to indicate where in the world we want the data to be stored.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "f36cb8c5-f305-4cb2-a5cc-0c9fd8592fb4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "bucket: essentials-learner-2021-12-17 region: us-west2\n" + ] + } + ], + "source": [ + "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", + "REGION=\"us-west2\"\n", + "echo \"bucket: $BUCKET region: $REGION\"" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "c2ae2b74-5e93-4c55-8bd7-63337f7dcbb8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating gs://essentials-learner-2021-12-17/...\n" + ] + } + ], + "source": [ + "gsutil mb -b on -l $REGION --pap enforced \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "22cc1043-386b-4990-b3dc-62cbdd7ba133", + "metadata": {}, + "source": [ + "Now copy the output data to the bucket. The `-r` flag recursively copies the output directory and `-m` copies the files in parallel. Note the locations of the `-m` and `-r` switches as they apply globally and to the `cp` command respectively." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "681e6b1d-98bb-448a-a57e-f5674214effd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml [Content-Type=application/xml]...\n", + "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png [Content-Type=image/png]...\n", + "==> NOTE: You are uploading one or more large file(s), which would run \n", + "significantly faster if you enable parallel composite uploads. This\n", + "feature can be enabled by editing the\n", + "\"parallel_composite_upload_threshold\" value in your .boto\n", + "configuration file. However, note that if you do this large files will\n", + "be uploaded as `composite objects\n", + "`_,which\n", + "means that any user who downloads such objects will need to have a\n", + "compiled crcmod installed (see \"gsutil help crcmod\"). This is because\n", + "without a compiled crcmod, computing checksums on composite objects is\n", + "so slow that gsutil disables downloads of composite objects.\n", + "\n", + "\\ [2/2 files][194.0 MiB/194.0 MiB] 100% Done \n", + "Operation completed over 2 objects/194.0 MiB. \n" + ] + } + ], + "source": [ + "gsutil -m cp -r \"output\" \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "857861ca-9721-4a1e-a4ae-efffe2143b43", + "metadata": {}, + "source": [ + "Verify that the output was uploaded." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-learner-2021-12-17/output/\n" + ] + } + ], + "source": [ + "gsutil ls \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "b1ea18e9-5861-4479-9948-3303952dee8a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "194.05 MiB 2021-12-17T19:03:54Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", + " 910 B 2021-12-17T19:03:52Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", + "TOTAL: 2 objects, 203478494 bytes (194.05 MiB)\n" + ] + } + ], + "source": [ + "gsutil ls -lh \"gs://$BUCKET/output\"" + ] + }, + { + "cell_type": "markdown", + "id": "66a41f97-3834-4d09-bfe5-c3f224628842", + "metadata": {}, + "source": [ + "## Viewing the Results\n", + "\n", + "You now can view the results by using the Google Cloud Platform Web Console and navigating to \"Cloud Storage\", selecting the bucket, and then the result object you wish to view (select the `.png` file). You will need to click the \"Preview\" button given the large size of the image." + ] + }, + { + "cell_type": "markdown", + "id": "a1c11268-f389-405f-8d96-3c319a49b882", + "metadata": {}, + "source": [ + "## Sharing Results\n", + "\n", + "In order to share resources outside a project we must use the Identity Access Management service. This is a powerful tool to grant and restrict access to resources, and if not done correctly It can have serious consequences. **Incorrect permissions can lead to exposure of sensitive data, destruction of data, and authorized use of resources that can result in huge bills.** When in doubt, seek help.\n", + "\n", + "The question \"What access is really needed?\" is the **Principal of Least Privilege** and is a major cornerstone of security. We need to determine the lowest set of permissions or roles that is needed. In our case we wish to grant the \"Collaborator\" the \"Viewer\" access to the \"results bucket\". This will allow them to view, list, and download all objects in the bucket. This illustrates that for a **resource** a **member** (identity) is granted a **permission** (think of it in this order). Together this is called a **policy**. Google also uses \"Roles\" as a collection of predefined and managed permissions." + ] + }, + { + "cell_type": "markdown", + "id": "50990a67-f901-48fd-9295-25ef73245142", + "metadata": { + "tags": [] + }, + "source": [ + "We will now add Members to a Bucket using the Web Console. We will use the Web Console to interactively build the policy binding by doing the following:\n", + " * Navigation Menu -> Storage/Cloud Storage -> Browser -> Click on the Bucket Name (**Bucket Details**) -> Select the **Permissions** tab -> Click **Add** next to \"Permissions\" above the permissions list.\n", + " * In the \"New Principals\" box add the Identity for the collaborator (another individual) as directed by the instructor.\n", + " * Select the \"Storage Object Viewer\" by typing \"Storage Object Viewer\" in the filter and then selecting \"Storage Object Viewer\". Do not use any \"Legacy Storage\" roles.\n", + " * Click \"Save\" to save the policy.\n", + " ![iam-storage-object-viewer](img/iam-storage-object-viewer.png)\n", + " * Verify the policy is listed in the \"Permissions\" table on the \"Bucket Details\" page (you should now be on this page).\n", + "\n", + "*Advanced Note: Changes in permissions will show up on the **Home** -> **Activity** page.*" + ] + }, + { + "cell_type": "markdown", + "id": "6a191922-7d5b-4125-9812-181eb056f86c", + "metadata": { + "tags": [] + }, + "source": [ + "Collaborators should now be able to see the contents of the bucket by explicitly naming it as shown below.\n", + "```\n", + "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17\n", + "gs://essentials-learner-2021-12-17/output/\n", + "\n", + "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17/output\n", + "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", + "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "40d9ab41-8920-45f2-8218-550baac5b069", + "metadata": {}, + "source": [ + "## Cleanup\n", + "\n", + "We will now leave the resources running in order to learn more about monitoring costs and will clean up all the resources as the end of Lesson. **Don't forget to do this!**" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Bash", + "language": "bash", + "name": "bash" + }, + "language_info": { + "codemirror_mode": "shell", + "file_extension": ".sh", + "mimetype": "text/x-sh", + "name": "bash" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/content/Azure/09_cleaning_up_resources.ipynb b/content/Azure/08_cleaning_up_resources.ipynb similarity index 100% rename from content/Azure/09_cleaning_up_resources.ipynb rename to content/Azure/08_cleaning_up_resources.ipynb diff --git a/content/Azure/08_running_analysis.ipynb b/content/Azure/08_running_analysis.ipynb deleted file mode 100644 index b16481f..0000000 --- a/content/Azure/08_running_analysis.ipynb +++ /dev/null @@ -1,47 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "dc57021c", - "metadata": {}, - "source": [ - "# Putting It All Together\n", - "\n", - "\n", - "```{admonition} Overview\n", - ":class: tip\n", - "\n", - "**Teaching:**\n", - "\n", - "**Exercises:**\n", - "\n", - "**Questions:**\n", - "\n", - "**Objectives:**\n", - "\n", - "```" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.8" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/content/Azure/10_basic_networking_extra.ipynb b/content/Azure/10_basic_networking_extra.ipynb deleted file mode 100644 index bc13496..0000000 --- a/content/Azure/10_basic_networking_extra.ipynb +++ /dev/null @@ -1,469 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "7b5d7170", - "metadata": {}, - "source": [ - "\n", - "## Basic Azure networking\n", - "\n", - "Learner Questions\n", - "\n", - "* How do I deploy a network in Azure using the CLI?\n", - "\n", - "Learning Objectives\n", - "\n", - "* Deploy a virtual network and subnet\n", - "* Seccure virtual network using network security groups (nsg)\n", - "* Deploy ngix container\n", - "* Clean up (delete) resouces\n", - "\n", - "# Setup\n", - "\n", - "First we will create a virtual network with a subnet. We will then create a network security groups \n", - "(ngs) for the subnet and define nsg rules to let traffic in and out (ports opened). Finally we will login to (execute commands on) the containers. After this exercise we will clean up and delete our resources." - ] - }, - { - "cell_type": "markdown", - "id": "98b49d7c", - "metadata": {}, - "source": [ - "Setup the environment variables" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f9b25424", - "metadata": {}, - "outputs": [], - "source": [ - "export RGROUP=\"ps-rg\"\n", - "export REGION=\"eastus\"" - ] - }, - { - "cell_type": "markdown", - "id": "cac47244", - "metadata": {}, - "source": [ - "All Azure projects start with a resouce group (RG)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f603213e", - "metadata": {}, - "outputs": [], - "source": [ - "az group create --name $RGROUP --location $REGION" - ] - }, - { - "cell_type": "markdown", - "id": "d1315f96", - "metadata": {}, - "source": [ - "Set virtual network and subnet environment variables" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2e674700", - "metadata": {}, - "outputs": [], - "source": [ - "export VNET=\"VNET\" # Virtual network\n", - "export SUBNET=\"SN\" # Public facing subnet" - ] - }, - { - "cell_type": "markdown", - "id": "73c5f2d9", - "metadata": {}, - "source": [ - "Create a virtual network and subnet" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "892d901e", - "metadata": {}, - "outputs": [], - "source": [ - "az network vnet create \\\n", - " --resource-group $RGROUP \\\n", - " --name $VNET \\\n", - " --address-prefix 10.0.0.0/16 \\\n", - " --subnet-name $SUBNET \\\n", - " --subnet-prefix 10.0.1.0/24" - ] - }, - { - "cell_type": "markdown", - "id": "bf965b43", - "metadata": {}, - "source": [ - "Create a public IP" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ab70080e", - "metadata": {}, - "outputs": [], - "source": [ - "export PUBIP=\"PublicIPAddress\" # Public IP address" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "b2b7b731", - "metadata": {}, - "outputs": [], - "source": [ - "az network public-ip create --resource-group $RGROUP --name $PUBIP" - ] - }, - { - "cell_type": "markdown", - "id": "6b9aa16e", - "metadata": {}, - "source": [ - "Create network security groups" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "d08a7a6c", - "metadata": {}, - "outputs": [], - "source": [ - "export NSGROUP=\"Container-Security-Group\" # Network Security Group" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "41760e52", - "metadata": {}, - "outputs": [], - "source": [ - "az network nsg create --resource-group $RGROUP --name $NSGROUP" - ] - }, - { - "cell_type": "markdown", - "id": "5a28245a", - "metadata": {}, - "source": [ - " Create a rule for the NSG for TCP." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "216bfdf2", - "metadata": {}, - "outputs": [], - "source": [ - "az network nsg rule create \\\n", - " --resource-group $RGROUP \\\n", - " --nsg-name $NSGROUP \\\n", - " --name demo-nsg \\\n", - " --access Allow \\\n", - " --protocol Tcp \\\n", - " --direction Inbound \\\n", - " --priority 300 \\\n", - " --source-address-prefix 10.0.1.0/24 \\\n", - " --source-port-range \"*\" \\\n", - " --destination-address-prefix \"*\" \\\n", - " --destination-port-range 80 8080 443" - ] - }, - { - "cell_type": "markdown", - "id": "6bfe734d", - "metadata": {}, - "source": [ - "Add rule to block all traffic from blah, Note --priority 100 is lower than the previous" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f2db1617", - "metadata": {}, - "outputs": [], - "source": [ - "az network nsg rule create \\\n", - " --resource-group $RGROUP \\\n", - " --nsg-name $NSGROUP \\\n", - " --name denyAll \\\n", - " --access Deny \\\n", - " --protocol Tcp \\\n", - " --direction Inbound \\\n", - " --priority 100 \\\n", - " --source-address-prefix \"*\" \\\n", - " --source-port-range \"*\" \\\n", - " --destination-address-prefix \"*\" \\\n", - " --destination-port-range \"*\"" - ] - }, - { - "cell_type": "markdown", - "id": "3da3e070", - "metadata": {}, - "source": [ - "Associate the NSG to the subnet." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "dc0c1bf3", - "metadata": {}, - "outputs": [], - "source": [ - "az network vnet subnet update \\\n", - " --vnet-name $VNET \\\n", - " --name $SUBNET \\\n", - " --resource-group $RGROUP \\\n", - " --network-security-group $NSGROUP" - ] - }, - { - "cell_type": "markdown", - "id": "8acdef78", - "metadata": {}, - "source": [ - "List NSG" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "0bb9fde5", - "metadata": {}, - "outputs": [], - "source": [ - "az network nsg list" - ] - }, - { - "cell_type": "markdown", - "id": "7a6b4e51", - "metadata": {}, - "source": [ - "**Deploy container**" - ] - }, - { - "cell_type": "markdown", - "id": "141b11b3", - "metadata": {}, - "source": [ - "Create container" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "822cbc51", - "metadata": {}, - "outputs": [], - "source": [ - "export CONT=\"demo-container\" # Public perfSONAR container" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1fd31b96", - "metadata": {}, - "outputs": [], - "source": [ - "az container create \\\n", - " --resource-group $RGROUP \\\n", - " --name $CONT \\\n", - " --image nginx \\\n", - " --restart-policy never \\\n", - " --vnet $VNET \\\n", - " --subnet $SUBNET" - ] - }, - { - "cell_type": "markdown", - "id": "89912f26", - "metadata": {}, - "source": [ - "Get front container IP address" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ec228f00", - "metadata": {}, - "outputs": [], - "source": [ - "az container show --resource-group $RGROUP --name $CONT --query ipAddress.ip --output tsv" - ] - }, - { - "cell_type": "markdown", - "id": "49e22332", - "metadata": {}, - "source": [ - "List containers" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "aa068645", - "metadata": {}, - "outputs": [], - "source": [ - "az container list --resource-group $RGROUP --output table" - ] - }, - { - "cell_type": "markdown", - "id": "022721a6", - "metadata": {}, - "source": [ - "Get logs from backend perfSONAR container" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "42bb79c8", - "metadata": {}, - "outputs": [], - "source": [ - "az container logs --resource-group $RGROUP --name $CONT" - ] - }, - { - "cell_type": "markdown", - "id": "bffb41e3", - "metadata": {}, - "source": [ - "Stream logs from backend perfSONAR container" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "79e78cdd", - "metadata": {}, - "outputs": [], - "source": [ - "az container attach --resource-group $RGROUP --name $SCONT" - ] - }, - { - "cell_type": "markdown", - "id": "a69a0109", - "metadata": {}, - "source": [ - "Execute command on contaioner" - ] - }, - { - "cell_type": "markdown", - "id": "c7746c24", - "metadata": {}, - "source": [ - "\"Login\" into container with bash shell" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "b2c66780", - "metadata": {}, - "outputs": [], - "source": [ - "az container exec --resource-group $RGROUP --name $CONT --exec-command \"/bin/bash\"" - ] - }, - { - "cell_type": "markdown", - "id": "f7328eb2", - "metadata": {}, - "source": [ - "Now execute \"ls\" command to see what is there." - ] - }, - { - "cell_type": "markdown", - "id": "8bbdad92", - "metadata": {}, - "source": [ - "Delete virtual network" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a5b45a8f", - "metadata": {}, - "outputs": [], - "source": [ - "az network vnet delete --resource-group $RGROUP --name $VNET" - ] - }, - { - "cell_type": "markdown", - "id": "96db8a9f", - "metadata": {}, - "source": [ - "Clean up: delete container & rg" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "4a0df1a5", - "metadata": {}, - "outputs": [], - "source": [ - "az container delete --resource-group $RGROUP --name -y\n", - "\n", - "az group delete --name $RGROUP" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.8" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/content/Azure/file b/content/Azure/file deleted file mode 100644 index 7e91e7c..0000000 --- a/content/Azure/file +++ /dev/null @@ -1 +0,0 @@ -this is a filee diff --git a/content/Azure/04_intro_to_compute_cli.ipynb b/content/Azure/intro_to_compute_cli.ipynb similarity index 100% rename from content/Azure/04_intro_to_compute_cli.ipynb rename to content/Azure/intro_to_compute_cli.ipynb diff --git a/content/_toc.yml b/content/_toc.yml index 3f1f58b..09aa2cf 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -25,15 +25,13 @@ parts: - file: Azure/intro_to_Azure_Essentials sections: - file: Azure/01_intro_to_cloud_console - - file: Azure/02_intro_to_cli - - file: Azure/03_intro_to_compute - - file: Azure/04_intro_to_compute_cli - - file: Azure/05_intro_to_cloud_storage - - file: Azure/06_intro_to_cloud_storage_cli + - file: Azure/02_intro_to_compute + - file: Azure/03_intro_to_cloud_storage + - file: Azure/04_intro_to_cli + - file: Azure/05_intro_to_cloud_storage_cli + - file: Azure/06_running_analysis - file: Azure/07_monitoring_costs - - file: Azure/08_running_analysis - - file: Azure/09_cleaning_up_resources - - file: Azure/10_basic_networking_extra + - file: Azure/08_cleaning_up_resources - file: GCP/intro_to_GCP_Essentials sections: