diff --git a/content/Azure/01_intro_to_cloud_console.ipynb b/content/Azure/01_intro_to_cloud_console.ipynb index efa722f..3cabc97 100644 --- a/content/Azure/01_intro_to_cloud_console.ipynb +++ b/content/Azure/01_intro_to_cloud_console.ipynb @@ -2,66 +2,87 @@ "cells": [ { "cell_type": "markdown", - "id": "976355b6", + "id": "b6f03192", "metadata": {}, "source": [ - "# Introduction to the Azure Console\n", - "\n", + "# Introduction to the Azure Cloud Console\n", "\n", "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 15 mins\n", + "**Teaching:**\n", "\n", - "**Exercises:** 5 mins\n", + "**Exercises:**\n", "\n", "**Questions:**\n", - "* How do I navigate the web console?\n", + "* How do I find things in the web console?\n", + "* How do I access the cloud through the command line?\n", "\n", "**Objectives:**\n", - "* Log in to the Azure web console\n", - "* Navigate the Azure web console\n", - "* Recognize important information about your cloud environment\n", + "* Login to the Azure web console\n", + "* Find and verify important information about your cloud environment.\n", + "* Find and open the Cloud Shell\n", "\n", "```" ] }, { "cell_type": "markdown", - "id": "61be7c43-99c1-4033-8252-6955b942bee4", + "id": "20e902b2", "metadata": {}, "source": [ "## Setup\n", "\n", - "The console is the point of entry where you will start your Azure journey. If you are participating in the CLASS Essentials Workshop, you will have received a csv file via email. This csv file contains your **username**, **temporary password** and **console login link**. \n", - "\n", - "```{admonition} Be Aware\n", - ":class: danger\n", - "If you are doing self-paced learning, your login information may be different!\n", - "```" + "The console is the point of entry where you will start your Azure journey. If you are participating in the CLASS Essentials Workshop, you will have received a csv file via email. This csv file contains your **username**, **temporary password** and **console login link**. \n" ] }, { "cell_type": "markdown", - "id": "f57d9cdf-8ed5-4cab-87f7-4219cb33bf2b", + "id": "5fab77a5", "metadata": {}, "source": [ - "## Logging in to the console\n", - "Paste your console login link into your web browser. You will see the Azure site requesting your username and password. Fill in your username and password that was assigned to you in the csv file. \n", + "#### Accounts (Authentication and Identity) & login\n", + "\n", + "The (Azure) **Account** is the identity used to access the Azure web console and is in the form of an email address and is associated with an individual. This is typically the account you use to access Azure services associated with your institution and not your personal account. We will now login to the Azure web console and verify our account.\n", "\n", - "You are now logged on. Welcome to the first step of your cloud journey! \n", + " * Login to the web console with your (Azure) account by going to [https://portal.azure.com//](https://portal.azure.com//)\n", + " * Fill in your username and password (assigned to you in the an email from CLASS or perhap from your institution. \n", + " * Find your user/account under the \"Avatar\" in the top right (see number 4 below).\n", "\n", - "The below is what your screen should look like when you log in to the Azure console. Next, let's take a closer look at some key concepts and components of the Azure console. \n", + "You are now logged on. Welcome to the first step of your cloud journey!" + ] + }, + { + "cell_type": "markdown", + "id": "43174583", + "metadata": {}, + "source": [ + "### Exercise\n", + "\n", + "Take a few moments to ask yourself the following questions:\n", + " * What account are you currently using in the Google Cloud web console?\n", + " * Are you logged into any other accounts?\n", + " * How do you know which account is being used?\n", + " \n", + " If you are not currently using the correct account, switch to it." + ] + }, + { + "cell_type": "markdown", + "id": "9cd64226", + "metadata": {}, + "source": [ + "The top of your screen should look very similar to the following when you log in to the Azure console. Next, let's take a closer look at some key concepts and components of the Azure console. \n", "\n", "![cloud-shell](images/azure_intro_console.jpg)" ] }, { "cell_type": "markdown", - "id": "27fc9127", + "id": "b1b0faae", "metadata": {}, "source": [ - "**1. Hamburger**\n", + "**1. Navigation Menu (often called the hamburger)**\n", ": If you click on the hamburger (three horizontal lines) you will get a dropdown menu and see a list of services that are offered by Azure. Currently, Azure has over N services ranging from compute to machine learning to networking to data storage. The list grows frequently!\n", "\n", "**2. Search bar**\n", @@ -70,36 +91,67 @@ "**3. Cloud Shell**\n", ": The cloud shell button invokes an interactive shell environment for either bash or PowerShell. The first time you invoke cloud shell you will also creare persistent storage specifically for cloud shell. The cloud shell comes with linux tools like bash, sh, & dig as well as the Azure CLI. Cloud shell also has editors like VIM and emacs, and tools like git, make, and configuration tools like ansible & terraform already installed. Language support for python, go, java, & PowerShell are also preinstalled. Azure Cloud Shell runs on Ubuntu 16.04 LTS with persistent storage.\n", "\n", - "**4. IAM Username and Organization**\n", + "**4. Directories & Subscriptions**\n", + ": This button will show the current Subsciption and Active Directory (cover later) associated with your account. The Subscription is your billing account. Using this dropdown to verify that are using the right subscription.\n", + "\n", + "**5. IAM Username and Organization**\n", ": The menu bar on the console also lists your username and the current organization that you are logged into. IAM is short for Identity and Access Management. Identity is a mechanism to authenticate users (e.g. password) and Access is a mechanism for authorizing what kinds of services users have access to. In the creation of an IAM user, the administrator (overlord) of the account assigned a particular permission scope (in AWS this is known as roles) to control access to a particular set of services and also assigned a password to authenticate user login. IAM is a key component in managing security on the cloud.\n" ] }, { "cell_type": "markdown", - "id": "e7ad8ac9", + "id": "6e3b3cbd", + "metadata": {}, + "source": [ + "## Services\n", + "\n", + "The web console is used to control and observe the cloud from the browser. It should only be used for simple and one-time tasks, exploring new services, accessing documentation, or for monitoring and debugging resources in the cloud. Programmatic control through the console, programming languages (for example Python), and other automation tools should be used for day to day activities to make the most out of the cloud and to help with the reproducibility of research and teaching.\n", + "\n", + "The Navigation Menu (number 1 above) is used to navigate to the various products, which are also sometimes called services.\n" + ] + }, + { + "cell_type": "markdown", + "id": "e94b53e2", "metadata": {}, "source": [ - "```{admonition} Exercise\n", - ":class: attention\n", + "## Open a Cloud Shell\n", + "\n", + "The Cloud Shell is an easy way to access Comand Line Interface (CLI) commands in a web browser. Under the hood Cloud Shell is Debian running in a container with 5GB of minimal-charge persistent storage and a lot of other useful tools (git, docker, jq, Docker, etc.) installed by default.\n", "\n", - "* What is your IAM username? \n", - "* Why is IAM important?\n", - "* What are the three ways you can find an Azure service on the console?\n", - "````" + "To use the Cloud Shell:\n", + " * Open up a Cloud Shell by clicking on the **Cloud Shell** icon in the right side of the top blue bar (number 3 above).\n", + " * Once the Cloud Shell is open, you should see \"Welcome to Azure Cloud Shell\".\n", + " * Click the middle box icon on the right side of the Cloud Shell bar to maximize the screen. You can find the correct icon by hovering over the icons to see their names.\n", + " * Note: You can open the Cloud Shell in a new tab by clicking on the **Open new session** icon (looks like a box with a \"+\") in the middle of the Cloud Shell bar.\n" + ] + }, + { + "cell_type": "markdown", + "id": "c0706cc3", + "metadata": {}, + "source": [ + "Now let's move on the to next lession" ] } ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/02_intro_to_compute.ipynb b/content/Azure/02_intro_to_compute.ipynb index dca45b7..58db0c5 100644 --- a/content/Azure/02_intro_to_compute.ipynb +++ b/content/Azure/02_intro_to_compute.ipynb @@ -148,15 +148,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/05_intro_to_cloud_storage.ipynb b/content/Azure/03_intro_to_cloud_storage.ipynb similarity index 95% rename from content/Azure/05_intro_to_cloud_storage.ipynb rename to content/Azure/03_intro_to_cloud_storage.ipynb index eaa38c6..e270130 100644 --- a/content/Azure/05_intro_to_cloud_storage.ipynb +++ b/content/Azure/03_intro_to_cloud_storage.ipynb @@ -182,15 +182,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/04_intro_to_cli.ipynb b/content/Azure/04_intro_to_cli.ipynb index 46c1fac..4ee4e92 100644 --- a/content/Azure/04_intro_to_cli.ipynb +++ b/content/Azure/04_intro_to_cli.ipynb @@ -2,10 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "bfb78c2d", + "id": "d448d2c1", "metadata": {}, "source": [ - "# Navigating the Command Line Interface (CLI)\n", + "# Introduction to the Azure CLI\n", "\n", "Learner Questions\n", " * How do I use the Cloud CLI?\n", @@ -448,15 +448,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/05_intro_to_cloud_storage_cli.ipynb b/content/Azure/05_intro_to_cloud_storage_cli.ipynb new file mode 100644 index 0000000..5451fe8 --- /dev/null +++ b/content/Azure/05_intro_to_cloud_storage_cli.ipynb @@ -0,0 +1,253 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Managing Cloud Storage from the Command Line\n", + "\n", + "**Overview**\n", + "\n", + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + "\n", + " * How do I store data in the cloud?\n", + "\n", + "**Objectives:**\n", + "\n", + " * Create a Storage Account \n", + " \n", + " * Create a bucket\n", + "\n", + " * Learn to copy data in/out of a bucket using the command line.\n", + "\n", + " * Clean up resources" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Open the cloud shell (See: intro_to_cli)\n", + "\n", + "Delpoy blob storage in Azure" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Define some environment variables" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "RG=\"demo_rg\"\n", + "LOCATION=eastus\n", + "BLOB_CON=\"demo_blob_cont\"\n", + "STGE_ACCT=\"demo_stor_ac\"\n", + "CTR=\"demo_stge_ctr\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create resource group" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az group create --name $RG --location $LOCATION" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create storage account\n", + "\n", + "All storage resources need to belong to a storage account." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage account create \\\n", + " --name $STGE_ACCT \\\n", + " --resource-group $RG \\\n", + " --location $LOCATION \\\n", + " --sku Standard_ZRS \\\n", + " --encryption-services blob\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create storage container\n", + "\n", + "The following example uses your Azure AD account to authorize the operation to create the container. Before you create the container, assign the Storage Blob Data Contributor role to yourself. Even if you are the account owner, you need explicit permissions to perform data operations against the storage account." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az ad signed-in-user show --query objectId -o tsv | az role assignment create \\\n", + " --role \"Storage Blob Data Contributor\" \\\n", + " --assignee @- \\\n", + " --scope \"/subscriptions//resourceGroups/$RG/providers/Microsoft.Storage/storageAccounts/$STGE_ACCT\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now create the storage Container " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage container create \\\n", + " --account-name $STGE_ACCT \\\n", + " --name $CTR \\\n", + " --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we will upload a file from your local machine" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create a file to upload" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "echo \"Demo text\" > demo_file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Copy demo_file to your blob." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage blob upload \\\n", + " --account-name $STGE_ACCT \\\n", + " --container-name $CTR \\\n", + " --name demo_file \\\n", + " --file demo_file \\\n", + " --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Download the file we just uploaded." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage blob download \\\n", + " --account-name $STGE_ACCT \\\n", + " --container-name $STGE_ACCT \\\n", + " --name demo_file \\\n", + " --file ~/destination/path/for/file \\\n", + " --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Clean up" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "By deleting the Resource Group, you will delete all resouce in that group.\n", + "\n", + "The --no-wait parameter keeps the CLI from blocking while the deletion takes place." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az group delete --name $RG --no-wait" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/content/Azure/06_running_analysis.ipynb b/content/Azure/06_running_analysis.ipynb index a0ec478..052ffe8 100644 --- a/content/Azure/06_running_analysis.ipynb +++ b/content/Azure/06_running_analysis.ipynb @@ -7,7 +7,6 @@ "source": [ "# Putting It All Together\n", "\n", - "\n", "```{admonition} Overview\n", ":class: tip\n", "\n", @@ -16,11 +15,1459 @@ "**Exercises:**\n", "\n", "**Questions:**\n", + "* Can you show me an example?\n", "\n", "**Objectives:**\n", + "* Create a simple workflow using a cloud VM and cloud object storage.\n", + "* Understand the basics of Identity and Access Management (IAM)\n", + "* Add collaborators to a Bucket with appropriate permissions.\n", + "\n", + "\n", + "\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "6291edee-c2df-4941-9b8e-de42649640f9", + "metadata": {}, + "source": [ + "## Create a VM\n", + "\n", + "Since we only create resources as we need them in the cloud. As an exercise you will now create a VM for our analysis. In this case will give the VM **Full** access to **Storage**. \n", + "\n", + "Using the console navigate to the \"Compute Engine\" service and create a new VM with the following properties.\n", + " * Call the VM \"essentials\"\n", + " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", + " * Feel free to select a larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + " \n", + "When you are done connect to the machine as described below." + ] + }, + { + "cell_type": "markdown", + "id": "c87e200c-1d18-44f5-b4e6-28b697b04b3c", + "metadata": {}, + "source": [ + "## Connect to the VM\n", + "\n", + "First login to the instance from the Cloud Shell by running the following command:\n", + "```\n", + "gcloud compute ssh essentials\n", + "```\n", + "If prompted for a zone select `n` to find it automatically. You can see an example session below." + ] + }, + { + "cell_type": "markdown", + "id": "44f32772-0abb-4fa7-a013-202fc646df5e", + "metadata": {}, + "source": [ + "```\n", + "learner@cloudshell:~ (just-armor-301114)$ gcloud compute ssh essentials\n", + "Did you mean zone [us-central1-b] for instance: [essentials] (Y/n)? n\n", + "\n", + "No zone specified. Using zone [us-west2-c] for instance: [essentials].\n", + "Linux essentials 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64\n", + "\n", + "The programs included with the Debian GNU/Linux system are free software;\n", + "the exact distribution terms for each program are described in the\n", + "individual files in /usr/share/doc/*/copyright.\n", + "\n", + "Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent\n", + "permitted by applicable law.\n", + "Last login: Tue Nov 9 20:12:49 2021 from 34.133.99.196\n", + "learner@essentials:~$\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "0ab6ad28-24bf-4c0f-b4dd-3be2ec25aaa7", + "metadata": {}, + "source": [ + "## Secure the VM\n", + "\n", + "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs security patches." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "0f2df994-202e-4e3e-96c2-1536d8adb036", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\n", + "Hit:2 http://deb.debian.org/debian buster InRelease\n", + "Hit:3 http://deb.debian.org/debian buster-updates InRelease\n", + "Hit:4 http://deb.debian.org/debian buster-backports InRelease\n", + "Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease\n", + "Hit:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease\n", + "Get:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease [5526 B]\n", + "Fetched 70.9 kB in 1s (91.3 kB/s) \u001b[0m\u001b[33m\n", + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "2 packages can be upgraded. Run 'apt list --upgradable' to see them.\n" + ] + } + ], + "source": [ + "sudo apt update\n", + "sudo unattended-upgrades" + ] + }, + { + "cell_type": "markdown", + "id": "b7c4db9e-f098-41bc-80d2-b524444eec7f", + "metadata": {}, + "source": [ + "## Get Example Code\n", + "\n", + "We will now install `git` and use it to download the example code into your home directory. For those of you who are unfamiliar with git, it is a way to collaboratively manage files and we will only be using it to download the example that we will be using. " + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "05fadd30-4f35-4a93-b713-cc132131b948", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "git is already the newest version (1:2.20.1-2+deb10u3).\n", + "0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.\n" + ] + } + ], + "source": [ + "sudo apt-get install --yes git" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", + "metadata": {}, + "outputs": [], + "source": [ + "cd ~" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cloning into 'CLASS-Examples'...\n", + "remote: Enumerating objects: 66, done.\u001b[K\n", + "remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66\u001b[K\n", + "Unpacking objects: 100% (66/66), done.\n" + ] + } + ], + "source": [ + "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" + ] + }, + { + "cell_type": "markdown", + "id": "3d2eb9fc-bec9-449d-a7ab-f3127f59d512", + "metadata": {}, + "source": [ + "We now change the current directory to the `landsat` directory in the `CLASS-Examples` directory that was just created by the previous git command." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", + "metadata": {}, + "outputs": [], + "source": [ + "cd ~/CLASS-Examples/landsat/" + ] + }, + { + "cell_type": "markdown", + "id": "6ea81d23-e1b0-4cac-a3ac-fed7a9723eca", + "metadata": {}, + "source": [ + "Your prompt should now change showing the current directory as follows.\n", + "```\n", + "learner@essentials:~/CLASS-Examples/landsat$\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 28\n", + "-rw-r--r-- 1 learner learner 960 Dec 17 19:00 ReadMe.md\n", + "-rw-r--r-- 1 learner learner 72 Dec 17 19:00 clean.sh\n", + "-rw-r--r-- 1 learner learner 613 Dec 17 19:00 combine.py\n", + "-rw-r--r-- 1 learner learner 280 Dec 17 19:00 download.sh\n", + "-rw-r--r-- 1 learner learner 314 Dec 17 19:00 get-index.sh\n", + "-rw-r--r-- 1 learner learner 76 Dec 17 19:00 search.json\n", + "-rw-r--r-- 1 learner learner 783 Dec 17 19:00 search.py\n" + ] + } + ], + "source": [ + "ls -l" + ] + }, + { + "cell_type": "markdown", + "id": "2b8b3144-7453-4350-a1cf-7fa74af2bcbf", + "metadata": {}, + "source": [ + "## Access the bucket\n", + "\n", + "First test that our tools are working and that we can access the public bucket that we will be using." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://gcp-public-data-landsat/index.csv.gz\n", + "gs://gcp-public-data-landsat/LC08/\n", + "gs://gcp-public-data-landsat/LE07/\n", + "gs://gcp-public-data-landsat/LM01/\n", + "gs://gcp-public-data-landsat/LM02/\n", + "gs://gcp-public-data-landsat/LM03/\n", + "gs://gcp-public-data-landsat/LM04/\n", + "gs://gcp-public-data-landsat/LM05/\n", + "gs://gcp-public-data-landsat/LO08/\n", + "gs://gcp-public-data-landsat/LT04/\n", + "gs://gcp-public-data-landsat/LT05/\n", + "gs://gcp-public-data-landsat/LT08/\n" + ] + } + ], + "source": [ + "gsutil ls gs://gcp-public-data-landsat/" + ] + }, + { + "cell_type": "markdown", + "id": "89acffba-cbce-436c-98dd-05467b6675a6", + "metadata": {}, + "source": [ + "The index file is a list of all the files in the bucket and we can use it to search and filter files.\n", + "\n", + "We will get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mkdir: created directory 'data'\n" + ] + } + ], + "source": [ + "mkdir -v data" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying gs://gcp-public-data-landsat/index.csv.gz...\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "\\ [1 files][761.0 MiB/761.0 MiB] 42.0 MiB/s \n", + "Operation completed over 1 objects/761.0 MiB. \n" + ] + } + ], + "source": [ + "gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/" + ] + }, + { + "cell_type": "markdown", + "id": "76b905b4-1c2a-4960-a14f-974b77f671cd", + "metadata": {}, + "source": [ + "We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", + "metadata": {}, + "outputs": [], + "source": [ + "gzip -df data/index.csv.gz" + ] + }, + { + "cell_type": "markdown", + "id": "75faa74a-b458-472b-9018-034c89e75315", + "metadata": {}, + "source": [ + "Again, check what happened." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "b005876c-f9af-43d6-80c6-f04295413b9b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 2.6G\n", + "-rw-r--r-- 1 learner learner 2.6G Dec 17 19:01 index.csv\n" + ] + } + ], + "source": [ + "ls -lh data" + ] + }, + { + "cell_type": "markdown", + "id": "fcde8334-f58d-4c3d-995a-2491be0f95ea", + "metadata": {}, + "source": [ + "We will now explore the data. The `head` command simply displays the first few lines of the file." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL\n", + "LM51940171984262KIS00,LM05_L1GS_194017_19840918_20210913_01_T2,LANDSAT_5,MSS,1984-09-18,01,T2,1984-09-18T09:36:27.0970000Z,L1GS,194,17,51.0,62.58023,60.38359,14.45307,19.13901,34758212,gs://gcp-public-data-landsat/LM05/01/194/017/LM05_L1GS_194017_19840918_20210913_01_T2\n", + "LM10420331972311AAA04,LM01_L1TP_042033_19721106_20180429_01_T2,LANDSAT_1,MSS,1972-11-06,01,T2,1972-11-06T17:49:00.5000000Z,L1TP,42,33,3.0,39.68891,37.707,-115.59896,-112.90344,26932395,gs://gcp-public-data-landsat/LM01/01/042/033/LM01_L1TP_042033_19721106_20180429_01_T2\n", + "LT50150151992246PAC00,LT05_L1TP_015015_19920902_20170121_01_T1,LANDSAT_5,TM,1992-09-02,01,T1,1992-09-02T15:01:34.1390130Z,L1TP,15,15,90.0,65.32118,63.11726,-67.3438,-61.89671,118785510,gs://gcp-public-data-landsat/LT05/01/015/015/LT05_L1TP_015015_19920902_20170121_01_T1\n" + ] + } + ], + "source": [ + "head --lines=4 data/index.csv" + ] + }, + { + "cell_type": "markdown", + "id": "532e6da3-302a-4e8a-8570-752995f30f1d", + "metadata": {}, + "source": [ + "## Search for Data\n", + "\n", + "We can see the data is well formed and what we expect. We will now use this data to download data related to a specific point and for the Landsat 8. The following script does a simple filter." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "#!/usr/bin/python3\n", + "import json\n", + "import csv\n", + "import sys\n", + "\n", + "# Example: Burr Oak Tree\n", + "# 38.899313,-92.464562 (Lat north+, Long west-) ; Landsat Path 025, Row 033\n", + "config=json.load(open(\"search.json\"))\n", + "lat,lon=config['lat'],config['lon']\n", + "landsat=config['landsat']\n", + "\n", + "reader=csv.reader(sys.stdin)\n", + "header=next(reader) # skip header\n", + "for l in reader:\n", + " SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL=l\n", + " west,east=float(WEST_LON),float(EAST_LON)\n", + " north,south=float(NORTH_LAT),float(SOUTH_LAT)\n", + " if SPACECRAFT_ID==landsat and north >= lat and south <= lat and west <= lon and east >= lon:\n", + " print(BASE_URL) # output BASE_URL\n" + ] + } + ], + "source": [ + "cat search.py" + ] + }, + { + "cell_type": "markdown", + "id": "4aa3de47-3dd4-4a0f-9f07-f2f004de7054", + "metadata": {}, + "source": [ + "We can see that the actual search data comes from the file `search.json`. The program reads the data from the standard input and iterates over all rows in the CSV file. It filters the results for which the image contains the pint and prints out the bucket URL for them. We are interested in all products that contain the Burr Oak Tree." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"lat\": 38.899313,\n", + " \"lon\": -92.464562,\n", + " \"landsat\": \"LANDSAT_8\"\n", + "}\n" + ] + } + ], + "source": [ + "cat search.json" + ] + }, + { + "cell_type": "markdown", + "id": "cbb27235-6bc4-4eb6-b668-5c30427a28b8", + "metadata": {}, + "source": [ + "Now lets test this on a subset of the data." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20150127_20170302_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20161028_20170219_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20191224_20200110_01_T1\n" + ] + } + ], + "source": [ + "head --lines=200000 data/index.csv | python3 search.py" + ] + }, + { + "cell_type": "markdown", + "id": "a76f24f8-3b2d-4c0d-880b-f2911b9d9b84", + "metadata": {}, + "source": [ + "## Download the Data\n", + "\n", + "Now that we have a list of folders we are interested, we will now download them with a simple script that takes bucket addresses (URL's) and downloads them with the `gsutil` program." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "3572c518-df83-4906-bfa6-a37bde2a5063", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "#!/bin/bash\n", + "\n", + "# Read space separated URL from STDIN and download \n", + "while read -r URL ; do\n", + " echo \"+++ $URL\"\n", + " # -m parallel\n", + " # -n no-clobber (do not re-download data)\n", + " # -r recursive (download all the data in the specified URL)\n", + " gsutil -m cp -n -r \"${URL}/\" data/\n", + "done\n" + ] + } + ], + "source": [ + "cat download.sh" + ] + }, + { + "cell_type": "markdown", + "id": "a02100bd-f8c5-42dd-975e-a9eb5369bc81", + "metadata": {}, + "source": [ + "Get the first 2 datasets" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B9.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B8.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B11.TIF...\n", + "==> NOTE: You are downloading one or more large file(s), which would\n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_MTL.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B5.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B7.TIF...\n", + "| [14/14 files][997.2 MiB/997.2 MiB] 100% Done 65.3 MiB/s ETA 00:00:00 \n", + "Operation completed over 14 objects/997.2 MiB. \n", + "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B8.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B9.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B7.TIF...\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", + "run significantly faster if you enabled sliced object downloads. This\n", + "feature is enabled by default but requires that compiled crcmod be\n", + "installed (see \"gsutil help crcmod\").\n", + "\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_MTL.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B5.TIF...\n", + "- [14/14 files][979.2 MiB/979.2 MiB] 100% Done 14.9 MiB/s ETA 00:00:00 \n", + "Operation completed over 14 objects/979.2 MiB. \n" + ] + } + ], + "source": [ + "head --lines=200000 data/index.csv | python3 search.py | head --lines=2 | bash download.sh" + ] + }, + { + "cell_type": "markdown", + "id": "14c23aab-a6e8-439d-8e55-44b7148f74d5", + "metadata": {}, + "source": [ + "Check that the data was downloaded" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 2665792\n", + "drwxr-xr-x 2 learner learner 4096 Dec 17 19:02 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20170913_20170928_01_T1\u001b[0m\n", + "drwxr-xr-x 2 learner learner 4096 Dec 17 19:01 \u001b[01;34mLC08_L1TP_025033_20211010_20211018_01_T1\u001b[0m\n", + "-rw-r--r-- 1 learner learner 2729754843 Dec 17 19:01 index.csv\n" + ] + } + ], + "source": [ + "ls -l data" + ] + }, + { + "cell_type": "markdown", + "id": "6073d5f6-68ef-41df-8044-73f221ce8780", + "metadata": {}, + "source": [ + "## Processing the Data\n", + "\n", + "We will now use simple script to combine the color bands and export it as a PNG" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "#!/usr/bin/python3\n", + "import os\n", + "import rasterio\n", + "\n", + "# Open the first directory in data, could walk entire tree\n", + "for dirname, dirs, files in os.walk('data'):\n", + " source = dirs[0]\n", + " break\n", + "\n", + "# Open band (B2/Blue) and copy metadata for result.\n", + "with rasterio.open(\"data/%s/%s_B2.TIF\" % (source, source)) as band2:\n", + " meta = band2.meta\n", + "\n", + "# Combine bands into PNG\n", + "meta.update(count = 3, driver='PNG')\n", + "with rasterio.open(\"output/result-%s.png\" % source, 'w+', **meta) as output:\n", + " for i in range(1, 4):\n", + " print(source, i)\n", + " output.write_band(i,rasterio.open(\"data/%s/%s_B%d.TIF\" % (source, source, i+3)).read(1))\n" + ] + } + ], + "source": [ + "cat combine.py" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Traceback (most recent call last):\n", + " File \"combine.py\", line 3, in \n", + " import rasterio\n", + "ModuleNotFoundError: No module named 'rasterio'\n" + ] + } + ], + "source": [ + "python3 combine.py\n", + "/bin/true # ignore this line used for jupyter" + ] + }, + { + "cell_type": "markdown", + "id": "52ada3ec-26cd-4e8b-94c2-8ed33e98be6e", + "metadata": {}, + "source": [ + "Oops, let's install the library (note: the output will be slightly different due to how this Lesson is built)\n", + "and while we are at it we also need to create the output directory." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mkdir: created directory 'output/'\n", + "Reading package lists... Done\n", + "Building dependency tree \n", + "Reading state information... Done\n", + "The following additional packages will be installed:\n", + " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", + " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", + " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", + " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", + " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", + " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", + " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", + " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", + " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", + " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", + " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", + " python3-chardet python3-click python3-click-plugins python3-cligj\n", + " python3-colorama python3-html5lib python3-lxml python3-numpy\n", + " python3-pyparsing python3-snuggs python3-soupsieve python3-webencodings\n", + "Suggested packages:\n", + " geotiff-bin gdal-bin libgeotiff-epsg libhdf4-doc libhdf4-alt-dev hdf4-tools\n", + " liblcms2-utils libmyodbc odbc-postgresql tdsodbc unixodbc-bin ogdi-bin\n", + " poppler-utils ghostscript fonts-japanese-mincho | fonts-ipafont-mincho\n", + " fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai\n", + " fonts-arphic-uming fonts-nanum python-attr-doc python3-genshi\n", + " python3-lxml-dbg python-lxml-doc gfortran python-numpy-doc python3-pytest\n", + " python3-numpy-dbg python-pyparsing-doc\n", + "The following NEW packages will be installed:\n", + " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", + " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", + " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", + " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", + " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", + " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", + " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", + " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", + " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", + " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", + " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", + " python3-chardet python3-click python3-click-plugins python3-cligj\n", + " python3-colorama python3-html5lib python3-lxml python3-numpy\n", + " python3-pyparsing python3-rasterio python3-snuggs python3-soupsieve\n", + " python3-webencodings\n", + "0 upgraded, 79 newly installed, 0 to remove and 2 not upgraded.\n", + "Need to get 46.8 MB of archives.\n", + "After this operation, 172 MB of additional disk space will be used.\n", + "Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", + "Get:2 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", + "Get:3 http://deb.debian.org/debian buster/main amd64 fonts-dejavu-core all 2.37-1 [1068 kB]\n", + "Get:4 http://deb.debian.org/debian buster/main amd64 fontconfig-config all 2.13.1-2 [280 kB]\n", + "Get:5 http://deb.debian.org/debian buster/main amd64 gdal-data all 2.4.0+dfsg-1 [744 kB]\n", + "Get:6 http://deb.debian.org/debian buster/main amd64 libaec0 amd64 1.0.2-1 [20.1 kB]\n", + "Get:7 http://deb.debian.org/debian buster/main amd64 libgfortran5 amd64 8.3.0-6 [581 kB]\n", + "Get:8 http://deb.debian.org/debian buster/main amd64 libblas3 amd64 3.8.0-2 [148 kB]\n", + "Get:9 http://deb.debian.org/debian buster/main amd64 liblapack3 amd64 3.8.0-2 [2110 kB]\n", + "Get:10 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", + "Get:11 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", + "Get:12 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", + "Get:13 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", + "Get:14 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", + "Get:15 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", + "Get:16 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", + "Get:17 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", + "Get:18 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", + "Get:19 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", + "Get:20 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", + "Get:21 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", + "Get:22 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", + "Get:23 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", + "Get:24 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", + "Get:25 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", + "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", + "Get:27 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", + "Get:28 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", + "Get:29 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", + "Get:30 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", + "Get:31 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", + "Get:32 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", + "Get:33 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", + "Get:34 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", + "Get:35 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", + "Get:36 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", + "Get:37 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", + "Get:38 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", + "Get:39 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", + "Get:40 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", + "Get:41 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", + "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlconvenience1 amd64 1.3.0-7 [50.1 kB]\n", + "Get:43 http://deb.debian.org/debian buster/main amd64 libkmlregionator1 amd64 1.3.0-7 [25.1 kB]\n", + "Get:44 http://deb.debian.org/debian buster/main amd64 libkmlxsd1 amd64 1.3.0-7 [34.6 kB]\n", + "Get:45 http://deb.debian.org/debian buster/main amd64 mysql-common all 5.8+1.0.5 [7324 B]\n", + "Get:46 http://deb.debian.org/debian buster/main amd64 mariadb-common all 1:10.3.31-0+deb10u1 [32.7 kB]\n", + "Get:47 http://deb.debian.org/debian buster/main amd64 libmariadb3 amd64 1:10.3.31-0+deb10u1 [173 kB]\n", + "Get:48 http://deb.debian.org/debian buster/main amd64 libnetcdf13 amd64 1:4.6.2-1 [403 kB]\n", + "Get:49 http://deb.debian.org/debian buster/main amd64 libltdl7 amd64 2.4.6-9 [390 kB]\n", + "Get:50 http://deb.debian.org/debian buster/main amd64 libodbc1 amd64 2.3.6-0.1 [223 kB]\n", + "Get:51 http://deb.debian.org/debian buster/main amd64 libogdi3.2 amd64 3.2.1+ds-4 [241 kB]\n", + "Get:52 http://deb.debian.org/debian buster/main amd64 libopenjp2-7 amd64 2.3.0-2+deb10u2 [158 kB]\n", + "Get:53 http://deb.debian.org/debian buster/main amd64 liblcms2-2 amd64 2.9-3 [145 kB]\n", + "Get:54 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]\n", + "Get:55 http://deb.debian.org/debian buster/main amd64 libpoppler82 amd64 0.71.0-5 [1507 kB]\n", + "Get:56 http://deb.debian.org/debian buster/main amd64 libqhull7 amd64 2015.2-4 [196 kB]\n", + "Get:57 http://deb.debian.org/debian buster/main amd64 libspatialite7 amd64 4.3.0a-5+b2 [1256 kB]\n", + "Get:58 http://deb.debian.org/debian buster/main amd64 libxerces-c3.2 amd64 3.2.2+debian-1+deb10u1 [886 kB]\n", + "Get:59 http://deb.debian.org/debian buster/main amd64 odbcinst amd64 2.3.6-0.1 [48.4 kB]\n", + "Get:60 http://deb.debian.org/debian buster/main amd64 odbcinst1debian2 amd64 2.3.6-0.1 [78.1 kB]\n", + "Get:61 http://deb.debian.org/debian buster/main amd64 libgdal20 amd64 2.4.0+dfsg-1+b1 [6171 kB]\n", + "Get:62 http://deb.debian.org/debian buster/main amd64 libxslt1.1 amd64 1.1.32-2.2~deb10u1 [237 kB]\n", + "Get:63 http://deb.debian.org/debian buster/main amd64 proj-bin amd64 5.2.0-1 [105 kB]\n", + "Get:64 http://deb.debian.org/debian buster/main amd64 python3-affine all 2.2.2-1 [14.5 kB]\n", + "Get:65 http://deb.debian.org/debian buster/main amd64 python3-attr all 18.2.0-1 [37.3 kB]\n", + "Get:66 http://deb.debian.org/debian buster/main amd64 python3-soupsieve all 1.8+dfsg-1 [27.6 kB]\n", + "Get:67 http://deb.debian.org/debian buster/main amd64 python3-bs4 all 4.7.1-1 [94.1 kB]\n", + "Get:68 http://deb.debian.org/debian buster/main amd64 python3-chardet all 3.0.4-3 [80.5 kB]\n", + "Get:69 http://deb.debian.org/debian buster/main amd64 python3-colorama all 0.3.7-1 [18.1 kB]\n", + "Get:70 http://deb.debian.org/debian buster/main amd64 python3-click all 7.0-1 [73.6 kB]\n", + "Get:71 http://deb.debian.org/debian buster/main amd64 python3-click-plugins all 1.0.4-1 [7540 B]\n", + "Get:72 http://deb.debian.org/debian buster/main amd64 python3-cligj all 0.5.0-1 [8212 B]\n", + "Get:73 http://deb.debian.org/debian buster/main amd64 python3-webencodings all 0.5.1-1 [10.9 kB]\n", + "Get:74 http://deb.debian.org/debian buster/main amd64 python3-html5lib all 1.0.1-1 [89.5 kB]\n", + "Get:75 http://deb.debian.org/debian buster/main amd64 python3-lxml amd64 4.3.2-1+deb10u3 [1162 kB]\n", + "Get:76 http://deb.debian.org/debian buster/main amd64 python3-numpy amd64 1:1.16.2-1 [2119 kB]\n", + "Get:77 http://deb.debian.org/debian buster/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [89.6 kB]\n", + "Get:78 http://deb.debian.org/debian buster/main amd64 python3-snuggs all 1.4.3-1 [7228 B]\n", + "Get:79 http://deb.debian.org/debian buster/main amd64 python3-rasterio amd64 1.0.21-1 [818 kB]\n", + "Fetched 46.8 MB in 1s (85.4 MB/s) \n", + "Extracting templates from packages: 100%\n", + "Preconfiguring packages ...\n", + "Selecting previously unselected package poppler-data.\n", + "(Reading database ... 57529 files and directories currently installed.)\n", + "Preparing to unpack .../00-poppler-data_0.4.9-2_all.deb ...\n", + "Unpacking poppler-data (0.4.9-2) ...\n", + "Selecting previously unselected package fonts-dejavu-core.\n", + "Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ...\n", + "Unpacking fonts-dejavu-core (2.37-1) ...\n", + "Selecting previously unselected package fontconfig-config.\n", + "Preparing to unpack .../02-fontconfig-config_2.13.1-2_all.deb ...\n", + "Unpacking fontconfig-config (2.13.1-2) ...\n", + "Selecting previously unselected package gdal-data.\n", + "Preparing to unpack .../03-gdal-data_2.4.0+dfsg-1_all.deb ...\n", + "Unpacking gdal-data (2.4.0+dfsg-1) ...\n", + "Selecting previously unselected package libaec0:amd64.\n", + "Preparing to unpack .../04-libaec0_1.0.2-1_amd64.deb ...\n", + "Unpacking libaec0:amd64 (1.0.2-1) ...\n", + "Selecting previously unselected package libgfortran5:amd64.\n", + "Preparing to unpack .../05-libgfortran5_8.3.0-6_amd64.deb ...\n", + "Unpacking libgfortran5:amd64 (8.3.0-6) ...\n", + "Selecting previously unselected package libblas3:amd64.\n", + "Preparing to unpack .../06-libblas3_3.8.0-2_amd64.deb ...\n", + "Unpacking libblas3:amd64 (3.8.0-2) ...\n", + "Selecting previously unselected package liblapack3:amd64.\n", + "Preparing to unpack .../07-liblapack3_3.8.0-2_amd64.deb ...\n", + "Unpacking liblapack3:amd64 (3.8.0-2) ...\n", + "Selecting previously unselected package libarpack2:amd64.\n", + "Preparing to unpack .../08-libarpack2_3.7.0-2_amd64.deb ...\n", + "Unpacking libarpack2:amd64 (3.7.0-2) ...\n", + "Selecting previously unselected package libsuperlu5:amd64.\n", + "Preparing to unpack .../09-libsuperlu5_5.2.1+dfsg1-4_amd64.deb ...\n", + "Unpacking libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", + "Selecting previously unselected package libarmadillo9.\n", + "Preparing to unpack .../10-libarmadillo9_1%3a9.200.7+dfsg-1_amd64.deb ...\n", + "Unpacking libarmadillo9 (1:9.200.7+dfsg-1) ...\n", + "Selecting previously unselected package libcharls2:amd64.\n", + "Preparing to unpack .../11-libcharls2_2.0.0+dfsg-1_amd64.deb ...\n", + "Unpacking libcharls2:amd64 (2.0.0+dfsg-1) ...\n", + "Selecting previously unselected package libicu63:amd64.\n", + "Preparing to unpack .../12-libicu63_63.1-6+deb10u2_amd64.deb ...\n", + "Unpacking libicu63:amd64 (63.1-6+deb10u2) ...\n", + "Selecting previously unselected package libxml2:amd64.\n", + "Preparing to unpack .../13-libxml2_2.9.4+dfsg1-7+deb10u2_amd64.deb ...\n", + "Unpacking libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", + "Selecting previously unselected package libdap25:amd64.\n", + "Preparing to unpack .../14-libdap25_3.20.3-1_amd64.deb ...\n", + "Unpacking libdap25:amd64 (3.20.3-1) ...\n", + "Selecting previously unselected package libdapclient6v5:amd64.\n", + "Preparing to unpack .../15-libdapclient6v5_3.20.3-1_amd64.deb ...\n", + "Unpacking libdapclient6v5:amd64 (3.20.3-1) ...\n", + "Selecting previously unselected package libdapserver7v5:amd64.\n", + "Preparing to unpack .../16-libdapserver7v5_3.20.3-1_amd64.deb ...\n", + "Unpacking libdapserver7v5:amd64 (3.20.3-1) ...\n", + "Selecting previously unselected package libepsilon1:amd64.\n", + "Preparing to unpack .../17-libepsilon1_0.9.2+dfsg-4_amd64.deb ...\n", + "Unpacking libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", + "Selecting previously unselected package libfontconfig1:amd64.\n", + "Preparing to unpack .../18-libfontconfig1_2.13.1-2_amd64.deb ...\n", + "Unpacking libfontconfig1:amd64 (2.13.1-2) ...\n", + "Selecting previously unselected package libfreexl1:amd64.\n", + "Preparing to unpack .../19-libfreexl1_1.0.5-3_amd64.deb ...\n", + "Unpacking libfreexl1:amd64 (1.0.5-3) ...\n", + "Selecting previously unselected package libfyba0:amd64.\n", + "Preparing to unpack .../20-libfyba0_4.1.1-6_amd64.deb ...\n", + "Unpacking libfyba0:amd64 (4.1.1-6) ...\n", + "Selecting previously unselected package libgeos-3.7.1:amd64.\n", + "Preparing to unpack .../21-libgeos-3.7.1_3.7.1-1_amd64.deb ...\n", + "Unpacking libgeos-3.7.1:amd64 (3.7.1-1) ...\n", + "Selecting previously unselected package libgeos-c1v5:amd64.\n", + "Preparing to unpack .../22-libgeos-c1v5_3.7.1-1_amd64.deb ...\n", + "Unpacking libgeos-c1v5:amd64 (3.7.1-1) ...\n", + "Selecting previously unselected package proj-data.\n", + "Preparing to unpack .../23-proj-data_5.2.0-1_all.deb ...\n", + "Unpacking proj-data (5.2.0-1) ...\n", + "Selecting previously unselected package libproj13:amd64.\n", + "Preparing to unpack .../24-libproj13_5.2.0-1_amd64.deb ...\n", + "Unpacking libproj13:amd64 (5.2.0-1) ...\n", + "Selecting previously unselected package libjbig0:amd64.\n", + "Preparing to unpack .../25-libjbig0_2.1-3.1+b2_amd64.deb ...\n", + "Unpacking libjbig0:amd64 (2.1-3.1+b2) ...\n", + "Selecting previously unselected package libjpeg62-turbo:amd64.\n", + "Preparing to unpack .../26-libjpeg62-turbo_1%3a1.5.2-2+deb10u1_amd64.deb ...\n", + "Unpacking libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", + "Selecting previously unselected package libwebp6:amd64.\n", + "Preparing to unpack .../27-libwebp6_0.6.1-2+deb10u1_amd64.deb ...\n", + "Unpacking libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", + "Selecting previously unselected package libtiff5:amd64.\n", + "Preparing to unpack .../28-libtiff5_4.1.0+git191117-2~deb10u3_amd64.deb ...\n", + "Unpacking libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", + "Selecting previously unselected package libgeotiff2:amd64.\n", + "Preparing to unpack .../29-libgeotiff2_1.4.3-1_amd64.deb ...\n", + "Unpacking libgeotiff2:amd64 (1.4.3-1) ...\n", + "Selecting previously unselected package libgif7:amd64.\n", + "Preparing to unpack .../30-libgif7_5.1.4-3_amd64.deb ...\n", + "Unpacking libgif7:amd64 (5.1.4-3) ...\n", + "Selecting previously unselected package libhdf4-0-alt.\n", + "Preparing to unpack .../31-libhdf4-0-alt_4.2.13-4_amd64.deb ...\n", + "Unpacking libhdf4-0-alt (4.2.13-4) ...\n", + "Selecting previously unselected package libsz2:amd64.\n", + "Preparing to unpack .../32-libsz2_1.0.2-1_amd64.deb ...\n", + "Unpacking libsz2:amd64 (1.0.2-1) ...\n", + "Selecting previously unselected package libhdf5-103:amd64.\n", + "Preparing to unpack .../33-libhdf5-103_1.10.4+repack-10_amd64.deb ...\n", + "Unpacking libhdf5-103:amd64 (1.10.4+repack-10) ...\n", + "Selecting previously unselected package libminizip1:amd64.\n", + "Preparing to unpack .../34-libminizip1_1.1-8+b1_amd64.deb ...\n", + "Unpacking libminizip1:amd64 (1.1-8+b1) ...\n", + "Selecting previously unselected package liburiparser1:amd64.\n", + "Preparing to unpack .../35-liburiparser1_0.9.1-1_amd64.deb ...\n", + "Unpacking liburiparser1:amd64 (0.9.1-1) ...\n", + "Selecting previously unselected package libkmlbase1:amd64.\n", + "Preparing to unpack .../36-libkmlbase1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlbase1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmldom1:amd64.\n", + "Preparing to unpack .../37-libkmldom1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmldom1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlengine1:amd64.\n", + "Preparing to unpack .../38-libkmlengine1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlengine1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlconvenience1:amd64.\n", + "Preparing to unpack .../39-libkmlconvenience1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlconvenience1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlregionator1:amd64.\n", + "Preparing to unpack .../40-libkmlregionator1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlregionator1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package libkmlxsd1:amd64.\n", + "Preparing to unpack .../41-libkmlxsd1_1.3.0-7_amd64.deb ...\n", + "Unpacking libkmlxsd1:amd64 (1.3.0-7) ...\n", + "Selecting previously unselected package mysql-common.\n", + "Preparing to unpack .../42-mysql-common_5.8+1.0.5_all.deb ...\n", + "Unpacking mysql-common (5.8+1.0.5) ...\n", + "Selecting previously unselected package mariadb-common.\n", + "Preparing to unpack .../43-mariadb-common_1%3a10.3.31-0+deb10u1_all.deb ...\n", + "Unpacking mariadb-common (1:10.3.31-0+deb10u1) ...\n", + "Selecting previously unselected package libmariadb3:amd64.\n", + "Preparing to unpack .../44-libmariadb3_1%3a10.3.31-0+deb10u1_amd64.deb ...\n", + "Unpacking libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", + "Selecting previously unselected package libnetcdf13:amd64.\n", + "Preparing to unpack .../45-libnetcdf13_1%3a4.6.2-1_amd64.deb ...\n", + "Unpacking libnetcdf13:amd64 (1:4.6.2-1) ...\n", + "Selecting previously unselected package libltdl7:amd64.\n", + "Preparing to unpack .../46-libltdl7_2.4.6-9_amd64.deb ...\n", + "Unpacking libltdl7:amd64 (2.4.6-9) ...\n", + "Selecting previously unselected package libodbc1:amd64.\n", + "Preparing to unpack .../47-libodbc1_2.3.6-0.1_amd64.deb ...\n", + "Unpacking libodbc1:amd64 (2.3.6-0.1) ...\n", + "Selecting previously unselected package libogdi3.2.\n", + "Preparing to unpack .../48-libogdi3.2_3.2.1+ds-4_amd64.deb ...\n", + "Unpacking libogdi3.2 (3.2.1+ds-4) ...\n", + "Selecting previously unselected package libopenjp2-7:amd64.\n", + "Preparing to unpack .../49-libopenjp2-7_2.3.0-2+deb10u2_amd64.deb ...\n", + "Unpacking libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", + "Selecting previously unselected package liblcms2-2:amd64.\n", + "Preparing to unpack .../50-liblcms2-2_2.9-3_amd64.deb ...\n", + "Unpacking liblcms2-2:amd64 (2.9-3) ...\n", + "Selecting previously unselected package libnspr4:amd64.\n", + "Preparing to unpack .../51-libnspr4_2%3a4.20-1_amd64.deb ...\n", + "Unpacking libnspr4:amd64 (2:4.20-1) ...\n", + "Selecting previously unselected package libnss3:amd64.\n", + "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u4_amd64.deb ...\n", + "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Selecting previously unselected package libpoppler82:amd64.\n", + "Preparing to unpack .../53-libpoppler82_0.71.0-5_amd64.deb ...\n", + "Unpacking libpoppler82:amd64 (0.71.0-5) ...\n", + "Selecting previously unselected package libpq5:amd64.\n", + "Preparing to unpack .../54-libpq5_11.14-0+deb10u1_amd64.deb ...\n", + "Unpacking libpq5:amd64 (11.14-0+deb10u1) ...\n", + "Selecting previously unselected package libqhull7:amd64.\n", + "Preparing to unpack .../55-libqhull7_2015.2-4_amd64.deb ...\n", + "Unpacking libqhull7:amd64 (2015.2-4) ...\n", + "Selecting previously unselected package libspatialite7:amd64.\n", + "Preparing to unpack .../56-libspatialite7_4.3.0a-5+b2_amd64.deb ...\n", + "Unpacking libspatialite7:amd64 (4.3.0a-5+b2) ...\n", + "Selecting previously unselected package libxerces-c3.2:amd64.\n", + "Preparing to unpack .../57-libxerces-c3.2_3.2.2+debian-1+deb10u1_amd64.deb ...\n", + "Unpacking libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", + "Selecting previously unselected package odbcinst.\n", + "Preparing to unpack .../58-odbcinst_2.3.6-0.1_amd64.deb ...\n", + "Unpacking odbcinst (2.3.6-0.1) ...\n", + "Selecting previously unselected package odbcinst1debian2:amd64.\n", + "Preparing to unpack .../59-odbcinst1debian2_2.3.6-0.1_amd64.deb ...\n", + "Unpacking odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", + "Selecting previously unselected package libgdal20.\n", + "Preparing to unpack .../60-libgdal20_2.4.0+dfsg-1+b1_amd64.deb ...\n", + "Unpacking libgdal20 (2.4.0+dfsg-1+b1) ...\n", + "Selecting previously unselected package libxslt1.1:amd64.\n", + "Preparing to unpack .../61-libxslt1.1_1.1.32-2.2~deb10u1_amd64.deb ...\n", + "Unpacking libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", + "Selecting previously unselected package proj-bin.\n", + "Preparing to unpack .../62-proj-bin_5.2.0-1_amd64.deb ...\n", + "Unpacking proj-bin (5.2.0-1) ...\n", + "Selecting previously unselected package python3-affine.\n", + "Preparing to unpack .../63-python3-affine_2.2.2-1_all.deb ...\n", + "Unpacking python3-affine (2.2.2-1) ...\n", + "Selecting previously unselected package python3-attr.\n", + "Preparing to unpack .../64-python3-attr_18.2.0-1_all.deb ...\n", + "Unpacking python3-attr (18.2.0-1) ...\n", + "Selecting previously unselected package python3-soupsieve.\n", + "Preparing to unpack .../65-python3-soupsieve_1.8+dfsg-1_all.deb ...\n", + "Unpacking python3-soupsieve (1.8+dfsg-1) ...\n", + "Selecting previously unselected package python3-bs4.\n", + "Preparing to unpack .../66-python3-bs4_4.7.1-1_all.deb ...\n", + "Unpacking python3-bs4 (4.7.1-1) ...\n", + "Selecting previously unselected package python3-chardet.\n", + "Preparing to unpack .../67-python3-chardet_3.0.4-3_all.deb ...\n", + "Unpacking python3-chardet (3.0.4-3) ...\n", + "Selecting previously unselected package python3-colorama.\n", + "Preparing to unpack .../68-python3-colorama_0.3.7-1_all.deb ...\n", + "Unpacking python3-colorama (0.3.7-1) ...\n", + "Selecting previously unselected package python3-click.\n", + "Preparing to unpack .../69-python3-click_7.0-1_all.deb ...\n", + "Unpacking python3-click (7.0-1) ...\n", + "Selecting previously unselected package python3-click-plugins.\n", + "Preparing to unpack .../70-python3-click-plugins_1.0.4-1_all.deb ...\n", + "Unpacking python3-click-plugins (1.0.4-1) ...\n", + "Selecting previously unselected package python3-cligj.\n", + "Preparing to unpack .../71-python3-cligj_0.5.0-1_all.deb ...\n", + "Unpacking python3-cligj (0.5.0-1) ...\n", + "Selecting previously unselected package python3-webencodings.\n", + "Preparing to unpack .../72-python3-webencodings_0.5.1-1_all.deb ...\n", + "Unpacking python3-webencodings (0.5.1-1) ...\n", + "Selecting previously unselected package python3-html5lib.\n", + "Preparing to unpack .../73-python3-html5lib_1.0.1-1_all.deb ...\n", + "Unpacking python3-html5lib (1.0.1-1) ...\n", + "Selecting previously unselected package python3-lxml:amd64.\n", + "Preparing to unpack .../74-python3-lxml_4.3.2-1+deb10u3_amd64.deb ...\n", + "Unpacking python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", + "Selecting previously unselected package python3-numpy.\n", + "Preparing to unpack .../75-python3-numpy_1%3a1.16.2-1_amd64.deb ...\n", + "Unpacking python3-numpy (1:1.16.2-1) ...\n", + "Selecting previously unselected package python3-pyparsing.\n", + "Preparing to unpack .../76-python3-pyparsing_2.2.0+dfsg1-2_all.deb ...\n", + "Unpacking python3-pyparsing (2.2.0+dfsg1-2) ...\n", + "Selecting previously unselected package python3-snuggs.\n", + "Preparing to unpack .../77-python3-snuggs_1.4.3-1_all.deb ...\n", + "Unpacking python3-snuggs (1.4.3-1) ...\n", + "Selecting previously unselected package python3-rasterio.\n", + "Preparing to unpack .../78-python3-rasterio_1.0.21-1_amd64.deb ...\n", + "Unpacking python3-rasterio (1.0.21-1) ...\n", + "Setting up liblcms2-2:amd64 (2.9-3) ...\n", + "Setting up python3-attr (18.2.0-1) ...\n", + "Setting up mysql-common (5.8+1.0.5) ...\n", + "update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", + "Setting up proj-data (5.2.0-1) ...\n", + "Setting up python3-colorama (0.3.7-1) ...\n", + "Setting up libcharls2:amd64 (2.0.0+dfsg-1) ...\n", + "Setting up libminizip1:amd64 (1.1-8+b1) ...\n", + "Setting up libproj13:amd64 (5.2.0-1) ...\n", + "Setting up libpq5:amd64 (11.14-0+deb10u1) ...\n", + "Setting up proj-bin (5.2.0-1) ...\n", + "Setting up libqhull7:amd64 (2015.2-4) ...\n", + "Setting up python3-click (7.0-1) ...\n", + "Setting up libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", + "Setting up libjbig0:amd64 (2.1-3.1+b2) ...\n", + "Setting up python3-webencodings (0.5.1-1) ...\n", + "Setting up libgeos-3.7.1:amd64 (3.7.1-1) ...\n", + "Setting up libaec0:amd64 (1.0.2-1) ...\n", + "Setting up libicu63:amd64 (63.1-6+deb10u2) ...\n", + "Setting up gdal-data (2.4.0+dfsg-1) ...\n", + "Setting up poppler-data (0.4.9-2) ...\n", + "Setting up mariadb-common (1:10.3.31-0+deb10u1) ...\n", + "update-alternatives: using /etc/mysql/mariadb.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", + "Setting up python3-chardet (3.0.4-3) ...\n", + "Setting up libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", + "Setting up python3-pyparsing (2.2.0+dfsg1-2) ...\n", + "Setting up libnspr4:amd64 (2:4.20-1) ...\n", + "Setting up libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", + "Setting up libgeos-c1v5:amd64 (3.7.1-1) ...\n", + "Setting up python3-click-plugins (1.0.4-1) ...\n", + "Setting up libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", + "Setting up fonts-dejavu-core (2.37-1) ...\n", + "Setting up python3-html5lib (1.0.1-1) ...\n", + "Setting up libltdl7:amd64 (2.4.6-9) ...\n", + "Setting up libgfortran5:amd64 (8.3.0-6) ...\n", + "Setting up libhdf4-0-alt (4.2.13-4) ...\n", + "Setting up libgif7:amd64 (5.1.4-3) ...\n", + "Setting up liburiparser1:amd64 (0.9.1-1) ...\n", + "Setting up libfreexl1:amd64 (1.0.5-3) ...\n", + "Setting up libfyba0:amd64 (4.1.1-6) ...\n", + "Setting up libkmlbase1:amd64 (1.3.0-7) ...\n", + "Setting up libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", + "Setting up libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", + "Setting up python3-affine (2.2.2-1) ...\n", + "Setting up libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", + "Setting up python3-soupsieve (1.8+dfsg-1) ...\n", + "Setting up libsz2:amd64 (1.0.2-1) ...\n", + "Setting up libkmlxsd1:amd64 (1.3.0-7) ...\n", + "Setting up libkmldom1:amd64 (1.3.0-7) ...\n", + "Setting up libspatialite7:amd64 (4.3.0a-5+b2) ...\n", + "Setting up libogdi3.2 (3.2.1+ds-4) ...\n", + "Setting up libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", + "Setting up libkmlengine1:amd64 (1.3.0-7) ...\n", + "Setting up libkmlconvenience1:amd64 (1.3.0-7) ...\n", + "Setting up fontconfig-config (2.13.1-2) ...\n", + "Setting up python3-cligj (0.5.0-1) ...\n", + "Setting up libdap25:amd64 (3.20.3-1) ...\n", + "Setting up libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Setting up libdapserver7v5:amd64 (3.20.3-1) ...\n", + "Setting up python3-bs4 (4.7.1-1) ...\n", + "Setting up libgeotiff2:amd64 (1.4.3-1) ...\n", + "Setting up libblas3:amd64 (3.8.0-2) ...\n", + "update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode\n", + "Setting up libkmlregionator1:amd64 (1.3.0-7) ...\n", + "Setting up libhdf5-103:amd64 (1.10.4+repack-10) ...\n", + "Setting up libodbc1:amd64 (2.3.6-0.1) ...\n", + "Setting up libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", + "Setting up libdapclient6v5:amd64 (3.20.3-1) ...\n", + "Setting up libfontconfig1:amd64 (2.13.1-2) ...\n", + "Setting up liblapack3:amd64 (3.8.0-2) ...\n", + "update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3 to provide /usr/lib/x86_64-linux-gnu/liblapack.so.3 (liblapack.so.3-x86_64-linux-gnu) in auto mode\n", + "Setting up libarpack2:amd64 (3.7.0-2) ...\n", + "Setting up libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", + "Setting up libpoppler82:amd64 (0.71.0-5) ...\n", + "Setting up libnetcdf13:amd64 (1:4.6.2-1) ...\n", + "Setting up python3-numpy (1:1.16.2-1) ...\n", + "Setting up python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", + "Setting up libarmadillo9 (1:9.200.7+dfsg-1) ...\n", + "Setting up python3-snuggs (1.4.3-1) ...\n", + "Setting up odbcinst (2.3.6-0.1) ...\n", + "Setting up odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", + "Setting up libgdal20 (2.4.0+dfsg-1+b1) ...\n", + "Setting up python3-rasterio (1.0.21-1) ...\n", + "Processing triggers for libc-bin (2.28-10) ...\n", + "Processing triggers for man-db (2.8.5-2) ...\n" + ] + } + ], + "source": [ + "mkdir -v output/\n", + "sudo apt-get install python3-rasterio --yes" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "LC08_L1TP_025033_20211010_20211018_01_T1 1\n", + "LC08_L1TP_025033_20211010_20211018_01_T1 2\n", + "LC08_L1TP_025033_20211010_20211018_01_T1 3\n", + "ERROR 4: output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png: No such file or directory\n" + ] + } + ], + "source": [ + "/usr/bin/python3 combine.py" + ] + }, + { + "cell_type": "markdown", + "id": "4199910a-1056-455c-ae9d-669296441aed", + "metadata": {}, + "source": [ + "Note, the \"ERROR 4\" is expected." + ] + }, + { + "cell_type": "markdown", + "id": "a3a89af5-993e-45d6-9e7c-2ca0a61dff7a", + "metadata": {}, + "source": [ + "Verify the results were created." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total 195M\n", + "-rw-r--r-- 1 learner learner 195M Dec 17 19:03 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20211010_20211018_01_T1.png\u001b[0m\n", + "-rw-r--r-- 1 learner learner 910 Dec 17 19:03 result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n" + ] + } + ], + "source": [ + "ls -lh output" + ] + }, + { + "cell_type": "markdown", + "id": "c0257075-537c-4510-bafd-72e9756db17b", + "metadata": {}, + "source": [ + "## Exporting the Results\n", + "\n", + "Now that we have the output data we will create a bucket to place the results. We will first create a bucket with a reasonable set of options.\n", + "\n", + "We fisrt store the bucket name in the `BUCKET` environment variable for future use. This time we will specify a realistic set of options for a private bucket used for computation.\n", + "\n", + "Options (run `gsutil mb --help` for more information):\n", + " * `-b on` specifies uniform bucket-level access.\n", + " * `-l $REGION` puts the data in a specific region for lower cost and lower latency.\n", + " * `--pap enforced` turns on public access prevention to help keep data private. \n", + " \n", + "The uniform bucket level access (Bucket Policy Only enabled: true) puts the data access permissions (ACL) on the entire bucket, not on each object in the bucket. This makes the permissions obvious and makes security much more predictable.\n", + " \n", + "As usual, we must set our environment. In this case we also set a `REGION` environment variable to indicate where in the world we want the data to be stored.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "f36cb8c5-f305-4cb2-a5cc-0c9fd8592fb4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "bucket: essentials-learner-2021-12-17 region: us-west2\n" + ] + } + ], + "source": [ + "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", + "REGION=\"us-west2\"\n", + "echo \"bucket: $BUCKET region: $REGION\"" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "c2ae2b74-5e93-4c55-8bd7-63337f7dcbb8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating gs://essentials-learner-2021-12-17/...\n" + ] + } + ], + "source": [ + "gsutil mb -b on -l $REGION --pap enforced \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "22cc1043-386b-4990-b3dc-62cbdd7ba133", + "metadata": {}, + "source": [ + "Now copy the output data to the bucket. The `-r` flag recursively copies the output directory and `-m` copies the files in parallel. Note the locations of the `-m` and `-r` switches as they apply globally and to the `cp` command respectively." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "681e6b1d-98bb-448a-a57e-f5674214effd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml [Content-Type=application/xml]...\n", + "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png [Content-Type=image/png]...\n", + "==> NOTE: You are uploading one or more large file(s), which would run \n", + "significantly faster if you enable parallel composite uploads. This\n", + "feature can be enabled by editing the\n", + "\"parallel_composite_upload_threshold\" value in your .boto\n", + "configuration file. However, note that if you do this large files will\n", + "be uploaded as `composite objects\n", + "`_,which\n", + "means that any user who downloads such objects will need to have a\n", + "compiled crcmod installed (see \"gsutil help crcmod\"). This is because\n", + "without a compiled crcmod, computing checksums on composite objects is\n", + "so slow that gsutil disables downloads of composite objects.\n", + "\n", + "\\ [2/2 files][194.0 MiB/194.0 MiB] 100% Done \n", + "Operation completed over 2 objects/194.0 MiB. \n" + ] + } + ], + "source": [ + "gsutil -m cp -r \"output\" \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "857861ca-9721-4a1e-a4ae-efffe2143b43", + "metadata": {}, + "source": [ + "Verify that the output was uploaded." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-learner-2021-12-17/output/\n" + ] + } + ], + "source": [ + "gsutil ls \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "b1ea18e9-5861-4479-9948-3303952dee8a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "194.05 MiB 2021-12-17T19:03:54Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", + " 910 B 2021-12-17T19:03:52Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", + "TOTAL: 2 objects, 203478494 bytes (194.05 MiB)\n" + ] + } + ], + "source": [ + "gsutil ls -lh \"gs://$BUCKET/output\"" + ] + }, + { + "cell_type": "markdown", + "id": "66a41f97-3834-4d09-bfe5-c3f224628842", + "metadata": {}, + "source": [ + "## Viewing the Results\n", "\n", + "You now can view the results by using the Google Cloud Platform Web Console and navigating to \"Cloud Storage\", selecting the bucket, and then the result object you wish to view (select the `.png` file). You will need to click the \"Preview\" button given the large size of the image." + ] + }, + { + "cell_type": "markdown", + "id": "a1c11268-f389-405f-8d96-3c319a49b882", + "metadata": {}, + "source": [ + "## Sharing Results\n", + "\n", + "In order to share resources outside a project we must use the Identity Access Management service. This is a powerful tool to grant and restrict access to resources, and if not done correctly It can have serious consequences. **Incorrect permissions can lead to exposure of sensitive data, destruction of data, and authorized use of resources that can result in huge bills.** When in doubt, seek help.\n", + "\n", + "The question \"What access is really needed?\" is the **Principal of Least Privilege** and is a major cornerstone of security. We need to determine the lowest set of permissions or roles that is needed. In our case we wish to grant the \"Collaborator\" the \"Viewer\" access to the \"results bucket\". This will allow them to view, list, and download all objects in the bucket. This illustrates that for a **resource** a **member** (identity) is granted a **permission** (think of it in this order). Together this is called a **policy**. Google also uses \"Roles\" as a collection of predefined and managed permissions." + ] + }, + { + "cell_type": "markdown", + "id": "50990a67-f901-48fd-9295-25ef73245142", + "metadata": { + "tags": [] + }, + "source": [ + "We will now add Members to a Bucket using the Web Console. We will use the Web Console to interactively build the policy binding by doing the following:\n", + " * Navigation Menu -> Storage/Cloud Storage -> Browser -> Click on the Bucket Name (**Bucket Details**) -> Select the **Permissions** tab -> Click **Add** next to \"Permissions\" above the permissions list.\n", + " * In the \"New Principals\" box add the Identity for the collaborator (another individual) as directed by the instructor.\n", + " * Select the \"Storage Object Viewer\" by typing \"Storage Object Viewer\" in the filter and then selecting \"Storage Object Viewer\". Do not use any \"Legacy Storage\" roles.\n", + " * Click \"Save\" to save the policy.\n", + " ![iam-storage-object-viewer](img/iam-storage-object-viewer.png)\n", + " * Verify the policy is listed in the \"Permissions\" table on the \"Bucket Details\" page (you should now be on this page).\n", + "\n", + "*Advanced Note: Changes in permissions will show up on the **Home** -> **Activity** page.*" + ] + }, + { + "cell_type": "markdown", + "id": "6a191922-7d5b-4125-9812-181eb056f86c", + "metadata": { + "tags": [] + }, + "source": [ + "Collaborators should now be able to see the contents of the bucket by explicitly naming it as shown below.\n", + "```\n", + "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17\n", + "gs://essentials-learner-2021-12-17/output/\n", + "\n", + "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17/output\n", + "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", + "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", "```" ] + }, + { + "cell_type": "markdown", + "id": "40d9ab41-8920-45f2-8218-550baac5b069", + "metadata": {}, + "source": [ + "## Cleanup\n", + "\n", + "We will now leave the resources running in order to learn more about monitoring costs and will clean up all the resources as the end of Lesson. **Don't forget to do this!**" + ] } ], "metadata": { diff --git a/content/Azure/07_monitoring_costs.ipynb b/content/Azure/07_monitoring_costs.ipynb index 0c280fa..e0b5d2d 100644 --- a/content/Azure/07_monitoring_costs.ipynb +++ b/content/Azure/07_monitoring_costs.ipynb @@ -26,15 +26,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/08_cleaning_up_resources.ipynb b/content/Azure/08_cleaning_up_resources.ipynb index 52b682f..6b11af4 100644 --- a/content/Azure/08_cleaning_up_resources.ipynb +++ b/content/Azure/08_cleaning_up_resources.ipynb @@ -111,15 +111,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/images/azure_intro_console.jpg b/content/Azure/images/azure_intro_console.jpg index c3cd325..1f77f2e 100644 Binary files a/content/Azure/images/azure_intro_console.jpg and b/content/Azure/images/azure_intro_console.jpg differ diff --git a/content/Azure/images/azure_intro_console_orig.jpg b/content/Azure/images/azure_intro_console_orig.jpg new file mode 100644 index 0000000..c3cd325 Binary files /dev/null and b/content/Azure/images/azure_intro_console_orig.jpg differ diff --git a/content/Azure/intro_to_Azure_Essentials.ipynb b/content/Azure/intro_to_Azure_Essentials.ipynb index c7ec784..2984146 100644 --- a/content/Azure/intro_to_Azure_Essentials.ipynb +++ b/content/Azure/intro_to_Azure_Essentials.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "21ff6943", + "id": "30c652f8", "metadata": {}, "source": [ "# Microsoft Azure\n", @@ -10,6 +10,7 @@ "The modules in this lesson are: \n", "\n", "1. [Introduction to the Azure Cloud Portal](./01_intro_to_cloud_console)\n", + "2. [Introduction to the Azure CLI](./02_intro_to_cli.ipynb)\n", "2. [Introduction to Cloud Compute (Azure Virtual Machines)](./02_intro_to_compute)\n", "3. [Introduction to the Azure CLI](./04_intro_to_cli)\n", "4. [Introduction to Cloud Storage (Blob Storage)](./05_intro_to_cloud_storage)\n", @@ -17,11 +18,19 @@ "6. [Monitoring Costs](./07_monitoring_costs)\n", "7. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)" ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "92d161a5", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "Python 3", "language": "python", "name": "python3" }, @@ -35,7 +44,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.10" + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/intro_to_compute_cli.ipynb b/content/Azure/intro_to_compute_cli.ipynb new file mode 100644 index 0000000..0bb8ebf --- /dev/null +++ b/content/Azure/intro_to_compute_cli.ipynb @@ -0,0 +1,260 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Intro to Compute CLI\n", + "\n", + "**Overview**\n", + " \n", + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + " * How do I create my own computer in the cloud?\n", + "\n", + "**Objectives:**\n", + " * Allocate a virtual machine in Azure.\n", + " * Learn how to connect to a virtual machine using ssh in the cloud shell." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Open the cloud shell (See: intro_to_cli)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Define some environment variables" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "RG=\"demo_rg\"\n", + "LOCATION=eastus\n", + "VM=\"demo_vm\"\n", + "STGE_ACCT=\"demo_stor_ac\"\n", + "CTR=\"demo_stge_ctr\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create resource group" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az group create --name $RG --location $LOCATION" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create a VM" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If this is the fist time running this command (or your ~/.ssh dir does not have an id_rsa key pair). It will generate a new key pair id_rsa in ~/.ssh. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az vm create --resource-group $RG \\\n", + " --name $VM \\\n", + " --image UbuntuLTS \\\n", + " --admin-username demouser \\\n", + " --generate-ssh-keys \\\n", + " --output json \\\n", + " --verbose " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To create a VM with an existing pair first create a new key pair" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ssh-keygen" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You will be prompted to \"Enter file in which to save the key\", type \"id_rsa\". This will create a key pair in the current dir." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here you should enter a passphrase (a series of words) to protect the key. You will be asked to re-enter the passphrase as shown below:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Enter same passphrase again:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After successfully retyping the passphrase the key generation process will continue showing similar message as follows (the fingerprint and art will look different):" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note: The following using \"--ssh-key-values\" to define which keys to use." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az vm create --name $VM \n", + " --resource-group $RG \n", + " --image UbuntuLTS \n", + " --admin-username demouser \n", + " --ssh-key-values id_rsa.pub" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Show VM details (all)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az vm show --name $VM --resource-group $RG" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Login into VM" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Get the public IP address from the output above.\n", + "If you generated a new key pair using \"--generate-ssh-keys\", login with the following command." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ssh " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you used the command with \"--ssh-key-values id_rsa.pub\" to use existing keys, use the following command to logiin:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ssh -i /id_rsa " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "clean up\n", + "\n", + "The --no-wait parameter keeps the CLI from blocking while the deletion takes place." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az group delete --name $RG --no-wait" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/content/_toc.yml b/content/_toc.yml index 450f420..09aa2cf 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -26,8 +26,9 @@ parts: sections: - file: Azure/01_intro_to_cloud_console - file: Azure/02_intro_to_compute + - file: Azure/03_intro_to_cloud_storage - file: Azure/04_intro_to_cli - - file: Azure/05_intro_to_cloud_storage + - file: Azure/05_intro_to_cloud_storage_cli - file: Azure/06_running_analysis - file: Azure/07_monitoring_costs - file: Azure/08_cleaning_up_resources