diff --git a/Build.md b/Build.md index c4033d1..2a88a6d 100644 --- a/Build.md +++ b/Build.md @@ -32,14 +32,27 @@ This copies the contents of `content/_build/html/` to `docs/` so GitHub Pages wi ## GCP -Using Jupyter launched from a remote Visual Code Studio session inside GCP Cloud-Shell requires some local settings stored in `local.sh`, use the following as a template replacing the project and email with the proper vaules: -``` -gcloud config set account student31@class.internet2.edu -export GOOGLE_CLOUD_PROJECT=just-armor-301114 -export DEVSHELL_PROJECT_ID=$GOOGLE_CLOUD_PROJECT -gcloud config set project $GOOGLE_CLOUD_PROJECT +Checkout the project so it is not seen by the students. Run `./scripts/gcp-cloudshell.sh` to open the shell with the agent and ports 8080 and 8081 forwarded and run the following commands (onetime setup). Be sure to check the project and account values. + +```bash +git clone git@github.internet2.edu:/CLASS/CLASS-Essentials.git .essentials +cd .essentials/ +git checkout -b gcp-dev --track origin/gcp-dev + +ACCOUNT=learner@class.internet2.edu +PROJECT=essentials-learner +cat > .essentials/local.sh < Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at [creativecommons.org/policies](http://creativecommons.org/policies), Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. +> +> Creative Commons may be contacted at creativecommons.org diff --git a/README.md b/README.md index f62fac3..56b09b3 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,21 @@ -# CLASS-core -Curriculum development for Cloud for Researchers, A Primer +# CLASS-Essentials +This is the repository for the CLASS Essentials Curriculum material for AWS, GCP and Azure. -For development see the [build instructions](Build.md). +# Contribution Workflow +We use a `Feature` branch workflow: each user works on a specific Feature branch associated with a specific cloud provider. The core idea behind the `Feature` Branch Workflow is that all feature development should take place in a dedicated branch instead of the main branch. The Feature branches in this repository currently are `aws-dev`, `gcp-dev`, and `azure-dev`. More information can be found on [this Atlassian page](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow). + +Best Practices: +1. Do not edit directly in a branch that you did not create. Use Github issues or pull requests to suggest or request changes. +2. Make pull requests before merging to main. + +# Maintainers +[John Hicks](jhicks@internet2.edu) + +[Tim Middlekoop](tmiddelkoop@internet2.edu) + +[Ananya Ravipati](aravipati@internet2.edu) + +[Amanda Tan](atan@internet2.edu) + +# License +Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. diff --git a/content/Azure/01_intro_to_cloud_console.ipynb b/content/Azure/01_intro_to_cloud_console.ipynb index d3e30aa..7619591 100644 --- a/content/Azure/01_intro_to_cloud_console.ipynb +++ b/content/Azure/01_intro_to_cloud_console.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "b6f03192", + "id": "b3a74289", "metadata": {}, "source": [ "# Introduction to the Azure Cloud Console\n", @@ -28,19 +28,18 @@ }, { "cell_type": "markdown", - "id": "abb35991", + "id": "7ba16e4b", "metadata": {}, "source": [ "## The Who, What, and Where of the Cloud\n", "\n", "Before we get started we must first define, and find, a few key pieces of information about your environment that will be used throughout this lesson. This information is also important to ensure that this information is what you expect, otherwise you may use the wrong account or run or store things in the wrong place. It is important to be clear about these terms as they are often different in other cloud providers. \n", - "\n", - "(need glossary for Azure) We provide a [glossary of Azure terms](glossary.ipynb) for your reference." + "We provide a [glossary of Azure terms](glossary.ipynb) for your reference." ] }, { "cell_type": "markdown", - "id": "3e128379", + "id": "b8747e81", "metadata": {}, "source": [ "## Accounts (Authentication and Identity) & login\n", @@ -48,21 +47,21 @@ "The (Azure) **Account** is the identity used to access the Azure web console and is in the form of an email address and is associated with an individual. This is typically the account you use to access Azure services associated with your institution and not your personal account. We will now login to the Azure web console and verify our account.\n", "\n", " * Login to the web console with your (Azure) account by going to [https://portal.azure.com//](https://portal.azure.com//)\n", - " * Fill in your username and password (assigned to you in the an email from CLASS or perhap from your institution. \n", - " * Find your user/account under the \"Avatar\" in the top right (see number 4 below).\n", + " * Fill in your username and password (assigned to you in the an email from CLASS or perhap from your institution). \n", + " * Find your user/account under the \"Avatar\" in the top right (see number 5 below).\n", "\n", - "You are now logged on. Welcome to the first step of your cloud journey!" + "You are now logged in. Welcome to the first step of your cloud journey!" ] }, { "cell_type": "markdown", - "id": "90066c19", + "id": "6f9af1ea", "metadata": {}, "source": [ "### Exercise\n", "\n", "Take a few moments to ask yourself the following questions:\n", - " * What account are you currently using in the Google Cloud web console?\n", + " * What account are you currently using in the Azure Cloud web console?\n", " * Are you logged into any other accounts?\n", " * How do you know which account is being used?\n", " \n", @@ -71,66 +70,50 @@ }, { "cell_type": "markdown", - "id": "160353bd", + "id": "4901d3bb", "metadata": {}, "source": [ "## Subscriptions\n", " \n", - "Click on the *Directories & Subscriptions* buttton (#4).\n", - "Verify that you are in the right *Directory and Subscription* (briefly discribe D&S).\n", + "Click on the *Directories & Subscriptions* buttton (number 4 below).\n", + "Verify that you are in the right *Directory and Subscription* (For more information on Directories and Subscriptions, see the *glossary of Azure terms* link above).\n", " \n", "Take a few moments to ask yourself the following questions:\n", - " * What is the *name* of the Project you are currently using in the Google Cloud Platform web console?\n", - " * What is the *Project ID* of that project?\n", - " * What is the *Project Number* of that project?\n", - " * Do you have multiple projects?\n", - " * How do you know which project you are currently using?\n", - "\n", - "If you are currently not using the correct project for this lesson, switch to the correct one." - ] - }, - { - "cell_type": "markdown", - "id": "6e3b3cbd", - "metadata": {}, - "source": [ - "## Services\n", - "\n", - "The web console is used to control and observe the cloud from the browser. It should only be used for simple and one-time tasks, exploring new services, accessing documentation, or for monitoring and debugging resources in the cloud. Programmatic control through the console, programming languages (for example Python), and other automation tools should be used for day to day activities to make the most out of the cloud and to help with the reproducibility of research and teaching.\n", + " * What is the *name* of the Subscription you are currently using in the Azure web console?\n", + " * Do you have multiple subscriptions?\n", + " * How do you know which subscription you are currently using?\n", "\n", - "The Navigation Menu (number 1 above) is used to navigate to the various products, which are also sometimes called services.\n" + "If you are currently not using the correct subscription for this lesson, switch to the correct one." ] }, { "cell_type": "markdown", - "id": "ed7e581f", + "id": "d2611421", "metadata": {}, "source": [ "## Services\n", "\n", "The web console is used to control and observe the cloud from the browser. It should only be used for simple and one-time tasks, exploring new services, accessing documentation, or for monitoring and debugging resources in the cloud. Programmatic control through the console, programming languages (for example Python), and other automation tools should be used for day to day activities to make the most out of the cloud and to help with the reproducibility of research and teaching.\n", "\n", - "The Navigation Menu (often called the hamburger) is used to navigate to the various products, which are also sometimes called services.\n", + "The Navigation Menu (number 1 below) is used to navigate to the various products, which are also called services.\n", "\n", - "![hamburger-navigation](img/hamburger-navigation.png)\n", + "![hamburger-navigation](images/Azure-services-menu.png)\n", "\n", - " * Click on a product to navigate the product page\n", - " * You can pin frequently used items on the top of this page by clicking on the pin icon. \n", + " * Click on a product to navigate to the product page \n", " * Click the hamburger again to hide the Left Sidebar.\n", "\n", - "All the different products and services are the \"What\" of the cloud. We will start with compute, called *Google Compute Engine* in the next Episode. \n", + "All the different products and services are the \"What\" of the cloud. We will start with compute, called *Azure VM* in the next Episode. \n", "\n", "### Exercise \n", "\n", - "Take a few moments to navigate a few key services.\n", - " * Navigate to the \"Compute Engine\" service under the \"Compute\" product group.\n", - " * You will probably need to \"Enable\" this service first by clicking on the \"Enable\" button on the \"Compute Engine API\" page. This will only need to be done once per project.\n", - " * Navigate to \"Cloud Storage\" under the \"Storage\" product group and enable the service if necessary." + "Take a few moments to navigate to a few key services.\n", + " * Navigate to the \"Virtual Machine\" service in the Left Sidebar menu.\n", + " * Navigate to \"Storage Accounts\" service in the Left Sidebar menu.\n" ] }, { "cell_type": "markdown", - "id": "4537d270", + "id": "13817be0", "metadata": {}, "source": [ "## Everything Else\n", @@ -160,7 +143,7 @@ }, { "cell_type": "markdown", - "id": "e94b53e2", + "id": "5ab062d4", "metadata": {}, "source": [ "## Open a Cloud Shell\n", @@ -169,14 +152,80 @@ "\n", "To use the Cloud Shell:\n", " * Open up a Cloud Shell by clicking on the **Cloud Shell** icon in the right side of the top blue bar (number 3 above).\n", - " * Once the Cloud Shell is open, you should see \"Welcome to Azure Cloud Shell\".\n", + " * Once the Cloud Shell is open and has initialized, you should see \"Welcome to Azure Cloud Shell\".\n", " * Click the middle box icon on the right side of the Cloud Shell bar to maximize the screen. You can find the correct icon by hovering over the icons to see their names.\n", - " * Note: You can open the Cloud Shell in a new tab by clicking on the **Open new session** icon (looks like a box with a \"+\") in the middle of the Cloud Shell bar.\n" + " * Note: You can open the Cloud Shell in a new tab by clicking on the **Open new session** icon (looks like a box with a \"+\") in the middle of the Cloud Shell bar.\n", + "\n", + "![Azure-cloud-shell](images/Azure-cloud-shell.png)\n", + "\n", + "You will notice that in the upper left corner of the cloud shell there is a pull down menu. This should be set to **bash** (The other option is PowerShell). We will now prepare for the next lessons by creating an ssh key." + ] + }, + { + "cell_type": "markdown", + "id": "53d98252", + "metadata": {}, + "source": [ + "Now we will create a ssh key pair to use with some of the other lessons (e.g. we need a ssh key when creating a VM in order to login securely).\n", + "\n", + "At the **$** prompt, enter **ssh-keygen** and follow the prompts." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "be8b4eb8", + "metadata": {}, + "outputs": [], + "source": [ + "$ ssh-keygen\n", + "Generating public/private rsa key pair.\n", + "Enter file in which to save the key (/home/john/.ssh/id_rsa):" + ] + }, + { + "cell_type": "markdown", + "id": "af178e4d", + "metadata": {}, + "source": [ + "This prompt is asking you to create a .ssh directory in your home directory and generate ssh public/private keys named id_rsa.pub/id_rsa (default).\n", + "\n", + "Press enter to continue to create the key. You will see another message asking you for a passphrase, similar to the following:\n", + "```\n", + "Generating public/private rsa key pair.\n", + "Enter passphrase (empty for no passphrase):\n", + "```\n", + "Here you should enter a passphrase (a series of words) to protect the key. You will be asked to re-enter the passphrase as shown below:\n", + "```\n", + "Enter same passphrase again:\n", + "```\n", + "\n", + "After successfully retyping the passphrase the key generation process will continue showing similar message as follows (the fingerprint and art will look different):\n", + "```\n", + "Your identification has been saved in /home/learner/.ssh/id_rsa.\n", + "Your public key has been saved in /home/learner/.ssh/id_rsa.pub.\n", + "The key fingerprint is:\n", + "SHA256:D9yj1nZEUigahirvjkIFoCP7RfjDu67Ku88M7nHQftI learner@cs-748001990186-default-boost-r9phq\n", + "The key's randomart image is:\n", + "+---[RSA 2048]----+\n", + "|o . .. |\n", + "|.. .. o . .. |\n", + "|+ o... o .. . |\n", + "|ooo= .. . o |\n", + "|.+..= S o . |\n", + "| o+..o = o |\n", + "|.+.+.E o + . |\n", + "|+.B o. . . . |\n", + "|=O**o |\n", + "+----[SHA256]-----+\n", + "```\n", + "\n", + "The ssh key generation is now complete. This will only happen once for your account. Later during the VM create proces you will be asked to Cut & Paste the public key into a form. " ] }, { "cell_type": "markdown", - "id": "e17944ad", + "id": "779c8715", "metadata": {}, "source": [ "## Exercise\n", @@ -197,7 +246,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -211,7 +260,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.8" + "version": "3.9.7" } }, "nbformat": 4, diff --git a/content/Azure/02_intro_to_compute.ipynb b/content/Azure/02_intro_to_compute.ipynb index 5c47b31..d401cb0 100644 --- a/content/Azure/02_intro_to_compute.ipynb +++ b/content/Azure/02_intro_to_compute.ipynb @@ -38,7 +38,7 @@ }, { "cell_type": "markdown", - "id": "3565e61f", + "id": "eb6326d6", "metadata": {}, "source": [ "## Azure Virtual Machines" @@ -46,11 +46,15 @@ }, { "cell_type": "markdown", - "id": "4efebb57", + "id": "ce935ec7", "metadata": {}, "source": [ "Before we begin this lesson/episode we need to create a ssh key in order to login into the VM we about to create. We will use the public key when we create the VM.\n", "\n", + "**Note:** Step 1. below is not required if you have already created a ssh key in the [Intro to Cloud Console](01_intro_to_cloud_console.ipynb) lesson. If you have already created a key go directly to step 2.\n", + "\n", + "**Step 1.**\n", + "\n", "If you have not used the cloud shell to connect to a *VM Instance* before you will need to create a new *ssh key*. The Compute Engine will use this key to allow you to access the *VM instance* in a secure manner. If this is the case you will see a message similar to the following:\n", "```\n", "WARNING: The private SSH key file for gcloud does not exist.\n", @@ -95,20 +99,17 @@ "\n", "The ssh key generation is now complete. This will only happen once for your account. Later during the VM create proces you will be asked to Cut & Paste the public key into a form. \n", "\n", - "Copy the public key to your clip board \n", - "\n", "\n", - "future connections.\n" + "**Step 2.**\n", + "Copy the public key to your clip board future connections.\n" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "7245d24e", + "cell_type": "markdown", + "id": "0f47f212", "metadata": {}, - "outputs": [], "source": [ - "cat /home/learner/.ssh/azure_compute_engine.pub" + "cat /home/learner/.ssh/id_rsa.pub" ] }, { @@ -294,7 +295,7 @@ }, { "cell_type": "markdown", - "id": "bb819970", + "id": "41119145", "metadata": {}, "source": [ "## Clean up\n", @@ -314,7 +315,9 @@ "* Select the Resource Group to delete.\n", "* Click delete\n", "* A warning will ask you to enter the name of the Resource Group to make sure you want to continue with this perminent operation.\n", - "* Confirm that you want ot delete the Resouce Group." + "* Confirm that you want ot delete the Resouce Group.\n", + "\n", + "Note: By deleting the Resource Group you will delete all services under it." ] }, { @@ -328,7 +331,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -342,7 +345,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.8" + "version": "3.9.7" } }, "nbformat": 4, diff --git a/content/Azure/04_intro_to_cli.ipynb b/content/Azure/04_intro_to_cli.ipynb index 09f62d7..f0b0cea 100644 --- a/content/Azure/04_intro_to_cli.ipynb +++ b/content/Azure/04_intro_to_cli.ipynb @@ -2,18 +2,18 @@ "cells": [ { "cell_type": "markdown", - "id": "f5be7a6c", + "id": "0fb9d517", "metadata": {}, "source": [ "# Introduction to the Azure CLI\n", "\n", "Learner Questions\n", " * How do I use the Cloud CLI?\n", - " * How do I use `az`?\n", + " * How do I use az?\n", "\n", "Learning Objectives\n", " * Find and use the \"Cloud Shell\" in the web console\n", - " * Use basic cloud CLI commands (`az`)\n", + " * Use basic cloud CLI commands (az)\n", " * Determine basic user/account settings.\n", " * Use environment variables for configuration - No hard coding\n", " * Understand the importance of using variables for configuration.\n", @@ -24,7 +24,7 @@ }, { "cell_type": "markdown", - "id": "b609b6ed", + "id": "7ec8b5bf", "metadata": {}, "source": [ "## Cloud Automation\n", @@ -38,7 +38,7 @@ }, { "cell_type": "markdown", - "id": "e8cda207", + "id": "68bd896e", "metadata": {}, "source": [ "## Open the Cloud Shell\n", @@ -48,13 +48,13 @@ "To use the Azure Cloud Shell:\n", " \n", " * Open up a Cloud Shell by clicking on the cloud shell icon (greater than sign followed by underscore) on the blue bar just to the right of search window. On first launch, Cloud Shell prompts to create a resource group, storage account, and Azure Files share on your behalf. This will open a cloud shell at the bottum of your browser window. You can choose a bash shell or Power shell.\n", - " * If you want a new cloud shell in a different window, click the new cloud shell button (file icon with '+' in upper right corner) on the cloud shell menu bar. This will create a new cloud shell window tab. **Note:** you may need to allow popups or enable popups for this website depending on the browser you use. You can also pull up the tab to increase the shell screen size.\n", + " * If you want a new cloud shell in a different window, click the new cloud shell button (file icon with + in upper right corner) on the cloud shell menu bar. This will create a new cloud shell window tab. **Note:** you may need to allow popups or enable popups for this website depending on the browser you use. You can also pull up the tab to increase the shell screen size.\n", " * To quit cloud shell type exit at the prompt." ] }, { "cell_type": "markdown", - "id": "1799e8dd", + "id": "d9583740", "metadata": {}, "source": [ "## Exercise\n", @@ -67,62 +67,37 @@ }, { "cell_type": "markdown", - "id": "da18d22e", + "id": "366108a9", "metadata": {}, "source": [ "## Basic Cloud Commands\n", "\n", "The cloud can also be controlled using a command line interface (CLI) or a programming language such as Python. Collectively these tools interact with the cloud over a Application Programming Interface (API) and this capability forms the basis of the cloud, the ability to control infrastructure programmatically. \n", "\n", - "For information about installing the Azure CLI for your OS see: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli. If you are on your own OS you will need to login via `az login`.\n", + "For information about installing the Azure CLI for your OS see: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli. If you are on your own OS you will need to login via az login.\n", "\n", - "The `az` command is used to control most aspects of Azure Cloud." + "The az command is used to control most aspects of Azure Cloud." ] }, { "cell_type": "markdown", - "id": "f7b90dba", + "id": "a68f3190", "metadata": {}, "source": [ "Now let us look at your account information (your details will be different). " ] }, { - "cell_type": "code", - "execution_count": 1, - "id": "27170c24", + "cell_type": "markdown", + "id": "c61af998", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[\n", - " {\n", - " \"cloudName\": \"AzureCloud\",\n", - " \"id\": \"975b9702-e6b9-47b5-94a5-ca5f2b110ca8\",\n", - " \"isDefault\": true,\n", - " \"name\": \"class-dev\",\n", - " \"state\": \"Enabled\",\n", - " \"tenantId\": \"78ea4b46-9f08-4ef5-949b-2dae057c55d8\",\n", - " \"user\": {\n", - " \"assignedIdentityInfo\": \"MSI\",\n", - " \"name\": \"systemAssignedIdentity\",\n", - " \"type\": \"servicePrincipal\"\n", - " }\n", - " }\n", - "]\n", - "\u001b[0m" - ] - } - ], "source": [ - "az account list" + "# az account list" ] }, { "cell_type": "markdown", - "id": "e8feb7f2", + "id": "8b51b583", "metadata": {}, "source": [ "## Environment Variables\n", @@ -133,36 +108,24 @@ ] }, { - "cell_type": "code", - "execution_count": 2, - "id": "b8f74fe9-0943-4bd9-a91b-da09c001f8a1", + "cell_type": "markdown", + "id": "a67e26c4", "metadata": {}, - "outputs": [], "source": [ "TEST=1" ] }, { - "cell_type": "code", - "execution_count": 3, - "id": "b937f6c7-3697-4bc0-8e32-da0b0faec593", + "cell_type": "markdown", + "id": "2af53bf6", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "1\n" - ] - } - ], "source": [ "echo $TEST" ] }, { "cell_type": "markdown", - "id": "78fe7013-a5ae-498f-aeab-ba1f3c710de1", + "id": "5d784f5c", "metadata": {}, "source": [ "Although we do not always need to, we should always enclose shell variables in double quotes and enclose the variable name in curly braces (funny things may happen otherwise). In most cases it is easier to do this every time than try to figure out when it is possible to use the short form. We only use the short form here when it is used frequently and known to always work.\n", @@ -171,108 +134,68 @@ ] }, { - "cell_type": "code", - "execution_count": 4, - "id": "bae0587c-0379-40e3-9c31-16ba5505487a", + "cell_type": "markdown", + "id": "f70908e5", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "learner\n" - ] - } - ], "source": [ "echo \"${USER}\"" ] }, { - "cell_type": "code", - "execution_count": 5, - "id": "52b4add5-0d18-4faf-a304-b7e0ee8ba82e", + "cell_type": "markdown", + "id": "6c5bfd86", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "essentials\n" - ] - } - ], "source": [ "echo \"${HOSTNAME}\"" ] }, { "cell_type": "markdown", - "id": "0be0c5dc-8f65-4749-a56b-f47bbdeb6106", + "id": "0722637d", "metadata": {}, "source": [ - "If you are using the cloudshell then your hostname will look similar to `cc-2510c465-7677c9dc4c-fcl8k`" + "If you are using the cloudshell then your hostname will look similar to cc-2510c465-7677c9dc4c-fcl8k" ] }, { "cell_type": "markdown", - "id": "ef895f38-ad16-4f2d-aa18-ec913732dd69", + "id": "e07f6919", "metadata": {}, "source": [ "We can also use the output of commands to set string values." ] }, { - "cell_type": "code", - "execution_count": 6, - "id": "776806e4-209f-45fa-b14e-f771f5d5089d", + "cell_type": "markdown", + "id": "a90996d2", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Thu Dec 2 17:16:22 UTC 2021\n" - ] - } - ], "source": [ - "date" + "# date" ] }, { - "cell_type": "code", - "execution_count": 7, - "id": "560fde63-0eb8-4c52-9787-4debfe35ec23", + "cell_type": "markdown", + "id": "20a5bdaa", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Date: Thu Dec 2 17:16:23 UTC 2021\n" - ] - } - ], "source": [ - "echo \"Date: $(date)\"" + "# echo \"Date: $(date)\"" ] }, { "cell_type": "markdown", - "id": "49c2302b", + "id": "108b3e76", "metadata": { "tags": [] }, "source": [ - "First, as with all Azure work we need to create a Resource Group (RG). A resource group is a logical container into which Azure resources are deployed and managed. Now let us combine all of the above to create a `RG` but first let us set a RG name and region env variable so we do not need to use the actual value in future commands.\n", + "First, as with all Azure work we need to create a Resource Group (RG). A resource group is a logical container into which Azure resources are deployed and managed. Now let us combine all of the above to create a RG but first let us set a RG name and region env variable so we do not need to use the actual value in future commands.\n", "\n", "Example:" ] }, { "cell_type": "markdown", - "id": "1f73131a", + "id": "1c87694c", "metadata": {}, "source": [ "Define some enviroment variables." @@ -280,7 +203,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 13, "id": "d00a37e4", "metadata": {}, "outputs": [], @@ -290,175 +213,68 @@ ] }, { - "cell_type": "code", - "execution_count": 9, - "id": "d7988c45", + "cell_type": "markdown", + "id": "6b5db328", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "rg-essentials\n" - ] - } - ], "source": [ - "echo $RG" + " # echo $RG" ] }, { - "cell_type": "code", - "execution_count": 13, - "id": "49f484a6", + "cell_type": "markdown", + "id": "b11e6783", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"id\": \"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials\",\n", - " \"location\": \"eastus\",\n", - " \"managedBy\": null,\n", - " \"name\": \"rg-essentials\",\n", - " \"properties\": {\n", - " \"provisioningState\": \"Succeeded\"\n", - " },\n", - " \"tags\": null,\n", - " \"type\": \"Microsoft.Resources/resourceGroups\"\n", - "}\n", - "\u001b[0m" - ] - } - ], "source": [ - "az group create --name $RG --location $REGION" + "# az group create --name $RG --location $REGION" ] }, { "cell_type": "markdown", - "id": "7fe71e66", + "id": "a22ac088", "metadata": {}, "source": [ - "Now use the `RG` environment variable to get information about your resource group. This information is available at any time with the following command not just at creation." + "Now use the RG environment variable to get information about your resource group. This information is available at any time with the following command not just at creation." ] }, { - "cell_type": "code", - "execution_count": 14, - "id": "31b01f78-94b1-44f0-a2af-d657058941b2", + "cell_type": "markdown", + "id": "1a7686fc", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"id\": \"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials\",\n", - " \"location\": \"eastus\",\n", - " \"managedBy\": null,\n", - " \"name\": \"rg-essentials\",\n", - " \"properties\": {\n", - " \"provisioningState\": \"Succeeded\"\n", - " },\n", - " \"tags\": null,\n", - " \"type\": \"Microsoft.Resources/resourceGroups\"\n", - "}\n", - "\u001b[0m" - ] - } - ], "source": [ - "az group show --resource-group $RG" + "# az group show --resource-group $RG" ] }, { "cell_type": "markdown", - "id": "270e0abb-d5e9-476a-af91-060cc0ea0349", + "id": "0eb11c11", "metadata": {}, "source": [ "## Exercise\n", "FIXME:\n", " * What command shows the project number? (hint: we used it already)\n", " * What is the name of this variable? (hint: it is case sensitive)\n", - " * What command shows *just* the Project Number (hint: use `--format`)\n", - " * Set the `PROJECT_NUMBER` environment variable with the correct value. (hint: be careful with punctuation and matching punctuation)" + " * What command shows *just* the Project Number (hint: use --format)\n", + " * Set the *PROJECT_NUMBER* environment variable with the correct value. (hint: be careful with punctuation and matching punctuation)" ] }, { "cell_type": "markdown", - "id": "3b744cd4-3c7e-43f2-83ae-0ff02ec9de83", + "id": "8da1b71e", "metadata": {}, "source": [ "## Advanced Call-Out: Formating Output.\n", "\n", - "The following example shows how to output the configuration as `JSON`." + "The following example shows how to output the configuration as JSON." ] }, { - "cell_type": "code", - "execution_count": 18, - "id": "678a1735-8cad-403c-b302-639de1fb0f4b", + "cell_type": "markdown", + "id": "8e241cd0", "metadata": { "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1;39m[\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/cloudshell\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"centralus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"cloudshell\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials-dev\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"centralus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"rg-essentials-dev\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/NetworkWatcherRG\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"eastus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"NetworkWatcherRG\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"eastus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"rg-essentials\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m\n", - "\u001b[1;39m]\u001b[0m\n" - ] - } - ], "source": [ - "az group list --output json | jq" + "# az group list --output json | jq" ] } ], diff --git a/content/Azure/05_intro_to_cloud_storage_cli.ipynb b/content/Azure/05_intro_to_cloud_storage_cli.ipynb index 5451fe8..b082779 100644 --- a/content/Azure/05_intro_to_cloud_storage_cli.ipynb +++ b/content/Azure/05_intro_to_cloud_storage_cli.ipynb @@ -45,12 +45,12 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "RG=\"demo_rg\"\n", - "LOCATION=eastus\n", + "LOCATION=\"eastus\"\n", "BLOB_CON=\"demo_blob_cont\"\n", "STGE_ACCT=\"demo_stor_ac\"\n", "CTR=\"demo_stge_ctr\"" @@ -83,16 +83,20 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "ename": "SyntaxError", + "evalue": "invalid syntax (838669738.py, line 1)", + "output_type": "error", + "traceback": [ + "\u001b[0;36m File \u001b[0;32m\"/var/folders/ty/h84mzh251vdbfwf3zsph_xrh0000gq/T/ipykernel_80687/838669738.py\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m az storage account create --name $STGE_ACCT --resource-group $RG --location $LOCATION --sku Standard_ZRS --encryption-services blob\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n" + ] + } + ], "source": [ - "az storage account create \\\n", - " --name $STGE_ACCT \\\n", - " --resource-group $RG \\\n", - " --location $LOCATION \\\n", - " --sku Standard_ZRS \\\n", - " --encryption-services blob\n" + "az storage account create --name $STGE_ACCT --resource-group $RG --location $LOCATION --sku Standard_ZRS --encryption-services blob" ] }, { @@ -105,10 +109,8 @@ ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ "az ad signed-in-user show --query objectId -o tsv | az role assignment create \\\n", " --role \"Storage Blob Data Contributor\" \\\n", @@ -124,15 +126,10 @@ ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "az storage container create \\\n", - " --account-name $STGE_ACCT \\\n", - " --name $CTR \\\n", - " --auth-mode login" + "az storage container create --account-name $STGE_ACCT --name $CTR --auth-mode login" ] }, { @@ -165,18 +162,27 @@ "Copy demo_file to your blob." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "az storage blob upload --account-name $STGE_ACCT --container-name $CTR --name demo_file --file demo_file --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify that the blob is there by Listing the blobs in the container. " + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "az storage blob upload \\\n", - " --account-name $STGE_ACCT \\\n", - " --container-name $CTR \\\n", - " --name demo_file \\\n", - " --file demo_file \\\n", - " --auth-mode login" + "az storage blob list --account-name $STGE_ACCT --container-name $CTR --output table --auth-mode login" ] }, { @@ -186,18 +192,43 @@ "Download the file we just uploaded." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "az storage blob download --account-name $STGE_ACCT --container-name $CTR --name demo_file --file demo_file_download file --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In Cloud Shell type the following command:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ls demo_file_download" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify that the downloaded file is the same file we uploaded." + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "az storage blob download \\\n", - " --account-name $STGE_ACCT \\\n", - " --container-name $STGE_ACCT \\\n", - " --name demo_file \\\n", - " --file ~/destination/path/for/file \\\n", - " --auth-mode login" + "diff demo_file demo_file_download" ] }, { @@ -211,6 +242,75 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "We will clean up the resources in this lesson in the reverse order that we created them. \n", + "\n", + "## Blob\n", + "Delete a blob" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage blob delete --account-name $STGE_ACCT --container-name $CTR --name demo_file --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify the blob was deleted" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage blob list --account-name $STGE_ACCT --container-name $CTR --output table --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Storage Account\n", + "Delete the Storage Account." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage account delete -n $STGE_ACCT -g RG" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify that the Storage Account has been deleted." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage account list | grep $STGE_ACCT" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Source Group\n", "By deleting the Resource Group, you will delete all resouce in that group.\n", "\n", "The --no-wait parameter keeps the CLI from blocking while the deletion takes place." @@ -231,7 +331,7 @@ "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49" }, "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -245,7 +345,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.8" + "version": "3.9.7" } }, "nbformat": 4, diff --git a/content/Azure/06_running_analysis.ipynb b/content/Azure/06_running_analysis.ipynb index 2abf7ac..b5ce5ae 100644 --- a/content/Azure/06_running_analysis.ipynb +++ b/content/Azure/06_running_analysis.ipynb @@ -30,7 +30,7 @@ }, { "cell_type": "markdown", - "id": "6291edee-c2df-4941-9b8e-de42649640f9", + "id": "3a0fb40b", "metadata": {}, "source": [ "## Create a VM\n", @@ -47,7 +47,7 @@ }, { "cell_type": "markdown", - "id": "c87e200c-1d18-44f5-b4e6-28b697b04b3c", + "id": "26f414ef", "metadata": {}, "source": [ "## Connect to the VM\n", @@ -61,7 +61,7 @@ }, { "cell_type": "markdown", - "id": "44f32772-0abb-4fa7-a013-202fc646df5e", + "id": "41c471fb", "metadata": {}, "source": [ "```\n", @@ -84,7 +84,7 @@ }, { "cell_type": "markdown", - "id": "0ab6ad28-24bf-4c0f-b4dd-3be2ec25aaa7", + "id": "aa2c0f15", "metadata": {}, "source": [ "## Secure the VM\n", @@ -93,30 +93,9 @@ ] }, { - "cell_type": "code", - "execution_count": 1, - "id": "0f2df994-202e-4e3e-96c2-1536d8adb036", + "cell_type": "markdown", + "id": "d95ece63", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\n", - "Hit:2 http://deb.debian.org/debian buster InRelease\n", - "Hit:3 http://deb.debian.org/debian buster-updates InRelease\n", - "Hit:4 http://deb.debian.org/debian buster-backports InRelease\n", - "Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease\n", - "Hit:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease\n", - "Get:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease [5526 B]\n", - "Fetched 70.9 kB in 1s (91.3 kB/s) \u001b[0m\u001b[33m\n", - "Reading package lists... Done\n", - "Building dependency tree \n", - "Reading state information... Done\n", - "2 packages can be upgraded. Run 'apt list --upgradable' to see them.\n" - ] - } - ], "source": [ "sudo apt update\n", "sudo unattended-upgrades" @@ -124,7 +103,7 @@ }, { "cell_type": "markdown", - "id": "b7c4db9e-f098-41bc-80d2-b524444eec7f", + "id": "da999ac0", "metadata": {}, "source": [ "## Get Example Code\n", @@ -134,30 +113,18 @@ }, { "cell_type": "code", - "execution_count": 2, - "id": "05fadd30-4f35-4a93-b713-cc132131b948", + "execution_count": null, + "id": "0fc21993", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Reading package lists... Done\n", - "Building dependency tree \n", - "Reading state information... Done\n", - "git is already the newest version (1:2.20.1-2+deb10u3).\n", - "0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.\n" - ] - } - ], + "outputs": [], "source": [ "sudo apt-get install --yes git" ] }, { "cell_type": "code", - "execution_count": 3, - "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", + "execution_count": null, + "id": "3d30db18", "metadata": {}, "outputs": [], "source": [ @@ -166,28 +133,17 @@ }, { "cell_type": "code", - "execution_count": 4, - "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", + "execution_count": null, + "id": "4a86fa33", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Cloning into 'CLASS-Examples'...\n", - "remote: Enumerating objects: 66, done.\u001b[K\n", - "remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66\u001b[K\n", - "Unpacking objects: 100% (66/66), done.\n" - ] - } - ], + "outputs": [], "source": [ "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" ] }, { "cell_type": "markdown", - "id": "3d2eb9fc-bec9-449d-a7ab-f3127f59d512", + "id": "5067f044", "metadata": {}, "source": [ "We now change the current directory to the `landsat` directory in the `CLASS-Examples` directory that was just created by the previous git command." @@ -195,8 +151,8 @@ }, { "cell_type": "code", - "execution_count": 5, - "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", + "execution_count": null, + "id": "7510d393", "metadata": {}, "outputs": [], "source": [ @@ -205,7 +161,7 @@ }, { "cell_type": "markdown", - "id": "6ea81d23-e1b0-4cac-a3ac-fed7a9723eca", + "id": "80788411", "metadata": {}, "source": [ "Your prompt should now change showing the current directory as follows.\n", @@ -216,32 +172,17 @@ }, { "cell_type": "code", - "execution_count": 6, - "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", + "execution_count": null, + "id": "6f213bfd", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 28\n", - "-rw-r--r-- 1 learner learner 960 Dec 17 19:00 ReadMe.md\n", - "-rw-r--r-- 1 learner learner 72 Dec 17 19:00 clean.sh\n", - "-rw-r--r-- 1 learner learner 613 Dec 17 19:00 combine.py\n", - "-rw-r--r-- 1 learner learner 280 Dec 17 19:00 download.sh\n", - "-rw-r--r-- 1 learner learner 314 Dec 17 19:00 get-index.sh\n", - "-rw-r--r-- 1 learner learner 76 Dec 17 19:00 search.json\n", - "-rw-r--r-- 1 learner learner 783 Dec 17 19:00 search.py\n" - ] - } - ], + "outputs": [], "source": [ "ls -l" ] }, { "cell_type": "markdown", - "id": "2b8b3144-7453-4350-a1cf-7fa74af2bcbf", + "id": "13168790", "metadata": {}, "source": [ "## Access the bucket\n", @@ -251,36 +192,17 @@ }, { "cell_type": "code", - "execution_count": 7, - "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", + "execution_count": null, + "id": "245f32a2", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://gcp-public-data-landsat/index.csv.gz\n", - "gs://gcp-public-data-landsat/LC08/\n", - "gs://gcp-public-data-landsat/LE07/\n", - "gs://gcp-public-data-landsat/LM01/\n", - "gs://gcp-public-data-landsat/LM02/\n", - "gs://gcp-public-data-landsat/LM03/\n", - "gs://gcp-public-data-landsat/LM04/\n", - "gs://gcp-public-data-landsat/LM05/\n", - "gs://gcp-public-data-landsat/LO08/\n", - "gs://gcp-public-data-landsat/LT04/\n", - "gs://gcp-public-data-landsat/LT05/\n", - "gs://gcp-public-data-landsat/LT08/\n" - ] - } - ], + "outputs": [], "source": [ "gsutil ls gs://gcp-public-data-landsat/" ] }, { "cell_type": "markdown", - "id": "89acffba-cbce-436c-98dd-05467b6675a6", + "id": "9153424d", "metadata": {}, "source": [ "The index file is a list of all the files in the bucket and we can use it to search and filter files.\n", @@ -290,50 +212,27 @@ }, { "cell_type": "code", - "execution_count": 8, - "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", + "execution_count": null, + "id": "e4ffd7e4", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "mkdir: created directory 'data'\n" - ] - } - ], + "outputs": [], "source": [ "mkdir -v data" ] }, { "cell_type": "code", - "execution_count": 9, - "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", + "execution_count": null, + "id": "f90b7dc4", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Copying gs://gcp-public-data-landsat/index.csv.gz...\n", - "==> NOTE: You are downloading one or more large file(s), which would \n", - "run significantly faster if you enabled sliced object downloads. This\n", - "feature is enabled by default but requires that compiled crcmod be\n", - "installed (see \"gsutil help crcmod\").\n", - "\n", - "\\ [1 files][761.0 MiB/761.0 MiB] 42.0 MiB/s \n", - "Operation completed over 1 objects/761.0 MiB. \n" - ] - } - ], + "outputs": [], "source": [ "gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/" ] }, { "cell_type": "markdown", - "id": "76b905b4-1c2a-4960-a14f-974b77f671cd", + "id": "0837cf43", "metadata": {}, "source": [ "We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using." @@ -341,8 +240,8 @@ }, { "cell_type": "code", - "execution_count": 10, - "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", + "execution_count": null, + "id": "9ed5a46b", "metadata": {}, "outputs": [], "source": [ @@ -359,19 +258,10 @@ }, { "cell_type": "code", - "execution_count": 11, - "id": "b005876c-f9af-43d6-80c6-f04295413b9b", + "execution_count": null, + "id": "5f4031b0", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 2.6G\n", - "-rw-r--r-- 1 learner learner 2.6G Dec 17 19:01 index.csv\n" - ] - } - ], + "outputs": [], "source": [ "ls -lh data" ] @@ -386,21 +276,10 @@ }, { "cell_type": "code", - "execution_count": 12, - "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", + "execution_count": null, + "id": "990f4c3f", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL\n", - "LM51940171984262KIS00,LM05_L1GS_194017_19840918_20210913_01_T2,LANDSAT_5,MSS,1984-09-18,01,T2,1984-09-18T09:36:27.0970000Z,L1GS,194,17,51.0,62.58023,60.38359,14.45307,19.13901,34758212,gs://gcp-public-data-landsat/LM05/01/194/017/LM05_L1GS_194017_19840918_20210913_01_T2\n", - "LM10420331972311AAA04,LM01_L1TP_042033_19721106_20180429_01_T2,LANDSAT_1,MSS,1972-11-06,01,T2,1972-11-06T17:49:00.5000000Z,L1TP,42,33,3.0,39.68891,37.707,-115.59896,-112.90344,26932395,gs://gcp-public-data-landsat/LM01/01/042/033/LM01_L1TP_042033_19721106_20180429_01_T2\n", - "LT50150151992246PAC00,LT05_L1TP_015015_19920902_20170121_01_T1,LANDSAT_5,TM,1992-09-02,01,T1,1992-09-02T15:01:34.1390130Z,L1TP,15,15,90.0,65.32118,63.11726,-67.3438,-61.89671,118785510,gs://gcp-public-data-landsat/LT05/01/015/015/LT05_L1TP_015015_19920902_20170121_01_T1\n" - ] - } - ], + "outputs": [], "source": [ "head --lines=4 data/index.csv" ] @@ -417,36 +296,10 @@ }, { "cell_type": "code", - "execution_count": 13, - "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", + "execution_count": null, + "id": "64ddd71e", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "#!/usr/bin/python3\n", - "import json\n", - "import csv\n", - "import sys\n", - "\n", - "# Example: Burr Oak Tree\n", - "# 38.899313,-92.464562 (Lat north+, Long west-) ; Landsat Path 025, Row 033\n", - "config=json.load(open(\"search.json\"))\n", - "lat,lon=config['lat'],config['lon']\n", - "landsat=config['landsat']\n", - "\n", - "reader=csv.reader(sys.stdin)\n", - "header=next(reader) # skip header\n", - "for l in reader:\n", - " SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL=l\n", - " west,east=float(WEST_LON),float(EAST_LON)\n", - " north,south=float(NORTH_LAT),float(SOUTH_LAT)\n", - " if SPACECRAFT_ID==landsat and north >= lat and south <= lat and west <= lon and east >= lon:\n", - " print(BASE_URL) # output BASE_URL\n" - ] - } - ], + "outputs": [], "source": [ "cat search.py" ] @@ -461,22 +314,10 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", + "execution_count": null, + "id": "b7d84cb5", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"lat\": 38.899313,\n", - " \"lon\": -92.464562,\n", - " \"landsat\": \"LANDSAT_8\"\n", - "}\n" - ] - } - ], + "outputs": [], "source": [ "cat search.json" ] @@ -491,22 +332,10 @@ }, { "cell_type": "code", - "execution_count": 15, - "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", + "execution_count": null, + "id": "a201fd0e", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20150127_20170302_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20161028_20170219_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20191224_20200110_01_T1\n" - ] - } - ], + "outputs": [], "source": [ "head --lines=200000 data/index.csv | python3 search.py" ] @@ -523,27 +352,10 @@ }, { "cell_type": "code", - "execution_count": 16, - "id": "3572c518-df83-4906-bfa6-a37bde2a5063", + "execution_count": null, + "id": "3ac78b3c", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "#!/bin/bash\n", - "\n", - "# Read space separated URL from STDIN and download \n", - "while read -r URL ; do\n", - " echo \"+++ $URL\"\n", - " # -m parallel\n", - " # -n no-clobber (do not re-download data)\n", - " # -r recursive (download all the data in the specified URL)\n", - " gsutil -m cp -n -r \"${URL}/\" data/\n", - "done\n" - ] - } - ], + "outputs": [], "source": [ "cat download.sh" ] @@ -558,64 +370,12 @@ }, { "cell_type": "code", - "execution_count": 17, - "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", + "execution_count": null, + "id": "2e93f083", "metadata": { - "scrolled": true, "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_ANG.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B3.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B9.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B8.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B6.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_BQA.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B11.TIF...\n", - "==> NOTE: You are downloading one or more large file(s), which would\n", - "run significantly faster if you enabled sliced object downloads. This\n", - "feature is enabled by default but requires that compiled crcmod be\n", - "installed (see \"gsutil help crcmod\").\n", - "\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B10.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_MTL.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B5.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B2.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B7.TIF...\n", - "| [14/14 files][997.2 MiB/997.2 MiB] 100% Done 65.3 MiB/s ETA 00:00:00 \n", - "Operation completed over 14 objects/997.2 MiB. \n", - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_ANG.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B11.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B10.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B8.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B9.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B3.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_BQA.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B6.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B7.TIF...\n", - "==> NOTE: You are downloading one or more large file(s), which would \n", - "run significantly faster if you enabled sliced object downloads. This\n", - "feature is enabled by default but requires that compiled crcmod be\n", - "installed (see \"gsutil help crcmod\").\n", - "\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B2.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_MTL.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B5.TIF...\n", - "- [14/14 files][979.2 MiB/979.2 MiB] 100% Done 14.9 MiB/s ETA 00:00:00 \n", - "Operation completed over 14 objects/979.2 MiB. \n" - ] - } - ], + "outputs": [], "source": [ "head --lines=200000 data/index.csv | python3 search.py | head --lines=2 | bash download.sh" ] @@ -630,21 +390,10 @@ }, { "cell_type": "code", - "execution_count": 18, - "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", + "execution_count": null, + "id": "271c7c4c", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 2665792\n", - "drwxr-xr-x 2 learner learner 4096 Dec 17 19:02 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20170913_20170928_01_T1\u001b[0m\n", - "drwxr-xr-x 2 learner learner 4096 Dec 17 19:01 \u001b[01;34mLC08_L1TP_025033_20211010_20211018_01_T1\u001b[0m\n", - "-rw-r--r-- 1 learner learner 2729754843 Dec 17 19:01 index.csv\n" - ] - } - ], + "outputs": [], "source": [ "ls -l data" ] @@ -661,57 +410,20 @@ }, { "cell_type": "code", - "execution_count": 19, - "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", + "execution_count": null, + "id": "e41b58c6", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "#!/usr/bin/python3\n", - "import os\n", - "import rasterio\n", - "\n", - "# Open the first directory in data, could walk entire tree\n", - "for dirname, dirs, files in os.walk('data'):\n", - " source = dirs[0]\n", - " break\n", - "\n", - "# Open band (B2/Blue) and copy metadata for result.\n", - "with rasterio.open(\"data/%s/%s_B2.TIF\" % (source, source)) as band2:\n", - " meta = band2.meta\n", - "\n", - "# Combine bands into PNG\n", - "meta.update(count = 3, driver='PNG')\n", - "with rasterio.open(\"output/result-%s.png\" % source, 'w+', **meta) as output:\n", - " for i in range(1, 4):\n", - " print(source, i)\n", - " output.write_band(i,rasterio.open(\"data/%s/%s_B%d.TIF\" % (source, source, i+3)).read(1))\n" - ] - } - ], + "outputs": [], "source": [ "cat combine.py" ] }, { "cell_type": "code", - "execution_count": 20, - "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", + "execution_count": null, + "id": "194c8cef", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Traceback (most recent call last):\n", - " File \"combine.py\", line 3, in \n", - " import rasterio\n", - "ModuleNotFoundError: No module named 'rasterio'\n" - ] - } - ], + "outputs": [], "source": [ "python3 combine.py\n", "/bin/true # ignore this line used for jupyter" @@ -728,471 +440,12 @@ }, { "cell_type": "code", - "execution_count": 21, - "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", + "execution_count": null, + "id": "659ed9d0", "metadata": { - "scrolled": true, "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "mkdir: created directory 'output/'\n", - "Reading package lists... Done\n", - "Building dependency tree \n", - "Reading state information... Done\n", - "The following additional packages will be installed:\n", - " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", - " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", - " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", - " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", - " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", - " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", - " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", - " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", - " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", - " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", - " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", - " python3-chardet python3-click python3-click-plugins python3-cligj\n", - " python3-colorama python3-html5lib python3-lxml python3-numpy\n", - " python3-pyparsing python3-snuggs python3-soupsieve python3-webencodings\n", - "Suggested packages:\n", - " geotiff-bin gdal-bin libgeotiff-epsg libhdf4-doc libhdf4-alt-dev hdf4-tools\n", - " liblcms2-utils libmyodbc odbc-postgresql tdsodbc unixodbc-bin ogdi-bin\n", - " poppler-utils ghostscript fonts-japanese-mincho | fonts-ipafont-mincho\n", - " fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai\n", - " fonts-arphic-uming fonts-nanum python-attr-doc python3-genshi\n", - " python3-lxml-dbg python-lxml-doc gfortran python-numpy-doc python3-pytest\n", - " python3-numpy-dbg python-pyparsing-doc\n", - "The following NEW packages will be installed:\n", - " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", - " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", - " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", - " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", - " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", - " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", - " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", - " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", - " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", - " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", - " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", - " python3-chardet python3-click python3-click-plugins python3-cligj\n", - " python3-colorama python3-html5lib python3-lxml python3-numpy\n", - " python3-pyparsing python3-rasterio python3-snuggs python3-soupsieve\n", - " python3-webencodings\n", - "0 upgraded, 79 newly installed, 0 to remove and 2 not upgraded.\n", - "Need to get 46.8 MB of archives.\n", - "After this operation, 172 MB of additional disk space will be used.\n", - "Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", - "Get:2 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", - "Get:3 http://deb.debian.org/debian buster/main amd64 fonts-dejavu-core all 2.37-1 [1068 kB]\n", - "Get:4 http://deb.debian.org/debian buster/main amd64 fontconfig-config all 2.13.1-2 [280 kB]\n", - "Get:5 http://deb.debian.org/debian buster/main amd64 gdal-data all 2.4.0+dfsg-1 [744 kB]\n", - "Get:6 http://deb.debian.org/debian buster/main amd64 libaec0 amd64 1.0.2-1 [20.1 kB]\n", - "Get:7 http://deb.debian.org/debian buster/main amd64 libgfortran5 amd64 8.3.0-6 [581 kB]\n", - "Get:8 http://deb.debian.org/debian buster/main amd64 libblas3 amd64 3.8.0-2 [148 kB]\n", - "Get:9 http://deb.debian.org/debian buster/main amd64 liblapack3 amd64 3.8.0-2 [2110 kB]\n", - "Get:10 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", - "Get:11 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", - "Get:12 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", - "Get:13 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", - "Get:14 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", - "Get:15 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", - "Get:16 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", - "Get:17 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", - "Get:18 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", - "Get:19 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", - "Get:20 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", - "Get:21 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", - "Get:22 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", - "Get:23 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", - "Get:24 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", - "Get:25 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", - "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", - "Get:27 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", - "Get:28 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", - "Get:29 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", - "Get:30 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", - "Get:31 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", - "Get:32 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", - "Get:33 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", - "Get:34 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", - "Get:35 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", - "Get:36 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", - "Get:37 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", - "Get:38 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", - "Get:39 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", - "Get:40 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", - "Get:41 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", - "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlconvenience1 amd64 1.3.0-7 [50.1 kB]\n", - "Get:43 http://deb.debian.org/debian buster/main amd64 libkmlregionator1 amd64 1.3.0-7 [25.1 kB]\n", - "Get:44 http://deb.debian.org/debian buster/main amd64 libkmlxsd1 amd64 1.3.0-7 [34.6 kB]\n", - "Get:45 http://deb.debian.org/debian buster/main amd64 mysql-common all 5.8+1.0.5 [7324 B]\n", - "Get:46 http://deb.debian.org/debian buster/main amd64 mariadb-common all 1:10.3.31-0+deb10u1 [32.7 kB]\n", - "Get:47 http://deb.debian.org/debian buster/main amd64 libmariadb3 amd64 1:10.3.31-0+deb10u1 [173 kB]\n", - "Get:48 http://deb.debian.org/debian buster/main amd64 libnetcdf13 amd64 1:4.6.2-1 [403 kB]\n", - "Get:49 http://deb.debian.org/debian buster/main amd64 libltdl7 amd64 2.4.6-9 [390 kB]\n", - "Get:50 http://deb.debian.org/debian buster/main amd64 libodbc1 amd64 2.3.6-0.1 [223 kB]\n", - "Get:51 http://deb.debian.org/debian buster/main amd64 libogdi3.2 amd64 3.2.1+ds-4 [241 kB]\n", - "Get:52 http://deb.debian.org/debian buster/main amd64 libopenjp2-7 amd64 2.3.0-2+deb10u2 [158 kB]\n", - "Get:53 http://deb.debian.org/debian buster/main amd64 liblcms2-2 amd64 2.9-3 [145 kB]\n", - "Get:54 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]\n", - "Get:55 http://deb.debian.org/debian buster/main amd64 libpoppler82 amd64 0.71.0-5 [1507 kB]\n", - "Get:56 http://deb.debian.org/debian buster/main amd64 libqhull7 amd64 2015.2-4 [196 kB]\n", - "Get:57 http://deb.debian.org/debian buster/main amd64 libspatialite7 amd64 4.3.0a-5+b2 [1256 kB]\n", - "Get:58 http://deb.debian.org/debian buster/main amd64 libxerces-c3.2 amd64 3.2.2+debian-1+deb10u1 [886 kB]\n", - "Get:59 http://deb.debian.org/debian buster/main amd64 odbcinst amd64 2.3.6-0.1 [48.4 kB]\n", - "Get:60 http://deb.debian.org/debian buster/main amd64 odbcinst1debian2 amd64 2.3.6-0.1 [78.1 kB]\n", - "Get:61 http://deb.debian.org/debian buster/main amd64 libgdal20 amd64 2.4.0+dfsg-1+b1 [6171 kB]\n", - "Get:62 http://deb.debian.org/debian buster/main amd64 libxslt1.1 amd64 1.1.32-2.2~deb10u1 [237 kB]\n", - "Get:63 http://deb.debian.org/debian buster/main amd64 proj-bin amd64 5.2.0-1 [105 kB]\n", - "Get:64 http://deb.debian.org/debian buster/main amd64 python3-affine all 2.2.2-1 [14.5 kB]\n", - "Get:65 http://deb.debian.org/debian buster/main amd64 python3-attr all 18.2.0-1 [37.3 kB]\n", - "Get:66 http://deb.debian.org/debian buster/main amd64 python3-soupsieve all 1.8+dfsg-1 [27.6 kB]\n", - "Get:67 http://deb.debian.org/debian buster/main amd64 python3-bs4 all 4.7.1-1 [94.1 kB]\n", - "Get:68 http://deb.debian.org/debian buster/main amd64 python3-chardet all 3.0.4-3 [80.5 kB]\n", - "Get:69 http://deb.debian.org/debian buster/main amd64 python3-colorama all 0.3.7-1 [18.1 kB]\n", - "Get:70 http://deb.debian.org/debian buster/main amd64 python3-click all 7.0-1 [73.6 kB]\n", - "Get:71 http://deb.debian.org/debian buster/main amd64 python3-click-plugins all 1.0.4-1 [7540 B]\n", - "Get:72 http://deb.debian.org/debian buster/main amd64 python3-cligj all 0.5.0-1 [8212 B]\n", - "Get:73 http://deb.debian.org/debian buster/main amd64 python3-webencodings all 0.5.1-1 [10.9 kB]\n", - "Get:74 http://deb.debian.org/debian buster/main amd64 python3-html5lib all 1.0.1-1 [89.5 kB]\n", - "Get:75 http://deb.debian.org/debian buster/main amd64 python3-lxml amd64 4.3.2-1+deb10u3 [1162 kB]\n", - "Get:76 http://deb.debian.org/debian buster/main amd64 python3-numpy amd64 1:1.16.2-1 [2119 kB]\n", - "Get:77 http://deb.debian.org/debian buster/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [89.6 kB]\n", - "Get:78 http://deb.debian.org/debian buster/main amd64 python3-snuggs all 1.4.3-1 [7228 B]\n", - "Get:79 http://deb.debian.org/debian buster/main amd64 python3-rasterio amd64 1.0.21-1 [818 kB]\n", - "Fetched 46.8 MB in 1s (85.4 MB/s) \n", - "Extracting templates from packages: 100%\n", - "Preconfiguring packages ...\n", - "Selecting previously unselected package poppler-data.\n", - "(Reading database ... 57529 files and directories currently installed.)\n", - "Preparing to unpack .../00-poppler-data_0.4.9-2_all.deb ...\n", - "Unpacking poppler-data (0.4.9-2) ...\n", - "Selecting previously unselected package fonts-dejavu-core.\n", - "Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ...\n", - "Unpacking fonts-dejavu-core (2.37-1) ...\n", - "Selecting previously unselected package fontconfig-config.\n", - "Preparing to unpack .../02-fontconfig-config_2.13.1-2_all.deb ...\n", - "Unpacking fontconfig-config (2.13.1-2) ...\n", - "Selecting previously unselected package gdal-data.\n", - "Preparing to unpack .../03-gdal-data_2.4.0+dfsg-1_all.deb ...\n", - "Unpacking gdal-data (2.4.0+dfsg-1) ...\n", - "Selecting previously unselected package libaec0:amd64.\n", - "Preparing to unpack .../04-libaec0_1.0.2-1_amd64.deb ...\n", - "Unpacking libaec0:amd64 (1.0.2-1) ...\n", - "Selecting previously unselected package libgfortran5:amd64.\n", - "Preparing to unpack .../05-libgfortran5_8.3.0-6_amd64.deb ...\n", - "Unpacking libgfortran5:amd64 (8.3.0-6) ...\n", - "Selecting previously unselected package libblas3:amd64.\n", - "Preparing to unpack .../06-libblas3_3.8.0-2_amd64.deb ...\n", - "Unpacking libblas3:amd64 (3.8.0-2) ...\n", - "Selecting previously unselected package liblapack3:amd64.\n", - "Preparing to unpack .../07-liblapack3_3.8.0-2_amd64.deb ...\n", - "Unpacking liblapack3:amd64 (3.8.0-2) ...\n", - "Selecting previously unselected package libarpack2:amd64.\n", - "Preparing to unpack .../08-libarpack2_3.7.0-2_amd64.deb ...\n", - "Unpacking libarpack2:amd64 (3.7.0-2) ...\n", - "Selecting previously unselected package libsuperlu5:amd64.\n", - "Preparing to unpack .../09-libsuperlu5_5.2.1+dfsg1-4_amd64.deb ...\n", - "Unpacking libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", - "Selecting previously unselected package libarmadillo9.\n", - "Preparing to unpack .../10-libarmadillo9_1%3a9.200.7+dfsg-1_amd64.deb ...\n", - "Unpacking libarmadillo9 (1:9.200.7+dfsg-1) ...\n", - "Selecting previously unselected package libcharls2:amd64.\n", - "Preparing to unpack .../11-libcharls2_2.0.0+dfsg-1_amd64.deb ...\n", - "Unpacking libcharls2:amd64 (2.0.0+dfsg-1) ...\n", - "Selecting previously unselected package libicu63:amd64.\n", - "Preparing to unpack .../12-libicu63_63.1-6+deb10u2_amd64.deb ...\n", - "Unpacking libicu63:amd64 (63.1-6+deb10u2) ...\n", - "Selecting previously unselected package libxml2:amd64.\n", - "Preparing to unpack .../13-libxml2_2.9.4+dfsg1-7+deb10u2_amd64.deb ...\n", - "Unpacking libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", - "Selecting previously unselected package libdap25:amd64.\n", - "Preparing to unpack .../14-libdap25_3.20.3-1_amd64.deb ...\n", - "Unpacking libdap25:amd64 (3.20.3-1) ...\n", - "Selecting previously unselected package libdapclient6v5:amd64.\n", - "Preparing to unpack .../15-libdapclient6v5_3.20.3-1_amd64.deb ...\n", - "Unpacking libdapclient6v5:amd64 (3.20.3-1) ...\n", - "Selecting previously unselected package libdapserver7v5:amd64.\n", - "Preparing to unpack .../16-libdapserver7v5_3.20.3-1_amd64.deb ...\n", - "Unpacking libdapserver7v5:amd64 (3.20.3-1) ...\n", - "Selecting previously unselected package libepsilon1:amd64.\n", - "Preparing to unpack .../17-libepsilon1_0.9.2+dfsg-4_amd64.deb ...\n", - "Unpacking libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", - "Selecting previously unselected package libfontconfig1:amd64.\n", - "Preparing to unpack .../18-libfontconfig1_2.13.1-2_amd64.deb ...\n", - "Unpacking libfontconfig1:amd64 (2.13.1-2) ...\n", - "Selecting previously unselected package libfreexl1:amd64.\n", - "Preparing to unpack .../19-libfreexl1_1.0.5-3_amd64.deb ...\n", - "Unpacking libfreexl1:amd64 (1.0.5-3) ...\n", - "Selecting previously unselected package libfyba0:amd64.\n", - "Preparing to unpack .../20-libfyba0_4.1.1-6_amd64.deb ...\n", - "Unpacking libfyba0:amd64 (4.1.1-6) ...\n", - "Selecting previously unselected package libgeos-3.7.1:amd64.\n", - "Preparing to unpack .../21-libgeos-3.7.1_3.7.1-1_amd64.deb ...\n", - "Unpacking libgeos-3.7.1:amd64 (3.7.1-1) ...\n", - "Selecting previously unselected package libgeos-c1v5:amd64.\n", - "Preparing to unpack .../22-libgeos-c1v5_3.7.1-1_amd64.deb ...\n", - "Unpacking libgeos-c1v5:amd64 (3.7.1-1) ...\n", - "Selecting previously unselected package proj-data.\n", - "Preparing to unpack .../23-proj-data_5.2.0-1_all.deb ...\n", - "Unpacking proj-data (5.2.0-1) ...\n", - "Selecting previously unselected package libproj13:amd64.\n", - "Preparing to unpack .../24-libproj13_5.2.0-1_amd64.deb ...\n", - "Unpacking libproj13:amd64 (5.2.0-1) ...\n", - "Selecting previously unselected package libjbig0:amd64.\n", - "Preparing to unpack .../25-libjbig0_2.1-3.1+b2_amd64.deb ...\n", - "Unpacking libjbig0:amd64 (2.1-3.1+b2) ...\n", - "Selecting previously unselected package libjpeg62-turbo:amd64.\n", - "Preparing to unpack .../26-libjpeg62-turbo_1%3a1.5.2-2+deb10u1_amd64.deb ...\n", - "Unpacking libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", - "Selecting previously unselected package libwebp6:amd64.\n", - "Preparing to unpack .../27-libwebp6_0.6.1-2+deb10u1_amd64.deb ...\n", - "Unpacking libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", - "Selecting previously unselected package libtiff5:amd64.\n", - "Preparing to unpack .../28-libtiff5_4.1.0+git191117-2~deb10u3_amd64.deb ...\n", - "Unpacking libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", - "Selecting previously unselected package libgeotiff2:amd64.\n", - "Preparing to unpack .../29-libgeotiff2_1.4.3-1_amd64.deb ...\n", - "Unpacking libgeotiff2:amd64 (1.4.3-1) ...\n", - "Selecting previously unselected package libgif7:amd64.\n", - "Preparing to unpack .../30-libgif7_5.1.4-3_amd64.deb ...\n", - "Unpacking libgif7:amd64 (5.1.4-3) ...\n", - "Selecting previously unselected package libhdf4-0-alt.\n", - "Preparing to unpack .../31-libhdf4-0-alt_4.2.13-4_amd64.deb ...\n", - "Unpacking libhdf4-0-alt (4.2.13-4) ...\n", - "Selecting previously unselected package libsz2:amd64.\n", - "Preparing to unpack .../32-libsz2_1.0.2-1_amd64.deb ...\n", - "Unpacking libsz2:amd64 (1.0.2-1) ...\n", - "Selecting previously unselected package libhdf5-103:amd64.\n", - "Preparing to unpack .../33-libhdf5-103_1.10.4+repack-10_amd64.deb ...\n", - "Unpacking libhdf5-103:amd64 (1.10.4+repack-10) ...\n", - "Selecting previously unselected package libminizip1:amd64.\n", - "Preparing to unpack .../34-libminizip1_1.1-8+b1_amd64.deb ...\n", - "Unpacking libminizip1:amd64 (1.1-8+b1) ...\n", - "Selecting previously unselected package liburiparser1:amd64.\n", - "Preparing to unpack .../35-liburiparser1_0.9.1-1_amd64.deb ...\n", - "Unpacking liburiparser1:amd64 (0.9.1-1) ...\n", - "Selecting previously unselected package libkmlbase1:amd64.\n", - "Preparing to unpack .../36-libkmlbase1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlbase1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmldom1:amd64.\n", - "Preparing to unpack .../37-libkmldom1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmldom1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlengine1:amd64.\n", - "Preparing to unpack .../38-libkmlengine1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlengine1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlconvenience1:amd64.\n", - "Preparing to unpack .../39-libkmlconvenience1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlconvenience1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlregionator1:amd64.\n", - "Preparing to unpack .../40-libkmlregionator1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlregionator1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlxsd1:amd64.\n", - "Preparing to unpack .../41-libkmlxsd1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlxsd1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package mysql-common.\n", - "Preparing to unpack .../42-mysql-common_5.8+1.0.5_all.deb ...\n", - "Unpacking mysql-common (5.8+1.0.5) ...\n", - "Selecting previously unselected package mariadb-common.\n", - "Preparing to unpack .../43-mariadb-common_1%3a10.3.31-0+deb10u1_all.deb ...\n", - "Unpacking mariadb-common (1:10.3.31-0+deb10u1) ...\n", - "Selecting previously unselected package libmariadb3:amd64.\n", - "Preparing to unpack .../44-libmariadb3_1%3a10.3.31-0+deb10u1_amd64.deb ...\n", - "Unpacking libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", - "Selecting previously unselected package libnetcdf13:amd64.\n", - "Preparing to unpack .../45-libnetcdf13_1%3a4.6.2-1_amd64.deb ...\n", - "Unpacking libnetcdf13:amd64 (1:4.6.2-1) ...\n", - "Selecting previously unselected package libltdl7:amd64.\n", - "Preparing to unpack .../46-libltdl7_2.4.6-9_amd64.deb ...\n", - "Unpacking libltdl7:amd64 (2.4.6-9) ...\n", - "Selecting previously unselected package libodbc1:amd64.\n", - "Preparing to unpack .../47-libodbc1_2.3.6-0.1_amd64.deb ...\n", - "Unpacking libodbc1:amd64 (2.3.6-0.1) ...\n", - "Selecting previously unselected package libogdi3.2.\n", - "Preparing to unpack .../48-libogdi3.2_3.2.1+ds-4_amd64.deb ...\n", - "Unpacking libogdi3.2 (3.2.1+ds-4) ...\n", - "Selecting previously unselected package libopenjp2-7:amd64.\n", - "Preparing to unpack .../49-libopenjp2-7_2.3.0-2+deb10u2_amd64.deb ...\n", - "Unpacking libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", - "Selecting previously unselected package liblcms2-2:amd64.\n", - "Preparing to unpack .../50-liblcms2-2_2.9-3_amd64.deb ...\n", - "Unpacking liblcms2-2:amd64 (2.9-3) ...\n", - "Selecting previously unselected package libnspr4:amd64.\n", - "Preparing to unpack .../51-libnspr4_2%3a4.20-1_amd64.deb ...\n", - "Unpacking libnspr4:amd64 (2:4.20-1) ...\n", - "Selecting previously unselected package libnss3:amd64.\n", - "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u4_amd64.deb ...\n", - "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", - "Selecting previously unselected package libpoppler82:amd64.\n", - "Preparing to unpack .../53-libpoppler82_0.71.0-5_amd64.deb ...\n", - "Unpacking libpoppler82:amd64 (0.71.0-5) ...\n", - "Selecting previously unselected package libpq5:amd64.\n", - "Preparing to unpack .../54-libpq5_11.14-0+deb10u1_amd64.deb ...\n", - "Unpacking libpq5:amd64 (11.14-0+deb10u1) ...\n", - "Selecting previously unselected package libqhull7:amd64.\n", - "Preparing to unpack .../55-libqhull7_2015.2-4_amd64.deb ...\n", - "Unpacking libqhull7:amd64 (2015.2-4) ...\n", - "Selecting previously unselected package libspatialite7:amd64.\n", - "Preparing to unpack .../56-libspatialite7_4.3.0a-5+b2_amd64.deb ...\n", - "Unpacking libspatialite7:amd64 (4.3.0a-5+b2) ...\n", - "Selecting previously unselected package libxerces-c3.2:amd64.\n", - "Preparing to unpack .../57-libxerces-c3.2_3.2.2+debian-1+deb10u1_amd64.deb ...\n", - "Unpacking libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", - "Selecting previously unselected package odbcinst.\n", - "Preparing to unpack .../58-odbcinst_2.3.6-0.1_amd64.deb ...\n", - "Unpacking odbcinst (2.3.6-0.1) ...\n", - "Selecting previously unselected package odbcinst1debian2:amd64.\n", - "Preparing to unpack .../59-odbcinst1debian2_2.3.6-0.1_amd64.deb ...\n", - "Unpacking odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", - "Selecting previously unselected package libgdal20.\n", - "Preparing to unpack .../60-libgdal20_2.4.0+dfsg-1+b1_amd64.deb ...\n", - "Unpacking libgdal20 (2.4.0+dfsg-1+b1) ...\n", - "Selecting previously unselected package libxslt1.1:amd64.\n", - "Preparing to unpack .../61-libxslt1.1_1.1.32-2.2~deb10u1_amd64.deb ...\n", - "Unpacking libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", - "Selecting previously unselected package proj-bin.\n", - "Preparing to unpack .../62-proj-bin_5.2.0-1_amd64.deb ...\n", - "Unpacking proj-bin (5.2.0-1) ...\n", - "Selecting previously unselected package python3-affine.\n", - "Preparing to unpack .../63-python3-affine_2.2.2-1_all.deb ...\n", - "Unpacking python3-affine (2.2.2-1) ...\n", - "Selecting previously unselected package python3-attr.\n", - "Preparing to unpack .../64-python3-attr_18.2.0-1_all.deb ...\n", - "Unpacking python3-attr (18.2.0-1) ...\n", - "Selecting previously unselected package python3-soupsieve.\n", - "Preparing to unpack .../65-python3-soupsieve_1.8+dfsg-1_all.deb ...\n", - "Unpacking python3-soupsieve (1.8+dfsg-1) ...\n", - "Selecting previously unselected package python3-bs4.\n", - "Preparing to unpack .../66-python3-bs4_4.7.1-1_all.deb ...\n", - "Unpacking python3-bs4 (4.7.1-1) ...\n", - "Selecting previously unselected package python3-chardet.\n", - "Preparing to unpack .../67-python3-chardet_3.0.4-3_all.deb ...\n", - "Unpacking python3-chardet (3.0.4-3) ...\n", - "Selecting previously unselected package python3-colorama.\n", - "Preparing to unpack .../68-python3-colorama_0.3.7-1_all.deb ...\n", - "Unpacking python3-colorama (0.3.7-1) ...\n", - "Selecting previously unselected package python3-click.\n", - "Preparing to unpack .../69-python3-click_7.0-1_all.deb ...\n", - "Unpacking python3-click (7.0-1) ...\n", - "Selecting previously unselected package python3-click-plugins.\n", - "Preparing to unpack .../70-python3-click-plugins_1.0.4-1_all.deb ...\n", - "Unpacking python3-click-plugins (1.0.4-1) ...\n", - "Selecting previously unselected package python3-cligj.\n", - "Preparing to unpack .../71-python3-cligj_0.5.0-1_all.deb ...\n", - "Unpacking python3-cligj (0.5.0-1) ...\n", - "Selecting previously unselected package python3-webencodings.\n", - "Preparing to unpack .../72-python3-webencodings_0.5.1-1_all.deb ...\n", - "Unpacking python3-webencodings (0.5.1-1) ...\n", - "Selecting previously unselected package python3-html5lib.\n", - "Preparing to unpack .../73-python3-html5lib_1.0.1-1_all.deb ...\n", - "Unpacking python3-html5lib (1.0.1-1) ...\n", - "Selecting previously unselected package python3-lxml:amd64.\n", - "Preparing to unpack .../74-python3-lxml_4.3.2-1+deb10u3_amd64.deb ...\n", - "Unpacking python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", - "Selecting previously unselected package python3-numpy.\n", - "Preparing to unpack .../75-python3-numpy_1%3a1.16.2-1_amd64.deb ...\n", - "Unpacking python3-numpy (1:1.16.2-1) ...\n", - "Selecting previously unselected package python3-pyparsing.\n", - "Preparing to unpack .../76-python3-pyparsing_2.2.0+dfsg1-2_all.deb ...\n", - "Unpacking python3-pyparsing (2.2.0+dfsg1-2) ...\n", - "Selecting previously unselected package python3-snuggs.\n", - "Preparing to unpack .../77-python3-snuggs_1.4.3-1_all.deb ...\n", - "Unpacking python3-snuggs (1.4.3-1) ...\n", - "Selecting previously unselected package python3-rasterio.\n", - "Preparing to unpack .../78-python3-rasterio_1.0.21-1_amd64.deb ...\n", - "Unpacking python3-rasterio (1.0.21-1) ...\n", - "Setting up liblcms2-2:amd64 (2.9-3) ...\n", - "Setting up python3-attr (18.2.0-1) ...\n", - "Setting up mysql-common (5.8+1.0.5) ...\n", - "update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", - "Setting up proj-data (5.2.0-1) ...\n", - "Setting up python3-colorama (0.3.7-1) ...\n", - "Setting up libcharls2:amd64 (2.0.0+dfsg-1) ...\n", - "Setting up libminizip1:amd64 (1.1-8+b1) ...\n", - "Setting up libproj13:amd64 (5.2.0-1) ...\n", - "Setting up libpq5:amd64 (11.14-0+deb10u1) ...\n", - "Setting up proj-bin (5.2.0-1) ...\n", - "Setting up libqhull7:amd64 (2015.2-4) ...\n", - "Setting up python3-click (7.0-1) ...\n", - "Setting up libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", - "Setting up libjbig0:amd64 (2.1-3.1+b2) ...\n", - "Setting up python3-webencodings (0.5.1-1) ...\n", - "Setting up libgeos-3.7.1:amd64 (3.7.1-1) ...\n", - "Setting up libaec0:amd64 (1.0.2-1) ...\n", - "Setting up libicu63:amd64 (63.1-6+deb10u2) ...\n", - "Setting up gdal-data (2.4.0+dfsg-1) ...\n", - "Setting up poppler-data (0.4.9-2) ...\n", - "Setting up mariadb-common (1:10.3.31-0+deb10u1) ...\n", - "update-alternatives: using /etc/mysql/mariadb.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", - "Setting up python3-chardet (3.0.4-3) ...\n", - "Setting up libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", - "Setting up python3-pyparsing (2.2.0+dfsg1-2) ...\n", - "Setting up libnspr4:amd64 (2:4.20-1) ...\n", - "Setting up libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", - "Setting up libgeos-c1v5:amd64 (3.7.1-1) ...\n", - "Setting up python3-click-plugins (1.0.4-1) ...\n", - "Setting up libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", - "Setting up fonts-dejavu-core (2.37-1) ...\n", - "Setting up python3-html5lib (1.0.1-1) ...\n", - "Setting up libltdl7:amd64 (2.4.6-9) ...\n", - "Setting up libgfortran5:amd64 (8.3.0-6) ...\n", - "Setting up libhdf4-0-alt (4.2.13-4) ...\n", - "Setting up libgif7:amd64 (5.1.4-3) ...\n", - "Setting up liburiparser1:amd64 (0.9.1-1) ...\n", - "Setting up libfreexl1:amd64 (1.0.5-3) ...\n", - "Setting up libfyba0:amd64 (4.1.1-6) ...\n", - "Setting up libkmlbase1:amd64 (1.3.0-7) ...\n", - "Setting up libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", - "Setting up libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", - "Setting up python3-affine (2.2.2-1) ...\n", - "Setting up libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", - "Setting up python3-soupsieve (1.8+dfsg-1) ...\n", - "Setting up libsz2:amd64 (1.0.2-1) ...\n", - "Setting up libkmlxsd1:amd64 (1.3.0-7) ...\n", - "Setting up libkmldom1:amd64 (1.3.0-7) ...\n", - "Setting up libspatialite7:amd64 (4.3.0a-5+b2) ...\n", - "Setting up libogdi3.2 (3.2.1+ds-4) ...\n", - "Setting up libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", - "Setting up libkmlengine1:amd64 (1.3.0-7) ...\n", - "Setting up libkmlconvenience1:amd64 (1.3.0-7) ...\n", - "Setting up fontconfig-config (2.13.1-2) ...\n", - "Setting up python3-cligj (0.5.0-1) ...\n", - "Setting up libdap25:amd64 (3.20.3-1) ...\n", - "Setting up libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", - "Setting up libdapserver7v5:amd64 (3.20.3-1) ...\n", - "Setting up python3-bs4 (4.7.1-1) ...\n", - "Setting up libgeotiff2:amd64 (1.4.3-1) ...\n", - "Setting up libblas3:amd64 (3.8.0-2) ...\n", - "update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode\n", - "Setting up libkmlregionator1:amd64 (1.3.0-7) ...\n", - "Setting up libhdf5-103:amd64 (1.10.4+repack-10) ...\n", - "Setting up libodbc1:amd64 (2.3.6-0.1) ...\n", - "Setting up libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", - "Setting up libdapclient6v5:amd64 (3.20.3-1) ...\n", - "Setting up libfontconfig1:amd64 (2.13.1-2) ...\n", - "Setting up liblapack3:amd64 (3.8.0-2) ...\n", - "update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3 to provide /usr/lib/x86_64-linux-gnu/liblapack.so.3 (liblapack.so.3-x86_64-linux-gnu) in auto mode\n", - "Setting up libarpack2:amd64 (3.7.0-2) ...\n", - "Setting up libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", - "Setting up libpoppler82:amd64 (0.71.0-5) ...\n", - "Setting up libnetcdf13:amd64 (1:4.6.2-1) ...\n", - "Setting up python3-numpy (1:1.16.2-1) ...\n", - "Setting up python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", - "Setting up libarmadillo9 (1:9.200.7+dfsg-1) ...\n", - "Setting up python3-snuggs (1.4.3-1) ...\n", - "Setting up odbcinst (2.3.6-0.1) ...\n", - "Setting up odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", - "Setting up libgdal20 (2.4.0+dfsg-1+b1) ...\n", - "Setting up python3-rasterio (1.0.21-1) ...\n", - "Processing triggers for libc-bin (2.28-10) ...\n", - "Processing triggers for man-db (2.8.5-2) ...\n" - ] - } - ], + "outputs": [], "source": [ "mkdir -v output/\n", "sudo apt-get install python3-rasterio --yes" @@ -1200,21 +453,10 @@ }, { "cell_type": "code", - "execution_count": 22, - "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", + "execution_count": null, + "id": "45a25550", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "LC08_L1TP_025033_20211010_20211018_01_T1 1\n", - "LC08_L1TP_025033_20211010_20211018_01_T1 2\n", - "LC08_L1TP_025033_20211010_20211018_01_T1 3\n", - "ERROR 4: output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png: No such file or directory\n" - ] - } - ], + "outputs": [], "source": [ "/usr/bin/python3 combine.py" ] @@ -1237,20 +479,10 @@ }, { "cell_type": "code", - "execution_count": 23, - "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", + "execution_count": null, + "id": "7f3f6f0b", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 195M\n", - "-rw-r--r-- 1 learner learner 195M Dec 17 19:03 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20211010_20211018_01_T1.png\u001b[0m\n", - "-rw-r--r-- 1 learner learner 910 Dec 17 19:03 result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n" - ] - } - ], + "outputs": [], "source": [ "ls -lh output" ] @@ -1278,18 +510,10 @@ }, { "cell_type": "code", - "execution_count": 24, - "id": "f36cb8c5-f305-4cb2-a5cc-0c9fd8592fb4", + "execution_count": null, + "id": "715291b5", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "bucket: essentials-learner-2021-12-17 region: us-west2\n" - ] - } - ], + "outputs": [], "source": [ "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", "REGION=\"us-west2\"\n", @@ -1297,19 +521,9 @@ ] }, { - "cell_type": "code", - "execution_count": 25, - "id": "c2ae2b74-5e93-4c55-8bd7-63337f7dcbb8", + "cell_type": "markdown", + "id": "920ff5ef", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Creating gs://essentials-learner-2021-12-17/...\n" - ] - } - ], "source": [ "gsutil mb -b on -l $REGION --pap enforced \"gs://$BUCKET\"" ] @@ -1324,33 +538,10 @@ }, { "cell_type": "code", - "execution_count": 26, - "id": "681e6b1d-98bb-448a-a57e-f5674214effd", + "execution_count": null, + "id": "9340b10d", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml [Content-Type=application/xml]...\n", - "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png [Content-Type=image/png]...\n", - "==> NOTE: You are uploading one or more large file(s), which would run \n", - "significantly faster if you enable parallel composite uploads. This\n", - "feature can be enabled by editing the\n", - "\"parallel_composite_upload_threshold\" value in your .boto\n", - "configuration file. However, note that if you do this large files will\n", - "be uploaded as `composite objects\n", - "`_,which\n", - "means that any user who downloads such objects will need to have a\n", - "compiled crcmod installed (see \"gsutil help crcmod\"). This is because\n", - "without a compiled crcmod, computing checksums on composite objects is\n", - "so slow that gsutil disables downloads of composite objects.\n", - "\n", - "\\ [2/2 files][194.0 MiB/194.0 MiB] 100% Done \n", - "Operation completed over 2 objects/194.0 MiB. \n" - ] - } - ], + "outputs": [], "source": [ "gsutil -m cp -r \"output\" \"gs://$BUCKET\"" ] @@ -1365,38 +556,20 @@ }, { "cell_type": "code", - "execution_count": 27, - "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", + "execution_count": null, + "id": "754c7eb4", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://essentials-learner-2021-12-17/output/\n" - ] - } - ], + "outputs": [], "source": [ "gsutil ls \"gs://$BUCKET\"" ] }, { "cell_type": "code", - "execution_count": 28, - "id": "b1ea18e9-5861-4479-9948-3303952dee8a", + "execution_count": null, + "id": "0e927a0a", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "194.05 MiB 2021-12-17T19:03:54Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", - " 910 B 2021-12-17T19:03:52Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", - "TOTAL: 2 objects, 203478494 bytes (194.05 MiB)\n" - ] - } - ], + "outputs": [], "source": [ "gsutil ls -lh \"gs://$BUCKET/output\"" ] @@ -1425,7 +598,7 @@ }, { "cell_type": "markdown", - "id": "50990a67-f901-48fd-9295-25ef73245142", + "id": "85237d4a", "metadata": { "tags": [] }, @@ -1435,7 +608,7 @@ " * In the \"New Principals\" box add the Identity for the collaborator (another individual) as directed by the instructor.\n", " * Select the \"Storage Object Viewer\" by typing \"Storage Object Viewer\" in the filter and then selecting \"Storage Object Viewer\". Do not use any \"Legacy Storage\" roles.\n", " * Click \"Save\" to save the policy.\n", - " ![iam-storage-object-viewer](img/iam-storage-object-viewer.png)\n", + " \n", " * Verify the policy is listed in the \"Permissions\" table on the \"Bucket Details\" page (you should now be on this page).\n", "\n", "*Advanced Note: Changes in permissions will show up on the **Home** -> **Activity** page.*" diff --git a/content/Azure/08_cleaning_up_resources.ipynb b/content/Azure/08_cleaning_up_resources.ipynb index 6b11af4..0494252 100644 --- a/content/Azure/08_cleaning_up_resources.ipynb +++ b/content/Azure/08_cleaning_up_resources.ipynb @@ -35,19 +35,9 @@ ] }, { - "cell_type": "code", - "execution_count": 7, - "id": "7a134850-8c4f-47c6-9816-6d487a88eda1", + "cell_type": "markdown", + "id": "7171ff08", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "rg-essentials\n" - ] - } - ], "source": [ "RG=rg-essentials\n", "echo $RG" @@ -62,20 +52,9 @@ ] }, { - "cell_type": "code", - "execution_count": 8, - "id": "6ccd79ab-0068-4a2a-9aca-2c47454f9d85", + "cell_type": "markdown", + "id": "b5d3261e", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "\u001b[0m" - ] - } - ], "source": [ "az vm list --resource-group $RG --output table" ] @@ -91,19 +70,9 @@ ] }, { - "cell_type": "code", - "execution_count": 9, - "id": "3ff9bb93-aadf-4319-aed4-b5fc1f2518c8", + "cell_type": "markdown", + "id": "92c8ae09", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[K\u001b[0minished .." - ] - } - ], "source": [ "az group delete --resource-group $RG --yes" ] diff --git a/content/Azure/glossary.ipynb b/content/Azure/glossary.ipynb index e690bf9..f7501e9 100644 --- a/content/Azure/glossary.ipynb +++ b/content/Azure/glossary.ipynb @@ -2,31 +2,43 @@ "cells": [ { "cell_type": "markdown", - "id": "1f0027c2-df7b-426e-b4f4-3e1db9916d14", + "id": "efd09804", "metadata": {}, "source": [ "# Glossary\n", "\n", - "* **Account** - The Google account (email) associated with an individual. This is the identity that you use to access console and takes the form of an email address.\n", - "* **Project** - Resources and services in Google Cloud are created within a single Project. \n", - "* **Project Name** - The long human readable name of your project. You can change this.\n", - "* **Project ID** - an alpha-numeric string identifier for the project. This is not changeable.\n", - "* **Project Number** - a long number that also identifies the project but is rarely used (it is an internal identifier). \n", - "* **Billing Account** - an account used to pay for Google Cloud services. A Billing Account is associated with a Project and enables resource allocation. A project has only one Billing Account associated with it. A Billing account can be used for multiple Projects." + "* **Active Directory** - This is were your identity is defined. This is typically setup by the institution that you belong to.\n", + "* **Subscription** - Resources and services in Azure are created within a Subsrciption and this acts as your billing account. \n", + "* **Account** - The Azure account (email) associated with an individual. This is the identity that you use to access console.\n", + "* **Resource Group** - A resource group is a *container* (in the literal sense not to be confused with a Docker container for example) that holds related resources that you want to manage as a group." ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "471d2fe5", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.7" } }, "nbformat": 4, diff --git a/content/Azure/images/Azure-cloud-shell.png b/content/Azure/images/Azure-cloud-shell.png new file mode 100644 index 0000000..5d86347 Binary files /dev/null and b/content/Azure/images/Azure-cloud-shell.png differ diff --git a/content/Azure/images/Azure-services-menu.png b/content/Azure/images/Azure-services-menu.png new file mode 100644 index 0000000..416acdc Binary files /dev/null and b/content/Azure/images/Azure-services-menu.png differ diff --git a/content/Azure/images/hamburger-navigation.png b/content/Azure/images/hamburger-navigation.png new file mode 100644 index 0000000..41423f2 Binary files /dev/null and b/content/Azure/images/hamburger-navigation.png differ diff --git a/content/Azure/intro_to_compute_cli.ipynb b/content/Azure/intro_to_compute_cli.ipynb deleted file mode 100644 index 0bb8ebf..0000000 --- a/content/Azure/intro_to_compute_cli.ipynb +++ /dev/null @@ -1,260 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Intro to Compute CLI\n", - "\n", - "**Overview**\n", - " \n", - "**Teaching:**\n", - "\n", - "**Exercises:**\n", - "\n", - "**Questions:**\n", - " * How do I create my own computer in the cloud?\n", - "\n", - "**Objectives:**\n", - " * Allocate a virtual machine in Azure.\n", - " * Learn how to connect to a virtual machine using ssh in the cloud shell." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Open the cloud shell (See: intro_to_cli)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Define some environment variables" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "RG=\"demo_rg\"\n", - "LOCATION=eastus\n", - "VM=\"demo_vm\"\n", - "STGE_ACCT=\"demo_stor_ac\"\n", - "CTR=\"demo_stge_ctr\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Create resource group" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az group create --name $RG --location $LOCATION" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Create a VM" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If this is the fist time running this command (or your ~/.ssh dir does not have an id_rsa key pair). It will generate a new key pair id_rsa in ~/.ssh. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az vm create --resource-group $RG \\\n", - " --name $VM \\\n", - " --image UbuntuLTS \\\n", - " --admin-username demouser \\\n", - " --generate-ssh-keys \\\n", - " --output json \\\n", - " --verbose " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To create a VM with an existing pair first create a new key pair" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "ssh-keygen" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You will be prompted to \"Enter file in which to save the key\", type \"id_rsa\". This will create a key pair in the current dir." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here you should enter a passphrase (a series of words) to protect the key. You will be asked to re-enter the passphrase as shown below:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Enter same passphrase again:" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "After successfully retyping the passphrase the key generation process will continue showing similar message as follows (the fingerprint and art will look different):" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note: The following using \"--ssh-key-values\" to define which keys to use." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az vm create --name $VM \n", - " --resource-group $RG \n", - " --image UbuntuLTS \n", - " --admin-username demouser \n", - " --ssh-key-values id_rsa.pub" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Show VM details (all)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az vm show --name $VM --resource-group $RG" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Login into VM" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Get the public IP address from the output above.\n", - "If you generated a new key pair using \"--generate-ssh-keys\", login with the following command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "ssh " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you used the command with \"--ssh-key-values id_rsa.pub\" to use existing keys, use the following command to logiin:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "ssh -i /id_rsa " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "clean up\n", - "\n", - "The --no-wait parameter keeps the CLI from blocking while the deletion takes place." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az group delete --name $RG --no-wait" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49" - }, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.8" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/content/GCP/02_intro_to_compute.ipynb b/content/GCP/02_intro_to_compute.ipynb index 7097fbc..fda48cd 100644 --- a/content/GCP/02_intro_to_compute.ipynb +++ b/content/GCP/02_intro_to_compute.ipynb @@ -309,7 +309,7 @@ "Did you \"Follow\" the VM instance by looking at the **activity** page as discussed above?\n", "\n", "Since we care about paying for resources we are not using we review our project by visiting the *compute engine* service and reviewing that we no longer have any *VM instances* running. \n", - " * Go to **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances** and look for your virtual machine.\n", + " * Go to **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances** and look for your virtual machine. It should be gone!\n", "\n", "In a later episode we will show how to easily list all the resources in a project." ] diff --git a/content/GCP/04_intro_to_cli.ipynb b/content/GCP/04_intro_to_cli.ipynb index 9136726..72499e4 100644 --- a/content/GCP/04_intro_to_cli.ipynb +++ b/content/GCP/04_intro_to_cli.ipynb @@ -62,8 +62,11 @@ "id": "8eb44568-1108-4963-aebf-7f6b98ddaffc", "metadata": {}, "source": [ - "### Advanced call-out\n", - "You can access the cloud shell from your laptop via `gcloud cloud-shell ssh` if you have the [Google Cloud SDK installed](https://cloud.google.com/sdk/docs/install). Installing the Google Cloud SDK will also allow you to run `gcloud` and `gsutil` commands directly from your laptop, workstation, or cluster. *Pro-tip, you cannot use a service account to access the cloud shell.*" + "```{admonition} Tip\n", + ":class: tip\n", + "\n", + "You can access the cloud shell from your laptop via `gcloud cloud-shell ssh` if you have the [Google Cloud SDK installed](https://cloud.google.com/sdk/docs/install). Installing the Google Cloud SDK will also allow you to run `gcloud` and `gsutil` commands directly from your laptop, workstation, or cluster. *Pro-tip, you cannot use a service account to access the cloud shell.*\n", + "```" ] }, { @@ -101,7 +104,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 1, "id": "3ce9e6fb-9f5f-42e1-8512-35c0564f8e85", "metadata": {}, "outputs": [ @@ -109,7 +112,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "998995103712-compute@developer.gserviceaccount.com\n" + "Your active configuration is: [cloudshell-2071]\n", + "learner@class.internet2.edu\n" ] } ], @@ -127,7 +131,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "id": "cfb10587-e007-45ae-839f-7bcfe088c711", "metadata": {}, "outputs": [ @@ -135,6 +139,7 @@ "name": "stdout", "output_type": "stream", "text": [ + "Your active configuration is: [cloudshell-2071]\n", "essentials-learner\n" ] } @@ -153,7 +158,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 3, "id": "617325c9-d853-4291-a1db-938ab9439fee", "metadata": {}, "outputs": [ @@ -236,7 +241,13 @@ "\n", "*To make the point even clearer, we use environment variables in the Cloud Essentials notebooks instead of hard coding them, which makes it easy for anyone to run, use, and edit and thus collaborate and contribute to them!*\n", "\n", - "We now demonstrate the basic use environment variables. As a reminder, environment variables are used to store configuration information that can be easily passed between programs and programming languages." + "We now demonstrate the basic use environment variables. As a reminder, environment variables are used to store configuration information that can be easily passed between programs and programming languages.\n", + "\n", + "```{admonition} Advanced Tip\n", + ":class: tip\n", + "\n", + "Although we do not always need to, we should always enclose shell variables in double quotes and enclose the variable name in curly braces (funny things may happen otherwise) when writing scripts, for example `\"${TEST}\"`. In most cases it is easier to do this every time than try to figure out when it is possible to use the short form. We only use the short form (`$TEST`) when it is used frequently and known to always work (most cases here), in our case it is `$PROJECT` and `$BUCKET`. When in doubt, use the explicit form.\n", + "```" ] }, { @@ -272,8 +283,6 @@ "id": "78fe7013-a5ae-498f-aeab-ba1f3c710de1", "metadata": {}, "source": [ - "Although we do not always need to, we should always enclose shell variables in double quotes and enclose the variable name in curly braces (funny things may happen otherwise). In most cases it is easier to do this every time than try to figure out when it is possible to use the short form. We only use the short form (`$TEST`) when it is used frequently and known to always work, in our case it is `$PROJECT` and `$BUCKET`. When in doubt, use the explicit form.\n", - "\n", "There are a number of environment variables already set that contain useful information, for example:" ] }, @@ -287,12 +296,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "student31\n" + "learner\n" ] } ], "source": [ - "echo \"${USER}\"" + "echo $USER" ] }, { @@ -305,12 +314,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "cs-730348008320-default-default-wwhtf\n" + "cs-194566722441-default\n" ] } ], "source": [ - "echo \"${HOSTNAME}\"" + "echo $HOSTNAME" ] }, { @@ -331,7 +340,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Tue 02 Nov 2021 07:15:33 PM UTC\n" + "Thu 03 Feb 2022 09:50:31 PM UTC\n" ] } ], @@ -349,7 +358,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Date: Tue 02 Nov 2021 07:15:34 PM UTC\n" + "Date: Thu 03 Feb 2022 09:50:31 PM UTC\n" ] } ], @@ -357,6 +366,14 @@ "echo \"Date: $(date)\"" ] }, + { + "cell_type": "markdown", + "id": "4ba54cbd-573e-43f6-ad7c-ccf6dac871b0", + "metadata": {}, + "source": [ + "We use double quotes to make it a single string." + ] + }, { "cell_type": "markdown", "id": "6323f7bf-b2fe-4c94-a87a-2d35e75a1a49", @@ -377,8 +394,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "Your active configuration is: [cloudshell-17640]\n", - "just-armor-301114\n" + "Your active configuration is: [cloudshell-2071]\n", + "essentials-learner\n" ] } ], @@ -398,12 +415,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Your active configuration is: [cloudshell-17640]\n" + "Your active configuration is: [cloudshell-2071]\n" ] } ], "source": [ - "PROJECT=\"$(gcloud config get-value project)\"" + "PROJECT=$(gcloud config get-value project)" ] }, { @@ -416,7 +433,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "just-armor-301114\n" + "essentials-learner\n" ] } ], @@ -429,7 +446,17 @@ "id": "9d3a8504-a75c-4945-ab54-85f1d9a7e2b2", "metadata": {}, "source": [ - "Note the \"Your active configuration is: ...\" line is a \"warning\" and not actually part of the output of the program." + "Note the \"Your active configuration is: ...\" line is a \"warning\" and not actually part of the output of the program.\n", + "\n", + "````{admonition} Advanced Tip\n", + ":class: tip\n", + "\n", + "You can also use \n", + "```\n", + "PROJECT=$(gcloud config list --format='value(core.project)')\n", + "```\n", + "to get the project id without the warning.\n", + "````" ] }, { @@ -437,7 +464,7 @@ "id": "c25e6431-4f92-454a-bd10-e77cb28deec6", "metadata": {}, "source": [ - "Now use the `PROJECT` envrionment variable to get more information about the active project." + "Now use the `PROJECT` environment variable to get more information about the active project (you may be prompted to enable the API)." ] }, { @@ -450,11 +477,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "createTime: '2021-01-08T14:17:35.831Z'\n", + "createTime: '2022-01-14T19:03:26.486Z'\n", "lifecycleState: ACTIVE\n", - "name: CLASS Project\n", - "projectId: just-armor-301114\n", - "projectNumber: '1002111293252'\n" + "name: essentials-learner\n", + "parent:\n", + " id: '1072231596131'\n", + " type: folder\n", + "projectId: essentials-learner\n", + "projectNumber: '998995103712'\n" ] } ], @@ -462,106 +492,6 @@ "gcloud projects describe $PROJECT" ] }, - { - "cell_type": "markdown", - "id": "ed2ab9ce-6b0a-4939-88c7-8103e29413b7", - "metadata": {}, - "source": [ - "Note the extraneous message about the active configuration produced by the `gcloud config get-value project` command. A slightly different way to access the data can be used to remove the extraneous message by using the following command:" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "845b2aaf-0032-4043-a40a-d2482693d23e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[accessibility]\n", - "screen_reader = True\n", - "[component_manager]\n", - "disable_update_check = True\n", - "[compute]\n", - "gce_metadata_read_timeout_sec = 30\n", - "[core]\n", - "account = student31@class.internet2.edu\n", - "disable_usage_reporting = True\n", - "project = just-armor-301114\n", - "[metrics]\n", - "environment = devshell\n", - "\n", - "Your active configuration is: [cloudshell-17640]\n" - ] - } - ], - "source": [ - "gcloud config list" - ] - }, - { - "cell_type": "markdown", - "id": "d919b26d-2f64-4a43-9c76-34f7798fe853", - "metadata": {}, - "source": [ - "We can now use the `--format='value()'` option to get specific values. The following command returns the project number without the extraneous message." - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "9714ebe2-2921-498b-9e0e-69ec6a85896a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "just-armor-301114\n" - ] - } - ], - "source": [ - "gcloud config list --format='value(core.project)'" - ] - }, - { - "cell_type": "markdown", - "id": "ba660d2b-5ddf-449b-8049-56a40f2e366b", - "metadata": {}, - "source": [ - "Note where the `core` part comes in the output of the `gcloud config list` command." - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "7d803ddd-acca-4ad1-933a-4d6e6044def6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "just-armor-301114\n" - ] - } - ], - "source": [ - "PROJECT=\"$(gcloud config list --format='value(core.project)')\"\n", - "echo $PROJECT" - ] - }, - { - "cell_type": "markdown", - "id": "d40b2c7b-7331-412d-80e1-f1a77204075a", - "metadata": {}, - "source": [ - "We will use this form from now on. Since we know the Project ID only contains numbers, letters, and dashes with no spaces there is no need to use quotes." - ] - }, { "cell_type": "markdown", "id": "270e0abb-d5e9-476a-af91-060cc0ea0349", @@ -569,10 +499,8 @@ "source": [ "```{admonition} Exercise\n", "\n", - " * What command shows the project number? (hint: we used it already)\n", - " * What is the name of this variable? (hint: it is case sensitive)\n", - " * What command shows *just* the Project Number (hint: use `--format`)\n", - " * Set the `PROJECT_NUMBER` environment variable with the correct value. (hint: be careful with punctuation (single and double quotes) and matching brackets and punctuation)\n", + " * What command shows the account? (hint: we used it already)\n", + " * Set the `ACCOUNT` environment variable with the correct value. (hint: be careful with punctuation (single and double quotes) and matching brackets and punctuation)\n", "```" ] }, @@ -583,12 +511,12 @@ "source": [ "## Formating and Scripting Output (Optional)\n", "\n", - "The following example shows how to output the configuration as `JSON`. The command `jq` can also be used to parse the JSON data." + "The following example shows how to output the configuration as `JSON`. The command `jq` can also be used to parse the JSON data in addition to the `--format` option." ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 15, "id": "678a1735-8cad-403c-b302-639de1fb0f4b", "metadata": { "tags": [] @@ -599,34 +527,26 @@ "output_type": "stream", "text": [ "\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"accessibility\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"screen_reader\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"True\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"component_manager\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"disable_update_check\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"True\"\u001b[0m\u001b[1;39m\n", + " \u001b[0m\u001b[34;1m\"createTime\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"2022-01-14T19:03:26.486Z\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"lifecycleState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ACTIVE\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"essentials-learner\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"parent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", + " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1072231596131\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"folder\"\u001b[0m\u001b[1;39m\n", " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"compute\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"gce_metadata_read_timeout_sec\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"30\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"core\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"account\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"student31@class.internet2.edu\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"disable_usage_reporting\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"True\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"project\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"just-armor-301114\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"metrics\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"environment\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"devshell\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m\n", + " \u001b[0m\u001b[34;1m\"projectId\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"essentials-learner\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"projectNumber\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"998995103712\"\u001b[0m\u001b[1;39m\n", "\u001b[1;39m}\u001b[0m\n" ] } ], "source": [ - "gcloud config list --format=json |jq" + "gcloud projects describe $PROJECT --format=json | jq" ] }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 16, "id": "b0dec5ed-dc7f-4003-b228-f6605119ce47", "metadata": {}, "outputs": [ @@ -634,12 +554,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "just-armor-301114\n" + "998995103712\n" ] } ], "source": [ - "gcloud config list --format=json |jq -r .core.project" + "gcloud projects describe $PROJECT --format=json | jq -r .projectNumber" ] } ], diff --git a/content/GCP/05_cli_storage.ipynb b/content/GCP/05_cli_storage.ipynb index 4076e4f..878850e 100644 --- a/content/GCP/05_cli_storage.ipynb +++ b/content/GCP/05_cli_storage.ipynb @@ -53,7 +53,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "998995103712-compute@developer.gserviceaccount.com\n" + "Your active configuration is: [cloudshell-27767]\n", + "learner@class.internet2.edu\n" ] } ], @@ -71,6 +72,7 @@ "name": "stdout", "output_type": "stream", "text": [ + "Your active configuration is: [cloudshell-27767]\n", "essentials-learner\n" ] } @@ -82,6 +84,25 @@ { "cell_type": "code", "execution_count": 3, + "id": "44f0b255-508a-4488-abde-87aae67cd5b3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Your active configuration is: [cloudshell-27767]\n", + "learner@class.internet2.edu\n" + ] + } + ], + "source": [ + "ACCOUNT=$(gcloud config get-value account) ; echo $ACCOUNT" + ] + }, + { + "cell_type": "code", + "execution_count": 4, "id": "36fcc48b-8a5b-4158-9704-f1ea7c9951eb", "metadata": {}, "outputs": [ @@ -89,13 +110,13 @@ "name": "stdout", "output_type": "stream", "text": [ + "Your active configuration is: [cloudshell-27767]\n", "essentials-learner\n" ] } ], "source": [ - "PROJECT=$(gcloud config list --format='value(core.project)')\n", - "echo $PROJECT" + "PROJECT=$(gcloud config get-value project) ; echo $PROJECT" ] }, { @@ -118,7 +139,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "id": "255eb914-a06a-4bfd-adbb-7ca2f94678e7", "metadata": {}, "outputs": [ @@ -126,18 +147,18 @@ "name": "stdout", "output_type": "stream", "text": [ - "essentials-learner-2022-01-21\n" + "essentials-learner-2022-02-03\n" ] } ], "source": [ - "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", + "BUCKET=\"essentials-$USER-$(date +%F)\"\n", "echo $BUCKET" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 6, "id": "b0661b4c-b56b-4d6d-ba3c-e1c5f38699c6", "metadata": {}, "outputs": [ @@ -145,12 +166,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Creating gs://essentials-learner-2022-01-21/...\n" + "Creating gs://essentials-learner-2022-02-03/...\n" ] } ], "source": [ - "gsutil mb \"gs://$BUCKET\"" + "gsutil mb gs://$BUCKET" ] }, { @@ -175,7 +196,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "id": "8ae2988a-fca8-4601-a7e5-458627fe68e3", "metadata": {}, "outputs": [ @@ -183,7 +204,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-03/\n" ] } ], @@ -209,15 +230,18 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 8, "id": "ecc0e4fe-c9d0-4edd-9a0c-56b5b9ec7c54", - "metadata": {}, + "metadata": { + "scrolled": true, + "tags": [] + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/ :\n", + "gs://essentials-learner-2022-02-03/ :\n", "\tStorage class:\t\t\tSTANDARD\n", "\tLocation type:\t\t\tmulti-region\n", "\tLocation constraint:\t\tUS\n", @@ -229,71 +253,17 @@ "\tRequester Pays enabled:\t\tNone\n", "\tLabels:\t\t\t\tNone\n", "\tDefault KMS key:\t\tNone\n", - "\tTime created:\t\t\tFri, 21 Jan 2022 23:23:39 GMT\n", - "\tTime updated:\t\t\tFri, 21 Jan 2022 23:23:39 GMT\n", + "\tTime created:\t\t\tThu, 03 Feb 2022 22:19:56 GMT\n", + "\tTime updated:\t\t\tThu, 03 Feb 2022 22:19:56 GMT\n", "\tMetageneration:\t\t\t1\n", "\tBucket Policy Only enabled:\tFalse\n", "\tPublic access prevention:\tinherited\n", - "\tRPO:\t\t\t\tDEFAULT\n", - "\tACL:\t\t\t\t\n", - "\t [\n", - "\t {\n", - "\t \"entity\": \"project-owners-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"owners\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-editors-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"editors\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-viewers-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"viewers\"\n", - "\t },\n", - "\t \"role\": \"READER\"\n", - "\t }\n", - "\t ]\n", - "\tDefault ACL:\t\t\t\n", - "\t [\n", - "\t {\n", - "\t \"entity\": \"project-owners-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"owners\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-editors-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"editors\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-viewers-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"viewers\"\n", - "\t },\n", - "\t \"role\": \"READER\"\n", - "\t }\n", - "\t ]\n" + "\tRPO:\t\t\t\tDEFAULT\n" ] } ], "source": [ - "gsutil ls -L -b \"gs://$BUCKET\"" + "gsutil ls -L -b gs://$BUCKET | head -18" ] }, { @@ -303,14 +273,14 @@ "source": [ "You can see that the bucket is \"multi-region\" and uses the \"standard\" storage class (this is the type of storage). The standard storage class is best used for frequently used buckets. Other storage classes are designed for less frequent use, are less expensive, but come with more restrictions and more complex access costs and should only be used after careful consideration.\n", "\n", - "Ignore the \"ACL\" section for now.\n", + "*We used the `| head -18 ` command here to limit the response to the first few lines (we ignore ACL for now)*\n", "\n", "You may also want to verify that you can see the newly created bucket in the web console dashboard or \"Cloud Storage\" page and explore the properties there." ] }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 9, "id": "8fc85600-a793-413b-b139-642e4cf08a8e", "metadata": {}, "outputs": [], @@ -331,7 +301,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 10, "id": "bb8e7270-2f64-4fbd-b312-b18a7960700f", "metadata": { "scrolled": true, @@ -343,76 +313,20 @@ "output_type": "stream", "text": [ "---\n", - "insertId: -o4omdgeg7sge\n", + "insertId: -oywycvf1xe8xy\n", "logName: projects/essentials-learner/logs/cloudaudit.googleapis.com%2Factivity\n", "protoPayload:\n", " '@type': type.googleapis.com/google.cloud.audit.AuditLog\n", " authenticationInfo:\n", - " principalEmail: 998995103712-compute@developer.gserviceaccount.com\n", - " serviceAccountDelegationInfo:\n", - " - firstPartyPrincipal:\n", - " principalEmail: service-998995103712@compute-system.iam.gserviceaccount.com\n", + " principalEmail: learner@class.internet2.edu\n", " authorizationInfo:\n", " - granted: true\n", - " permission: storage.buckets.create\n", - " resource: projects/_/buckets/essentials-learner-2022-01-21\n", - " resourceAttributes: {}\n", - " metadata:\n", - " rpo: DEFAULT\n", - " methodName: storage.buckets.create\n", - " request:\n", - " defaultObjectAcl:\n", - " '@type': type.googleapis.com/google.iam.v1.Policy\n", - " bindings:\n", - " - members:\n", - " - projectViewer:essentials-learner\n", - " role: roles/storage.legacyObjectReader\n", - " - members:\n", - " - projectOwner:essentials-learner\n", - " - projectEditor:essentials-learner\n", - " role: roles/storage.legacyObjectOwner\n", - " requestMetadata:\n", - " callerIp: 34.102.126.22\n", - " callerNetwork: //compute.googleapis.com/projects/essentials-learner/global/networks/__unknown__\n", - " callerSuppliedUserAgent: apitools Python/3.7.3 gsutil/5.5 (linux) analytics/disabled\n", - " interactive/True command/mb google-cloud-sdk/366.0.0,gzip(gfe)\n", - " destinationAttributes: {}\n", - " requestAttributes:\n", - " auth: {}\n", - " time: '2022-01-21T23:23:39.060742076Z'\n", - " resourceLocation:\n", - " currentLocations:\n", - " - us\n", - " resourceName: projects/_/buckets/essentials-learner-2022-01-21\n", - " serviceData:\n", - " '@type': type.googleapis.com/google.iam.v1.logging.AuditData\n", - " policyDelta:\n", - " bindingDeltas:\n", - " - action: ADD\n", - " member: projectEditor:essentials-learner\n", - " role: roles/storage.legacyBucketOwner\n", - " - action: ADD\n", - " member: projectOwner:essentials-learner\n", - " role: roles/storage.legacyBucketOwner\n", - " - action: ADD\n", - " member: projectViewer:essentials-learner\n", - " role: roles/storage.legacyBucketReader\n", - " serviceName: storage.googleapis.com\n", - " status: {}\n", - "receiveTimestamp: '2022-01-21T23:23:39.729940907Z'\n", - "resource:\n", - " labels:\n", - " bucket_name: essentials-learner-2022-01-21\n", - " location: us\n", - " project_id: essentials-learner\n", - " type: gcs_bucket\n", - "severity: NOTICE\n", - "timestamp: '2022-01-21T23:23:39.055676566Z'\n" + " permission: storage.buckets.create\n" ] } ], "source": [ - "gcloud logging read --limit 1" + "gcloud logging read --limit 1 | head" ] }, { @@ -452,7 +366,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 11, "id": "f9a90752-871f-4156-b122-4c46fd4bfa43", "metadata": {}, "outputs": [], @@ -470,7 +384,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 12, "id": "bc0fb778-cc2e-4af4-8aa8-ca90d6b4f877", "metadata": {}, "outputs": [ @@ -496,7 +410,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 13, "id": "6384974b-d422-4ac1-90b7-cf9feec9aa8c", "metadata": {}, "outputs": [ @@ -511,12 +425,12 @@ } ], "source": [ - "gsutil cp one.txt \"gs://$BUCKET/1\"" + "gsutil cp one.txt gs://$BUCKET/1" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 14, "id": "49589119-56bb-4c7b-b0a6-36e8f1133056", "metadata": {}, "outputs": [ @@ -524,12 +438,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/1\n" + "gs://essentials-learner-2022-02-03/1\n" ] } ], "source": [ - "gsutil ls \"gs://$BUCKET\"" + "gsutil ls gs://$BUCKET" ] }, { @@ -540,6 +454,24 @@ "Copy the object to the standard out (displays it on the screen)." ] }, + { + "cell_type": "code", + "execution_count": 15, + "id": "79d0f1ac-e42f-46c4-aeec-6548a2651015", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "test one\n" + ] + } + ], + "source": [ + "gsutil cat gs://$BUCKET/1" + ] + }, { "cell_type": "markdown", "id": "ef366240-1152-4b30-b99f-b7835f63788d", @@ -550,7 +482,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 16, "id": "442cc51f-8856-41d4-a3a6-072c8569beef", "metadata": {}, "outputs": [ @@ -568,7 +500,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 17, "id": "4e3c0de9-8a23-428a-9689-0cf896228afe", "metadata": {}, "outputs": [ @@ -581,7 +513,7 @@ } ], "source": [ - "gsutil cat \"gs://$BUCKET/1\"" + "gsutil cat gs://$BUCKET/1" ] }, { @@ -594,7 +526,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 18, "id": "13f94b36-7dea-46b9-9e8b-baa8b4e59de6", "metadata": {}, "outputs": [ @@ -602,19 +534,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "Copying gs://essentials-learner-2022-01-21/1...\n", + "Copying gs://essentials-learner-2022-02-03/1...\n", "/ [1 files][ 9.0 B/ 9.0 B] \n", "Operation completed over 1 objects/9.0 B. \n" ] } ], "source": [ - "gsutil cp \"gs://$BUCKET/1\" one.txt" + "gsutil cp gs://$BUCKET/1 one.txt" ] }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 19, "id": "3982f21c-b827-4772-aa8c-3ac67e478b0d", "metadata": {}, "outputs": [ @@ -640,7 +572,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 20, "id": "6b57a3fc-94ea-4c25-b7ac-2744d7b28852", "metadata": {}, "outputs": [ @@ -666,7 +598,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 21, "id": "5a7fc453-fca7-4ab4-8f0a-cb4de53bc992", "metadata": {}, "outputs": [ @@ -681,7 +613,7 @@ } ], "source": [ - "date | gsutil cp - \"gs://$BUCKET/2\"" + "date | gsutil cp - gs://$BUCKET/2" ] }, { @@ -694,7 +626,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 22, "id": "1be58d86-5de7-420c-a3b3-0b82839ac90f", "metadata": {}, "outputs": [ @@ -702,8 +634,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/1\n", - "gs://essentials-learner-2022-01-21/2\n" + "gs://essentials-learner-2022-02-03/1\n", + "gs://essentials-learner-2022-02-03/2\n" ] } ], @@ -718,13 +650,13 @@ "source": [ "```{admonition} Exercise\n", "\n", - "* Display the date in the bucket.\n", + "* Display the object with the date in the bucket.\n", "```" ] }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 23, "id": "e3d0585b-4d81-4429-ad90-b3ff51eec13d", "metadata": {}, "outputs": [ @@ -732,12 +664,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Fri Jan 21 23:23:53 UTC 2022\n" + "Thu 03 Feb 2022 10:20:26 PM UTC\n" ] } ], "source": [ - "gsutil cat \"gs://$BUCKET/2\"" + "gsutil cat gs://$BUCKET/2" ] }, { @@ -750,7 +682,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 24, "id": "9ad460d9-7a4f-4e3f-befb-2bbb10c51993", "metadata": {}, "outputs": [ @@ -758,19 +690,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/1...\n", + "Removing gs://essentials-learner-2022-02-03/1...\n", "/ [1 objects] \n", "Operation completed over 1 objects. \n" ] } ], "source": [ - "gsutil rm \"gs://$BUCKET/1\"" + "gsutil rm gs://$BUCKET/1" ] }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 25, "id": "8fd782f2-700a-4862-970b-1b4751009d5f", "metadata": {}, "outputs": [ @@ -778,12 +710,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/2\n" + "gs://essentials-learner-2022-02-03/2\n" ] } ], "source": [ - "gsutil ls \"gs://$BUCKET\"" + "gsutil ls gs://$BUCKET" ] }, { @@ -796,7 +728,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 26, "id": "e1350695-0f01-4205-a604-91d62896783e", "metadata": {}, "outputs": [ @@ -804,13 +736,13 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/...\n", - "NotEmptyException: 409 BucketNotEmpty (essentials-learner-2022-01-21)\n" + "Removing gs://essentials-learner-2022-02-03/...\n", + "NotEmptyException: 409 BucketNotEmpty (essentials-learner-2022-02-03)\n" ] } ], "source": [ - "gsutil rb \"gs://$BUCKET\"\n", + "gsutil rb gs://$BUCKET\n", "/bin/true # ignore expected error in Jupyter" ] }, @@ -824,7 +756,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 27, "id": "ba2671f8-521f-48b4-9231-491d128ba67c", "metadata": {}, "outputs": [ @@ -832,19 +764,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/2...\n", + "Removing gs://essentials-learner-2022-02-03/2...\n", "/ [1 objects] \n", "Operation completed over 1 objects. \n" ] } ], "source": [ - "gsutil rm \"gs://$BUCKET/2\"" + "gsutil rm gs://$BUCKET/2" ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 28, "id": "73d8b299-8e8e-443e-ae5b-4ecf7efa0442", "metadata": {}, "outputs": [ @@ -852,7 +784,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-03/\n" ] } ], @@ -871,7 +803,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 29, "id": "7b32764d-e7b5-42d6-8f88-bc6936b2024a", "metadata": {}, "outputs": [ @@ -879,7 +811,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-03/\n" ] } ], @@ -897,7 +829,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 30, "id": "82487bd8-14f4-4811-8f5a-dd346042bd0f", "metadata": {}, "outputs": [ @@ -905,7 +837,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Bucket: essentials-learner-2022-01-21\n" + "Bucket: essentials-learner-2022-02-03\n" ] } ], @@ -913,9 +845,35 @@ "echo \"Bucket: $BUCKET\"" ] }, + { + "cell_type": "markdown", + "id": "6120eee3-47d0-4e28-a1dd-b1090297a573", + "metadata": {}, + "source": [ + "Check to see if the bucket is empty" + ] + }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 31, + "id": "a52e1838-02f4-406c-a58f-6a709d9aaf2e", + "metadata": {}, + "outputs": [], + "source": [ + "gsutil ls gs://$BUCKET" + ] + }, + { + "cell_type": "markdown", + "id": "6eb97c7e-aed6-493a-b724-a4d6ba3cb806", + "metadata": {}, + "source": [ + "Remove the bucket" + ] + }, + { + "cell_type": "code", + "execution_count": 32, "id": "8635db94-2f3c-46d2-acc2-76a176fb37e7", "metadata": {}, "outputs": [ @@ -923,12 +881,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/...\n" + "Removing gs://essentials-learner-2022-02-03/...\n" ] } ], "source": [ - "gsutil rb \"gs://$BUCKET\"" + "gsutil rb gs://$BUCKET" ] }, { @@ -936,12 +894,12 @@ "id": "3e2178ca", "metadata": {}, "source": [ - "We verify that the bucket has been removed. In this example there is no output since there are no more buckets." + "We verify that the bucket has been removed." ] }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 33, "id": "ed6f0204-0de2-4d20-ab00-3b7a1023fd86", "metadata": {}, "outputs": [], diff --git a/content/GCP/06_running_analysis.ipynb b/content/GCP/06_running_analysis.ipynb index 72780cd..f5bfdd1 100644 --- a/content/GCP/06_running_analysis.ipynb +++ b/content/GCP/06_running_analysis.ipynb @@ -12,13 +12,22 @@ "\n", "**Teaching:** 80 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** 8 min\n", "\n", "**Questions:**\n", - "* Can you show me an example?\n", + " * Can you show me an example?\n", "\n", "**Objectives:**\n", - "* Create a simple workflow using a cloud VM and cloud object storage.\n", + " * Create a simple workflow using a cloud VM and cloud object storage.\n", + " * Update a VM Instance software for important security updates.\n", + " * Create a VM Instance with the appropriate storage scope.\n", + " * Create a private regional storage bucket with appropriate security settings.\n", + " * Using the CLI to install software.\n", + " * Download source code using git\n", + " * Retrieve data from a bucket\n", + " * Run the python analysis code\n", + " * Store results in a bucket\n", + " * View the results in the Cloud Storage browser.\n", "\n", "```" ] @@ -30,30 +39,55 @@ "source": [ "## A Research Computational and Data Workflow - Drew's story\n", "\n", - "Drew needs to do some analysis on the data. They need data (satellite images stored in the cloud), computational resources (a virtual machine), some software (we will supply this), and a place to store the results (Cloud Storage). We will assemble all these parts in the cloud \n" + "Drew needs to do some analysis on the data. They need data (satellite images stored in the cloud), computational resources (a virtual machine), some software (we will supply this), and a place to store the results (Cloud Storage). We will assemble and process all these parts in the cloud with a simple example.\n" ] }, { "cell_type": "markdown", - "id": "6291edee-c2df-4941-9b8e-de42649640f9", + "id": "245ffbcb-bf77-4ad1-8bde-1c7d2717ab46", "metadata": {}, "source": [ "## Create a VM\n", "\n", - "Since we only create resources as we need them in the cloud, we will now create a new virtual machine (VM) for Drew to use for their analysis.\n", + "Since we only create resources as we need them in the cloud, we will now create a new virtual machine (VM)instance for Drew to use for their analysis.\n", "\n", - "We will do this as an exercise to give you practice in creating resources. Since the virtual machine will need access to storage on your behalf, you will need to change the **access scope** to give **Full** access to the **Storage** API to the virtual machine. \n", - "\n", - "### Exercise\n", + "We will do this as an exercise to give you practice in creating resources. Since the virtual machine will need access to storage on your behalf, you will need to change the **access scope** to give **Full** access to the **Storage** API to the virtual machine." + ] + }, + { + "cell_type": "markdown", + "id": "45cb5d1d-c28a-4e0e-a0f3-bf68add065be", + "metadata": {}, + "source": [ + "````{admonition} Exercise\n", "\n", "Using the console navigate to the \"Compute Engine\" service and create a new VM with the following properties.\n", " * Call the VM \"essentials\"\n", " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" on the \"create an instance\" page and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", - " * Feel free to select a bit larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + " * Select a bit larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + "````" + ] + }, + { + "cell_type": "markdown", + "id": "009fbfee-f1ce-4046-8223-0aa11b21bd61", + "metadata": {}, + "source": [ + "*Instructor: place the above exercise instructions on the screen*\n", + "\n", + "*When you are done feel free to connect to the virtual machine on your own for additional practice. Once everyone has created their VM we will connect to the machine as described below.*\n", + "\n", + "Please verify that the virtual machine was created as above. If you are unsure delete the virtual machine instance and create it again.\n", "\n", - "*Instructor: place these instructions on the screen*\n", + "Verify that the **Compute Engine default service account** is being used.\n", + "![compute-iam-service-account](img/compute-iam-service-account.png)\n", "\n", - "*When you are done feel free to connect to the virtual machine on your own for additional practice. Once everyone has created their VM we will connect to the machine as described below.*" + "Change **Access scopes** to **Set access for each API**\n", + "![compute-iam-scope-top](img/compute-iam-scope-top.png)\n", + "\n", + "And set **Storage** to **Full**.\n", + "![compute-iam-scope-storage-full](img/compute-iam-scope-storage-full.png)\n", + "\n" ] }, { @@ -100,7 +134,7 @@ "source": [ "## Secure the VM\n", "\n", - "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs security patches." + "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs important security packages and does not upgrade all packages." ] }, { @@ -113,20 +147,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\n", - "Hit:2 http://deb.debian.org/debian buster InRelease \u001b[0m\u001b[33m\n", - "Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]\n", - "Get:4 http://deb.debian.org/debian buster-backports InRelease [46.7 kB]\n", - "Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease\n", + "Hit:1 http://security.debian.org/debian-security buster/updates InRelease\n", + "Hit:2 http://deb.debian.org/debian buster InRelease\n", + "Hit:3 http://deb.debian.org/debian buster-updates InRelease\n", + "Hit:4 http://deb.debian.org/debian buster-backports InRelease\n", + "Get:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease [6780 B]\n", "Hit:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease\n", "Hit:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease\n", - "Get:8 http://deb.debian.org/debian buster-backports/main amd64 Packages.diff/Index [27.8 kB]\n", - "Get:9 http://deb.debian.org/debian buster-backports/main Translation-en.diff/Index [27.8 kB]\n", - "Get:10 http://deb.debian.org/debian buster-backports/main amd64 Packages 2022-01-21-2002.00.pdiff [2515 B]\n", - "Get:11 http://deb.debian.org/debian buster-backports/main Translation-en 2022-01-21-2002.00.pdiff [3068 B]\n", - "Get:10 http://deb.debian.org/debian buster-backports/main amd64 Packages 2022-01-21-2002.00.pdiff [2515 B]\n", - "Get:11 http://deb.debian.org/debian buster-backports/main Translation-en 2022-01-21-2002.00.pdiff [3068 B]\n", - "Fetched 225 kB in 1s (278 kB/s)0m\u001b[33m \u001b[0m\u001b[33m\n", + "Fetched 6780 B in 1s (8978 B/s)\n", "Reading package lists... Done\n", "Building dependency tree \n", "Reading state information... Done\n", @@ -172,7 +200,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "bucket: essentials-learner-2022-01-21 region: us-west2\n" + "bucket: essentials-learner-2022-02-07 region: us-west2\n" ] } ], @@ -192,7 +220,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Creating gs://essentials-learner-2022-01-21/...\n" + "Creating gs://essentials-learner-2022-02-07/...\n" ] } ], @@ -218,7 +246,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-07/\n" ] } ], @@ -260,7 +288,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 11, "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", "metadata": {}, "outputs": [], @@ -270,7 +298,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 12, "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", "metadata": {}, "outputs": [ @@ -279,9 +307,11 @@ "output_type": "stream", "text": [ "Cloning into 'CLASS-Examples'...\n", - "remote: Enumerating objects: 66, done.\u001b[K\n", - "remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66\u001b[K\n", - "Unpacking objects: 100% (66/66), done.\n" + "remote: Enumerating objects: 4, done.\u001b[K\n", + "remote: Counting objects: 100% (4/4), done.\u001b[K\n", + "remote: Compressing objects: 100% (4/4), done.\u001b[K\n", + "remote: Total 70 (delta 0), reused 1 (delta 0), pack-reused 66\u001b[K\n", + "Unpacking objects: 100% (70/70), done.\n" ] } ], @@ -299,7 +329,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 13, "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", "metadata": {}, "outputs": [], @@ -320,7 +350,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 14, "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", "metadata": {}, "outputs": [ @@ -329,13 +359,13 @@ "output_type": "stream", "text": [ "total 28\n", - "-rw-r--r-- 1 learner learner 960 Jan 21 21:32 ReadMe.md\n", - "-rw-r--r-- 1 learner learner 72 Jan 21 21:32 clean.sh\n", - "-rw-r--r-- 1 learner learner 613 Jan 21 21:32 combine.py\n", - "-rw-r--r-- 1 learner learner 280 Jan 21 21:32 download.sh\n", - "-rw-r--r-- 1 learner learner 314 Jan 21 21:32 get-index.sh\n", - "-rw-r--r-- 1 learner learner 76 Jan 21 21:32 search.json\n", - "-rw-r--r-- 1 learner learner 783 Jan 21 21:32 search.py\n" + "-rw-r--r-- 1 learner learner 964 Feb 7 16:04 ReadMe.md\n", + "-rw-r--r-- 1 learner learner 72 Feb 7 16:04 clean.sh\n", + "-rw-r--r-- 1 learner learner 280 Feb 7 16:04 download.sh\n", + "-rw-r--r-- 1 learner learner 314 Feb 7 16:04 get-index.sh\n", + "-rw-r--r-- 1 learner learner 613 Feb 7 16:04 process_sat.py\n", + "-rw-r--r-- 1 learner learner 76 Feb 7 16:04 search.json\n", + "-rw-r--r-- 1 learner learner 783 Feb 7 16:04 search.py\n" ] } ], @@ -357,7 +387,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 15, "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", "metadata": {}, "outputs": [ @@ -406,7 +436,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 16, "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", "metadata": {}, "outputs": [ @@ -424,7 +454,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 17, "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", "metadata": {}, "outputs": [ @@ -438,7 +468,7 @@ "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", "\n", - "| [1 files][731.9 MiB/731.9 MiB] \n", + "/ [1 files][731.9 MiB/731.9 MiB] 63.0 MiB/s \n", "Operation completed over 1 objects/731.9 MiB. \n" ] } @@ -457,7 +487,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 18, "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", "metadata": {}, "outputs": [], @@ -475,7 +505,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 19, "id": "b005876c-f9af-43d6-80c6-f04295413b9b", "metadata": {}, "outputs": [ @@ -484,7 +514,7 @@ "output_type": "stream", "text": [ "total 2.5G\n", - "-rw-r--r-- 1 learner learner 2.5G Jan 21 21:32 index.csv\n" + "-rw-r--r-- 1 learner learner 2.5G Feb 7 16:04 index.csv\n" ] } ], @@ -502,7 +532,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 20, "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", "metadata": {}, "outputs": [ @@ -521,6 +551,18 @@ "head --lines=4 data/index.csv" ] }, + { + "cell_type": "markdown", + "id": "f98c38de-87fa-4e66-9d4b-186fbf81b3b2", + "metadata": {}, + "source": [ + "````{admonition} Tip\n", + ":class: Tip\n", + "\n", + "Now our virtual machine instance is ready and we can access the code and data. Now is a great time to take a short break.\n", + "````" + ] + }, { "cell_type": "markdown", "id": "532e6da3-302a-4e8a-8570-752995f30f1d", @@ -533,7 +575,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 21, "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", "metadata": {}, "outputs": [ @@ -577,7 +619,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 22, "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", "metadata": {}, "outputs": [ @@ -607,7 +649,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 23, "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", "metadata": {}, "outputs": [ @@ -638,7 +680,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 24, "id": "3572c518-df83-4906-bfa6-a37bde2a5063", "metadata": {}, "outputs": [ @@ -673,7 +715,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 25, "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", "metadata": { "scrolled": true, @@ -687,14 +729,14 @@ "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_ANG.txt...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B3.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B2.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B6.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B5.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B11.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B7.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B2.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B8.TIF...\n", "==> NOTE: You are downloading one or more large file(s), which would\n", "run significantly faster if you enabled sliced object downloads. This\n", @@ -704,21 +746,21 @@ "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B9.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_BQA.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_MTL.txt...\n", - "- [14/14 files][952.9 MiB/952.9 MiB] 100% Done \n", + "| [14/14 files][952.9 MiB/952.9 MiB] 100% Done 18.6 MiB/s ETA 00:00:00 \n", "Operation completed over 14 objects/952.9 MiB. \n", "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_ANG.txt...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B3.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B10.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B5.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B6.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B10.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B5.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B7.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B6.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B8.TIF...\n", - "==> NOTE: You are downloading one or more large file(s), which would\n", + "==> NOTE: You are downloading one or more large file(s), which would \n", "run significantly faster if you enabled sliced object downloads. This\n", "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", @@ -726,7 +768,7 @@ "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B9.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_BQA.TIF...\n", "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_MTL.txt...\n", - "/ [14/14 files][ 1.0 GiB/ 1.0 GiB] 100% Done 30.3 MiB/s ETA 00:00:00 \n", + "- [14/14 files][ 1.0 GiB/ 1.0 GiB] 100% Done 19.1 MiB/s ETA 00:00:00 \n", "Operation completed over 14 objects/1.0 GiB. \n" ] } @@ -745,7 +787,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 26, "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", "metadata": {}, "outputs": [ @@ -754,9 +796,9 @@ "output_type": "stream", "text": [ "total 2564796\n", - "drwxr-xr-x 2 learner learner 4096 Jan 21 21:33 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20201007_20201016_01_T1\u001b[0m\n", - "drwxr-xr-x 2 learner learner 4096 Jan 21 21:33 \u001b[01;34mLC08_L1TP_025033_20210519_20210528_01_T1\u001b[0m\n", - "-rw-r--r-- 1 learner learner 2626336574 Jan 21 21:32 index.csv\n" + "drwxr-xr-x 2 learner learner 4096 Feb 7 16:07 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20201007_20201016_01_T1\u001b[0m\n", + "drwxr-xr-x 2 learner learner 4096 Feb 7 16:08 \u001b[01;34mLC08_L1TP_025033_20210519_20210528_01_T1\u001b[0m\n", + "-rw-r--r-- 1 learner learner 2626336574 Feb 7 16:04 index.csv\n" ] } ], @@ -776,7 +818,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 27, "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", "metadata": {}, "outputs": [ @@ -807,12 +849,12 @@ } ], "source": [ - "cat combine.py" + "cat process_sat.py" ] }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 28, "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", "metadata": {}, "outputs": [ @@ -821,14 +863,14 @@ "output_type": "stream", "text": [ "Traceback (most recent call last):\n", - " File \"combine.py\", line 3, in \n", + " File \"process_sat.py\", line 3, in \n", " import rasterio\n", "ModuleNotFoundError: No module named 'rasterio'\n" ] } ], "source": [ - "python3 combine.py\n", + "python3 process_sat.py\n", "/bin/true # ignore this line used for jupyter" ] }, @@ -843,7 +885,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 29, "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", "metadata": { "scrolled": true, @@ -900,8 +942,8 @@ "0 upgraded, 79 newly installed, 0 to remove and 2 not upgraded.\n", "Need to get 46.8 MB of archives.\n", "After this operation, 172 MB of additional disk space will be used.\n", - "Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", - "Get:2 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", + "Get:1 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", + "Get:2 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", "Get:3 http://deb.debian.org/debian buster/main amd64 fonts-dejavu-core all 2.37-1 [1068 kB]\n", "Get:4 http://deb.debian.org/debian buster/main amd64 fontconfig-config all 2.13.1-2 [280 kB]\n", "Get:5 http://deb.debian.org/debian buster/main amd64 gdal-data all 2.4.0+dfsg-1 [744 kB]\n", @@ -909,36 +951,36 @@ "Get:7 http://deb.debian.org/debian buster/main amd64 libgfortran5 amd64 8.3.0-6 [581 kB]\n", "Get:8 http://deb.debian.org/debian buster/main amd64 libblas3 amd64 3.8.0-2 [148 kB]\n", "Get:9 http://deb.debian.org/debian buster/main amd64 liblapack3 amd64 3.8.0-2 [2110 kB]\n", - "Get:10 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", - "Get:11 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", - "Get:12 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", - "Get:13 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", - "Get:14 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", - "Get:15 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", - "Get:16 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", - "Get:17 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", - "Get:18 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", - "Get:19 http://security.debian.org/debian-security buster/updates/main amd64 python3-lxml amd64 4.3.2-1+deb10u4 [1163 kB]\n", - "Get:20 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", - "Get:21 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", - "Get:22 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", - "Get:23 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", - "Get:24 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", - "Get:25 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", - "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", - "Get:27 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", - "Get:28 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", - "Get:29 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", - "Get:30 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", - "Get:31 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", - "Get:32 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", - "Get:33 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", - "Get:34 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", - "Get:35 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", - "Get:36 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", - "Get:37 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", - "Get:38 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", - "Get:39 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", + "Get:10 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", + "Get:11 http://security.debian.org/debian-security buster/updates/main amd64 liburiparser1 amd64 0.9.1-1+deb10u1 [48.1 kB]\n", + "Get:12 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u5 [1160 kB]\n", + "Get:13 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", + "Get:14 http://security.debian.org/debian-security buster/updates/main amd64 python3-lxml amd64 4.3.2-1+deb10u4 [1163 kB]\n", + "Get:15 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", + "Get:16 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", + "Get:17 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", + "Get:18 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", + "Get:19 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", + "Get:20 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", + "Get:21 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", + "Get:22 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", + "Get:23 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", + "Get:24 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", + "Get:25 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", + "Get:26 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", + "Get:27 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", + "Get:28 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", + "Get:29 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", + "Get:30 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", + "Get:31 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", + "Get:32 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", + "Get:33 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", + "Get:34 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", + "Get:35 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", + "Get:36 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", + "Get:37 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", + "Get:38 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", + "Get:39 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", "Get:40 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", "Get:41 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", @@ -979,11 +1021,11 @@ "Get:77 http://deb.debian.org/debian buster/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [89.6 kB]\n", "Get:78 http://deb.debian.org/debian buster/main amd64 python3-snuggs all 1.4.3-1 [7228 B]\n", "Get:79 http://deb.debian.org/debian buster/main amd64 python3-rasterio amd64 1.0.21-1 [818 kB]\n", - "Fetched 46.8 MB in 2s (28.9 MB/s) \n", + "Fetched 46.8 MB in 5s (8830 kB/s) \n", "Extracting templates from packages: 100%\n", "Preconfiguring packages ...\n", "Selecting previously unselected package poppler-data.\n", - "(Reading database ... 57529 files and directories currently installed.)\n", + "(Reading database ... 57683 files and directories currently installed.)\n", "Preparing to unpack .../00-poppler-data_0.4.9-2_all.deb ...\n", "Unpacking poppler-data (0.4.9-2) ...\n", "Selecting previously unselected package fonts-dejavu-core.\n", @@ -1089,8 +1131,8 @@ "Preparing to unpack .../34-libminizip1_1.1-8+b1_amd64.deb ...\n", "Unpacking libminizip1:amd64 (1.1-8+b1) ...\n", "Selecting previously unselected package liburiparser1:amd64.\n", - "Preparing to unpack .../35-liburiparser1_0.9.1-1_amd64.deb ...\n", - "Unpacking liburiparser1:amd64 (0.9.1-1) ...\n", + "Preparing to unpack .../35-liburiparser1_0.9.1-1+deb10u1_amd64.deb ...\n", + "Unpacking liburiparser1:amd64 (0.9.1-1+deb10u1) ...\n", "Selecting previously unselected package libkmlbase1:amd64.\n", "Preparing to unpack .../36-libkmlbase1_1.3.0-7_amd64.deb ...\n", "Unpacking libkmlbase1:amd64 (1.3.0-7) ...\n", @@ -1140,8 +1182,8 @@ "Preparing to unpack .../51-libnspr4_2%3a4.20-1_amd64.deb ...\n", "Unpacking libnspr4:amd64 (2:4.20-1) ...\n", "Selecting previously unselected package libnss3:amd64.\n", - "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u4_amd64.deb ...\n", - "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u5_amd64.deb ...\n", + "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u5) ...\n", "Selecting previously unselected package libpoppler82:amd64.\n", "Preparing to unpack .../53-libpoppler82_0.71.0-5_amd64.deb ...\n", "Unpacking libpoppler82:amd64 (0.71.0-5) ...\n", @@ -1257,7 +1299,7 @@ "Setting up libgfortran5:amd64 (8.3.0-6) ...\n", "Setting up libhdf4-0-alt (4.2.13-4) ...\n", "Setting up libgif7:amd64 (5.1.4-3) ...\n", - "Setting up liburiparser1:amd64 (0.9.1-1) ...\n", + "Setting up liburiparser1:amd64 (0.9.1-1+deb10u1) ...\n", "Setting up libfreexl1:amd64 (1.0.5-3) ...\n", "Setting up libfyba0:amd64 (4.1.1-6) ...\n", "Setting up libkmlbase1:amd64 (1.3.0-7) ...\n", @@ -1277,7 +1319,7 @@ "Setting up fontconfig-config (2.13.1-2) ...\n", "Setting up python3-cligj (0.5.0-1) ...\n", "Setting up libdap25:amd64 (3.20.3-1) ...\n", - "Setting up libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", + "Setting up libnss3:amd64 (2:3.42.1-1+deb10u5) ...\n", "Setting up libdapserver7v5:amd64 (3.20.3-1) ...\n", "Setting up python3-bs4 (4.7.1-1) ...\n", "Setting up libgeotiff2:amd64 (1.4.3-1) ...\n", @@ -1315,7 +1357,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 31, "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", "metadata": {}, "outputs": [ @@ -1331,7 +1373,7 @@ } ], "source": [ - "/usr/bin/python3 combine.py" + "/usr/bin/python3 process_sat.py" ] }, { @@ -1352,7 +1394,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 32, "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", "metadata": {}, "outputs": [ @@ -1361,8 +1403,8 @@ "output_type": "stream", "text": [ "total 192M\n", - "-rw-r--r-- 1 learner learner 192M Jan 21 21:34 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20201007_20201016_01_T1.png\u001b[0m\n", - "-rw-r--r-- 1 learner learner 910 Jan 21 21:34 result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n" + "-rw-r--r-- 1 learner learner 192M Feb 7 16:09 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20201007_20201016_01_T1.png\u001b[0m\n", + "-rw-r--r-- 1 learner learner 910 Feb 7 16:09 result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n" ] } ], @@ -1382,7 +1424,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 33, "id": "9345472f-4ef3-490b-a80e-2462cd534c89", "metadata": {}, "outputs": [ @@ -1390,7 +1432,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "essentials-learner-2022-01-21\n" + "essentials-learner-2022-02-07\n" ] } ], @@ -1400,7 +1442,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 34, "id": "27dfae96-faf2-4d5d-8a78-97781841f172", "metadata": {}, "outputs": [ @@ -1408,7 +1450,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-07/\n" ] } ], @@ -1428,7 +1470,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 35, "id": "681e6b1d-98bb-448a-a57e-f5674214effd", "metadata": {}, "outputs": [ @@ -1469,7 +1511,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 36, "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", "metadata": {}, "outputs": [ @@ -1477,7 +1519,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/output/\n" + "gs://essentials-learner-2022-02-07/output/\n" ] } ], @@ -1487,7 +1529,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 37, "id": "b1ea18e9-5861-4479-9948-3303952dee8a", "metadata": {}, "outputs": [ @@ -1495,8 +1537,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "191.58 MiB 2022-01-21T21:34:56Z gs://essentials-learner-2022-01-21/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png\n", - " 910 B 2022-01-21T21:34:54Z gs://essentials-learner-2022-01-21/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n", + "191.58 MiB 2022-02-07T16:09:41Z gs://essentials-learner-2022-02-07/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png\n", + " 910 B 2022-02-07T16:09:39Z gs://essentials-learner-2022-02-07/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n", "TOTAL: 2 objects, 200890195 bytes (191.58 MiB)\n" ] } @@ -1507,7 +1549,7 @@ }, { "cell_type": "markdown", - "id": "66a41f97-3834-4d09-bfe5-c3f224628842", + "id": "36530927-2205-4586-bca7-10b8698f5a15", "metadata": {}, "source": [ "## Viewing the Results\n", @@ -1515,6 +1557,30 @@ "You now can view the results by using the Google Cloud Platform Web Console and navigating to \"Cloud Storage\", selecting the bucket, and then the result object you wish to view (select the `.png` file). You will need to click the \"Preview\" button given the large size of the image." ] }, + { + "cell_type": "markdown", + "id": "4de32b26-f4cb-4f33-b7d5-e3852769840a", + "metadata": {}, + "source": [ + "````{admonition} Exercise\n", + "\n", + " * Try to find and view the results on your own\n", + "\n", + "````" + ] + }, + { + "cell_type": "markdown", + "id": "9f7a4893-104a-4467-84ac-871d21db6050", + "metadata": {}, + "source": [ + "Navigate to **Cloud Storage** -> **Browser** -> **Bucket** -> **output** folder and then click on the **result** object.\n", + "![example-object](img/example-object.png)\n", + "\n", + "And press the **Preview** button below the object details (check the details) and you should see something similar to the following:\n", + "![example-object-preview](img/example-object-preview.png)" + ] + }, { "cell_type": "markdown", "id": "40d9ab41-8920-45f2-8218-550baac5b069", diff --git a/content/GCP/06b_sharing_results.ipynb b/content/GCP/06_sharing_results.ipynb similarity index 100% rename from content/GCP/06b_sharing_results.ipynb rename to content/GCP/06_sharing_results.ipynb diff --git a/content/GCP/glossary.ipynb b/content/GCP/glossary.ipynb index e690bf9..b0ef7a6 100644 --- a/content/GCP/glossary.ipynb +++ b/content/GCP/glossary.ipynb @@ -7,12 +7,21 @@ "source": [ "# Glossary\n", "\n", - "* **Account** - The Google account (email) associated with an individual. This is the identity that you use to access console and takes the form of an email address.\n", - "* **Project** - Resources and services in Google Cloud are created within a single Project. \n", - "* **Project Name** - The long human readable name of your project. You can change this.\n", - "* **Project ID** - an alpha-numeric string identifier for the project. This is not changeable.\n", - "* **Project Number** - a long number that also identifies the project but is rarely used (it is an internal identifier). \n", - "* **Billing Account** - an account used to pay for Google Cloud services. A Billing Account is associated with a Project and enables resource allocation. A project has only one Billing Account associated with it. A Billing account can be used for multiple Projects." + " * **Account** - The Google account (email) associated with an individual. This is the identity that you use to access console and takes the form of an email address.\n", + " * **API** - Application Programming Interface (API). An external interface for a library or software, usually well documented. The cloud uses remote accessible APIs to allocate and control resources.\n", + " * **Billing Account** - an account used to pay for Google Cloud services. A Billing Account is associated with a Project and enables resource allocation. A project has only one Billing Account associated with it. A Billing account can be used for multiple Projects.\n", + " * **Bucket** - A collection of key value pairs (similar to a Python dictionary) and is often called object storage. Object storage in Google is called Cloud Storage and is similar to Amazon S3.\n", + " * **Cloud Shell** - A Linux command line interface (CLI) running inside the GCP web console. \n", + " * **Disk Image** - The operating system disk for a **VM Instance** such as Ubuntu, Debian, and other Linux distributions.\n", + " * **Project** - Resources and services in Google Cloud are created within a single Project. \n", + " * **Project Name** - The long human readable name of your project. You can change this.\n", + " * **Project ID** - an alpha-numeric string identifier for the project. This is not changeable.\n", + " * **Project Number** - a long number that also identifies the project but is rarely used (it is an internal identifier). \n", + " * **Region** - A geographical region within the world (typically a city) in which a number of physical or logical data centers are located (each is a different **zone**). An example of a **region** is `us-west2`, which is located in Los Angeles.\n", + " * **Service Account** - A machine **account** or identity that is used to give permissions to a resource, such as a **VM Instance** in GCP. These take the form of an email, usually ending with `gserviceaccount.com`.\n", + " * **Label** - A key value pair attached to a resource to aid in finding resources, allocating billing, debugging, etc.. A Label is similar to a *tag* in AWS and Azure. \n", + " * **VM Instance** - A Virtual Machine (VM) is a virtual computer running in a **zone**.\n", + " * **Zone** - A logical or physical data center (a building with computers in it) in which cloud resources physically reside. Each **zone** in a **region** have different resources. An example of a **zone** is `us-west2-c`." ] } ], diff --git a/content/GCP/img/compute-iam-scope-storage-full.png b/content/GCP/img/compute-iam-scope-storage-full.png new file mode 100644 index 0000000..670ce48 Binary files /dev/null and b/content/GCP/img/compute-iam-scope-storage-full.png differ diff --git a/content/GCP/img/compute-iam-scope-top.png b/content/GCP/img/compute-iam-scope-top.png new file mode 100644 index 0000000..1400754 Binary files /dev/null and b/content/GCP/img/compute-iam-scope-top.png differ diff --git a/content/GCP/img/compute-iam-service-account.png b/content/GCP/img/compute-iam-service-account.png new file mode 100644 index 0000000..6e08ea2 Binary files /dev/null and b/content/GCP/img/compute-iam-service-account.png differ diff --git a/content/GCP/img/example-object-preview.png b/content/GCP/img/example-object-preview.png new file mode 100644 index 0000000..efcc6e3 Binary files /dev/null and b/content/GCP/img/example-object-preview.png differ diff --git a/content/GCP/img/example-object.png b/content/GCP/img/example-object.png new file mode 100644 index 0000000..7bcb81f Binary files /dev/null and b/content/GCP/img/example-object.png differ diff --git a/content/GCP/intro_to_GCP_Essentials.ipynb b/content/GCP/intro_to_GCP_Essentials.ipynb index 6bff3c8..e486192 100644 --- a/content/GCP/intro_to_GCP_Essentials.ipynb +++ b/content/GCP/intro_to_GCP_Essentials.ipynb @@ -15,16 +15,10 @@ "4. [Introduction to the Cloud CLI](./04_intro_to_cli)\n", "4. [Using the Cloud Storage CLI](./05_cli_storage)\n", "6. [Running Analysis on the Cloud](./06_running_analysis)\n", - " and [Sharing Results](./06b_sharing_results)\n", + " and [Sharing Results](./06_sharing_results) (optional)\n", "7. [Monitoring Costs](./07_monitoring_costs)\n", "8. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)\n" ] - }, - { - "cell_type": "markdown", - "id": "78058910", - "metadata": {}, - "source": [] } ], "metadata": { diff --git a/content/_toc.yml b/content/_toc.yml index 8dc8e54..7f86396 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -31,6 +31,7 @@ parts: - file: Azure/06_running_analysis - file: Azure/07_monitoring_costs - file: Azure/08_cleaning_up_resources + - file: Azure/glossary - file: GCP/intro_to_GCP_Essentials sections: @@ -40,7 +41,7 @@ parts: - file: GCP/04_intro_to_cli - file: GCP/05_cli_storage - file: GCP/06_running_analysis - - file: GCP/06b_sharing_results + - file: GCP/06_sharing_results - file: GCP/07_monitoring_costs - file: GCP/08_cleaning_up_resources - file: GCP/glossary diff --git a/scripts/aws-create.sh b/scripts/aws-create.sh index 643f523..2f8a073 100755 --- a/scripts/aws-create.sh +++ b/scripts/aws-create.sh @@ -52,6 +52,7 @@ echo "+++ configuring VM" ssh ec2-user@$IP -A < .ssh/known_hosts git config --global color.ui auto @@ -60,6 +61,7 @@ git config --global pull.ff only git config --global user.name "$(git config user.name)" git config --global user.email "$(git config user.email)" git clone --branch $BRANCH $REPO +echo $(cat ~/.ssh/id_essentials-cloudshell.pub) >> .ssh/authorized_keys EOF echo "+++ configure ~/.ssh/$VM.config" diff --git a/scripts/gcp-cloudshell-jupyter-lab.sh b/scripts/gcp-cloudshell-jupyter-lab.sh new file mode 100755 index 0000000..311cd44 --- /dev/null +++ b/scripts/gcp-cloudshell-jupyter-lab.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +gcloud cloud-shell ssh --authorize-session --ssh-flag=-A --ssh-flag=-L8080:localhost:8080 --ssh-flag=-L8081:localhost:8081 \ + --command="cd .essentials && ./scripts/jupyter-lab.sh" diff --git a/scripts/gcp-cloudshell.sh b/scripts/gcp-cloudshell.sh new file mode 100755 index 0000000..4f071a6 --- /dev/null +++ b/scripts/gcp-cloudshell.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +gcloud cloud-shell ssh --authorize-session --ssh-flag=-A --ssh-flag=-L8080:localhost:8080 --ssh-flag=-L8081:localhost:8081 + diff --git a/scripts/gcp-create.sh b/scripts/gcp-create.sh index 8a0fd2d..efbc780 100755 --- a/scripts/gcp-create.sh +++ b/scripts/gcp-create.sh @@ -41,7 +41,7 @@ done echo "+++ configuring VM" gcloud compute ssh --zone=$ZONE $NAME@$VM --ssh-flag='-A' < .ssh/known_hosts git config --global color.ui auto git config --global push.default simple