diff --git a/content/Azure/01_intro_to_cloud_console.ipynb b/content/Azure/01_intro_to_cloud_console.ipynb index d3e30aa..ccdbac9 100644 --- a/content/Azure/01_intro_to_cloud_console.ipynb +++ b/content/Azure/01_intro_to_cloud_console.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "b6f03192", + "id": "b3a74289", "metadata": {}, "source": [ "# Introduction to the Azure Cloud Console\n", @@ -28,7 +28,7 @@ }, { "cell_type": "markdown", - "id": "abb35991", + "id": "926df543", "metadata": {}, "source": [ "## The Who, What, and Where of the Cloud\n", @@ -103,7 +103,7 @@ }, { "cell_type": "markdown", - "id": "ed7e581f", + "id": "b474c54d", "metadata": {}, "source": [ "## Services\n", @@ -112,7 +112,7 @@ "\n", "The Navigation Menu (often called the hamburger) is used to navigate to the various products, which are also sometimes called services.\n", "\n", - "![hamburger-navigation](img/hamburger-navigation.png)\n", + "![hamburger-navigation](images/hamburger-navigation.png)\n", "\n", " * Click on a product to navigate the product page\n", " * You can pin frequently used items on the top of this page by clicking on the pin icon. \n", @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "id": "4537d270", + "id": "a25ba0f1", "metadata": {}, "source": [ "## Everything Else\n", diff --git a/content/Azure/04_intro_to_cli.ipynb b/content/Azure/04_intro_to_cli.ipynb index 09f62d7..f0b0cea 100644 --- a/content/Azure/04_intro_to_cli.ipynb +++ b/content/Azure/04_intro_to_cli.ipynb @@ -2,18 +2,18 @@ "cells": [ { "cell_type": "markdown", - "id": "f5be7a6c", + "id": "0fb9d517", "metadata": {}, "source": [ "# Introduction to the Azure CLI\n", "\n", "Learner Questions\n", " * How do I use the Cloud CLI?\n", - " * How do I use `az`?\n", + " * How do I use az?\n", "\n", "Learning Objectives\n", " * Find and use the \"Cloud Shell\" in the web console\n", - " * Use basic cloud CLI commands (`az`)\n", + " * Use basic cloud CLI commands (az)\n", " * Determine basic user/account settings.\n", " * Use environment variables for configuration - No hard coding\n", " * Understand the importance of using variables for configuration.\n", @@ -24,7 +24,7 @@ }, { "cell_type": "markdown", - "id": "b609b6ed", + "id": "7ec8b5bf", "metadata": {}, "source": [ "## Cloud Automation\n", @@ -38,7 +38,7 @@ }, { "cell_type": "markdown", - "id": "e8cda207", + "id": "68bd896e", "metadata": {}, "source": [ "## Open the Cloud Shell\n", @@ -48,13 +48,13 @@ "To use the Azure Cloud Shell:\n", " \n", " * Open up a Cloud Shell by clicking on the cloud shell icon (greater than sign followed by underscore) on the blue bar just to the right of search window. On first launch, Cloud Shell prompts to create a resource group, storage account, and Azure Files share on your behalf. This will open a cloud shell at the bottum of your browser window. You can choose a bash shell or Power shell.\n", - " * If you want a new cloud shell in a different window, click the new cloud shell button (file icon with '+' in upper right corner) on the cloud shell menu bar. This will create a new cloud shell window tab. **Note:** you may need to allow popups or enable popups for this website depending on the browser you use. You can also pull up the tab to increase the shell screen size.\n", + " * If you want a new cloud shell in a different window, click the new cloud shell button (file icon with + in upper right corner) on the cloud shell menu bar. This will create a new cloud shell window tab. **Note:** you may need to allow popups or enable popups for this website depending on the browser you use. You can also pull up the tab to increase the shell screen size.\n", " * To quit cloud shell type exit at the prompt." ] }, { "cell_type": "markdown", - "id": "1799e8dd", + "id": "d9583740", "metadata": {}, "source": [ "## Exercise\n", @@ -67,62 +67,37 @@ }, { "cell_type": "markdown", - "id": "da18d22e", + "id": "366108a9", "metadata": {}, "source": [ "## Basic Cloud Commands\n", "\n", "The cloud can also be controlled using a command line interface (CLI) or a programming language such as Python. Collectively these tools interact with the cloud over a Application Programming Interface (API) and this capability forms the basis of the cloud, the ability to control infrastructure programmatically. \n", "\n", - "For information about installing the Azure CLI for your OS see: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli. If you are on your own OS you will need to login via `az login`.\n", + "For information about installing the Azure CLI for your OS see: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli. If you are on your own OS you will need to login via az login.\n", "\n", - "The `az` command is used to control most aspects of Azure Cloud." + "The az command is used to control most aspects of Azure Cloud." ] }, { "cell_type": "markdown", - "id": "f7b90dba", + "id": "a68f3190", "metadata": {}, "source": [ "Now let us look at your account information (your details will be different). " ] }, { - "cell_type": "code", - "execution_count": 1, - "id": "27170c24", + "cell_type": "markdown", + "id": "c61af998", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[\n", - " {\n", - " \"cloudName\": \"AzureCloud\",\n", - " \"id\": \"975b9702-e6b9-47b5-94a5-ca5f2b110ca8\",\n", - " \"isDefault\": true,\n", - " \"name\": \"class-dev\",\n", - " \"state\": \"Enabled\",\n", - " \"tenantId\": \"78ea4b46-9f08-4ef5-949b-2dae057c55d8\",\n", - " \"user\": {\n", - " \"assignedIdentityInfo\": \"MSI\",\n", - " \"name\": \"systemAssignedIdentity\",\n", - " \"type\": \"servicePrincipal\"\n", - " }\n", - " }\n", - "]\n", - "\u001b[0m" - ] - } - ], "source": [ - "az account list" + "# az account list" ] }, { "cell_type": "markdown", - "id": "e8feb7f2", + "id": "8b51b583", "metadata": {}, "source": [ "## Environment Variables\n", @@ -133,36 +108,24 @@ ] }, { - "cell_type": "code", - "execution_count": 2, - "id": "b8f74fe9-0943-4bd9-a91b-da09c001f8a1", + "cell_type": "markdown", + "id": "a67e26c4", "metadata": {}, - "outputs": [], "source": [ "TEST=1" ] }, { - "cell_type": "code", - "execution_count": 3, - "id": "b937f6c7-3697-4bc0-8e32-da0b0faec593", + "cell_type": "markdown", + "id": "2af53bf6", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "1\n" - ] - } - ], "source": [ "echo $TEST" ] }, { "cell_type": "markdown", - "id": "78fe7013-a5ae-498f-aeab-ba1f3c710de1", + "id": "5d784f5c", "metadata": {}, "source": [ "Although we do not always need to, we should always enclose shell variables in double quotes and enclose the variable name in curly braces (funny things may happen otherwise). In most cases it is easier to do this every time than try to figure out when it is possible to use the short form. We only use the short form here when it is used frequently and known to always work.\n", @@ -171,108 +134,68 @@ ] }, { - "cell_type": "code", - "execution_count": 4, - "id": "bae0587c-0379-40e3-9c31-16ba5505487a", + "cell_type": "markdown", + "id": "f70908e5", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "learner\n" - ] - } - ], "source": [ "echo \"${USER}\"" ] }, { - "cell_type": "code", - "execution_count": 5, - "id": "52b4add5-0d18-4faf-a304-b7e0ee8ba82e", + "cell_type": "markdown", + "id": "6c5bfd86", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "essentials\n" - ] - } - ], "source": [ "echo \"${HOSTNAME}\"" ] }, { "cell_type": "markdown", - "id": "0be0c5dc-8f65-4749-a56b-f47bbdeb6106", + "id": "0722637d", "metadata": {}, "source": [ - "If you are using the cloudshell then your hostname will look similar to `cc-2510c465-7677c9dc4c-fcl8k`" + "If you are using the cloudshell then your hostname will look similar to cc-2510c465-7677c9dc4c-fcl8k" ] }, { "cell_type": "markdown", - "id": "ef895f38-ad16-4f2d-aa18-ec913732dd69", + "id": "e07f6919", "metadata": {}, "source": [ "We can also use the output of commands to set string values." ] }, { - "cell_type": "code", - "execution_count": 6, - "id": "776806e4-209f-45fa-b14e-f771f5d5089d", + "cell_type": "markdown", + "id": "a90996d2", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Thu Dec 2 17:16:22 UTC 2021\n" - ] - } - ], "source": [ - "date" + "# date" ] }, { - "cell_type": "code", - "execution_count": 7, - "id": "560fde63-0eb8-4c52-9787-4debfe35ec23", + "cell_type": "markdown", + "id": "20a5bdaa", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Date: Thu Dec 2 17:16:23 UTC 2021\n" - ] - } - ], "source": [ - "echo \"Date: $(date)\"" + "# echo \"Date: $(date)\"" ] }, { "cell_type": "markdown", - "id": "49c2302b", + "id": "108b3e76", "metadata": { "tags": [] }, "source": [ - "First, as with all Azure work we need to create a Resource Group (RG). A resource group is a logical container into which Azure resources are deployed and managed. Now let us combine all of the above to create a `RG` but first let us set a RG name and region env variable so we do not need to use the actual value in future commands.\n", + "First, as with all Azure work we need to create a Resource Group (RG). A resource group is a logical container into which Azure resources are deployed and managed. Now let us combine all of the above to create a RG but first let us set a RG name and region env variable so we do not need to use the actual value in future commands.\n", "\n", "Example:" ] }, { "cell_type": "markdown", - "id": "1f73131a", + "id": "1c87694c", "metadata": {}, "source": [ "Define some enviroment variables." @@ -280,7 +203,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 13, "id": "d00a37e4", "metadata": {}, "outputs": [], @@ -290,175 +213,68 @@ ] }, { - "cell_type": "code", - "execution_count": 9, - "id": "d7988c45", + "cell_type": "markdown", + "id": "6b5db328", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "rg-essentials\n" - ] - } - ], "source": [ - "echo $RG" + " # echo $RG" ] }, { - "cell_type": "code", - "execution_count": 13, - "id": "49f484a6", + "cell_type": "markdown", + "id": "b11e6783", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"id\": \"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials\",\n", - " \"location\": \"eastus\",\n", - " \"managedBy\": null,\n", - " \"name\": \"rg-essentials\",\n", - " \"properties\": {\n", - " \"provisioningState\": \"Succeeded\"\n", - " },\n", - " \"tags\": null,\n", - " \"type\": \"Microsoft.Resources/resourceGroups\"\n", - "}\n", - "\u001b[0m" - ] - } - ], "source": [ - "az group create --name $RG --location $REGION" + "# az group create --name $RG --location $REGION" ] }, { "cell_type": "markdown", - "id": "7fe71e66", + "id": "a22ac088", "metadata": {}, "source": [ - "Now use the `RG` environment variable to get information about your resource group. This information is available at any time with the following command not just at creation." + "Now use the RG environment variable to get information about your resource group. This information is available at any time with the following command not just at creation." ] }, { - "cell_type": "code", - "execution_count": 14, - "id": "31b01f78-94b1-44f0-a2af-d657058941b2", + "cell_type": "markdown", + "id": "1a7686fc", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"id\": \"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials\",\n", - " \"location\": \"eastus\",\n", - " \"managedBy\": null,\n", - " \"name\": \"rg-essentials\",\n", - " \"properties\": {\n", - " \"provisioningState\": \"Succeeded\"\n", - " },\n", - " \"tags\": null,\n", - " \"type\": \"Microsoft.Resources/resourceGroups\"\n", - "}\n", - "\u001b[0m" - ] - } - ], "source": [ - "az group show --resource-group $RG" + "# az group show --resource-group $RG" ] }, { "cell_type": "markdown", - "id": "270e0abb-d5e9-476a-af91-060cc0ea0349", + "id": "0eb11c11", "metadata": {}, "source": [ "## Exercise\n", "FIXME:\n", " * What command shows the project number? (hint: we used it already)\n", " * What is the name of this variable? (hint: it is case sensitive)\n", - " * What command shows *just* the Project Number (hint: use `--format`)\n", - " * Set the `PROJECT_NUMBER` environment variable with the correct value. (hint: be careful with punctuation and matching punctuation)" + " * What command shows *just* the Project Number (hint: use --format)\n", + " * Set the *PROJECT_NUMBER* environment variable with the correct value. (hint: be careful with punctuation and matching punctuation)" ] }, { "cell_type": "markdown", - "id": "3b744cd4-3c7e-43f2-83ae-0ff02ec9de83", + "id": "8da1b71e", "metadata": {}, "source": [ "## Advanced Call-Out: Formating Output.\n", "\n", - "The following example shows how to output the configuration as `JSON`." + "The following example shows how to output the configuration as JSON." ] }, { - "cell_type": "code", - "execution_count": 18, - "id": "678a1735-8cad-403c-b302-639de1fb0f4b", + "cell_type": "markdown", + "id": "8e241cd0", "metadata": { "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1;39m[\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/cloudshell\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"centralus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"cloudshell\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials-dev\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"centralus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"rg-essentials-dev\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/NetworkWatcherRG\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"eastus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"NetworkWatcherRG\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/subscriptions/975b9702-e6b9-47b5-94a5-ca5f2b110ca8/resourceGroups/rg-essentials\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"location\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"eastus\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"managedBy\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"rg-essentials\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"properties\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"provisioningState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Succeeded\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"tags\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Microsoft.Resources/resourceGroups\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m\n", - "\u001b[1;39m]\u001b[0m\n" - ] - } - ], "source": [ - "az group list --output json | jq" + "# az group list --output json | jq" ] } ], diff --git a/content/Azure/05_intro_to_cloud_storage_cli.ipynb b/content/Azure/05_intro_to_cloud_storage_cli.ipynb index 5451fe8..ef2660e 100644 --- a/content/Azure/05_intro_to_cloud_storage_cli.ipynb +++ b/content/Azure/05_intro_to_cloud_storage_cli.ipynb @@ -45,9 +45,21 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "ename": "NameError", + "evalue": "name 'eastus' is not defined", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mRG\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"demo_rg\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mLOCATION\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meastus\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mBLOB_CON\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"demo_blob_cont\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mSTGE_ACCT\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"demo_stor_ac\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mCTR\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"demo_stge_ctr\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mNameError\u001b[0m: name 'eastus' is not defined" + ] + } + ], "source": [ "RG=\"demo_rg\"\n", "LOCATION=eastus\n", @@ -64,10 +76,8 @@ ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ "az group create --name $RG --location $LOCATION" ] @@ -87,12 +97,7 @@ "metadata": {}, "outputs": [], "source": [ - "az storage account create \\\n", - " --name $STGE_ACCT \\\n", - " --resource-group $RG \\\n", - " --location $LOCATION \\\n", - " --sku Standard_ZRS \\\n", - " --encryption-services blob\n" + "az storage account create --name $STGE_ACCT --resource-group $RG --location $LOCATION --sku Standard_ZRS --encryption-services blob" ] }, { @@ -105,10 +110,8 @@ ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ "az ad signed-in-user show --query objectId -o tsv | az role assignment create \\\n", " --role \"Storage Blob Data Contributor\" \\\n", @@ -124,15 +127,10 @@ ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "az storage container create \\\n", - " --account-name $STGE_ACCT \\\n", - " --name $CTR \\\n", - " --auth-mode login" + "az storage container create --account-name $STGE_ACCT --name $CTR --auth-mode login" ] }, { @@ -165,18 +163,27 @@ "Copy demo_file to your blob." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "az storage blob upload --account-name $STGE_ACCT --container-name $CTR --name demo_file --file demo_file --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify that the blob is there by Listing the blobs in the container. " + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "az storage blob upload \\\n", - " --account-name $STGE_ACCT \\\n", - " --container-name $CTR \\\n", - " --name demo_file \\\n", - " --file demo_file \\\n", - " --auth-mode login" + "az storage blob list --account-name $STGE_ACCT --container-name $CTR --output table --auth-mode login" ] }, { @@ -186,18 +193,43 @@ "Download the file we just uploaded." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "az storage blob download --account-name $STGE_ACCT --container-name $CTR --name demo_file --file demo_file_download file --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In Cloud Shell type the following command:" + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "az storage blob download \\\n", - " --account-name $STGE_ACCT \\\n", - " --container-name $STGE_ACCT \\\n", - " --name demo_file \\\n", - " --file ~/destination/path/for/file \\\n", - " --auth-mode login" + "ls demo_file_download" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify that the downloaded file is the same file we uploaded." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "diff demo_file demo_file_download" ] }, { @@ -211,6 +243,75 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "We will clean up the resources in this lesson in the reverse order that we created them. \n", + "\n", + "## Blob\n", + "Delete a blob" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage blob delete --account-name $STGE_ACCT --container-name $CTR --name demo_file --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify the blob was deleted" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage blob list --account-name $STGE_ACCT --container-name $CTR --output table --auth-mode login" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Storage Account\n", + "Delete the Storage Account." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage account delete -n $STGE_ACCT -g RG" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Verify that the Storage Account has been deleted." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "az storage account list | grep $STGE_ACCT" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Source Group\n", "By deleting the Resource Group, you will delete all resouce in that group.\n", "\n", "The --no-wait parameter keeps the CLI from blocking while the deletion takes place." diff --git a/content/Azure/06_running_analysis.ipynb b/content/Azure/06_running_analysis.ipynb index 2abf7ac..b5ce5ae 100644 --- a/content/Azure/06_running_analysis.ipynb +++ b/content/Azure/06_running_analysis.ipynb @@ -30,7 +30,7 @@ }, { "cell_type": "markdown", - "id": "6291edee-c2df-4941-9b8e-de42649640f9", + "id": "3a0fb40b", "metadata": {}, "source": [ "## Create a VM\n", @@ -47,7 +47,7 @@ }, { "cell_type": "markdown", - "id": "c87e200c-1d18-44f5-b4e6-28b697b04b3c", + "id": "26f414ef", "metadata": {}, "source": [ "## Connect to the VM\n", @@ -61,7 +61,7 @@ }, { "cell_type": "markdown", - "id": "44f32772-0abb-4fa7-a013-202fc646df5e", + "id": "41c471fb", "metadata": {}, "source": [ "```\n", @@ -84,7 +84,7 @@ }, { "cell_type": "markdown", - "id": "0ab6ad28-24bf-4c0f-b4dd-3be2ec25aaa7", + "id": "aa2c0f15", "metadata": {}, "source": [ "## Secure the VM\n", @@ -93,30 +93,9 @@ ] }, { - "cell_type": "code", - "execution_count": 1, - "id": "0f2df994-202e-4e3e-96c2-1536d8adb036", + "cell_type": "markdown", + "id": "d95ece63", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\n", - "Hit:2 http://deb.debian.org/debian buster InRelease\n", - "Hit:3 http://deb.debian.org/debian buster-updates InRelease\n", - "Hit:4 http://deb.debian.org/debian buster-backports InRelease\n", - "Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease\n", - "Hit:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease\n", - "Get:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease [5526 B]\n", - "Fetched 70.9 kB in 1s (91.3 kB/s) \u001b[0m\u001b[33m\n", - "Reading package lists... Done\n", - "Building dependency tree \n", - "Reading state information... Done\n", - "2 packages can be upgraded. Run 'apt list --upgradable' to see them.\n" - ] - } - ], "source": [ "sudo apt update\n", "sudo unattended-upgrades" @@ -124,7 +103,7 @@ }, { "cell_type": "markdown", - "id": "b7c4db9e-f098-41bc-80d2-b524444eec7f", + "id": "da999ac0", "metadata": {}, "source": [ "## Get Example Code\n", @@ -134,30 +113,18 @@ }, { "cell_type": "code", - "execution_count": 2, - "id": "05fadd30-4f35-4a93-b713-cc132131b948", + "execution_count": null, + "id": "0fc21993", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Reading package lists... Done\n", - "Building dependency tree \n", - "Reading state information... Done\n", - "git is already the newest version (1:2.20.1-2+deb10u3).\n", - "0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.\n" - ] - } - ], + "outputs": [], "source": [ "sudo apt-get install --yes git" ] }, { "cell_type": "code", - "execution_count": 3, - "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", + "execution_count": null, + "id": "3d30db18", "metadata": {}, "outputs": [], "source": [ @@ -166,28 +133,17 @@ }, { "cell_type": "code", - "execution_count": 4, - "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", + "execution_count": null, + "id": "4a86fa33", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Cloning into 'CLASS-Examples'...\n", - "remote: Enumerating objects: 66, done.\u001b[K\n", - "remote: Total 66 (delta 0), reused 0 (delta 0), pack-reused 66\u001b[K\n", - "Unpacking objects: 100% (66/66), done.\n" - ] - } - ], + "outputs": [], "source": [ "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" ] }, { "cell_type": "markdown", - "id": "3d2eb9fc-bec9-449d-a7ab-f3127f59d512", + "id": "5067f044", "metadata": {}, "source": [ "We now change the current directory to the `landsat` directory in the `CLASS-Examples` directory that was just created by the previous git command." @@ -195,8 +151,8 @@ }, { "cell_type": "code", - "execution_count": 5, - "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", + "execution_count": null, + "id": "7510d393", "metadata": {}, "outputs": [], "source": [ @@ -205,7 +161,7 @@ }, { "cell_type": "markdown", - "id": "6ea81d23-e1b0-4cac-a3ac-fed7a9723eca", + "id": "80788411", "metadata": {}, "source": [ "Your prompt should now change showing the current directory as follows.\n", @@ -216,32 +172,17 @@ }, { "cell_type": "code", - "execution_count": 6, - "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", + "execution_count": null, + "id": "6f213bfd", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 28\n", - "-rw-r--r-- 1 learner learner 960 Dec 17 19:00 ReadMe.md\n", - "-rw-r--r-- 1 learner learner 72 Dec 17 19:00 clean.sh\n", - "-rw-r--r-- 1 learner learner 613 Dec 17 19:00 combine.py\n", - "-rw-r--r-- 1 learner learner 280 Dec 17 19:00 download.sh\n", - "-rw-r--r-- 1 learner learner 314 Dec 17 19:00 get-index.sh\n", - "-rw-r--r-- 1 learner learner 76 Dec 17 19:00 search.json\n", - "-rw-r--r-- 1 learner learner 783 Dec 17 19:00 search.py\n" - ] - } - ], + "outputs": [], "source": [ "ls -l" ] }, { "cell_type": "markdown", - "id": "2b8b3144-7453-4350-a1cf-7fa74af2bcbf", + "id": "13168790", "metadata": {}, "source": [ "## Access the bucket\n", @@ -251,36 +192,17 @@ }, { "cell_type": "code", - "execution_count": 7, - "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", + "execution_count": null, + "id": "245f32a2", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://gcp-public-data-landsat/index.csv.gz\n", - "gs://gcp-public-data-landsat/LC08/\n", - "gs://gcp-public-data-landsat/LE07/\n", - "gs://gcp-public-data-landsat/LM01/\n", - "gs://gcp-public-data-landsat/LM02/\n", - "gs://gcp-public-data-landsat/LM03/\n", - "gs://gcp-public-data-landsat/LM04/\n", - "gs://gcp-public-data-landsat/LM05/\n", - "gs://gcp-public-data-landsat/LO08/\n", - "gs://gcp-public-data-landsat/LT04/\n", - "gs://gcp-public-data-landsat/LT05/\n", - "gs://gcp-public-data-landsat/LT08/\n" - ] - } - ], + "outputs": [], "source": [ "gsutil ls gs://gcp-public-data-landsat/" ] }, { "cell_type": "markdown", - "id": "89acffba-cbce-436c-98dd-05467b6675a6", + "id": "9153424d", "metadata": {}, "source": [ "The index file is a list of all the files in the bucket and we can use it to search and filter files.\n", @@ -290,50 +212,27 @@ }, { "cell_type": "code", - "execution_count": 8, - "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", + "execution_count": null, + "id": "e4ffd7e4", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "mkdir: created directory 'data'\n" - ] - } - ], + "outputs": [], "source": [ "mkdir -v data" ] }, { "cell_type": "code", - "execution_count": 9, - "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", + "execution_count": null, + "id": "f90b7dc4", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Copying gs://gcp-public-data-landsat/index.csv.gz...\n", - "==> NOTE: You are downloading one or more large file(s), which would \n", - "run significantly faster if you enabled sliced object downloads. This\n", - "feature is enabled by default but requires that compiled crcmod be\n", - "installed (see \"gsutil help crcmod\").\n", - "\n", - "\\ [1 files][761.0 MiB/761.0 MiB] 42.0 MiB/s \n", - "Operation completed over 1 objects/761.0 MiB. \n" - ] - } - ], + "outputs": [], "source": [ "gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/" ] }, { "cell_type": "markdown", - "id": "76b905b4-1c2a-4960-a14f-974b77f671cd", + "id": "0837cf43", "metadata": {}, "source": [ "We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using." @@ -341,8 +240,8 @@ }, { "cell_type": "code", - "execution_count": 10, - "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", + "execution_count": null, + "id": "9ed5a46b", "metadata": {}, "outputs": [], "source": [ @@ -359,19 +258,10 @@ }, { "cell_type": "code", - "execution_count": 11, - "id": "b005876c-f9af-43d6-80c6-f04295413b9b", + "execution_count": null, + "id": "5f4031b0", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 2.6G\n", - "-rw-r--r-- 1 learner learner 2.6G Dec 17 19:01 index.csv\n" - ] - } - ], + "outputs": [], "source": [ "ls -lh data" ] @@ -386,21 +276,10 @@ }, { "cell_type": "code", - "execution_count": 12, - "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", + "execution_count": null, + "id": "990f4c3f", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL\n", - "LM51940171984262KIS00,LM05_L1GS_194017_19840918_20210913_01_T2,LANDSAT_5,MSS,1984-09-18,01,T2,1984-09-18T09:36:27.0970000Z,L1GS,194,17,51.0,62.58023,60.38359,14.45307,19.13901,34758212,gs://gcp-public-data-landsat/LM05/01/194/017/LM05_L1GS_194017_19840918_20210913_01_T2\n", - "LM10420331972311AAA04,LM01_L1TP_042033_19721106_20180429_01_T2,LANDSAT_1,MSS,1972-11-06,01,T2,1972-11-06T17:49:00.5000000Z,L1TP,42,33,3.0,39.68891,37.707,-115.59896,-112.90344,26932395,gs://gcp-public-data-landsat/LM01/01/042/033/LM01_L1TP_042033_19721106_20180429_01_T2\n", - "LT50150151992246PAC00,LT05_L1TP_015015_19920902_20170121_01_T1,LANDSAT_5,TM,1992-09-02,01,T1,1992-09-02T15:01:34.1390130Z,L1TP,15,15,90.0,65.32118,63.11726,-67.3438,-61.89671,118785510,gs://gcp-public-data-landsat/LT05/01/015/015/LT05_L1TP_015015_19920902_20170121_01_T1\n" - ] - } - ], + "outputs": [], "source": [ "head --lines=4 data/index.csv" ] @@ -417,36 +296,10 @@ }, { "cell_type": "code", - "execution_count": 13, - "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", + "execution_count": null, + "id": "64ddd71e", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "#!/usr/bin/python3\n", - "import json\n", - "import csv\n", - "import sys\n", - "\n", - "# Example: Burr Oak Tree\n", - "# 38.899313,-92.464562 (Lat north+, Long west-) ; Landsat Path 025, Row 033\n", - "config=json.load(open(\"search.json\"))\n", - "lat,lon=config['lat'],config['lon']\n", - "landsat=config['landsat']\n", - "\n", - "reader=csv.reader(sys.stdin)\n", - "header=next(reader) # skip header\n", - "for l in reader:\n", - " SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL=l\n", - " west,east=float(WEST_LON),float(EAST_LON)\n", - " north,south=float(NORTH_LAT),float(SOUTH_LAT)\n", - " if SPACECRAFT_ID==landsat and north >= lat and south <= lat and west <= lon and east >= lon:\n", - " print(BASE_URL) # output BASE_URL\n" - ] - } - ], + "outputs": [], "source": [ "cat search.py" ] @@ -461,22 +314,10 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", + "execution_count": null, + "id": "b7d84cb5", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\n", - " \"lat\": 38.899313,\n", - " \"lon\": -92.464562,\n", - " \"landsat\": \"LANDSAT_8\"\n", - "}\n" - ] - } - ], + "outputs": [], "source": [ "cat search.json" ] @@ -491,22 +332,10 @@ }, { "cell_type": "code", - "execution_count": 15, - "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", + "execution_count": null, + "id": "a201fd0e", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20150127_20170302_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20161028_20170219_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20191224_20200110_01_T1\n" - ] - } - ], + "outputs": [], "source": [ "head --lines=200000 data/index.csv | python3 search.py" ] @@ -523,27 +352,10 @@ }, { "cell_type": "code", - "execution_count": 16, - "id": "3572c518-df83-4906-bfa6-a37bde2a5063", + "execution_count": null, + "id": "3ac78b3c", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "#!/bin/bash\n", - "\n", - "# Read space separated URL from STDIN and download \n", - "while read -r URL ; do\n", - " echo \"+++ $URL\"\n", - " # -m parallel\n", - " # -n no-clobber (do not re-download data)\n", - " # -r recursive (download all the data in the specified URL)\n", - " gsutil -m cp -n -r \"${URL}/\" data/\n", - "done\n" - ] - } - ], + "outputs": [], "source": [ "cat download.sh" ] @@ -558,64 +370,12 @@ }, { "cell_type": "code", - "execution_count": 17, - "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", + "execution_count": null, + "id": "2e93f083", "metadata": { - "scrolled": true, "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_ANG.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B3.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B9.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B8.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B6.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_BQA.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B11.TIF...\n", - "==> NOTE: You are downloading one or more large file(s), which would\n", - "run significantly faster if you enabled sliced object downloads. This\n", - "feature is enabled by default but requires that compiled crcmod be\n", - "installed (see \"gsutil help crcmod\").\n", - "\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B10.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_MTL.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B5.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B2.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B7.TIF...\n", - "| [14/14 files][997.2 MiB/997.2 MiB] 100% Done 65.3 MiB/s ETA 00:00:00 \n", - "Operation completed over 14 objects/997.2 MiB. \n", - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_ANG.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B11.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B10.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B8.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B9.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B3.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_BQA.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B6.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B7.TIF...\n", - "==> NOTE: You are downloading one or more large file(s), which would \n", - "run significantly faster if you enabled sliced object downloads. This\n", - "feature is enabled by default but requires that compiled crcmod be\n", - "installed (see \"gsutil help crcmod\").\n", - "\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B2.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_MTL.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B5.TIF...\n", - "- [14/14 files][979.2 MiB/979.2 MiB] 100% Done 14.9 MiB/s ETA 00:00:00 \n", - "Operation completed over 14 objects/979.2 MiB. \n" - ] - } - ], + "outputs": [], "source": [ "head --lines=200000 data/index.csv | python3 search.py | head --lines=2 | bash download.sh" ] @@ -630,21 +390,10 @@ }, { "cell_type": "code", - "execution_count": 18, - "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", + "execution_count": null, + "id": "271c7c4c", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 2665792\n", - "drwxr-xr-x 2 learner learner 4096 Dec 17 19:02 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20170913_20170928_01_T1\u001b[0m\n", - "drwxr-xr-x 2 learner learner 4096 Dec 17 19:01 \u001b[01;34mLC08_L1TP_025033_20211010_20211018_01_T1\u001b[0m\n", - "-rw-r--r-- 1 learner learner 2729754843 Dec 17 19:01 index.csv\n" - ] - } - ], + "outputs": [], "source": [ "ls -l data" ] @@ -661,57 +410,20 @@ }, { "cell_type": "code", - "execution_count": 19, - "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", + "execution_count": null, + "id": "e41b58c6", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "#!/usr/bin/python3\n", - "import os\n", - "import rasterio\n", - "\n", - "# Open the first directory in data, could walk entire tree\n", - "for dirname, dirs, files in os.walk('data'):\n", - " source = dirs[0]\n", - " break\n", - "\n", - "# Open band (B2/Blue) and copy metadata for result.\n", - "with rasterio.open(\"data/%s/%s_B2.TIF\" % (source, source)) as band2:\n", - " meta = band2.meta\n", - "\n", - "# Combine bands into PNG\n", - "meta.update(count = 3, driver='PNG')\n", - "with rasterio.open(\"output/result-%s.png\" % source, 'w+', **meta) as output:\n", - " for i in range(1, 4):\n", - " print(source, i)\n", - " output.write_band(i,rasterio.open(\"data/%s/%s_B%d.TIF\" % (source, source, i+3)).read(1))\n" - ] - } - ], + "outputs": [], "source": [ "cat combine.py" ] }, { "cell_type": "code", - "execution_count": 20, - "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", + "execution_count": null, + "id": "194c8cef", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Traceback (most recent call last):\n", - " File \"combine.py\", line 3, in \n", - " import rasterio\n", - "ModuleNotFoundError: No module named 'rasterio'\n" - ] - } - ], + "outputs": [], "source": [ "python3 combine.py\n", "/bin/true # ignore this line used for jupyter" @@ -728,471 +440,12 @@ }, { "cell_type": "code", - "execution_count": 21, - "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", + "execution_count": null, + "id": "659ed9d0", "metadata": { - "scrolled": true, "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "mkdir: created directory 'output/'\n", - "Reading package lists... Done\n", - "Building dependency tree \n", - "Reading state information... Done\n", - "The following additional packages will be installed:\n", - " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", - " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", - " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", - " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", - " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", - " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", - " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", - " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", - " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", - " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", - " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", - " python3-chardet python3-click python3-click-plugins python3-cligj\n", - " python3-colorama python3-html5lib python3-lxml python3-numpy\n", - " python3-pyparsing python3-snuggs python3-soupsieve python3-webencodings\n", - "Suggested packages:\n", - " geotiff-bin gdal-bin libgeotiff-epsg libhdf4-doc libhdf4-alt-dev hdf4-tools\n", - " liblcms2-utils libmyodbc odbc-postgresql tdsodbc unixodbc-bin ogdi-bin\n", - " poppler-utils ghostscript fonts-japanese-mincho | fonts-ipafont-mincho\n", - " fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai\n", - " fonts-arphic-uming fonts-nanum python-attr-doc python3-genshi\n", - " python3-lxml-dbg python-lxml-doc gfortran python-numpy-doc python3-pytest\n", - " python3-numpy-dbg python-pyparsing-doc\n", - "The following NEW packages will be installed:\n", - " fontconfig-config fonts-dejavu-core gdal-data libaec0 libarmadillo9\n", - " libarpack2 libblas3 libcharls2 libdap25 libdapclient6v5 libdapserver7v5\n", - " libepsilon1 libfontconfig1 libfreexl1 libfyba0 libgdal20 libgeos-3.7.1\n", - " libgeos-c1v5 libgeotiff2 libgfortran5 libgif7 libhdf4-0-alt libhdf5-103\n", - " libicu63 libjbig0 libjpeg62-turbo libkmlbase1 libkmlconvenience1 libkmldom1\n", - " libkmlengine1 libkmlregionator1 libkmlxsd1 liblapack3 liblcms2-2 libltdl7\n", - " libmariadb3 libminizip1 libnetcdf13 libnspr4 libnss3 libodbc1 libogdi3.2\n", - " libopenjp2-7 libpoppler82 libpq5 libproj13 libqhull7 libspatialite7\n", - " libsuperlu5 libsz2 libtiff5 liburiparser1 libwebp6 libxerces-c3.2 libxml2\n", - " libxslt1.1 mariadb-common mysql-common odbcinst odbcinst1debian2\n", - " poppler-data proj-bin proj-data python3-affine python3-attr python3-bs4\n", - " python3-chardet python3-click python3-click-plugins python3-cligj\n", - " python3-colorama python3-html5lib python3-lxml python3-numpy\n", - " python3-pyparsing python3-rasterio python3-snuggs python3-soupsieve\n", - " python3-webencodings\n", - "0 upgraded, 79 newly installed, 0 to remove and 2 not upgraded.\n", - "Need to get 46.8 MB of archives.\n", - "After this operation, 172 MB of additional disk space will be used.\n", - "Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libicu63 amd64 63.1-6+deb10u2 [8300 kB]\n", - "Get:2 http://deb.debian.org/debian buster/main amd64 poppler-data all 0.4.9-2 [1473 kB]\n", - "Get:3 http://deb.debian.org/debian buster/main amd64 fonts-dejavu-core all 2.37-1 [1068 kB]\n", - "Get:4 http://deb.debian.org/debian buster/main amd64 fontconfig-config all 2.13.1-2 [280 kB]\n", - "Get:5 http://deb.debian.org/debian buster/main amd64 gdal-data all 2.4.0+dfsg-1 [744 kB]\n", - "Get:6 http://deb.debian.org/debian buster/main amd64 libaec0 amd64 1.0.2-1 [20.1 kB]\n", - "Get:7 http://deb.debian.org/debian buster/main amd64 libgfortran5 amd64 8.3.0-6 [581 kB]\n", - "Get:8 http://deb.debian.org/debian buster/main amd64 libblas3 amd64 3.8.0-2 [148 kB]\n", - "Get:9 http://deb.debian.org/debian buster/main amd64 liblapack3 amd64 3.8.0-2 [2110 kB]\n", - "Get:10 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", - "Get:11 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", - "Get:12 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", - "Get:13 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", - "Get:14 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", - "Get:15 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", - "Get:16 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", - "Get:17 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", - "Get:18 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", - "Get:19 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", - "Get:20 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", - "Get:21 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", - "Get:22 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", - "Get:23 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", - "Get:24 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", - "Get:25 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", - "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", - "Get:27 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", - "Get:28 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", - "Get:29 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", - "Get:30 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", - "Get:31 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", - "Get:32 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", - "Get:33 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", - "Get:34 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", - "Get:35 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", - "Get:36 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", - "Get:37 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", - "Get:38 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", - "Get:39 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", - "Get:40 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", - "Get:41 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", - "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlconvenience1 amd64 1.3.0-7 [50.1 kB]\n", - "Get:43 http://deb.debian.org/debian buster/main amd64 libkmlregionator1 amd64 1.3.0-7 [25.1 kB]\n", - "Get:44 http://deb.debian.org/debian buster/main amd64 libkmlxsd1 amd64 1.3.0-7 [34.6 kB]\n", - "Get:45 http://deb.debian.org/debian buster/main amd64 mysql-common all 5.8+1.0.5 [7324 B]\n", - "Get:46 http://deb.debian.org/debian buster/main amd64 mariadb-common all 1:10.3.31-0+deb10u1 [32.7 kB]\n", - "Get:47 http://deb.debian.org/debian buster/main amd64 libmariadb3 amd64 1:10.3.31-0+deb10u1 [173 kB]\n", - "Get:48 http://deb.debian.org/debian buster/main amd64 libnetcdf13 amd64 1:4.6.2-1 [403 kB]\n", - "Get:49 http://deb.debian.org/debian buster/main amd64 libltdl7 amd64 2.4.6-9 [390 kB]\n", - "Get:50 http://deb.debian.org/debian buster/main amd64 libodbc1 amd64 2.3.6-0.1 [223 kB]\n", - "Get:51 http://deb.debian.org/debian buster/main amd64 libogdi3.2 amd64 3.2.1+ds-4 [241 kB]\n", - "Get:52 http://deb.debian.org/debian buster/main amd64 libopenjp2-7 amd64 2.3.0-2+deb10u2 [158 kB]\n", - "Get:53 http://deb.debian.org/debian buster/main amd64 liblcms2-2 amd64 2.9-3 [145 kB]\n", - "Get:54 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]\n", - "Get:55 http://deb.debian.org/debian buster/main amd64 libpoppler82 amd64 0.71.0-5 [1507 kB]\n", - "Get:56 http://deb.debian.org/debian buster/main amd64 libqhull7 amd64 2015.2-4 [196 kB]\n", - "Get:57 http://deb.debian.org/debian buster/main amd64 libspatialite7 amd64 4.3.0a-5+b2 [1256 kB]\n", - "Get:58 http://deb.debian.org/debian buster/main amd64 libxerces-c3.2 amd64 3.2.2+debian-1+deb10u1 [886 kB]\n", - "Get:59 http://deb.debian.org/debian buster/main amd64 odbcinst amd64 2.3.6-0.1 [48.4 kB]\n", - "Get:60 http://deb.debian.org/debian buster/main amd64 odbcinst1debian2 amd64 2.3.6-0.1 [78.1 kB]\n", - "Get:61 http://deb.debian.org/debian buster/main amd64 libgdal20 amd64 2.4.0+dfsg-1+b1 [6171 kB]\n", - "Get:62 http://deb.debian.org/debian buster/main amd64 libxslt1.1 amd64 1.1.32-2.2~deb10u1 [237 kB]\n", - "Get:63 http://deb.debian.org/debian buster/main amd64 proj-bin amd64 5.2.0-1 [105 kB]\n", - "Get:64 http://deb.debian.org/debian buster/main amd64 python3-affine all 2.2.2-1 [14.5 kB]\n", - "Get:65 http://deb.debian.org/debian buster/main amd64 python3-attr all 18.2.0-1 [37.3 kB]\n", - "Get:66 http://deb.debian.org/debian buster/main amd64 python3-soupsieve all 1.8+dfsg-1 [27.6 kB]\n", - "Get:67 http://deb.debian.org/debian buster/main amd64 python3-bs4 all 4.7.1-1 [94.1 kB]\n", - "Get:68 http://deb.debian.org/debian buster/main amd64 python3-chardet all 3.0.4-3 [80.5 kB]\n", - "Get:69 http://deb.debian.org/debian buster/main amd64 python3-colorama all 0.3.7-1 [18.1 kB]\n", - "Get:70 http://deb.debian.org/debian buster/main amd64 python3-click all 7.0-1 [73.6 kB]\n", - "Get:71 http://deb.debian.org/debian buster/main amd64 python3-click-plugins all 1.0.4-1 [7540 B]\n", - "Get:72 http://deb.debian.org/debian buster/main amd64 python3-cligj all 0.5.0-1 [8212 B]\n", - "Get:73 http://deb.debian.org/debian buster/main amd64 python3-webencodings all 0.5.1-1 [10.9 kB]\n", - "Get:74 http://deb.debian.org/debian buster/main amd64 python3-html5lib all 1.0.1-1 [89.5 kB]\n", - "Get:75 http://deb.debian.org/debian buster/main amd64 python3-lxml amd64 4.3.2-1+deb10u3 [1162 kB]\n", - "Get:76 http://deb.debian.org/debian buster/main amd64 python3-numpy amd64 1:1.16.2-1 [2119 kB]\n", - "Get:77 http://deb.debian.org/debian buster/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [89.6 kB]\n", - "Get:78 http://deb.debian.org/debian buster/main amd64 python3-snuggs all 1.4.3-1 [7228 B]\n", - "Get:79 http://deb.debian.org/debian buster/main amd64 python3-rasterio amd64 1.0.21-1 [818 kB]\n", - "Fetched 46.8 MB in 1s (85.4 MB/s) \n", - "Extracting templates from packages: 100%\n", - "Preconfiguring packages ...\n", - "Selecting previously unselected package poppler-data.\n", - "(Reading database ... 57529 files and directories currently installed.)\n", - "Preparing to unpack .../00-poppler-data_0.4.9-2_all.deb ...\n", - "Unpacking poppler-data (0.4.9-2) ...\n", - "Selecting previously unselected package fonts-dejavu-core.\n", - "Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ...\n", - "Unpacking fonts-dejavu-core (2.37-1) ...\n", - "Selecting previously unselected package fontconfig-config.\n", - "Preparing to unpack .../02-fontconfig-config_2.13.1-2_all.deb ...\n", - "Unpacking fontconfig-config (2.13.1-2) ...\n", - "Selecting previously unselected package gdal-data.\n", - "Preparing to unpack .../03-gdal-data_2.4.0+dfsg-1_all.deb ...\n", - "Unpacking gdal-data (2.4.0+dfsg-1) ...\n", - "Selecting previously unselected package libaec0:amd64.\n", - "Preparing to unpack .../04-libaec0_1.0.2-1_amd64.deb ...\n", - "Unpacking libaec0:amd64 (1.0.2-1) ...\n", - "Selecting previously unselected package libgfortran5:amd64.\n", - "Preparing to unpack .../05-libgfortran5_8.3.0-6_amd64.deb ...\n", - "Unpacking libgfortran5:amd64 (8.3.0-6) ...\n", - "Selecting previously unselected package libblas3:amd64.\n", - "Preparing to unpack .../06-libblas3_3.8.0-2_amd64.deb ...\n", - "Unpacking libblas3:amd64 (3.8.0-2) ...\n", - "Selecting previously unselected package liblapack3:amd64.\n", - "Preparing to unpack .../07-liblapack3_3.8.0-2_amd64.deb ...\n", - "Unpacking liblapack3:amd64 (3.8.0-2) ...\n", - "Selecting previously unselected package libarpack2:amd64.\n", - "Preparing to unpack .../08-libarpack2_3.7.0-2_amd64.deb ...\n", - "Unpacking libarpack2:amd64 (3.7.0-2) ...\n", - "Selecting previously unselected package libsuperlu5:amd64.\n", - "Preparing to unpack .../09-libsuperlu5_5.2.1+dfsg1-4_amd64.deb ...\n", - "Unpacking libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", - "Selecting previously unselected package libarmadillo9.\n", - "Preparing to unpack .../10-libarmadillo9_1%3a9.200.7+dfsg-1_amd64.deb ...\n", - "Unpacking libarmadillo9 (1:9.200.7+dfsg-1) ...\n", - "Selecting previously unselected package libcharls2:amd64.\n", - "Preparing to unpack .../11-libcharls2_2.0.0+dfsg-1_amd64.deb ...\n", - "Unpacking libcharls2:amd64 (2.0.0+dfsg-1) ...\n", - "Selecting previously unselected package libicu63:amd64.\n", - "Preparing to unpack .../12-libicu63_63.1-6+deb10u2_amd64.deb ...\n", - "Unpacking libicu63:amd64 (63.1-6+deb10u2) ...\n", - "Selecting previously unselected package libxml2:amd64.\n", - "Preparing to unpack .../13-libxml2_2.9.4+dfsg1-7+deb10u2_amd64.deb ...\n", - "Unpacking libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", - "Selecting previously unselected package libdap25:amd64.\n", - "Preparing to unpack .../14-libdap25_3.20.3-1_amd64.deb ...\n", - "Unpacking libdap25:amd64 (3.20.3-1) ...\n", - "Selecting previously unselected package libdapclient6v5:amd64.\n", - "Preparing to unpack .../15-libdapclient6v5_3.20.3-1_amd64.deb ...\n", - "Unpacking libdapclient6v5:amd64 (3.20.3-1) ...\n", - "Selecting previously unselected package libdapserver7v5:amd64.\n", - "Preparing to unpack .../16-libdapserver7v5_3.20.3-1_amd64.deb ...\n", - "Unpacking libdapserver7v5:amd64 (3.20.3-1) ...\n", - "Selecting previously unselected package libepsilon1:amd64.\n", - "Preparing to unpack .../17-libepsilon1_0.9.2+dfsg-4_amd64.deb ...\n", - "Unpacking libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", - "Selecting previously unselected package libfontconfig1:amd64.\n", - "Preparing to unpack .../18-libfontconfig1_2.13.1-2_amd64.deb ...\n", - "Unpacking libfontconfig1:amd64 (2.13.1-2) ...\n", - "Selecting previously unselected package libfreexl1:amd64.\n", - "Preparing to unpack .../19-libfreexl1_1.0.5-3_amd64.deb ...\n", - "Unpacking libfreexl1:amd64 (1.0.5-3) ...\n", - "Selecting previously unselected package libfyba0:amd64.\n", - "Preparing to unpack .../20-libfyba0_4.1.1-6_amd64.deb ...\n", - "Unpacking libfyba0:amd64 (4.1.1-6) ...\n", - "Selecting previously unselected package libgeos-3.7.1:amd64.\n", - "Preparing to unpack .../21-libgeos-3.7.1_3.7.1-1_amd64.deb ...\n", - "Unpacking libgeos-3.7.1:amd64 (3.7.1-1) ...\n", - "Selecting previously unselected package libgeos-c1v5:amd64.\n", - "Preparing to unpack .../22-libgeos-c1v5_3.7.1-1_amd64.deb ...\n", - "Unpacking libgeos-c1v5:amd64 (3.7.1-1) ...\n", - "Selecting previously unselected package proj-data.\n", - "Preparing to unpack .../23-proj-data_5.2.0-1_all.deb ...\n", - "Unpacking proj-data (5.2.0-1) ...\n", - "Selecting previously unselected package libproj13:amd64.\n", - "Preparing to unpack .../24-libproj13_5.2.0-1_amd64.deb ...\n", - "Unpacking libproj13:amd64 (5.2.0-1) ...\n", - "Selecting previously unselected package libjbig0:amd64.\n", - "Preparing to unpack .../25-libjbig0_2.1-3.1+b2_amd64.deb ...\n", - "Unpacking libjbig0:amd64 (2.1-3.1+b2) ...\n", - "Selecting previously unselected package libjpeg62-turbo:amd64.\n", - "Preparing to unpack .../26-libjpeg62-turbo_1%3a1.5.2-2+deb10u1_amd64.deb ...\n", - "Unpacking libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", - "Selecting previously unselected package libwebp6:amd64.\n", - "Preparing to unpack .../27-libwebp6_0.6.1-2+deb10u1_amd64.deb ...\n", - "Unpacking libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", - "Selecting previously unselected package libtiff5:amd64.\n", - "Preparing to unpack .../28-libtiff5_4.1.0+git191117-2~deb10u3_amd64.deb ...\n", - "Unpacking libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", - "Selecting previously unselected package libgeotiff2:amd64.\n", - "Preparing to unpack .../29-libgeotiff2_1.4.3-1_amd64.deb ...\n", - "Unpacking libgeotiff2:amd64 (1.4.3-1) ...\n", - "Selecting previously unselected package libgif7:amd64.\n", - "Preparing to unpack .../30-libgif7_5.1.4-3_amd64.deb ...\n", - "Unpacking libgif7:amd64 (5.1.4-3) ...\n", - "Selecting previously unselected package libhdf4-0-alt.\n", - "Preparing to unpack .../31-libhdf4-0-alt_4.2.13-4_amd64.deb ...\n", - "Unpacking libhdf4-0-alt (4.2.13-4) ...\n", - "Selecting previously unselected package libsz2:amd64.\n", - "Preparing to unpack .../32-libsz2_1.0.2-1_amd64.deb ...\n", - "Unpacking libsz2:amd64 (1.0.2-1) ...\n", - "Selecting previously unselected package libhdf5-103:amd64.\n", - "Preparing to unpack .../33-libhdf5-103_1.10.4+repack-10_amd64.deb ...\n", - "Unpacking libhdf5-103:amd64 (1.10.4+repack-10) ...\n", - "Selecting previously unselected package libminizip1:amd64.\n", - "Preparing to unpack .../34-libminizip1_1.1-8+b1_amd64.deb ...\n", - "Unpacking libminizip1:amd64 (1.1-8+b1) ...\n", - "Selecting previously unselected package liburiparser1:amd64.\n", - "Preparing to unpack .../35-liburiparser1_0.9.1-1_amd64.deb ...\n", - "Unpacking liburiparser1:amd64 (0.9.1-1) ...\n", - "Selecting previously unselected package libkmlbase1:amd64.\n", - "Preparing to unpack .../36-libkmlbase1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlbase1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmldom1:amd64.\n", - "Preparing to unpack .../37-libkmldom1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmldom1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlengine1:amd64.\n", - "Preparing to unpack .../38-libkmlengine1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlengine1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlconvenience1:amd64.\n", - "Preparing to unpack .../39-libkmlconvenience1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlconvenience1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlregionator1:amd64.\n", - "Preparing to unpack .../40-libkmlregionator1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlregionator1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package libkmlxsd1:amd64.\n", - "Preparing to unpack .../41-libkmlxsd1_1.3.0-7_amd64.deb ...\n", - "Unpacking libkmlxsd1:amd64 (1.3.0-7) ...\n", - "Selecting previously unselected package mysql-common.\n", - "Preparing to unpack .../42-mysql-common_5.8+1.0.5_all.deb ...\n", - "Unpacking mysql-common (5.8+1.0.5) ...\n", - "Selecting previously unselected package mariadb-common.\n", - "Preparing to unpack .../43-mariadb-common_1%3a10.3.31-0+deb10u1_all.deb ...\n", - "Unpacking mariadb-common (1:10.3.31-0+deb10u1) ...\n", - "Selecting previously unselected package libmariadb3:amd64.\n", - "Preparing to unpack .../44-libmariadb3_1%3a10.3.31-0+deb10u1_amd64.deb ...\n", - "Unpacking libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", - "Selecting previously unselected package libnetcdf13:amd64.\n", - "Preparing to unpack .../45-libnetcdf13_1%3a4.6.2-1_amd64.deb ...\n", - "Unpacking libnetcdf13:amd64 (1:4.6.2-1) ...\n", - "Selecting previously unselected package libltdl7:amd64.\n", - "Preparing to unpack .../46-libltdl7_2.4.6-9_amd64.deb ...\n", - "Unpacking libltdl7:amd64 (2.4.6-9) ...\n", - "Selecting previously unselected package libodbc1:amd64.\n", - "Preparing to unpack .../47-libodbc1_2.3.6-0.1_amd64.deb ...\n", - "Unpacking libodbc1:amd64 (2.3.6-0.1) ...\n", - "Selecting previously unselected package libogdi3.2.\n", - "Preparing to unpack .../48-libogdi3.2_3.2.1+ds-4_amd64.deb ...\n", - "Unpacking libogdi3.2 (3.2.1+ds-4) ...\n", - "Selecting previously unselected package libopenjp2-7:amd64.\n", - "Preparing to unpack .../49-libopenjp2-7_2.3.0-2+deb10u2_amd64.deb ...\n", - "Unpacking libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", - "Selecting previously unselected package liblcms2-2:amd64.\n", - "Preparing to unpack .../50-liblcms2-2_2.9-3_amd64.deb ...\n", - "Unpacking liblcms2-2:amd64 (2.9-3) ...\n", - "Selecting previously unselected package libnspr4:amd64.\n", - "Preparing to unpack .../51-libnspr4_2%3a4.20-1_amd64.deb ...\n", - "Unpacking libnspr4:amd64 (2:4.20-1) ...\n", - "Selecting previously unselected package libnss3:amd64.\n", - "Preparing to unpack .../52-libnss3_2%3a3.42.1-1+deb10u4_amd64.deb ...\n", - "Unpacking libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", - "Selecting previously unselected package libpoppler82:amd64.\n", - "Preparing to unpack .../53-libpoppler82_0.71.0-5_amd64.deb ...\n", - "Unpacking libpoppler82:amd64 (0.71.0-5) ...\n", - "Selecting previously unselected package libpq5:amd64.\n", - "Preparing to unpack .../54-libpq5_11.14-0+deb10u1_amd64.deb ...\n", - "Unpacking libpq5:amd64 (11.14-0+deb10u1) ...\n", - "Selecting previously unselected package libqhull7:amd64.\n", - "Preparing to unpack .../55-libqhull7_2015.2-4_amd64.deb ...\n", - "Unpacking libqhull7:amd64 (2015.2-4) ...\n", - "Selecting previously unselected package libspatialite7:amd64.\n", - "Preparing to unpack .../56-libspatialite7_4.3.0a-5+b2_amd64.deb ...\n", - "Unpacking libspatialite7:amd64 (4.3.0a-5+b2) ...\n", - "Selecting previously unselected package libxerces-c3.2:amd64.\n", - "Preparing to unpack .../57-libxerces-c3.2_3.2.2+debian-1+deb10u1_amd64.deb ...\n", - "Unpacking libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", - "Selecting previously unselected package odbcinst.\n", - "Preparing to unpack .../58-odbcinst_2.3.6-0.1_amd64.deb ...\n", - "Unpacking odbcinst (2.3.6-0.1) ...\n", - "Selecting previously unselected package odbcinst1debian2:amd64.\n", - "Preparing to unpack .../59-odbcinst1debian2_2.3.6-0.1_amd64.deb ...\n", - "Unpacking odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", - "Selecting previously unselected package libgdal20.\n", - "Preparing to unpack .../60-libgdal20_2.4.0+dfsg-1+b1_amd64.deb ...\n", - "Unpacking libgdal20 (2.4.0+dfsg-1+b1) ...\n", - "Selecting previously unselected package libxslt1.1:amd64.\n", - "Preparing to unpack .../61-libxslt1.1_1.1.32-2.2~deb10u1_amd64.deb ...\n", - "Unpacking libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", - "Selecting previously unselected package proj-bin.\n", - "Preparing to unpack .../62-proj-bin_5.2.0-1_amd64.deb ...\n", - "Unpacking proj-bin (5.2.0-1) ...\n", - "Selecting previously unselected package python3-affine.\n", - "Preparing to unpack .../63-python3-affine_2.2.2-1_all.deb ...\n", - "Unpacking python3-affine (2.2.2-1) ...\n", - "Selecting previously unselected package python3-attr.\n", - "Preparing to unpack .../64-python3-attr_18.2.0-1_all.deb ...\n", - "Unpacking python3-attr (18.2.0-1) ...\n", - "Selecting previously unselected package python3-soupsieve.\n", - "Preparing to unpack .../65-python3-soupsieve_1.8+dfsg-1_all.deb ...\n", - "Unpacking python3-soupsieve (1.8+dfsg-1) ...\n", - "Selecting previously unselected package python3-bs4.\n", - "Preparing to unpack .../66-python3-bs4_4.7.1-1_all.deb ...\n", - "Unpacking python3-bs4 (4.7.1-1) ...\n", - "Selecting previously unselected package python3-chardet.\n", - "Preparing to unpack .../67-python3-chardet_3.0.4-3_all.deb ...\n", - "Unpacking python3-chardet (3.0.4-3) ...\n", - "Selecting previously unselected package python3-colorama.\n", - "Preparing to unpack .../68-python3-colorama_0.3.7-1_all.deb ...\n", - "Unpacking python3-colorama (0.3.7-1) ...\n", - "Selecting previously unselected package python3-click.\n", - "Preparing to unpack .../69-python3-click_7.0-1_all.deb ...\n", - "Unpacking python3-click (7.0-1) ...\n", - "Selecting previously unselected package python3-click-plugins.\n", - "Preparing to unpack .../70-python3-click-plugins_1.0.4-1_all.deb ...\n", - "Unpacking python3-click-plugins (1.0.4-1) ...\n", - "Selecting previously unselected package python3-cligj.\n", - "Preparing to unpack .../71-python3-cligj_0.5.0-1_all.deb ...\n", - "Unpacking python3-cligj (0.5.0-1) ...\n", - "Selecting previously unselected package python3-webencodings.\n", - "Preparing to unpack .../72-python3-webencodings_0.5.1-1_all.deb ...\n", - "Unpacking python3-webencodings (0.5.1-1) ...\n", - "Selecting previously unselected package python3-html5lib.\n", - "Preparing to unpack .../73-python3-html5lib_1.0.1-1_all.deb ...\n", - "Unpacking python3-html5lib (1.0.1-1) ...\n", - "Selecting previously unselected package python3-lxml:amd64.\n", - "Preparing to unpack .../74-python3-lxml_4.3.2-1+deb10u3_amd64.deb ...\n", - "Unpacking python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", - "Selecting previously unselected package python3-numpy.\n", - "Preparing to unpack .../75-python3-numpy_1%3a1.16.2-1_amd64.deb ...\n", - "Unpacking python3-numpy (1:1.16.2-1) ...\n", - "Selecting previously unselected package python3-pyparsing.\n", - "Preparing to unpack .../76-python3-pyparsing_2.2.0+dfsg1-2_all.deb ...\n", - "Unpacking python3-pyparsing (2.2.0+dfsg1-2) ...\n", - "Selecting previously unselected package python3-snuggs.\n", - "Preparing to unpack .../77-python3-snuggs_1.4.3-1_all.deb ...\n", - "Unpacking python3-snuggs (1.4.3-1) ...\n", - "Selecting previously unselected package python3-rasterio.\n", - "Preparing to unpack .../78-python3-rasterio_1.0.21-1_amd64.deb ...\n", - "Unpacking python3-rasterio (1.0.21-1) ...\n", - "Setting up liblcms2-2:amd64 (2.9-3) ...\n", - "Setting up python3-attr (18.2.0-1) ...\n", - "Setting up mysql-common (5.8+1.0.5) ...\n", - "update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", - "Setting up proj-data (5.2.0-1) ...\n", - "Setting up python3-colorama (0.3.7-1) ...\n", - "Setting up libcharls2:amd64 (2.0.0+dfsg-1) ...\n", - "Setting up libminizip1:amd64 (1.1-8+b1) ...\n", - "Setting up libproj13:amd64 (5.2.0-1) ...\n", - "Setting up libpq5:amd64 (11.14-0+deb10u1) ...\n", - "Setting up proj-bin (5.2.0-1) ...\n", - "Setting up libqhull7:amd64 (2015.2-4) ...\n", - "Setting up python3-click (7.0-1) ...\n", - "Setting up libepsilon1:amd64 (0.9.2+dfsg-4) ...\n", - "Setting up libjbig0:amd64 (2.1-3.1+b2) ...\n", - "Setting up python3-webencodings (0.5.1-1) ...\n", - "Setting up libgeos-3.7.1:amd64 (3.7.1-1) ...\n", - "Setting up libaec0:amd64 (1.0.2-1) ...\n", - "Setting up libicu63:amd64 (63.1-6+deb10u2) ...\n", - "Setting up gdal-data (2.4.0+dfsg-1) ...\n", - "Setting up poppler-data (0.4.9-2) ...\n", - "Setting up mariadb-common (1:10.3.31-0+deb10u1) ...\n", - "update-alternatives: using /etc/mysql/mariadb.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode\n", - "Setting up python3-chardet (3.0.4-3) ...\n", - "Setting up libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...\n", - "Setting up python3-pyparsing (2.2.0+dfsg1-2) ...\n", - "Setting up libnspr4:amd64 (2:4.20-1) ...\n", - "Setting up libwebp6:amd64 (0.6.1-2+deb10u1) ...\n", - "Setting up libgeos-c1v5:amd64 (3.7.1-1) ...\n", - "Setting up python3-click-plugins (1.0.4-1) ...\n", - "Setting up libmariadb3:amd64 (1:10.3.31-0+deb10u1) ...\n", - "Setting up fonts-dejavu-core (2.37-1) ...\n", - "Setting up python3-html5lib (1.0.1-1) ...\n", - "Setting up libltdl7:amd64 (2.4.6-9) ...\n", - "Setting up libgfortran5:amd64 (8.3.0-6) ...\n", - "Setting up libhdf4-0-alt (4.2.13-4) ...\n", - "Setting up libgif7:amd64 (5.1.4-3) ...\n", - "Setting up liburiparser1:amd64 (0.9.1-1) ...\n", - "Setting up libfreexl1:amd64 (1.0.5-3) ...\n", - "Setting up libfyba0:amd64 (4.1.1-6) ...\n", - "Setting up libkmlbase1:amd64 (1.3.0-7) ...\n", - "Setting up libopenjp2-7:amd64 (2.3.0-2+deb10u2) ...\n", - "Setting up libtiff5:amd64 (4.1.0+git191117-2~deb10u3) ...\n", - "Setting up python3-affine (2.2.2-1) ...\n", - "Setting up libxml2:amd64 (2.9.4+dfsg1-7+deb10u2) ...\n", - "Setting up python3-soupsieve (1.8+dfsg-1) ...\n", - "Setting up libsz2:amd64 (1.0.2-1) ...\n", - "Setting up libkmlxsd1:amd64 (1.3.0-7) ...\n", - "Setting up libkmldom1:amd64 (1.3.0-7) ...\n", - "Setting up libspatialite7:amd64 (4.3.0a-5+b2) ...\n", - "Setting up libogdi3.2 (3.2.1+ds-4) ...\n", - "Setting up libxerces-c3.2:amd64 (3.2.2+debian-1+deb10u1) ...\n", - "Setting up libkmlengine1:amd64 (1.3.0-7) ...\n", - "Setting up libkmlconvenience1:amd64 (1.3.0-7) ...\n", - "Setting up fontconfig-config (2.13.1-2) ...\n", - "Setting up python3-cligj (0.5.0-1) ...\n", - "Setting up libdap25:amd64 (3.20.3-1) ...\n", - "Setting up libnss3:amd64 (2:3.42.1-1+deb10u4) ...\n", - "Setting up libdapserver7v5:amd64 (3.20.3-1) ...\n", - "Setting up python3-bs4 (4.7.1-1) ...\n", - "Setting up libgeotiff2:amd64 (1.4.3-1) ...\n", - "Setting up libblas3:amd64 (3.8.0-2) ...\n", - "update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode\n", - "Setting up libkmlregionator1:amd64 (1.3.0-7) ...\n", - "Setting up libhdf5-103:amd64 (1.10.4+repack-10) ...\n", - "Setting up libodbc1:amd64 (2.3.6-0.1) ...\n", - "Setting up libxslt1.1:amd64 (1.1.32-2.2~deb10u1) ...\n", - "Setting up libdapclient6v5:amd64 (3.20.3-1) ...\n", - "Setting up libfontconfig1:amd64 (2.13.1-2) ...\n", - "Setting up liblapack3:amd64 (3.8.0-2) ...\n", - "update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3 to provide /usr/lib/x86_64-linux-gnu/liblapack.so.3 (liblapack.so.3-x86_64-linux-gnu) in auto mode\n", - "Setting up libarpack2:amd64 (3.7.0-2) ...\n", - "Setting up libsuperlu5:amd64 (5.2.1+dfsg1-4) ...\n", - "Setting up libpoppler82:amd64 (0.71.0-5) ...\n", - "Setting up libnetcdf13:amd64 (1:4.6.2-1) ...\n", - "Setting up python3-numpy (1:1.16.2-1) ...\n", - "Setting up python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", - "Setting up libarmadillo9 (1:9.200.7+dfsg-1) ...\n", - "Setting up python3-snuggs (1.4.3-1) ...\n", - "Setting up odbcinst (2.3.6-0.1) ...\n", - "Setting up odbcinst1debian2:amd64 (2.3.6-0.1) ...\n", - "Setting up libgdal20 (2.4.0+dfsg-1+b1) ...\n", - "Setting up python3-rasterio (1.0.21-1) ...\n", - "Processing triggers for libc-bin (2.28-10) ...\n", - "Processing triggers for man-db (2.8.5-2) ...\n" - ] - } - ], + "outputs": [], "source": [ "mkdir -v output/\n", "sudo apt-get install python3-rasterio --yes" @@ -1200,21 +453,10 @@ }, { "cell_type": "code", - "execution_count": 22, - "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", + "execution_count": null, + "id": "45a25550", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "LC08_L1TP_025033_20211010_20211018_01_T1 1\n", - "LC08_L1TP_025033_20211010_20211018_01_T1 2\n", - "LC08_L1TP_025033_20211010_20211018_01_T1 3\n", - "ERROR 4: output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png: No such file or directory\n" - ] - } - ], + "outputs": [], "source": [ "/usr/bin/python3 combine.py" ] @@ -1237,20 +479,10 @@ }, { "cell_type": "code", - "execution_count": 23, - "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", + "execution_count": null, + "id": "7f3f6f0b", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "total 195M\n", - "-rw-r--r-- 1 learner learner 195M Dec 17 19:03 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20211010_20211018_01_T1.png\u001b[0m\n", - "-rw-r--r-- 1 learner learner 910 Dec 17 19:03 result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n" - ] - } - ], + "outputs": [], "source": [ "ls -lh output" ] @@ -1278,18 +510,10 @@ }, { "cell_type": "code", - "execution_count": 24, - "id": "f36cb8c5-f305-4cb2-a5cc-0c9fd8592fb4", + "execution_count": null, + "id": "715291b5", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "bucket: essentials-learner-2021-12-17 region: us-west2\n" - ] - } - ], + "outputs": [], "source": [ "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", "REGION=\"us-west2\"\n", @@ -1297,19 +521,9 @@ ] }, { - "cell_type": "code", - "execution_count": 25, - "id": "c2ae2b74-5e93-4c55-8bd7-63337f7dcbb8", + "cell_type": "markdown", + "id": "920ff5ef", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Creating gs://essentials-learner-2021-12-17/...\n" - ] - } - ], "source": [ "gsutil mb -b on -l $REGION --pap enforced \"gs://$BUCKET\"" ] @@ -1324,33 +538,10 @@ }, { "cell_type": "code", - "execution_count": 26, - "id": "681e6b1d-98bb-448a-a57e-f5674214effd", + "execution_count": null, + "id": "9340b10d", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml [Content-Type=application/xml]...\n", - "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png [Content-Type=image/png]...\n", - "==> NOTE: You are uploading one or more large file(s), which would run \n", - "significantly faster if you enable parallel composite uploads. This\n", - "feature can be enabled by editing the\n", - "\"parallel_composite_upload_threshold\" value in your .boto\n", - "configuration file. However, note that if you do this large files will\n", - "be uploaded as `composite objects\n", - "`_,which\n", - "means that any user who downloads such objects will need to have a\n", - "compiled crcmod installed (see \"gsutil help crcmod\"). This is because\n", - "without a compiled crcmod, computing checksums on composite objects is\n", - "so slow that gsutil disables downloads of composite objects.\n", - "\n", - "\\ [2/2 files][194.0 MiB/194.0 MiB] 100% Done \n", - "Operation completed over 2 objects/194.0 MiB. \n" - ] - } - ], + "outputs": [], "source": [ "gsutil -m cp -r \"output\" \"gs://$BUCKET\"" ] @@ -1365,38 +556,20 @@ }, { "cell_type": "code", - "execution_count": 27, - "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", + "execution_count": null, + "id": "754c7eb4", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://essentials-learner-2021-12-17/output/\n" - ] - } - ], + "outputs": [], "source": [ "gsutil ls \"gs://$BUCKET\"" ] }, { "cell_type": "code", - "execution_count": 28, - "id": "b1ea18e9-5861-4479-9948-3303952dee8a", + "execution_count": null, + "id": "0e927a0a", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "194.05 MiB 2021-12-17T19:03:54Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", - " 910 B 2021-12-17T19:03:52Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", - "TOTAL: 2 objects, 203478494 bytes (194.05 MiB)\n" - ] - } - ], + "outputs": [], "source": [ "gsutil ls -lh \"gs://$BUCKET/output\"" ] @@ -1425,7 +598,7 @@ }, { "cell_type": "markdown", - "id": "50990a67-f901-48fd-9295-25ef73245142", + "id": "85237d4a", "metadata": { "tags": [] }, @@ -1435,7 +608,7 @@ " * In the \"New Principals\" box add the Identity for the collaborator (another individual) as directed by the instructor.\n", " * Select the \"Storage Object Viewer\" by typing \"Storage Object Viewer\" in the filter and then selecting \"Storage Object Viewer\". Do not use any \"Legacy Storage\" roles.\n", " * Click \"Save\" to save the policy.\n", - " ![iam-storage-object-viewer](img/iam-storage-object-viewer.png)\n", + " \n", " * Verify the policy is listed in the \"Permissions\" table on the \"Bucket Details\" page (you should now be on this page).\n", "\n", "*Advanced Note: Changes in permissions will show up on the **Home** -> **Activity** page.*" diff --git a/content/Azure/08_cleaning_up_resources.ipynb b/content/Azure/08_cleaning_up_resources.ipynb index 6b11af4..0494252 100644 --- a/content/Azure/08_cleaning_up_resources.ipynb +++ b/content/Azure/08_cleaning_up_resources.ipynb @@ -35,19 +35,9 @@ ] }, { - "cell_type": "code", - "execution_count": 7, - "id": "7a134850-8c4f-47c6-9816-6d487a88eda1", + "cell_type": "markdown", + "id": "7171ff08", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "rg-essentials\n" - ] - } - ], "source": [ "RG=rg-essentials\n", "echo $RG" @@ -62,20 +52,9 @@ ] }, { - "cell_type": "code", - "execution_count": 8, - "id": "6ccd79ab-0068-4a2a-9aca-2c47454f9d85", + "cell_type": "markdown", + "id": "b5d3261e", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "\u001b[0m" - ] - } - ], "source": [ "az vm list --resource-group $RG --output table" ] @@ -91,19 +70,9 @@ ] }, { - "cell_type": "code", - "execution_count": 9, - "id": "3ff9bb93-aadf-4319-aed4-b5fc1f2518c8", + "cell_type": "markdown", + "id": "92c8ae09", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[K\u001b[0minished .." - ] - } - ], "source": [ "az group delete --resource-group $RG --yes" ] diff --git a/content/Azure/glossary.ipynb b/content/Azure/glossary.ipynb new file mode 100644 index 0000000..e690bf9 --- /dev/null +++ b/content/Azure/glossary.ipynb @@ -0,0 +1,34 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "1f0027c2-df7b-426e-b4f4-3e1db9916d14", + "metadata": {}, + "source": [ + "# Glossary\n", + "\n", + "* **Account** - The Google account (email) associated with an individual. This is the identity that you use to access console and takes the form of an email address.\n", + "* **Project** - Resources and services in Google Cloud are created within a single Project. \n", + "* **Project Name** - The long human readable name of your project. You can change this.\n", + "* **Project ID** - an alpha-numeric string identifier for the project. This is not changeable.\n", + "* **Project Number** - a long number that also identifies the project but is rarely used (it is an internal identifier). \n", + "* **Billing Account** - an account used to pay for Google Cloud services. A Billing Account is associated with a Project and enables resource allocation. A project has only one Billing Account associated with it. A Billing account can be used for multiple Projects." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Bash", + "language": "bash", + "name": "bash" + }, + "language_info": { + "codemirror_mode": "shell", + "file_extension": ".sh", + "mimetype": "text/x-sh", + "name": "bash" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/content/Azure/intro_to_compute_cli.ipynb b/content/Azure/intro_to_compute_cli.ipynb deleted file mode 100644 index 0bb8ebf..0000000 --- a/content/Azure/intro_to_compute_cli.ipynb +++ /dev/null @@ -1,260 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Intro to Compute CLI\n", - "\n", - "**Overview**\n", - " \n", - "**Teaching:**\n", - "\n", - "**Exercises:**\n", - "\n", - "**Questions:**\n", - " * How do I create my own computer in the cloud?\n", - "\n", - "**Objectives:**\n", - " * Allocate a virtual machine in Azure.\n", - " * Learn how to connect to a virtual machine using ssh in the cloud shell." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Open the cloud shell (See: intro_to_cli)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Define some environment variables" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "RG=\"demo_rg\"\n", - "LOCATION=eastus\n", - "VM=\"demo_vm\"\n", - "STGE_ACCT=\"demo_stor_ac\"\n", - "CTR=\"demo_stge_ctr\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Create resource group" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az group create --name $RG --location $LOCATION" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Create a VM" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If this is the fist time running this command (or your ~/.ssh dir does not have an id_rsa key pair). It will generate a new key pair id_rsa in ~/.ssh. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az vm create --resource-group $RG \\\n", - " --name $VM \\\n", - " --image UbuntuLTS \\\n", - " --admin-username demouser \\\n", - " --generate-ssh-keys \\\n", - " --output json \\\n", - " --verbose " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To create a VM with an existing pair first create a new key pair" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "ssh-keygen" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You will be prompted to \"Enter file in which to save the key\", type \"id_rsa\". This will create a key pair in the current dir." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here you should enter a passphrase (a series of words) to protect the key. You will be asked to re-enter the passphrase as shown below:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Enter same passphrase again:" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "After successfully retyping the passphrase the key generation process will continue showing similar message as follows (the fingerprint and art will look different):" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note: The following using \"--ssh-key-values\" to define which keys to use." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az vm create --name $VM \n", - " --resource-group $RG \n", - " --image UbuntuLTS \n", - " --admin-username demouser \n", - " --ssh-key-values id_rsa.pub" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Show VM details (all)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az vm show --name $VM --resource-group $RG" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Login into VM" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Get the public IP address from the output above.\n", - "If you generated a new key pair using \"--generate-ssh-keys\", login with the following command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "ssh " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you used the command with \"--ssh-key-values id_rsa.pub\" to use existing keys, use the following command to logiin:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "ssh -i /id_rsa " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "clean up\n", - "\n", - "The --no-wait parameter keeps the CLI from blocking while the deletion takes place." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "az group delete --name $RG --no-wait" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49" - }, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.8" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/content/_toc.yml b/content/_toc.yml index 497354d..0ace1bd 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -32,6 +32,7 @@ parts: - file: Azure/06_running_analysis - file: Azure/07_monitoring_costs - file: Azure/08_cleaning_up_resources + - file: Azure/glossary - file: GCP/intro_to_GCP_Essentials sections: