diff --git a/content/Azure/01_intro_to_cloud_console.ipynb b/content/Azure/01_intro_to_cloud_console.ipynb index 3cabc97..d3e30aa 100644 --- a/content/Azure/01_intro_to_cloud_console.ipynb +++ b/content/Azure/01_intro_to_cloud_console.ipynb @@ -28,20 +28,22 @@ }, { "cell_type": "markdown", - "id": "20e902b2", + "id": "abb35991", "metadata": {}, "source": [ - "## Setup\n", + "## The Who, What, and Where of the Cloud\n", "\n", - "The console is the point of entry where you will start your Azure journey. If you are participating in the CLASS Essentials Workshop, you will have received a csv file via email. This csv file contains your **username**, **temporary password** and **console login link**. \n" + "Before we get started we must first define, and find, a few key pieces of information about your environment that will be used throughout this lesson. This information is also important to ensure that this information is what you expect, otherwise you may use the wrong account or run or store things in the wrong place. It is important to be clear about these terms as they are often different in other cloud providers. \n", + "\n", + "(need glossary for Azure) We provide a [glossary of Azure terms](glossary.ipynb) for your reference." ] }, { "cell_type": "markdown", - "id": "5fab77a5", + "id": "3e128379", "metadata": {}, "source": [ - "#### Accounts (Authentication and Identity) & login\n", + "## Accounts (Authentication and Identity) & login\n", "\n", "The (Azure) **Account** is the identity used to access the Azure web console and is in the form of an email address and is associated with an individual. This is typically the account you use to access Azure services associated with your institution and not your personal account. We will now login to the Azure web console and verify our account.\n", "\n", @@ -54,7 +56,7 @@ }, { "cell_type": "markdown", - "id": "43174583", + "id": "90066c19", "metadata": {}, "source": [ "### Exercise\n", @@ -69,19 +71,77 @@ }, { "cell_type": "markdown", - "id": "9cd64226", + "id": "160353bd", "metadata": {}, "source": [ - "The top of your screen should look very similar to the following when you log in to the Azure console. Next, let's take a closer look at some key concepts and components of the Azure console. \n", + "## Subscriptions\n", + " \n", + "Click on the *Directories & Subscriptions* buttton (#4).\n", + "Verify that you are in the right *Directory and Subscription* (briefly discribe D&S).\n", + " \n", + "Take a few moments to ask yourself the following questions:\n", + " * What is the *name* of the Project you are currently using in the Google Cloud Platform web console?\n", + " * What is the *Project ID* of that project?\n", + " * What is the *Project Number* of that project?\n", + " * Do you have multiple projects?\n", + " * How do you know which project you are currently using?\n", + "\n", + "If you are currently not using the correct project for this lesson, switch to the correct one." + ] + }, + { + "cell_type": "markdown", + "id": "6e3b3cbd", + "metadata": {}, + "source": [ + "## Services\n", + "\n", + "The web console is used to control and observe the cloud from the browser. It should only be used for simple and one-time tasks, exploring new services, accessing documentation, or for monitoring and debugging resources in the cloud. Programmatic control through the console, programming languages (for example Python), and other automation tools should be used for day to day activities to make the most out of the cloud and to help with the reproducibility of research and teaching.\n", + "\n", + "The Navigation Menu (number 1 above) is used to navigate to the various products, which are also sometimes called services.\n" + ] + }, + { + "cell_type": "markdown", + "id": "ed7e581f", + "metadata": {}, + "source": [ + "## Services\n", + "\n", + "The web console is used to control and observe the cloud from the browser. It should only be used for simple and one-time tasks, exploring new services, accessing documentation, or for monitoring and debugging resources in the cloud. Programmatic control through the console, programming languages (for example Python), and other automation tools should be used for day to day activities to make the most out of the cloud and to help with the reproducibility of research and teaching.\n", + "\n", + "The Navigation Menu (often called the hamburger) is used to navigate to the various products, which are also sometimes called services.\n", + "\n", + "![hamburger-navigation](img/hamburger-navigation.png)\n", "\n", - "![cloud-shell](images/azure_intro_console.jpg)" + " * Click on a product to navigate the product page\n", + " * You can pin frequently used items on the top of this page by clicking on the pin icon. \n", + " * Click the hamburger again to hide the Left Sidebar.\n", + "\n", + "All the different products and services are the \"What\" of the cloud. We will start with compute, called *Google Compute Engine* in the next Episode. \n", + "\n", + "### Exercise \n", + "\n", + "Take a few moments to navigate a few key services.\n", + " * Navigate to the \"Compute Engine\" service under the \"Compute\" product group.\n", + " * You will probably need to \"Enable\" this service first by clicking on the \"Enable\" button on the \"Compute Engine API\" page. This will only need to be done once per project.\n", + " * Navigate to \"Cloud Storage\" under the \"Storage\" product group and enable the service if necessary." ] }, { "cell_type": "markdown", - "id": "b1b0faae", + "id": "4537d270", "metadata": {}, "source": [ + "## Everything Else\n", + "\n", + "If you did not already notice, every page on the Azure web console has a blue navigation bar at the top of the page. This can be used to quickly navigate to the page you need.\n", + "\n", + "\n", + "The top of your screen should look very similar to the following when you log in to the Azure console. Next, let's take a closer look at some key concepts and components of the Azure console. \n", + "\n", + "![cloud-shell](images/azure_intro_console.jpg)\n", + "\n", "**1. Navigation Menu (often called the hamburger)**\n", ": If you click on the hamburger (three horizontal lines) you will get a dropdown menu and see a list of services that are offered by Azure. Currently, Azure has over N services ranging from compute to machine learning to networking to data storage. The list grows frequently!\n", "\n", @@ -98,18 +158,6 @@ ": The menu bar on the console also lists your username and the current organization that you are logged into. IAM is short for Identity and Access Management. Identity is a mechanism to authenticate users (e.g. password) and Access is a mechanism for authorizing what kinds of services users have access to. In the creation of an IAM user, the administrator (overlord) of the account assigned a particular permission scope (in AWS this is known as roles) to control access to a particular set of services and also assigned a password to authenticate user login. IAM is a key component in managing security on the cloud.\n" ] }, - { - "cell_type": "markdown", - "id": "6e3b3cbd", - "metadata": {}, - "source": [ - "## Services\n", - "\n", - "The web console is used to control and observe the cloud from the browser. It should only be used for simple and one-time tasks, exploring new services, accessing documentation, or for monitoring and debugging resources in the cloud. Programmatic control through the console, programming languages (for example Python), and other automation tools should be used for day to day activities to make the most out of the cloud and to help with the reproducibility of research and teaching.\n", - "\n", - "The Navigation Menu (number 1 above) is used to navigate to the various products, which are also sometimes called services.\n" - ] - }, { "cell_type": "markdown", "id": "e94b53e2", @@ -126,6 +174,18 @@ " * Note: You can open the Cloud Shell in a new tab by clicking on the **Open new session** icon (looks like a box with a \"+\") in the middle of the Cloud Shell bar.\n" ] }, + { + "cell_type": "markdown", + "id": "e17944ad", + "metadata": {}, + "source": [ + "## Exercise\n", + "\n", + "Before we move on, do the following:\n", + "* Double check that you are in the correct account that you will be using for this lesson.\n", + "* Take a moment to reflect on the \"Who, What, and Where\" of the cloud. Why is this important?" + ] + }, { "cell_type": "markdown", "id": "c0706cc3", diff --git a/content/Azure/02_intro_to_compute.ipynb b/content/Azure/02_intro_to_compute.ipynb index 58db0c5..5c47b31 100644 --- a/content/Azure/02_intro_to_compute.ipynb +++ b/content/Azure/02_intro_to_compute.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "8932d2c1", + "id": "4a903197", "metadata": {}, "source": [ "# Introduction to Virtual Machines (VM) in Azure\n", @@ -11,21 +11,117 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 45 mins\n", + "**Teaching:**\n", "\n", - "**Exercises:** 10 mins\n", + "**Exercises:**\n", "\n", "**Questions:**\n", + "* How do I create my own computer in the cloud?\n", "\n", "**Objectives:**\n", - "* Understand the concept of virtual servers.\n", - "\n", + "* Navigate the Google Compute Engine service and terminology.\n", + "* Allocate a virtual machine in Google Compute Engine.\n", + "* Choose a machine type based on the application.\n", + "* Use the cost estimator for Google Compute Engine.\n", + "* Understand what a service account is and how they are used.\n", + "* Learn how to connect to a virtual machine using ssh in the cloud shell with the web console.\n", "```" ] }, { "cell_type": "markdown", - "id": "05306c25", + "id": "92622861", + "metadata": {}, + "source": [ + "Drew, after working with a Research Computing and Data (RCD) facilitator, has decided to start their cloud journey with compute, specifically a Virtual Machine, to process data in the cloud with a large and powerful machine to do the analysis. " + ] + }, + { + "cell_type": "markdown", + "id": "3565e61f", + "metadata": {}, + "source": [ + "## Azure Virtual Machines" + ] + }, + { + "cell_type": "markdown", + "id": "4efebb57", + "metadata": {}, + "source": [ + "Before we begin this lesson/episode we need to create a ssh key in order to login into the VM we about to create. We will use the public key when we create the VM.\n", + "\n", + "If you have not used the cloud shell to connect to a *VM Instance* before you will need to create a new *ssh key*. The Compute Engine will use this key to allow you to access the *VM instance* in a secure manner. If this is the case you will see a message similar to the following:\n", + "```\n", + "WARNING: The private SSH key file for gcloud does not exist.\n", + "WARNING: The public SSH key file for gcloud does not exist.\n", + "WARNING: You do not have an SSH key for gcloud.\n", + "WARNING: SSH keygen will be executed to generate a key.\n", + "This tool needs to create the directory [/home/learner/.ssh] before\n", + " being able to generate SSH keys.\n", + "\n", + "Do you want to continue (Y/n)?\n", + "```\n", + "\n", + "Press enter to continue to create the key. You will see another message asking you for a passphrase, similar to the following:\n", + "```\n", + "Generating public/private rsa key pair.\n", + "Enter passphrase (empty for no passphrase):\n", + "```\n", + "Here you should enter a passphrase (a series of words) to protect the key. You will be asked to re-enter the passphrase as shown below:\n", + "```\n", + "Enter same passphrase again:\n", + "```\n", + "\n", + "After successfully retyping the passphrase the key generation process will continue showing similar message as follows (the fingerprint and art will look different):\n", + "```\n", + "Your identification has been saved in /home/learner/.ssh/azure_compute_engine.\n", + "Your public key has been saved in /home/learner/.ssh/azure_compute_engine.pub.\n", + "The key fingerprint is:\n", + "SHA256:D9yj1nZEUigahirvjkIFoCP7RfjDu67Ku88M7nHQftI learner@cs-748001990186-default-boost-r9phq\n", + "The key's randomart image is:\n", + "+---[RSA 2048]----+\n", + "|o . .. |\n", + "|.. .. o . .. |\n", + "|+ o... o .. . |\n", + "|ooo= .. . o |\n", + "|.+..= S o . |\n", + "| o+..o = o |\n", + "|.+.+.E o + . |\n", + "|+.B o. . . . |\n", + "|=O**o |\n", + "+----[SHA256]-----+\n", + "```\n", + "\n", + "The ssh key generation is now complete. This will only happen once for your account. Later during the VM create proces you will be asked to Cut & Paste the public key into a form. \n", + "\n", + "Copy the public key to your clip board \n", + "\n", + "\n", + "future connections.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7245d24e", + "metadata": {}, + "outputs": [], + "source": [ + "cat /home/learner/.ssh/azure_compute_engine.pub" + ] + }, + { + "cell_type": "markdown", + "id": "d7f0bf3c", + "metadata": {}, + "source": [ + "Highlight the output string with your cursor and use the copy command (depending on OS) to put it in your edit buffer." + ] + }, + { + "cell_type": "markdown", + "id": "9c8830ce", "metadata": {}, "source": [ "## Create a VM\n", @@ -44,12 +140,34 @@ "![vm-create](images/01_CLASS_Azure_VM_basic_01.png)\n", "\n", "**Part 2**\n", + "\n", + "Before we begin part 2, we need to supply a public key when we create the VM so we can securely login. Using the key we created earlier (assume - /home/learner/.ssh/azure_compute_engine.pub), we will copy the key by doing the following." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c8fa1b24", + "metadata": {}, + "outputs": [], + "source": [ + "cat /home/learner/.ssh/azure_compute_engine.pub" + ] + }, + { + "cell_type": "markdown", + "id": "5da66e3b", + "metadata": {}, + "source": [ + "Highlight the output string with your cursor and use the copy command (depending on OS) to put it in your edit buffer.\n", + "\n", "* Authentication type \n", " * ssh public key\n", "* Username\n", " * Choose usename (default: azeruser)\n", - "* ssh public key source\n", - " * Generate new key pair\n", + "* SSH public key source\n", + " * Select use existing public key\n", + " * Copy the key we create earlier and paste it in the red box below.\n", "* Key pair name\n", " * Choose name for key pair (default: auto generated based on VM name)\n", "* Ports (none)\n", @@ -60,7 +178,7 @@ }, { "cell_type": "markdown", - "id": "536af466", + "id": "65d977eb", "metadata": {}, "source": [ "Now let's continue to the disk configuration by clicking Next:Disk> on the bottum or on the top heading menu choose the Disk option.\n", @@ -99,7 +217,7 @@ }, { "cell_type": "markdown", - "id": "1e052657", + "id": "a1561d9f", "metadata": {}, "source": [ "Now let's continue to provision the VM. The next three menu options across the top are outside the scope of this tutorial so now we are ready to click Review & Create.\n", @@ -130,19 +248,81 @@ "* You should see your VM name\n", "* Click on VM name to get more information about your VM\n", "\n", - "![vm-networking](images/04_CLASS_Azure_VM_verify.png)" + "![vm-networking](images/04_CLASS_Azure_VM_verify.png)\n", + "\n", + "On the VM page, you will be looking at the overview page. The Overview page contains essential information like OS, public IP, RG, status, subscriptioin as well as other properties and information. Find and record the Public IP address. " + ] + }, + { + "cell_type": "markdown", + "id": "f8afd914", + "metadata": {}, + "source": [ + "## Security (need to azurize)\n", + "\n", + "Everything in the cloud requires permission (authorization). Ordinary we would configure and check security first but in the case of exploring services it is often easier to do things out of order. We noted that the *VM instance* was created with the *Compute Engine default service account*, and if the \"Allow full access to all Cloud Api's\" scope is enable, then everyone on the VM has access to all the resources in your project." ] }, { "cell_type": "markdown", - "id": "48a51891", + "id": "0d1d75a2", "metadata": {}, "source": [ - "```{admonition} Exercise\n", - ":class: attention\n", + "## Enumerate the VM Instances\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "89f6a274", + "metadata": {}, + "source": [ + "## Connect to the VM Instance\n", "\n", - "* How do Spot Instances help you optimize costs?\n", - "````" + "To connect to the *VM Instance* we will use the cloud shell in the web console. Open up a shell by clicking on the **Activate Cloud Shell** icon in the top blue bar if it is not already open. In Cloud Shell ssh to the VM you just created. (default username \"azureuser\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c03d81d8", + "metadata": {}, + "outputs": [], + "source": [ + "ssh @" + ] + }, + { + "cell_type": "markdown", + "id": "bb819970", + "metadata": {}, + "source": [ + "## Clean up\n", + "\n", + "To complete the resource life-cycle we simply delete the VM created in this lesson. This will leave you with a Resource Group that can be used again for additional services.\n", + "\n", + "To delete the VM:\n", + "\n", + "* In the Azure portal, navigate to the list of VMs in your Resource Group.\n", + "* Select the VM to delete.\n", + "* Select the More button (...), and select Delete.\n", + "* Confirm that you want to delete the VM.\n", + "\n", + "Delete the Resource Group (optional)\n", + "\n", + "* In the Azure portal, navigate to the Resource Group blade/button.\n", + "* Select the Resource Group to delete.\n", + "* Click delete\n", + "* A warning will ask you to enter the name of the Resource Group to make sure you want to continue with this perminent operation.\n", + "* Confirm that you want ot delete the Resouce Group." + ] + }, + { + "cell_type": "markdown", + "id": "937e31d4", + "metadata": {}, + "source": [ + "## Refernce" ] } ], diff --git a/content/Azure/03_intro_to_cloud_storage.ipynb b/content/Azure/03_intro_to_cloud_storage.ipynb index e270130..f96c948 100644 --- a/content/Azure/03_intro_to_cloud_storage.ipynb +++ b/content/Azure/03_intro_to_cloud_storage.ipynb @@ -2,30 +2,39 @@ "cells": [ { "cell_type": "markdown", - "id": "677549a8", + "id": "aebc9022", "metadata": {}, "source": [ - "# Introduction to Azure Cloud Storage\n", + "# Introduction to Azure Storage\n", "\n", - "Learner Questions\n", - " * How do I store data in the cloud?\n", + "```{admonition} Overview\n", + ":class: tip\n", "\n", - "Learning Objectives:\n", - " * Navigate the Azure Cloud Storage service and terminology\n", - " * Understand the roles and permissions needed to use Azure Cloud Storage\n", - " * Allocate storage in Azure Cloud Storage account\n", - " * Allocate storage in Azure Cloud Storage\n", - " * De-allocate Azure Cloud Storage storage" + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + "* How do I store data in the cloud?\n", + "\n", + "**Objectives:**\n", + "* Navigate the Azure Storage service and terminology\n", + "* Understand the roles and permissions needed to use Azure Storage\n", + "* Create Storage Account\n", + "* Allocate storage in Azure Storage\n", + "* Find the cost estimator for Azure Storage\n", + "* Recognize that resources have a \"location\"\n", + "* Recognize danger of public access\n", + "* De-allocate Azure Storage storage\n", + "```" ] }, { "cell_type": "markdown", - "id": "09e26920", + "id": "b8a6b10f", "metadata": {}, "source": [ - "Drew, after working with a Research Computing and Data (RCD) facilitator, has decided to continue his cloud journey with storage.\n", - "\n", - "One of the most common and economic ways to store data in the cloud is to use object storage.\n" + "One of the most common and economic ways to store data in the cloud is to use object storage. In Azure object storage is called *blob storage*, which is similar to the Simple Storage Service, also known as S3, on Amazon Web Services (AWS). For object storage, information is stored as a collection of key-value pairs." ] }, { @@ -46,30 +55,20 @@ }, { "cell_type": "markdown", - "id": "89b2cc65", + "id": "63d54ced", "metadata": { "tags": [] }, "source": [ - "### Security (Still need to update for Azure)\n", - "\n", - "Everything in the cloud requires permission (authorization). Let's first verify that we have the permissions to create a bucket. A Bucket (a resource) is created within a project and inheres permissions from it.\n", - "\n", - "We are interested in what permissions that *your* account has for *your* project. To do this navigate to the IAM page (**Navigation Menu -> IAM & Admin -> IAM -> Permissions -> View By: Principals**). This shows the permissions for the project.\n", - "\n", - "*Note: There is a powerful filter box to limit the permissions shown.*\n", - "\n", - "You should see a row with your account shown in the Principal column. Here you should see the \"Owner\" Role in the Role column. A *role* is a collection of permissions managed by Google or someone else. The **Owner** or the **Storage Admin** role for a project will *allow* *you* to create, access, and delete Buckets *in* the project.\n", - "\n", - "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal or member), and the resource (project)." + "## Security (Still need to update for Azure)" ] }, { "cell_type": "markdown", - "id": "3e396a67", + "id": "af639990", "metadata": {}, "source": [ - "### Create Resource Group (Note sure if this is needed)\n", + "## Create Resource Group (Note sure if this is needed)\n", "\n", "Alocating any resource in Azure requires a Resource Goup. Resources Groups scope your resources under a common entity for \n", "resource interaction, IAM, billing (technically subsription but a resource group is under a subscription), ..., etc. \n", @@ -84,10 +83,10 @@ }, { "cell_type": "markdown", - "id": "4a8f2f30", + "id": "00376ac9", "metadata": {}, "source": [ - "### Create storage account\n", + "## Create storage account\n", "\n", "Like all things in Azure you will need an account to manage your service. In this case we need a storage account. To create a storage account, follow these steps.\n", "\n", @@ -144,12 +143,20 @@ }, { "cell_type": "markdown", - "id": "21adbab1", + "id": "b605e339", + "metadata": {}, + "source": [ + "## Follow the Store Allocation" + ] + }, + { + "cell_type": "markdown", + "id": "af63fa27", "metadata": { "tags": [] }, "source": [ - "### Enumerate the Buckets\n", + "## Enumerate the Buckets\n", "\n", "Now lets find and examine the bucket. To view a bucket we do the following:\n", "\n", @@ -161,12 +168,12 @@ }, { "cell_type": "markdown", - "id": "9f28062e", + "id": "52f2d430", "metadata": { "tags": [] }, "source": [ - "### Delete the Storage\n", + "## Delete the Storage\n", "\n", "To complete the resource life-cycle we simply delete the storage container created in this lesson. All blobs in the container will also be deleted. This will leave you with a storage account that can be used again for additional stoarge services.\n", "\n", @@ -178,6 +185,34 @@ "* Confirm that you want to delete the container.\n", "\n" ] + }, + { + "cell_type": "markdown", + "id": "85e0fd51", + "metadata": {}, + "source": [ + "## Discussion\n", + "\n", + "* What does the words \"Secure\", \"Allocate\", \"Follow\", and \"Enumerate\" spell?\n", + "* What happens when you add the \"R\" in \"Review?\"\n", + "* Is this useful?" + ] + }, + { + "cell_type": "markdown", + "id": "f9e74e9b", + "metadata": {}, + "source": [ + "## Resources in Google Cloud Platform - Review" + ] + }, + { + "cell_type": "markdown", + "id": "ccb93315", + "metadata": {}, + "source": [ + "## References" + ] } ], "metadata": { diff --git a/content/Azure/04_intro_to_cli.ipynb b/content/Azure/04_intro_to_cli.ipynb index 4ee4e92..09f62d7 100644 --- a/content/Azure/04_intro_to_cli.ipynb +++ b/content/Azure/04_intro_to_cli.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "d448d2c1", + "id": "f5be7a6c", "metadata": {}, "source": [ "# Introduction to the Azure CLI\n", @@ -17,7 +17,23 @@ " * Determine basic user/account settings.\n", " * Use environment variables for configuration - No hard coding\n", " * Understand the importance of using variables for configuration.\n", - " * Introduce value of reproducibility and automation." + " * Introduce value of reproducibility and automation.\n", + "\n", + "Resources" + ] + }, + { + "cell_type": "markdown", + "id": "b609b6ed", + "metadata": {}, + "source": [ + "## Cloud Automation\n", + "\n", + "Now that Drew can create a bucket in the web console they wish to learn how to use tools to automate the process. Drew noticed that the web console could be used to upload and share objects but wishes to learn the command line so that they can automate the and document the process (it is much easier to document commands then describing and showing mouse clicks).\n", + "\n", + "The cloud can be controlled using a Command Line Interface (CLI) or a programming language such as Python. Collectively these tools interact with the cloud over a Application Programming Interface (API) and this capability forms the basis of the cloud, the ability to control infrastructure programmatically.\n", + "\n", + "Just as with navigating the web console it is important to know the **who**, **what**, and **where** of CLI access to reduce the possibility of access mistakes. We will first verify the tools are installed and configured correctly. Next we get the Account being used (who) and the Subscription ID (where). We will then use the CLI to list some public Buckets (what)." ] }, { @@ -106,7 +122,7 @@ }, { "cell_type": "markdown", - "id": "cd6323ad", + "id": "e8feb7f2", "metadata": {}, "source": [ "## Environment Variables\n", diff --git a/content/Azure/06_running_analysis.ipynb b/content/Azure/06_running_analysis.ipynb index 052ffe8..2abf7ac 100644 --- a/content/Azure/06_running_analysis.ipynb +++ b/content/Azure/06_running_analysis.ipynb @@ -1472,15 +1472,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" } }, "nbformat": 4, diff --git a/content/Azure/07_monitoring_costs.ipynb b/content/Azure/07_monitoring_costs.ipynb index e0b5d2d..6a0ba88 100644 --- a/content/Azure/07_monitoring_costs.ipynb +++ b/content/Azure/07_monitoring_costs.ipynb @@ -22,6 +22,19 @@ "\n", "```" ] + }, + { + "cell_type": "markdown", + "id": "95459101", + "metadata": {}, + "source": [ + "* In the Azure portal navigate to the *Cost Management + Billing* from the hamburger menu.\n", + "* On the left menu choose *Cost Management*\n", + "* This will give cost overview of you subscription\n", + "* On this pages left menu choose *Cost analysis*\n", + "* This page will display the actual and forcasted cost for your subscription.\n", + "* This page also offers interactive graphs that assist in drilling down to look at cost for various resources." + ] } ], "metadata": { diff --git a/content/Azure/intro_to_Azure_Essentials.ipynb b/content/Azure/intro_to_Azure_Essentials.ipynb index 2984146..6c1b81f 100644 --- a/content/Azure/intro_to_Azure_Essentials.ipynb +++ b/content/Azure/intro_to_Azure_Essentials.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "30c652f8", + "id": "a7b4c217", "metadata": {}, "source": [ "# Microsoft Azure\n", @@ -10,22 +10,14 @@ "The modules in this lesson are: \n", "\n", "1. [Introduction to the Azure Cloud Portal](./01_intro_to_cloud_console)\n", - "2. [Introduction to the Azure CLI](./02_intro_to_cli.ipynb)\n", "2. [Introduction to Cloud Compute (Azure Virtual Machines)](./02_intro_to_compute)\n", - "3. [Introduction to the Azure CLI](./04_intro_to_cli)\n", - "4. [Introduction to Cloud Storage (Blob Storage)](./05_intro_to_cloud_storage)\n", - "5. [Running Analysis on the Cloud](./06_running_analysis)\n", - "6. [Monitoring Costs](./07_monitoring_costs)\n", - "7. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)" + "3. [Introduction to Cloud Storage (Blob Storage)](./03_intro_to_cloud_storage)\n", + "4. [Introduction to the Azure CLI](./04_intro_to_cli)\n", + "5. [Using the Cloud Storage CLI](./05_intro_to_cloud_storage_cli)\n", + "6. [Running Analysis on the Cloud](./06_running_analysis)\n", + "7. [Monitoring Costs](./07_monitoring_costs)\n", + "8. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "92d161a5", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/content/GCP/01_intro_to_cloud_console.ipynb b/content/GCP/01_intro_to_cloud_console.ipynb index f8e73fe..d7a5d60 100644 --- a/content/GCP/01_intro_to_cloud_console.ipynb +++ b/content/GCP/01_intro_to_cloud_console.ipynb @@ -12,9 +12,9 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 10 min.\n", "\n", - "**Exercises:**\n", + "**Exercises:** 6 min.\n", "\n", "**Questions:**\n", "* How do I find things in the web console?\n", @@ -29,7 +29,7 @@ }, { "cell_type": "markdown", - "id": "3316c7f9-b0c8-4565-bdc6-7582c4f16127", + "id": "10be57f6", "metadata": {}, "source": [ "## The Who, What, and Where of the Cloud\n", @@ -64,19 +64,20 @@ "tags": [] }, "source": [ - "### Exercise\n", + "```{admonition} Exercise\n", "\n", "Take a few moments to ask yourself the following questions:\n", " * What account are you currently using in the Google Cloud web console?\n", " * Are you logged into any other accounts?\n", " * How do you know which account is being used?\n", " \n", - " If you are not currently using the correct account, switch to it." + " If you are not currently using the correct account, switch to it.\n", + "```" ] }, { "cell_type": "markdown", - "id": "5cfe3d13-c050-49ee-873b-be823fda5b10", + "id": "847149e2-bce0-4377-b9fc-ae88d8ba41e8", "metadata": {}, "source": [ "## Projects\n", @@ -94,10 +95,16 @@ "\n", "A Project, just like the word, should be associated with a real world project (for example a research effort or grant, lab, or your Ph.D. Project. etc). For Drew, this is the image processing project. In this way it is easier to track and allocate costs and to manage permissions and access for resources within the project.\n", "\n", - "The active project is also shown next to the project icon (three hexagons) and clicking it brings up the project selection dialog. The current project has a check mark and and the active project can be changed by clicking on the project name or double clicking the row.\n", - " ![select-project](img/select-project.png) \n", - "\n", - "### Exercise\n", + "The active project is also shown next to the project icon (three hexagons) and clicking it brings up the project selection dialog. The current project has a check mark and the active project can be changed by clicking on the project name or double clicking the row.\n", + " ![select-project](img/select-project.png) " + ] + }, + { + "cell_type": "markdown", + "id": "45e76bfd-9392-4905-b554-f0a0ad815274", + "metadata": {}, + "source": [ + "```{admonition} Exercise\n", "\n", "Take a few moments to ask yourself the following questions:\n", " * What is the *name* of the Project you are currently using in the Google Cloud Platform web console?\n", @@ -106,12 +113,13 @@ " * Do you have multiple projects?\n", " * How do you know which project you are currently using?\n", "\n", - "If you are currently not using the correct project for this lesson, switch to the correct one." + "If you are currently not using the correct project for this lesson, switch to the correct one.\n", + "```" ] }, { "cell_type": "markdown", - "id": "294a3fa0-7a55-4547-af19-4ef542b8dae7", + "id": "a43bd3bd-2d12-4d24-a59e-707e389b9cd0", "metadata": { "tags": [] }, @@ -128,14 +136,23 @@ " * You can pin frequently used items on the top of this page by clicking on the pin icon. \n", " * Click the hamburger again to hide the Left Sidebar.\n", "\n", - "All the different products and services are the \"What\" of the cloud. We will start with compute, called *Google Compute Engine* in the next Episode. \n", - "\n", - "### Exercise \n", + "All the different products and services are the \"What\" of the cloud. We will start with compute, called *Google Compute Engine* in the next Episode. " + ] + }, + { + "cell_type": "markdown", + "id": "4da0bd01-47f7-40bd-a929-985f63bb1810", + "metadata": { + "tags": [] + }, + "source": [ + "```{admonition} Exercise\n", "\n", "Take a few moments to navigate a few key services.\n", " * Navigate to the \"Compute Engine\" service under the \"Compute\" product group.\n", " * You will probably need to \"Enable\" this service first by clicking on the \"Enable\" button on the \"Compute Engine API\" page. This will only need to be done once per project.\n", - " * Navigate to \"Cloud Storage\" under the \"Storage\" product group and enable the service if necessary." + " * Navigate to \"Cloud Storage\" under the \"Storage\" product group and enable the service if necessary.\n", + "```" ] }, { @@ -201,25 +218,42 @@ "tags": [] }, "source": [ - "## Exercise\n", + "```{admonition} Exercise\n", "\n", "Before we move on, do the following:\n", "* Double check that you are in the correct account and project that you will be using for this lesson.\n", - "* Take a moment to reflect on the \"Who, What, and Where\" of the cloud. Why is this important?\n" + "* Take a moment to reflect on the \"Who, What, and Where\" of the cloud. \n", + " * What is your Who, What, and Where?\n", + " * Why is this important?\n", + "```" ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "530f7a45-5708-4213-b120-71c6328f7026", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" } }, "nbformat": 4, diff --git a/content/GCP/02_intro_to_compute.ipynb b/content/GCP/02_intro_to_compute.ipynb index 374d54b..7097fbc 100644 --- a/content/GCP/02_intro_to_compute.ipynb +++ b/content/GCP/02_intro_to_compute.ipynb @@ -12,9 +12,9 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 45 min.\n", "\n", - "**Exercises:**\n", + "**Exercises:** 6 min\n", "\n", "**Questions:**\n", "* How do I create my own computer in the cloud?\n", @@ -24,8 +24,8 @@ "* Allocate a virtual machine in Google Compute Engine.\n", "* Choose a machine type based on the application.\n", "* Use the cost estimator for Google Compute Engine.\n", - "* Understand what a service account is and how they are used.\n", - "* Learn how to connect to a virtual machine using ssh in the cloud shell with the web console.\n", + "* Learn what a service account is and how they are used.\n", + "* Learn how to connect to a virtual machine using ssh in the Cloud Shell with the web console.\n", "```" ] }, @@ -75,8 +75,21 @@ " * For **Machine type** select *e2-medium* (default). Note that it shows the size of the VM in parentheses. \n", " * In the **Identity and API access** section note that for the **Service Accounts** -> **Service account** option that the **Compute Engine default service account** is selected. This allows the *VM Instance* to access resources in the project. Note: the *\"Allow full access to all Cloud APIs\"* allows *everyone* on the VM to *access and edit all resources in the project*.\n", " * Leave the remaining options as default.\n", - " * Note the **Monthly estimate** in the right side panel. This is only for running the *VM instance* and does not include other storage, networking, compute, and other charges that may occur by running software. Click **Details** show a breakdown of the *VM instance* only costs. **Exercise:** Find the most expensive VM.\n", - " * Finally, click **Create** to create the *VM Instance* (allocate the resource).\n" + " * Note the **Monthly estimate** in the right side panel. This is only for running the *VM instance* and does not include other storage, networking, compute, and other charges that may occur by running software. Click **Details** show a breakdown of the *VM instance* only costs.\n", + "![compute-create-instance](img/compute-create-instance.png) \n", + " * Finally, click **Create** to create the *VM Instance* (allocate the resource).\n", + "![compute-create-instance-ok](img/compute-create-instance-ok.png)\n" + ] + }, + { + "cell_type": "markdown", + "id": "6737d8ba-e6d4-437f-a652-5d9d4477b34c", + "metadata": {}, + "source": [ + "```{admonition} Exercise\n", + "* Find the most expensive VM. **DO NOT actually create! Just cancel!**\n", + "* Find the least expensive VM.\n", + "```" ] }, { @@ -86,7 +99,9 @@ "source": [ "## Security\n", "\n", - "Everything in the cloud requires permission (authorization). Ordinary we would configure and check security first but in the case of exploring services it is often easier to do things out of order. We noted that the *VM instance* was created with the *Compute Engine default service account*, and if the \"Allow full access to all Cloud Api's\" scope is enable, then everyone on the VM has access to all the resources in your project..\n", + "Everything in the cloud requires permission (authorization). Ordinary we would configure and check security first but in the case of exploring services it is often easier to do things out of order. We noted that the *VM instance* was created with the *Compute Engine default service account*, and if the \"Allow full access to all Cloud Api's\" scope is enable, then everyone on the VM has access to all the resources in your project.\n", + "\n", + "In the case of Drew, this is useful since they will be storing results of the analysis in Cloud Storage and the VM that will be doing the processing will need to have access to the storage. This is the mechanism that allows this.\n", "\n", "Specifically, we are interested in verifying what permissions that the *Compute Engine default service account* has for your *VM instances* . To do this navigate to the IAM page (**Navigation Menu** -> **IAM & Admin** -> **IAM** -> **Permissions** tab -> View By: **Principals** tab)\n", "\n", @@ -94,7 +109,13 @@ "\n", "In the **Name** column you should see a row with **Compute Engine default service account**. Here you should see the **Editor** role in the Role column. A *role* is a collection of permissions managed by Google or someone else. The **Editor** role for a project will *allow* a **VM instance** to modify resources *in* the **project**. The principal for the *Compute Engine default service account* should start with the *Project ID* of your project. It is possible to create a custom service account with the appropriate roles for the *VM Instance*.\n", "\n", - "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal or member), and the resource (project)." + "![iam-default-service-account](img/iam-default-service-account.png)\n", + "\n", + "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal or member), and the resource (project).\n", + "\n", + "There is one additional tool that is used, **scope**. Scope describes the services that are accessible by an identity and is a simple way of limiting access to services. \n", + "\n", + "When we previously created the virtual machine we selected the default **scope**, which limits what services the virtual machine has access to. In other words, these scopes can only *limit* what a virtual machine instance can do, it can not add any permissions that are not already granted to the service account through IAM. In general, access should be managed through IAM policy, not scope.\n" ] }, { @@ -130,7 +151,7 @@ "\n", "Now lets find and connect to the *VM Instance*.\n", " * Navigate to the Google Compute Engine page by clicking **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances**.\n", - " * **Find** the *VM instance* you just created. You can use the filter to find a bucket if there are a lot of them.\n", + " * **Find** the *VM instance* you just created. You can use the filter to find a *VM instance* if there are a lot of them.\n", " * Click on the *VM instance* name to open the **VM instance details** (it will display as a hyperlink when you hover over the *VM instance* name).\n", "\n", "Navigate to the **dashboard** and you will see **Compute Engine** in the *resources* card. You can click on this to quickly navigate to the compute engine page.\n" @@ -241,20 +262,56 @@ }, { "cell_type": "markdown", - "id": "0362e76c-3f03-4299-8217-24a2f39ee503", + "id": "c59b7e30-95a4-4eb7-91bf-ba30390ed817", + "metadata": { + "tags": [] + }, + "source": [ + "## Delete the VM Instance\n", + "\n", + "```{admonition} Danger\n", + ":class: danger\n", + "\n", + "This will destroy the VM instance and and ALL the data in it! You cannot undelete an instance!\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "31bcefe2-dcf8-42d7-a8b5-76edeb585c7d", "metadata": { "tags": [] }, "source": [ - "## Delete the VM - Exercise \n", + "Now that we have made that clear, we will now complete the resource life-cycle by deleting the *VM Instance* we just created. First, try to do this on your own.\n", "\n", - "We will now complete the resource life-cycle by deleting the *VM Instance* we just created. Think about the process we did for creating the bucket and follow a similar one. \n", + "```{admonition} Exercise\n", "\n", - "Did you \"Follow\" the bucket by looking at the **activity** page?\n", + "* Try to delete the ***VM instance*** on your own\n", + "* Try to follow the process we just learned (hint: *follow* and *enumerate*)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "de7fff43-be0f-45f1-a950-8d7e8326d328", + "metadata": { + "tags": [] + }, + "source": [ + "To delete a *VM Instance*:\n", + " * Go to **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances**.\n", + " * On the row of the virtual machines click the three vertical dots (**kabob**) menu and select **Delete**\n", + " * Click 'Delete' to confirm deleting the instance. \n", + "\n", + "![compute-delete-instance](img/compute-delete-instance.png)\n", "\n", - "Since we care about paying for resources we are not using we **review** our project by visiting the compute engine service and reviewing that we no longer have any virtual machines (VM) running. \n", + "Did you \"Follow\" the VM instance by looking at the **activity** page as discussed above?\n", "\n", - "In later episodes we will show how to easily list all the resources in a project." + "Since we care about paying for resources we are not using we review our project by visiting the *compute engine* service and reviewing that we no longer have any *VM instances* running. \n", + " * Go to **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances** and look for your virtual machine.\n", + "\n", + "In a later episode we will show how to easily list all the resources in a project." ] }, { @@ -263,21 +320,27 @@ "metadata": {}, "source": [ "## References\n", - " * Compute Engine Permissions (IAM): https://cloud.google.com/compute/docs/access/iam" + " * Compute Engine Permissions (IAM): [https://cloud.google.com/compute/docs/access/iam](https://cloud.google.com/compute/docs/access/iam)" ] } ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" } }, "nbformat": 4, diff --git a/content/GCP/03_intro_to_cloud_storage.ipynb b/content/GCP/03_intro_to_cloud_storage.ipynb index bbc64a4..452861e 100644 --- a/content/GCP/03_intro_to_cloud_storage.ipynb +++ b/content/GCP/03_intro_to_cloud_storage.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "fb98ae89-8494-4fd7-aa97-44380424d68d", + "id": "073b2176", "metadata": {}, "source": [ "# Introduction to Google Cloud Storage\n", @@ -10,9 +10,9 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 40 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** 5 min\n", "\n", "**Questions:**\n", "* How do I store data in the cloud?\n", @@ -33,7 +33,7 @@ "id": "9897048a-6aa8-4d85-a557-d85b802f3f1d", "metadata": {}, "source": [ - "One of the most common and economic ways to store data in the cloud is to use object storage. In GCP object storage is called *Google Cloud Storage*, which is similar to the Simple Storage Service, also known as S3, on Amazon Web Services (AWS). For object storage, information is stored as a collection of key-value pairs." + "There are many storage services in the cloud. One of the most common and economic ways to store data in the cloud is to use object storage. In GCP object storage is called *Google Cloud Storage*, which is similar to the Simple Storage Service, also known as S3, on Amazon Web Services (AWS). For object storage, information is stored as a collection of key-value pairs. This is different to how data is commonly stored on laptops and high performance computing clusters (supercomputers)." ] }, { @@ -52,7 +52,7 @@ }, { "cell_type": "markdown", - "id": "4b1d2914-cac0-465e-b61a-11bbc243cc8c", + "id": "07fb9096-2b40-4995-a742-be7bd9b2797c", "metadata": { "tags": [] }, @@ -67,7 +67,22 @@ "\n", "You should see a row with your account shown in the Principal column. Here you should see the \"Editor\" Role in the Role column. A *role* is a collection of permissions managed by Google or someone else. The **Editor**, **Owner**, or the **Storage Admin** role for a project will *allow* *you* to create, access, and delete Buckets *in* the project.\n", "\n", - "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal or member), and the resource (project)." + "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal), and the resource (project). This is another who (identity), what (permission), and where (resource)." + ] + }, + { + "cell_type": "markdown", + "id": "9acf29cf-660b-4922-bcb8-89fd9080fdea", + "metadata": { + "tags": [] + }, + "source": [ + "```{admonition} Exercise\n", + "\n", + "Answer the following questions:\n", + " * What is the \"Who, What, Where\" of the IAM policy that allows you to use your project?\n", + " * What else has permissions to do things in your project and state the \"Who, What, Where\"?\n", + "```" ] }, { @@ -102,6 +117,8 @@ " * *Data Retrieval* is only used for for long-term, non *standard*, storage.\n", " * *Operations* is the number of \"operations\" performed on an object, for example reading and writing an object. Storing and accessing many objects require many operations.\n", " * Finally, click **Create** to create the bucket (allocate the resource).\n", + "![storage-create-bucket](img/storage-create-bucket.png) \n", + "\n", " " ] }, @@ -164,18 +181,53 @@ }, { "cell_type": "markdown", - "id": "0362e76c-3f03-4299-8217-24a2f39ee503", + "id": "b44b27aa-d98e-4c9f-b00b-938e0a0fa783", + "metadata": { + "tags": [] + }, + "source": [ + "## Delete the Storage\n", + "\n", + "```{admonition} Danger\n", + ":class: danger\n", + "\n", + "This will destroy the storage *Bucket* and and ALL the data in it! You cannot undelete a Bucket!\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "59005efb-2aad-4dcd-ac7b-f153f4295abc", "metadata": { "tags": [] }, "source": [ - "## Delete the Storage - Exercise \n", + "We will now complete the resource life-cycle by deleting the bucket we just created. Think about the process we did for creating the bucket and follow a similar one.\n", "\n", - "We will now complete the resource life-cycle by deleting the bucket we just created. Think about the process we did for creating the bucket and follow a similar one. \n", + "```{admonition} Exercise\n", "\n", - "Did you \"Follow\" the bucket by looking at the **activity** page?\n", + " * Try to delete the storage bucket on your own\n", + " * Verify that the storage bucket has been deleted.\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "72048a39-e36b-411a-a6a8-bf1d35b4acb7", + "metadata": { + "tags": [] + }, + "source": [ + "To delete a *Bucket*:\n", + " * Go to **Navigation Menu** -> **Cloud Storage** (under Storage) -> **Browser**.\n", + " * On the row of the virtual machines click the three vertical dots (**kabob**) menu and select **Delete**\n", + " * Click 'Delete' to confirm deleting the instance. \n", + "\n", + "![storage-delete-bucket](img/storage-delete-bucket.png)\n", "\n", - "Since we care about paying for resources we are not using we **review** by visiting the **dashboard** and reviewing that we no longer have any buckets, if this was our only bucket. " + "Did you \"Follow\" the bucket by looking at the **activity** page as discussed above?\n", + "\n", + "Since we care about paying for resources we are not using we review our project by visiting the *compute storage* service and reviewing that we no longer have any *Buckets* allocated. " ] }, { @@ -183,7 +235,7 @@ "id": "3a28e28d-1d70-44fa-a952-4f3506ea85ec", "metadata": {}, "source": [ - "## Discussion\n", + "## Discussion (Optional)\n", "\n", "* What does the words \"Secure\", \"Allocate\", \"Follow\", and \"Enumerate\" spell?\n", "* What happens when you add the \"R\" in \"Review?\"\n", @@ -195,7 +247,7 @@ "id": "97d7ebc5-4a81-4f1a-aaf3-517adf70640a", "metadata": {}, "source": [ - "## Resources in Google Cloud Platform - Review\n", + "## Resources in Google Cloud Platform (Optional)\n", "\n", "Even though we only covered the Google Cloud Storage service in this episode, this process can be used for other *resources* allocated in the cloud. The term *resource* is used for the \"things\" that live in a Project, such as compute, storage, and networking and other services. Resources have the following characteristics:\n", "\n", @@ -214,22 +266,28 @@ "metadata": {}, "source": [ "## References\n", - " * API reference https://cloud.google.com/storage/docs/reference/libraries\n", - " * Permissions (IAM): https://cloud.google.com/storage/docs/access-control/iam-permissions" + " * API reference [https://cloud.google.com/storage/docs/reference/libraries](https://cloud.google.com/storage/docs/reference/libraries)\n", + " * Permissions (IAM): [https://cloud.google.com/storage/docs/access-control/iam-permissions](https://cloud.google.com/storage/docs/access-control/iam-permissions)" ] } ], "metadata": { "kernelspec": { - "display_name": "Bash", - "language": "bash", - "name": "bash" + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, "language_info": { - "codemirror_mode": "shell", - "file_extension": ".sh", - "mimetype": "text/x-sh", - "name": "bash" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" } }, "nbformat": 4, diff --git a/content/GCP/04_intro_to_cli.ipynb b/content/GCP/04_intro_to_cli.ipynb index dd4d0df..9136726 100644 --- a/content/GCP/04_intro_to_cli.ipynb +++ b/content/GCP/04_intro_to_cli.ipynb @@ -2,18 +2,17 @@ "cells": [ { "cell_type": "markdown", - "id": "5439e525-b985-495f-85a6-e4c8d7452956", + "id": "eba8b2f8-0d38-4c7a-aaf0-a08b38e6ca8f", "metadata": {}, "source": [ - "# Introduction to the gcloud CLI\n", - "\n", + "# Introduction to the Command Line Interface (CLI)\n", "\n", "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 40 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** 5 min\n", "\n", "**Questions:**\n", "* How do I use the Cloud CLI?\n", @@ -26,12 +25,18 @@ "* Use environment variables for configuration.\n", "* Understand the importance of using variables for configuration.\n", "* Recognize the value of reproducibility and automation.\n", - "```\n", - " \n", + "``` " + ] + }, + { + "cell_type": "markdown", + "id": "43a7d426-ce18-4289-b70f-639a3f1070ce", + "metadata": {}, + "source": [ "Resources\n", - " * `gcloud` documentation: https://cloud.google.com/sdk/gcloud\n", - " * `gsutil` documentation: https://cloud.google.com/storage/docs/gsutil\n", - " * Instructions to install `gcloud` and `gsutil` on your local machine: https://cloud.google.com/sdk/docs/install" + " * `gcloud` documentation: [https://cloud.google.com/sdk/gcloud](https://cloud.google.com/sdk/gcloud)\n", + " * `gsutil` documentation: [https://cloud.google.com/storage/docs/gsutil](https://cloud.google.com/storage/docs/gsutil)\n", + " * Instructions to install `gcloud` and `gsutil` on your local machine: [https://cloud.google.com/sdk/docs/install](https://cloud.google.com/sdk/docs/install)" ] }, { @@ -63,23 +68,40 @@ }, { "cell_type": "markdown", - "id": "e4fa29a6-7d8e-4591-af3f-8539b94b3bef", + "id": "90229d38-69d3-4bf2-a0ec-c7b2e6c452e4", "metadata": {}, "source": [ - "## Verify the Configuration (Who, What, Where)" + "## Start a Cloud Shell Session\n", + "\n", + "Open up a Cloud shell session by clicking on the terminal icon on the blue bar on the top right of the page (labeled 5 below).\n", + "\n", + "![blue-bar](img/blue-bar.png)\n", + "\n", + "You may wish to push the maximize button (hover over the icons to see the names) on the terminal to make it full screen and change the font size (found in the gear icon) to your liking.\n", + "\n", + "Now test that the `gcloud` command works and you can get help. This is done by running the following:\n", + "```bash\n", + "gcloud help\n", + "```\n", + "\n", + "Use the arrows, page up and page down (also space), to navigate the help screen and press `q` to exit the help screen. (Pressing `h` will give you more information about how to navigate)" ] }, { "cell_type": "markdown", - "id": "f786f92c-7127-4f26-b0c0-fc27a364aca4", - "metadata": {}, + "id": "b91e0142-a26d-4bfd-9d6d-c455fdbf49f4", + "metadata": { + "tags": [] + }, "source": [ + "## Verify the Configuration (Who, What, Where)\n", + "\n", "First, let's verify that the Account being used for access (who) is what we expect." ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "id": "3ce9e6fb-9f5f-42e1-8512-35c0564f8e85", "metadata": {}, "outputs": [ @@ -87,8 +109,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Your active configuration is: [cloudshell-17640]\n", - "student31@class.internet2.edu\n" + "998995103712-compute@developer.gserviceaccount.com\n" ] } ], @@ -106,7 +127,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "id": "cfb10587-e007-45ae-839f-7bcfe088c711", "metadata": {}, "outputs": [ @@ -114,8 +135,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Your active configuration is: [cloudshell-17640]\n", - "just-armor-301114\n" + "essentials-learner\n" ] } ], @@ -133,7 +153,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "id": "617325c9-d853-4291-a1db-938ab9439fee", "metadata": {}, "outputs": [ @@ -547,11 +567,13 @@ "id": "270e0abb-d5e9-476a-af91-060cc0ea0349", "metadata": {}, "source": [ - "## Exercise\n", + "```{admonition} Exercise\n", + "\n", " * What command shows the project number? (hint: we used it already)\n", " * What is the name of this variable? (hint: it is case sensitive)\n", " * What command shows *just* the Project Number (hint: use `--format`)\n", - " * Set the `PROJECT_NUMBER` environment variable with the correct value. (hint: be careful with punctuation (single and double quotes) and matching brackets and punctuation)" + " * Set the `PROJECT_NUMBER` environment variable with the correct value. (hint: be careful with punctuation (single and double quotes) and matching brackets and punctuation)\n", + "```" ] }, { @@ -559,7 +581,7 @@ "id": "bda44ee1-337a-447d-8377-e8f7ae563316", "metadata": {}, "source": [ - "## Advanced Call-Out: Formating Output.\n", + "## Formating and Scripting Output (Optional)\n", "\n", "The following example shows how to output the configuration as `JSON`. The command `jq` can also be used to parse the JSON data." ] diff --git a/content/GCP/05_cli_storage.ipynb b/content/GCP/05_cli_storage.ipynb index fca81f9..4076e4f 100644 --- a/content/GCP/05_cli_storage.ipynb +++ b/content/GCP/05_cli_storage.ipynb @@ -11,9 +11,9 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 20 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** 2 min\n", "\n", "**Questions:**\n", "* How do I store data in a Bucket?\n", @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "3bcdd299-5d83-4bb6-b2bb-daa85b392a19", + "id": "7825fd5a", "metadata": {}, "source": [ "Now that Drew understands how to create create buckets using the web console and to use the Google command line tools they are now going to explore managing buckets with the command line." @@ -53,8 +53,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Your active configuration is: [cloudshell-31923]\n", - "student31@class.internet2.edu\n" + "998995103712-compute@developer.gserviceaccount.com\n" ] } ], @@ -72,8 +71,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Your active configuration is: [cloudshell-31923]\n", - "just-armor-301114\n" + "essentials-learner\n" ] } ], @@ -91,7 +89,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "just-armor-301114\n" + "essentials-learner\n" ] } ], @@ -112,15 +110,15 @@ }, { "cell_type": "markdown", - "id": "f020bfcf-2e0e-4b8f-97f3-6b4753d95146", + "id": "658e1324", "metadata": {}, "source": [ - "We first generate a bucket name and store it in an environment variable for future use. Bucket names are globally unique, so here we use \"essentials\" with your Cloud Shell username (the username part of the Google Account) as as a prefix to name the bucket something unique and easy to understand where it came from." + "We first generate a bucket name and store it in an environment variable for future use. Bucket names are globally unique, so here we use \"essentials\" with your Cloud Shell username (the username part of the Google Account) as a prefix to name the bucket something unique and easy to understand where it came from." ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 4, "id": "255eb914-a06a-4bfd-adbb-7ca2f94678e7", "metadata": {}, "outputs": [ @@ -128,7 +126,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "essentials-learner-2021-12-16\n" + "essentials-learner-2022-01-21\n" ] } ], @@ -139,7 +137,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 5, "id": "b0661b4c-b56b-4d6d-ba3c-e1c5f38699c6", "metadata": {}, "outputs": [ @@ -147,7 +145,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Creating gs://essentials-learner-2021-12-16/...\n" + "Creating gs://essentials-learner-2022-01-21/...\n" ] } ], @@ -185,7 +183,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-student31-2021-10-26/\n" + "gs://essentials-learner-2022-01-21/\n" ] } ], @@ -203,10 +201,10 @@ }, { "cell_type": "markdown", - "id": "00973c8c-b3bb-41ba-b3aa-9a425284c70a", + "id": "838a4110", "metadata": {}, "source": [ - "Just like the unix `ls` command, the `gsutil ls` command has a lot of (similar) options. Let's get detailed information about the bucket. Note we must specify the bucket by using the `-b` option to specify the bucket. Type `gsutil ls --help` for more information on the command and options." + "Just like the Linux `ls` command, the `gsutil ls` command has a lot of (similar) options. Let's get detailed information about the bucket. Note we must specify the bucket by using the `-b` option to specify the bucket. Type `gsutil ls --help` for more information on the command and options." ] }, { @@ -219,7 +217,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-student31-2021-10-26/ :\n", + "gs://essentials-learner-2022-01-21/ :\n", "\tStorage class:\t\t\tSTANDARD\n", "\tLocation type:\t\t\tmulti-region\n", "\tLocation constraint:\t\tUS\n", @@ -231,34 +229,34 @@ "\tRequester Pays enabled:\t\tNone\n", "\tLabels:\t\t\t\tNone\n", "\tDefault KMS key:\t\tNone\n", - "\tTime created:\t\t\tTue, 26 Oct 2021 20:47:06 GMT\n", - "\tTime updated:\t\t\tTue, 26 Oct 2021 20:47:06 GMT\n", + "\tTime created:\t\t\tFri, 21 Jan 2022 23:23:39 GMT\n", + "\tTime updated:\t\t\tFri, 21 Jan 2022 23:23:39 GMT\n", "\tMetageneration:\t\t\t1\n", "\tBucket Policy Only enabled:\tFalse\n", - "\tPublic access prevention:\tunspecified\n", + "\tPublic access prevention:\tinherited\n", "\tRPO:\t\t\t\tDEFAULT\n", "\tACL:\t\t\t\t\n", "\t [\n", "\t {\n", - "\t \"entity\": \"project-owners-1002111293252\",\n", + "\t \"entity\": \"project-owners-998995103712\",\n", "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"projectNumber\": \"998995103712\",\n", "\t \"team\": \"owners\"\n", "\t },\n", "\t \"role\": \"OWNER\"\n", "\t },\n", "\t {\n", - "\t \"entity\": \"project-editors-1002111293252\",\n", + "\t \"entity\": \"project-editors-998995103712\",\n", "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"projectNumber\": \"998995103712\",\n", "\t \"team\": \"editors\"\n", "\t },\n", "\t \"role\": \"OWNER\"\n", "\t },\n", "\t {\n", - "\t \"entity\": \"project-viewers-1002111293252\",\n", + "\t \"entity\": \"project-viewers-998995103712\",\n", "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"projectNumber\": \"998995103712\",\n", "\t \"team\": \"viewers\"\n", "\t },\n", "\t \"role\": \"READER\"\n", @@ -267,25 +265,25 @@ "\tDefault ACL:\t\t\t\n", "\t [\n", "\t {\n", - "\t \"entity\": \"project-owners-1002111293252\",\n", + "\t \"entity\": \"project-owners-998995103712\",\n", "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"projectNumber\": \"998995103712\",\n", "\t \"team\": \"owners\"\n", "\t },\n", "\t \"role\": \"OWNER\"\n", "\t },\n", "\t {\n", - "\t \"entity\": \"project-editors-1002111293252\",\n", + "\t \"entity\": \"project-editors-998995103712\",\n", "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"projectNumber\": \"998995103712\",\n", "\t \"team\": \"editors\"\n", "\t },\n", "\t \"role\": \"OWNER\"\n", "\t },\n", "\t {\n", - "\t \"entity\": \"project-viewers-1002111293252\",\n", + "\t \"entity\": \"project-viewers-998995103712\",\n", "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"projectNumber\": \"998995103712\",\n", "\t \"team\": \"viewers\"\n", "\t },\n", "\t \"role\": \"READER\"\n", @@ -300,14 +298,14 @@ }, { "cell_type": "markdown", - "id": "1e7fceeb-661e-4a42-8fa4-5c2c50503ea6", + "id": "3a59ca7c", "metadata": {}, "source": [ - "You can see that the bucket is \"multi-region\" and uses the \"standard\" storage class (this the type of storage). The standard storage class is best used for frequently used buckets. Other storage classes are designed for less frequent use, are less expensive, but come with more restrictions and more complex access costs and should be only used after careful consideration.\n", + "You can see that the bucket is \"multi-region\" and uses the \"standard\" storage class (this is the type of storage). The standard storage class is best used for frequently used buckets. Other storage classes are designed for less frequent use, are less expensive, but come with more restrictions and more complex access costs and should only be used after careful consideration.\n", "\n", "Ignore the \"ACL\" section for now.\n", "\n", - "You may also want to verify that you can see the newly created bucket in the web console dashboard or \"Cloud Storage\" page and explore the properties here." + "You may also want to verify that you can see the newly created bucket in the web console dashboard or \"Cloud Storage\" page and explore the properties there." ] }, { @@ -322,13 +320,13 @@ }, { "cell_type": "markdown", - "id": "c9480ea5-41b3-4bde-ad01-3f290792bc0a", + "id": "3cdb9158", "metadata": {}, "source": [ "## Bucket Activity\n", "Next we will check/follow the Activity log to ensure that the bucket was created. This command assumes that there was no other activity in the account, you may need to increase the `--limit` value to something larger to find the activity. \n", "\n", - "Activity logs are used track important project level activity (such as bucket creation and deletion) and can be used for security and tracking resources and cannot be deleted. This can be used to debug if something goes wrong or in a security audit or investigation.\n" + "Activity logs are used to track important project level activity (such as bucket creation and deletion) and can be used for security and tracking resources and cannot be deleted. This can be used to debug if something goes wrong or in a security audit or investigation.\n" ] }, { @@ -345,65 +343,71 @@ "output_type": "stream", "text": [ "---\n", - "insertId: -z8eaicef61t3\n", - "logName: projects/just-armor-301114/logs/cloudaudit.googleapis.com%2Factivity\n", + "insertId: -o4omdgeg7sge\n", + "logName: projects/essentials-learner/logs/cloudaudit.googleapis.com%2Factivity\n", "protoPayload:\n", " '@type': type.googleapis.com/google.cloud.audit.AuditLog\n", " authenticationInfo:\n", - " principalEmail: student31@class.internet2.edu\n", + " principalEmail: 998995103712-compute@developer.gserviceaccount.com\n", + " serviceAccountDelegationInfo:\n", + " - firstPartyPrincipal:\n", + " principalEmail: service-998995103712@compute-system.iam.gserviceaccount.com\n", " authorizationInfo:\n", " - granted: true\n", " permission: storage.buckets.create\n", - " resource: projects/_/buckets/essentials-student31-2021-10-26\n", + " resource: projects/_/buckets/essentials-learner-2022-01-21\n", " resourceAttributes: {}\n", + " metadata:\n", + " rpo: DEFAULT\n", " methodName: storage.buckets.create\n", " request:\n", " defaultObjectAcl:\n", " '@type': type.googleapis.com/google.iam.v1.Policy\n", " bindings:\n", " - members:\n", - " - projectViewer:just-armor-301114\n", + " - projectViewer:essentials-learner\n", " role: roles/storage.legacyObjectReader\n", " - members:\n", - " - projectOwner:just-armor-301114\n", - " - projectEditor:just-armor-301114\n", + " - projectOwner:essentials-learner\n", + " - projectEditor:essentials-learner\n", " role: roles/storage.legacyObjectOwner\n", " requestMetadata:\n", - " callerIp: 35.239.199.87\n", - " callerSuppliedUserAgent: apitools Python/3.7.3 gsutil/5.4 (linux) analytics/disabled\n", - " interactive/True command/mb google-cloud-sdk/361.0.0,gzip(gfe)\n", + " callerIp: 34.102.126.22\n", + " callerNetwork: //compute.googleapis.com/projects/essentials-learner/global/networks/__unknown__\n", + " callerSuppliedUserAgent: apitools Python/3.7.3 gsutil/5.5 (linux) analytics/disabled\n", + " interactive/True command/mb google-cloud-sdk/366.0.0,gzip(gfe)\n", " destinationAttributes: {}\n", " requestAttributes:\n", " auth: {}\n", - " time: '2021-10-26T20:47:05.790066575Z'\n", + " time: '2022-01-21T23:23:39.060742076Z'\n", " resourceLocation:\n", " currentLocations:\n", " - us\n", - " resourceName: projects/_/buckets/essentials-student31-2021-10-26\n", + " resourceName: projects/_/buckets/essentials-learner-2022-01-21\n", " serviceData:\n", " '@type': type.googleapis.com/google.iam.v1.logging.AuditData\n", " policyDelta:\n", " bindingDeltas:\n", " - action: ADD\n", - " member: projectEditor:just-armor-301114\n", + " member: projectEditor:essentials-learner\n", " role: roles/storage.legacyBucketOwner\n", " - action: ADD\n", - " member: projectOwner:just-armor-301114\n", + " member: projectOwner:essentials-learner\n", " role: roles/storage.legacyBucketOwner\n", " - action: ADD\n", - " member: projectViewer:just-armor-301114\n", + " member: projectViewer:essentials-learner\n", " role: roles/storage.legacyBucketReader\n", " serviceName: storage.googleapis.com\n", " status: {}\n", - "receiveTimestamp: '2021-10-26T20:47:06.793562424Z'\n", + "receiveTimestamp: '2022-01-21T23:23:39.729940907Z'\n", "resource:\n", " labels:\n", - " bucket_name: essentials-student31-2021-10-26\n", + " bucket_name: essentials-learner-2022-01-21\n", " location: us\n", - " project_id: just-armor-301114\n", + " project_id: essentials-learner\n", " type: gcs_bucket\n", "severity: NOTICE\n", - "timestamp: '2021-10-26T20:47:05.785456012Z'\n" + "timestamp: '2022-01-21T23:23:39.055676566Z'\n" ] } ], @@ -411,6 +415,23 @@ "gcloud logging read --limit 1" ] }, + { + "cell_type": "markdown", + "id": "e9e48571-80fa-40cc-838f-ec5ea3000bd7", + "metadata": {}, + "source": [ + "````{admonition} Note\n", + ":class: note\n", + "\n", + "You will probably need to enable the logging api to continue, press `y` to accept as shown below:\n", + "```\n", + "API [logging.googleapis.com] not enabled on project [998995103712]. Would you \n", + "like to enable and retry (this will take a few minutes)? (y/N)?\n", + "```\n", + "\n", + "````" + ] + }, { "cell_type": "markdown", "id": "fcdf7049-620f-4721-bc62-fa8f7fe97dcb", @@ -431,7 +452,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 10, "id": "f9a90752-871f-4156-b122-4c46fd4bfa43", "metadata": {}, "outputs": [], @@ -449,7 +470,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 11, "id": "bc0fb778-cc2e-4af4-8aa8-ca90d6b4f877", "metadata": {}, "outputs": [ @@ -475,7 +496,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 12, "id": "6384974b-d422-4ac1-90b7-cf9feec9aa8c", "metadata": {}, "outputs": [ @@ -495,7 +516,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 13, "id": "49589119-56bb-4c7b-b0a6-36e8f1133056", "metadata": {}, "outputs": [ @@ -503,7 +524,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-student31-2021-10-26/1\n" + "gs://essentials-learner-2022-01-21/1\n" ] } ], @@ -529,7 +550,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 14, "id": "442cc51f-8856-41d4-a3a6-072c8569beef", "metadata": {}, "outputs": [ @@ -547,7 +568,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 15, "id": "4e3c0de9-8a23-428a-9689-0cf896228afe", "metadata": {}, "outputs": [ @@ -573,7 +594,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 16, "id": "13f94b36-7dea-46b9-9e8b-baa8b4e59de6", "metadata": {}, "outputs": [ @@ -581,7 +602,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Copying gs://essentials-student31-2021-10-26/1...\n", + "Copying gs://essentials-learner-2022-01-21/1...\n", "/ [1 files][ 9.0 B/ 9.0 B] \n", "Operation completed over 1 objects/9.0 B. \n" ] @@ -593,7 +614,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 17, "id": "3982f21c-b827-4772-aa8c-3ac67e478b0d", "metadata": {}, "outputs": [ @@ -619,7 +640,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 18, "id": "6b57a3fc-94ea-4c25-b7ac-2744d7b28852", "metadata": {}, "outputs": [ @@ -645,7 +666,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 19, "id": "5a7fc453-fca7-4ab4-8f0a-cb4de53bc992", "metadata": {}, "outputs": [ @@ -681,8 +702,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-student31-2021-10-26/1\n", - "gs://essentials-student31-2021-10-26/2\n" + "gs://essentials-learner-2022-01-21/1\n", + "gs://essentials-learner-2022-01-21/2\n" ] } ], @@ -695,7 +716,10 @@ "id": "1427e837-f65f-4d35-8088-6a82677b5ed6", "metadata": {}, "source": [ - "Exercise: display the date in the bucket." + "```{admonition} Exercise\n", + "\n", + "* Display the date in the bucket.\n", + "```" ] }, { @@ -708,7 +732,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Tue 26 Oct 2021 07:49:59 PM UTC\n" + "Fri Jan 21 23:23:53 UTC 2022\n" ] } ], @@ -726,7 +750,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 22, "id": "9ad460d9-7a4f-4e3f-befb-2bbb10c51993", "metadata": {}, "outputs": [ @@ -734,7 +758,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-student31-2021-10-26/1...\n", + "Removing gs://essentials-learner-2022-01-21/1...\n", "/ [1 objects] \n", "Operation completed over 1 objects. \n" ] @@ -746,7 +770,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 23, "id": "8fd782f2-700a-4862-970b-1b4751009d5f", "metadata": {}, "outputs": [ @@ -754,7 +778,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-student31-2021-10-26/2\n" + "gs://essentials-learner-2022-01-21/2\n" ] } ], @@ -772,7 +796,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 24, "id": "e1350695-0f01-4205-a604-91d62896783e", "metadata": {}, "outputs": [ @@ -780,8 +804,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-student31-2021-10-26/...\n", - "NotEmptyException: 409 BucketNotEmpty (essentials-student31-2021-10-26)\n" + "Removing gs://essentials-learner-2022-01-21/...\n", + "NotEmptyException: 409 BucketNotEmpty (essentials-learner-2022-01-21)\n" ] } ], @@ -800,7 +824,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 25, "id": "ba2671f8-521f-48b4-9231-491d128ba67c", "metadata": {}, "outputs": [ @@ -808,7 +832,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-student31-2021-10-26/2...\n", + "Removing gs://essentials-learner-2022-01-21/2...\n", "/ [1 objects] \n", "Operation completed over 1 objects. \n" ] @@ -818,20 +842,20 @@ "gsutil rm \"gs://$BUCKET/2\"" ] }, - { - "cell_type": "markdown", - "id": "9540ba16-1662-49b5-9bf7-87c5c49b1e2c", - "metadata": {}, - "source": [ - "Finally we can remove the bucket. We first double check the value of `$BUCKET`" - ] - }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 26, "id": "73d8b299-8e8e-443e-ae5b-4ecf7efa0442", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-learner-2022-01-21/\n" + ] + } + ], "source": [ "gsutil ls" ] @@ -847,7 +871,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 27, "id": "7b32764d-e7b5-42d6-8f88-bc6936b2024a", "metadata": {}, "outputs": [ @@ -855,7 +879,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-student31-2021-10-26/\n" + "gs://essentials-learner-2022-01-21/\n" ] } ], @@ -873,7 +897,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 28, "id": "82487bd8-14f4-4811-8f5a-dd346042bd0f", "metadata": {}, "outputs": [ @@ -881,7 +905,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Bucket: essentials-student31-2021-10-26\n" + "Bucket: essentials-learner-2022-01-21\n" ] } ], @@ -891,7 +915,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 29, "id": "8635db94-2f3c-46d2-acc2-76a176fb37e7", "metadata": {}, "outputs": [ @@ -899,7 +923,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-student31-2021-10-26/...\n" + "Removing gs://essentials-learner-2022-01-21/...\n" ] } ], @@ -909,15 +933,15 @@ }, { "cell_type": "markdown", - "id": "9002604e-6908-4a86-ab45-b9dd4cb11f75", + "id": "3e2178ca", "metadata": {}, "source": [ - "We verify that the bucket has been removed. In this example there is not output since there are no more buckets." + "We verify that the bucket has been removed. In this example there is no output since there are no more buckets." ] }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 30, "id": "ed6f0204-0de2-4d20-ab00-3b7a1023fd86", "metadata": {}, "outputs": [], diff --git a/content/GCP/06_running_analysis.ipynb b/content/GCP/06_running_analysis.ipynb index 052ffe8..72780cd 100644 --- a/content/GCP/06_running_analysis.ipynb +++ b/content/GCP/06_running_analysis.ipynb @@ -10,7 +10,7 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 80 min\n", "\n", "**Exercises:**\n", "\n", @@ -19,15 +19,20 @@ "\n", "**Objectives:**\n", "* Create a simple workflow using a cloud VM and cloud object storage.\n", - "* Understand the basics of Identity and Access Management (IAM)\n", - "* Add collaborators to a Bucket with appropriate permissions.\n", - "\n", - "\n", - "\n", "\n", "```" ] }, + { + "cell_type": "markdown", + "id": "06ef4791-862e-4c05-afc8-98d440c47f73", + "metadata": {}, + "source": [ + "## A Research Computational and Data Workflow - Drew's story\n", + "\n", + "Drew needs to do some analysis on the data. They need data (satellite images stored in the cloud), computational resources (a virtual machine), some software (we will supply this), and a place to store the results (Cloud Storage). We will assemble all these parts in the cloud \n" + ] + }, { "cell_type": "markdown", "id": "6291edee-c2df-4941-9b8e-de42649640f9", @@ -35,14 +40,20 @@ "source": [ "## Create a VM\n", "\n", - "Since we only create resources as we need them in the cloud. As an exercise you will now create a VM for our analysis. In this case will give the VM **Full** access to **Storage**. \n", + "Since we only create resources as we need them in the cloud, we will now create a new virtual machine (VM) for Drew to use for their analysis.\n", + "\n", + "We will do this as an exercise to give you practice in creating resources. Since the virtual machine will need access to storage on your behalf, you will need to change the **access scope** to give **Full** access to the **Storage** API to the virtual machine. \n", + "\n", + "### Exercise\n", "\n", "Using the console navigate to the \"Compute Engine\" service and create a new VM with the following properties.\n", " * Call the VM \"essentials\"\n", - " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", - " * Feel free to select a larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", - " \n", - "When you are done connect to the machine as described below." + " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" on the \"create an instance\" page and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", + " * Feel free to select a bit larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", + "\n", + "*Instructor: place these instructions on the screen*\n", + "\n", + "*When you are done feel free to connect to the virtual machine on your own for additional practice. Once everyone has created their VM we will connect to the machine as described below.*" ] }, { @@ -52,7 +63,7 @@ "source": [ "## Connect to the VM\n", "\n", - "First login to the instance from the Cloud Shell by running the following command:\n", + "Now login to the new virtual machine instance by opening up the Cloud Shell and by running the following command:\n", "```\n", "gcloud compute ssh essentials\n", "```\n", @@ -103,13 +114,19 @@ "output_type": "stream", "text": [ "Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]\n", - "Hit:2 http://deb.debian.org/debian buster InRelease\n", - "Hit:3 http://deb.debian.org/debian buster-updates InRelease\n", - "Hit:4 http://deb.debian.org/debian buster-backports InRelease\n", + "Hit:2 http://deb.debian.org/debian buster InRelease \u001b[0m\u001b[33m\n", + "Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]\n", + "Get:4 http://deb.debian.org/debian buster-backports InRelease [46.7 kB]\n", "Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease\n", "Hit:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease\n", - "Get:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease [5526 B]\n", - "Fetched 70.9 kB in 1s (91.3 kB/s) \u001b[0m\u001b[33m\n", + "Hit:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease\n", + "Get:8 http://deb.debian.org/debian buster-backports/main amd64 Packages.diff/Index [27.8 kB]\n", + "Get:9 http://deb.debian.org/debian buster-backports/main Translation-en.diff/Index [27.8 kB]\n", + "Get:10 http://deb.debian.org/debian buster-backports/main amd64 Packages 2022-01-21-2002.00.pdiff [2515 B]\n", + "Get:11 http://deb.debian.org/debian buster-backports/main Translation-en 2022-01-21-2002.00.pdiff [3068 B]\n", + "Get:10 http://deb.debian.org/debian buster-backports/main amd64 Packages 2022-01-21-2002.00.pdiff [2515 B]\n", + "Get:11 http://deb.debian.org/debian buster-backports/main Translation-en 2022-01-21-2002.00.pdiff [3068 B]\n", + "Fetched 225 kB in 1s (278 kB/s)0m\u001b[33m \u001b[0m\u001b[33m\n", "Reading package lists... Done\n", "Building dependency tree \n", "Reading state information... Done\n", @@ -122,6 +139,93 @@ "sudo unattended-upgrades" ] }, + { + "cell_type": "markdown", + "id": "cd0c7010-ef68-4648-9766-ab26e9bd6ecc", + "metadata": { + "tags": [] + }, + "source": [ + "## Setup Storage\n", + "\n", + "Before we do any work we will first create a bucket to place the results with a reasonable set of options. We do this first to make sure we can store the results when we are done, it is easier to fix problems now than later. \n", + "\n", + "We first store the bucket name in the `BUCKET` environment variable for future use. This time we will specify a realistic set of options for a private bucket used for computation.\n", + "\n", + "Options (run `gsutil mb --help` for more information):\n", + " * `-b on` specifies uniform bucket-level access.\n", + " * `-l $REGION` puts the data in a specific region for lower cost and lower latency.\n", + " * `--pap enforced` turns on public access prevention to help keep data private. \n", + " \n", + "The uniform bucket level access (Bucket Policy Only enabled: true) puts the data access permissions (ACL) on the entire bucket, not on each object in the bucket. This makes the permissions obvious and makes security much more predictable.\n", + " \n", + "As usual, we must set our environment. In this case we also set a `REGION` environment variable to indicate where in the world we want the data to be stored.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "f36cb8c5-f305-4cb2-a5cc-0c9fd8592fb4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "bucket: essentials-learner-2022-01-21 region: us-west2\n" + ] + } + ], + "source": [ + "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", + "REGION=\"us-west2\"\n", + "echo \"bucket: $BUCKET region: $REGION\"" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "c2ae2b74-5e93-4c55-8bd7-63337f7dcbb8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating gs://essentials-learner-2022-01-21/...\n" + ] + } + ], + "source": [ + "gsutil mb -b on -l $REGION --pap enforced \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "2c71d0de-8bdb-476f-922f-62dc19e8bbeb", + "metadata": {}, + "source": [ + "And verify the bucket was created" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "10f8c773-97fc-46af-a6b5-1bb832472b33", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-learner-2022-01-21/\n" + ] + } + ], + "source": [ + "gsutil ls -b gs://$BUCKET" + ] + }, { "cell_type": "markdown", "id": "b7c4db9e-f098-41bc-80d2-b524444eec7f", @@ -134,7 +238,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 5, "id": "05fadd30-4f35-4a93-b713-cc132131b948", "metadata": {}, "outputs": [ @@ -156,7 +260,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 6, "id": "96db6a66-3fbf-419a-b8c8-dbb27639e990", "metadata": {}, "outputs": [], @@ -166,7 +270,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 7, "id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7", "metadata": {}, "outputs": [ @@ -195,7 +299,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 8, "id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab", "metadata": {}, "outputs": [], @@ -216,7 +320,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 9, "id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2", "metadata": {}, "outputs": [ @@ -225,13 +329,13 @@ "output_type": "stream", "text": [ "total 28\n", - "-rw-r--r-- 1 learner learner 960 Dec 17 19:00 ReadMe.md\n", - "-rw-r--r-- 1 learner learner 72 Dec 17 19:00 clean.sh\n", - "-rw-r--r-- 1 learner learner 613 Dec 17 19:00 combine.py\n", - "-rw-r--r-- 1 learner learner 280 Dec 17 19:00 download.sh\n", - "-rw-r--r-- 1 learner learner 314 Dec 17 19:00 get-index.sh\n", - "-rw-r--r-- 1 learner learner 76 Dec 17 19:00 search.json\n", - "-rw-r--r-- 1 learner learner 783 Dec 17 19:00 search.py\n" + "-rw-r--r-- 1 learner learner 960 Jan 21 21:32 ReadMe.md\n", + "-rw-r--r-- 1 learner learner 72 Jan 21 21:32 clean.sh\n", + "-rw-r--r-- 1 learner learner 613 Jan 21 21:32 combine.py\n", + "-rw-r--r-- 1 learner learner 280 Jan 21 21:32 download.sh\n", + "-rw-r--r-- 1 learner learner 314 Jan 21 21:32 get-index.sh\n", + "-rw-r--r-- 1 learner learner 76 Jan 21 21:32 search.json\n", + "-rw-r--r-- 1 learner learner 783 Jan 21 21:32 search.py\n" ] } ], @@ -246,12 +350,14 @@ "source": [ "## Access the bucket\n", "\n", - "First test that our tools are working and that we can access the public bucket that we will be using." + "Now we need to verify that Drew has access to the analysis data. \n", + "\n", + "We do this by testing that our tools are working and that we can access the public bucket that we will be using." ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 10, "id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd", "metadata": {}, "outputs": [ @@ -278,19 +384,29 @@ "gsutil ls gs://gcp-public-data-landsat/" ] }, + { + "cell_type": "markdown", + "id": "9e16e8b5-a178-492a-aa80-5affe721b6ca", + "metadata": {}, + "source": [ + "## Getting the data" + ] + }, { "cell_type": "markdown", "id": "89acffba-cbce-436c-98dd-05467b6675a6", "metadata": {}, "source": [ - "The index file is a list of all the files in the bucket and we can use it to search and filter files.\n", + "Since the Landsat data is *huge* we do not, and cannot, download everything to the virtual machine. We will only analyzing a subset of the data.\n", + "\n", + "We will use the the `index.csv.gz` file, which is a list of all the files and additional metadata in the bucket and we can use it to search and filter the data.\n", "\n", - "We will get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region." + "We will first get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region." ] }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 11, "id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e", "metadata": {}, "outputs": [ @@ -308,7 +424,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 12, "id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38", "metadata": {}, "outputs": [ @@ -322,8 +438,8 @@ "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", "\n", - "\\ [1 files][761.0 MiB/761.0 MiB] 42.0 MiB/s \n", - "Operation completed over 1 objects/761.0 MiB. \n" + "| [1 files][731.9 MiB/731.9 MiB] \n", + "Operation completed over 1 objects/731.9 MiB. \n" ] } ], @@ -336,12 +452,12 @@ "id": "76b905b4-1c2a-4960-a14f-974b77f671cd", "metadata": {}, "source": [ - "We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using." + "We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using. (This is also why it is good to write scripts to do the entire process)." ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 13, "id": "2cdaf24c-c4aa-4e80-9236-939e7c982916", "metadata": {}, "outputs": [], @@ -359,7 +475,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 14, "id": "b005876c-f9af-43d6-80c6-f04295413b9b", "metadata": {}, "outputs": [ @@ -367,8 +483,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "total 2.6G\n", - "-rw-r--r-- 1 learner learner 2.6G Dec 17 19:01 index.csv\n" + "total 2.5G\n", + "-rw-r--r-- 1 learner learner 2.5G Jan 21 21:32 index.csv\n" ] } ], @@ -386,7 +502,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 15, "id": "ffe969db-d207-44fe-8957-8d129c76ee8f", "metadata": {}, "outputs": [ @@ -395,9 +511,9 @@ "output_type": "stream", "text": [ "SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL\n", - "LM51940171984262KIS00,LM05_L1GS_194017_19840918_20210913_01_T2,LANDSAT_5,MSS,1984-09-18,01,T2,1984-09-18T09:36:27.0970000Z,L1GS,194,17,51.0,62.58023,60.38359,14.45307,19.13901,34758212,gs://gcp-public-data-landsat/LM05/01/194/017/LM05_L1GS_194017_19840918_20210913_01_T2\n", - "LM10420331972311AAA04,LM01_L1TP_042033_19721106_20180429_01_T2,LANDSAT_1,MSS,1972-11-06,01,T2,1972-11-06T17:49:00.5000000Z,L1TP,42,33,3.0,39.68891,37.707,-115.59896,-112.90344,26932395,gs://gcp-public-data-landsat/LM01/01/042/033/LM01_L1TP_042033_19721106_20180429_01_T2\n", - "LT50150151992246PAC00,LT05_L1TP_015015_19920902_20170121_01_T1,LANDSAT_5,TM,1992-09-02,01,T1,1992-09-02T15:01:34.1390130Z,L1TP,15,15,90.0,65.32118,63.11726,-67.3438,-61.89671,118785510,gs://gcp-public-data-landsat/LT05/01/015/015/LT05_L1TP_015015_19920902_20170121_01_T1\n" + "LE71800592011134ASN00,LE07_L1TP_180059_20110514_20161209_01_T1,LANDSAT_7,ETM,2011-05-14,01,T1,2011-05-14T08:50:07.5251363Z,L1TP,180,59,74.0,2.39913,0.50961,18.09062,20.2487,181281962,gs://gcp-public-data-landsat/LE07/01/180/059/LE07_L1TP_180059_20110514_20161209_01_T1\n", + "LT51360422008226BKT00,LT05_L1GS_136042_20080813_20161030_01_T2,LANDSAT_5,TM,2008-08-13,01,T2,2008-08-13T04:03:49.0450690Z,L1GS,136,42,92.0,26.9495,25.03915,91.40541,93.81099,141994748,gs://gcp-public-data-landsat/LT05/01/136/042/LT05_L1GS_136042_20080813_20161030_01_T2\n", + "LE71760312020339NSG00,LE07_L1TP_176031_20201204_20201230_01_T1,LANDSAT_7,ETM,2020-12-04,01,T1,2020-12-04T07:41:11.6084536Z,L1TP,176,31,3.0,42.75649,40.7935,33.66313,36.65653,188511155,gs://gcp-public-data-landsat/LE07/01/176/031/LE07_L1TP_176031_20201204_20201230_01_T1\n" ] } ], @@ -417,7 +533,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 16, "id": "c5e300c3-e1f3-4cd4-9679-77725e61c4db", "metadata": {}, "outputs": [ @@ -461,7 +577,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 17, "id": "c9872510-4265-4b0e-aeb5-5a829ff69b24", "metadata": {}, "outputs": [ @@ -491,7 +607,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 18, "id": "6912a9ec-0f9b-4500-ba20-d4280592b323", "metadata": {}, "outputs": [ @@ -499,11 +615,10 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20150127_20170302_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20161028_20170219_01_T1\n", - "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20191224_20200110_01_T1\n" + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20200601_20200608_01_T1\n", + "gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1GT_025033_20150111_20170302_01_T2\n" ] } ], @@ -523,7 +638,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 19, "id": "3572c518-df83-4906-bfa6-a37bde2a5063", "metadata": {}, "outputs": [ @@ -558,7 +673,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 20, "id": "cccec3e1-0dcd-4e3b-a059-a884f5219b66", "metadata": { "scrolled": true, @@ -569,50 +684,50 @@ "name": "stdout", "output_type": "stream", "text": [ - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_ANG.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B3.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B9.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B8.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B6.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_BQA.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B11.TIF...\n", + "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B5.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B7.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B8.TIF...\n", "==> NOTE: You are downloading one or more large file(s), which would\n", "run significantly faster if you enabled sliced object downloads. This\n", "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", "\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B10.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_MTL.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B5.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B2.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20211010_20211018_01_T1/LC08_L1TP_025033_20211010_20211018_01_T1_B7.TIF...\n", - "| [14/14 files][997.2 MiB/997.2 MiB] 100% Done 65.3 MiB/s ETA 00:00:00 \n", - "Operation completed over 14 objects/997.2 MiB. \n", - "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_ANG.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B1.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B11.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B10.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B8.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B9.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B3.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_BQA.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B6.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B7.TIF...\n", - "==> NOTE: You are downloading one or more large file(s), which would \n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B9.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_MTL.txt...\n", + "- [14/14 files][952.9 MiB/952.9 MiB] 100% Done \n", + "Operation completed over 14 objects/952.9 MiB. \n", + "+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_ANG.txt...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B3.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B1.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B10.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B4.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B5.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B6.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B11.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B2.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B7.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B8.TIF...\n", + "==> NOTE: You are downloading one or more large file(s), which would\n", "run significantly faster if you enabled sliced object downloads. This\n", "feature is enabled by default but requires that compiled crcmod be\n", "installed (see \"gsutil help crcmod\").\n", "\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B2.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_MTL.txt...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B4.TIF...\n", - "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20170913_20170928_01_T1/LC08_L1TP_025033_20170913_20170928_01_T1_B5.TIF...\n", - "- [14/14 files][979.2 MiB/979.2 MiB] 100% Done 14.9 MiB/s ETA 00:00:00 \n", - "Operation completed over 14 objects/979.2 MiB. \n" + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_B9.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_BQA.TIF...\n", + "Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20210519_20210528_01_T1/LC08_L1TP_025033_20210519_20210528_01_T1_MTL.txt...\n", + "/ [14/14 files][ 1.0 GiB/ 1.0 GiB] 100% Done 30.3 MiB/s ETA 00:00:00 \n", + "Operation completed over 14 objects/1.0 GiB. \n" ] } ], @@ -630,7 +745,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 21, "id": "a37c1567-14b5-4dc7-bc27-d1b84411fce1", "metadata": {}, "outputs": [ @@ -638,10 +753,10 @@ "name": "stdout", "output_type": "stream", "text": [ - "total 2665792\n", - "drwxr-xr-x 2 learner learner 4096 Dec 17 19:02 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20170913_20170928_01_T1\u001b[0m\n", - "drwxr-xr-x 2 learner learner 4096 Dec 17 19:01 \u001b[01;34mLC08_L1TP_025033_20211010_20211018_01_T1\u001b[0m\n", - "-rw-r--r-- 1 learner learner 2729754843 Dec 17 19:01 index.csv\n" + "total 2564796\n", + "drwxr-xr-x 2 learner learner 4096 Jan 21 21:33 \u001b[0m\u001b[01;34mLC08_L1TP_025033_20201007_20201016_01_T1\u001b[0m\n", + "drwxr-xr-x 2 learner learner 4096 Jan 21 21:33 \u001b[01;34mLC08_L1TP_025033_20210519_20210528_01_T1\u001b[0m\n", + "-rw-r--r-- 1 learner learner 2626336574 Jan 21 21:32 index.csv\n" ] } ], @@ -661,7 +776,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 22, "id": "0c027e92-ae6f-4152-b8d6-5a70172de3e2", "metadata": {}, "outputs": [ @@ -697,7 +812,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 23, "id": "77999a80-7cfd-46d3-86a6-05d199f7e66d", "metadata": {}, "outputs": [ @@ -728,7 +843,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 24, "id": "b5a2b29b-9c1d-4376-a3eb-4a3dc4bac160", "metadata": { "scrolled": true, @@ -796,75 +911,75 @@ "Get:9 http://deb.debian.org/debian buster/main amd64 liblapack3 amd64 3.8.0-2 [2110 kB]\n", "Get:10 http://deb.debian.org/debian buster/main amd64 libarpack2 amd64 3.7.0-2 [102 kB]\n", "Get:11 http://deb.debian.org/debian buster/main amd64 libsuperlu5 amd64 5.2.1+dfsg1-4 [161 kB]\n", - "Get:12 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", - "Get:13 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", - "Get:14 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", - "Get:15 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", - "Get:16 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", + "Get:12 http://security.debian.org/debian-security buster/updates/main amd64 libtiff5 amd64 4.1.0+git191117-2~deb10u3 [271 kB]\n", + "Get:13 http://deb.debian.org/debian buster/main amd64 libarmadillo9 amd64 1:9.200.7+dfsg-1 [88.6 kB]\n", + "Get:14 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u4 [1161 kB]\n", + "Get:15 http://deb.debian.org/debian buster/main amd64 libcharls2 amd64 2.0.0+dfsg-1 [64.3 kB]\n", + "Get:16 http://deb.debian.org/debian buster/main amd64 libxml2 amd64 2.9.4+dfsg1-7+deb10u2 [689 kB]\n", "Get:17 http://deb.debian.org/debian buster/main amd64 libdap25 amd64 3.20.3-1 [557 kB]\n", - "Get:18 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", - "Get:19 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", - "Get:20 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", - "Get:21 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", - "Get:22 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", - "Get:23 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", - "Get:24 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", - "Get:25 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", - "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", - "Get:27 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", - "Get:28 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", - "Get:29 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", - "Get:30 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", - "Get:31 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", - "Get:32 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", - "Get:33 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", - "Get:34 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", - "Get:35 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", - "Get:36 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", - "Get:37 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", - "Get:38 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", - "Get:39 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", - "Get:40 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", - "Get:41 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", - "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlconvenience1 amd64 1.3.0-7 [50.1 kB]\n", - "Get:43 http://deb.debian.org/debian buster/main amd64 libkmlregionator1 amd64 1.3.0-7 [25.1 kB]\n", - "Get:44 http://deb.debian.org/debian buster/main amd64 libkmlxsd1 amd64 1.3.0-7 [34.6 kB]\n", - "Get:45 http://deb.debian.org/debian buster/main amd64 mysql-common all 5.8+1.0.5 [7324 B]\n", - "Get:46 http://deb.debian.org/debian buster/main amd64 mariadb-common all 1:10.3.31-0+deb10u1 [32.7 kB]\n", - "Get:47 http://deb.debian.org/debian buster/main amd64 libmariadb3 amd64 1:10.3.31-0+deb10u1 [173 kB]\n", - "Get:48 http://deb.debian.org/debian buster/main amd64 libnetcdf13 amd64 1:4.6.2-1 [403 kB]\n", - "Get:49 http://deb.debian.org/debian buster/main amd64 libltdl7 amd64 2.4.6-9 [390 kB]\n", - "Get:50 http://deb.debian.org/debian buster/main amd64 libodbc1 amd64 2.3.6-0.1 [223 kB]\n", - "Get:51 http://deb.debian.org/debian buster/main amd64 libogdi3.2 amd64 3.2.1+ds-4 [241 kB]\n", - "Get:52 http://deb.debian.org/debian buster/main amd64 libopenjp2-7 amd64 2.3.0-2+deb10u2 [158 kB]\n", - "Get:53 http://deb.debian.org/debian buster/main amd64 liblcms2-2 amd64 2.9-3 [145 kB]\n", - "Get:54 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]\n", - "Get:55 http://deb.debian.org/debian buster/main amd64 libpoppler82 amd64 0.71.0-5 [1507 kB]\n", - "Get:56 http://deb.debian.org/debian buster/main amd64 libqhull7 amd64 2015.2-4 [196 kB]\n", - "Get:57 http://deb.debian.org/debian buster/main amd64 libspatialite7 amd64 4.3.0a-5+b2 [1256 kB]\n", - "Get:58 http://deb.debian.org/debian buster/main amd64 libxerces-c3.2 amd64 3.2.2+debian-1+deb10u1 [886 kB]\n", - "Get:59 http://deb.debian.org/debian buster/main amd64 odbcinst amd64 2.3.6-0.1 [48.4 kB]\n", - "Get:60 http://deb.debian.org/debian buster/main amd64 odbcinst1debian2 amd64 2.3.6-0.1 [78.1 kB]\n", - "Get:61 http://deb.debian.org/debian buster/main amd64 libgdal20 amd64 2.4.0+dfsg-1+b1 [6171 kB]\n", - "Get:62 http://deb.debian.org/debian buster/main amd64 libxslt1.1 amd64 1.1.32-2.2~deb10u1 [237 kB]\n", - "Get:63 http://deb.debian.org/debian buster/main amd64 proj-bin amd64 5.2.0-1 [105 kB]\n", - "Get:64 http://deb.debian.org/debian buster/main amd64 python3-affine all 2.2.2-1 [14.5 kB]\n", - "Get:65 http://deb.debian.org/debian buster/main amd64 python3-attr all 18.2.0-1 [37.3 kB]\n", - "Get:66 http://deb.debian.org/debian buster/main amd64 python3-soupsieve all 1.8+dfsg-1 [27.6 kB]\n", - "Get:67 http://deb.debian.org/debian buster/main amd64 python3-bs4 all 4.7.1-1 [94.1 kB]\n", - "Get:68 http://deb.debian.org/debian buster/main amd64 python3-chardet all 3.0.4-3 [80.5 kB]\n", - "Get:69 http://deb.debian.org/debian buster/main amd64 python3-colorama all 0.3.7-1 [18.1 kB]\n", - "Get:70 http://deb.debian.org/debian buster/main amd64 python3-click all 7.0-1 [73.6 kB]\n", - "Get:71 http://deb.debian.org/debian buster/main amd64 python3-click-plugins all 1.0.4-1 [7540 B]\n", - "Get:72 http://deb.debian.org/debian buster/main amd64 python3-cligj all 0.5.0-1 [8212 B]\n", - "Get:73 http://deb.debian.org/debian buster/main amd64 python3-webencodings all 0.5.1-1 [10.9 kB]\n", - "Get:74 http://deb.debian.org/debian buster/main amd64 python3-html5lib all 1.0.1-1 [89.5 kB]\n", - "Get:75 http://deb.debian.org/debian buster/main amd64 python3-lxml amd64 4.3.2-1+deb10u3 [1162 kB]\n", + "Get:18 http://security.debian.org/debian-security buster/updates/main amd64 libpq5 amd64 11.14-0+deb10u1 [171 kB]\n", + "Get:19 http://security.debian.org/debian-security buster/updates/main amd64 python3-lxml amd64 4.3.2-1+deb10u4 [1163 kB]\n", + "Get:20 http://deb.debian.org/debian buster/main amd64 libdapclient6v5 amd64 3.20.3-1 [202 kB]\n", + "Get:21 http://deb.debian.org/debian buster/main amd64 libdapserver7v5 amd64 3.20.3-1 [131 kB]\n", + "Get:22 http://deb.debian.org/debian buster/main amd64 libepsilon1 amd64 0.9.2+dfsg-4 [42.0 kB]\n", + "Get:23 http://deb.debian.org/debian buster/main amd64 libfontconfig1 amd64 2.13.1-2 [346 kB]\n", + "Get:24 http://deb.debian.org/debian buster/main amd64 libfreexl1 amd64 1.0.5-3 [34.1 kB]\n", + "Get:25 http://deb.debian.org/debian buster/main amd64 libfyba0 amd64 4.1.1-6 [114 kB]\n", + "Get:26 http://deb.debian.org/debian buster/main amd64 libgeos-3.7.1 amd64 3.7.1-1 [735 kB]\n", + "Get:27 http://deb.debian.org/debian buster/main amd64 libgeos-c1v5 amd64 3.7.1-1 [299 kB]\n", + "Get:28 http://deb.debian.org/debian buster/main amd64 proj-data all 5.2.0-1 [6986 kB]\n", + "Get:29 http://deb.debian.org/debian buster/main amd64 libproj13 amd64 5.2.0-1 [225 kB]\n", + "Get:30 http://deb.debian.org/debian buster/main amd64 libjbig0 amd64 2.1-3.1+b2 [31.0 kB]\n", + "Get:31 http://deb.debian.org/debian buster/main amd64 libjpeg62-turbo amd64 1:1.5.2-2+deb10u1 [133 kB]\n", + "Get:32 http://deb.debian.org/debian buster/main amd64 libwebp6 amd64 0.6.1-2+deb10u1 [261 kB]\n", + "Get:33 http://deb.debian.org/debian buster/main amd64 libgeotiff2 amd64 1.4.3-1 [72.0 kB]\n", + "Get:34 http://deb.debian.org/debian buster/main amd64 libgif7 amd64 5.1.4-3 [43.3 kB]\n", + "Get:35 http://deb.debian.org/debian buster/main amd64 libhdf4-0-alt amd64 4.2.13-4 [277 kB]\n", + "Get:36 http://deb.debian.org/debian buster/main amd64 libsz2 amd64 1.0.2-1 [6676 B]\n", + "Get:37 http://deb.debian.org/debian buster/main amd64 libhdf5-103 amd64 1.10.4+repack-10 [1325 kB]\n", + "Get:38 http://deb.debian.org/debian buster/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]\n", + "Get:39 http://deb.debian.org/debian buster/main amd64 liburiparser1 amd64 0.9.1-1 [47.8 kB]\n", + "Get:40 http://deb.debian.org/debian buster/main amd64 libkmlbase1 amd64 1.3.0-7 [48.9 kB]\n", + "Get:41 http://deb.debian.org/debian buster/main amd64 libkmldom1 amd64 1.3.0-7 [172 kB]\n", + "Get:42 http://deb.debian.org/debian buster/main amd64 libkmlengine1 amd64 1.3.0-7 [78.4 kB]\n", + "Get:43 http://deb.debian.org/debian buster/main amd64 libkmlconvenience1 amd64 1.3.0-7 [50.1 kB]\n", + "Get:44 http://deb.debian.org/debian buster/main amd64 libkmlregionator1 amd64 1.3.0-7 [25.1 kB]\n", + "Get:45 http://deb.debian.org/debian buster/main amd64 libkmlxsd1 amd64 1.3.0-7 [34.6 kB]\n", + "Get:46 http://deb.debian.org/debian buster/main amd64 mysql-common all 5.8+1.0.5 [7324 B]\n", + "Get:47 http://deb.debian.org/debian buster/main amd64 mariadb-common all 1:10.3.31-0+deb10u1 [32.7 kB]\n", + "Get:48 http://deb.debian.org/debian buster/main amd64 libmariadb3 amd64 1:10.3.31-0+deb10u1 [173 kB]\n", + "Get:49 http://deb.debian.org/debian buster/main amd64 libnetcdf13 amd64 1:4.6.2-1 [403 kB]\n", + "Get:50 http://deb.debian.org/debian buster/main amd64 libltdl7 amd64 2.4.6-9 [390 kB]\n", + "Get:51 http://deb.debian.org/debian buster/main amd64 libodbc1 amd64 2.3.6-0.1 [223 kB]\n", + "Get:52 http://deb.debian.org/debian buster/main amd64 libogdi3.2 amd64 3.2.1+ds-4 [241 kB]\n", + "Get:53 http://deb.debian.org/debian buster/main amd64 libopenjp2-7 amd64 2.3.0-2+deb10u2 [158 kB]\n", + "Get:54 http://deb.debian.org/debian buster/main amd64 liblcms2-2 amd64 2.9-3 [145 kB]\n", + "Get:55 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]\n", + "Get:56 http://deb.debian.org/debian buster/main amd64 libpoppler82 amd64 0.71.0-5 [1507 kB]\n", + "Get:57 http://deb.debian.org/debian buster/main amd64 libqhull7 amd64 2015.2-4 [196 kB]\n", + "Get:58 http://deb.debian.org/debian buster/main amd64 libspatialite7 amd64 4.3.0a-5+b2 [1256 kB]\n", + "Get:59 http://deb.debian.org/debian buster/main amd64 libxerces-c3.2 amd64 3.2.2+debian-1+deb10u1 [886 kB]\n", + "Get:60 http://deb.debian.org/debian buster/main amd64 odbcinst amd64 2.3.6-0.1 [48.4 kB]\n", + "Get:61 http://deb.debian.org/debian buster/main amd64 odbcinst1debian2 amd64 2.3.6-0.1 [78.1 kB]\n", + "Get:62 http://deb.debian.org/debian buster/main amd64 libgdal20 amd64 2.4.0+dfsg-1+b1 [6171 kB]\n", + "Get:63 http://deb.debian.org/debian buster/main amd64 libxslt1.1 amd64 1.1.32-2.2~deb10u1 [237 kB]\n", + "Get:64 http://deb.debian.org/debian buster/main amd64 proj-bin amd64 5.2.0-1 [105 kB]\n", + "Get:65 http://deb.debian.org/debian buster/main amd64 python3-affine all 2.2.2-1 [14.5 kB]\n", + "Get:66 http://deb.debian.org/debian buster/main amd64 python3-attr all 18.2.0-1 [37.3 kB]\n", + "Get:67 http://deb.debian.org/debian buster/main amd64 python3-soupsieve all 1.8+dfsg-1 [27.6 kB]\n", + "Get:68 http://deb.debian.org/debian buster/main amd64 python3-bs4 all 4.7.1-1 [94.1 kB]\n", + "Get:69 http://deb.debian.org/debian buster/main amd64 python3-chardet all 3.0.4-3 [80.5 kB]\n", + "Get:70 http://deb.debian.org/debian buster/main amd64 python3-colorama all 0.3.7-1 [18.1 kB]\n", + "Get:71 http://deb.debian.org/debian buster/main amd64 python3-click all 7.0-1 [73.6 kB]\n", + "Get:72 http://deb.debian.org/debian buster/main amd64 python3-click-plugins all 1.0.4-1 [7540 B]\n", + "Get:73 http://deb.debian.org/debian buster/main amd64 python3-cligj all 0.5.0-1 [8212 B]\n", + "Get:74 http://deb.debian.org/debian buster/main amd64 python3-webencodings all 0.5.1-1 [10.9 kB]\n", + "Get:75 http://deb.debian.org/debian buster/main amd64 python3-html5lib all 1.0.1-1 [89.5 kB]\n", "Get:76 http://deb.debian.org/debian buster/main amd64 python3-numpy amd64 1:1.16.2-1 [2119 kB]\n", "Get:77 http://deb.debian.org/debian buster/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [89.6 kB]\n", "Get:78 http://deb.debian.org/debian buster/main amd64 python3-snuggs all 1.4.3-1 [7228 B]\n", "Get:79 http://deb.debian.org/debian buster/main amd64 python3-rasterio amd64 1.0.21-1 [818 kB]\n", - "Fetched 46.8 MB in 1s (85.4 MB/s) \n", + "Fetched 46.8 MB in 2s (28.9 MB/s) \n", "Extracting templates from packages: 100%\n", "Preconfiguring packages ...\n", "Selecting previously unselected package poppler-data.\n", @@ -1091,8 +1206,8 @@ "Preparing to unpack .../73-python3-html5lib_1.0.1-1_all.deb ...\n", "Unpacking python3-html5lib (1.0.1-1) ...\n", "Selecting previously unselected package python3-lxml:amd64.\n", - "Preparing to unpack .../74-python3-lxml_4.3.2-1+deb10u3_amd64.deb ...\n", - "Unpacking python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", + "Preparing to unpack .../74-python3-lxml_4.3.2-1+deb10u4_amd64.deb ...\n", + "Unpacking python3-lxml:amd64 (4.3.2-1+deb10u4) ...\n", "Selecting previously unselected package python3-numpy.\n", "Preparing to unpack .../75-python3-numpy_1%3a1.16.2-1_amd64.deb ...\n", "Unpacking python3-numpy (1:1.16.2-1) ...\n", @@ -1181,7 +1296,7 @@ "Setting up libpoppler82:amd64 (0.71.0-5) ...\n", "Setting up libnetcdf13:amd64 (1:4.6.2-1) ...\n", "Setting up python3-numpy (1:1.16.2-1) ...\n", - "Setting up python3-lxml:amd64 (4.3.2-1+deb10u3) ...\n", + "Setting up python3-lxml:amd64 (4.3.2-1+deb10u4) ...\n", "Setting up libarmadillo9 (1:9.200.7+dfsg-1) ...\n", "Setting up python3-snuggs (1.4.3-1) ...\n", "Setting up odbcinst (2.3.6-0.1) ...\n", @@ -1200,7 +1315,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 25, "id": "b9e367a0-26ce-42ce-bb04-6a432f41876e", "metadata": {}, "outputs": [ @@ -1208,10 +1323,10 @@ "name": "stdout", "output_type": "stream", "text": [ - "LC08_L1TP_025033_20211010_20211018_01_T1 1\n", - "LC08_L1TP_025033_20211010_20211018_01_T1 2\n", - "LC08_L1TP_025033_20211010_20211018_01_T1 3\n", - "ERROR 4: output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png: No such file or directory\n" + "LC08_L1TP_025033_20201007_20201016_01_T1 1\n", + "LC08_L1TP_025033_20201007_20201016_01_T1 2\n", + "LC08_L1TP_025033_20201007_20201016_01_T1 3\n", + "ERROR 4: output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png: No such file or directory\n" ] } ], @@ -1237,7 +1352,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 26, "id": "db9f26aa-6317-4834-8bf1-972c8b3cc032", "metadata": {}, "outputs": [ @@ -1245,9 +1360,9 @@ "name": "stdout", "output_type": "stream", "text": [ - "total 195M\n", - "-rw-r--r-- 1 learner learner 195M Dec 17 19:03 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20211010_20211018_01_T1.png\u001b[0m\n", - "-rw-r--r-- 1 learner learner 910 Dec 17 19:03 result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n" + "total 192M\n", + "-rw-r--r-- 1 learner learner 192M Jan 21 21:34 \u001b[0m\u001b[01;35mresult-LC08_L1TP_025033_20201007_20201016_01_T1.png\u001b[0m\n", + "-rw-r--r-- 1 learner learner 910 Jan 21 21:34 result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n" ] } ], @@ -1257,74 +1372,63 @@ }, { "cell_type": "markdown", - "id": "c0257075-537c-4510-bafd-72e9756db17b", + "id": "6c63e0c4-1476-4f0b-93b0-e5bd2b406e60", "metadata": {}, "source": [ - "## Exporting the Results\n", + "## Saving the Results\n", "\n", - "Now that we have the output data we will create a bucket to place the results. We will first create a bucket with a reasonable set of options.\n", - "\n", - "We fisrt store the bucket name in the `BUCKET` environment variable for future use. This time we will specify a realistic set of options for a private bucket used for computation.\n", - "\n", - "Options (run `gsutil mb --help` for more information):\n", - " * `-b on` specifies uniform bucket-level access.\n", - " * `-l $REGION` puts the data in a specific region for lower cost and lower latency.\n", - " * `--pap enforced` turns on public access prevention to help keep data private. \n", - " \n", - "The uniform bucket level access (Bucket Policy Only enabled: true) puts the data access permissions (ACL) on the entire bucket, not on each object in the bucket. This makes the permissions obvious and makes security much more predictable.\n", - " \n", - "As usual, we must set our environment. In this case we also set a `REGION` environment variable to indicate where in the world we want the data to be stored.\n" + "We now will store the data in the bucket we created in the beginning of the episode. First we verify the environment variable and that it exists.\n" ] }, { "cell_type": "code", - "execution_count": 24, - "id": "f36cb8c5-f305-4cb2-a5cc-0c9fd8592fb4", + "execution_count": 27, + "id": "9345472f-4ef3-490b-a80e-2462cd534c89", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "bucket: essentials-learner-2021-12-17 region: us-west2\n" + "essentials-learner-2022-01-21\n" ] } ], "source": [ - "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", - "REGION=\"us-west2\"\n", - "echo \"bucket: $BUCKET region: $REGION\"" + "echo $BUCKET" ] }, { "cell_type": "code", - "execution_count": 25, - "id": "c2ae2b74-5e93-4c55-8bd7-63337f7dcbb8", + "execution_count": 28, + "id": "27dfae96-faf2-4d5d-8a78-97781841f172", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Creating gs://essentials-learner-2021-12-17/...\n" + "gs://essentials-learner-2022-01-21/\n" ] } ], "source": [ - "gsutil mb -b on -l $REGION --pap enforced \"gs://$BUCKET\"" + "gsutil ls -b gs://$BUCKET" ] }, { "cell_type": "markdown", - "id": "22cc1043-386b-4990-b3dc-62cbdd7ba133", - "metadata": {}, + "id": "939bcbaa-14fe-479f-a394-087f116ec7cc", + "metadata": { + "tags": [] + }, "source": [ "Now copy the output data to the bucket. The `-r` flag recursively copies the output directory and `-m` copies the files in parallel. Note the locations of the `-m` and `-r` switches as they apply globally and to the `cp` command respectively." ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 29, "id": "681e6b1d-98bb-448a-a57e-f5674214effd", "metadata": {}, "outputs": [ @@ -1332,8 +1436,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml [Content-Type=application/xml]...\n", - "Copying file://output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png [Content-Type=image/png]...\n", + "Copying file://output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml [Content-Type=application/xml]...\n", + "Copying file://output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png [Content-Type=image/png]...\n", "==> NOTE: You are uploading one or more large file(s), which would run \n", "significantly faster if you enable parallel composite uploads. This\n", "feature can be enabled by editing the\n", @@ -1346,8 +1450,8 @@ "without a compiled crcmod, computing checksums on composite objects is\n", "so slow that gsutil disables downloads of composite objects.\n", "\n", - "\\ [2/2 files][194.0 MiB/194.0 MiB] 100% Done \n", - "Operation completed over 2 objects/194.0 MiB. \n" + "\\ [2/2 files][191.6 MiB/191.6 MiB] 100% Done \n", + "Operation completed over 2 objects/191.6 MiB. \n" ] } ], @@ -1365,7 +1469,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 30, "id": "248c47be-625f-44f5-a6b6-919e8d8baafd", "metadata": {}, "outputs": [ @@ -1373,7 +1477,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2021-12-17/output/\n" + "gs://essentials-learner-2022-01-21/output/\n" ] } ], @@ -1383,7 +1487,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 31, "id": "b1ea18e9-5861-4479-9948-3303952dee8a", "metadata": {}, "outputs": [ @@ -1391,9 +1495,9 @@ "name": "stdout", "output_type": "stream", "text": [ - "194.05 MiB 2021-12-17T19:03:54Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", - " 910 B 2021-12-17T19:03:52Z gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", - "TOTAL: 2 objects, 203478494 bytes (194.05 MiB)\n" + "191.58 MiB 2022-01-21T21:34:56Z gs://essentials-learner-2022-01-21/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png\n", + " 910 B 2022-01-21T21:34:54Z gs://essentials-learner-2022-01-21/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml\n", + "TOTAL: 2 objects, 200890195 bytes (191.58 MiB)\n" ] } ], @@ -1413,61 +1517,19 @@ }, { "cell_type": "markdown", - "id": "a1c11268-f389-405f-8d96-3c319a49b882", + "id": "40d9ab41-8920-45f2-8218-550baac5b069", "metadata": {}, "source": [ - "## Sharing Results\n", - "\n", - "In order to share resources outside a project we must use the Identity Access Management service. This is a powerful tool to grant and restrict access to resources, and if not done correctly It can have serious consequences. **Incorrect permissions can lead to exposure of sensitive data, destruction of data, and authorized use of resources that can result in huge bills.** When in doubt, seek help.\n", + "## Cleanup\n", "\n", - "The question \"What access is really needed?\" is the **Principal of Least Privilege** and is a major cornerstone of security. We need to determine the lowest set of permissions or roles that is needed. In our case we wish to grant the \"Collaborator\" the \"Viewer\" access to the \"results bucket\". This will allow them to view, list, and download all objects in the bucket. This illustrates that for a **resource** a **member** (identity) is granted a **permission** (think of it in this order). Together this is called a **policy**. Google also uses \"Roles\" as a collection of predefined and managed permissions." - ] - }, - { - "cell_type": "markdown", - "id": "50990a67-f901-48fd-9295-25ef73245142", - "metadata": { - "tags": [] - }, - "source": [ - "We will now add Members to a Bucket using the Web Console. We will use the Web Console to interactively build the policy binding by doing the following:\n", - " * Navigation Menu -> Storage/Cloud Storage -> Browser -> Click on the Bucket Name (**Bucket Details**) -> Select the **Permissions** tab -> Click **Add** next to \"Permissions\" above the permissions list.\n", - " * In the \"New Principals\" box add the Identity for the collaborator (another individual) as directed by the instructor.\n", - " * Select the \"Storage Object Viewer\" by typing \"Storage Object Viewer\" in the filter and then selecting \"Storage Object Viewer\". Do not use any \"Legacy Storage\" roles.\n", - " * Click \"Save\" to save the policy.\n", - " ![iam-storage-object-viewer](img/iam-storage-object-viewer.png)\n", - " * Verify the policy is listed in the \"Permissions\" table on the \"Bucket Details\" page (you should now be on this page).\n", + "We will now leave the resources running in order to learn more about sharing and monitoring costs and will clean up all the resources as the end of Lesson. \n", "\n", - "*Advanced Note: Changes in permissions will show up on the **Home** -> **Activity** page.*" - ] - }, - { - "cell_type": "markdown", - "id": "6a191922-7d5b-4125-9812-181eb056f86c", - "metadata": { - "tags": [] - }, - "source": [ - "Collaborators should now be able to see the contents of the bucket by explicitly naming it as shown below.\n", - "```\n", - "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17\n", - "gs://essentials-learner-2021-12-17/output/\n", + "```{admonition} Danger\n", + ":class: danger\n", "\n", - "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17/output\n", - "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", - "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", + "**Do not forget to do remove the Cloud Storage Bucket and the Compute Engine Instance (Virtual Machine) when you are done with this Lesson!** Running resources and stored data costs money! Cleanup when you are done!\n", "```" ] - }, - { - "cell_type": "markdown", - "id": "40d9ab41-8920-45f2-8218-550baac5b069", - "metadata": {}, - "source": [ - "## Cleanup\n", - "\n", - "We will now leave the resources running in order to learn more about monitoring costs and will clean up all the resources as the end of Lesson. **Don't forget to do this!**" - ] } ], "metadata": { diff --git a/content/GCP/06b_sharing_results.ipynb b/content/GCP/06b_sharing_results.ipynb new file mode 100644 index 0000000..4537754 --- /dev/null +++ b/content/GCP/06b_sharing_results.ipynb @@ -0,0 +1,116 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "dc57021c", + "metadata": {}, + "source": [ + "# Sharing Data (Optional)\n", + "\n", + "```{admonition} Overview\n", + ":class: tip\n", + "\n", + "**Teaching:** 30 min\n", + "\n", + "**Exercises:** 5 min (optional)\n", + "\n", + "**Questions:**\n", + "* How do I share my results?\n", + "\n", + "**Objectives:**\n", + "* Understand the basics of Identity and Access Management (IAM)\n", + "* Add collaborators to a Bucket with appropriate permissions.\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "a1c11268-f389-405f-8d96-3c319a49b882", + "metadata": {}, + "source": [ + "## Sharing Results\n", + "\n", + "In order to share resources outside a project we must use the Identity Access Management service. This is a powerful tool to grant and restrict access to resources, and if not done correctly It can have serious consequences. **Incorrect permissions can lead to exposure of sensitive data, destruction of data, and authorized use of resources that can result in huge bills.** When in doubt, seek help.\n", + "\n", + "The question \"What access is really needed?\" is the **Principal of Least Privilege** and is a major cornerstone of security. We need to determine the lowest set of permissions or roles that is needed. In our case we wish to grant the \"Collaborator\" the \"Viewer\" access to the \"results bucket\". This will allow them to view, list, and download all objects in the bucket. This illustrates that for a **resource** a **member** (identity) is granted a **permission** (think of it in this order). Together this is called a **policy**. Google also uses \"Roles\" as a collection of predefined and managed permissions.\n", + "\n", + "What we do not want to do is add the collaborator to the project! This would give them access to all storage buckets and all resources." + ] + }, + { + "cell_type": "markdown", + "id": "50990a67-f901-48fd-9295-25ef73245142", + "metadata": { + "tags": [] + }, + "source": [ + "We will now add Members to a Bucket using the Web Console. We will use the Web Console to interactively build the policy binding by doing the following:\n", + " * Navigation Menu -> **Storage/Cloud Storage** -> Browser -> Click on the Bucket Name (**Bucket Details**) -> Select the **Permissions** tab -> Click **Add** next to \"Permissions\" above the permissions list.\n", + " * In the \"New Principals\" box add the Identity for the collaborator (another individual) as directed by the instructor.\n", + " * Select the \"Storage Object Viewer\" by typing \"Storage Object Viewer\" in the filter and then selecting \"Storage Object Viewer\". Do not use any \"Legacy Storage\" roles.\n", + " * Click \"Save\" to save the policy.\n", + " ![iam-storage-object-viewer](img/iam-storage-object-viewer.png)\n", + " * Verify the policy is listed in the \"Permissions\" table on the \"Bucket Details\" page (you should now be on this page).\n", + "\n", + "*Advanced Note: Changes in permissions will show up on the **Home** -> **Activity** page.*" + ] + }, + { + "cell_type": "markdown", + "id": "6a191922-7d5b-4125-9812-181eb056f86c", + "metadata": { + "tags": [] + }, + "source": [ + "Collaborators should now be able to see the contents of the bucket by explicitly naming the bucket. Below shows student321 accessing the bucket (note the prompt).\n", + "```bash\n", + "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17\n", + "gs://essentials-learner-2021-12-17/output/\n", + "\n", + "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17/output\n", + "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", + "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "e35d3a0b-aa8c-4c12-87d9-b218e2c16142", + "metadata": {}, + "source": [ + "Now remove the access by selecting the checkbox on the row with the principal identity and click `remove`. Now verify that the collaborator does not have access\n", + "\n", + "```bash\n", + "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-tmiddelkoop-$(date +%F)/output\n", + "AccessDeniedException: 403 student231@class.internet2.edu does not have storage.objects.list access to the Google Cloud Storage bucket.\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "4b93362c-c22b-453c-910b-db4a14ca58af", + "metadata": {}, + "source": [ + "```{admonition} Exercise\n", + "\n", + "*Instructors: You may want to have students share these buckets for the example to reduce screen flipping and involve the students.*\n", + "```" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Bash", + "language": "bash", + "name": "bash" + }, + "language_info": { + "codemirror_mode": "shell", + "file_extension": ".sh", + "mimetype": "text/x-sh", + "name": "bash" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/content/GCP/07_monitoring_costs.ipynb b/content/GCP/07_monitoring_costs.ipynb index d41ec1d..920bb05 100644 --- a/content/GCP/07_monitoring_costs.ipynb +++ b/content/GCP/07_monitoring_costs.ipynb @@ -10,12 +10,12 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 30 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** none\n", "\n", "**Questions:**\n", - "* How do I find out how much I have spent and what things cost?\n", + "* How do I find out how much I have spent?\n", "* How do I find what is running and stored in my project?\n", "\n", "**Objectives:**\n", @@ -24,9 +24,9 @@ "* Determine the permissions you have on the Billing Account\n", "* Find and monitor your spend with billing reports\n", "* Find where to set Billing Alerts.\n", - "* List all resources in a Project\n", - "* List resources in a service\n", - "* Optional: Understand how labels are used\n", + "* Find all resources in a Project\n", + "* Extra: List assets with the CLI\n", + "* Extra: Understand how labels are used\n", " \n", "```" ] @@ -45,11 +45,12 @@ " * Double check that the Project ID and name in the \"Project Info\" card is correct.\n", " * Note the billing card, but do not click on it.\n", " * **Navigation Menu -> Billing** \n", - " * If you get a \"You have multiple billing accounts\" select \"Go to linked billing account\"\n", + " * If you get a \"You have multiple billing accounts\" select \"**Go to linked billing account**\"\n", " * If you get a \"This project has no billing account\" you cannot incur any charges on this account. Since you do not have a billing account you have no charges to monitor or set alerts on. You will need to associate or create a billing account to do anything.\n", " * Select **Account Management** on the *Left Sidebar* of the Billing page.\n", " * Note down the \"Billing Account ID\" near the top left of the billing page It should be in the form *01ABC2-A0B34C-DE5F6*. This information should not be made public.\n", " * Open the right sidebar Info Panel if closed. To do so click **Show Info Panel** button on top right under the blue bar (don't click the \"Hide Info Panel\" button!)\n", + " ![billing-management](img/billing-management.png)\n", " * If you get \"you don't have permission to view the permissions of the selected resource\" in the side bar you have no privileges and cannot set Billing Alerts.\n", " * If you have permission to view permissions (you did not get the previous message):\n", " * Ensure **Show inherited permissions** is \"on\" (blue and to the right)\n", @@ -84,10 +85,12 @@ " * Bar chart (bar chart graph on the right, \"Daily\" is on the left)\n", " * **Save** view\n", " * If you cannot save the view you can also click \"Share\" and use the URL to create a bookmark for easy access.\n", - " \n", - "These charts should be monitored daily for active projects and weekly for ongoing projects. For projects that are inactive the billing should be disabled or the project should be deleted.\n", "\n", - "There reports do not show consumption in real time. In order to get an idea of what is generating costs (and you can estimate these costs based on past expereince), we will next show how to *enumerate* running resources. " + "![billing-report](img/billing-report.png)\n", + "\n", + "This is the most reliable and direct way to monitor costs and these charts should be monitored daily for active projects and weekly for ongoing projects. For projects that are inactive the billing should be disabled or the project should be deleted.\n", + "\n", + "There reports do not show consumption in real time. In order to get an idea of what is generating costs (and you can estimate these costs based on past experience), later we will show how to *enumerate* running resources. " ] }, { @@ -101,7 +104,26 @@ "\n", "Due to the diversity in the way Billing Accounts are setup at institutions we will not show you have to setup billing alerts. They are simple and straight forward to setup once you have access and navigate to it. If you do not have access to Billing Alerts you can accomplish similar results by navigating to the Billing Reports every day.\n", "\n", - "You can find the Billing Alerts under **Billing** -> **Budgets & Alerts**." + "You can find the Billing Alerts under **Billing** -> **Budgets & Alerts**.\n", + "\n", + "There are also 3rd party cost monitoring services that your institution may use that can set billing alerts, contact your research computing and data professional for more information." + ] + }, + { + "cell_type": "markdown", + "id": "1c382ee4-6fbc-4616-8154-4396fe724fa6", + "metadata": {}, + "source": [ + "## Asset Inventory\n", + "\n", + "In order to get an overview of what is running and stored in your account and where we use the *Asset Inventory* page in the *IAM* Service. To see what is in your *Project* do the following:\n", + "\n", + " * Go to **Navigation Menu** -> **IAM & Admin** -> **Asset Inventory** page.\n", + " * Filter important resources by navigating to **overview** -> **Asset type: View More** and selecting important resources (compute.Instance, compute.Disk, storage.Bucket) and click **Apply** on on the bottom.\n", + " ![iam-asset-navigation](img/iam-asset-navigation.png)\n", + "\n", + "Use the *Overview* and *Resource* to learn what is running and where (in the world) it is running.\n", + "\n" ] }, { @@ -109,7 +131,7 @@ "id": "cdfc0b3b-8bc3-44a3-a712-1005531c9d30", "metadata": {}, "source": [ - "## Enumerate All Resources\n", + "## Enumerate All Resources (CLI - Extra)\n", "\n", "Before we clean up we should always make sure that we can see the resources we created and that after destroying them they are actually removed. This is the most powerful tool for managing cost as it gives a real-time indication on what is running.\n", "\n", @@ -304,7 +326,7 @@ "id": "a670902f-53d3-4fce-acb0-0ace9608cd5d", "metadata": {}, "source": [ - "## Enumerate Service Resources\n", + "## Enumerate Service Resources (CLI - Extra)\n", "\n", "To get up-to-date information about a service the `gcloud` command can be used to list allocated resources. It is good to write a simple script to show all the resources in all the services you use. This, combined with the `gcloud asset` command is a powerful tool for " ] @@ -370,7 +392,7 @@ "id": "45681123-b27a-48d8-ac4a-30dc3b46d2c1", "metadata": {}, "source": [ - "## Optional: Labeling Resources\n", + "## Label Resources (CLI - Extra)\n", "\n", "In order to track resources it is important to Label them. Labeling places key-value pairs (key=value) in the resources metadata (information about the resource) and most of the Google Cloud tools can use this information to search, filter, and group information. In the literature, the example most used is the \"env\" Label and indicates if the resource is in development (dev), testing (test), or production (prod); the lables would be env=test, env=dev and env=prod, respectively. \n", "\n", diff --git a/content/GCP/08_cleaning_up_resources.ipynb b/content/GCP/08_cleaning_up_resources.ipynb index b5049bd..3880c63 100644 --- a/content/GCP/08_cleaning_up_resources.ipynb +++ b/content/GCP/08_cleaning_up_resources.ipynb @@ -10,9 +10,9 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 5 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** 10 min\n", "\n", "**Questions:**\n", "* How do I clean up after I'm done for the day?\n", @@ -27,317 +27,81 @@ }, { "cell_type": "markdown", - "id": "7621ba3b-c110-48aa-8300-14bbe5ee845f", + "id": "65fb0615-f942-4af0-9087-8f0696df95f9", "metadata": {}, "source": [ - "## Enumerating Existing Resources\n", + "## Cleanup VMs\n", "\n", - "We will first use the method from the previous lesson (see if you can do this on your own first) to check which resources are currently running so we can verify that they get removed." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "c8ee7307-4ed5-437f-b4e2-400b64190399", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//storage.googleapis.com/essentials-learner-2021-12-16\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --asset-types='compute.googleapis.com/Disk,compute.googleapis.com/Instance,storage.googleapis.com/Bucket' --format='value(name)'" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "67f75321-d684-4930-bd16-8112d4ced868", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n", - "//storage.googleapis.com/essentials-learner-2021-12-16\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort" - ] - }, - { - "cell_type": "markdown", - "id": "d43bd86d-acf7-416a-9b35-506f5625f2c3", - "metadata": {}, - "source": [ - "## Cleanup Buckets\n", + "```{admonition} Danger\n", + ":class: danger\n", "\n", - "First set the environment variables and verify them. If they are previously set you can just verify them. You can find the zone from the previous commands." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "03226ee4-00d3-4944-b60c-bd1ad62f1077", - "metadata": {}, - "outputs": [], - "source": [ - "ACCOUNT=$(gcloud config list --format='value(core.account)')\n", - "PROJECT=$(gcloud config list --format='value(core.project)')\n", - "BUCKET=\"essentials-${USER}-*\"\n", - "REGION=\"us-west2\"\n", - "ZONE=\"us-west2-c\"" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "75bf0dd4-4a62-497a-b491-2bc80285aa44", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Account: learner@class.internet2.edu, Project: essentials-project-335018, Region: us-west2 Zone: us-west2-c\n", - "Bucket: essentials-learner-*\n" - ] - } - ], - "source": [ - "echo \"Account: $ACCOUNT, Project: $PROJECT, Region: $REGION Zone: $ZONE\"\n", - "echo \"Bucket: $BUCKET\"" + "This will destroy the VM instance and and ALL the data in it! You cannot undelete an instance!\n", + "```\n", + "\n", + "```{admonition} Exercise\n", + "\n", + " * Try to delete the **VM Instance** on your own\n", + "```\n", + "\n", + "Navigate to the **Compute Engine** service and in the **VM Instances** page and remove the *VM Instance*.\n" ] }, { "cell_type": "markdown", - "id": "b4b93fcb-d72b-4b7c-ab40-5e5ad83fce86", + "id": "92321a60-13c6-4c87-aff5-92fb708e8bff", "metadata": {}, "source": [ "## Cleanup Buckets\n", "\n", - "We will now list the buckets that we will be deleting. Note the wild-card (GLOB) from above will match all dates (and all strings)." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "93add660-5bdb-4393-a05a-031ec301d7f8", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://essentials-learner-2021-12-16/\n" - ] - } - ], - "source": [ - "gsutil ls -b \"gs://$BUCKET\"" - ] - }, - { - "cell_type": "markdown", - "id": "8daa2ec9-d64d-4586-8ab9-33f03c1b1138", - "metadata": {}, - "source": [ - "**WARNING: All of the above buckets will be deleted if the following commands are run**" - ] - }, - { - "cell_type": "markdown", - "id": "be020ee5-d3af-4ada-a674-51775648f373", - "metadata": {}, - "source": [ - "**Delete** all the objects first" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "24616c20-ae7e-4d0f-9f9c-6c8752891fa7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Removing gs://essentials-learner-2021-12-16/1#1639684995415818...\n", - "/ [1/1 objects] 100% Done \n", - "Operation completed over 1 objects. \n" - ] - } - ], - "source": [ - "gsutil -m rm -r \"gs://$BUCKET/*\"" - ] - }, - { - "cell_type": "markdown", - "id": "194c0181-6a47-43c5-af40-29230d858164", - "metadata": {}, - "source": [ - "Delete the empty buckets" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "a1310b76-6212-48fa-b023-0a15e06d4332", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Removing gs://essentials-learner-2021-12-16/...\n" - ] - } - ], - "source": [ - "gsutil rb \"gs://$BUCKET/\"" - ] - }, - { - "cell_type": "markdown", - "id": "4bf0ed5b-b4f3-4061-ae65-50bf9d2500f7", - "metadata": {}, - "source": [ - "## Enumerating Resources\n", + "```{admonition} Danger\n", + ":class: danger\n", "\n", - "Now we will check that all the resources were removed." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "b088681d-633d-4991-ba7d-2ecfb41c36dd", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --asset-types='compute.googleapis.com/Disk,compute.googleapis.com/Instance,storage.googleapis.com/Bucket' --format='value(name)'" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "74d0e8e7-f097-41c5-88fe-929edcb0c1c1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort" - ] - }, - { - "cell_type": "markdown", - "id": "f275a53a-a3d1-4a0f-a44c-21e2ba7186ba", - "metadata": {}, - "source": [ - "Looks like we forgot to delete the VM." - ] - }, - { - "cell_type": "markdown", - "id": "74fd626a-6102-4815-a8c2-27aade6ebc3b", - "metadata": {}, - "source": [ - "## CLeanup VMs\n", - "We can also create and delete VMs using the `gcloud` command. Note, the `--quiet` disables conformation. **WARNING: This command will immediately delete the VM called essentials**" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "2ab8b339-b923-4623-942c-ddd424e3bcf7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Deleted [https://www.googleapis.com/compute/v1/projects/essentials-project-335018/zones/us-west2-c/instances/essentials].\n" - ] - } - ], - "source": [ - "gcloud compute instances delete essentials --zone=$ZONE --quiet" - ] - }, - { - "cell_type": "markdown", - "id": "4a215b73-ac5e-4fa6-88b7-8af26ee4090f", - "metadata": {}, - "source": [ - "Check again" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "e53cb55c-3195-4fc3-9aab-cabad44d865e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort" + "This will destroy the storage *Bucket* and and ALL the data in it! You cannot undelete a Bucket!\n", + "```\n", + "\n", + "```{admonition} Exercise\n", + "\n", + " * Try to delete the storage **bucket* on your own\n", + "```\n", + "\n", + "Navigate to the **Cloud Storage** service and in the **browser** page and remove the *essentials* Bucket." ] }, { "cell_type": "markdown", - "id": "631a08af-88bb-463c-b388-8993021ee216", + "id": "6753b415-1fa9-4716-8c41-8d98228af51c", "metadata": {}, "source": [ - "Success! The only resource left is the project (both are project resources)" + "## Verify Cleanup\n", + "\n", + "```{admonition} Exercise\n", + "\n", + " * There are at least three ways to verify resource removal, what are they?\n", + " * Try to verify that the resources were removed.\n", + "```\n", + "\n", + "```{admonition} Tip\n", + ":class: tip\n", + "You can also quickly navigate to resource page for most services from the **Resources** tab on the project **Dashboard**\n", + "```\n", + "\n", + "To verify that the resources have been deleted do the following:\n", + " * Navigate to the **VM Instances** page and **Cloud Storage** **Browser** to verify the resources have been deleted.\n", + " * Navigate to the **IAM & Admin** service and in the **Asset Inventory** page to verify that the resources have been deleted.\n", + " * Navigate to the **Activity** page and verify the deletion events. **Note:** the information collected in the asset inventory is often delayed.\n", + "\n", + "```{admonition} Tip\n", + ":class: tip\n", + "\n", + "If this is your own account, remember to visit the Billing Report page that you bookmarked daily.\n", + "```\n", + "\n", + "```{admonition} Tip\n", + ":class: tip\n", + "\n", + "You can also create, list, and destroy resources easily from the cli with the `gcloud` and `gsutil` commands.\n", + "```" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cac28557-fec2-43f4-aaa5-4f82b2257c55", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/content/GCP/img/api-enable-compute-engine.png b/content/GCP/img/api-enable-compute-engine.png new file mode 100644 index 0000000..f5d4d5a Binary files /dev/null and b/content/GCP/img/api-enable-compute-engine.png differ diff --git a/content/GCP/img/billing-management.png b/content/GCP/img/billing-management.png new file mode 100644 index 0000000..fb4f874 Binary files /dev/null and b/content/GCP/img/billing-management.png differ diff --git a/content/GCP/img/billing-report.png b/content/GCP/img/billing-report.png new file mode 100644 index 0000000..b4b9202 Binary files /dev/null and b/content/GCP/img/billing-report.png differ diff --git a/content/GCP/img/compute-create-instance-ok.png b/content/GCP/img/compute-create-instance-ok.png new file mode 100644 index 0000000..3d1614d Binary files /dev/null and b/content/GCP/img/compute-create-instance-ok.png differ diff --git a/content/GCP/img/compute-create-instance.png b/content/GCP/img/compute-create-instance.png new file mode 100644 index 0000000..1de2241 Binary files /dev/null and b/content/GCP/img/compute-create-instance.png differ diff --git a/content/GCP/img/compute-delete-instance.png b/content/GCP/img/compute-delete-instance.png new file mode 100644 index 0000000..e67a38a Binary files /dev/null and b/content/GCP/img/compute-delete-instance.png differ diff --git a/content/GCP/img/iam-asset-navigation.png b/content/GCP/img/iam-asset-navigation.png new file mode 100644 index 0000000..c323d30 Binary files /dev/null and b/content/GCP/img/iam-asset-navigation.png differ diff --git a/content/GCP/img/iam-default-service-account.png b/content/GCP/img/iam-default-service-account.png new file mode 100644 index 0000000..f291fb0 Binary files /dev/null and b/content/GCP/img/iam-default-service-account.png differ diff --git a/content/GCP/img/storage-create-bucket.png b/content/GCP/img/storage-create-bucket.png new file mode 100644 index 0000000..2c0f70f Binary files /dev/null and b/content/GCP/img/storage-create-bucket.png differ diff --git a/content/GCP/img/storage-delete-bucket.png b/content/GCP/img/storage-delete-bucket.png new file mode 100644 index 0000000..26004fa Binary files /dev/null and b/content/GCP/img/storage-delete-bucket.png differ diff --git a/content/GCP/intro_to_GCP_Essentials.ipynb b/content/GCP/intro_to_GCP_Essentials.ipynb index 2bd2d17..6bff3c8 100644 --- a/content/GCP/intro_to_GCP_Essentials.ipynb +++ b/content/GCP/intro_to_GCP_Essentials.ipynb @@ -15,9 +15,16 @@ "4. [Introduction to the Cloud CLI](./04_intro_to_cli)\n", "4. [Using the Cloud Storage CLI](./05_cli_storage)\n", "6. [Running Analysis on the Cloud](./06_running_analysis)\n", + " and [Sharing Results](./06b_sharing_results)\n", "7. [Monitoring Costs](./07_monitoring_costs)\n", "8. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)\n" ] + }, + { + "cell_type": "markdown", + "id": "78058910", + "metadata": {}, + "source": [] } ], "metadata": { diff --git a/content/_toc.yml b/content/_toc.yml index cc8007e..497354d 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -41,6 +41,7 @@ parts: - file: GCP/04_intro_to_cli - file: GCP/05_cli_storage - file: GCP/06_running_analysis + - file: GCP/06b_sharing_results - file: GCP/07_monitoring_costs - file: GCP/08_cleaning_up_resources - file: GCP/glossary