From acbc4834116c26e60778944d36613668e26f506b Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 16 Dec 2021 21:25:07 +0000 Subject: [PATCH] Pull missing questions, objectives and material --- content/GCP/01_intro_to_cloud_console.ipynb | 45 +++++- content/GCP/02_intro_to_compute.ipynb | 30 ++-- content/GCP/03_intro_to_cloud_storage.ipynb | 30 ++-- content/GCP/04_intro_to_cli.ipynb | 48 +++--- content/GCP/05_cli_storage.ipynb | 20 ++- content/GCP/06_running_analysis.ipynb | 4 +- content/GCP/07_monitoring_costs.ipynb | 155 ++++++++++++++++++-- content/GCP/08_cleaning_up_resources.ipynb | 8 +- 8 files changed, 262 insertions(+), 78 deletions(-) diff --git a/content/GCP/01_intro_to_cloud_console.ipynb b/content/GCP/01_intro_to_cloud_console.ipynb index ff4e087..4659fa6 100644 --- a/content/GCP/01_intro_to_cloud_console.ipynb +++ b/content/GCP/01_intro_to_cloud_console.ipynb @@ -9,12 +9,22 @@ "source": [ "# Introduction to the GCP Cloud Console\n", "\n", - "Learner Questions\n", - " * How do I find things in the web console?\n", + "```{admonition} Overview\n", + ":class: tip\n", "\n", - "Learning Objectives\n", - " * Login to the Google Cloud web console\n", - " * Find and verify important information about your cloud environment.\n" + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + " * How do I find things in the web console?\n", + " * How do I access the cloud through the command line?\n", + "\n", + "**Objectives:**\n", + " * Login to the Google Cloud web console\n", + " * Find and verify important information about your cloud environment.\n", + " * Find and open the Cloud Shell\n", + "```" ] }, { @@ -151,6 +161,31 @@ " 9. Google Account (avatar at the far right) - verify and switch accounts." ] }, + { + "cell_type": "markdown", + "id": "50fddb1f-c826-412c-bb3c-127fee6b4949", + "metadata": { + "tags": [] + }, + "source": [ + "## Open a Cloud Shell\n", + "\n", + "The Cloud Shell is an easy way to access Comand Line Interface (CLI) commands in a web browser. Under the hood Cloud Shell is Debian running in a container with 5GB of no-charge persistent storage with most of the Google uti and a lot of other useful tools (git, docker, jq, Docker, etc.) installed by default.\n", + "\n", + "To use the Cloud Shell:\n", + " * Open up a Cloud Shell by clicking on the **Activate Cloud Shell** icon in the right side of the top blue bar. If a \"Authorize Cloud Shell\" dialog appears click \"Authorize\" to allow the Cloud Shell to access the project.\n", + " * Click the **Maximize** icon on the right side of the Cloud Shell bar. You can find the correct icon by hovering over the icons to see their names.\n", + " * If a shell is associated with a project it will show it in the tab and in **yellow text** in round parenthesis in the shell prompt. **Verify the Project ID**.\n", + " * To open up a shell with a specific project click the \"down arrow\" icon next to the plus(+) to open a new tab with a specific project. Selecting the plus (+) will usually open up a shell with the default project.\n", + " * Note: You can open the Cloud Shell in a new tab by clicking on icon **Open in a New Window** on the far right of the Cloud Shell bar.\n", + "\n", + "\n", + "### Opening Cloud Shell in a Tab - Optional\n", + "\n", + " * Click the **Open in a New Window** icon on the far left of the \"Cloud Shell\" tab on the bottom to open up in a separate tab. Hover the mouse over the icons to see their name/function. **Note:** you may need to allow popups or enable popups for this website depending on the browser you use. You can also pull up the tab to increase the shell screen size.\n", + " * Click the **\"Close editor\"** button (the pencil icon in the top right of the window)." + ] + }, { "cell_type": "markdown", "id": "118254c6-a5d9-4bd0-8a0f-64d4f72b3cf0", diff --git a/content/GCP/02_intro_to_compute.ipynb b/content/GCP/02_intro_to_compute.ipynb index 4216c03..04310b7 100644 --- a/content/GCP/02_intro_to_compute.ipynb +++ b/content/GCP/02_intro_to_compute.ipynb @@ -9,16 +9,24 @@ "source": [ "# Introduction to Cloud Compute (Google Compute Engine)\n", "\n", - "Learner Questions\n", - " * How do I compute in the cloud?\n", - "\n", - "Learning Objectives:\n", - " * Navigate the Google Compute Engine service and terminology.\n", - " * Allocate a virtual machine in Google Compute Engine.\n", - " * Choose a machine type based on the application.\n", - " * Understand how storage is managed for virtual machines.\n", - " * Use the cost estimator for Google Compute Engine.\n", - " * Understand what a service account is and how they are used.\n" + "```{admonition} Overview\n", + ":class: tip\n", + "\n", + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + " * How do I create my own computer in the cloud?\n", + "\n", + "**Objectives:**\n", + " * Navigate the Google Compute Engine service and terminology.\n", + " * Allocate a virtual machine in Google Compute Engine.\n", + " * Choose a machine type based on the application.\n", + " * Use the cost estimator for Google Compute Engine.\n", + " * Understand what a service account is and how they are used.\n", + " * Learn how to connect to a virtual machine using ssh in the cloud shell with the web console.\n", + "```" ] }, { @@ -139,8 +147,6 @@ "\n", "To connect to the *VM Instance* we will use the cloud shell in the web console. Open up a shell by clicking on the **Activate Cloud Shell** icon in the top blue bar if it is not already open.\n", "\n", - "If you have not used Cloud Shell you may need read through some dialog boxes to create a new Cloud Shell and \"Allow\" the Cloud Shell to access your account.\n", - "\n", "We first verify that the cloud shell is using the correct identity and *project id* by looking at the welcome message and the prompt. If your shell does not show a project you will have to use the little \"Down Arrow\" next to the \"+\" to create a new tab to select the correct project.\n", "\n", "![cloud-shell](img/cloud-shell.png)\n", diff --git a/content/GCP/03_intro_to_cloud_storage.ipynb b/content/GCP/03_intro_to_cloud_storage.ipynb index f53c14f..ada3730 100644 --- a/content/GCP/03_intro_to_cloud_storage.ipynb +++ b/content/GCP/03_intro_to_cloud_storage.ipynb @@ -7,17 +7,25 @@ "source": [ "# Introduction to Google Cloud Storage\n", "\n", - "Learner Questions\n", - " * How do I store data in the cloud?\n", - "\n", - "Learning Objectives:\n", - " * Navigate the Google Cloud Storage service and terminology\n", - " * Understand the roles and permissions needed to use Google Cloud Storage in projects\n", - " * Allocate storage in Google Cloud Storage\n", - " * Find the cost estimator for Google Cloud Storage\n", - " * Recognize that resources have a \"location\"\n", - " * Recognize danger of public access\n", - " * De-allocate Google Cloud Storage storage" + "```{admonition} Overview\n", + ":class: tip\n", + "\n", + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + " * How do I store data in the cloud?\n", + "\n", + "**Objectives:**\n", + " * Navigate the Google Cloud Storage service and terminology\n", + " * Understand the roles and permissions needed to use Google Cloud Storage in projects\n", + " * Allocate storage in Google Cloud Storage\n", + " * Find the cost estimator for Google Cloud Storage\n", + " * Recognize that resources have a \"location\"\n", + " * Recognize danger of public access\n", + " * De-allocate Google Cloud Storage storage\n", + "```" ] }, { diff --git a/content/GCP/04_intro_to_cli.ipynb b/content/GCP/04_intro_to_cli.ipynb index 9f8e713..0eaedc6 100644 --- a/content/GCP/04_intro_to_cli.ipynb +++ b/content/GCP/04_intro_to_cli.ipynb @@ -7,17 +7,26 @@ "source": [ "# Introduction to the gcloud CLI\n", "\n", - "Learner Questions\n", - " * How do I use the Cloud CLI?\n", - " * How do I use `gcloud`?\n", "\n", - "Learning Objectives\n", - " * Find and use the \"Cloud Shell\" in the web console.\n", - " * Use basic cloud CLI commands (`gcloud` and `gsutil`).\n", - " * Verify basic settings.\n", - " * Use environment variables for configuration.\n", - " * Understand the importance of using variables for configuration.\n", - " * Recognize the value of reproducibility and automation.\n", + "```{admonition} Overview\n", + ":class: tip\n", + "\n", + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + " * How do I use the Cloud CLI?\n", + " * How do I use `gcloud`?\n", + "\n", + "**Objectives:**\n", + " * Find and use the \"Cloud Shell\" in the web console.\n", + " * Use basic cloud CLI commands (`gcloud` and `gsutil`).\n", + " * Verify basic settings.\n", + " * Use environment variables for configuration.\n", + " * Understand the importance of using variables for configuration.\n", + " * Recognize the value of reproducibility and automation.\n", + "```\n", " \n", "Resources\n", " * `gcloud` documentation: https://cloud.google.com/sdk/gcloud\n", @@ -45,26 +54,9 @@ }, { "cell_type": "markdown", - "id": "ce598878-1b1c-4e6b-9122-e679e84c7dbb", + "id": "8eb44568-1108-4963-aebf-7f6b98ddaffc", "metadata": {}, "source": [ - "## Open a Cloud Shell\n", - "\n", - "The Cloud Shell is an easy way to access Comand Line Interface (CLI) commands in a web browser. Under the hood Cloud Shell is Debian running in a container with 5GB of no-charge persistent storage with most of the Google uti and a lot of other useful tools (git, docker, jq, Docker, etc.) installed by default.\n", - "\n", - "To use the Cloud Shell:\n", - " * Open up a Cloud Shell by clicking on the **Activate Cloud Shell** icon in the right side of the top blue bar. If a \"Authorize Cloud Shell\" dialog appears click \"Authorize\" to allow the Cloud Shell to access the project.\n", - " * Click the **Maximize** icon on the right side of the Cloud Shell bar. You can find the correct icon by hovering over the icons to see their names.\n", - " * If a shell is associated with a project it will show it in the tab and in **yellow text** in round parenthesis in the shell prompt. **Verify the Project ID**.\n", - " * To open up a shell with a specific project click the \"down arrow\" icon next to the plus(+) to open a new tab with a specific project. Selecting the plus (+) will usually open up a shell with the default project.\n", - " * Note: You can open the Cloud Shell in a new tab by clicking on icon **Open in a New Window** on the far right of the Cloud Shell bar.\n", - "\n", - "\n", - "### Opening Cloud Shell in a Tab - Optional\n", - "\n", - " * Click the **Open in a New Window** icon on the far left of the \"Cloud Shell\" tab on the bottom to open up in a separate tab. Hover the mouse over the icons to see their name/function. **Note:** you may need to allow popups or enable popups for this website depending on the browser you use. You can also pull up the tab to increase the shell screen size.\n", - " * Click the **\"Close editor\"** button (the pencil icon in the top right of the window).\n", - "\n", "### Advanced call-out\n", "You can access the cloud shell from your laptop via `gcloud cloud-shell ssh` if you have the [Google Cloud SDK installed](https://cloud.google.com/sdk/docs/install). Installing the Google Cloud SDK will also allow you to run `gcloud` and `gsutil` commands directly from your laptop, workstation, or cluster. *Pro-tip, you cannot use a service account to access the cloud shell.*" ] diff --git a/content/GCP/05_cli_storage.ipynb b/content/GCP/05_cli_storage.ipynb index 1a70c23..dd02551 100644 --- a/content/GCP/05_cli_storage.ipynb +++ b/content/GCP/05_cli_storage.ipynb @@ -7,12 +7,22 @@ "source": [ "# Managing Cloud Storage from the Command Line\n", "\n", - "Learner Questions\n", - " * How do I store data in a Bucket?\n", "\n", - "Learning Objectives\n", - " * Manage buckets using the command line.\n", - " * Create a bucket with a sensible configuration. \n" + "```{admonition} Overview\n", + ":class: tip\n", + "\n", + "**Teaching:**\n", + "\n", + "**Exercises:**\n", + "\n", + "**Questions:**\n", + " * How do I store data in a Bucket?\n", + "\n", + "**Objectives:**\n", + " * Manage buckets using the command line.\n", + " * Create a bucket with a sensible configuration.\n", + " * Learn to manage data (add, list, retrieve, delete) in a bucket using the command line.\n", + "```\n" ] }, { diff --git a/content/GCP/06_running_analysis.ipynb b/content/GCP/06_running_analysis.ipynb index 8e297cf..d5e30a5 100644 --- a/content/GCP/06_running_analysis.ipynb +++ b/content/GCP/06_running_analysis.ipynb @@ -15,10 +15,10 @@ "**Exercises:**\n", "\n", "**Questions:**\n", - " * \n", + " * Can you show me an example?\n", "\n", "**Objectives:**\n", - " * \n", + " * Create a simple workflow using a cloud VM and cloud object storage.\n", "\n", "```" ] diff --git a/content/GCP/07_monitoring_costs.ipynb b/content/GCP/07_monitoring_costs.ipynb index bf10918..d9c6ac1 100644 --- a/content/GCP/07_monitoring_costs.ipynb +++ b/content/GCP/07_monitoring_costs.ipynb @@ -15,13 +15,19 @@ "**Exercises:**\n", "\n", "**Questions:**\n", - " * How do I find out how much I am spending?\n", + " * How do I find out how much I have spent and what things cost?\n", + " * How do I find what is running and stored in my project?\n", "\n", "**Objectives:**\n", - " * Learn about Billing Accounts\n", - " * Find information about the Billing Account associated with your project\n", - " * Determine the permissions you have on the Billing Account\n", - " * Find and monitor your spend with billing reports\n", + " * Learn about Billing Accounts\n", + " * Find information about the Billing Account associated with your project\n", + " * Determine the permissions you have on the Billing Account\n", + " * Find and monitor your spend with billing reports\n", + " * Find where to set Billing Alerts.\n", + " * List all resources in a Project\n", + " * List resources in a service\n", + " * Understand how labels are used\n", + " \n", "```" ] }, @@ -56,7 +62,7 @@ }, { "cell_type": "markdown", - "id": "b6ac1492-727f-4bda-8ecb-78b8ca13f222", + "id": "1d149251-ccd3-482b-9ea7-05b79adaaf93", "metadata": { "tags": [] }, @@ -84,12 +90,26 @@ "There reports do not show consumption in real time. In order to get an idea of what is generating costs (and you can estimate these costs based on past expereince), we will next show how to *enumerate* running resources. " ] }, + { + "cell_type": "markdown", + "id": "87e1a77d-d6b2-4618-8451-e6c0a1fd4293", + "metadata": { + "tags": [] + }, + "source": [ + "## Billing Alerts\n", + "\n", + "Due to the diversity in the way Billing Accounts are setup at institutions we will not show you have to setup billing alerts. They are simple and straight forward to setup once you have access and navigate to it. If you do not have access to Billing Alerts you can accomplish similar results by navigating to the Billing Reports every day.\n", + "\n", + "You can find the Billing Alerts under **Billing** -> **Budgets & Alerts**." + ] + }, { "cell_type": "markdown", "id": "cdfc0b3b-8bc3-44a3-a712-1005531c9d30", "metadata": {}, "source": [ - "## Enumerate Allocated Resources\n", + "## Enumerate All Resources\n", "\n", "Before we clean up we should always make sure that we can see the resources we created and that after destroying them they are actually removed. This is the most powerful tool for managing cost as it gives a real-time indication on what is running.\n", "\n", @@ -284,7 +304,9 @@ "id": "a670902f-53d3-4fce-acb0-0ace9608cd5d", "metadata": {}, "source": [ - "To get up-to-date information about a service the `gcloud` command can be used to list allocated resources." + "## Enumerate Service Resources\n", + "\n", + "To get up-to-date information about a service the `gcloud` command can be used to list allocated resources. It is good to write a simple script to show all the resources in all the services you use. This, combined with the `gcloud asset` command is a powerful tool for " ] }, { @@ -343,13 +365,122 @@ "gcloud alpha storage ls" ] }, + { + "cell_type": "markdown", + "id": "45681123-b27a-48d8-ac4a-30dc3b46d2c1", + "metadata": {}, + "source": [ + "## Optional: Labeling Resources\n", + "\n", + "In order to track resources it is important to Label them. Labeling places key-value pairs (key=value) in the resources metadata (information about the resource) and most of the Google Cloud tools can use this information to search, filter, and group information. In the literature, the example most used is the \"env\" Label and indicates if the resource is in development (dev), testing (test), or production (prod); the lables would be env=test, env=dev and env=prod, respectively. \n", + "\n", + "For research and teaching, Labels could be used to track funding, grants, students, or other uses of resources. For large efforts (projects) it is better to use different Projects to separate access and billing and protect the projects from each other.\n", + "\n", + "*Pro-tip: Labeling is called Tagging on AWS and Azure - tagging in GCP is related to firewalls).*\n" + ] + }, + { + "cell_type": "markdown", + "id": "ed821808-0f4c-4c83-854f-4e3867224e07", + "metadata": {}, + "source": [ + "In this Lesson we use the `cce` Label to identify resources associated with this Lesson.\n", + "First, we label our bucket:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "ea8360f5-5e78-4601-bfa0-4731c5c9a962", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Setting label configuration on gs://cce-data-student31-2021-08-25/...\n" + ] + } + ], + "source": [ + "gsutil label ch -l cce:data \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "1d6a46f9-9494-4766-b488-6ca9bb9f0061", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://cce-data-student31-2021-08-25/ :\n", + "\tStorage class:\t\t\tSTANDARD\n", + "\tLocation type:\t\t\tregion\n", + "\tLocation constraint:\t\tUS-CENTRAL1\n", + "\tVersioning enabled:\t\tNone\n", + "\tLogging configuration:\t\tNone\n", + "\tWebsite configuration:\t\tNone\n", + "\tCORS configuration: \t\tNone\n", + "\tLifecycle configuration:\tNone\n", + "\tRequester Pays enabled:\t\tNone\n", + "\tLabels:\t\t\t\t\n", + "\t {\n", + "\t \"cce\": \"data\"\n", + "\t }\n", + "\tDefault KMS key:\t\tNone\n", + "\tTime created:\t\t\tWed, 25 Aug 2021 18:37:25 GMT\n", + "\tTime updated:\t\t\tWed, 25 Aug 2021 20:49:49 GMT\n", + "\tMetageneration:\t\t\t10\n", + "\tBucket Policy Only enabled:\tTrue\n", + "\tPublic access prevention:\tenforced\n", + "\tACL:\t\t\t\t[]\n", + "\tDefault ACL:\t\t\t[]\n" + ] + } + ], + "source": [ + "gsutil ls -L -b \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "70a91e91-6bd3-4056-9037-b5a4bed29079", + "metadata": {}, + "source": [ + "Now we can use it to filter our report" + ] + }, { "cell_type": "code", - "execution_count": null, - "id": "d815546b-5a0a-4280-af29-aa032a790f1d", + "execution_count": 7, + "id": "26549482-d508-41ac-8c80-92e4ec8bc3d1", "metadata": {}, - "outputs": [], - "source": [] + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "---\n", + "assetType: storage.googleapis.com/Bucket\n", + "createTime: '2021-08-25T18:37:25Z'\n", + "displayName: cce-data-student31-2021-08-25\n", + "labels:\n", + " cce: data\n", + "location: us-central1\n", + "name: //storage.googleapis.com/cce-data-student31-2021-08-25\n", + "parentAssetType: cloudresourcemanager.googleapis.com/Project\n", + "parentFullResourceName: //cloudresourcemanager.googleapis.com/projects/just-armor-301114\n", + "project: projects/1002111293252\n", + "updateTime: '2021-08-25T20:49:49Z'\n" + ] + } + ], + "source": [ + "gcloud asset search-all-resources --query=labels.cce:data" + ] } ], "metadata": { diff --git a/content/GCP/08_cleaning_up_resources.ipynb b/content/GCP/08_cleaning_up_resources.ipynb index de00ca1..dc69866 100644 --- a/content/GCP/08_cleaning_up_resources.ipynb +++ b/content/GCP/08_cleaning_up_resources.ipynb @@ -15,10 +15,12 @@ "**Exercises:**\n", "\n", "**Questions:**\n", - " * \n", + " * How do I clean up after I'm done for the day?\n", + " * How do I make sure everything is gone?\n", "\n", "**Objectives:**\n", - " * This Episode is used to cleanup resources created by the Lesson.\n", + " * Cleanup resources created by the Lesson.\n", + " * Verify that all the resources have been removed.\n", "\n", "```" ] @@ -82,7 +84,7 @@ "source": [ "## Cleanup Buckets\n", "\n", - "First set the environment variables and verify them. If they are previously set you can just verify them." + "First set the environment variables and verify them. If they are previously set you can just verify them. You can find the zone from the previous commands." ] }, {