From b4dab85ac9a956cd3126d122ce082f9729dd3426 Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 3 Feb 2022 18:38:39 +0000 Subject: [PATCH 1/6] Rename sharing result to 06. --- ...06b_sharing_results.ipynb => 06_sharing_results.ipynb} | 0 content/GCP/intro_to_GCP_Essentials.ipynb | 8 +------- content/_toc.yml | 2 +- 3 files changed, 2 insertions(+), 8 deletions(-) rename content/GCP/{06b_sharing_results.ipynb => 06_sharing_results.ipynb} (100%) diff --git a/content/GCP/06b_sharing_results.ipynb b/content/GCP/06_sharing_results.ipynb similarity index 100% rename from content/GCP/06b_sharing_results.ipynb rename to content/GCP/06_sharing_results.ipynb diff --git a/content/GCP/intro_to_GCP_Essentials.ipynb b/content/GCP/intro_to_GCP_Essentials.ipynb index 6bff3c8..e486192 100644 --- a/content/GCP/intro_to_GCP_Essentials.ipynb +++ b/content/GCP/intro_to_GCP_Essentials.ipynb @@ -15,16 +15,10 @@ "4. [Introduction to the Cloud CLI](./04_intro_to_cli)\n", "4. [Using the Cloud Storage CLI](./05_cli_storage)\n", "6. [Running Analysis on the Cloud](./06_running_analysis)\n", - " and [Sharing Results](./06b_sharing_results)\n", + " and [Sharing Results](./06_sharing_results) (optional)\n", "7. [Monitoring Costs](./07_monitoring_costs)\n", "8. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)\n" ] - }, - { - "cell_type": "markdown", - "id": "78058910", - "metadata": {}, - "source": [] } ], "metadata": { diff --git a/content/_toc.yml b/content/_toc.yml index 3a2f2a8..fe9bcd4 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -41,7 +41,7 @@ parts: - file: GCP/04_intro_to_cli - file: GCP/05_cli_storage - file: GCP/06_running_analysis - - file: GCP/06b_sharing_results + - file: GCP/06_sharing_results - file: GCP/07_monitoring_costs - file: GCP/08_cleaning_up_resources - file: GCP/glossary From df67f49a436fa2d6b7c001a0e63ab8d7563af1c0 Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 3 Feb 2022 19:36:05 +0000 Subject: [PATCH 2/6] GCP: Glossary additions --- content/GCP/05_cli_storage.ipynb | 5 ++++- content/GCP/glossary.ipynb | 21 +++++++++++++++------ 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/content/GCP/05_cli_storage.ipynb b/content/GCP/05_cli_storage.ipynb index 4076e4f..00d761a 100644 --- a/content/GCP/05_cli_storage.ipynb +++ b/content/GCP/05_cli_storage.ipynb @@ -211,7 +211,10 @@ "cell_type": "code", "execution_count": 7, "id": "ecc0e4fe-c9d0-4edd-9a0c-56b5b9ec7c54", - "metadata": {}, + "metadata": { + "scrolled": true, + "tags": [] + }, "outputs": [ { "name": "stdout", diff --git a/content/GCP/glossary.ipynb b/content/GCP/glossary.ipynb index e690bf9..b0ef7a6 100644 --- a/content/GCP/glossary.ipynb +++ b/content/GCP/glossary.ipynb @@ -7,12 +7,21 @@ "source": [ "# Glossary\n", "\n", - "* **Account** - The Google account (email) associated with an individual. This is the identity that you use to access console and takes the form of an email address.\n", - "* **Project** - Resources and services in Google Cloud are created within a single Project. \n", - "* **Project Name** - The long human readable name of your project. You can change this.\n", - "* **Project ID** - an alpha-numeric string identifier for the project. This is not changeable.\n", - "* **Project Number** - a long number that also identifies the project but is rarely used (it is an internal identifier). \n", - "* **Billing Account** - an account used to pay for Google Cloud services. A Billing Account is associated with a Project and enables resource allocation. A project has only one Billing Account associated with it. A Billing account can be used for multiple Projects." + " * **Account** - The Google account (email) associated with an individual. This is the identity that you use to access console and takes the form of an email address.\n", + " * **API** - Application Programming Interface (API). An external interface for a library or software, usually well documented. The cloud uses remote accessible APIs to allocate and control resources.\n", + " * **Billing Account** - an account used to pay for Google Cloud services. A Billing Account is associated with a Project and enables resource allocation. A project has only one Billing Account associated with it. A Billing account can be used for multiple Projects.\n", + " * **Bucket** - A collection of key value pairs (similar to a Python dictionary) and is often called object storage. Object storage in Google is called Cloud Storage and is similar to Amazon S3.\n", + " * **Cloud Shell** - A Linux command line interface (CLI) running inside the GCP web console. \n", + " * **Disk Image** - The operating system disk for a **VM Instance** such as Ubuntu, Debian, and other Linux distributions.\n", + " * **Project** - Resources and services in Google Cloud are created within a single Project. \n", + " * **Project Name** - The long human readable name of your project. You can change this.\n", + " * **Project ID** - an alpha-numeric string identifier for the project. This is not changeable.\n", + " * **Project Number** - a long number that also identifies the project but is rarely used (it is an internal identifier). \n", + " * **Region** - A geographical region within the world (typically a city) in which a number of physical or logical data centers are located (each is a different **zone**). An example of a **region** is `us-west2`, which is located in Los Angeles.\n", + " * **Service Account** - A machine **account** or identity that is used to give permissions to a resource, such as a **VM Instance** in GCP. These take the form of an email, usually ending with `gserviceaccount.com`.\n", + " * **Label** - A key value pair attached to a resource to aid in finding resources, allocating billing, debugging, etc.. A Label is similar to a *tag* in AWS and Azure. \n", + " * **VM Instance** - A Virtual Machine (VM) is a virtual computer running in a **zone**.\n", + " * **Zone** - A logical or physical data center (a building with computers in it) in which cloud resources physically reside. Each **zone** in a **region** have different resources. An example of a **zone** is `us-west2-c`." ] } ], From 0d114c5bbc70954d7a12991baf1115064b69ab2b Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 3 Feb 2022 20:19:52 +0000 Subject: [PATCH 3/6] GCP: Simpler use of environment variables. --- content/GCP/02_intro_to_compute.ipynb | 2 +- content/GCP/04_intro_to_cli.ipynb | 149 +++++++++++++------------- 2 files changed, 73 insertions(+), 78 deletions(-) diff --git a/content/GCP/02_intro_to_compute.ipynb b/content/GCP/02_intro_to_compute.ipynb index 7097fbc..fda48cd 100644 --- a/content/GCP/02_intro_to_compute.ipynb +++ b/content/GCP/02_intro_to_compute.ipynb @@ -309,7 +309,7 @@ "Did you \"Follow\" the VM instance by looking at the **activity** page as discussed above?\n", "\n", "Since we care about paying for resources we are not using we review our project by visiting the *compute engine* service and reviewing that we no longer have any *VM instances* running. \n", - " * Go to **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances** and look for your virtual machine.\n", + " * Go to **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances** and look for your virtual machine. It should be gone!\n", "\n", "In a later episode we will show how to easily list all the resources in a project." ] diff --git a/content/GCP/04_intro_to_cli.ipynb b/content/GCP/04_intro_to_cli.ipynb index 9136726..6ffab2b 100644 --- a/content/GCP/04_intro_to_cli.ipynb +++ b/content/GCP/04_intro_to_cli.ipynb @@ -62,8 +62,11 @@ "id": "8eb44568-1108-4963-aebf-7f6b98ddaffc", "metadata": {}, "source": [ - "### Advanced call-out\n", - "You can access the cloud shell from your laptop via `gcloud cloud-shell ssh` if you have the [Google Cloud SDK installed](https://cloud.google.com/sdk/docs/install). Installing the Google Cloud SDK will also allow you to run `gcloud` and `gsutil` commands directly from your laptop, workstation, or cluster. *Pro-tip, you cannot use a service account to access the cloud shell.*" + "```{admonition} Tip\n", + ":class: tip\n", + "\n", + "You can access the cloud shell from your laptop via `gcloud cloud-shell ssh` if you have the [Google Cloud SDK installed](https://cloud.google.com/sdk/docs/install). Installing the Google Cloud SDK will also allow you to run `gcloud` and `gsutil` commands directly from your laptop, workstation, or cluster. *Pro-tip, you cannot use a service account to access the cloud shell.*\n", + "```" ] }, { @@ -236,12 +239,18 @@ "\n", "*To make the point even clearer, we use environment variables in the Cloud Essentials notebooks instead of hard coding them, which makes it easy for anyone to run, use, and edit and thus collaborate and contribute to them!*\n", "\n", - "We now demonstrate the basic use environment variables. As a reminder, environment variables are used to store configuration information that can be easily passed between programs and programming languages." + "We now demonstrate the basic use environment variables. As a reminder, environment variables are used to store configuration information that can be easily passed between programs and programming languages.\n", + "\n", + "```{admonition} Advanced Tip\n", + ":class: tip\n", + "\n", + "Although we do not always need to, we should always enclose shell variables in double quotes and enclose the variable name in curly braces (funny things may happen otherwise) when writing scripts, for example `\"${TEST}\"`. In most cases it is easier to do this every time than try to figure out when it is possible to use the short form. We only use the short form (`$TEST`) when it is used frequently and known to always work (most cases here), in our case it is `$PROJECT` and `$BUCKET`. When in doubt, use the explicit form.\n", + "```" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 3, "id": "b8f74fe9-0943-4bd9-a91b-da09c001f8a1", "metadata": {}, "outputs": [], @@ -251,7 +260,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 4, "id": "b937f6c7-3697-4bc0-8e32-da0b0faec593", "metadata": {}, "outputs": [ @@ -272,14 +281,12 @@ "id": "78fe7013-a5ae-498f-aeab-ba1f3c710de1", "metadata": {}, "source": [ - "Although we do not always need to, we should always enclose shell variables in double quotes and enclose the variable name in curly braces (funny things may happen otherwise). In most cases it is easier to do this every time than try to figure out when it is possible to use the short form. We only use the short form (`$TEST`) when it is used frequently and known to always work, in our case it is `$PROJECT` and `$BUCKET`. When in doubt, use the explicit form.\n", - "\n", "There are a number of environment variables already set that contain useful information, for example:" ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 5, "id": "bae0587c-0379-40e3-9c31-16ba5505487a", "metadata": {}, "outputs": [ @@ -287,17 +294,17 @@ "name": "stdout", "output_type": "stream", "text": [ - "student31\n" + "learner\n" ] } ], "source": [ - "echo \"${USER}\"" + "echo $USER" ] }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 9, "id": "52b4add5-0d18-4faf-a304-b7e0ee8ba82e", "metadata": {}, "outputs": [ @@ -305,12 +312,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "cs-730348008320-default-default-wwhtf\n" + "essentials\n" ] } ], "source": [ - "echo \"${HOSTNAME}\"" + "echo $HOSTNAME" ] }, { @@ -323,7 +330,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 12, "id": "776806e4-209f-45fa-b14e-f771f5d5089d", "metadata": {}, "outputs": [ @@ -331,7 +338,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Tue 02 Nov 2021 07:15:33 PM UTC\n" + "Thu Feb 3 20:10:51 UTC 2022\n" ] } ], @@ -341,7 +348,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 13, "id": "560fde63-0eb8-4c52-9787-4debfe35ec23", "metadata": {}, "outputs": [ @@ -349,7 +356,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Date: Tue 02 Nov 2021 07:15:34 PM UTC\n" + "Date: Thu Feb 3 20:10:52 UTC 2022\n" ] } ], @@ -357,6 +364,14 @@ "echo \"Date: $(date)\"" ] }, + { + "cell_type": "markdown", + "id": "4ba54cbd-573e-43f6-ad7c-ccf6dac871b0", + "metadata": {}, + "source": [ + "We use double quotes to make it a single string." + ] + }, { "cell_type": "markdown", "id": "6323f7bf-b2fe-4c94-a87a-2d35e75a1a49", @@ -369,7 +384,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 14, "id": "18f40620-8537-42b5-948f-dd581e604a36", "metadata": {}, "outputs": [ @@ -377,8 +392,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Your active configuration is: [cloudshell-17640]\n", - "just-armor-301114\n" + "essentials-learner\n" ] } ], @@ -388,27 +402,19 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 15, "id": "a959f857-8a20-4356-bc30-dbf8c75be96d", "metadata": { "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Your active configuration is: [cloudshell-17640]\n" - ] - } - ], + "outputs": [], "source": [ "PROJECT=\"$(gcloud config get-value project)\"" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 16, "id": "c1d27e89-cc69-4979-82c4-79c3fb36bccb", "metadata": {}, "outputs": [ @@ -416,7 +422,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "just-armor-301114\n" + "essentials-learner\n" ] } ], @@ -437,12 +443,12 @@ "id": "c25e6431-4f92-454a-bd10-e77cb28deec6", "metadata": {}, "source": [ - "Now use the `PROJECT` envrionment variable to get more information about the active project." + "Now use the `PROJECT` environment variable to get more information about the active project (you may be prompted to enable the API)." ] }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 19, "id": "31b01f78-94b1-44f0-a2af-d657058941b2", "metadata": {}, "outputs": [ @@ -450,11 +456,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "createTime: '2021-01-08T14:17:35.831Z'\n", + "createTime: '2022-01-14T19:03:26.486Z'\n", "lifecycleState: ACTIVE\n", - "name: CLASS Project\n", - "projectId: just-armor-301114\n", - "projectNumber: '1002111293252'\n" + "name: essentials-learner\n", + "parent:\n", + " id: '1072231596131'\n", + " type: folder\n", + "projectId: essentials-learner\n", + "projectNumber: '998995103712'\n" ] } ], @@ -472,7 +481,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 21, "id": "845b2aaf-0032-4043-a40a-d2482693d23e", "metadata": {}, "outputs": [ @@ -480,20 +489,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "[accessibility]\n", - "screen_reader = True\n", - "[component_manager]\n", - "disable_update_check = True\n", - "[compute]\n", - "gce_metadata_read_timeout_sec = 30\n", "[core]\n", - "account = student31@class.internet2.edu\n", + "account = 998995103712-compute@developer.gserviceaccount.com\n", "disable_usage_reporting = True\n", - "project = just-armor-301114\n", - "[metrics]\n", - "environment = devshell\n", + "project = essentials-learner\n", "\n", - "Your active configuration is: [cloudshell-17640]\n" + "Your active configuration is: [default]\n" ] } ], @@ -511,7 +512,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 23, "id": "9714ebe2-2921-498b-9e0e-69ec6a85896a", "metadata": {}, "outputs": [ @@ -519,7 +520,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "just-armor-301114\n" + "essentials-learner\n" ] } ], @@ -537,7 +538,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 25, "id": "7d803ddd-acca-4ad1-933a-4d6e6044def6", "metadata": {}, "outputs": [ @@ -545,12 +546,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "just-armor-301114\n" + "essentials-learner\n" ] } ], "source": [ - "PROJECT=\"$(gcloud config list --format='value(core.project)')\"\n", + "PROJECT=$(gcloud config list --format='value(core.project)')\n", "echo $PROJECT" ] }, @@ -588,7 +589,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 26, "id": "678a1735-8cad-403c-b302-639de1fb0f4b", "metadata": { "tags": [] @@ -598,26 +599,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"accessibility\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"screen_reader\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"True\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"component_manager\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"disable_update_check\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"True\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"compute\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"gce_metadata_read_timeout_sec\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"30\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"core\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"account\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"student31@class.internet2.edu\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"disable_usage_reporting\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"True\"\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"project\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"just-armor-301114\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", - " \u001b[0m\u001b[34;1m\"metrics\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", - " \u001b[0m\u001b[34;1m\"environment\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"devshell\"\u001b[0m\u001b[1;39m\n", - " \u001b[1;39m}\u001b[0m\u001b[1;39m\n", - "\u001b[1;39m}\u001b[0m\n" + "bash: jq: command not found\n" ] + }, + { + "ename": "", + "evalue": "127", + "output_type": "error", + "traceback": [] } ], "source": [ @@ -626,7 +615,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 27, "id": "b0dec5ed-dc7f-4003-b228-f6605119ce47", "metadata": {}, "outputs": [ @@ -634,8 +623,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "just-armor-301114\n" + "bash: jq: command not found\n" ] + }, + { + "ename": "", + "evalue": "127", + "output_type": "error", + "traceback": [] } ], "source": [ From 9c77425071ae559f7f1b187b4717845d76138322 Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 3 Feb 2022 21:56:49 +0000 Subject: [PATCH 4/6] GCP: Simplify the use of ENV var's. --- content/GCP/04_intro_to_cli.ipynb | 195 +++++++++--------------------- 1 file changed, 60 insertions(+), 135 deletions(-) diff --git a/content/GCP/04_intro_to_cli.ipynb b/content/GCP/04_intro_to_cli.ipynb index 6ffab2b..72499e4 100644 --- a/content/GCP/04_intro_to_cli.ipynb +++ b/content/GCP/04_intro_to_cli.ipynb @@ -104,7 +104,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 1, "id": "3ce9e6fb-9f5f-42e1-8512-35c0564f8e85", "metadata": {}, "outputs": [ @@ -112,7 +112,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "998995103712-compute@developer.gserviceaccount.com\n" + "Your active configuration is: [cloudshell-2071]\n", + "learner@class.internet2.edu\n" ] } ], @@ -130,7 +131,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "id": "cfb10587-e007-45ae-839f-7bcfe088c711", "metadata": {}, "outputs": [ @@ -138,6 +139,7 @@ "name": "stdout", "output_type": "stream", "text": [ + "Your active configuration is: [cloudshell-2071]\n", "essentials-learner\n" ] } @@ -156,7 +158,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 3, "id": "617325c9-d853-4291-a1db-938ab9439fee", "metadata": {}, "outputs": [ @@ -250,7 +252,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "id": "b8f74fe9-0943-4bd9-a91b-da09c001f8a1", "metadata": {}, "outputs": [], @@ -260,7 +262,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 6, "id": "b937f6c7-3697-4bc0-8e32-da0b0faec593", "metadata": {}, "outputs": [ @@ -286,7 +288,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 7, "id": "bae0587c-0379-40e3-9c31-16ba5505487a", "metadata": {}, "outputs": [ @@ -304,7 +306,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "id": "52b4add5-0d18-4faf-a304-b7e0ee8ba82e", "metadata": {}, "outputs": [ @@ -312,7 +314,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "essentials\n" + "cs-194566722441-default\n" ] } ], @@ -330,7 +332,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 9, "id": "776806e4-209f-45fa-b14e-f771f5d5089d", "metadata": {}, "outputs": [ @@ -338,7 +340,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Thu Feb 3 20:10:51 UTC 2022\n" + "Thu 03 Feb 2022 09:50:31 PM UTC\n" ] } ], @@ -348,7 +350,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 10, "id": "560fde63-0eb8-4c52-9787-4debfe35ec23", "metadata": {}, "outputs": [ @@ -356,7 +358,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Date: Thu Feb 3 20:10:52 UTC 2022\n" + "Date: Thu 03 Feb 2022 09:50:31 PM UTC\n" ] } ], @@ -384,7 +386,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 11, "id": "18f40620-8537-42b5-948f-dd581e604a36", "metadata": {}, "outputs": [ @@ -392,6 +394,7 @@ "name": "stdout", "output_type": "stream", "text": [ + "Your active configuration is: [cloudshell-2071]\n", "essentials-learner\n" ] } @@ -402,19 +405,27 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 12, "id": "a959f857-8a20-4356-bc30-dbf8c75be96d", "metadata": { "tags": [] }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Your active configuration is: [cloudshell-2071]\n" + ] + } + ], "source": [ - "PROJECT=\"$(gcloud config get-value project)\"" + "PROJECT=$(gcloud config get-value project)" ] }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 13, "id": "c1d27e89-cc69-4979-82c4-79c3fb36bccb", "metadata": {}, "outputs": [ @@ -435,7 +446,17 @@ "id": "9d3a8504-a75c-4945-ab54-85f1d9a7e2b2", "metadata": {}, "source": [ - "Note the \"Your active configuration is: ...\" line is a \"warning\" and not actually part of the output of the program." + "Note the \"Your active configuration is: ...\" line is a \"warning\" and not actually part of the output of the program.\n", + "\n", + "````{admonition} Advanced Tip\n", + ":class: tip\n", + "\n", + "You can also use \n", + "```\n", + "PROJECT=$(gcloud config list --format='value(core.project)')\n", + "```\n", + "to get the project id without the warning.\n", + "````" ] }, { @@ -448,7 +469,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 14, "id": "31b01f78-94b1-44f0-a2af-d657058941b2", "metadata": {}, "outputs": [ @@ -471,98 +492,6 @@ "gcloud projects describe $PROJECT" ] }, - { - "cell_type": "markdown", - "id": "ed2ab9ce-6b0a-4939-88c7-8103e29413b7", - "metadata": {}, - "source": [ - "Note the extraneous message about the active configuration produced by the `gcloud config get-value project` command. A slightly different way to access the data can be used to remove the extraneous message by using the following command:" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "845b2aaf-0032-4043-a40a-d2482693d23e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[core]\n", - "account = 998995103712-compute@developer.gserviceaccount.com\n", - "disable_usage_reporting = True\n", - "project = essentials-learner\n", - "\n", - "Your active configuration is: [default]\n" - ] - } - ], - "source": [ - "gcloud config list" - ] - }, - { - "cell_type": "markdown", - "id": "d919b26d-2f64-4a43-9c76-34f7798fe853", - "metadata": {}, - "source": [ - "We can now use the `--format='value()'` option to get specific values. The following command returns the project number without the extraneous message." - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "id": "9714ebe2-2921-498b-9e0e-69ec6a85896a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "essentials-learner\n" - ] - } - ], - "source": [ - "gcloud config list --format='value(core.project)'" - ] - }, - { - "cell_type": "markdown", - "id": "ba660d2b-5ddf-449b-8049-56a40f2e366b", - "metadata": {}, - "source": [ - "Note where the `core` part comes in the output of the `gcloud config list` command." - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "id": "7d803ddd-acca-4ad1-933a-4d6e6044def6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "essentials-learner\n" - ] - } - ], - "source": [ - "PROJECT=$(gcloud config list --format='value(core.project)')\n", - "echo $PROJECT" - ] - }, - { - "cell_type": "markdown", - "id": "d40b2c7b-7331-412d-80e1-f1a77204075a", - "metadata": {}, - "source": [ - "We will use this form from now on. Since we know the Project ID only contains numbers, letters, and dashes with no spaces there is no need to use quotes." - ] - }, { "cell_type": "markdown", "id": "270e0abb-d5e9-476a-af91-060cc0ea0349", @@ -570,10 +499,8 @@ "source": [ "```{admonition} Exercise\n", "\n", - " * What command shows the project number? (hint: we used it already)\n", - " * What is the name of this variable? (hint: it is case sensitive)\n", - " * What command shows *just* the Project Number (hint: use `--format`)\n", - " * Set the `PROJECT_NUMBER` environment variable with the correct value. (hint: be careful with punctuation (single and double quotes) and matching brackets and punctuation)\n", + " * What command shows the account? (hint: we used it already)\n", + " * Set the `ACCOUNT` environment variable with the correct value. (hint: be careful with punctuation (single and double quotes) and matching brackets and punctuation)\n", "```" ] }, @@ -584,12 +511,12 @@ "source": [ "## Formating and Scripting Output (Optional)\n", "\n", - "The following example shows how to output the configuration as `JSON`. The command `jq` can also be used to parse the JSON data." + "The following example shows how to output the configuration as `JSON`. The command `jq` can also be used to parse the JSON data in addition to the `--format` option." ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 15, "id": "678a1735-8cad-403c-b302-639de1fb0f4b", "metadata": { "tags": [] @@ -599,23 +526,27 @@ "name": "stdout", "output_type": "stream", "text": [ - "bash: jq: command not found\n" + "\u001b[1;39m{\n", + " \u001b[0m\u001b[34;1m\"createTime\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"2022-01-14T19:03:26.486Z\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"lifecycleState\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ACTIVE\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"essentials-learner\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"parent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n", + " \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1072231596131\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"folder\"\u001b[0m\u001b[1;39m\n", + " \u001b[1;39m}\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"projectId\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"essentials-learner\"\u001b[0m\u001b[1;39m,\n", + " \u001b[0m\u001b[34;1m\"projectNumber\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"998995103712\"\u001b[0m\u001b[1;39m\n", + "\u001b[1;39m}\u001b[0m\n" ] - }, - { - "ename": "", - "evalue": "127", - "output_type": "error", - "traceback": [] } ], "source": [ - "gcloud config list --format=json |jq" + "gcloud projects describe $PROJECT --format=json | jq" ] }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 16, "id": "b0dec5ed-dc7f-4003-b228-f6605119ce47", "metadata": {}, "outputs": [ @@ -623,18 +554,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "bash: jq: command not found\n" + "998995103712\n" ] - }, - { - "ename": "", - "evalue": "127", - "output_type": "error", - "traceback": [] } ], "source": [ - "gcloud config list --format=json |jq -r .core.project" + "gcloud projects describe $PROJECT --format=json | jq -r .projectNumber" ] } ], From 2a18af844fd3fb74d13c88a0f30d8f15be02cbc8 Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 3 Feb 2022 22:22:16 +0000 Subject: [PATCH 5/6] GCP: simplify syntax and re-render in cloudshell --- content/GCP/05_cli_storage.ipynb | 319 +++++++++++++------------------ 1 file changed, 137 insertions(+), 182 deletions(-) diff --git a/content/GCP/05_cli_storage.ipynb b/content/GCP/05_cli_storage.ipynb index 00d761a..878850e 100644 --- a/content/GCP/05_cli_storage.ipynb +++ b/content/GCP/05_cli_storage.ipynb @@ -53,7 +53,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "998995103712-compute@developer.gserviceaccount.com\n" + "Your active configuration is: [cloudshell-27767]\n", + "learner@class.internet2.edu\n" ] } ], @@ -71,6 +72,7 @@ "name": "stdout", "output_type": "stream", "text": [ + "Your active configuration is: [cloudshell-27767]\n", "essentials-learner\n" ] } @@ -82,6 +84,25 @@ { "cell_type": "code", "execution_count": 3, + "id": "44f0b255-508a-4488-abde-87aae67cd5b3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Your active configuration is: [cloudshell-27767]\n", + "learner@class.internet2.edu\n" + ] + } + ], + "source": [ + "ACCOUNT=$(gcloud config get-value account) ; echo $ACCOUNT" + ] + }, + { + "cell_type": "code", + "execution_count": 4, "id": "36fcc48b-8a5b-4158-9704-f1ea7c9951eb", "metadata": {}, "outputs": [ @@ -89,13 +110,13 @@ "name": "stdout", "output_type": "stream", "text": [ + "Your active configuration is: [cloudshell-27767]\n", "essentials-learner\n" ] } ], "source": [ - "PROJECT=$(gcloud config list --format='value(core.project)')\n", - "echo $PROJECT" + "PROJECT=$(gcloud config get-value project) ; echo $PROJECT" ] }, { @@ -118,7 +139,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "id": "255eb914-a06a-4bfd-adbb-7ca2f94678e7", "metadata": {}, "outputs": [ @@ -126,18 +147,18 @@ "name": "stdout", "output_type": "stream", "text": [ - "essentials-learner-2022-01-21\n" + "essentials-learner-2022-02-03\n" ] } ], "source": [ - "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", + "BUCKET=\"essentials-$USER-$(date +%F)\"\n", "echo $BUCKET" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 6, "id": "b0661b4c-b56b-4d6d-ba3c-e1c5f38699c6", "metadata": {}, "outputs": [ @@ -145,12 +166,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Creating gs://essentials-learner-2022-01-21/...\n" + "Creating gs://essentials-learner-2022-02-03/...\n" ] } ], "source": [ - "gsutil mb \"gs://$BUCKET\"" + "gsutil mb gs://$BUCKET" ] }, { @@ -175,7 +196,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "id": "8ae2988a-fca8-4601-a7e5-458627fe68e3", "metadata": {}, "outputs": [ @@ -183,7 +204,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-03/\n" ] } ], @@ -209,7 +230,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 8, "id": "ecc0e4fe-c9d0-4edd-9a0c-56b5b9ec7c54", "metadata": { "scrolled": true, @@ -220,7 +241,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/ :\n", + "gs://essentials-learner-2022-02-03/ :\n", "\tStorage class:\t\t\tSTANDARD\n", "\tLocation type:\t\t\tmulti-region\n", "\tLocation constraint:\t\tUS\n", @@ -232,71 +253,17 @@ "\tRequester Pays enabled:\t\tNone\n", "\tLabels:\t\t\t\tNone\n", "\tDefault KMS key:\t\tNone\n", - "\tTime created:\t\t\tFri, 21 Jan 2022 23:23:39 GMT\n", - "\tTime updated:\t\t\tFri, 21 Jan 2022 23:23:39 GMT\n", + "\tTime created:\t\t\tThu, 03 Feb 2022 22:19:56 GMT\n", + "\tTime updated:\t\t\tThu, 03 Feb 2022 22:19:56 GMT\n", "\tMetageneration:\t\t\t1\n", "\tBucket Policy Only enabled:\tFalse\n", "\tPublic access prevention:\tinherited\n", - "\tRPO:\t\t\t\tDEFAULT\n", - "\tACL:\t\t\t\t\n", - "\t [\n", - "\t {\n", - "\t \"entity\": \"project-owners-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"owners\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-editors-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"editors\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-viewers-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"viewers\"\n", - "\t },\n", - "\t \"role\": \"READER\"\n", - "\t }\n", - "\t ]\n", - "\tDefault ACL:\t\t\t\n", - "\t [\n", - "\t {\n", - "\t \"entity\": \"project-owners-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"owners\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-editors-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"editors\"\n", - "\t },\n", - "\t \"role\": \"OWNER\"\n", - "\t },\n", - "\t {\n", - "\t \"entity\": \"project-viewers-998995103712\",\n", - "\t \"projectTeam\": {\n", - "\t \"projectNumber\": \"998995103712\",\n", - "\t \"team\": \"viewers\"\n", - "\t },\n", - "\t \"role\": \"READER\"\n", - "\t }\n", - "\t ]\n" + "\tRPO:\t\t\t\tDEFAULT\n" ] } ], "source": [ - "gsutil ls -L -b \"gs://$BUCKET\"" + "gsutil ls -L -b gs://$BUCKET | head -18" ] }, { @@ -306,14 +273,14 @@ "source": [ "You can see that the bucket is \"multi-region\" and uses the \"standard\" storage class (this is the type of storage). The standard storage class is best used for frequently used buckets. Other storage classes are designed for less frequent use, are less expensive, but come with more restrictions and more complex access costs and should only be used after careful consideration.\n", "\n", - "Ignore the \"ACL\" section for now.\n", + "*We used the `| head -18 ` command here to limit the response to the first few lines (we ignore ACL for now)*\n", "\n", "You may also want to verify that you can see the newly created bucket in the web console dashboard or \"Cloud Storage\" page and explore the properties there." ] }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 9, "id": "8fc85600-a793-413b-b139-642e4cf08a8e", "metadata": {}, "outputs": [], @@ -334,7 +301,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 10, "id": "bb8e7270-2f64-4fbd-b312-b18a7960700f", "metadata": { "scrolled": true, @@ -346,76 +313,20 @@ "output_type": "stream", "text": [ "---\n", - "insertId: -o4omdgeg7sge\n", + "insertId: -oywycvf1xe8xy\n", "logName: projects/essentials-learner/logs/cloudaudit.googleapis.com%2Factivity\n", "protoPayload:\n", " '@type': type.googleapis.com/google.cloud.audit.AuditLog\n", " authenticationInfo:\n", - " principalEmail: 998995103712-compute@developer.gserviceaccount.com\n", - " serviceAccountDelegationInfo:\n", - " - firstPartyPrincipal:\n", - " principalEmail: service-998995103712@compute-system.iam.gserviceaccount.com\n", + " principalEmail: learner@class.internet2.edu\n", " authorizationInfo:\n", " - granted: true\n", - " permission: storage.buckets.create\n", - " resource: projects/_/buckets/essentials-learner-2022-01-21\n", - " resourceAttributes: {}\n", - " metadata:\n", - " rpo: DEFAULT\n", - " methodName: storage.buckets.create\n", - " request:\n", - " defaultObjectAcl:\n", - " '@type': type.googleapis.com/google.iam.v1.Policy\n", - " bindings:\n", - " - members:\n", - " - projectViewer:essentials-learner\n", - " role: roles/storage.legacyObjectReader\n", - " - members:\n", - " - projectOwner:essentials-learner\n", - " - projectEditor:essentials-learner\n", - " role: roles/storage.legacyObjectOwner\n", - " requestMetadata:\n", - " callerIp: 34.102.126.22\n", - " callerNetwork: //compute.googleapis.com/projects/essentials-learner/global/networks/__unknown__\n", - " callerSuppliedUserAgent: apitools Python/3.7.3 gsutil/5.5 (linux) analytics/disabled\n", - " interactive/True command/mb google-cloud-sdk/366.0.0,gzip(gfe)\n", - " destinationAttributes: {}\n", - " requestAttributes:\n", - " auth: {}\n", - " time: '2022-01-21T23:23:39.060742076Z'\n", - " resourceLocation:\n", - " currentLocations:\n", - " - us\n", - " resourceName: projects/_/buckets/essentials-learner-2022-01-21\n", - " serviceData:\n", - " '@type': type.googleapis.com/google.iam.v1.logging.AuditData\n", - " policyDelta:\n", - " bindingDeltas:\n", - " - action: ADD\n", - " member: projectEditor:essentials-learner\n", - " role: roles/storage.legacyBucketOwner\n", - " - action: ADD\n", - " member: projectOwner:essentials-learner\n", - " role: roles/storage.legacyBucketOwner\n", - " - action: ADD\n", - " member: projectViewer:essentials-learner\n", - " role: roles/storage.legacyBucketReader\n", - " serviceName: storage.googleapis.com\n", - " status: {}\n", - "receiveTimestamp: '2022-01-21T23:23:39.729940907Z'\n", - "resource:\n", - " labels:\n", - " bucket_name: essentials-learner-2022-01-21\n", - " location: us\n", - " project_id: essentials-learner\n", - " type: gcs_bucket\n", - "severity: NOTICE\n", - "timestamp: '2022-01-21T23:23:39.055676566Z'\n" + " permission: storage.buckets.create\n" ] } ], "source": [ - "gcloud logging read --limit 1" + "gcloud logging read --limit 1 | head" ] }, { @@ -455,7 +366,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 11, "id": "f9a90752-871f-4156-b122-4c46fd4bfa43", "metadata": {}, "outputs": [], @@ -473,7 +384,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 12, "id": "bc0fb778-cc2e-4af4-8aa8-ca90d6b4f877", "metadata": {}, "outputs": [ @@ -499,7 +410,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 13, "id": "6384974b-d422-4ac1-90b7-cf9feec9aa8c", "metadata": {}, "outputs": [ @@ -514,12 +425,12 @@ } ], "source": [ - "gsutil cp one.txt \"gs://$BUCKET/1\"" + "gsutil cp one.txt gs://$BUCKET/1" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 14, "id": "49589119-56bb-4c7b-b0a6-36e8f1133056", "metadata": {}, "outputs": [ @@ -527,12 +438,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/1\n" + "gs://essentials-learner-2022-02-03/1\n" ] } ], "source": [ - "gsutil ls \"gs://$BUCKET\"" + "gsutil ls gs://$BUCKET" ] }, { @@ -543,6 +454,24 @@ "Copy the object to the standard out (displays it on the screen)." ] }, + { + "cell_type": "code", + "execution_count": 15, + "id": "79d0f1ac-e42f-46c4-aeec-6548a2651015", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "test one\n" + ] + } + ], + "source": [ + "gsutil cat gs://$BUCKET/1" + ] + }, { "cell_type": "markdown", "id": "ef366240-1152-4b30-b99f-b7835f63788d", @@ -553,7 +482,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 16, "id": "442cc51f-8856-41d4-a3a6-072c8569beef", "metadata": {}, "outputs": [ @@ -571,7 +500,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 17, "id": "4e3c0de9-8a23-428a-9689-0cf896228afe", "metadata": {}, "outputs": [ @@ -584,7 +513,7 @@ } ], "source": [ - "gsutil cat \"gs://$BUCKET/1\"" + "gsutil cat gs://$BUCKET/1" ] }, { @@ -597,7 +526,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 18, "id": "13f94b36-7dea-46b9-9e8b-baa8b4e59de6", "metadata": {}, "outputs": [ @@ -605,19 +534,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "Copying gs://essentials-learner-2022-01-21/1...\n", + "Copying gs://essentials-learner-2022-02-03/1...\n", "/ [1 files][ 9.0 B/ 9.0 B] \n", "Operation completed over 1 objects/9.0 B. \n" ] } ], "source": [ - "gsutil cp \"gs://$BUCKET/1\" one.txt" + "gsutil cp gs://$BUCKET/1 one.txt" ] }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 19, "id": "3982f21c-b827-4772-aa8c-3ac67e478b0d", "metadata": {}, "outputs": [ @@ -643,7 +572,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 20, "id": "6b57a3fc-94ea-4c25-b7ac-2744d7b28852", "metadata": {}, "outputs": [ @@ -669,7 +598,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 21, "id": "5a7fc453-fca7-4ab4-8f0a-cb4de53bc992", "metadata": {}, "outputs": [ @@ -684,7 +613,7 @@ } ], "source": [ - "date | gsutil cp - \"gs://$BUCKET/2\"" + "date | gsutil cp - gs://$BUCKET/2" ] }, { @@ -697,7 +626,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 22, "id": "1be58d86-5de7-420c-a3b3-0b82839ac90f", "metadata": {}, "outputs": [ @@ -705,8 +634,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/1\n", - "gs://essentials-learner-2022-01-21/2\n" + "gs://essentials-learner-2022-02-03/1\n", + "gs://essentials-learner-2022-02-03/2\n" ] } ], @@ -721,13 +650,13 @@ "source": [ "```{admonition} Exercise\n", "\n", - "* Display the date in the bucket.\n", + "* Display the object with the date in the bucket.\n", "```" ] }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 23, "id": "e3d0585b-4d81-4429-ad90-b3ff51eec13d", "metadata": {}, "outputs": [ @@ -735,12 +664,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Fri Jan 21 23:23:53 UTC 2022\n" + "Thu 03 Feb 2022 10:20:26 PM UTC\n" ] } ], "source": [ - "gsutil cat \"gs://$BUCKET/2\"" + "gsutil cat gs://$BUCKET/2" ] }, { @@ -753,7 +682,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 24, "id": "9ad460d9-7a4f-4e3f-befb-2bbb10c51993", "metadata": {}, "outputs": [ @@ -761,19 +690,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/1...\n", + "Removing gs://essentials-learner-2022-02-03/1...\n", "/ [1 objects] \n", "Operation completed over 1 objects. \n" ] } ], "source": [ - "gsutil rm \"gs://$BUCKET/1\"" + "gsutil rm gs://$BUCKET/1" ] }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 25, "id": "8fd782f2-700a-4862-970b-1b4751009d5f", "metadata": {}, "outputs": [ @@ -781,12 +710,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/2\n" + "gs://essentials-learner-2022-02-03/2\n" ] } ], "source": [ - "gsutil ls \"gs://$BUCKET\"" + "gsutil ls gs://$BUCKET" ] }, { @@ -799,7 +728,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 26, "id": "e1350695-0f01-4205-a604-91d62896783e", "metadata": {}, "outputs": [ @@ -807,13 +736,13 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/...\n", - "NotEmptyException: 409 BucketNotEmpty (essentials-learner-2022-01-21)\n" + "Removing gs://essentials-learner-2022-02-03/...\n", + "NotEmptyException: 409 BucketNotEmpty (essentials-learner-2022-02-03)\n" ] } ], "source": [ - "gsutil rb \"gs://$BUCKET\"\n", + "gsutil rb gs://$BUCKET\n", "/bin/true # ignore expected error in Jupyter" ] }, @@ -827,7 +756,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 27, "id": "ba2671f8-521f-48b4-9231-491d128ba67c", "metadata": {}, "outputs": [ @@ -835,19 +764,19 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/2...\n", + "Removing gs://essentials-learner-2022-02-03/2...\n", "/ [1 objects] \n", "Operation completed over 1 objects. \n" ] } ], "source": [ - "gsutil rm \"gs://$BUCKET/2\"" + "gsutil rm gs://$BUCKET/2" ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 28, "id": "73d8b299-8e8e-443e-ae5b-4ecf7efa0442", "metadata": {}, "outputs": [ @@ -855,7 +784,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-03/\n" ] } ], @@ -874,7 +803,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 29, "id": "7b32764d-e7b5-42d6-8f88-bc6936b2024a", "metadata": {}, "outputs": [ @@ -882,7 +811,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "gs://essentials-learner-2022-01-21/\n" + "gs://essentials-learner-2022-02-03/\n" ] } ], @@ -900,7 +829,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 30, "id": "82487bd8-14f4-4811-8f5a-dd346042bd0f", "metadata": {}, "outputs": [ @@ -908,7 +837,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Bucket: essentials-learner-2022-01-21\n" + "Bucket: essentials-learner-2022-02-03\n" ] } ], @@ -916,9 +845,35 @@ "echo \"Bucket: $BUCKET\"" ] }, + { + "cell_type": "markdown", + "id": "6120eee3-47d0-4e28-a1dd-b1090297a573", + "metadata": {}, + "source": [ + "Check to see if the bucket is empty" + ] + }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 31, + "id": "a52e1838-02f4-406c-a58f-6a709d9aaf2e", + "metadata": {}, + "outputs": [], + "source": [ + "gsutil ls gs://$BUCKET" + ] + }, + { + "cell_type": "markdown", + "id": "6eb97c7e-aed6-493a-b724-a4d6ba3cb806", + "metadata": {}, + "source": [ + "Remove the bucket" + ] + }, + { + "cell_type": "code", + "execution_count": 32, "id": "8635db94-2f3c-46d2-acc2-76a176fb37e7", "metadata": {}, "outputs": [ @@ -926,12 +881,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Removing gs://essentials-learner-2022-01-21/...\n" + "Removing gs://essentials-learner-2022-02-03/...\n" ] } ], "source": [ - "gsutil rb \"gs://$BUCKET\"" + "gsutil rb gs://$BUCKET" ] }, { @@ -939,12 +894,12 @@ "id": "3e2178ca", "metadata": {}, "source": [ - "We verify that the bucket has been removed. In this example there is no output since there are no more buckets." + "We verify that the bucket has been removed." ] }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 33, "id": "ed6f0204-0de2-4d20-ab00-3b7a1023fd86", "metadata": {}, "outputs": [], From 48c028b24c8972f8e0afa54fa8e13ca71e721fb5 Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Thu, 3 Feb 2022 16:35:52 -0600 Subject: [PATCH 6/6] GCP: Scripts to run jupyterlab in cloudshell --- Build.md | 25 +++++++++++++++++++------ scripts/gcp-cloudshell-jupyter-lab.sh | 4 ++++ scripts/gcp-cloudshell.sh | 4 ++++ 3 files changed, 27 insertions(+), 6 deletions(-) create mode 100755 scripts/gcp-cloudshell-jupyter-lab.sh create mode 100755 scripts/gcp-cloudshell.sh diff --git a/Build.md b/Build.md index c4033d1..2a88a6d 100644 --- a/Build.md +++ b/Build.md @@ -32,14 +32,27 @@ This copies the contents of `content/_build/html/` to `docs/` so GitHub Pages wi ## GCP -Using Jupyter launched from a remote Visual Code Studio session inside GCP Cloud-Shell requires some local settings stored in `local.sh`, use the following as a template replacing the project and email with the proper vaules: -``` -gcloud config set account student31@class.internet2.edu -export GOOGLE_CLOUD_PROJECT=just-armor-301114 -export DEVSHELL_PROJECT_ID=$GOOGLE_CLOUD_PROJECT -gcloud config set project $GOOGLE_CLOUD_PROJECT +Checkout the project so it is not seen by the students. Run `./scripts/gcp-cloudshell.sh` to open the shell with the agent and ports 8080 and 8081 forwarded and run the following commands (onetime setup). Be sure to check the project and account values. + +```bash +git clone git@github.internet2.edu:/CLASS/CLASS-Essentials.git .essentials +cd .essentials/ +git checkout -b gcp-dev --track origin/gcp-dev + +ACCOUNT=learner@class.internet2.edu +PROJECT=essentials-learner +cat > .essentials/local.sh <