From c1318ad4da38d09bc13744c6e4f75e6c1988debf Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Tue, 15 Feb 2022 23:32:23 +0000 Subject: [PATCH] GCP: Cleanup and Speedup. --- content/GCP/01_intro_to_cloud_console.ipynb | 2 +- content/GCP/02_intro_to_compute.ipynb | 4 +- content/GCP/03_intro_to_cloud_storage.ipynb | 70 +++------------------ content/GCP/04_intro_to_cli.ipynb | 56 ++--------------- content/GCP/06_running_analysis.ipynb | 45 ++++++++++--- content/GCP/06_sharing_results.ipynb | 35 +++++++++++ content/GCP/07_monitoring_costs.ipynb | 2 +- scripts/gcp-create.sh | 2 +- 8 files changed, 91 insertions(+), 125 deletions(-) diff --git a/content/GCP/01_intro_to_cloud_console.ipynb b/content/GCP/01_intro_to_cloud_console.ipynb index 7740272..60e21b6 100644 --- a/content/GCP/01_intro_to_cloud_console.ipynb +++ b/content/GCP/01_intro_to_cloud_console.ipynb @@ -12,7 +12,7 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 10 min.\n", + "**Teaching:** 15 min.\n", "\n", "**Exercises:** 6 min.\n", "\n", diff --git a/content/GCP/02_intro_to_compute.ipynb b/content/GCP/02_intro_to_compute.ipynb index cd7504d..f210727 100644 --- a/content/GCP/02_intro_to_compute.ipynb +++ b/content/GCP/02_intro_to_compute.ipynb @@ -12,7 +12,7 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 45 min.\n", + "**Teaching:** 30 min.\n", "\n", "**Exercises:** 6 min\n", "\n", @@ -147,7 +147,7 @@ "tags": [] }, "source": [ - "## Find the VM Instances\n", + "## Find the VM Instance\n", "\n", "Now lets find and connect to the *VM Instance*.\n", " * Navigate to the Google Compute Engine page by clicking **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances**.\n", diff --git a/content/GCP/03_intro_to_cloud_storage.ipynb b/content/GCP/03_intro_to_cloud_storage.ipynb index 452861e..a8e3028 100644 --- a/content/GCP/03_intro_to_cloud_storage.ipynb +++ b/content/GCP/03_intro_to_cloud_storage.ipynb @@ -10,7 +10,7 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 40 min\n", + "**Teaching:** 20 min\n", "\n", "**Exercises:** 5 min\n", "\n", @@ -19,7 +19,6 @@ "\n", "**Objectives:**\n", "* Navigate the Google Cloud Storage service and terminology\n", - "* Understand the roles and permissions needed to use Google Cloud Storage in projects\n", "* Allocate storage in Google Cloud Storage\n", "* Find the cost estimator for Google Cloud Storage\n", "* Recognize that resources have a \"location\"\n", @@ -50,41 +49,6 @@ "We now take Drew through the process of creating a Google Cloud Storage bucket." ] }, - { - "cell_type": "markdown", - "id": "07fb9096-2b40-4995-a742-be7bd9b2797c", - "metadata": { - "tags": [] - }, - "source": [ - "## Security\n", - "\n", - "Everything in the cloud requires permission (authorization). Let's first verify that we have the permissions to create a bucket. A Bucket (a resource) is created within a project and inheres permissions from it.\n", - "\n", - "We are interested in what permissions that *your* account has for *your* project. To do this navigate to the IAM page (**Navigation Menu -> IAM & Admin -> IAM -> Permissions -> View By: Principals**). This shows the permissions for the project.\n", - "\n", - "*Note: There is a powerful filter box to limit the permissions shown.*\n", - "\n", - "You should see a row with your account shown in the Principal column. Here you should see the \"Editor\" Role in the Role column. A *role* is a collection of permissions managed by Google or someone else. The **Editor**, **Owner**, or the **Storage Admin** role for a project will *allow* *you* to create, access, and delete Buckets *in* the project.\n", - "\n", - "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal), and the resource (project). This is another who (identity), what (permission), and where (resource)." - ] - }, - { - "cell_type": "markdown", - "id": "9acf29cf-660b-4922-bcb8-89fd9080fdea", - "metadata": { - "tags": [] - }, - "source": [ - "```{admonition} Exercise\n", - "\n", - "Answer the following questions:\n", - " * What is the \"Who, What, Where\" of the IAM policy that allows you to use your project?\n", - " * What else has permissions to do things in your project and state the \"Who, What, Where\"?\n", - "```" - ] - }, { "cell_type": "markdown", "id": "c5430b40-1a5f-40df-9e13-529ef3ece4ce", @@ -94,12 +58,12 @@ "source": [ "## Allocate Google Cloud Storage\n", "\n", - "Now that we have verified the permissions we can now create a bucket. Buckets are where objects are stored and have a globally unique name.\n", + "Buckets are where objects are stored and have a globally unique name.\n", "\n", "To create a bucket we do the following:\n", " * Click **Navigation Menu** -> **Cloud Storage** (under Storage) -> **Browser** -> **+Create Bucket** (just under the blue bar) to open the *Create a bucket* page.\n", " * In *Name your bucket*:\n", - " * For **Name**, enter a globally unique name for the bucket (example \"essentials-test-myname-2021-01-01\")\n", + " * For **Name**, enter a globally unique name for the bucket (example \"**essentials-test-myname-2022-01-01**\")\n", " * Click **Continue**\n", " * In *Choose where to store your data*:\n", " * For *Location Type* select **Region** (cheapest and fastest)\n", @@ -129,9 +93,9 @@ "tags": [] }, "source": [ - "## Follow the Storage Allocation\n", + "## Track the Storage Allocation\n", "\n", - "Just as with compute, we will audit (follow) the bucket creation by examining at the project *activity*.\n", + "Just as with compute, we will track (follow) the bucket creation by examining at the project *activity*.\n", "\n", "To view the project activity we do the following:\n", "\n", @@ -150,15 +114,13 @@ "tags": [] }, "source": [ - "## Enumerate the Buckets\n", + "## List the Buckets\n", "\n", "Now lets find and examine the bucket. To view a bucket we do the following:\n", "\n", " * Navigate to the Google Storage page by clicking **Navigation Menu** -> **Cloud Storage** (under Storage) -> **Browser**. \n", " * **Find** the bucket you just created. You can use the filter to find a bucket if there are a lot of them.\n", - " * Click on the bucket name to open the **bucket details** (it will display as a hyperlink when you hover over the bucket name).\n", - "\n", - "Navigate to the **dashboard** and you will now see \"Storage\" in the *resources* card under. You can click on this to quickly navigate to the storage page." + " * Click on the bucket name to open the **bucket details** (it will display as a hyperlink when you hover over the bucket name).\n" ] }, { @@ -170,12 +132,12 @@ "source": [ "## Review what is Important\n", "\n", - "It is always important to review what is important to you. It may be cost, or keeping the data secure. Later on we will show how to monitor overall costs.\n", + "It is always important to review what is important to you. It may be cost, or keeping the data secure. Later on we will show how to monitor overall costs. We will also learn how to use the \"info panel\" to show more information about a bucket.\n", "\n", "For Drew, we will review that the bucket **public access** is *not public* by doing the following:\n", " * Go to **Navigation Menu -> Cloud Storage -> Browser**\n", " * Select the bucket of interest by **checking the box** next to the Bucket name.\n", - " * In the Right Side Bar (open if necessary) in the **Permissions** tab in the **Public Access** card you should see **Not Public**. This means that public access prevention is turned on.\n", + " * In the **Info Panel** (click show \"Info Panel\" if necessary) in the **Permissions** tab in the **Public Access** card you should see **Not Public**. This means that public access prevention is turned on.\n", " * You can also see the **permissions** for the bucket in the bottom of the bar." ] }, @@ -225,23 +187,11 @@ "\n", "![storage-delete-bucket](img/storage-delete-bucket.png)\n", "\n", - "Did you \"Follow\" the bucket by looking at the **activity** page as discussed above?\n", + "Did you \"Track\" the bucket by looking at the **activity** page as discussed above?\n", "\n", "Since we care about paying for resources we are not using we review our project by visiting the *compute storage* service and reviewing that we no longer have any *Buckets* allocated. " ] }, - { - "cell_type": "markdown", - "id": "3a28e28d-1d70-44fa-a952-4f3506ea85ec", - "metadata": {}, - "source": [ - "## Discussion (Optional)\n", - "\n", - "* What does the words \"Secure\", \"Allocate\", \"Follow\", and \"Enumerate\" spell?\n", - "* What happens when you add the \"R\" in \"Review?\"\n", - "* Is this useful?" - ] - }, { "cell_type": "markdown", "id": "97d7ebc5-4a81-4f1a-aaf3-517adf70640a", diff --git a/content/GCP/04_intro_to_cli.ipynb b/content/GCP/04_intro_to_cli.ipynb index 72499e4..957463e 100644 --- a/content/GCP/04_intro_to_cli.ipynb +++ b/content/GCP/04_intro_to_cli.ipynb @@ -10,7 +10,7 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 40 min\n", + "**Teaching:** 20 min\n", "\n", "**Exercises:** 5 min\n", "\n", @@ -23,8 +23,7 @@ "* Use basic cloud CLI commands (`gcloud` and `gsutil`).\n", "* Verify basic settings.\n", "* Use environment variables for configuration.\n", - "* Understand the importance of using variables for configuration.\n", - "* Recognize the value of reproducibility and automation.\n", + "* Understand the importance of using variables reproducibility and automation.\n", "``` " ] }, @@ -50,7 +49,7 @@ "\n", "The cloud can be controlled using a Command Line Interface (CLI) or a programming language such as Python. Collectively these tools interact with the cloud over a Application Programming Interface (API) and this capability forms the basis of the cloud, the ability to control infrastructure programmatically.\n", "\n", - "Just as with navigating the web console it is important to know the **who**, **what**, and **where** of CLI access to reduce the possibility of access mistakes. We will first verify the tools are installed and configured correctly. Next we get the Account being used (who) and the Project ID of the active project (where) using the `gcloud` command. We will then use the `gcloud` and `gsutil` commands to list some public Buckets (what).\n", + "Just as with navigating the web console it is important to know the **who**, **where**, and **what** of CLI access to reduce the possibility of access mistakes. We will first verify the tools are installed and configured correctly. Next we get the Account being used (who) and the Project ID of the active project (where) using the `gcloud` command. We will then use the `gcloud` and `gsutil` commands to list some public Buckets (what).\n", "\n", "The `gcloud` command is used to control most aspects of GCP and the `gsutil` command is used to control Google Cloud Storage Buckets. To access the manual pages for a command just add `--help` to the end of the command or run `gcloud help` for more information.\n", "\n", @@ -97,7 +96,7 @@ "tags": [] }, "source": [ - "## Verify the Configuration (Who, What, Where)\n", + "## Verify the Configuration (Who, Where, What)\n", "\n", "First, let's verify that the Account being used for access (who) is what we expect." ] @@ -148,57 +147,12 @@ "gcloud config get-value project" ] }, - { - "cell_type": "markdown", - "id": "c2972d7b-f393-42b5-8330-cf8292d28afb", - "metadata": {}, - "source": [ - "Now we will use `gcloud` to list a well known public bucket (what). " - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "617325c9-d853-4291-a1db-938ab9439fee", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://gcp-public-data-landsat/index.csv.gz\n", - "gs://gcp-public-data-landsat/LC08/\n", - "gs://gcp-public-data-landsat/LE07/\n", - "gs://gcp-public-data-landsat/LM01/\n", - "gs://gcp-public-data-landsat/LM02/\n", - "gs://gcp-public-data-landsat/LM03/\n", - "gs://gcp-public-data-landsat/LM04/\n", - "gs://gcp-public-data-landsat/LM05/\n", - "gs://gcp-public-data-landsat/LO08/\n", - "gs://gcp-public-data-landsat/LT04/\n", - "gs://gcp-public-data-landsat/LT05/\n", - "gs://gcp-public-data-landsat/LT08/\n" - ] - } - ], - "source": [ - "gcloud alpha storage ls gs://gcp-public-data-landsat" - ] - }, - { - "cell_type": "markdown", - "id": "42e6ad5b-186d-4cd1-ba03-e7c85ad40e38", - "metadata": {}, - "source": [ - "*Advanced Callout: The `alpha` (and `beta`) command allows us to access commands that have not been released for production and care should be taken when using these in a production environment. At this time this is not the recommended way to access storage buckets, but it does help verify that everything is working correctly.*" - ] - }, { "cell_type": "markdown", "id": "1389ca4f-7234-4ea3-9ad8-85914d88ede5", "metadata": {}, "source": [ - "Finally, we will verify that the separate and preferred `gsutil` command is installed and working by listing the same well known public bucket. " + "Now we will use `gsutil` to list a well known public bucket (what). The `gsutil` command is how we access Google Cloud Storage, most other services use the `gcloud` command." ] }, { diff --git a/content/GCP/06_running_analysis.ipynb b/content/GCP/06_running_analysis.ipynb index e6adc25..5a725e7 100644 --- a/content/GCP/06_running_analysis.ipynb +++ b/content/GCP/06_running_analysis.ipynb @@ -10,7 +10,7 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 80 min\n", + "**Teaching:** 60 min\n", "\n", "**Exercises:** 8 min\n", "\n", @@ -377,7 +377,7 @@ "id": "2b8b3144-7453-4350-a1cf-7fa74af2bcbf", "metadata": {}, "source": [ - "## Access the bucket\n", + "## Access the Bucket\n", "\n", "Now we need to verify that Drew has access to the analysis data. \n", "\n", @@ -418,7 +418,7 @@ "id": "9e16e8b5-a178-492a-aa80-5affe721b6ca", "metadata": {}, "source": [ - "## Getting the data" + "## Getting the Metadata" ] }, { @@ -552,11 +552,25 @@ }, { "cell_type": "markdown", - "id": "f98c38de-87fa-4e66-9d4b-186fbf81b3b2", + "id": "588ddd30-af8f-4378-b543-290c4c6f0840", "metadata": {}, "source": [ "````{admonition} Tip\n", - ":class: Tip\n", + ":class: tip\n", + "To run the above commands in one step run\n", + "\n", + "```\n", + "bash get-index.sh\n", + "```\n", + "````" + ] + }, + { + "cell_type": "markdown", + "id": "f98c38de-87fa-4e66-9d4b-186fbf81b3b2", + "metadata": {}, + "source": [ + "````{admonition} Break (Optional)\n", "\n", "Now our virtual machine instance is ready and we can access the code and data. Now is a great time to take a short break.\n", "````" @@ -567,7 +581,7 @@ "id": "532e6da3-302a-4e8a-8570-752995f30f1d", "metadata": {}, "source": [ - "## Search for Data\n", + "## Getting the Data\n", "\n", "We can see the data is well formed and what we expect. We will now use this data to download data related to a specific point and for the Landsat 8. The following script does a simple filter." ] @@ -672,8 +686,6 @@ "id": "a76f24f8-3b2d-4c0d-880b-f2911b9d9b84", "metadata": {}, "source": [ - "## Download the Data\n", - "\n", "Now that we have a list of folders we are interested, we will now download them with a simple script that takes bucket addresses (URL's) and downloads them with the `gsutil` program." ] }, @@ -805,6 +817,21 @@ "ls -l data" ] }, + { + "cell_type": "markdown", + "id": "0ca2e2ff-8276-4092-8a9e-ca754db5078e", + "metadata": {}, + "source": [ + "````{admonition} Tip\n", + ":class: tip\n", + "To run the above analysis in one step run\n", + "\n", + "```\n", + "bash get-data.sh\n", + "```\n", + "````" + ] + }, { "cell_type": "markdown", "id": "6073d5f6-68ef-41df-8044-73f221ce8780", @@ -1396,7 +1423,7 @@ } ], "source": [ - "/usr/bin/python3 process_sat.py" + "python3 process_sat.py" ] }, { diff --git a/content/GCP/06_sharing_results.ipynb b/content/GCP/06_sharing_results.ipynb index 3c777dc..edc6e0b 100644 --- a/content/GCP/06_sharing_results.ipynb +++ b/content/GCP/06_sharing_results.ipynb @@ -23,6 +23,41 @@ "```" ] }, + { + "cell_type": "markdown", + "id": "c08aca46-0c53-4d44-aded-74a41eb38fdf", + "metadata": { + "tags": [] + }, + "source": [ + "## Security\n", + "\n", + "Everything in the cloud requires permission (authorization). Let's first verify that we have the permissions to create a bucket. A Bucket (a resource) is created within a project and inheres permissions from it.\n", + "\n", + "We are interested in what permissions that *your* account has for *your* project. To do this navigate to the IAM page (**Navigation Menu -> IAM & Admin -> IAM -> Permissions -> View By: Principals**). This shows the permissions for the project.\n", + "\n", + "*Note: There is a powerful filter box to limit the permissions shown.*\n", + "\n", + "You should see a row with your account shown in the Principal column. Here you should see the \"Editor\" Role in the Role column. A *role* is a collection of permissions managed by Google or someone else. The **Editor**, **Owner**, or the **Storage Admin** role for a project will *allow* *you* to create, access, and delete Buckets *in* the project.\n", + "\n", + "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal), and the resource (project). This is another who (identity), what (permission), and where (resource)." + ] + }, + { + "cell_type": "markdown", + "id": "c37b2071-b51a-42fd-acdf-e05a53e7fc8e", + "metadata": { + "tags": [] + }, + "source": [ + "```{admonition} Exercise\n", + "\n", + "Answer the following questions:\n", + " * What is the \"Who, What, Where\" of the IAM policy that allows you to use your project?\n", + " * What else has permissions to do things in your project and state the \"Who, What, Where\"?\n", + "```" + ] + }, { "cell_type": "markdown", "id": "a1c11268-f389-405f-8d96-3c319a49b882", diff --git a/content/GCP/07_monitoring_costs.ipynb b/content/GCP/07_monitoring_costs.ipynb index 920bb05..781ba4e 100644 --- a/content/GCP/07_monitoring_costs.ipynb +++ b/content/GCP/07_monitoring_costs.ipynb @@ -10,7 +10,7 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:** 30 min\n", + "**Teaching:** 10 min\n", "\n", "**Exercises:** none\n", "\n", diff --git a/scripts/gcp-create.sh b/scripts/gcp-create.sh index efbc780..aaaefed 100755 --- a/scripts/gcp-create.sh +++ b/scripts/gcp-create.sh @@ -18,7 +18,7 @@ echo "=== gcp-dev.sh $PROJECT $BRANCH" echo "+++ creating/starting VM" gcloud compute instances create --zone=$ZONE $VM \ --scopes=cloud-platform \ - --machine-type=e2-highcpu-2 --preemptible + --machine-type=e2-standard-2 --preemptible gcloud compute instances start --zone=$ZONE $VM IP=$(gcloud compute instances describe essentials --zone=$ZONE --format='value(networkInterfaces.accessConfigs[0].natIP)')