From e0fa9ac40e4e6702f4f95bf7b46e339e3c558912 Mon Sep 17 00:00:00 2001 From: Timothy Middelkoop Date: Fri, 21 Jan 2022 23:07:13 +0000 Subject: [PATCH] GCP: Cleanup and move to Web Console for Cleanup --- content/GCP/07_monitoring_costs.ipynb | 2 +- content/GCP/08_cleaning_up_resources.ipynb | 350 ++++----------------- 2 files changed, 58 insertions(+), 294 deletions(-) diff --git a/content/GCP/07_monitoring_costs.ipynb b/content/GCP/07_monitoring_costs.ipynb index 1b2ef8c..920bb05 100644 --- a/content/GCP/07_monitoring_costs.ipynb +++ b/content/GCP/07_monitoring_costs.ipynb @@ -392,7 +392,7 @@ "id": "45681123-b27a-48d8-ac4a-30dc3b46d2c1", "metadata": {}, "source": [ - "## (CLI - Extra)\n", + "## Label Resources (CLI - Extra)\n", "\n", "In order to track resources it is important to Label them. Labeling places key-value pairs (key=value) in the resources metadata (information about the resource) and most of the Google Cloud tools can use this information to search, filter, and group information. In the literature, the example most used is the \"env\" Label and indicates if the resource is in development (dev), testing (test), or production (prod); the lables would be env=test, env=dev and env=prod, respectively. \n", "\n", diff --git a/content/GCP/08_cleaning_up_resources.ipynb b/content/GCP/08_cleaning_up_resources.ipynb index aee3e3d..3880c63 100644 --- a/content/GCP/08_cleaning_up_resources.ipynb +++ b/content/GCP/08_cleaning_up_resources.ipynb @@ -10,9 +10,9 @@ "```{admonition} Overview\n", ":class: tip\n", "\n", - "**Teaching:**\n", + "**Teaching:** 5 min\n", "\n", - "**Exercises:**\n", + "**Exercises:** 10 min\n", "\n", "**Questions:**\n", "* How do I clean up after I'm done for the day?\n", @@ -27,317 +27,81 @@ }, { "cell_type": "markdown", - "id": "7621ba3b-c110-48aa-8300-14bbe5ee845f", + "id": "65fb0615-f942-4af0-9087-8f0696df95f9", "metadata": {}, "source": [ - "## Enumerating Existing Resources\n", + "## Cleanup VMs\n", "\n", - "We will first use the method from the previous lesson (see if you can do this on your own first) to check which resources are currently running so we can verify that they get removed." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "c8ee7307-4ed5-437f-b4e2-400b64190399", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//storage.googleapis.com/essentials-learner-2021-12-16\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --asset-types='compute.googleapis.com/Disk,compute.googleapis.com/Instance,storage.googleapis.com/Bucket' --format='value(name)'" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "67f75321-d684-4930-bd16-8112d4ced868", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n", - "//storage.googleapis.com/essentials-learner-2021-12-16\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort" - ] - }, - { - "cell_type": "markdown", - "id": "d43bd86d-acf7-416a-9b35-506f5625f2c3", - "metadata": {}, - "source": [ - "## Cleanup Buckets\n", + "```{admonition} Danger\n", + ":class: danger\n", "\n", - "First set the environment variables and verify them. If they are previously set you can just verify them. You can find the zone from the previous commands." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "03226ee4-00d3-4944-b60c-bd1ad62f1077", - "metadata": {}, - "outputs": [], - "source": [ - "ACCOUNT=$(gcloud config list --format='value(core.account)')\n", - "PROJECT=$(gcloud config list --format='value(core.project)')\n", - "BUCKET=\"essentials-${USER}-*\"\n", - "REGION=\"us-west2\"\n", - "ZONE=\"us-west2-c\"" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "75bf0dd4-4a62-497a-b491-2bc80285aa44", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Account: learner@class.internet2.edu, Project: essentials-project-335018, Region: us-west2 Zone: us-west2-c\n", - "Bucket: essentials-learner-*\n" - ] - } - ], - "source": [ - "echo \"Account: $ACCOUNT, Project: $PROJECT, Region: $REGION Zone: $ZONE\"\n", - "echo \"Bucket: $BUCKET\"" + "This will destroy the VM instance and and ALL the data in it! You cannot undelete an instance!\n", + "```\n", + "\n", + "```{admonition} Exercise\n", + "\n", + " * Try to delete the **VM Instance** on your own\n", + "```\n", + "\n", + "Navigate to the **Compute Engine** service and in the **VM Instances** page and remove the *VM Instance*.\n" ] }, { "cell_type": "markdown", - "id": "b4b93fcb-d72b-4b7c-ab40-5e5ad83fce86", + "id": "92321a60-13c6-4c87-aff5-92fb708e8bff", "metadata": {}, "source": [ "## Cleanup Buckets\n", "\n", - "We will now list the buckets that we will be deleting. Note the wild-card (GLOB) from above will match all dates (and all strings)." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "93add660-5bdb-4393-a05a-031ec301d7f8", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gs://essentials-learner-2021-12-16/\n" - ] - } - ], - "source": [ - "gsutil ls -b \"gs://$BUCKET\"" - ] - }, - { - "cell_type": "markdown", - "id": "8daa2ec9-d64d-4586-8ab9-33f03c1b1138", - "metadata": {}, - "source": [ - "**WARNING: All of the above buckets will be deleted if the following commands are run**" - ] - }, - { - "cell_type": "markdown", - "id": "be020ee5-d3af-4ada-a674-51775648f373", - "metadata": {}, - "source": [ - "**Delete** all the objects first" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "24616c20-ae7e-4d0f-9f9c-6c8752891fa7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Removing gs://essentials-learner-2021-12-16/1#1639684995415818...\n", - "/ [1/1 objects] 100% Done \n", - "Operation completed over 1 objects. \n" - ] - } - ], - "source": [ - "gsutil -m rm -r \"gs://$BUCKET/*\"" - ] - }, - { - "cell_type": "markdown", - "id": "194c0181-6a47-43c5-af40-29230d858164", - "metadata": {}, - "source": [ - "Delete the empty buckets" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "a1310b76-6212-48fa-b023-0a15e06d4332", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Removing gs://essentials-learner-2021-12-16/...\n" - ] - } - ], - "source": [ - "gsutil rb \"gs://$BUCKET/\"" - ] - }, - { - "cell_type": "markdown", - "id": "4bf0ed5b-b4f3-4061-ae65-50bf9d2500f7", - "metadata": {}, - "source": [ - "## Enumerating Resources\n", + "```{admonition} Danger\n", + ":class: danger\n", "\n", - "Now we will check that all the resources were removed." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "b088681d-633d-4991-ba7d-2ecfb41c36dd", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --asset-types='compute.googleapis.com/Disk,compute.googleapis.com/Instance,storage.googleapis.com/Bucket' --format='value(name)'" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "74d0e8e7-f097-41c5-88fe-929edcb0c1c1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n", - "//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort" - ] - }, - { - "cell_type": "markdown", - "id": "f275a53a-a3d1-4a0f-a44c-21e2ba7186ba", - "metadata": {}, - "source": [ - "Looks like we forgot to delete the VM." - ] - }, - { - "cell_type": "markdown", - "id": "74fd626a-6102-4815-a8c2-27aade6ebc3b", - "metadata": {}, - "source": [ - "## Cleanup VMs\n", - "We can also create and delete VMs using the `gcloud` command. Note, the `--quiet` disables conformation. **WARNING: This command will immediately delete the VM called essentials**" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "2ab8b339-b923-4623-942c-ddd424e3bcf7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Deleted [https://www.googleapis.com/compute/v1/projects/essentials-project-335018/zones/us-west2-c/instances/essentials].\n" - ] - } - ], - "source": [ - "gcloud compute instances delete essentials --zone=$ZONE --quiet" - ] - }, - { - "cell_type": "markdown", - "id": "4a215b73-ac5e-4fa6-88b7-8af26ee4090f", - "metadata": {}, - "source": [ - "Check again" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "e53cb55c-3195-4fc3-9aab-cabad44d865e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n", - "//compute.googleapis.com/projects/essentials-project-335018\n" - ] - } - ], - "source": [ - "gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort" + "This will destroy the storage *Bucket* and and ALL the data in it! You cannot undelete a Bucket!\n", + "```\n", + "\n", + "```{admonition} Exercise\n", + "\n", + " * Try to delete the storage **bucket* on your own\n", + "```\n", + "\n", + "Navigate to the **Cloud Storage** service and in the **browser** page and remove the *essentials* Bucket." ] }, { "cell_type": "markdown", - "id": "631a08af-88bb-463c-b388-8993021ee216", + "id": "6753b415-1fa9-4716-8c41-8d98228af51c", "metadata": {}, "source": [ - "Success! The only resource left is the project (both are project resources)" + "## Verify Cleanup\n", + "\n", + "```{admonition} Exercise\n", + "\n", + " * There are at least three ways to verify resource removal, what are they?\n", + " * Try to verify that the resources were removed.\n", + "```\n", + "\n", + "```{admonition} Tip\n", + ":class: tip\n", + "You can also quickly navigate to resource page for most services from the **Resources** tab on the project **Dashboard**\n", + "```\n", + "\n", + "To verify that the resources have been deleted do the following:\n", + " * Navigate to the **VM Instances** page and **Cloud Storage** **Browser** to verify the resources have been deleted.\n", + " * Navigate to the **IAM & Admin** service and in the **Asset Inventory** page to verify that the resources have been deleted.\n", + " * Navigate to the **Activity** page and verify the deletion events. **Note:** the information collected in the asset inventory is often delayed.\n", + "\n", + "```{admonition} Tip\n", + ":class: tip\n", + "\n", + "If this is your own account, remember to visit the Billing Report page that you bookmarked daily.\n", + "```\n", + "\n", + "```{admonition} Tip\n", + ":class: tip\n", + "\n", + "You can also create, list, and destroy resources easily from the cli with the `gcloud` and `gsutil` commands.\n", + "```" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cac28557-fec2-43f4-aaa5-4f82b2257c55", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": {