Skip to content
This repository was archived by the owner on Dec 12, 2025. It is now read-only.

Commit

Permalink
Continue the cleanup and monitoring story
Browse files Browse the repository at this point in the history
  • Loading branch information
@tmiddelkoop
tmiddelkoop committed Dec 15, 2021
1 parent 59f030d commit a73e1e6
Show file tree
Hide file tree
Showing 2 changed files with 265 additions and 148 deletions.
192 changes: 188 additions & 4 deletions content/GCP/07_monitoring_costs.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,197 @@
"```"
]
},
{
"cell_type": "markdown",
"id": "cdfc0b3b-8bc3-44a3-a712-1005531c9d30",
"metadata": {},
"source": [
"## Enumerate Allocated Resources\n",
"\n",
"Before we clean up we should always make sure that we can see the resources we created and that after destroying them they are actually removed. This is the most powerful tool for managing cost as it gives a real-time indication on what is running.\n",
"\n",
"We can do this by listing the name of all resources (you can also get all the details for every resource). For more information on how to filter this information see `gcloud asset search-all-resources --help`. **Note: The resource list updates are often delayed.**"
]
},
{
"cell_type": "markdown",
"id": "18482bf1-c059-4946-b540-10022abab8c3",
"metadata": {},
"source": [
"First, we just list every single resource and sort it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1c5f78a5-ec20-46a8-9fc4-eb8fcc6c4482",
"execution_count": 4,
"id": "bee03b96-53f6-47c0-bcbf-0bf73fe99f73",
"metadata": {
"scrolled": true,
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n",
"//compute.googleapis.com/projects/essentials-project-335018\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-icmp\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-internal\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-rdp\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-ssh\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/networks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-026a26ff3e7b8291\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-0de54427d8115731\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-25e455ae2742fadd\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-29f6f8e5afe7eec3\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-2e031a808fb5c309\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-2f64262dcf6c2b01\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-35affd8ef359d282\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-3e179d55aa7a9312\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-4d0f16a3e97fd5ff\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-652acc5329dbfeae\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-6a75a97222655f3a\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-7d67fd6f0db9e78d\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-92f4928b25f3339f\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-93f19fa060dd07b0\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-951072b4e7869d60\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-9647a69124af3a9d\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-9aad70cb0a157c15\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-a1756848ee4c91f9\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-a8406cb97cda576b\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-bc1a73e144b265a1\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-beb7cf2071614bc7\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-d6d7479aa3edfbaa\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e1998a2a25002f09\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e2db952ee9e05045\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e8baa00ef9d1f1ec\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e97a548861a5bb5d\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-efb583a306f2e63e\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-f534dbfe59970bd5\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-fc29d50e7bb3cad4\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-fc66c490b29216e2\n",
"//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-fcea5cc3384e3a2c\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-east1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-east2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-northeast1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-northeast2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-northeast3/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-south1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-south2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-southeast1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/asia-southeast2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/australia-southeast1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/australia-southeast2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/europe-central2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/europe-north1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west3/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west4/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west6/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/northamerica-northeast1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/northamerica-northeast2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/southamerica-east1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/southamerica-west1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-central1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-east1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-east4/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-east7/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-west1/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-west2/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-west3/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/regions/us-west4/subnetworks/default\n",
"//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n",
"//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n",
"//iam.googleapis.com/projects/essentials-project-335018/serviceAccounts/281138444384-compute@developer.gserviceaccount.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/bigquery.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/bigquerystorage.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/cloudapis.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/cloudasset.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/clouddebugger.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/cloudtrace.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/compute.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/datastore.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/logging.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/monitoring.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/oslogin.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/servicemanagement.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/serviceusage.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/sql-component.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/storage-api.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/storage-component.googleapis.com\n",
"//serviceusage.googleapis.com/projects/281138444384/services/storage.googleapis.com\n"
]
}
],
"source": [
"gcloud asset search-all-resources --format='value(name)' | sort"
]
},
{
"cell_type": "markdown",
"id": "ee12bccd-5bcb-4b4f-8a31-9d0ae14b07c4",
"metadata": {},
"outputs": [],
"source": []
"source": [
"This is a lot information and most of it (services, networks, routes etc.) is at no or minimal charge. We can also list by resource \"Type\". Just remember, as you use a new service to add the corresponding resource type and verify that it is listed here. It is important to always ensure that you see a resource before you remove it so you can use the command (stored in a script) in the future."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "7ac47880-3760-4da0-a56d-8d59a0a51f24",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n",
"//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n"
]
}
],
"source": [
"gcloud asset search-all-resources --asset-types='compute.googleapis.com/Disk,compute.googleapis.com/Instance,storage.googleapis.com/Bucket' --format='value(name)'"
]
},
{
"cell_type": "markdown",
"id": "7902e352-b833-49d0-bdfc-959236bdd888",
"metadata": {},
"source": [
"We can also approach the problem from another direction by removing known common resources that have little or no cost."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "57caf876-c8b0-4263-bb96-19adcbba779a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"//cloudresourcemanager.googleapis.com/projects/essentials-project-335018\n",
"//compute.googleapis.com/projects/essentials-project-335018\n",
"//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials\n",
"//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials\n"
]
}
],
"source": [
"gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort"
]
},
{
"cell_type": "markdown",
"id": "d63531f1-160d-48b5-8aff-c7d92f97d34a",
"metadata": {},
"source": [
"Now we can see that the list of resources is what we expect. Of interest here is the \"disk\" associated with the virtual machine. If these are not removed as well they can run up costs if you don't know where to look."
]
}
],
"metadata": {
Expand Down
Loading

0 comments on commit a73e1e6

Please sign in to comment.