Monitoring Costs#

Overview

Teaching: 10 min

Exercises: none

Questions:

  • How do I find out how much I have spent?

  • How do I find what is running and stored in my project?

Objectives:

  • Learn about Billing Accounts

  • Find information about the Billing Account associated with your project

  • Determine the permissions you have on the Billing Account

  • Find and monitor your spend with billing reports

  • Find where to set Billing Alerts.

  • Find all resources in a Project

  • Extra: List assets with the CLI

  • Extra: Understand how labels are used

Billing#

Projects and Billing Accounts are two different things in Google Cloud. In order for a Project operate it must have a Billing Account to charge any expenses to. Even if a project is utilizing credits it must have a Billing Account associated with it. Projects within an organization will often have the Billing Account controlled directly by the organization.

To find out more about the Billing Account associated with your project do the following:

  • Click Google Cloud (this will ensure your project is selected)

    • Double check that the Project ID and name is correct.

  • Select Navigation Menu -> Billing

    • If you get a “You have multiple billing accounts” select “Go to linked billing account”

    • If you get a “This project has no billing account” you cannot incur any charges on this account. Since you do not have a billing account you have no charges to monitor or set alerts on. You will need to associate or create a billing account to do anything.

  • Select Account Management on the Left Sidebar of the Billing page.

    • Note down the “Billing Account ID” near the top left of the billing page It should be in the form 01ABC2-A0B34C-DE5F6. This information should not be made public.

    • Open the right sidebar Info Panel if closed. To do so click Show Info Panel button on top right under the blue bar (don’t click the “Hide Info Panel” button!) billing-management

      • If you get “you don’t have permission to view the permissions of the selected resource” in the side bar you have no privileges and cannot set Billing Alerts.

      • If you have permission to view permissions (you did not get the previous message):

        • Ensure Show inherited permissions is “on” (blue and to the right)

        • Expand the all the “arrows” on panel (for example “Billing Account Administrator”) for each Role.

        • When you see your Account/User (Member) in a section you have that permission.

        • Check if you have Billing Account Administrator or Billing Account Cost Manager Role you have the permission (Role) to set alerts.

The Billing Account Administrator Role on the Billing Account gives you full control over the Billing Account. The Billing Account Cost Manager Role allows you to view information and set Billing Alerts.

Billing Reports#

To monitor spending we look at the billing reports and configure to to see the information we need. Specifically we will turn off discounts, promotions, and credits to see activity, even if we are not billed for it. We will also switch to the bar-chart to see easily spikes and constant expenses.

  • Navigate to Billing -> Billing Account -> Reports (see billing section)

  • Get detailed reports for the past 30 days.

    • In the filters on the Right Sidebar (open if necessary)

      • Time Range “Last 30 days”

      • Group By “Sku”

      • Projects: All or your current project (should be already selected)

      • All Services (should be already selected)

      • All SKUs (should be already selected)

    • In the main chart change options on the top of the graph to

      • “Daily”

      • Bar chart (bar chart graph on the right, “Daily” is on the left)

    • Save view

      • If you cannot save the view you can also click “Share” and use the URL to create a bookmark for easy access.

billing-report

This is the most reliable and direct way to monitor costs and these charts should be monitored daily for active projects and weekly for ongoing projects. For projects that are inactive the billing should be disabled or the project should be deleted.

There reports do not show consumption in real time. In order to get an idea of what is generating costs (and you can estimate these costs based on past experience), later we will show how to enumerate running resources.

Billing Alerts#

Due to the diversity in the way Billing Accounts are setup at institutions we will not show you have to setup billing alerts. They are simple and straight forward to setup once you have access and navigate to it. If you do not have access to Billing Alerts you can accomplish similar results by navigating to the Billing Reports every day.

You can find the Billing Alerts under Billing -> Budgets & Alerts.

There are also 3rd party cost monitoring services that your institution may use that can set billing alerts, contact your research computing and data professional for more information.

Asset Inventory#

In order to get an overview of what is running and stored in your account and where we use the Asset Inventory page in the IAM Service. To see what is in your Project do the following:

  • Go to Navigation Menu -> IAM & Admin -> Asset Inventory page.

  • Filter important resources by navigating to overview -> Asset type: View More and selecting important resources (compute.Instance, compute.Disk, storage.Bucket) and click Apply on on the bottom. iam-asset-navigation

Use the Overview and Resource to learn what is running and where (in the world) it is running.

Enumerate All Resources (CLI - Extra)#

Before we clean up we should always make sure that we can see the resources we created and that after destroying them they are actually removed. This is the most powerful tool for managing cost as it gives a real-time indication on what is running.

We can do this by listing the name of all resources (you can also get all the details for every resource). For more information on how to filter this information see gcloud asset search-all-resources --help. Note: The resource list updates are often delayed.

First, we just list every single resource and sort it.

gcloud asset search-all-resources --format='value(name)' | sort

//cloudresourcemanager.googleapis.com/projects/essentials-project-335018
//compute.googleapis.com/projects/essentials-project-335018
//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-icmp
//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-internal
//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-rdp
//compute.googleapis.com/projects/essentials-project-335018/global/firewalls/default-allow-ssh
//compute.googleapis.com/projects/essentials-project-335018/global/networks/default
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-026a26ff3e7b8291
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-0de54427d8115731
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-25e455ae2742fadd
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-29f6f8e5afe7eec3
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-2e031a808fb5c309
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-2f64262dcf6c2b01
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-35affd8ef359d282
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-3e179d55aa7a9312
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-4d0f16a3e97fd5ff
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-652acc5329dbfeae
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-6a75a97222655f3a
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-7d67fd6f0db9e78d
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-92f4928b25f3339f
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-93f19fa060dd07b0
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-951072b4e7869d60
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-9647a69124af3a9d
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-9aad70cb0a157c15
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-a1756848ee4c91f9
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-a8406cb97cda576b
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-bc1a73e144b265a1
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-beb7cf2071614bc7
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-d6d7479aa3edfbaa
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e1998a2a25002f09
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e2db952ee9e05045
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e8baa00ef9d1f1ec
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-e97a548861a5bb5d
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-efb583a306f2e63e
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-f534dbfe59970bd5
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-fc29d50e7bb3cad4
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-fc66c490b29216e2
//compute.googleapis.com/projects/essentials-project-335018/global/routes/default-route-fcea5cc3384e3a2c
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-east1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-east2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-northeast1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-northeast2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-northeast3/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-south1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-south2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-southeast1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/asia-southeast2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/australia-southeast1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/australia-southeast2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/europe-central2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/europe-north1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west3/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west4/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/europe-west6/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/northamerica-northeast1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/northamerica-northeast2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/southamerica-east1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/southamerica-west1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-central1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-east1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-east4/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-east7/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-west1/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-west2/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-west3/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/regions/us-west4/subnetworks/default
//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials
//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials
//iam.googleapis.com/projects/essentials-project-335018/serviceAccounts/281138444384-compute@developer.gserviceaccount.com
//serviceusage.googleapis.com/projects/281138444384/services/bigquery.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/bigquerystorage.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/cloudapis.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/cloudasset.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/clouddebugger.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/cloudtrace.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/compute.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/datastore.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/logging.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/monitoring.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/oslogin.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/servicemanagement.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/serviceusage.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/sql-component.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/storage-api.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/storage-component.googleapis.com
//serviceusage.googleapis.com/projects/281138444384/services/storage.googleapis.com
//storage.googleapis.com/essentials-learner-2021-12-16

This is a lot information and most of it (services, networks, routes etc.) is at no or minimal charge. We can also list by resource “Type”. Just remember, as you use a new service to add the corresponding resource type and verify that it is listed here. It is important to always ensure that you see a resource before you remove it so you can use the command (stored in a script) in the future.

gcloud asset search-all-resources --asset-types='compute.googleapis.com/Disk,compute.googleapis.com/Instance,storage.googleapis.com/Bucket' --format='value(name)'

//storage.googleapis.com/essentials-learner-2021-12-16
//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials
//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials

We can also approach the problem from another direction by removing known common resources that have little or no cost.

gcloud asset search-all-resources --format='value(name)' |grep -vE '/(services|networks|routes|subnetworks|firewalls|serviceAccounts)/' | sort

//cloudresourcemanager.googleapis.com/projects/essentials-project-335018
//compute.googleapis.com/projects/essentials-project-335018
//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/disks/essentials
//compute.googleapis.com/projects/essentials-project-335018/zones/us-west2-c/instances/essentials
//storage.googleapis.com/essentials-learner-2021-12-16

Now we can see that the list of resources is what we expect. Of interest here is the “disk” associated with the virtual machine. If these are not removed as well they can run up costs if you don’t know where to look.

Enumerate Service Resources (CLI - Extra)#

To get up-to-date information about a service the gcloud command can be used to list allocated resources. It is good to write a simple script to show all the resources in all the services you use. This, combined with the gcloud asset command is a powerful tool for

gcloud compute instances list

NAME        ZONE        MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
essentials  us-west2-c  e2-highcpu-4  true         10.168.0.4   34.94.174.128  RUNNING

gcloud compute disks list

NAME        LOCATION    LOCATION_SCOPE  SIZE_GB  TYPE         STATUS
essentials  us-west2-c  zone            10       pd-standard  READY

gcloud alpha storage ls

gs://essentials-learner-2021-12-16/

Label Resources (CLI - Extra)#

In order to track resources it is important to Label them. Labeling places key-value pairs (key=value) in the resources metadata (information about the resource) and most of the Google Cloud tools can use this information to search, filter, and group information. In the literature, the example most used is the “env” Label and indicates if the resource is in development (dev), testing (test), or production (prod); the lables would be env=test, env=dev and env=prod, respectively.

For research and teaching, Labels could be used to track funding, grants, students, or other uses of resources. For large efforts (projects) it is better to use different Projects to separate access and billing and protect the projects from each other.

Pro Tip

Labeling is called Tagging on AWS and Azure - tagging in Google Cloud is related to firewalls).

In this Lesson we use the cce Label to identify resources associated with this Lesson. First, we label our bucket:

gsutil label ch -l cce:data "gs://$BUCKET"

Setting label configuration on gs://cce-data-student31-2021-08-25/...

gsutil ls -L -b "gs://$BUCKET"

gs://cce-data-student31-2021-08-25/ :
	Storage class:			STANDARD
	Location type:			region
	Location constraint:		US-CENTRAL1
	Versioning enabled:		None
	Logging configuration:		None
	Website configuration:		None
	CORS configuration: 		None
	Lifecycle configuration:	None
	Requester Pays enabled:		None
	Labels:				
	  {
	    "cce": "data"
	  }
	Default KMS key:		None
	Time created:			Wed, 25 Aug 2021 18:37:25 GMT
	Time updated:			Wed, 25 Aug 2021 20:49:49 GMT
	Metageneration:			10
	Bucket Policy Only enabled:	True
	Public access prevention:	enforced
	ACL:				[]
	Default ACL:			[]

Now we can use it to filter our report

gcloud asset search-all-resources --query=labels.cce:data

---
assetType: storage.googleapis.com/Bucket
createTime: '2021-08-25T18:37:25Z'
displayName: cce-data-student31-2021-08-25
labels:
  cce: data
location: us-central1
name: //storage.googleapis.com/cce-data-student31-2021-08-25
parentAssetType: cloudresourcemanager.googleapis.com/Project
parentFullResourceName: //cloudresourcemanager.googleapis.com/projects/just-armor-301114
project: projects/1002111293252
updateTime: '2021-08-25T20:49:49Z'