Introduction to the Command Line Interface (CLI)#

Overview

Teaching: 20 min

Exercises: 5 min

Questions:

  • How do I use the Cloud CLI?

  • How do I use gcloud?

Objectives:

  • Find and use the “Cloud Shell” in the web console.

  • Use basic cloud CLI commands (gcloud and gsutil).

  • Verify basic settings.

  • Use environment variables for configuration.

  • Understand the importance of using variables reproducibility and automation.

Resources

Cloud Automation#

Now that Drew can create a bucket in the web console they wish to learn how to use tools to automate the process. Drew noticed that the web console could be used to upload and share objects but wishes to learn the command line so that they can automate the and document the process (it is much easier to document commands then describing and showing mouse clicks).

The cloud can be controlled using a Command Line Interface (CLI) or a programming language such as Python. Collectively these tools interact with the cloud over a Application Programming Interface (API) and this capability forms the basis of the cloud, the ability to control infrastructure programmatically.

Just as with navigating the web console it is important to know the who, where, and what of CLI access to reduce the possibility of access mistakes. We will first verify the tools are installed and configured correctly. Next we get the Account being used (who) and the Project ID of the active project (where) using the gcloud command. We will then use the gcloud and gsutil commands to list some public Buckets (what).

The gcloud command is used to control most aspects of Google Cloud and the gsutil command is used to control Google Cloud Storage Buckets. To access the manual pages for a command just add --help to the end of the command or run gcloud help for more information.

When running CLI commands you may get a dialog box to authorize the command. You should “Authorize” the request to allow the Cloud Shell to access the Project using your (Google) Account.

Tip

You can access the cloud shell from your laptop via gcloud cloud-shell ssh if you have the Google Cloud SDK installed. Installing the Google Cloud SDK will also allow you to run gcloud and gsutil commands directly from your laptop, workstation, or cluster. Pro-tip, you cannot use a service account to access the cloud shell.

Start a Cloud Shell Session#

Open up a Cloud shell session by clicking on the terminal icon on the blue bar on the top right of the page (labeled 5 below).

blue-bar

You may wish to push the maximize button (hover over the icons to see the names) on the terminal to make it full screen and change the font size (found in the gear icon) to your liking.

Now test that the gcloud command works and you can get help. This is done by running the following:

gcloud help

Use the arrows, page up and page down (also space), to navigate the help screen and press q to exit the help screen. (Pressing h will give you more information about how to navigate)

Verify the Configuration (Who, Where, What)#

First, let’s verify that the Account being used for access (who) is what we expect.

gcloud config get-value account

Your active configuration is: [cloudshell-2071]
learner@class.internet2.edu

Next, show verify the active project (where).

gcloud config get-value project

Your active configuration is: [cloudshell-2071]
essentials-learner

We will first use the gsutil command, which is similar to gcloud but older but more comprehensive. We will first check to see if the command working by showing the help by issuing the following command:

gcloud help

Now we will use gsutil to list a well known public bucket (what). The gsutil command is how we access Google Cloud Storage, most other services use the gcloud command.

gsutil ls gs://gcp-public-data-landsat

gs://gcp-public-data-landsat/index.csv.gz
gs://gcp-public-data-landsat/LC08/
gs://gcp-public-data-landsat/LE07/
gs://gcp-public-data-landsat/LM01/
gs://gcp-public-data-landsat/LM02/
gs://gcp-public-data-landsat/LM03/
gs://gcp-public-data-landsat/LM04/
gs://gcp-public-data-landsat/LM05/
gs://gcp-public-data-landsat/LO08/
gs://gcp-public-data-landsat/LT04/
gs://gcp-public-data-landsat/LT05/
gs://gcp-public-data-landsat/LT08/

Environment Variables#

When using the CLI you quickly notice that you have to enter many of the details, such as the Project ID, many times (Drew is also Dyslexic, which makes it even harder for them). Not only is this tedious, it also makes moving to another project or using another bucket difficult making reusability and collaboration painful. We will demonstrate the use environment variables to address this issue. Most cloud environments and tools rely heavily on this technique.

To make the point even clearer, we use environment variables in the Cloud Essentials notebooks instead of hard coding them, which makes it easy for anyone to run, use, and edit and thus collaborate and contribute to them!

We now demonstrate the basic use environment variables. As a reminder, environment variables are used to store configuration information that can be easily passed between programs and programming languages.

Tip

Although we do not always need to, we should always enclose shell variables in double quotes and enclose the variable name in curly braces (funny things may happen otherwise) when writing scripts, for example "${TEST}". In most cases it is easier to do this every time than try to figure out when it is possible to use the short form. We only use the short form ($TEST) when it is used frequently and known to always work (most cases here), in our case it is $PROJECT and $BUCKET. When in doubt, use the explicit form.

TEST=1

echo $TEST

1

There are a number of environment variables already set that contain useful information, for example:

echo $USER

learner

echo $HOSTNAME

cs-194566722441-default

We can also use the output of commands to set values.

date

Thu 03 Feb 2022 09:50:31 PM UTC

echo "Date: $(date)"

Date: Thu 03 Feb 2022 09:50:31 PM UTC

We use double quotes to make it a single string.

Now lets combine all this to set the PROJECT environment variable to the default project so we do not need to use the actual value in future commands.

gcloud config get-value project

Your active configuration is: [cloudshell-2071]
essentials-learner

PROJECT=$(gcloud config get-value project)

Your active configuration is: [cloudshell-2071]

echo $PROJECT

essentials-learner

Note the “Your active configuration is: …” line is a “warning” and not actually part of the output of the program.

Advanced Tip

You can also use

PROJECT=$(gcloud config list --format='value(core.project)')

to get the project id without the warning.

Now use the PROJECT environment variable to get more information about the active project (you may be prompted to enable the API).

gcloud projects describe $PROJECT

createTime: '2022-01-14T19:03:26.486Z'
lifecycleState: ACTIVE
name: essentials-learner
parent:
  id: '1072231596131'
  type: folder
projectId: essentials-learner
projectNumber: '998995103712'

Exercise

  • What command shows the account? (hint: we used it already)

  • Set the ACCOUNT environment variable with the correct value. (hint: be careful with punctuation (single and double quotes) and matching brackets and punctuation)

Naming Resources#

Many times we will create resources (which have names) that we will need to use later. In the next episode we will be creating a Cloud Storage bucket (a resource). Bucket names are globally unique, so here we use “essentials” with your Cloud Shell username (the username part of the Google Account) as a prefix to name the bucket something unique and easy to understand where it came from.

export BUCKET="essentials-$USER-$(date +%F)"

As usual, we will verify that it was set correctly.

echo $BUCKET

essentials-learner-2022-05-02

Environment Configuration#

We will use what we just learned to get ready for the next episode by configuring the environment in one step. These commands can be included in your scripts, however, we will be simply copy and pasting them in the Cloud Shell for the remainder of the lesson.

PROJECT=$(gcloud config get-value project)
ACCOUNT=$(gcloud config get-value account)
BUCKET="essentials-$USER-$(date +%F)"
echo "Account: $ACCOUNT Project: $PROJECT Bucket: $BUCKET"

Account: 998995103712-compute@developer.gserviceaccount.com Project: essentials-learner Bucket: essentials-learner-2022-05-02

You must verify the output since this will be the “Who, Where, and What” of what you are doing.

Formating and Scripting Output (Optional)#

The following example shows how to output the configuration as JSON. The command jq can also be used to parse the JSON data in addition to the --format option.

gcloud projects describe $PROJECT --format=json | jq

{
  "createTime": "2022-01-14T19:03:26.486Z",
  "lifecycleState": "ACTIVE",
  "name": "essentials-learner",
  "parent": {
    "id": "1072231596131",
    "type": "folder"
  },
  "projectId": "essentials-learner",
  "projectNumber": "998995103712"
}

gcloud projects describe $PROJECT --format=json | jq -r .projectNumber

998995103712