Skip to content
This repository was archived by the owner on Dec 12, 2025. It is now read-only.

Commit

Permalink
GCP: Make copy/paste of config easy.
Browse files Browse the repository at this point in the history
  • Loading branch information
@tmiddelkoop
tmiddelkoop committed May 2, 2022
1 parent 46f525e commit 5fdc9cb
Show file tree
Hide file tree
Showing 3 changed files with 113 additions and 66 deletions.
2 changes: 1 addition & 1 deletion content/GCP/03_intro_to_cloud_storage.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
"id": "9897048a-6aa8-4d85-a557-d85b802f3f1d",
"metadata": {},
"source": [
"There are many storage services in the cloud. One of the most common and economic ways to store data in the cloud is to use object storage. In GCP object storage is called *Google Cloud Storage*, which is similar to the Simple Storage Service, also known as S3, on Amazon Web Services (AWS). For object storage, information is stored as a collection of key-value pairs. This is different to how data is commonly stored on laptops and high performance computing clusters (supercomputers)."
"There are many storage products and services in the cloud. One of the most common and economic ways to store data in the cloud is to use object storage. In GCP object storage is called *Google Cloud Storage*, which is similar to the Simple Storage Service, also known as S3, on Amazon Web Services (AWS). For object storage, information is stored as a collection of key-value pairs. This is different to how data is commonly stored on laptops and high performance computing clusters (supercomputers)."
]
},
{
Expand Down
85 changes: 85 additions & 0 deletions content/GCP/04_intro_to_cli.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,91 @@
"```"
]
},
{
"cell_type": "markdown",
"id": "4ba1b3a1-15fc-40f3-ad7e-f260a85ce0a3",
"metadata": {},
"source": [
"### Naming Resources\n",
"\n",
"Many times we will create resources (which have names) that we will need to use later. In the next episode we will be creating a *Cloud Storage* bucket (a resource). Bucket names are globally unique, so here we use \"essentials\" with your Cloud Shell username (the username part of the Google Account) as a prefix to name the bucket something unique and easy to understand where it came from."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "247a515d-04ed-4dd1-bd7b-aa52df948ce8",
"metadata": {},
"outputs": [],
"source": [
"export BUCKET=\"essentials-$USER-$(date +%F)\""
]
},
{
"cell_type": "markdown",
"id": "17a58166-e9e2-419d-9a3c-5ce746e0bf8e",
"metadata": {},
"source": [
"As usual, we will verify that it was set correctly."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "13a99343-4de3-436b-9f45-6b4f4eff4895",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"essentials-learner-2022-05-02\n"
]
}
],
"source": [
"echo $BUCKET"
]
},
{
"cell_type": "markdown",
"id": "bde1ca1b-5904-4aa2-92ac-a31185d42d35",
"metadata": {},
"source": [
"## Environment Configuration\n",
"\n",
"We will use what we just learned to get ready for the next episode by configuring the environment in one step. These commands can be included in your scripts, however, we will be simply copy and pasting them in the Cloud Shell for the remainder of the lesson."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "95ac29b1-2bf0-49e8-a409-dff8f77b754f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Account: 998995103712-compute@developer.gserviceaccount.com Project: essentials-learner Bucket: essentials-learner-2022-05-02\n"
]
}
],
"source": [
"PROJECT=$(gcloud config get-value project)\n",
"ACCOUNT=$(gcloud config get-value account)\n",
"BUCKET=\"essentials-$USER-$(date +%F)\"\n",
"echo \"Account: $ACCOUNT Project: $PROJECT Bucket: $BUCKET\""
]
},
{
"cell_type": "markdown",
"id": "24cf2c5f-742c-4b00-9f3a-05251a8889fb",
"metadata": {},
"source": [
"You must verify the output since this will be the \"Who, Where, and What\" of what you are doing."
]
},
{
"cell_type": "markdown",
"id": "bda44ee1-337a-447d-8377-e8f7ae563316",
Expand Down
92 changes: 27 additions & 65 deletions content/GCP/05_cli_storage.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
"source": [
"## Configuration\n",
"\n",
"It is important to verify the Account and Project information every time you start interacting with the cloud. We also use this opportunity to set the configuration environment variables (`PROJECT`) for the Episode.\n"
"We will use what we learned in the previous episode to set the configuration by entering the commands below. Remember, it is important to verify the Account and Project information every time you start interacting with the cloud. "
]
},
{
Expand All @@ -53,70 +53,27 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Your active configuration is: [cloudshell-27767]\n",
"learner@class.internet2.edu\n"
"Account: 998995103712-compute@developer.gserviceaccount.com Project: essentials-learner Bucket: essentials-learner-2022-05-02\n"
]
}
],
"source": [
"gcloud config get-value account"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9db0f3ca-2402-4002-b9d3-e634580e8f7f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your active configuration is: [cloudshell-27767]\n",
"essentials-learner\n"
]
}
],
"source": [
"gcloud config get-value project"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "44f0b255-508a-4488-abde-87aae67cd5b3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your active configuration is: [cloudshell-27767]\n",
"learner@class.internet2.edu\n"
]
}
],
"source": [
"ACCOUNT=$(gcloud config get-value account) ; echo $ACCOUNT"
"PROJECT=$(gcloud config get-value project)\n",
"ACCOUNT=$(gcloud config get-value account)\n",
"BUCKET=\"essentials-$USER-$(date +%F)\"\n",
"echo \"Account: $ACCOUNT Project: $PROJECT Bucket: $BUCKET\""
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "36fcc48b-8a5b-4158-9704-f1ea7c9951eb",
"cell_type": "markdown",
"id": "0727548c-bb05-4664-8ed5-cc33ba1b8021",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your active configuration is: [cloudshell-27767]\n",
"essentials-learner\n"
]
}
],
"source": [
"PROJECT=$(gcloud config get-value project) ; echo $PROJECT"
"```{admonition} Top\n",
":class: tip\n",
"\n",
"If you need to setup a new Cloud Shell you can copy and paste the above commands into the new shell.\n",
"```"
]
},
{
Expand All @@ -126,36 +83,43 @@
"source": [
"## Create a Bucket\n",
"\n",
"We will first make a new bucket (mb) with mostly default values. We will explore the newly created bucket with default values then immediately destroy it. This is a typical pattern when learning a new service by reducing chances to make a mistake (errors can be hard to understand). In most cases resources should not be created with all default values. We will re-create the bucket in the next section with more sensible values. "
"We will first make a new bucket (mb) with the default values. We will explore the newly created bucket with default values then immediately destroy it. This is a typical pattern when learning a new product by reducing chances to make a mistake (errors can be hard to understand). In most cases resources should not be created with all default values. We will re-create the bucket in the next section with more sensible values. "
]
},
{
"cell_type": "markdown",
"id": "658e1324",
"metadata": {},
"source": [
"We first generate a bucket name and store it in an environment variable for future use. Bucket names are globally unique, so here we use \"essentials\" with your Cloud Shell username (the username part of the Google Account) as a prefix to name the bucket something unique and easy to understand where it came from."
"We first verify that `$BUCKET` is set correctly."
]
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 2,
"id": "255eb914-a06a-4bfd-adbb-7ca2f94678e7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"essentials-learner-2022-02-03\n"
"essentials-learner-2022-05-02\n"
]
}
],
"source": [
"BUCKET=\"essentials-$USER-$(date +%F)\"\n",
"echo $BUCKET"
]
},
{
"cell_type": "markdown",
"id": "9a5789df-ed6b-44a5-8f06-7937c62d6035",
"metadata": {},
"source": [
"And now we create the bucket. Note that most times we must use `gs://` to indicate that it is a bucket. "
]
},
{
"cell_type": "code",
"execution_count": 6,
Expand All @@ -179,9 +143,7 @@
"id": "42584633-f77f-43b4-bb23-49ae4364af07",
"metadata": {},
"source": [
"If the bucket creation fails with a \"ServiceException: 409 A Cloud Storage bucket named ... already exists.\" it means that the name is not unique or you have already created the bucket. You will need to either delete the bucket or choose another name.\n",
"\n",
"*Advanced Note: Bucket creation may fail if you are not using the Cloud Shell and the machine you are using has generic account name such as `pi`, `admin`, or other standard username. If this is the case just set `BUCKET` to some unique value.*"
"If the bucket creation fails with a \"ServiceException: 409 A Cloud Storage bucket named ... already exists.\" it means that the name is not unique or you have already created the bucket. You will need to either delete the bucket or choose another name."
]
},
{
Expand All @@ -191,7 +153,7 @@
"source": [
"## Show the Bucket\n",
"\n",
"We will now list (Enumerate) the resource to verify that the resource (bucket) has been created by using the `gsutil ls` command to list all the buckets."
"We will now list all the buckets (resources) to verify that the bucket has been created by using the `gsutil ls` command."
]
},
{
Expand Down

0 comments on commit 5fdc9cb

Please sign in to comment.