diff --git a/content/GCP/05_intro_to_cloud_storage.ipynb b/content/GCP/03_intro_to_cloud_storage.ipynb similarity index 100% rename from content/GCP/05_intro_to_cloud_storage.ipynb rename to content/GCP/03_intro_to_cloud_storage.ipynb diff --git a/content/GCP/05_cli_storage.ipynb b/content/GCP/05_cli_storage.ipynb new file mode 100644 index 0000000..bef8efe --- /dev/null +++ b/content/GCP/05_cli_storage.ipynb @@ -0,0 +1,517 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c73369d2-8d8c-4764-bcab-cf8a0ec71e57", + "metadata": {}, + "source": [ + "## Managing Cloud Storage from the Command Line\n", + "\n", + "Learner Questions\n", + " * How do I store data in a Bucket?\n", + "\n", + "Learning Objectives\n", + " * Manage buckets using the command line.\n", + " * Create a bucket with a sensible configuration. \n" + ] + }, + { + "cell_type": "markdown", + "id": "3bcdd299-5d83-4bb6-b2bb-daa85b392a19", + "metadata": {}, + "source": [ + "Now that Drew understands how to create create buckets using the web console and to use the Google command line tools they are now going to explore managing buckets with the command line." + ] + }, + { + "cell_type": "markdown", + "id": "86da8dfa-44d6-45e7-8b43-36ef24626955", + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "It is important to verify the Account and Project information every time you start interacting with the cloud. We also use this opportunity to set the configuration environment variables (`PROJECT`) for the Episode.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "8e3ef8dc-e41b-45bb-8f5c-cba30add845b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Your active configuration is: [cloudshell-31923]\n", + "student31@class.internet2.edu\n" + ] + } + ], + "source": [ + "gcloud config get-value account" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "9db0f3ca-2402-4002-b9d3-e634580e8f7f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Your active configuration is: [cloudshell-31923]\n", + "just-armor-301114\n" + ] + } + ], + "source": [ + "gcloud config get-value project" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "36fcc48b-8a5b-4158-9704-f1ea7c9951eb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "just-armor-301114\n" + ] + } + ], + "source": [ + "PROJECT=$(gcloud config list --format='value(core.project)')\n", + "echo $PROJECT" + ] + }, + { + "cell_type": "markdown", + "id": "a8a166be-e959-497c-8ba0-44014442310e", + "metadata": {}, + "source": [ + "### Create a Bucket\n", + "\n", + "We will first make a new bucket (mb) with mostly default values. We will explore the newly created bucket with default values then immediately destroy it. This is a typical pattern when learning a new service by reducing chances to make a mistake (errors can be hard to understand). In most cases resources should not be created with all default values. We will re-create the bucket in the next section with more sensible values. " + ] + }, + { + "cell_type": "markdown", + "id": "f020bfcf-2e0e-4b8f-97f3-6b4753d95146", + "metadata": {}, + "source": [ + "We first generate a bucket name and store it in an environment variable for future use. Bucket names are globally unique, so here we use \"essentials\" with your Cloud Shell username (the username part of the Google Account) as as a prefix to name the bucket something unique and easy to understand where it came from." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "255eb914-a06a-4bfd-adbb-7ca2f94678e7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "essentials-student31-2021-10-26\n" + ] + } + ], + "source": [ + "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", + "echo $BUCKET" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "b0661b4c-b56b-4d6d-ba3c-e1c5f38699c6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating gs://essentials-student31-2021-10-26/...\n" + ] + } + ], + "source": [ + "gsutil mb \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "42584633-f77f-43b4-bb23-49ae4364af07", + "metadata": {}, + "source": [ + "If the bucket creation fails with a \"ServiceException: 409 A Cloud Storage bucket named ... already exists.\" it means that the name is not unique or you have already created the bucket. You will need to either delete the bucket or choose another name.\n", + "\n", + "*Advanced Note: Bucket creation may fail if you are not using the Cloud Shell and the machine you are using has generic account name such as `pi`, `admin`, or other standard username. If this is the case just set `BUCKET` to some unique value.*" + ] + }, + { + "cell_type": "markdown", + "id": "7cc6dd9a-bae4-4e59-8e7c-48007aaccd28", + "metadata": {}, + "source": [ + "### Show the Bucket\n", + "\n", + "We will now list (Enumerate) the resource to verify that the resource (bucket) has been created by using the `gsutil ls` command to list all the buckets." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "8ae2988a-fca8-4601-a7e5-458627fe68e3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-student31-2021-10-26/\n" + ] + } + ], + "source": [ + "gsutil ls" + ] + }, + { + "cell_type": "markdown", + "id": "7d50532e-1c0f-488d-8710-439a62617529", + "metadata": {}, + "source": [ + "In this case we see that there is only one bucket and it is the one we just created. Your project may contain other buckets." + ] + }, + { + "cell_type": "markdown", + "id": "00973c8c-b3bb-41ba-b3aa-9a425284c70a", + "metadata": {}, + "source": [ + "Just like the unix `ls` command, the `gsutil ls` command has a lot of (similar) options. Let's get detailed information about the bucket. Note we must specify the bucket by using the `-b` option to specify the bucket. Type `gsutil ls --help` for more information on the command and options." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "ecc0e4fe-c9d0-4edd-9a0c-56b5b9ec7c54", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-student31-2021-10-26/ :\n", + "\tStorage class:\t\t\tSTANDARD\n", + "\tLocation type:\t\t\tmulti-region\n", + "\tLocation constraint:\t\tUS\n", + "\tVersioning enabled:\t\tNone\n", + "\tLogging configuration:\t\tNone\n", + "\tWebsite configuration:\t\tNone\n", + "\tCORS configuration: \t\tNone\n", + "\tLifecycle configuration:\tNone\n", + "\tRequester Pays enabled:\t\tNone\n", + "\tLabels:\t\t\t\tNone\n", + "\tDefault KMS key:\t\tNone\n", + "\tTime created:\t\t\tTue, 26 Oct 2021 20:47:06 GMT\n", + "\tTime updated:\t\t\tTue, 26 Oct 2021 20:47:06 GMT\n", + "\tMetageneration:\t\t\t1\n", + "\tBucket Policy Only enabled:\tFalse\n", + "\tPublic access prevention:\tunspecified\n", + "\tRPO:\t\t\t\tDEFAULT\n", + "\tACL:\t\t\t\t\n", + "\t [\n", + "\t {\n", + "\t \"entity\": \"project-owners-1002111293252\",\n", + "\t \"projectTeam\": {\n", + "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"team\": \"owners\"\n", + "\t },\n", + "\t \"role\": \"OWNER\"\n", + "\t },\n", + "\t {\n", + "\t \"entity\": \"project-editors-1002111293252\",\n", + "\t \"projectTeam\": {\n", + "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"team\": \"editors\"\n", + "\t },\n", + "\t \"role\": \"OWNER\"\n", + "\t },\n", + "\t {\n", + "\t \"entity\": \"project-viewers-1002111293252\",\n", + "\t \"projectTeam\": {\n", + "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"team\": \"viewers\"\n", + "\t },\n", + "\t \"role\": \"READER\"\n", + "\t }\n", + "\t ]\n", + "\tDefault ACL:\t\t\t\n", + "\t [\n", + "\t {\n", + "\t \"entity\": \"project-owners-1002111293252\",\n", + "\t \"projectTeam\": {\n", + "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"team\": \"owners\"\n", + "\t },\n", + "\t \"role\": \"OWNER\"\n", + "\t },\n", + "\t {\n", + "\t \"entity\": \"project-editors-1002111293252\",\n", + "\t \"projectTeam\": {\n", + "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"team\": \"editors\"\n", + "\t },\n", + "\t \"role\": \"OWNER\"\n", + "\t },\n", + "\t {\n", + "\t \"entity\": \"project-viewers-1002111293252\",\n", + "\t \"projectTeam\": {\n", + "\t \"projectNumber\": \"1002111293252\",\n", + "\t \"team\": \"viewers\"\n", + "\t },\n", + "\t \"role\": \"READER\"\n", + "\t }\n", + "\t ]\n" + ] + } + ], + "source": [ + "gsutil ls -L -b \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "1e7fceeb-661e-4a42-8fa4-5c2c50503ea6", + "metadata": {}, + "source": [ + "You can see that the bucket is \"multi-region\" and uses the \"standard\" storage class (this the type of storage). The standard storage class is best used for frequently used buckets. Other storage classes are designed for less frequent use, are less expensive, but come with more restrictions and more complex access costs and should be only used after careful consideration.\n", + "\n", + "Ignore the \"ACL\" section for now.\n", + "\n", + "You may also want to verify that you can see the newly created bucket in the web console dashboard or \"Cloud Storage\" page and explore the properties here." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "8fc85600-a793-413b-b139-642e4cf08a8e", + "metadata": {}, + "outputs": [], + "source": [ + "sleep 2 # used to ensure that the next command runs correctly in Jupyter" + ] + }, + { + "cell_type": "markdown", + "id": "c9480ea5-41b3-4bde-ad01-3f290792bc0a", + "metadata": {}, + "source": [ + "### Bucket Activity\n", + "Next we will check/follow the Activity log to ensure that the bucket was created. This command assumes that there was no other activity in the account, you may need to increase the `--limit` value to something larger to find the activity. \n", + "\n", + "Activity logs are used track important project level activity (such as bucket creation and deletion) and can be used for security and tracking resources and cannot be deleted. This can be used to debug if something goes wrong or in a security audit or investigation.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "bb8e7270-2f64-4fbd-b312-b18a7960700f", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "---\n", + "insertId: -z8eaicef61t3\n", + "logName: projects/just-armor-301114/logs/cloudaudit.googleapis.com%2Factivity\n", + "protoPayload:\n", + " '@type': type.googleapis.com/google.cloud.audit.AuditLog\n", + " authenticationInfo:\n", + " principalEmail: student31@class.internet2.edu\n", + " authorizationInfo:\n", + " - granted: true\n", + " permission: storage.buckets.create\n", + " resource: projects/_/buckets/essentials-student31-2021-10-26\n", + " resourceAttributes: {}\n", + " methodName: storage.buckets.create\n", + " request:\n", + " defaultObjectAcl:\n", + " '@type': type.googleapis.com/google.iam.v1.Policy\n", + " bindings:\n", + " - members:\n", + " - projectViewer:just-armor-301114\n", + " role: roles/storage.legacyObjectReader\n", + " - members:\n", + " - projectOwner:just-armor-301114\n", + " - projectEditor:just-armor-301114\n", + " role: roles/storage.legacyObjectOwner\n", + " requestMetadata:\n", + " callerIp: 35.239.199.87\n", + " callerSuppliedUserAgent: apitools Python/3.7.3 gsutil/5.4 (linux) analytics/disabled\n", + " interactive/True command/mb google-cloud-sdk/361.0.0,gzip(gfe)\n", + " destinationAttributes: {}\n", + " requestAttributes:\n", + " auth: {}\n", + " time: '2021-10-26T20:47:05.790066575Z'\n", + " resourceLocation:\n", + " currentLocations:\n", + " - us\n", + " resourceName: projects/_/buckets/essentials-student31-2021-10-26\n", + " serviceData:\n", + " '@type': type.googleapis.com/google.iam.v1.logging.AuditData\n", + " policyDelta:\n", + " bindingDeltas:\n", + " - action: ADD\n", + " member: projectEditor:just-armor-301114\n", + " role: roles/storage.legacyBucketOwner\n", + " - action: ADD\n", + " member: projectOwner:just-armor-301114\n", + " role: roles/storage.legacyBucketOwner\n", + " - action: ADD\n", + " member: projectViewer:just-armor-301114\n", + " role: roles/storage.legacyBucketReader\n", + " serviceName: storage.googleapis.com\n", + " status: {}\n", + "receiveTimestamp: '2021-10-26T20:47:06.793562424Z'\n", + "resource:\n", + " labels:\n", + " bucket_name: essentials-student31-2021-10-26\n", + " location: us\n", + " project_id: just-armor-301114\n", + " type: gcs_bucket\n", + "severity: NOTICE\n", + "timestamp: '2021-10-26T20:47:05.785456012Z'\n" + ] + } + ], + "source": [ + "gcloud logging read --limit 1" + ] + }, + { + "cell_type": "markdown", + "id": "a322d55d-5f1c-4ca9-8aec-ec6a4579cea1", + "metadata": {}, + "source": [ + "### Remove a Bucket\n", + "Since we are done exploring the bucket we will remove the bucket (rb). This is a common pattern in cloud computing, to remove resources once we are done with them otherwise they will just sit around incurring costs." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "7b32764d-e7b5-42d6-8f88-bc6936b2024a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-student31-2021-10-26/\n" + ] + } + ], + "source": [ + "gsutil ls" + ] + }, + { + "cell_type": "markdown", + "id": "b3ff49c2-01b5-4c1b-a3e9-100f55c2f774", + "metadata": {}, + "source": [ + "We first verify that our environment variable is set. This is a useful pattern to catch simple bugs faster and to aid in debugging." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "82487bd8-14f4-4811-8f5a-dd346042bd0f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Bucket: essentials-student31-2021-10-26\n" + ] + } + ], + "source": [ + "echo \"Bucket: $BUCKET\"" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "8635db94-2f3c-46d2-acc2-76a176fb37e7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Removing gs://essentials-student31-2021-10-26/...\n" + ] + } + ], + "source": [ + "gsutil rb \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "9002604e-6908-4a86-ab45-b9dd4cb11f75", + "metadata": {}, + "source": [ + "We verify that the bucket has been removed. In this example there is not output since there are no more buckets." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "ed6f0204-0de2-4d20-ab00-3b7a1023fd86", + "metadata": {}, + "outputs": [], + "source": [ + "gsutil ls" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "242645f3-5803-4579-a051-bb2fe88c0c11", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Bash", + "language": "bash", + "name": "bash" + }, + "language_info": { + "codemirror_mode": "shell", + "file_extension": ".sh", + "mimetype": "text/x-sh", + "name": "bash" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/content/GCP/intro_to_GCP_Essentials.ipynb b/content/GCP/intro_to_GCP_Essentials.ipynb index 4e9c5ab..2bd2d17 100644 --- a/content/GCP/intro_to_GCP_Essentials.ipynb +++ b/content/GCP/intro_to_GCP_Essentials.ipynb @@ -11,11 +11,12 @@ "\n", "1. [Introduction to the GCP Cloud Console](./01_intro_to_cloud_console)\n", "2. [Introduction to Cloud Compute](./02_intro_to_compute)\n", - "3. [Introduction to the Cloud CLI](./04_intro_to_cli)\n", - "4. [Introduction to Cloud Storage](./05_intro_to_cloud_storage)\n", - "5. [Running Analysis on the Cloud](./06_running_analysis)\n", - "6. [Monitoring Costs](./07_monitoring_costs)\n", - "7. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)\n" + "3. [Introduction to Cloud Storage](./03_intro_to_cloud_storage)\n", + "4. [Introduction to the Cloud CLI](./04_intro_to_cli)\n", + "4. [Using the Cloud Storage CLI](./05_cli_storage)\n", + "6. [Running Analysis on the Cloud](./06_running_analysis)\n", + "7. [Monitoring Costs](./07_monitoring_costs)\n", + "8. [Cleaning up Resources and Best Practices](./08_cleaning_up_resources)\n" ] } ], diff --git a/content/_toc.yml b/content/_toc.yml index e771863..38a8fcb 100644 --- a/content/_toc.yml +++ b/content/_toc.yml @@ -37,13 +37,14 @@ parts: sections: - file: GCP/01_intro_to_cloud_console - file: GCP/02_intro_to_compute + - file: GCP/03_intro_to_cloud_storage - file: GCP/04_intro_to_cli - - file: GCP/05_intro_to_cloud_storage + - file: GCP/05_cli_storage - file: GCP/06_running_analysis - file: GCP/07_monitoring_costs - file: GCP/08_cleaning_up_resources - file: GCP/glossary - + - caption: Extra Learning Materials chapters: - file: ELM/01_bash_shell