diff --git a/content/GCP/05_cli_storage.ipynb b/content/GCP/05_cli_storage.ipynb index 02ca086..948f9ca 100644 --- a/content/GCP/05_cli_storage.ipynb +++ b/content/GCP/05_cli_storage.ipynb @@ -401,12 +401,437 @@ "gcloud logging read --limit 1" ] }, + { + "cell_type": "markdown", + "id": "fcdf7049-620f-4721-bc62-fa8f7fe97dcb", + "metadata": {}, + "source": [ + "## Use the Bucket\n", + "\n", + "Once the bucket is created, we can now store objects in them and work with them. It is possible to copy objects (files, strings, etc) in and out of the bucket using the `gsutil cp` command. The bucket can take objects from the command line or local files" + ] + }, + { + "cell_type": "markdown", + "id": "2493ad10-e552-4c3c-87c2-73f68a2675b2", + "metadata": {}, + "source": [ + "First create a simple file called `one.txt` with the contents \"test one\"" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "f9a90752-871f-4156-b122-4c46fd4bfa43", + "metadata": {}, + "outputs": [], + "source": [ + "echo \"test one\" > one.txt" + ] + }, + { + "cell_type": "markdown", + "id": "0860eebf-8dba-446d-abd3-56bd4957cfb9", + "metadata": {}, + "source": [ + "Display the contents of the file by using the `cat` command." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "bc0fb778-cc2e-4af4-8aa8-ca90d6b4f877", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "test one\n" + ] + } + ], + "source": [ + "cat one.txt" + ] + }, + { + "cell_type": "markdown", + "id": "8fbac238-e2a0-4bab-aede-eea051ce611b", + "metadata": {}, + "source": [ + "Copy the file `one.txt` into the bucket as object \"1\"." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "6384974b-d422-4ac1-90b7-cf9feec9aa8c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying file://one.txt [Content-Type=text/plain]...\n", + "/ [1 files][ 9.0 B/ 9.0 B] \n", + "Operation completed over 1 objects/9.0 B. \n" + ] + } + ], + "source": [ + "gsutil cp one.txt \"gs://$BUCKET/1\"" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "49589119-56bb-4c7b-b0a6-36e8f1133056", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-student31-2021-10-26/1\n" + ] + } + ], + "source": [ + "gsutil ls \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "ae542a46-f31d-4998-9bb5-29b8451cf941", + "metadata": {}, + "source": [ + "Copy the object to the standard out (displays it on the screen)." + ] + }, + { + "cell_type": "markdown", + "id": "ef366240-1152-4b30-b99f-b7835f63788d", + "metadata": {}, + "source": [ + "Remove the orignal file to make sure we are actually seeing the object in the bucket, not the local file." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "442cc51f-8856-41d4-a3a6-072c8569beef", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "removed 'one.txt'\n" + ] + } + ], + "source": [ + "rm -v one.txt" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "4e3c0de9-8a23-428a-9689-0cf896228afe", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "test one\n" + ] + } + ], + "source": [ + "gsutil cat \"gs://$BUCKET/1\"" + ] + }, + { + "cell_type": "markdown", + "id": "5727a33c-7142-45a9-acc2-3c372aa012ea", + "metadata": {}, + "source": [ + "We can also copy an object to a local file." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "13f94b36-7dea-46b9-9e8b-baa8b4e59de6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying gs://essentials-student31-2021-10-26/1...\n", + "/ [1 files][ 9.0 B/ 9.0 B] \n", + "Operation completed over 1 objects/9.0 B. \n" + ] + } + ], + "source": [ + "gsutil cp \"gs://$BUCKET/1\" one.txt" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "3982f21c-b827-4772-aa8c-3ac67e478b0d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "test one\n" + ] + } + ], + "source": [ + "cat one.txt" + ] + }, + { + "cell_type": "markdown", + "id": "b0badb1c-2a44-43bb-9135-399e1ae78b16", + "metadata": {}, + "source": [ + "Remove the local copy." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "6b57a3fc-94ea-4c25-b7ac-2744d7b28852", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "removed 'one.txt'\n" + ] + } + ], + "source": [ + "rm -v one.txt" + ] + }, + { + "cell_type": "markdown", + "id": "6cb774c1-dd47-4a8f-9f2b-8d527deec3c5", + "metadata": {}, + "source": [ + "We can also copy the output of a command directly into an object in a bucket. Since output of the `date` command changes we can be sure that we are not seeing old data in an object." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "5a7fc453-fca7-4ab4-8f0a-cb4de53bc992", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Copying from ...\n", + "/ [1 files][ 0.0 B/ 0.0 B] \n", + "Operation completed over 1 objects. \n" + ] + } + ], + "source": [ + "date | gsutil cp - \"gs://$BUCKET/2\"" + ] + }, + { + "cell_type": "markdown", + "id": "f28ee999-67f6-4079-9a4f-9f1b31b33e81", + "metadata": {}, + "source": [ + "List all the objects in the buckets and verify the contents of object \"2\"." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "1be58d86-5de7-420c-a3b3-0b82839ac90f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-student31-2021-10-26/1\n", + "gs://essentials-student31-2021-10-26/2\n" + ] + } + ], + "source": [ + "gsutil ls \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "1427e837-f65f-4d35-8088-6a82677b5ed6", + "metadata": {}, + "source": [ + "Exercise: display the date in the bucket." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "e3d0585b-4d81-4429-ad90-b3ff51eec13d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Tue 26 Oct 2021 07:49:59 PM UTC\n" + ] + } + ], + "source": [ + "gsutil cat \"gs://$BUCKET/2\"" + ] + }, + { + "cell_type": "markdown", + "id": "2ad8bdca-759c-40a6-b505-accc91b2ac12", + "metadata": {}, + "source": [ + "Objects can also be removed from buckets. We will remove object \"1\" from the Bucket." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "9ad460d9-7a4f-4e3f-befb-2bbb10c51993", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Removing gs://essentials-student31-2021-10-26/1...\n", + "/ [1 objects] \n", + "Operation completed over 1 objects. \n" + ] + } + ], + "source": [ + "gsutil rm \"gs://$BUCKET/1\"" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "8fd782f2-700a-4862-970b-1b4751009d5f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gs://essentials-student31-2021-10-26/2\n" + ] + } + ], + "source": [ + "gsutil ls \"gs://$BUCKET\"" + ] + }, + { + "cell_type": "markdown", + "id": "cf5d3942-8266-446a-b873-faaa593ac1c4", + "metadata": {}, + "source": [ + "Try to remove the bucket." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "e1350695-0f01-4205-a604-91d62896783e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Removing gs://essentials-student31-2021-10-26/...\n", + "NotEmptyException: 409 BucketNotEmpty (essentials-student31-2021-10-26)\n" + ] + } + ], + "source": [ + "gsutil rb \"gs://$BUCKET\"\n", + "/bin/true # ignore expected error in Jupyter" + ] + }, + { + "cell_type": "markdown", + "id": "6191aa2f-3fc4-4b34-83fd-8b62d845e844", + "metadata": {}, + "source": [ + "Buckets must be empty before they can be deleted (just like subdirectories). (The `/bin/true` ignores the expected error in Jupter so Jupyter does not stop processing the entire notebook) " + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "ba2671f8-521f-48b4-9231-491d128ba67c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Removing gs://essentials-student31-2021-10-26/2...\n", + "/ [1 objects] \n", + "Operation completed over 1 objects. \n" + ] + } + ], + "source": [ + "gsutil rm \"gs://$BUCKET/2\"" + ] + }, + { + "cell_type": "markdown", + "id": "9540ba16-1662-49b5-9bf7-87c5c49b1e2c", + "metadata": {}, + "source": [ + "Finally we can remove the bucket. We first double check the value of `$BUCKET`" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "73d8b299-8e8e-443e-ae5b-4ecf7efa0442", + "metadata": {}, + "outputs": [], + "source": [ + "gsutil ls" + ] + }, { "cell_type": "markdown", "id": "a322d55d-5f1c-4ca9-8aec-ec6a4579cea1", "metadata": {}, "source": [ - "## Remove a Bucket\n", + "## Remove the Bucket\n", "Since we are done exploring the bucket we will remove the bucket (rb). This is a common pattern in cloud computing, to remove resources once we are done with them otherwise they will just sit around incurring costs." ] }, @@ -489,6 +914,14 @@ "source": [ "gsutil ls" ] + }, + { + "cell_type": "markdown", + "id": "42c535e3-ba9e-4537-85bd-5ad6ee6274ff", + "metadata": {}, + "source": [ + "In this case the empty response indicates that there are no Buckets in the project." + ] } ], "metadata": {