Skip to content
This repository was archived by the owner on Dec 12, 2025. It is now read-only.

Commit

Permalink
Basic VM/code/Bucket usage
Browse files Browse the repository at this point in the history
  • Loading branch information
Timothy Middelkoop committed Nov 9, 2021
1 parent 951f032 commit 774dd53
Showing 1 changed file with 259 additions and 0 deletions.
259 changes: 259 additions & 0 deletions content/GCP/06_running_analysis.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,265 @@
"\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "c87e200c-1d18-44f5-b4e6-28b697b04b3c",
"metadata": {},
"source": [
"## Connect to the VM\n",
"\n",
"First login to the instance from the Cloud Shell\n",
"```\n",
"gcloud compute ssh instance-1\n",
"```\n",
"If prompted for a zone select `n` to find it automatically. You can see an example session below."
]
},
{
"cell_type": "markdown",
"id": "44f32772-0abb-4fa7-a013-202fc646df5e",
"metadata": {},
"source": [
"```\n",
"student31@cloudshell:~ (just-armor-301114)$ gcloud compute ssh instance-1\n",
"Did you mean zone [us-central1-b] for instance: [instance-1] (Y/n)? n\n",
"\n",
"No zone specified. Using zone [us-west2-c] for instance: [instance-1].\n",
"Linux instance-1 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64\n",
"\n",
"The programs included with the Debian GNU/Linux system are free software;\n",
"the exact distribution terms for each program are described in the\n",
"individual files in /usr/share/doc/*/copyright.\n",
"\n",
"Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent\n",
"permitted by applicable law.\n",
"Last login: Tue Nov 9 20:12:49 2021 from 34.133.99.196\n",
"student31@instance-1:~$\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "b7c4db9e-f098-41bc-80d2-b524444eec7f",
"metadata": {},
"source": [
"## Get Example Code\n",
"\n",
"We will now install `git` and use it to download the example code and change into the directory."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "05fadd30-4f35-4a93-b713-cc132131b948",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reading package lists... Done\n",
"Building dependency tree \n",
"Reading state information... Done\n",
"git is already the newest version (1:2.20.1-2+deb10u3).\n",
"0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.\n"
]
}
],
"source": [
"sudo apt install --yes git"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "36554c99-ba08-4733-8ef2-e68d42d0d2b7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cloning into 'CLASS-Examples'...\n",
"remote: Enumerating objects: 16, done.\u001b[K\n",
"remote: Counting objects: 100% (16/16), done.\u001b[K\n",
"remote: Compressing objects: 100% (13/13), done.\u001b[K\n",
"remote: Total 41 (delta 4), reused 15 (delta 3), pack-reused 25\u001b[K\n",
"Unpacking objects: 100% (41/41), done.\n"
]
}
],
"source": [
"git clone https://github.internet2.edu/CLASS/CLASS-Examples.git"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "90c1cda7-60d4-44bb-84f8-e776a77a94ab",
"metadata": {},
"outputs": [],
"source": [
"cd CLASS-Examples/landsat/"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "55b628d5-6e5c-45a5-9cd3-c129db9cdcd2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 24\n",
"-rw-r--r-- 1 student31 student31 72 Nov 9 20:18 clean.sh\n",
"-rw-r--r-- 1 student31 student31 256 Nov 9 20:18 download.sh\n",
"-rw-r--r-- 1 student31 student31 314 Nov 9 20:18 get-index.sh\n",
"-rw-r--r-- 1 student31 student31 841 Nov 9 20:18 ReadMe.md\n",
"-rw-r--r-- 1 student31 student31 110 Nov 9 20:18 search.json\n",
"-rw-r--r-- 1 student31 student31 1447 Nov 9 20:18 search.py\n"
]
}
],
"source": [
"ls -l"
]
},
{
"cell_type": "markdown",
"id": "2b8b3144-7453-4350-a1cf-7fa74af2bcbf",
"metadata": {},
"source": [
"## Access the bucket\n",
"\n",
"First test bucket access by listing the bucket"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "e56ab74a-ae6d-4602-a26b-4a2656bd40cd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"gs://gcp-public-data-landsat/index.csv.gz\n",
"gs://gcp-public-data-landsat/LC08/\n",
"gs://gcp-public-data-landsat/LE07/\n",
"gs://gcp-public-data-landsat/LM01/\n",
"gs://gcp-public-data-landsat/LM02/\n",
"gs://gcp-public-data-landsat/LM03/\n",
"gs://gcp-public-data-landsat/LM04/\n",
"gs://gcp-public-data-landsat/LM05/\n",
"gs://gcp-public-data-landsat/LO08/\n",
"gs://gcp-public-data-landsat/LT04/\n",
"gs://gcp-public-data-landsat/LT05/\n",
"gs://gcp-public-data-landsat/LT08/\n"
]
}
],
"source": [
"gsutil ls gs://gcp-public-data-landsat/"
]
},
{
"cell_type": "markdown",
"id": "89acffba-cbce-436c-98dd-05467b6675a6",
"metadata": {},
"source": [
"Get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "bbe85b75-c7cd-40ed-a3b0-37cbd0a5f52e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mkdir: created directory 'data'\n"
]
}
],
"source": [
"mkdir -v data"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "18a9b71c-5871-4ce2-a202-b48ad04e8d38",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Copying gs://gcp-public-data-landsat/index.csv.gz...\n",
"==> NOTE: You are downloading one or more large file(s), which would \n",
"run significantly faster if you enabled sliced object downloads. This\n",
"feature is enabled by default but requires that compiled crcmod be\n",
"installed (see \"gsutil help crcmod\").\n",
"\n",
"| [1 files][757.2 MiB/757.2 MiB] 737.1 KiB/s \n",
"Operation completed over 1 objects/757.2 MiB. \n"
]
}
],
"source": [
"gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "2cdaf24c-c4aa-4e80-9236-939e7c982916",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"gzip: data/index.csv: No space left on device\n"
]
},
{
"ename": "",
"evalue": "1",
"output_type": "error",
"traceback": []
}
],
"source": [
"gzip -d data/index.csv.gz"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b005876c-f9af-43d6-80c6-f04295413b9b",
"metadata": {},
"outputs": [],
"source": [
"ls -lh data"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bbb26cfb-4b0d-4458-bc24-13a069556c3a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand Down

0 comments on commit 774dd53

Please sign in to comment.