diff --git a/content/AWS/02_intro_to_compute_part1.ipynb b/content/AWS/02_intro_to_compute_part1.ipynb index 02795cd..b18baa6 100644 --- a/content/AWS/02_intro_to_compute_part1.ipynb +++ b/content/AWS/02_intro_to_compute_part1.ipynb @@ -5,7 +5,7 @@ "id": "dc57021c", "metadata": {}, "source": [ - "# Introduction to Elastic Cloud Compute (EC2)\n", + "# Introduction to Elastic Cloud Compute (EC2) - Part 1\n", "\n", "\n", "```{admonition} Overview\n", @@ -23,6 +23,7 @@ "**Objectives:**\n", "* Understand the concept of virtual servers.\n", "* Understand what an Elastic Cloud Compute (EC2) instance is.\n", + "* Understand how to launch an EC2 instance. \n", "\n", "```" ] @@ -124,7 +125,7 @@ "Select ```Next: Configure Instance Details```.\n", "\n", "## Step 3: Configure Instance Details\n", - "Step 3 in creating an EC2 instance involves a rudimentary understanding of several key AWS and cloud jargon (Figure 4). While delving deeper into some of the terminology is outside of the scope of CLASS Essentials, we go will through these terms in brief as we learn how to configure your EC2 instance. \n", + "Step 3 in creating an EC2 instance involves a rudimentary understanding of several key AWS and cloud jargon (Figure 4). While delving deeper into some of the terminology is outside of the scope of CLASS Essentials, we go will through these terms in brief as we learn how to configure your EC2 instance. For the most part, we will **leave the settings as default**. CLASS Intermediate offers a more in depth discussion on cloud concepts. \n", "\n", "![Configure Instance Details](images/ec2-configure.png)\n", "\n", @@ -150,11 +151,25 @@ "* How do Spot Instances help you optimize costs?\n", "````" ] + }, + { + "cell_type": "markdown", + "id": "1b8642f4", + "metadata": {}, + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fff35db8", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "Python 3", "language": "python", "name": "python3" }, @@ -168,7 +183,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.10" + "version": "3.9.5" } }, "nbformat": 4, diff --git a/content/AWS/03_intro_to_compute_part2.ipynb b/content/AWS/03_intro_to_compute_part2.ipynb index 28269db..675056a 100644 --- a/content/AWS/03_intro_to_compute_part2.ipynb +++ b/content/AWS/03_intro_to_compute_part2.ipynb @@ -5,7 +5,7 @@ "id": "dc57021c", "metadata": {}, "source": [ - "# Instance Storage, Tags & Security Groups\n", + "# Introduction to Elastic Cloud Compute (EC2) - Part 2\n", "\n", "\n", "```{admonition} Overview\n", @@ -16,13 +16,14 @@ "**Exercises:** 10 mins\n", "\n", "**Questions:**\n", - "* What is an EC2 instance?\n", - "* When would I use an EC2 instance?\n", - "* How do I launch an EC2 instance?\n", + "* How do I allocate storage to an instance?\n", + "* Why should I attached tags to an instance?\n", + "* What are security groups?\n", "\n", "**Objectives:**\n", - "* Understand the concept of virtual servers.\n", + "* Understand the concept of storage, tags and security groups.\n", "* Understand what an Elastic Cloud Compute (EC2) instance is.\n", + "* Understand how to launch an EC2 instance. \n", "\n", "```" ] @@ -32,32 +33,8 @@ "id": "502f2360", "metadata": {}, "source": [ - "Recall that the two fundamental components of cloud computing is compute and storage. On AWS, a \"virtual server\" or \"virtual computer\" is known as an **Elastic Cloud Compute (EC2) instance**; sometimes it's called \"EC2\", sometimes it's called an \"instance\" to denote that the ability to build and terminate this server instantaneously, but they all mean the same thing. An EC2 instance is no different from a server that sits under your desk, or your local departmental cluster, or even your local HPC cluster. You even boot up an EC2 instance through the web console, install software and then shut down your instance just like you would a real computer, except that Amazon takes care of the physical machinery while you are in charge of process of creating the computer. In some sense, you can think of utilizing an EC2 instance as renting a server or computer from Amazon! \n", + "In the previous lesson we learned how to launch a virtual machine from the AWS console. We selected an Amazon Machine Image (AMI), Chose an Instance and Configured Launch Settings. Recall that there are 7 steps to walk through to create a new EC2 instance; we will go through each in detail: \n", "\n", - "In cloud jargon, the term **elasticity** denotes the ability to quickly expand or decrease computer processing, memory, and storage resources to meet changing demands. In that way, you can expand the size of your CPU, RAM and disk size on your EC2 instance almost instantenously. Since EC2 forms the backbone of most of AWS's core infrastructure, it is an important part of your cloud journey. \n", - " \n", - "\n", - "Let's walk through some of the steps on getting an EC2 instance up and running. \n" - ] - }, - { - "cell_type": "markdown", - "id": "bc5d082d", - "metadata": {}, - "source": [ - "We begin with the AWS console again. Under the \"Build a Solution\" panel, select `Launch a Virtual Machine`\n", - "\n", - "![Figure 1: Start page for the AWS console](images/console_ec2.png)\n", - "\n", - "This will then lead you through a series of steps to get a **Free Tier** EC2 instance up and running. \n", - "\n", - "```{admonition} Note\n", - ":class: note\n", - "\n", - "AWS Free Tier refers to several of the services that AWS offers to help users gain more hands on experience on the AWS platform without being charged. [Click here](https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=*all&awsf.Free%20Tier%20Categories=*all) for more info about the AWS Free Tier [external link] . \n", - "```\n", - "\n", - "There are 7 steps to walk through to create a new EC2 instance; we will go through each in detail: \n", "1. Select an AMI\n", "2. Choose Instance Type\n", "3. Configure Instance \n", @@ -72,19 +49,11 @@ "id": "b9809503", "metadata": {}, "source": [ - "## 1. Select an AMI\n", - "\n", - "An Amazon Machine Image (AMI) is a template that Amazon uses to describe the operating system, disk type and all the software configuration that is needed to make sure a computer runs smoothly. Imagine that you are purchasing a new laptop; fresh out of the box, the laptop is pre-configured with an operating system (e.g. Windows, Mac OS, Ubuntu etc.), configuration files that tells the laptop what peripherals are attached, and pre-installed software like Adobe PDF reader. An AMI contains all this information so that your EC2 instance runs exactly like it would a new laptop out of the box! There is much more to learn about AMIs and how they can used for collaboration and data sharing but that is not within the scope of CLASS Essentials. \n", - "\n", - "As you scroll through the AMI list shown below you will notice that the list contains offerings from various vendors (e.g. Amazon, RedHat, Windows, etc.). We will be choosing the Ubuntu operating system for flexibility and versatility (can be used outside of the AWS ecosystem). \n", - "\n", - "To list all the Free Tier AMIs, check the box on the right that says ```Free tier only```.\n", + "## Step 4. Add Storage\n", "\n", - "![Step 1 - Select an AMI - Free Tier Only](images/ec2-ami.png)\n", + "Storage on an EC2 instance is akin to a hard drive. Here we will leave the default settings but it is important to know that a hard drive on an EC2 instance is known as [Elastic Block Storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html). EBS volumes behave like raw, unformatted block devices.\n", "\n", - "Scroll to ```Ubuntu Server 20.04 LTS(HVM), SSD Volume Type``` shown below. Select ```64-bit(x86)```. \n", - "\n", - "![Step 1 - Select an AMI - Operating System Selection](images/ec2-ubuntu.png)" + "" ] }, { @@ -92,36 +61,51 @@ "id": "17597535", "metadata": {}, "source": [ - "## Step 2: Choose an Instance Type\n", - "\n", - "Choosing an instance type is choosing the hardware for your computing system: you get to pick the number of CPUs and memory size for your instance. \n", + "## Step 5: Add Tags\n", "\n", - "Instance types are group by [**families**](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html) and denotes whether, for example, an instance is optimized for batch processing (compute-optimized, C-family), optimized for databases (memory-optimized, R-family) or has accelerated hardware (GPUs) for AI or Machine Learning pipelines. \n", + "[Tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) are labels that you assign to an AWS resource. Each tag consists of a key and an optional value, both of which you define.Tags enable you to categorize your AWS resources in different ways, for example, by purpose, owner, or environment. e\n", "\n", - "When you choose an Instance Type (Figure 3), the screen show additional information about the selected instance type including the number of CPUs, the memory size, the type of storage and information about networking. \n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "4ee655a3", + "metadata": {}, + "source": [ + "## Step 6: Configure Security Group\n", "\n", - "In the Instance Storage (GB) column, you will notice a term called **EBS**. EBS is the acronym for **Elastic Block Storage** and is analogous to the hard disk or boot drive on your personal computer or laptop. More details about EBS and different kinds of disk storage on EC2 instances are beyond the scope of CLASS Essentials. \n", + "A [security group](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) acts as a virtual firewall for your EC2 instances to control inbound and outbound traffic. Again, we will not delve too much into networking protocols in CLASS Essentials. We will leave the default values of opening port 22 so that we can securely log in to the EC2 instance that we create. \n", "\n", - "```{admonition} Note\n", - ":class: note\n", - "The four most common types of storage you will encounter on AWS are: Elastic Block Storage (EBS), Elastic File Storage (EFS), Simple Storage Service (s3) and s3 Glacier. In the simplest terms, EBS is analogous to a computer hard drive and EFS is analogous to a network file system (NFS) or shared file system. s3 is AWS's object storage which is discussed [here](05_intro_to_cloud_storage). s3 Glacier is a cost-effective way of storing s3 files that you do not need to access frequently. \n", - "```\n" + "" ] }, { "cell_type": "markdown", - "id": "4ee655a3", + "id": "5235bac7", "metadata": {}, "source": [ - "![Step 2 - Choose an Instance Type](images/ec2-type.png)\n", + "## Step 7: Review Instance Launch\n", + "\n", + "\n", + "\n", + "\n", "\n", - "Select ```Next: Configure Instance Details``` and we'll move on to the next chapter. " + "" ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "113ba95a", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "Python 3", "language": "python", "name": "python3" }, @@ -135,7 +119,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.10" + "version": "3.9.5" } }, "nbformat": 4, diff --git a/content/AWS/04_intro_to_cli.ipynb b/content/AWS/04_intro_to_cli.ipynb deleted file mode 100644 index 6a01cb7..0000000 --- a/content/AWS/04_intro_to_cli.ipynb +++ /dev/null @@ -1,47 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "dc57021c", - "metadata": {}, - "source": [ - "# Introduction to the AWS CLI\n", - "\n", - "\n", - "```{admonition} Overview\n", - ":class: tip\n", - "\n", - "**Teaching:**\n", - "\n", - "**Exercises:**\n", - "\n", - "**Questions:**\n", - "\n", - "**Objectives:**\n", - "\n", - "```" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/content/AWS/05_intro_to_cloud_storage.ipynb b/content/AWS/05_intro_to_cloud_storage.ipynb deleted file mode 100644 index 79993ad..0000000 --- a/content/AWS/05_intro_to_cloud_storage.ipynb +++ /dev/null @@ -1,47 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "dc57021c", - "metadata": {}, - "source": [ - "# Introduction to Cloud Storage\n", - "\n", - "\n", - "```{admonition} Overview\n", - ":class: tip\n", - "\n", - "**Teaching:**\n", - "\n", - "**Exercises:**\n", - "\n", - "**Questions:**\n", - "\n", - "**Objectives:**\n", - "\n", - "```" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/content/GCP/01_intro_to_cloud_console.ipynb b/content/GCP/01_intro_to_cloud_console.ipynb index 2983c0a..f8e73fe 100644 --- a/content/GCP/01_intro_to_cloud_console.ipynb +++ b/content/GCP/01_intro_to_cloud_console.ipynb @@ -50,7 +50,7 @@ "\n", " * Login to the web console with your (Google) Account by going to [https://console.cloud.google.com/](https://console.cloud.google.com/)\n", " * Login with your credentials if you have not done so already.\n", - " * Find your user/account under the \"Avatar\" in the top right (see below).\n", + " * Find your user/account under the \"Avatar\" in the top right (see below) and click on the avatar.\n", " ![avatar-account](img/avatar-account.png) and click on it if it is not the top one.\n", " * If you do not see your account click on \"Add Account\" and repeat the previous step.\n", "\n", @@ -92,7 +92,7 @@ " \n", "For a Project to do anything useful it must also have an enabled Billing Account associated with it. An enabled Billing Account is a prerequisite for this lesson.\n", "\n", - "A Project, just like the word, should be associated with a real world project (for example a research effort or grant, lab, or your Ph.D. Project. etc). For Drew, this will be the Acceleration Lab project. In this way it is easier to track and allocate costs and to manage permissions and access for resources within the project.\n", + "A Project, just like the word, should be associated with a real world project (for example a research effort or grant, lab, or your Ph.D. Project. etc). For Drew, this is the image processing project. In this way it is easier to track and allocate costs and to manage permissions and access for resources within the project.\n", "\n", "The active project is also shown next to the project icon (three hexagons) and clicking it brings up the project selection dialog. The current project has a check mark and and the active project can be changed by clicking on the project name or double clicking the row.\n", " ![select-project](img/select-project.png) \n", @@ -128,8 +128,14 @@ " * You can pin frequently used items on the top of this page by clicking on the pin icon. \n", " * Click the hamburger again to hide the Left Sidebar.\n", "\n", - "All the different products and services are the \"What\" of the cloud. We will start with Cloud Storage in the next Episode. \n", - "\n" + "All the different products and services are the \"What\" of the cloud. We will start with compute, called *Google Compute Engine* in the next Episode. \n", + "\n", + "### Exercise \n", + "\n", + "Take a few moments to navigate a few key services.\n", + " * Navigate to the \"Compute Engine\" service under the \"Compute\" product group.\n", + " * You will probably need to \"Enable\" this service first by clicking on the \"Enable\" button on the \"Compute Engine API\" page. This will only need to be done once per project.\n", + " * Navigate to \"Cloud Storage\" under the \"Storage\" product group and enable the service if necessary." ] }, { @@ -170,14 +176,16 @@ "source": [ "## Open a Cloud Shell\n", "\n", - "The Cloud Shell is an easy way to access Comand Line Interface (CLI) commands in a web browser. Under the hood Cloud Shell is Debian running in a container with 5GB of no-charge persistent storage with most of the Google uti and a lot of other useful tools (git, docker, jq, Docker, etc.) installed by default.\n", + "The Cloud Shell is an easy way to access the Comand Line Interface (CLI) in a web browser. Under the hood Cloud Shell is Linux (Debian) running in a container with 5GB of no-charge persistent storage with most of the Google utilities and a lot of other useful tools such as git, docker, jq, Docker, etc. installed by default.\n", "\n", "To use the Cloud Shell:\n", - " * Open up a Cloud Shell by clicking on the **Activate Cloud Shell** icon in the right side of the top blue bar. If a \"Authorize Cloud Shell\" dialog appears click \"Authorize\" to allow the Cloud Shell to access the project.\n", + " * Open up a Cloud Shell by clicking on the **Activate Cloud Shell** icon in the right side of the top blue bar. If a \"Authorize Cloud Shell\" dialog appears click \"Authorize\" to allow the Cloud Shell to access your Project.\n", " * Click the **Maximize** icon on the right side of the Cloud Shell bar. You can find the correct icon by hovering over the icons to see their names.\n", " * If a shell is associated with a project it will show it in the tab and in **yellow text** in round parenthesis in the shell prompt. **Verify the Project ID**.\n", " * To open up a shell with a specific project click the \"down arrow\" icon next to the plus(+) to open a new tab with a specific project. Selecting the plus (+) will usually open up a shell with the default project.\n", - " * Note: You can open the Cloud Shell in a new tab by clicking on icon **Open in a New Window** on the far right of the Cloud Shell bar.\n", + " * Note: You can open the Cloud Shell in a new tab by clicking on the **Open in a New Window** icon on the far right of the Cloud Shell bar.\n", + "\n", + "![cloud-shell](img/cloud-shell.png)\n", "\n", "\n", "### Opening Cloud Shell in a Tab - Optional\n", diff --git a/content/GCP/02_intro_to_compute.ipynb b/content/GCP/02_intro_to_compute.ipynb index 5eb30ff..374d54b 100644 --- a/content/GCP/02_intro_to_compute.ipynb +++ b/content/GCP/02_intro_to_compute.ipynb @@ -46,7 +46,7 @@ "source": [ "## Google Compute Engine\n", "\n", - "To learn more about the Google Compute Engine service we will use the web console to create, explore, and destroy a Virtual Machine. The ability to quickly create, explore, and then discard resources in the web-console is a powerful tool to explore a service. After an initial exploration with the web-console, programmatic access using scripts or configuration management systems should be used.\n", + "To learn more about the Google Compute Engine service we will use the web console to create, explore, and destroy a Virtual Machine. The ability to quickly create, explore, and then discard resources in the web-console is a powerful tool to learn more about a service. After an initial exploration with the web-console, programmatic access using scripts or configuration management systems should be used.\n", "\n", "Unfortunately, the terminology used for cloud computing differs from one commercial cloud provider to another. Google Compute Engine uses the term **VM instance** to refer to a running virtual machine, **image** for the boot disk containing the operating system, and **disk** for the writable storage used for a running *VM Instance*.\n", "\n", @@ -64,7 +64,7 @@ "\n", "We will first create an *VM instance* with defaults to explore the Google Compute Engine service.\n", "\n", - "To create a bucket we do the following:\n", + "To create a VM Instance we do the following:\n", " * Click **Navigation Menu** -> **Compute Engine** (under Compute) -> **VM Instances** -> **+Create Instance** (just under the blue bar) to open the *Create an instance* page.\n", " * In the **New VM instance** tab on the left (selected by default) configure the *VM instance* as follows:\n", " * For **Name**, enter a unique name for the instances (example: \"essentials-instance-1\")\n", @@ -72,12 +72,11 @@ " * For **Zone** leave the default (note how the name is constructed and that it is a separate data center) some zones have different capabilities.\n", " * In the **Machine configuration** section:\n", " * For **Series** select *E2* (default). The other options are for types of machines.\n", - " * For **Machine type** select *e2-medium* (default)\n", + " * For **Machine type** select *e2-medium* (default). Note that it shows the size of the VM in parentheses. \n", " * In the **Identity and API access** section note that for the **Service Accounts** -> **Service account** option that the **Compute Engine default service account** is selected. This allows the *VM Instance* to access resources in the project. Note: the *\"Allow full access to all Cloud APIs\"* allows *everyone* on the VM to *access and edit all resources in the project*.\n", " * Leave the remaining options as default.\n", - " * Note the **Monthly estimate** in the right side panel. This is only for running the *VM instance* and does not include other storage, networking, compute, and other charges that may occur by running software. Click **Details** show a breakdown of the *VM instance* only costs.\n", - " * Finally, click **Create** to create the *VM Instance* (allocate the resource).\n", - " " + " * Note the **Monthly estimate** in the right side panel. This is only for running the *VM instance* and does not include other storage, networking, compute, and other charges that may occur by running software. Click **Details** show a breakdown of the *VM instance* only costs. **Exercise:** Find the most expensive VM.\n", + " * Finally, click **Create** to create the *VM Instance* (allocate the resource).\n" ] }, { @@ -87,9 +86,9 @@ "source": [ "## Security\n", "\n", - "Everything in the cloud requires permission (authorization). Ordinary we would configure and check security first but in the case of exploring services it is often easier to do things out of order. We noted that the *VM instance* was created with the *Compute Engine default service account*, which by *default* has access to all the resources that your Google Account does.\n", + "Everything in the cloud requires permission (authorization). Ordinary we would configure and check security first but in the case of exploring services it is often easier to do things out of order. We noted that the *VM instance* was created with the *Compute Engine default service account*, and if the \"Allow full access to all Cloud Api's\" scope is enable, then everyone on the VM has access to all the resources in your project..\n", "\n", - "Specifically, we are interested in verifying what permissions that the *Compute Engine default service account* has for your *VM instances* . To do this navigate to the IAM page (**Navigation Menu** -> **IAM & Admin** -> **IAM** -> **Permissions** -> View By: **Principals**)\n", + "Specifically, we are interested in verifying what permissions that the *Compute Engine default service account* has for your *VM instances* . To do this navigate to the IAM page (**Navigation Menu** -> **IAM & Admin** -> **IAM** -> **Permissions** tab -> View By: **Principals** tab)\n", "\n", "*Note: There is a powerful filter box to limit the permissions shown.*\n", "\n", @@ -114,9 +113,10 @@ " * Navigate to the project **Dashboard** by clicking on \"Google Cloud Platform\" on the left side of the top blue bar.\n", " * Select **Activity** tab just below the top blue bar.\n", " * Look for the row with **Create VM** on the top of the table. This shows that the *VM Instance* was created.\n", - " * Expand the detail by clicking the **down arrow** on the right side of the row. You can see what identity created the bucket and the name. The rest of the information details the initial settings that you configured when creating the bucket. The details are not really important at this time, just know they exist.\n", + " * Expand the detail by clicking the **down arrow** on the right side of the row. You can see what identity created the bucket and the name. The rest of the information details the initial settings that you configured when creating the virtual machine. The details are not really important at this time, just know they exist.\n", + " * By default, this information cannot be deleted for security and auditing reasons.\n", "\n", - "Note that the activity is on the project, not the account.\n" + "Note: that the activity is on the project, not on your account.\n" ] }, { @@ -128,12 +128,12 @@ "source": [ "## Enumerate the VM Instances\n", "\n", - "To see an overview of resources in the project navigate to the **dashboard** and you will now see at least **1 instance** in the *resources* card under *Compute Engine*.\n", - "\n", "Now lets find and connect to the *VM Instance*.\n", " * Navigate to the Google Compute Engine page by clicking **Navigation Menu** -> **Compute Engine** (under Compute) -> **Instances**.\n", " * **Find** the *VM instance* you just created. You can use the filter to find a bucket if there are a lot of them.\n", - " * Click on the *VM instance* name to open the **VM instance details** (it will display as a hyperlink when you hover over the *VM instance* name)." + " * Click on the *VM instance* name to open the **VM instance details** (it will display as a hyperlink when you hover over the *VM instance* name).\n", + "\n", + "Navigate to the **dashboard** and you will see **Compute Engine** in the *resources* card. You can click on this to quickly navigate to the compute engine page.\n" ] }, { diff --git a/content/GCP/03_intro_to_cloud_storage.ipynb b/content/GCP/03_intro_to_cloud_storage.ipynb index 815b737..bbc64a4 100644 --- a/content/GCP/03_intro_to_cloud_storage.ipynb +++ b/content/GCP/03_intro_to_cloud_storage.ipynb @@ -33,8 +33,6 @@ "id": "9897048a-6aa8-4d85-a557-d85b802f3f1d", "metadata": {}, "source": [ - "Drew, after working with a Research Computing and Data (RCD) facilitator, has decided to start their cloud journey with storage. Data is usually the center of a research project and it makes sense to start here.\n", - "\n", "One of the most common and economic ways to store data in the cloud is to use object storage. In GCP object storage is called *Google Cloud Storage*, which is similar to the Simple Storage Service, also known as S3, on Amazon Web Services (AWS). For object storage, information is stored as a collection of key-value pairs." ] }, @@ -47,8 +45,6 @@ "source": [ "## Google Cloud Storage\n", "\n", - "To learn more about the Google Cloud Storage service we will use the web console to create, explore, and destroy a Bucket. The ability to quickly create, explore, and then discard resources in the web-console is a powerful tool to explore a service. After an initial exploration with the web-console, programmatic access using scripts and programming languages should be used.\n", - "\n", "Unfortunately, the terminology used for object storage differs from one commercial cloud provider to another. Google Cloud Storage uses the terms Bucket and Object. Data is stored in Objects and Objects are stored in Buckets. Objects (data) are stored as a key-value pair, which is similar to a Python dictionary but persistent. Values are stored and retrieved using a unique key and the value can contain any information (more on this later).\n", "\n", "We now take Drew through the process of creating a Google Cloud Storage bucket." @@ -69,7 +65,7 @@ "\n", "*Note: There is a powerful filter box to limit the permissions shown.*\n", "\n", - "You should see a row with your account shown in the Principal column. Here you should see the \"Owner\" Role in the Role column. A *role* is a collection of permissions managed by Google or someone else. The **Owner** or the **Storage Admin** role for a project will *allow* *you* to create, access, and delete Buckets *in* the project.\n", + "You should see a row with your account shown in the Principal column. Here you should see the \"Editor\" Role in the Role column. A *role* is a collection of permissions managed by Google or someone else. The **Editor**, **Owner**, or the **Storage Admin** role for a project will *allow* *you* to create, access, and delete Buckets *in* the project.\n", "\n", "There are three important pieces of information that work together to form the **IAM policy**. The permission (role), the identity (principal or member), and the resource (project)." ] @@ -102,7 +98,7 @@ " * Under *Access Control* Select **Uniform** (default). *Uniform* permissions are on the entire Bucket. *Fine-grained* permissions are on the objects.\n", " * Select **Continue**\n", " * Note the **monthly cost estimate** in the right side panel. This is only for reading, writing, and storing data and does not include networking, compute, and other charges that may occur by using the data.\n", - " * *Storage size* is the total amount of data stored.\n", + " * *Storage size* is the total amount of data stored. **Exercise:** Enter 1000 in the storage size box to see how much a TB costs per month.\n", " * *Data Retrieval* is only used for for long-term, non *standard*, storage.\n", " * *Operations* is the number of \"operations\" performed on an object, for example reading and writing an object. Storing and accessing many objects require many operations.\n", " * Finally, click **Create** to create the bucket (allocate the resource).\n", @@ -118,7 +114,7 @@ "source": [ "## Follow the Storage Allocation\n", "\n", - "Just as with security, we will audit (follow) the bucket creation by examining at the project *activity*.\n", + "Just as with compute, we will audit (follow) the bucket creation by examining at the project *activity*.\n", "\n", "To view the project activity we do the following:\n", "\n", @@ -145,7 +141,7 @@ " * **Find** the bucket you just created. You can use the filter to find a bucket if there are a lot of them.\n", " * Click on the bucket name to open the **bucket details** (it will display as a hyperlink when you hover over the bucket name).\n", "\n", - "Navigate to the **dashboard** and you will now see at least **1 bucket** in the *resources* card under storage." + "Navigate to the **dashboard** and you will now see \"Storage\" in the *resources* card under. You can click on this to quickly navigate to the storage page." ] }, { @@ -189,8 +185,9 @@ "source": [ "## Discussion\n", "\n", - "* What does the words \"Secure\", \"Allocate\", \"Follow\", and \"Enumerate\" Spell?\n", - "* Why do you think we use this framework?" + "* What does the words \"Secure\", \"Allocate\", \"Follow\", and \"Enumerate\" spell?\n", + "* What happens when you add the \"R\" in \"Review?\"\n", + "* Is this useful?" ] }, { @@ -198,7 +195,7 @@ "id": "97d7ebc5-4a81-4f1a-aaf3-517adf70640a", "metadata": {}, "source": [ - "## Resources in Google Cloud Platform - Optional\n", + "## Resources in Google Cloud Platform - Review\n", "\n", "Even though we only covered the Google Cloud Storage service in this episode, this process can be used for other *resources* allocated in the cloud. The term *resource* is used for the \"things\" that live in a Project, such as compute, storage, and networking and other services. Resources have the following characteristics:\n", "\n", diff --git a/content/GCP/07_monitoring_costs.ipynb b/content/GCP/07_monitoring_costs.ipynb index 39eb3db..d41ec1d 100644 --- a/content/GCP/07_monitoring_costs.ipynb +++ b/content/GCP/07_monitoring_costs.ipynb @@ -26,7 +26,7 @@ "* Find where to set Billing Alerts.\n", "* List all resources in a Project\n", "* List resources in a service\n", - "* Understand how labels are used\n", + "* Optional: Understand how labels are used\n", " \n", "```" ] diff --git a/content/GCP/img/avatar-account.png b/content/GCP/img/avatar-account.png index 6b8c543..beda63a 100644 Binary files a/content/GCP/img/avatar-account.png and b/content/GCP/img/avatar-account.png differ diff --git a/content/GCP/img/cloud-shell.png b/content/GCP/img/cloud-shell.png index 5e7960d..4949eb3 100644 Binary files a/content/GCP/img/cloud-shell.png and b/content/GCP/img/cloud-shell.png differ diff --git a/content/GCP/img/kabob-project.png b/content/GCP/img/kabob-project.png index 4653564..eef54f5 100644 Binary files a/content/GCP/img/kabob-project.png and b/content/GCP/img/kabob-project.png differ diff --git a/content/GCP/img/select-project.png b/content/GCP/img/select-project.png index 1e30739..57a98f8 100644 Binary files a/content/GCP/img/select-project.png and b/content/GCP/img/select-project.png differ diff --git a/content/intro_to_cloud.ipynb b/content/intro_to_cloud.ipynb index 5579a6e..35c9f48 100644 --- a/content/intro_to_cloud.ipynb +++ b/content/intro_to_cloud.ipynb @@ -70,7 +70,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -84,7 +84,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.5" + "version": "3.7.3" } }, "nbformat": 4, diff --git a/scripts/gcp-create.sh b/scripts/gcp-create.sh index 99af56c..8a0fd2d 100755 --- a/scripts/gcp-create.sh +++ b/scripts/gcp-create.sh @@ -18,9 +18,9 @@ echo "=== gcp-dev.sh $PROJECT $BRANCH" echo "+++ creating/starting VM" gcloud compute instances create --zone=$ZONE $VM \ --scopes=cloud-platform \ - --machine-type=e2-highcpu-4 --preemptible + --machine-type=e2-highcpu-2 --preemptible gcloud compute instances start --zone=$ZONE $VM -IP=$(gcloud compute instances describe essentials --format='value(networkInterfaces.accessConfigs[0].natIP)') +IP=$(gcloud compute instances describe essentials --zone=$ZONE --format='value(networkInterfaces.accessConfigs[0].natIP)') echo "+++ configure local ssh $IP" gcloud compute config-ssh @@ -51,15 +51,5 @@ git config --global user.email "$(git config user.email)" git clone --branch $BRANCH $REPO EOF -echo "+++ configure local ssh" -gcloud compute config-ssh -cat > ~/.ssh/$VM.config <