diff --git a/content/AWS/05_running_analysis.ipynb b/content/AWS/05_running_analysis.ipynb index edd93a7..0c40259 100644 --- a/content/AWS/05_running_analysis.ipynb +++ b/content/AWS/05_running_analysis.ipynb @@ -336,7 +336,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.2" + "version": "3.9.7" } }, "nbformat": 4, diff --git a/content/Azure/01_intro_to_cloud_console.ipynb b/content/Azure/01_intro_to_cloud_console.ipynb index fbc27c5..c4b5219 100644 --- a/content/Azure/01_intro_to_cloud_console.ipynb +++ b/content/Azure/01_intro_to_cloud_console.ipynb @@ -210,7 +210,7 @@ }, { "cell_type": "markdown", - "id": "a1c0e980", + "id": "c58c867b", "metadata": {}, "source": [ "This prompt is asking you to create a .ssh directory in your home directory and generate ssh public/private keys named id_rsa.pub/id_rsa (default) respectively.\n", @@ -245,7 +245,8 @@ "+----[SHA256]-----+\n", "```\n", "\n", - "The ssh key generation is now complete. This will only happen once for your account. Later during the VM create proces you will be asked to Cut & Paste the public key into a form. " + "The ssh key generation is now complete. You will only need to do this once for your account. Later during the VM \n", + "create proces you will be asked to Cut & Paste the public key into a form. " ] }, { diff --git a/content/Azure/02_intro_to_compute.ipynb b/content/Azure/02_intro_to_compute.ipynb index feed87a..3b526a0 100644 --- a/content/Azure/02_intro_to_compute.ipynb +++ b/content/Azure/02_intro_to_compute.ipynb @@ -30,59 +30,81 @@ }, { "cell_type": "markdown", - "id": "92622861", + "id": "cb09d264", "metadata": {}, "source": [ - "Drew, after working with a Research Computing and Data (RCD) facilitator, has decided to start their cloud journey with compute, specifically a Virtual Machine, to process data in the cloud with a large and powerful machine to do the analysis. " + "Drew, after working with a Research Computing and Data (RCD) facilitator, has decided to start their cloud journey\n", + "with compute to process data in the cloud with a Virtual Machine to do the analysis. " ] }, { - "cell_type": "code", - "execution_count": null, - "id": "b85b77ed", + "cell_type": "markdown", + "id": "485a754a", "metadata": {}, - "outputs": [], "source": [ "## Azure Virtual Machines" ] }, { "cell_type": "markdown", - "id": "ce935ec7", + "id": "0ef69f07", "metadata": {}, "source": [ - "Before we begin this lesson/episode we need to create a ssh key in order to login into the VM we about to create. We will use the public key when we create the VM.\n", + "Before we begin this lesson/episode and create a Virtual Machine (VM), we need to create a ssh \n", + "key so that we can securely login to the VM we are about to create. We need use a ssh public \n", + "key when we create the VM.\n", "\n", - "**Note:** Step 1. below is not required if you have already created a ssh key in the [Intro to Cloud Console](01_intro_to_cloud_console.ipynb) lesson. If you have already created a key go directly to step 2.\n", + "**Note:** Step 1. below is not required if you have already created a ssh key in the \n", + "[Intro to Cloud Console](02_intro_to_compute.ipynb) lesson. If you have already \n", + "created a key go directly to step 2.\n", "\n", "**Step 1.**\n", "\n", - "If you have not used the cloud shell to connect to a *VM Instance* before you will need to create a new *ssh key*. The Compute Engine will use this key to allow you to access the *VM instance* in a secure manner. If this is the case you will see a message similar to the following:\n", - "```\n", - "WARNING: The private SSH key file for gcloud does not exist.\n", - "WARNING: The public SSH key file for gcloud does not exist.\n", - "WARNING: You do not have an SSH key for gcloud.\n", - "WARNING: SSH keygen will be executed to generate a key.\n", - "This tool needs to create the directory [/home/learner/.ssh] before\n", - " being able to generate SSH keys.\n", + "If you have not used the cloud shell to connect to a *VM Instance* before, you will need to \n", + "create a new *ssh key*. The VM will use this key to allow you to access the *VM instance* \n", + "in a secure manner. \n", "\n", - "Do you want to continue (Y/n)?\n", - "```\n", + "Open up a Cloud Shell by clicking on the Cloud Shell icon in the right side of the top \n", + "blue bar (the icon looks like box with a greater than sign followed by an underscore) as \n", + "described in the [Intro to Cloud Console](01_intro_to_cloud_console.ipynb) lesson.\n", + "\n", + "At the $ prompt in the Cloud Shell, enter ssh-keygen and follow the prompts.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5889478", + "metadata": {}, + "outputs": [], + "source": [ + "$ssh-keygen\n", + "Generating public/private rsa key pair.\n", + "Enter file in which to save the key (/home/john/.ssh/id_rsa):" + ] + }, + { + "cell_type": "markdown", + "id": "164cdb8c", + "metadata": {}, + "source": [ + "This prompt is asking you to create a .ssh directory in your home directory and generate ssh \n", + "public/private keys named id_rsa.pub/id_rsa (default) respectively.\n", + "\n", + "Press enter to continue to create the key. You will see another message asking you for a \n", + "passphrase, similar to the following:\n", "\n", - "Press enter to continue to create the key. You will see another message asking you for a passphrase, similar to the following:\n", - "```\n", "Generating public/private rsa key pair.\n", "Enter passphrase (empty for no passphrase):\n", - "```\n", - "Here you should enter a passphrase (a series of words) to protect the key. You will be asked to re-enter the passphrase as shown below:\n", - "```\n", - "Enter same passphrase again:\n", - "```\n", + "Here you should enter a passphrase (a series of words) to protect the key. You will be asked \n", + "to re-enter the passphrase as shown below:\n", "\n", - "After successfully retyping the passphrase the key generation process will continue showing similar message as follows (the fingerprint and art will look different):\n", - "```\n", - "Your identification has been saved in /home/learner/.ssh/azure_compute_engine.\n", - "Your public key has been saved in /home/learner/.ssh/azure_compute_engine.pub.\n", + "Enter same passphrase again:\n", + "After successfully retyping the passphrase the key generation process will continue showing a \n", + "similar message as follows (the fingerprint and art will look different):\n", + "````\n", + "Your identification has been saved in /home/learner/.ssh/id_rsa.\n", + "Your public key has been saved in /home/learner/.ssh/id_rsa.pub.\n", "The key fingerprint is:\n", "SHA256:D9yj1nZEUigahirvjkIFoCP7RfjDu67Ku88M7nHQftI learner@cs-748001990186-default-boost-r9phq\n", "The key's randomart image is:\n", @@ -97,74 +119,78 @@ "|+.B o. . . . |\n", "|=O**o |\n", "+----[SHA256]-----+\n", - "```\n", - "\n", - "The ssh key generation is now complete. This will only happen once for your account. Later during the VM create proces you will be asked to Cut & Paste the public key into a form. \n", + "````\n", "\n", - "\n", - "**Step 2.**\n", - "Copy the public key to your clip board future connections.\n" + "The ssh key generation is now complete. You will only need to do this once for your account. \n", + "Next, you will need to Cut (copy) and later Paste the public key during the VM creation process." ] }, { "cell_type": "markdown", - "id": "0f47f212", + "id": "d2237859", "metadata": {}, + "source": [ + "**Step 2.**\n", + "\n", + "To Cut (copy) your public key, run the following command." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e4971654", + "metadata": {}, + "outputs": [], "source": [ "cat /home/learner/.ssh/id_rsa.pub" ] }, { "cell_type": "markdown", - "id": "d7f0bf3c", + "id": "b758abd3", "metadata": {}, "source": [ - "Highlight the output string with your cursor and use the copy command (depending on OS) to put it in your edit buffer." + "Highlight the output string with your cursor starting with ssh-rsa until the end of the string (white test) and \n", + "before you come to the green command prompt. With the public key highlighted, use the copy command (depending on OS) \n", + "to put it in your edit buffer (i.e. for mac use command c)." ] }, { - "cell_type": "code", - "execution_count": null, - "id": "0d5b8abd", + "cell_type": "markdown", + "id": "84ddff97", "metadata": {}, - "outputs": [], "source": [ "## Create a VM\n", "\n", - "Hamburger in upper left -> Virtual Machines -> Create (+) in upper left -> virtual machines ->\n", - "In the middle of the page under \"Create Virtual Machine\" heading there are menu options from left to right: Basic, Disk, Networking, Management, Advanced, Tags, Review & Create\n", + "From the hamburger menu in the upper left of the top blue bar, click *Virtual Machines*. On the *Virtual machines* \n", + "page click Create (+) on the menu in upper left of the page. Choose *Virtual machines*.\n", + "In the middle of the page under the \"Create Virtual Machine\" heading there are menu options from left to right: \n", + "Basic, Disk, Networking, Management, Advanced, Tags, Review & Create. We will start with the Basic tab (default).\n", "\n", "**Part 1**\n", - "* Check subscription \n", + "\n", + "Under the Basic tab (default)\n", + "\n", + "* Check subscription\n", + " * Use the pull down menu to change subscriptions if needed\n", "* select or create Resource Group\n", - "* VM name\n", + " * Click the create a new Resource Group and enter a name that conforms to the azure naming convention (only alphanumeric characters, periods, underscores, hyphens, and parenthesis and can not end in a period)\n", + "* Virtual machine name\n", + " * Enter a name that does not contain special characters including blah or begin *_* or end with *.* or *-* \n", "* Image\n", - "* Spot\n", + " * Use default image (Ubuntu 20.04 - Gen2)\n", "* Size\n", + " * Standard_D2s_v3 (default)\n", "\n", - "![vm-create](images/01_CLASS_Azure_VM_basic_01.png)\n", - "\n", - "**Part 2**\n", - "\n", - "Before we begin part 2, we need to supply a public key when we create the VM so we can securely login. Using the key we created earlier (assume - /home/learner/.ssh/azure_compute_engine.pub), we will copy the key by doing the following." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c8fa1b24", - "metadata": {}, - "outputs": [], - "source": [ - "cat /home/learner/.ssh/azure_compute_engine.pub" + "![vm-create](images/01_CLASS_Azure_VM_basic_01.png)" ] }, { "cell_type": "markdown", - "id": "5da66e3b", + "id": "18d099ed", "metadata": {}, "source": [ - "Highlight the output string with your cursor and use the copy command (depending on OS) to put it in your edit buffer.\n", + "**Part 2**\n", "\n", "* Authentication type \n", " * ssh public key\n", @@ -172,11 +198,13 @@ " * Choose usename (default: azeruser)\n", "* SSH public key source\n", " * Select use existing public key\n", - " * Copy the key we create earlier and paste it in the red box below.\n", + " * Use the key we create and Cut (copied) earlier, paste (depending on OS) it into the red box below (i.e for mac use command v). \n", "* Key pair name\n", " * Choose name for key pair (default: auto generated based on VM name)\n", - "* Ports (none)\n", - " * Open inbound ports (default: none)\n", + "* Public inbound ports\n", + " * Select Allow selected ports (default)\n", + "* Select inbound ports\n", + " * Select SSH (22) from the pull down menu (default)\n", "\n", "![vm-create](images/01_CLASS_Azure_VM_basic_02.png)" ] @@ -201,7 +229,7 @@ }, { "cell_type": "markdown", - "id": "007e8359", + "id": "4feb08e9", "metadata": {}, "source": [ "Now let's continue to the networking configuration by clicking Next:Networking> on the bottum or on the top heading menu choose the Networking option.\n", @@ -216,13 +244,15 @@ " * Allow selected port\n", "* Select inbound port(s)\n", " * Use 22 for ssh access\n", + "* Delete public IP and NIC when VM is deleted\n", + " * Check this box\n", "\n", "![vm-networking](images/03_CLASS_Azure_VM_networking.png)" ] }, { "cell_type": "markdown", - "id": "a1561d9f", + "id": "1f163321", "metadata": {}, "source": [ "Now let's continue to provision the VM. The next three menu options across the top are outside the scope of this tutorial so now we are ready to click Review & Create.\n", @@ -255,36 +285,29 @@ "\n", "![vm-networking](images/04_CLASS_Azure_VM_verify.png)\n", "\n", - "On the VM page, you will be looking at the overview page. The Overview page contains essential information like OS, public IP, RG, status, subscriptioin as well as other properties and information. Find and record the Public IP address. " - ] - }, - { - "cell_type": "markdown", - "id": "f8afd914", - "metadata": {}, - "source": [ - "## Security (need to azurize)\n", - "\n", - "Everything in the cloud requires permission (authorization). Ordinary we would configure and check security first but in the case of exploring services it is often easier to do things out of order. We noted that the *VM instance* was created with the *Compute Engine default service account*, and if the \"Allow full access to all Cloud Api's\" scope is enable, then everyone on the VM has access to all the resources in your project." + "You will be looking at the overview page. The Overview page contains essential information like OS, public IP, \n", + "RG, status, subscriptioin as well as other properties and information. Find and record the Public IP address. " ] }, { "cell_type": "markdown", - "id": "0d1d75a2", + "id": "62c6abd4", "metadata": {}, "source": [ - "## Enumerate the VM Instances\n", - "\n" + "## Security\n", + "## Enumerate the VM Instances" ] }, { "cell_type": "markdown", - "id": "89f6a274", + "id": "dd25a15a", "metadata": {}, "source": [ "## Connect to the VM Instance\n", "\n", - "To connect to the *VM Instance* we will use the cloud shell in the web console. Open up a shell by clicking on the **Activate Cloud Shell** icon in the top blue bar if it is not already open. In Cloud Shell ssh to the VM you just created. (default username \"azureuser\")" + "To connect to the *VM Instance* we will use the cloud shell in the web console. Open up a shell by clicking on the **Activate Cloud Shell** icon (on the top blue bar click\n", + "button the looks like a box with a greater than sign followed by an underscore) in the top blue bar if it is not \n", + "already open. In Cloud Shell ssh to the VM you just created. (default username \"azureuser\")" ] }, { diff --git a/content/Azure/06_running_analysis.ipynb b/content/Azure/06_running_analysis.ipynb index b5ce5ae..e6227b3 100644 --- a/content/Azure/06_running_analysis.ipynb +++ b/content/Azure/06_running_analysis.ipynb @@ -30,622 +30,504 @@ }, { "cell_type": "markdown", - "id": "3a0fb40b", + "id": "77c8568b", "metadata": {}, "source": [ - "## Create a VM\n", - "\n", - "Since we only create resources as we need them in the cloud. As an exercise you will now create a VM for our analysis. In this case will give the VM **Full** access to **Storage**. \n", - "\n", - "Using the console navigate to the \"Compute Engine\" service and create a new VM with the following properties.\n", - " * Call the VM \"essentials\"\n", - " * Allow the VM \"Full\" access to \"Storage\". This can be found under \"Identity and API\" and then selecting \"Set access for each API\" and change \"Storage\" to \"Full\". **This will allow the VM to create, read, write, and delete all storage buckets in the project\"**\n", - " * Feel free to select a larger VM by changing the machine type to something larger, for example an \"e2-standard-2\".\n", - " \n", - "When you are done connect to the machine as described below." + "## A Research Computational and Data Workflow - Drew's story\n", + "Drew needs to do some analysis on the data. They need data \n", + "(satellite images stored in the cloud), computational resources \n", + "(a virtual machine), some software (we will supply this), and a \n", + "place to store the results (Cloud Storage). We will assemble and \n", + "process all these parts in the cloud with a simple example." ] }, { "cell_type": "markdown", - "id": "26f414ef", + "id": "3934bac4", "metadata": {}, "source": [ - "## Connect to the VM\n", + "## Create a VM\n", "\n", - "First login to the instance from the Cloud Shell by running the following command:\n", - "```\n", - "gcloud compute ssh essentials\n", - "```\n", - "If prompted for a zone select `n` to find it automatically. You can see an example session below." + "For this exercise we will use the VM that we created in the [Intro to Cloud Console](01_intro_to_cloud_console.ipynb) \n", + "lesson." ] }, { "cell_type": "markdown", - "id": "41c471fb", + "id": "853b86d9", "metadata": {}, "source": [ - "```\n", - "learner@cloudshell:~ (just-armor-301114)$ gcloud compute ssh essentials\n", - "Did you mean zone [us-central1-b] for instance: [essentials] (Y/n)? n\n", - "\n", - "No zone specified. Using zone [us-west2-c] for instance: [essentials].\n", - "Linux essentials 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64\n", - "\n", - "The programs included with the Debian GNU/Linux system are free software;\n", - "the exact distribution terms for each program are described in the\n", - "individual files in /usr/share/doc/*/copyright.\n", + "## Connect to the VM\n", "\n", - "Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent\n", - "permitted by applicable law.\n", - "Last login: Tue Nov 9 20:12:49 2021 from 34.133.99.196\n", - "learner@essentials:~$\n", - "```\n" + "Recall the public IP and connect to the VM instance using the cloud shell in the web console. Open up a cloud\n", + "shell by clicking on the Activate Cloud Shell icon (on the top blue bar click button the looks like a box with a \n", + "greater than sign followed by an underscore) in the top blue bar if it is not already open. In Cloud Shell ssh \n", + "to the VM you just created. (default username \"azureuser\")." ] }, { "cell_type": "markdown", - "id": "aa2c0f15", + "id": "db8dfe14", "metadata": {}, "source": [ - "## Secure the VM\n", - "\n", - "We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the `sudo unattended-upgrades` command only installs security patches." + "````\n", + "ssh @\n", + "````" ] }, { "cell_type": "markdown", - "id": "d95ece63", + "id": "95e80449", "metadata": {}, "source": [ - "sudo apt update\n", - "sudo unattended-upgrades" - ] - }, - { - "cell_type": "markdown", - "id": "da999ac0", - "metadata": {}, - "source": [ - "## Get Example Code\n", - "\n", - "We will now install `git` and use it to download the example code into your home directory. For those of you who are unfamiliar with git, it is a way to collaboratively manage files and we will only be using it to download the example that we will be using. " + "If this is your first time logging in, you will be prompted with *The authenticity of host 'IP' can't be established.\n", + "type 'y' and this will add the host to the ~/.ssh/*know_hosts* file. After doing this once you will not be prompted\n", + "again (unless the host key changes or the host is removed from the ~/.ssh/*know_hosts* file." ] }, { "cell_type": "code", "execution_count": null, - "id": "0fc21993", + "id": "31813a2d", "metadata": {}, "outputs": [], "source": [ - "sudo apt-get install --yes git" + "ssh azureuser@52.168.77.0\n", + "The authenticity of host '52.168.77.0 (52.168.77.0)' can't be established.\n", + "ECDSA key fingerprint is SHA256:F/3poSNTVVNbfSp3Kc7zBGwiGzt3qwrmjiyfI1ZHnX4.\n", + "Are you sure you want to continue connecting (yes/no)?" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "3d30db18", + "cell_type": "markdown", + "id": "9e42ad8c", "metadata": {}, - "outputs": [], "source": [ - "cd ~" + "## Secure the VM\n", + "\n", + "First make sure that the VM is up to date with the latest versions of pachages and security patches by running the following commands." ] }, { "cell_type": "code", "execution_count": null, - "id": "4a86fa33", + "id": "6ca308e5", "metadata": {}, "outputs": [], "source": [ - "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" + "sudo apt-get update -y && sudo apt-get upgrade -y" ] }, { "cell_type": "markdown", - "id": "5067f044", + "id": "da999ac0", "metadata": {}, "source": [ - "We now change the current directory to the `landsat` directory in the `CLASS-Examples` directory that was just created by the previous git command." + "## Get Example Code\n", + "\n", + "We will now install `git` and use it to download the example code into your home directory. For those of you who are unfamiliar with git, it is a way to collaboratively manage files and we will only be using it to download the example that we will be using. " ] }, { "cell_type": "code", "execution_count": null, - "id": "7510d393", + "id": "0fc21993", "metadata": {}, "outputs": [], "source": [ - "cd ~/CLASS-Examples/landsat/" + "sudo apt-get install --yes git" ] }, { - "cell_type": "markdown", - "id": "80788411", + "cell_type": "code", + "execution_count": null, + "id": "3d30db18", "metadata": {}, + "outputs": [], "source": [ - "Your prompt should now change showing the current directory as follows.\n", - "```\n", - "learner@essentials:~/CLASS-Examples/landsat$\n", - "```" + "cd ~" ] }, { "cell_type": "code", "execution_count": null, - "id": "6f213bfd", + "id": "4a86fa33", "metadata": {}, "outputs": [], "source": [ - "ls -l" + "git clone https://github.internet2.edu/CLASS/CLASS-Examples.git" ] }, { "cell_type": "markdown", - "id": "13168790", + "id": "f3133c48", "metadata": {}, "source": [ - "## Access the bucket\n", - "\n", - "First test that our tools are working and that we can access the public bucket that we will be using." + "We now change the current directory to the `azure-landsat` directory in the `CLASS-Examples` directory that was just created by the previous git command." ] }, { "cell_type": "code", "execution_count": null, - "id": "245f32a2", + "id": "7510d393", "metadata": {}, "outputs": [], "source": [ - "gsutil ls gs://gcp-public-data-landsat/" + "cd ~/CLASS-Examples/azure-landsat/" ] }, { "cell_type": "markdown", - "id": "9153424d", + "id": "fb8a8f4b", "metadata": {}, "source": [ - "The index file is a list of all the files in the bucket and we can use it to search and filter files.\n", - "\n", - "We will get the index and uncompress the file placing it in the `data/` directory (this is ignored by git). This should take around 2 min with a `e2-medium` instance in the `us-west2` region." + "List the contents of the directory with the following command." ] }, { "cell_type": "code", "execution_count": null, - "id": "e4ffd7e4", + "id": "6f213bfd", "metadata": {}, "outputs": [], "source": [ - "mkdir -v data" + "ls -l" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "f90b7dc4", + "cell_type": "markdown", + "id": "6bb70256", "metadata": {}, - "outputs": [], "source": [ - "gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/" + "Verify that *landsat-hls-azure.py* & *landsat-env* are listed." ] }, { "cell_type": "markdown", - "id": "0837cf43", + "id": "0aa4805f", "metadata": {}, "source": [ - "We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using." + "Run the *landsat-env* file to setup the environment by installing the \n", + "packages needed for the data analysis workflow." ] }, { "cell_type": "code", "execution_count": null, - "id": "9ed5a46b", + "id": "eb1aff28", "metadata": {}, "outputs": [], "source": [ - "gzip -df data/index.csv.gz" + "#./landsat-env" ] }, { "cell_type": "markdown", - "id": "75faa74a-b458-472b-9018-034c89e75315", + "id": "6be82619", "metadata": {}, "source": [ - "Again, check what happened." + "This will install the following packages." ] }, { "cell_type": "code", "execution_count": null, - "id": "5f4031b0", + "id": "e3930da6", "metadata": {}, "outputs": [], "source": [ - "ls -lh data" + "sudo apt install python3-pip -y\n", + "pip3 install --upgrade pip\n", + "pip3 install numpy\n", + "pip3 install matplotlib\n", + "pip3 install pandas\n", + "pip3 install rasterio\n", + "pip3 install azure.storage.blob" ] }, { "cell_type": "markdown", - "id": "fcde8334-f58d-4c3d-995a-2491be0f95ea", - "metadata": {}, - "source": [ - "We will now explore the data. The `head` command simply displays the first few lines of the file." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "990f4c3f", + "id": "9d459eb2", "metadata": {}, - "outputs": [], "source": [ - "head --lines=4 data/index.csv" + "## Get and Process landsat data\n", + "\n", + "Lets take a quick look at the python scrip *landsat-hls-azure.py*. This code will find the HLS tile coresponding \n", + "to a given latatue and longitude at a specific time. It will then produce a .tiff image of the tile. We will then combine images with different \n", + "bands to create a composite image of the area of interest. " ] }, { "cell_type": "markdown", - "id": "532e6da3-302a-4e8a-8570-752995f30f1d", + "id": "79a3ed1f", "metadata": {}, "source": [ - "## Search for Data\n", - "\n", - "We can see the data is well formed and what we expect. We will now use this data to download data related to a specific point and for the Landsat 8. The following script does a simple filter." + "Run the analysis file with the following command." ] }, { "cell_type": "code", "execution_count": null, - "id": "64ddd71e", + "id": "24a03d9c", "metadata": {}, "outputs": [], "source": [ - "cat search.py" + "#./landsat-hls-azure.py" ] }, { "cell_type": "markdown", - "id": "4aa3de47-3dd4-4a0f-9f07-f2f004de7054", + "id": "6ea30b3c", "metadata": {}, "source": [ - "We can see that the actual search data comes from the file `search.json`. The program reads the data from the standard input and iterates over all rows in the CSV file. It filters the results for which the image contains the pint and prints out the bucket URL for them. We are interested in all products that contain the Burr Oak Tree." + "This will product the file *test.png*. This is an image file of the tile we reqeusted." ] }, { "cell_type": "code", "execution_count": null, - "id": "b7d84cb5", + "id": "b2e77765", "metadata": {}, "outputs": [], "source": [ - "cat search.json" - ] - }, - { - "cell_type": "markdown", - "id": "cbb27235-6bc4-4eb6-b668-5c30427a28b8", - "metadata": {}, - "source": [ - "Now lets test this on a subset of the data." + "ls -l" ] }, { "cell_type": "code", "execution_count": null, - "id": "a201fd0e", + "id": "f423488b", "metadata": {}, "outputs": [], "source": [ - "head --lines=200000 data/index.csv | python3 search.py" + "test.png" ] }, { "cell_type": "markdown", - "id": "a76f24f8-3b2d-4c0d-880b-f2911b9d9b84", + "id": "19158868", "metadata": {}, "source": [ - "## Download the Data\n", + "Install the azure CLI by executing the the curl command below.\n", "\n", - "Now that we have a list of folders we are interested, we will now download them with a simple script that takes bucket addresses (URL's) and downloads them with the `gsutil` program." + "This script does three fundamental things: #\n", + " 1. Add Microsoft's GPG Key has a trusted source of apt packages. #\n", + " 2. Add Microsoft's repositories as a source for apt packages. #\n", + " 3. Installs the Azure CLI from those repositories. " ] }, { "cell_type": "code", "execution_count": null, - "id": "3ac78b3c", + "id": "dca9b76f", "metadata": {}, "outputs": [], "source": [ - "cat download.sh" + "curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash" ] }, { "cell_type": "markdown", - "id": "a02100bd-f8c5-42dd-975e-a9eb5369bc81", + "id": "ab4c265a", "metadata": {}, "source": [ - "Get the first 2 datasets" + "Define some environment variables" ] }, { "cell_type": "code", "execution_count": null, - "id": "2e93f083", - "metadata": { - "tags": [] - }, + "id": "6fc41936", + "metadata": {}, "outputs": [], "source": [ - "head --lines=200000 data/index.csv | python3 search.py | head --lines=2 | bash download.sh" + "RG=\"demo_rg\"\n", + "LOCATION=\"eastus\"\n", + "BLOB_CON=\"demo_blob_cont\"\n", + "STGE_ACCT=\"demo_stor_ac\"\n", + "CTR=\"demo_stge_ctr\"" ] }, { "cell_type": "markdown", - "id": "14c23aab-a6e8-439d-8e55-44b7148f74d5", + "id": "3b4f9136", "metadata": {}, "source": [ - "Check that the data was downloaded" + "login into azure and verify info" ] }, { "cell_type": "code", "execution_count": null, - "id": "271c7c4c", + "id": "cdcbd1e2", "metadata": {}, "outputs": [], "source": [ - "ls -l data" - ] - }, - { - "cell_type": "markdown", - "id": "6073d5f6-68ef-41df-8044-73f221ce8780", - "metadata": {}, - "source": [ - "## Processing the Data\n", - "\n", - "We will now use simple script to combine the color bands and export it as a PNG" + "az login\n", + "az group list –output table\n", + "az storage account list --output table\n", + "az ad signed-in-user show --query objectId -o tsv | az role assignment create --role \"Storage Blob Data Contributor\" --assignee @- --scope \"/subscriptions/ae186240-1b93-42b5-bb33-e9b8c4e527de/resourceGroups/demo_rg/providers/Microsoft.Storage/storageAccounts/landsatdemosa\"" ] }, { "cell_type": "code", "execution_count": null, - "id": "e41b58c6", + "id": "3b89ddb9", "metadata": {}, "outputs": [], "source": [ - "cat combine.py" + "Use the existing container created ealier or create a new one.\n", + "\n", + "List existing containers" ] }, { "cell_type": "code", "execution_count": null, - "id": "194c8cef", + "id": "ab37a316", "metadata": {}, "outputs": [], "source": [ - "python3 combine.py\n", - "/bin/true # ignore this line used for jupyter" + "az storage container list --account-name $STGE_ACCT --auth-mode login" ] }, { "cell_type": "markdown", - "id": "52ada3ec-26cd-4e8b-94c2-8ed33e98be6e", + "id": "7c9f97db", "metadata": {}, "source": [ - "Oops, let's install the library (note: the output will be slightly different due to how this Lesson is built)\n", - "and while we are at it we also need to create the output directory." + "Create storage account if one doen't already exist" ] }, { "cell_type": "code", "execution_count": null, - "id": "659ed9d0", - "metadata": { - "tags": [] - }, + "id": "19ea43e7", + "metadata": {}, "outputs": [], "source": [ - "mkdir -v output/\n", - "sudo apt-get install python3-rasterio --yes" + "az storage container create --account-name $STGE_ACCT --name $BLOB_CON --auth-mode login" ] }, { "cell_type": "code", "execution_count": null, - "id": "45a25550", + "id": "c6047945", "metadata": {}, "outputs": [], "source": [ - "/usr/bin/python3 combine.py" - ] - }, - { - "cell_type": "markdown", - "id": "4199910a-1056-455c-ae9d-669296441aed", - "metadata": {}, - "source": [ - "Note, the \"ERROR 4\" is expected." + "az storage container list --account-name $STGE_ACCT" ] }, { "cell_type": "markdown", - "id": "a3a89af5-993e-45d6-9e7c-2ca0a61dff7a", + "id": "f3640eaa", "metadata": {}, "source": [ - "Verify the results were created." + "Upload the image we just created to the storage container" ] }, { "cell_type": "code", "execution_count": null, - "id": "7f3f6f0b", + "id": "8291f50e", "metadata": {}, "outputs": [], "source": [ - "ls -lh output" + "az storage blob upload --account-name $STGE_ACCT --container-name $BLOB_CON --name test.png --file test.png --auth-mode login" ] }, { "cell_type": "markdown", - "id": "c0257075-537c-4510-bafd-72e9756db17b", + "id": "27d93fe6", "metadata": {}, "source": [ - "## Exporting the Results\n", - "\n", - "Now that we have the output data we will create a bucket to place the results. We will first create a bucket with a reasonable set of options.\n", - "\n", - "We fisrt store the bucket name in the `BUCKET` environment variable for future use. This time we will specify a realistic set of options for a private bucket used for computation.\n", - "\n", - "Options (run `gsutil mb --help` for more information):\n", - " * `-b on` specifies uniform bucket-level access.\n", - " * `-l $REGION` puts the data in a specific region for lower cost and lower latency.\n", - " * `--pap enforced` turns on public access prevention to help keep data private. \n", - " \n", - "The uniform bucket level access (Bucket Policy Only enabled: true) puts the data access permissions (ACL) on the entire bucket, not on each object in the bucket. This makes the permissions obvious and makes security much more predictable.\n", - " \n", - "As usual, we must set our environment. In this case we also set a `REGION` environment variable to indicate where in the world we want the data to be stored.\n" + "Verify that the blob is in blob container" ] }, { "cell_type": "code", "execution_count": null, - "id": "715291b5", + "id": "4ef95e0c", "metadata": {}, "outputs": [], "source": [ - "BUCKET=\"essentials-${USER}-$(date +%F)\"\n", - "REGION=\"us-west2\"\n", - "echo \"bucket: $BUCKET region: $REGION\"" + "az storage blob list --account-name $STGE_ACCT --container-name $BLOB_CON --output table --auth-mode login" ] }, { "cell_type": "markdown", - "id": "920ff5ef", + "id": "4817719a", "metadata": {}, "source": [ - "gsutil mb -b on -l $REGION --pap enforced \"gs://$BUCKET\"" + "Now go back to the console. Open the hamburger menu and navigate to the Storage Account blade. In the right hand\n", + "menu under Data Storage, click on Containers. On this page you should see the container we just created, click on \n", + "it. On the container page, click the *Change access level* on the menu towards the top of the page. Click \n", + "on the *Public access level* pull down menu then select *Blob (anonymous read access for blobs only)*. Note the \n", + "message warning you that blobs in this container will have anonymous public access, Click *OK*. Now click on the \n", + "blob (test.png) that we uploaded." ] }, { "cell_type": "markdown", - "id": "22cc1043-386b-4990-b3dc-62cbdd7ba133", + "id": "467ad988", "metadata": {}, "source": [ - "Now copy the output data to the bucket. The `-r` flag recursively copies the output directory and `-m` copies the files in parallel. Note the locations of the `-m` and `-r` switches as they apply globally and to the `cp` command respectively." + "Now delete the blob (image,png file)" ] }, { "cell_type": "code", "execution_count": null, - "id": "9340b10d", + "id": "7a43446d", "metadata": {}, "outputs": [], "source": [ - "gsutil -m cp -r \"output\" \"gs://$BUCKET\"" - ] - }, - { - "cell_type": "markdown", - "id": "857861ca-9721-4a1e-a4ae-efffe2143b43", - "metadata": {}, - "source": [ - "Verify that the output was uploaded." + "az storage blob delete --account-name $STGE_ACCT --container-name landsat-demo-cont --name test.png --auth-mode login" ] }, { "cell_type": "code", "execution_count": null, - "id": "754c7eb4", + "id": "e6faacab", "metadata": {}, "outputs": [], "source": [ - "gsutil ls \"gs://$BUCKET\"" + "Delete the storage container" ] }, { "cell_type": "code", "execution_count": null, - "id": "0e927a0a", + "id": "9f18856d", "metadata": {}, "outputs": [], "source": [ - "gsutil ls -lh \"gs://$BUCKET/output\"" + "az storage container delete --account-name $STGE_ACCT --name $BLOB_CON --auth-mode login" ] }, { "cell_type": "markdown", - "id": "66a41f97-3834-4d09-bfe5-c3f224628842", + "id": "87091cc6", "metadata": {}, "source": [ - "## Viewing the Results\n", - "\n", - "You now can view the results by using the Google Cloud Platform Web Console and navigating to \"Cloud Storage\", selecting the bucket, and then the result object you wish to view (select the `.png` file). You will need to click the \"Preview\" button given the large size of the image." + "Delete the storage account" ] }, { - "cell_type": "markdown", - "id": "a1c11268-f389-405f-8d96-3c319a49b882", - "metadata": {}, - "source": [ - "## Sharing Results\n", - "\n", - "In order to share resources outside a project we must use the Identity Access Management service. This is a powerful tool to grant and restrict access to resources, and if not done correctly It can have serious consequences. **Incorrect permissions can lead to exposure of sensitive data, destruction of data, and authorized use of resources that can result in huge bills.** When in doubt, seek help.\n", - "\n", - "The question \"What access is really needed?\" is the **Principal of Least Privilege** and is a major cornerstone of security. We need to determine the lowest set of permissions or roles that is needed. In our case we wish to grant the \"Collaborator\" the \"Viewer\" access to the \"results bucket\". This will allow them to view, list, and download all objects in the bucket. This illustrates that for a **resource** a **member** (identity) is granted a **permission** (think of it in this order). Together this is called a **policy**. Google also uses \"Roles\" as a collection of predefined and managed permissions." - ] - }, - { - "cell_type": "markdown", - "id": "85237d4a", - "metadata": { - "tags": [] - }, - "source": [ - "We will now add Members to a Bucket using the Web Console. We will use the Web Console to interactively build the policy binding by doing the following:\n", - " * Navigation Menu -> Storage/Cloud Storage -> Browser -> Click on the Bucket Name (**Bucket Details**) -> Select the **Permissions** tab -> Click **Add** next to \"Permissions\" above the permissions list.\n", - " * In the \"New Principals\" box add the Identity for the collaborator (another individual) as directed by the instructor.\n", - " * Select the \"Storage Object Viewer\" by typing \"Storage Object Viewer\" in the filter and then selecting \"Storage Object Viewer\". Do not use any \"Legacy Storage\" roles.\n", - " * Click \"Save\" to save the policy.\n", - " \n", - " * Verify the policy is listed in the \"Permissions\" table on the \"Bucket Details\" page (you should now be on this page).\n", - "\n", - "*Advanced Note: Changes in permissions will show up on the **Home** -> **Activity** page.*" - ] - }, - { - "cell_type": "markdown", - "id": "6a191922-7d5b-4125-9812-181eb056f86c", - "metadata": { - "tags": [] - }, - "source": [ - "Collaborators should now be able to see the contents of the bucket by explicitly naming it as shown below.\n", - "```\n", - "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17\n", - "gs://essentials-learner-2021-12-17/output/\n", - "\n", - "student231@cloudshell:~ (t-monument-315019)$ gsutil ls gs://essentials-learner-2021-12-17/output\n", - "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png\n", - "gs://essentials-learner-2021-12-17/output/result-LC08_L1TP_025033_20211010_20211018_01_T1.png.aux.xml\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "40d9ab41-8920-45f2-8218-550baac5b069", + "cell_type": "code", + "execution_count": null, + "id": "2c21f537", "metadata": {}, + "outputs": [], "source": [ - "## Cleanup\n", - "\n", - "We will now leave the resources running in order to learn more about monitoring costs and will clean up all the resources as the end of Lesson. **Don't forget to do this!**" + "az storage account delete -n $STGE_ACCT" ] } ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -659,7 +541,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.8" + "version": "3.9.7" } }, "nbformat": 4, diff --git a/content/Azure/images/Accessing_Landsat_data_Azure b/content/Azure/images/Accessing_Landsat_data_Azure new file mode 100644 index 0000000..68e3702 --- /dev/null +++ b/content/Azure/images/Accessing_Landsat_data_Azure @@ -0,0 +1,178 @@ +#/usr/bin python3 +# https://nbviewer.org/github/microsoft/AIforEarthDataSets/blob/main/data/landsat-8.ipynb + +import os +import datetime +import numpy as np +import matplotlib.pyplot as plt +import rasterio +import rasterio.features +from rasterio.windows import Window +from pyproj import Transformer + +from azure.storage.blob import ContainerClient +from cmr import GranuleQuery + +from lxml import etree +import unicodedata + +# Let's take a look at an area near Auckland, New Zealand +query_lon = 174.92446857141755 +query_lat = -36.89307045544658 + +query_short_name = 'Landsat_8_OLI_TIRS_C1' + +query_start_date = datetime.datetime(2020, 1, 1, 0, 0, 0) +query_end_date = datetime.datetime(2020, 12, 1, 0, 0, 0) + +# This should be a text file with a SAS token for the Landsat container on the first line +sas_file = os.path.expanduser('~/tokens/landsat_ro_sas.txt') + +# Maps instrument names as they appear in CMR results to the short names used in filenames +instrument_mapping = { + 'Landsat 8 Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) Collection 1 V1': + 'oli-tirs'} + +# Choose bands for the RGB composite we're going to make +rgb_bands = ['B3','B2','B1'] + +# Normalization value for rendering +composite_norm_value = 25000 + +# Select a thumbnail for thumbnail rendering (six are available for Landsat C2 files) +thumbnail_index = -3 + +# When rendering whole images, how much should we downscale? +dsfactor = 10 + +# We're going to render a nice image at the end, set the size and "zoom" +target_size = [800,400] +oversample = 3.0 + +lines = [] +with open(sas_file,'r') as f: + lines = f.readlines() +assert len(lines) >= 1 +sas_token = lines[0].strip() + +storage_account_name = 'landsateuwest' +container_name = 'landsat-c2' +storage_account_url = 'https://' + storage_account_name + '.blob.core.windows.net/' + +container_client = ContainerClient(account_url=storage_account_url, + container_name=container_name, + credential=sas_token) +api = GranuleQuery() +json_granules = api.short_name(query_short_name).point(query_lon,query_lat).temporal(query_start_date, query_end_date).get() + +print('Found {} granules'.format(len(json_granules))) + +# Retrieve results in echo10 format, which is less convenient than the default .json, but +# includes cloud cover information, which the .json results don't (for Landsat) +api = GranuleQuery() +api.parameters( + short_name=query_short_name, + point=(query_lon, query_lat), + temporal=(query_start_date,query_end_date) +) +xml_results = api.format('echo10').get(len(json_granules)) +xml = xml_results[0] +xml = unicodedata.normalize('NFKD', xml).encode('ascii', 'ignore') +tree = etree.fromstring(xml) +xml_granules = tree.findall("result/Granule") + +lowest_cloud_cover = None +granule_name = None + +for granule in xml_granules: + attributes = granule.find('AdditionalAttributes') + for a in attributes: + name = a.find('Name').text + if name == 'LandCloudCover' or name == 'LandCloudCoverPct': + cc = float(a.find('Values').find('Value').text) + if lowest_cloud_cover is None or cc < lowest_cloud_cover: + lowest_cloud_cover = cc + granule_name = granule.find('GranuleUR').text + break +# Find the json-formatted granule with the same name +granule = [g for g in json_granules if g['title'] == granule_name] +assert len(granule) == 1 +granule = granule[0] + +# E.g. 'LE07_L1TP_074086_20200328_20200424_01_T1' +granule_id = granule['title'] +print('Accessing tile {}'.format(granule_id)) + +level = 'level-2' +category = 'standard' +sensor = instrument_mapping[granule['dataset_id']] + +# E.g., 2020-01-03T19:01:46.557Z +date = granule['time_start'] +year = date[0:4] +month = date[5:7] +day = date[8:10] + +path = granule_id[10:13] +row = granule_id[13:16] + +row_folder = '/'.join([level,category,sensor,year,path,row]) + +# E.g. 01152004 +granule_date_string = granule_id[11:19] + +granule_month = granule_date_string[0:2] +granule_day = granule_date_string[2:4] +granule_year = granule_date_string[4:8] + +# E.g. LC08_L1TP_047027_20200103 +scene_prefix = granule_id[0:25] + +# E.g. LC08_L2SP_047027_20200103 +scene_prefix = scene_prefix[0:5] + 'L2SP' + scene_prefix[9:] + +azure_scene_prefix = row_folder + '/' + scene_prefix + +generator = container_client.list_blobs(name_starts_with=azure_scene_prefix) +image_paths = [blob.name for blob in generator if blob.name.endswith('.TIF')] + +print('Found {} images:'.format(len(image_paths))) +for fn in image_paths: + print(fn.split('/')[-1]) + +azure_urls = [storage_account_url + container_name + '/' + p + sas_token for p in image_paths] + +# From https://www.usgs.gov/media/images/common-landsat-band-rgb-composites +rgb_bands = ['B3','B2','B1'] +rgb_urls = [] +for band_name in rgb_bands: + rgb_urls.append([s for s in azure_urls if band_name + '.TIF' in s][0]) + +thumbnail_data = [] + +# url = rgb_urls[0] +for url in rgb_urls: + + # From: + # + # https://automating-gis-processes.github.io/CSC/notebooks/L5/read-cogs.html + with rasterio.open(url) as raster: + + # List of overviews from biggest to smallest + oviews = raster.overviews(1) + + # Retrieve a small-ish thumbnail (six are available for Landsat files) + decimation_level = oviews[thumbnail_index] + h = int(raster.height/decimation_level) + w = int(raster.width/decimation_level) + + thumbnail_channel = raster.read(1, out_shape=(1, h, w)) / composite_norm_value + thumbnail_data.append(thumbnail_channel) + +rgb = np.dstack((thumbnail_data[0],thumbnail_data[1],thumbnail_data[2])) +np.clip(rgb,0,1,rgb) + +dpi = 100; fig = plt.figure(frameon=False,figsize=(w/dpi,h/dpi),dpi=dpi) +ax = plt.Axes(fig,[0., 0., 1., 1.]); ax.set_axis_off(); fig.add_axes(ax) + +plt.imshow(rgb); diff --git a/content/Azure/images/landsat-hls-asure.py b/content/Azure/images/landsat-hls-asure.py new file mode 100755 index 0000000..4fe7688 --- /dev/null +++ b/content/Azure/images/landsat-hls-asure.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +# need to install - pip install azure-storage-blob +# https://nbviewer.org/github/microsoft/AIforEarthDataSets/blob/main/data/hls.ipynb + +# Environment setup +import requests +import numpy as np +import io +import matplotlib.pyplot as plt +import pandas as pd + +import rasterio +from azure.storage.blob import ContainerClient + +container_name = 'hls' +storage_account_name = 'hlssa' +storage_account_url = 'https://' + storage_account_name + '.blob.core.windows.net/' +hls_blob_root = storage_account_url + container_name + +# This file is provided by NASA; it indicates the lat/lon extents of each +# hls tile. +# +# The file originally comes from: +# +# https://hls.gsfc.nasa.gov/wp-content/uploads/2016/10/S2_TilingSystem2-1.txt +# +# ...but as of 8/2019, there is a bug with the column names in the original file, so we +# access a copy with corrected column names. +hls_tile_extents_url = 'https://ai4edatasetspublicassets.blob.core.windows.net/assets/S2_TilingSystem2-1.txt' + +# Load this file into a table, where each row is: +# +# Tile ID, Xstart, Ystart, UZ, EPSG, MinLon, MaxLon, MinLon, MaxLon +s = requests.get(hls_tile_extents_url).content +hls_tile_extents = pd.read_csv(io.StringIO(s.decode('utf-8')),delimiter=r'\s+') +print('Read tile extents for {} tiles'.format(len(hls_tile_extents))) + +hls_container_client = ContainerClient(account_url=storage_account_url, + container_name=container_name, + credential=None) + +# Functions + +def list_available_tiles(prefix): + """ + List all blobs in an Azure blob container matching a prefix. + + We'll use this to query tiles by location and year. + """ + + files = [] + generator = hls_container_client.list_blobs(name_starts_with=prefix) + for blob in generator: + files.append(blob.name) + return files + + +def lat_lon_to_hls_tile_id(lat,lon): + """ + Get the hls tile ID for a given lat/lon coordinate pair. + """ + + found_matching_tile = False + + for i_row,row in hls_tile_extents.iterrows(): + found_matching_tile = lat >= row.MinLat and lat <= row.MaxLat \ + and lon >= row.MinLon and lon <= row.MaxLon + if found_matching_tile: + break + + if not found_matching_tile: + return None + else: + return row.TilID + +# Build a tile path from components, including lat/lon lookup +# Near Bear Lake, UT +# lat = 41.89655047211277; lon = -111.4132464403312 +lat = 39.768402; lon = -86.158066 + +tile_id = lat_lon_to_hls_tile_id(lat,lon) +print('Using Tile ID {}'.format(tile_id)) + +year = '2019' +daynum = '001' # 1-indexed day-of-year +product = 'S30' # 'S30' for Sentinel, 'L30' for Landsat +sband = '_01' +version = 'v1.4' # Currently always v1.4 + +blob_name = product + '/HLS.' + product + '.T' + tile_id + '.' + year + daynum + '.' + version \ + + sband + '.tif' +sentinel_url = hls_blob_root + '/' + blob_name +print('Constructed tile URL:\n{}'.format(sentinel_url)) + +# Display an RGB composite of the Sentinel-2 image we just chose +# Bands 2, 3, and 4 are B, G, and R in Sentinel-2 HLS images +composite_bands = [4,3,2] + +urls = [sentinel_url.replace(sband,'_' + str(b).zfill(2)) for b in composite_bands] +print('Rendering URLs:') +for s in urls: + print(s) + +image_data = [] + +composite_downsample_factor = 5 +composite_norm_value = 15000 + +for fn in urls: + with rasterio.open(fn,'r') as raster: + h = int(raster.height/composite_downsample_factor) + w = int(raster.width/composite_downsample_factor) + band_array = raster.read(1, out_shape=(1, h, w)) + raster.close() + band_array = band_array / composite_norm_value + image_data.append(band_array) + +rgb = np.dstack((image_data[0],image_data[1],image_data[2])) +np.clip(rgb,0,1,rgb) + +dpi = 100; fig = plt.figure(frameon=False,figsize=(w/dpi,h/dpi),dpi=dpi) +ax = plt.Axes(fig,[0., 0., 1., 1.]); ax.set_axis_off(); fig.add_axes(ax) +plt.imshow(rgb); +plt.imsave('test.png', rgb) + +i_daynum = int(daynum) +product = 'L30' + +# Try days sequentially until we find one where there's a Landsat image for this day +while True: + + daynum = str(i_daynum).zfill(3) + prefix = product + '/HLS.' + product + '.T' + tile_id + '.' + year + daynum + print('Finding tiles with prefix {}'.format(prefix)) + matches = list_available_tiles(prefix) + + if len(matches) == 0: + print('No matching tiles') + i_daynum += 1 + else: + break + +blob_name = matches[0] +landsat_url = hls_blob_root + '/' + blob_name +print('Found {} matching tiles, e.g.:\n{}'.format(len(matches),landsat_url)) + +# Display a composite of the Landsat image we found +urls = [landsat_url.replace(sband,'_' + str(b).zfill(2)) for b in composite_bands] +print('Rendering URLs:') +for s in urls: + print(s) + +image_data = [] +for fn in urls: + with rasterio.open(fn,'r') as raster: + h = int(raster.height/composite_downsample_factor) + w = int(raster.width/composite_downsample_factor) + band_array = raster.read(1, out_shape=(1, h, w)) + raster.close() + band_array = band_array / composite_norm_value + image_data.append(band_array) + +rgb = np.dstack((image_data[0],image_data[1],image_data[2])) +np.clip(rgb,0,1,rgb) + +dpi = 100; fig = plt.figure(frameon=False,figsize=(w/dpi,h/dpi),dpi=dpi) +ax = plt.Axes(fig,[0., 0., 1., 1.]); ax.set_axis_off(); fig.add_axes(ax) +plt.imshow(rgb); +plt.show()