Putting It All Together
Contents
Putting It All Together#
Overview
Teaching: 60 min
Exercises: 8 min
Questions:
Can you show me an example?
Objectives:
Create a simple workflow using a cloud VM and cloud object storage.
Update a VM Instance software for important security updates.
Create a VM Instance with the appropriate storage scope.
Create a private regional storage bucket with appropriate security settings.
Using the CLI to install software.
Download source code using git
Retrieve data from a bucket
Run the python analysis code
Store results in a bucket
View the results in the Cloud Storage browser.
A Research Computational and Data Workflow - Drew’s story#
Drew needs to do some analysis on the data. They need data (satellite images stored in the cloud), computational resources (a virtual machine), some software (we will supply this), and a place to store the results (Cloud Storage). We will assemble and process all these parts in the cloud with a simple example.
Tip
If you get disconnected, you will need to reconnect to the virtual machine (gcloud compute ssh essentials) and re-run the following commands:
BUCKET="essentials-$USER-$(date +%F)"
REGION="us-west2"
echo "bucket: $BUCKET region: $REGION"
cd ~/CLASS-Examples/landsat
Create a VM#
Since we only create resources as we need them in the cloud, we will now create a new virtual machine (VM)instance for Drew to use for their analysis.
We will do this as an exercise to give you practice in creating resources. Since the virtual machine will need access to storage on your behalf, you will need to change the access scope to give Full access to the Storage API to the virtual machine.
Before you do anything, think about (and check) your who, where, and what!
Instructor: place the exercise instructions below on the screen
When you are done feel free to connect to the virtual machine on your own for additional practice. Once everyone has created their VM we will connect to the machine as described below.
Exercise 6
Using the console navigate to the “Compute Engine” service and create a new VM with the following properties.
Set the VM instance name to “essentials”
Select a bit larger VM instance by changing the machine type to “e2-standard-2”.
Set the VM instance API access for “Storage” to “Full”. This can be found under “Identity and API access” on the “create an instance” page and then selecting “Set access for each API” and change “Storage” to “Full”. This will allow the VM to create, read, write, and delete all storage buckets in the project”
Please verify that the virtual machine was created as above. If you are unsure delete the virtual machine instance and create it again.
Verify that the Compute Engine default service account is being used.
Change Access scopes to Set access for each API
And set Storage to Full.
Connect to the VM#
Now login to the new virtual machine instance by opening up the Cloud Shell (don’t forget to check your who, where, and what) and then running the following command:
gcloud compute ssh essentials
If prompted for a zone select n to find it automatically. You can see an example session below.
learner@cloudshell:~ (just-armor-301114)$ gcloud compute ssh essentials
Did you mean zone [us-central1-b] for instance: [essentials] (Y/n)? n
No zone specified. Using zone [us-west2-c] for instance: [essentials].
Linux essentials 4.19.0-18-cloud-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Nov 9 20:12:49 2021 from 34.133.99.196
learner@essentials:~$
Secure the VM#
We first make sure that the VM is up to date with the latest security patches by running the following commands. Note: the sudo unattended-upgrades command only installs important security packages and does not upgrade all packages.
sudo apt update
sudo unattended-upgrades
Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Hit:4 http://deb.debian.org/debian buster-backports InRelease
Hit:5 http://packages.cloud.google.com/apt cloud-sdk-buster InRelease
Get:6 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster InRelease [5553 B]
Hit:7 http://packages.cloud.google.com/apt google-compute-engine-buster-stable InRelease
Get:8 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-buster/main amd64 Packages [389 B]
Fetched 5942 B in 1s (7839 B/s)33m0m
Reading package lists... Done
Building dependency tree
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
Setup Storage#
Before we do any work we will first create a bucket to place the results with a reasonable set of options. We do this first to make sure we can store the results when we are done, it is easier to fix problems now than later.
We first store the bucket name in the BUCKET environment variable for future use. This time we will specify a realistic set of options for a private bucket used for computation.
Options (run gsutil mb --help for more information):
-b onspecifies uniform bucket-level access.-l $REGIONputs the data in a specific region for lower cost and lower latency.--pap enforcedturns on public access prevention to help keep data private.
The uniform bucket level access (Bucket Policy Only enabled: true) puts the data access permissions (ACL) on the entire bucket, not on each object in the bucket. This makes the permissions obvious and makes security much more predictable.
As usual, we must set our environment. In this case we also set a REGION environment variable to indicate where in the world we want the data to be stored.
BUCKET="essentials-$USER-$(date +%F)"
REGION="us-west2"
echo "bucket: $BUCKET region: $REGION"
bucket: essentials-learner-2022-02-14 region: us-west2
gsutil mb -b on -l $REGION --pap enforced gs://$BUCKET
Creating gs://essentials-learner-2022-02-14/...
And verify the bucket was created
gsutil ls -b gs://$BUCKET
gs://essentials-learner-2022-02-14/
Get Example Code#
We will now install git and use it to download the example code into your home directory. For those of you who are unfamiliar with git, it is a way to collaboratively manage files and we will only be using it to download the example that we will be using.
sudo apt-get install --yes git
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
git is already the newest version (1:2.30.2-1).
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.
cd ~
git clone https://github.internet2.edu/CLASS/CLASS-Examples.git
Cloning into 'CLASS-Examples'...
remote: Enumerating objects: 117, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 117 (delta 1), reused 9 (delta 1), pack-reused 107
Receiving objects: 100% (117/117), 20.66 KiB | 542.00 KiB/s, done.
Resolving deltas: 100% (44/44), done.
We now change the current directory to the landsat directory in the CLASS-Examples directory that was just created by the previous git command.
cd ~/CLASS-Examples/landsat/
Your prompt should now change showing the current directory as follows.
learner@essentials:~/CLASS-Examples/landsat$
ls -l
total 32
-rw-r--r-- 1 learner learner 964 May 20 17:41 ReadMe.md
-rw-r--r-- 1 learner learner 72 May 20 17:41 clean.sh
-rw-r--r-- 1 learner learner 280 May 20 17:41 download.sh
-rw-r--r-- 1 learner learner 113 May 20 17:41 get-data.sh
-rw-r--r-- 1 learner learner 345 May 20 17:41 get-index.sh
-rw-r--r-- 1 learner learner 613 May 20 17:41 process_sat.py
-rw-r--r-- 1 learner learner 95 May 20 17:41 search.json
-rw-r--r-- 1 learner learner 851 May 20 17:41 search.py
Access the Bucket#
Now we need to verify that Drew has access to the analysis data.
We do this by testing that our tools are working and that we can access the public bucket that we will be using.
gsutil ls gs://gcp-public-data-landsat/
gs://gcp-public-data-landsat/index.csv.gz
gs://gcp-public-data-landsat/LC08/
gs://gcp-public-data-landsat/LE07/
gs://gcp-public-data-landsat/LM01/
gs://gcp-public-data-landsat/LM02/
gs://gcp-public-data-landsat/LM03/
gs://gcp-public-data-landsat/LM04/
gs://gcp-public-data-landsat/LM05/
gs://gcp-public-data-landsat/LO08/
gs://gcp-public-data-landsat/LT04/
gs://gcp-public-data-landsat/LT05/
gs://gcp-public-data-landsat/LT08/
Getting the Metadata#
Since the Landsat data is huge we do not, and cannot, download everything to the virtual machine. We will only analyzing a subset of the data.
We will use the the index.csv.gz file, which is a list of all the files and additional metadata in the bucket and we can use it to search and filter the data.
We will first get the index and uncompress the file placing it in the data/ directory (this is ignored by git). This should take around 2 min with a e2-medium instance in the us-west2 region.
mkdir -v data
mkdir: created directory 'data'
gsutil cp gs://gcp-public-data-landsat/index.csv.gz data/
Copying gs://gcp-public-data-landsat/index.csv.gz...
\ [1 files][731.9 MiB/731.9 MiB] 54.4 MiB/s
Operation completed over 1 objects/731.9 MiB.
We will now uncompress the index file to make it easier to use. This may take some time depending on the machine type you are using. (This is also why it is good to write scripts to do the entire process).
gzip -df data/index.csv.gz
Again, check what happened.
ls -lh data
total 2.5G
-rw-r--r-- 1 learner learner 2.5G May 20 17:42 index.csv
We will now explore the data. The head command simply displays the first few lines of the file.
head data/index.csv
SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL
LE71800592011134ASN00,LE07_L1TP_180059_20110514_20161209_01_T1,LANDSAT_7,ETM,2011-05-14,01,T1,2011-05-14T08:50:07.5251363Z,L1TP,180,59,74.0,2.39913,0.50961,18.09062,20.2487,181281962,gs://gcp-public-data-landsat/LE07/01/180/059/LE07_L1TP_180059_20110514_20161209_01_T1
LT51360422008226BKT00,LT05_L1GS_136042_20080813_20161030_01_T2,LANDSAT_5,TM,2008-08-13,01,T2,2008-08-13T04:03:49.0450690Z,L1GS,136,42,92.0,26.9495,25.03915,91.40541,93.81099,141994748,gs://gcp-public-data-landsat/LT05/01/136/042/LT05_L1GS_136042_20080813_20161030_01_T2
LE71760312020339NSG00,LE07_L1TP_176031_20201204_20201230_01_T1,LANDSAT_7,ETM,2020-12-04,01,T1,2020-12-04T07:41:11.6084536Z,L1TP,176,31,3.0,42.75649,40.7935,33.66313,36.65653,188511155,gs://gcp-public-data-landsat/LE07/01/176/031/LE07_L1TP_176031_20201204_20201230_01_T1
LE71930302019087NSG00,LE07_L1TP_193030_20190328_20190423_01_T1,LANDSAT_7,ETM,2019-03-28,01,T1,2019-03-28T09:57:50.8833259Z,L1TP,193,30,1.0,44.17399,42.20768,7.88666,10.97109,154211367,gs://gcp-public-data-landsat/LE07/01/193/030/LE07_L1TP_193030_20190328_20190423_01_T1
LM10420361973161AAA05,LM01_L1TP_042036_19730610_20180428_01_T2,LANDSAT_1,MSS,1973-06-10,01,T2,1973-06-10T17:50:11.5000000Z,L1TP,42,36,11.0,35.7197,33.79417,-116.97493,-114.47118,27823097,gs://gcp-public-data-landsat/LM01/01/042/036/LM01_L1TP_042036_19730610_20180428_01_T2
LT51780171998140KIS00,LT05_L1TP_178017_19980520_20161224_01_T1,LANDSAT_5,TM,1998-05-20,01,T1,1998-05-20T08:04:57.6290500Z,L1TP,178,17,88.0,62.53031,60.38833,39.29647,44.07323,110078792,gs://gcp-public-data-landsat/LT05/01/178/017/LT05_L1TP_178017_19980520_20161224_01_T1
LE71150302015314ASN00,LE07_L1GT_115030_20151110_20161017_01_T2,LANDSAT_7,ETM,2015-11-10,01,T2,2015-11-10T02:03:53.8882257Z,L1GT,115,30,94.0,44.16222,42.18812,128.52473,131.58518,199231999,gs://gcp-public-data-landsat/LE07/01/115/030/LE07_L1GT_115030_20151110_20161017_01_T2
LE71620272018219NPA00,LE07_L1TP_162027_20180807_20180902_01_T1,LANDSAT_7,ETM,2018-08-07,01,T1,2018-08-07T06:51:16.8070952Z,L1TP,162,27,16.0,48.44773,46.45227,57.25936,60.58404,242206283,gs://gcp-public-data-landsat/LE07/01/162/027/LE07_L1TP_162027_20180807_20180902_01_T1
LC81240642016196LGN02,LC08_L1TP_124064_20160714_20180523_01_T1,LANDSAT_8,OLI_TIRS,2016-07-14,01,T1,2016-07-14T03:12:23.6248870Z,L1TP,124,64,41.35,-4.73633,-6.8367,103.1169,105.17739,916587606,gs://gcp-public-data-landsat/LC08/01/124/064/LC08_L1TP_124064_20160714_20180523_01_T1
Tip
To run the above commands in one step run
bash get-index.sh
Break (Optional)
Now our virtual machine instance is ready and we can access the code and data. Now is a great time to take a short break.
Getting the Data#
We can see the data is well formed and what we expect. We will now use this data to download data related to a specific point and limit it to only Landsat 8. The following script does a simple filter.
cat search.py
#!/usr/bin/python3
import json
import csv
import sys
# Example: Burr Oak Tree
# 38.899313,-92.464562 (Lat north+, Long west-) ; Landsat Path 025, Row 033
config=json.load(open("search.json"))
lat,lon=config['lat'],config['lon']
landsat=config['landsat']
limit=config['limit']
reader=csv.reader(sys.stdin)
header=next(reader) # skip header
for l in reader:
SCENE_ID,PRODUCT_ID,SPACECRAFT_ID,SENSOR_ID,DATE_ACQUIRED,COLLECTION_NUMBER,COLLECTION_CATEGORY,SENSING_TIME,DATA_TYPE,WRS_PATH,WRS_ROW,CLOUD_COVER,NORTH_LAT,SOUTH_LAT,WEST_LON,EAST_LON,TOTAL_SIZE,BASE_URL=l
west,east=float(WEST_LON),float(EAST_LON)
north,south=float(NORTH_LAT),float(SOUTH_LAT)
if SPACECRAFT_ID==landsat and north >= lat and south <= lat and west <= lon and east >= lon:
print(BASE_URL) # output BASE_URL
limit and sys.exit(0) # limit results
We can see that the actual search data comes from the file search.json. The program reads the data from the standard input and iterates over all rows in the CSV file. It filters the results for which the image contains the pint and prints out the bucket URL for them. We are interested in all products that contain the Burr Oak Tree.
cat search.json
{
"lat": 38.899313,
"lon": -92.464562,
"landsat": "LANDSAT_8",
"limit": true
}
Now lets test this on a subset of the data (note the ‘limit’ option).
python3 search.py < data/index.csv
gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1
Now that we have a list of folders we are interested, we will now download them with a simple script that takes bucket addresses (URL’s) and downloads them with the gsutil program.
cat download.sh
#!/bin/bash
# Read space separated URL from STDIN and download
while read -r URL ; do
echo "+++ $URL"
# -m parallel
# -n no-clobber (do not re-download data)
# -r recursive (download all the data in the specified URL)
gsutil -m cp -n -r "${URL}/" data/
done
Get the first dataset (limit option)
python3 search.py < data/index.csv | bash download.sh
+++ gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_ANG.txt...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B10.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B1.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B3.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B4.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B11.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B5.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B6.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B2.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B7.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B8.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_B9.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_BQA.TIF...
Copying gs://gcp-public-data-landsat/LC08/01/025/033/LC08_L1TP_025033_20201007_20201016_01_T1/LC08_L1TP_025033_20201007_20201016_01_T1_MTL.txt...
| [14/14 files][952.9 MiB/952.9 MiB] 100% Done 18.8 MiB/s ETA 00:00:00
Operation completed over 14 objects/952.9 MiB.
Check that the data was downloaded
ls -l data
total 2564792
drwxr-xr-x 2 learner learner 4096 May 20 17:42 LC08_L1TP_025033_20201007_20201016_01_T1
-rw-r--r-- 1 learner learner 2626336574 May 20 17:42 index.csv
Tip
To run the above analysis in one step run
bash get-data.sh
Processing the Data#
We will now use simple script to combine the color bands and export it as a PNG
cat process_sat.py
#!/usr/bin/python3
import os
import rasterio
# Open the first directory in data, could walk entire tree
for dirname, dirs, files in os.walk('data'):
source = dirs[0]
break
# Open band (B2/Blue) and copy metadata for result.
with rasterio.open("data/%s/%s_B2.TIF" % (source, source)) as band2:
meta = band2.meta
# Combine bands into PNG
meta.update(count = 3, driver='PNG')
with rasterio.open("output/result-%s.png" % source, 'w+', **meta) as output:
for i in range(1, 4):
print(source, i)
output.write_band(i,rasterio.open("data/%s/%s_B%d.TIF" % (source, source, i+3)).read(1))
This code writes to the output folder, so let’s create it
mkdir -v output/
mkdir: created directory 'output/'
python3 process_sat.py
/bin/true # ignore this line used for jupyter
Traceback (most recent call last):
File "/home/learner/CLASS-Examples/landsat/process_sat.py", line 3, in <module>
import rasterio
ModuleNotFoundError: No module named 'rasterio'
Oops, let’s install the library (note: the output will be slightly different due to how this Lesson is built).
sudo apt-get install python3-rasterio --yes
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
gdal-data libaec0 libaom0 libarmadillo10 libarpack2 libblas3 libcfitsio9
libcharls2 libdap27 libdapclient6v5 libdav1d4 libde265-0 libepsilon1
libfreexl1 libfyba0 libgdal28 libgeos-3.9.0 libgeos-c1v5 libgeotiff5
libgfortran5 libgif7 libhdf4-0-alt libhdf5-103-1 libhdf5-hl-100 libheif1
libicu67 libkmlbase1 libkmldom1 libkmlengine1 liblapack3 liblcms2-2 libltdl7
libmariadb3 libminizip1 libnetcdf18 libnspr4 libnss3 libnuma1 libodbc1
libogdi4.1 libopenjp2-7 libpoppler102 libpq5 libproj19 libqhull8.0
librttopo1 libspatialite7 libsuperlu5 libsz2 liburiparser1 libx265-192
libxerces-c3.2 libxml2 libxslt1.1 mariadb-common mysql-common odbcinst
odbcinst1debian2 poppler-data proj-bin proj-data python3-affine python3-attr
python3-bs4 python3-certifi python3-chardet python3-click
python3-click-plugins python3-cligj python3-colorama python3-html5lib
python3-lxml python3-numpy python3-pyparsing python3-six python3-snuggs
python3-soupsieve python3-webencodings
Suggested packages:
geotiff-bin gdal-bin libgeotiff-epsg libhdf4-doc libhdf4-alt-dev hdf4-tools
liblcms2-utils libmyodbc odbc-postgresql tdsodbc unixodbc-bin ogdi-bin
poppler-utils ghostscript fonts-japanese-mincho | fonts-ipafont-mincho
fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai
fonts-arphic-uming fonts-nanum python-attr-doc python3-genshi
python3-lxml-dbg python-lxml-doc gfortran python-numpy-doc python3-numpy-dbg
python3-pytest python-pyparsing-doc
The following NEW packages will be installed:
gdal-data libaec0 libaom0 libarmadillo10 libarpack2 libblas3 libcfitsio9
libcharls2 libdap27 libdapclient6v5 libdav1d4 libde265-0 libepsilon1
libfreexl1 libfyba0 libgdal28 libgeos-3.9.0 libgeos-c1v5 libgeotiff5
libgfortran5 libgif7 libhdf4-0-alt libhdf5-103-1 libhdf5-hl-100 libheif1
libicu67 libkmlbase1 libkmldom1 libkmlengine1 liblapack3 liblcms2-2 libltdl7
libmariadb3 libminizip1 libnetcdf18 libnspr4 libnss3 libnuma1 libodbc1
libogdi4.1 libopenjp2-7 libpoppler102 libpq5 libproj19 libqhull8.0
librttopo1 libspatialite7 libsuperlu5 libsz2 liburiparser1 libx265-192
libxerces-c3.2 libxml2 libxslt1.1 mariadb-common mysql-common odbcinst
odbcinst1debian2 poppler-data proj-bin proj-data python3-affine python3-attr
python3-bs4 python3-certifi python3-chardet python3-click
python3-click-plugins python3-cligj python3-colorama python3-html5lib
python3-lxml python3-numpy python3-pyparsing python3-rasterio python3-six
python3-snuggs python3-soupsieve python3-webencodings
0 upgraded, 79 newly installed, 0 to remove and 11 not upgraded.
Need to get 53.2 MB of archives.
After this operation, 211 MB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security bullseye-security/main amd64 libpq5 amd64 13.7-0+deb11u1 [180 kB]
Get:2 http://deb.debian.org/debian bullseye/main amd64 poppler-data all 0.4.10-1 [1602 kB]
Get:3 http://deb.debian.org/debian bullseye/main amd64 gdal-data all 3.2.2+dfsg-2+deb11u1 [462 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 libaec0 amd64 1.0.4-1 [20.3 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 libaom0 amd64 1.0.0.errata1-3 [1158 kB]
Get:6 http://deb.debian.org/debian bullseye/main amd64 libblas3 amd64 3.9.0-3 [153 kB]
Get:7 http://deb.debian.org/debian bullseye/main amd64 libgfortran5 amd64 10.2.1-6 [727 kB]
Get:8 http://deb.debian.org/debian bullseye/main amd64 liblapack3 amd64 3.9.0-3 [2166 kB]
Get:9 http://deb.debian.org/debian bullseye/main amd64 libarpack2 amd64 3.8.0-1 [103 kB]
Get:10 http://deb.debian.org/debian bullseye/main amd64 libsuperlu5 amd64 5.2.2+dfsg1-2 [163 kB]
Get:11 http://deb.debian.org/debian bullseye/main amd64 libarmadillo10 amd64 1:10.1.2+dfsg-6+b1 [98.8 kB]
Get:12 http://deb.debian.org/debian bullseye/main amd64 libcfitsio9 amd64 3.490-3 [554 kB]
Get:13 http://deb.debian.org/debian bullseye/main amd64 libcharls2 amd64 2.2.0+dfsg-2 [79.4 kB]
Get:14 http://deb.debian.org/debian bullseye/main amd64 libicu67 amd64 67.1-7 [8622 kB]
Get:15 http://deb.debian.org/debian bullseye/main amd64 libxml2 amd64 2.9.10+dfsg-6.7+deb11u1 [693 kB]
Get:16 http://deb.debian.org/debian bullseye/main amd64 libdap27 amd64 3.20.7-6 [566 kB]
Get:17 http://deb.debian.org/debian bullseye/main amd64 libdapclient6v5 amd64 3.20.7-6 [208 kB]
Get:18 http://deb.debian.org/debian bullseye/main amd64 libdav1d4 amd64 0.7.1-3 [333 kB]
Get:19 http://deb.debian.org/debian bullseye/main amd64 libde265-0 amd64 1.0.8-1 [242 kB]
Get:20 http://deb.debian.org/debian bullseye/main amd64 libepsilon1 amd64 0.9.2+dfsg-5 [41.9 kB]
Get:21 http://deb.debian.org/debian bullseye/main amd64 libfreexl1 amd64 1.0.6-1 [34.2 kB]
Get:22 http://deb.debian.org/debian bullseye/main amd64 libfyba0 amd64 4.1.1-7 [114 kB]
Get:23 http://deb.debian.org/debian bullseye/main amd64 libgeos-3.9.0 amd64 3.9.0-1 [921 kB]
Get:24 http://deb.debian.org/debian bullseye/main amd64 libgeos-c1v5 amd64 3.9.0-1 [377 kB]
Get:25 http://deb.debian.org/debian bullseye/main amd64 proj-data all 7.2.1-1 [7940 kB]
Get:26 http://deb.debian.org/debian bullseye/main amd64 libproj19 amd64 7.2.1-1 [1132 kB]
Get:27 http://deb.debian.org/debian bullseye/main amd64 libgeotiff5 amd64 1.6.0-1 [70.3 kB]
Get:28 http://deb.debian.org/debian bullseye/main amd64 libgif7 amd64 5.1.9-2 [45.1 kB]
Get:29 http://deb.debian.org/debian bullseye/main amd64 libhdf4-0-alt amd64 4.2.15-3 [278 kB]
Get:30 http://deb.debian.org/debian bullseye/main amd64 libsz2 amd64 1.0.4-1 [6760 B]
Get:31 http://deb.debian.org/debian bullseye/main amd64 libhdf5-103-1 amd64 1.10.6+repack-4+deb11u1 [1189 kB]
Get:32 http://deb.debian.org/debian bullseye/main amd64 libnuma1 amd64 2.0.12-1+b1 [26.3 kB]
Get:33 http://deb.debian.org/debian bullseye/main amd64 libx265-192 amd64 3.4-2 [1095 kB]
Get:34 http://deb.debian.org/debian bullseye/main amd64 libheif1 amd64 1.11.0-1 [191 kB]
Get:35 http://deb.debian.org/debian bullseye/main amd64 libminizip1 amd64 1.1-8+b1 [20.4 kB]
Get:36 http://deb.debian.org/debian bullseye/main amd64 liburiparser1 amd64 0.9.4+dfsg-1+deb11u1 [45.9 kB]
Get:37 http://deb.debian.org/debian bullseye/main amd64 libkmlbase1 amd64 1.3.0-9 [48.7 kB]
Get:38 http://deb.debian.org/debian bullseye/main amd64 libkmldom1 amd64 1.3.0-9 [156 kB]
Get:39 http://deb.debian.org/debian bullseye/main amd64 libkmlengine1 amd64 1.3.0-9 [77.7 kB]
Get:40 http://deb.debian.org/debian bullseye/main amd64 mysql-common all 5.8+1.0.7 [7464 B]
Get:41 http://deb.debian.org/debian bullseye/main amd64 mariadb-common all 1:10.5.15-0+deb11u1 [36.7 kB]
Get:42 http://deb.debian.org/debian bullseye/main amd64 libmariadb3 amd64 1:10.5.15-0+deb11u1 [176 kB]
Get:43 http://deb.debian.org/debian bullseye/main amd64 libhdf5-hl-100 amd64 1.10.6+repack-4+deb11u1 [81.8 kB]
Get:44 http://deb.debian.org/debian bullseye/main amd64 libnetcdf18 amd64 1:4.7.4-1 [399 kB]
Get:45 http://deb.debian.org/debian bullseye/main amd64 libltdl7 amd64 2.4.6-15 [391 kB]
Get:46 http://deb.debian.org/debian bullseye/main amd64 libodbc1 amd64 2.3.6-0.1+b1 [224 kB]
Get:47 http://deb.debian.org/debian bullseye/main amd64 libogdi4.1 amd64 4.1.0+ds-5 [213 kB]
Get:48 http://deb.debian.org/debian bullseye/main amd64 libopenjp2-7 amd64 2.4.0-3 [172 kB]
Get:49 http://deb.debian.org/debian bullseye/main amd64 liblcms2-2 amd64 2.12~rc1-2 [150 kB]
Get:50 http://deb.debian.org/debian bullseye/main amd64 libnspr4 amd64 2:4.29-1 [112 kB]
Get:51 http://deb.debian.org/debian bullseye/main amd64 libnss3 amd64 2:3.61-1+deb11u2 [1306 kB]
Get:52 http://deb.debian.org/debian bullseye/main amd64 libpoppler102 amd64 20.09.0-3.1 [1675 kB]
Get:53 http://deb.debian.org/debian bullseye/main amd64 libqhull8.0 amd64 2020.2-3 [246 kB]
Get:54 http://deb.debian.org/debian bullseye/main amd64 librttopo1 amd64 1.1.0-2 [180 kB]
Get:55 http://deb.debian.org/debian bullseye/main amd64 libspatialite7 amd64 5.0.1-2 [1796 kB]
Get:56 http://deb.debian.org/debian bullseye/main amd64 libxerces-c3.2 amd64 3.2.3+debian-3 [870 kB]
Get:57 http://deb.debian.org/debian bullseye/main amd64 odbcinst amd64 2.3.6-0.1+b1 [48.7 kB]
Get:58 http://deb.debian.org/debian bullseye/main amd64 odbcinst1debian2 amd64 2.3.6-0.1+b1 [78.6 kB]
Get:59 http://deb.debian.org/debian bullseye/main amd64 libgdal28 amd64 3.2.2+dfsg-2+deb11u1 [7255 kB]
Get:60 http://deb.debian.org/debian bullseye/main amd64 libxslt1.1 amd64 1.1.34-4 [239 kB]
Get:61 http://deb.debian.org/debian bullseye/main amd64 proj-bin amd64 7.2.1-1 [189 kB]
Get:62 http://deb.debian.org/debian bullseye/main amd64 python3-affine all 2.3.0-2 [15.0 kB]
Get:63 http://deb.debian.org/debian bullseye/main amd64 python3-attr all 20.3.0-1 [52.9 kB]
Get:64 http://deb.debian.org/debian bullseye/main amd64 python3-soupsieve all 2.2.1-1 [34.7 kB]
Get:65 http://deb.debian.org/debian bullseye/main amd64 python3-bs4 all 4.9.3-1 [112 kB]
Get:66 http://deb.debian.org/debian bullseye/main amd64 python3-certifi all 2020.6.20-1 [151 kB]
Get:67 http://deb.debian.org/debian bullseye/main amd64 python3-chardet all 4.0.0-1 [99.0 kB]
Get:68 http://deb.debian.org/debian bullseye/main amd64 python3-colorama all 0.4.4-1 [28.5 kB]
Get:69 http://deb.debian.org/debian bullseye/main amd64 python3-click all 7.1.2-1 [75.7 kB]
Get:70 http://deb.debian.org/debian bullseye/main amd64 python3-click-plugins all 1.1.1-3 [10.3 kB]
Get:71 http://deb.debian.org/debian bullseye/main amd64 python3-cligj all 0.7.1-1 [9024 B]
Get:72 http://deb.debian.org/debian bullseye/main amd64 python3-six all 1.16.0-2 [17.5 kB]
Get:73 http://deb.debian.org/debian bullseye/main amd64 python3-webencodings all 0.5.1-2 [11.0 kB]
Get:74 http://deb.debian.org/debian bullseye/main amd64 python3-html5lib all 1.1-3 [93.0 kB]
Get:75 http://deb.debian.org/debian bullseye/main amd64 python3-lxml amd64 4.6.3+dfsg-0.1+deb11u1 [1093 kB]
Get:76 http://deb.debian.org/debian bullseye/main amd64 python3-numpy amd64 1:1.19.5-1 [2693 kB]
Get:77 http://deb.debian.org/debian bullseye/main amd64 python3-pyparsing all 2.4.7-1 [109 kB]
Get:78 http://deb.debian.org/debian bullseye/main amd64 python3-snuggs all 1.4.7-2 [7780 B]
Get:79 http://deb.debian.org/debian bullseye/main amd64 python3-rasterio amd64 1.2.0-1 [839 kB]
Fetched 53.2 MB in 1s (104 MB/s)
Extracting templates from packages: 100%
Selecting previously unselected package poppler-data.
(Reading database ... 63527 files and directories currently installed.)
Preparing to unpack .../00-poppler-data_0.4.10-1_all.deb ...
Unpacking poppler-data (0.4.10-1) ...
Selecting previously unselected package gdal-data.
Preparing to unpack .../01-gdal-data_3.2.2+dfsg-2+deb11u1_all.deb ...
Unpacking gdal-data (3.2.2+dfsg-2+deb11u1) ...
Selecting previously unselected package libaec0:amd64.
Preparing to unpack .../02-libaec0_1.0.4-1_amd64.deb ...
Unpacking libaec0:amd64 (1.0.4-1) ...
Selecting previously unselected package libaom0:amd64.
Preparing to unpack .../03-libaom0_1.0.0.errata1-3_amd64.deb ...
Unpacking libaom0:amd64 (1.0.0.errata1-3) ...
Selecting previously unselected package libblas3:amd64.
Preparing to unpack .../04-libblas3_3.9.0-3_amd64.deb ...
Unpacking libblas3:amd64 (3.9.0-3) ...
Selecting previously unselected package libgfortran5:amd64.
Preparing to unpack .../05-libgfortran5_10.2.1-6_amd64.deb ...
Unpacking libgfortran5:amd64 (10.2.1-6) ...
Selecting previously unselected package liblapack3:amd64.
Preparing to unpack .../06-liblapack3_3.9.0-3_amd64.deb ...
Unpacking liblapack3:amd64 (3.9.0-3) ...
Selecting previously unselected package libarpack2:amd64.
Preparing to unpack .../07-libarpack2_3.8.0-1_amd64.deb ...
Unpacking libarpack2:amd64 (3.8.0-1) ...
Selecting previously unselected package libsuperlu5:amd64.
Preparing to unpack .../08-libsuperlu5_5.2.2+dfsg1-2_amd64.deb ...
Unpacking libsuperlu5:amd64 (5.2.2+dfsg1-2) ...
Selecting previously unselected package libarmadillo10.
Preparing to unpack .../09-libarmadillo10_1%3a10.1.2+dfsg-6+b1_amd64.deb ...
Unpacking libarmadillo10 (1:10.1.2+dfsg-6+b1) ...
Selecting previously unselected package libcfitsio9:amd64.
Preparing to unpack .../10-libcfitsio9_3.490-3_amd64.deb ...
Unpacking libcfitsio9:amd64 (3.490-3) ...
Selecting previously unselected package libcharls2:amd64.
Preparing to unpack .../11-libcharls2_2.2.0+dfsg-2_amd64.deb ...
Unpacking libcharls2:amd64 (2.2.0+dfsg-2) ...
Selecting previously unselected package libicu67:amd64.
Preparing to unpack .../12-libicu67_67.1-7_amd64.deb ...
Unpacking libicu67:amd64 (67.1-7) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../13-libxml2_2.9.10+dfsg-6.7+deb11u1_amd64.deb ...
Unpacking libxml2:amd64 (2.9.10+dfsg-6.7+deb11u1) ...
Selecting previously unselected package libdap27:amd64.
Preparing to unpack .../14-libdap27_3.20.7-6_amd64.deb ...
Unpacking libdap27:amd64 (3.20.7-6) ...
Selecting previously unselected package libdapclient6v5:amd64.
Preparing to unpack .../15-libdapclient6v5_3.20.7-6_amd64.deb ...
Unpacking libdapclient6v5:amd64 (3.20.7-6) ...
Selecting previously unselected package libdav1d4:amd64.
Preparing to unpack .../16-libdav1d4_0.7.1-3_amd64.deb ...
Unpacking libdav1d4:amd64 (0.7.1-3) ...
Selecting previously unselected package libde265-0:amd64.
Preparing to unpack .../17-libde265-0_1.0.8-1_amd64.deb ...
Unpacking libde265-0:amd64 (1.0.8-1) ...
Selecting previously unselected package libepsilon1:amd64.
Preparing to unpack .../18-libepsilon1_0.9.2+dfsg-5_amd64.deb ...
Unpacking libepsilon1:amd64 (0.9.2+dfsg-5) ...
Selecting previously unselected package libfreexl1:amd64.
Preparing to unpack .../19-libfreexl1_1.0.6-1_amd64.deb ...
Unpacking libfreexl1:amd64 (1.0.6-1) ...
Selecting previously unselected package libfyba0:amd64.
Preparing to unpack .../20-libfyba0_4.1.1-7_amd64.deb ...
Unpacking libfyba0:amd64 (4.1.1-7) ...
Selecting previously unselected package libgeos-3.9.0:amd64.
Preparing to unpack .../21-libgeos-3.9.0_3.9.0-1_amd64.deb ...
Unpacking libgeos-3.9.0:amd64 (3.9.0-1) ...
Selecting previously unselected package libgeos-c1v5:amd64.
Preparing to unpack .../22-libgeos-c1v5_3.9.0-1_amd64.deb ...
Unpacking libgeos-c1v5:amd64 (3.9.0-1) ...
Selecting previously unselected package proj-data.
Preparing to unpack .../23-proj-data_7.2.1-1_all.deb ...
Unpacking proj-data (7.2.1-1) ...
Selecting previously unselected package libproj19:amd64.
Preparing to unpack .../24-libproj19_7.2.1-1_amd64.deb ...
Unpacking libproj19:amd64 (7.2.1-1) ...
Selecting previously unselected package libgeotiff5:amd64.
Preparing to unpack .../25-libgeotiff5_1.6.0-1_amd64.deb ...
Unpacking libgeotiff5:amd64 (1.6.0-1) ...
Selecting previously unselected package libgif7:amd64.
Preparing to unpack .../26-libgif7_5.1.9-2_amd64.deb ...
Unpacking libgif7:amd64 (5.1.9-2) ...
Selecting previously unselected package libhdf4-0-alt.
Preparing to unpack .../27-libhdf4-0-alt_4.2.15-3_amd64.deb ...
Unpacking libhdf4-0-alt (4.2.15-3) ...
Selecting previously unselected package libsz2:amd64.
Preparing to unpack .../28-libsz2_1.0.4-1_amd64.deb ...
Unpacking libsz2:amd64 (1.0.4-1) ...
Selecting previously unselected package libhdf5-103-1:amd64.
Preparing to unpack .../29-libhdf5-103-1_1.10.6+repack-4+deb11u1_amd64.deb ...
Unpacking libhdf5-103-1:amd64 (1.10.6+repack-4+deb11u1) ...
Selecting previously unselected package libnuma1:amd64.
Preparing to unpack .../30-libnuma1_2.0.12-1+b1_amd64.deb ...
Unpacking libnuma1:amd64 (2.0.12-1+b1) ...
Selecting previously unselected package libx265-192:amd64.
Preparing to unpack .../31-libx265-192_3.4-2_amd64.deb ...
Unpacking libx265-192:amd64 (3.4-2) ...
Selecting previously unselected package libheif1:amd64.
Preparing to unpack .../32-libheif1_1.11.0-1_amd64.deb ...
Unpacking libheif1:amd64 (1.11.0-1) ...
Selecting previously unselected package libminizip1:amd64.
Preparing to unpack .../33-libminizip1_1.1-8+b1_amd64.deb ...
Unpacking libminizip1:amd64 (1.1-8+b1) ...
Selecting previously unselected package liburiparser1:amd64.
Preparing to unpack .../34-liburiparser1_0.9.4+dfsg-1+deb11u1_amd64.deb ...
Unpacking liburiparser1:amd64 (0.9.4+dfsg-1+deb11u1) ...
Selecting previously unselected package libkmlbase1:amd64.
Preparing to unpack .../35-libkmlbase1_1.3.0-9_amd64.deb ...
Unpacking libkmlbase1:amd64 (1.3.0-9) ...
Selecting previously unselected package libkmldom1:amd64.
Preparing to unpack .../36-libkmldom1_1.3.0-9_amd64.deb ...
Unpacking libkmldom1:amd64 (1.3.0-9) ...
Selecting previously unselected package libkmlengine1:amd64.
Preparing to unpack .../37-libkmlengine1_1.3.0-9_amd64.deb ...
Unpacking libkmlengine1:amd64 (1.3.0-9) ...
Selecting previously unselected package mysql-common.
Preparing to unpack .../38-mysql-common_5.8+1.0.7_all.deb ...
Unpacking mysql-common (5.8+1.0.7) ...
Selecting previously unselected package mariadb-common.
Preparing to unpack .../39-mariadb-common_1%3a10.5.15-0+deb11u1_all.deb ...
Unpacking mariadb-common (1:10.5.15-0+deb11u1) ...
Selecting previously unselected package libmariadb3:amd64.
Preparing to unpack .../40-libmariadb3_1%3a10.5.15-0+deb11u1_amd64.deb ...
Unpacking libmariadb3:amd64 (1:10.5.15-0+deb11u1) ...
Selecting previously unselected package libhdf5-hl-100:amd64.
Preparing to unpack .../41-libhdf5-hl-100_1.10.6+repack-4+deb11u1_amd64.deb ...
Unpacking libhdf5-hl-100:amd64 (1.10.6+repack-4+deb11u1) ...
Selecting previously unselected package libnetcdf18:amd64.
Preparing to unpack .../42-libnetcdf18_1%3a4.7.4-1_amd64.deb ...
Unpacking libnetcdf18:amd64 (1:4.7.4-1) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../43-libltdl7_2.4.6-15_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-15) ...
Selecting previously unselected package libodbc1:amd64.
Preparing to unpack .../44-libodbc1_2.3.6-0.1+b1_amd64.deb ...
Unpacking libodbc1:amd64 (2.3.6-0.1+b1) ...
Selecting previously unselected package libogdi4.1.
Preparing to unpack .../45-libogdi4.1_4.1.0+ds-5_amd64.deb ...
Unpacking libogdi4.1 (4.1.0+ds-5) ...
Selecting previously unselected package libopenjp2-7:amd64.
Preparing to unpack .../46-libopenjp2-7_2.4.0-3_amd64.deb ...
Unpacking libopenjp2-7:amd64 (2.4.0-3) ...
Selecting previously unselected package liblcms2-2:amd64.
Preparing to unpack .../47-liblcms2-2_2.12~rc1-2_amd64.deb ...
Unpacking liblcms2-2:amd64 (2.12~rc1-2) ...
Selecting previously unselected package libnspr4:amd64.
Preparing to unpack .../48-libnspr4_2%3a4.29-1_amd64.deb ...
Unpacking libnspr4:amd64 (2:4.29-1) ...
Selecting previously unselected package libnss3:amd64.
Preparing to unpack .../49-libnss3_2%3a3.61-1+deb11u2_amd64.deb ...
Unpacking libnss3:amd64 (2:3.61-1+deb11u2) ...
Selecting previously unselected package libpoppler102:amd64.
Preparing to unpack .../50-libpoppler102_20.09.0-3.1_amd64.deb ...
Unpacking libpoppler102:amd64 (20.09.0-3.1) ...
Selecting previously unselected package libpq5:amd64.
Preparing to unpack .../51-libpq5_13.7-0+deb11u1_amd64.deb ...
Unpacking libpq5:amd64 (13.7-0+deb11u1) ...
Selecting previously unselected package libqhull8.0:amd64.
Preparing to unpack .../52-libqhull8.0_2020.2-3_amd64.deb ...
Unpacking libqhull8.0:amd64 (2020.2-3) ...
Selecting previously unselected package librttopo1:amd64.
Preparing to unpack .../53-librttopo1_1.1.0-2_amd64.deb ...
Unpacking librttopo1:amd64 (1.1.0-2) ...
Selecting previously unselected package libspatialite7:amd64.
Preparing to unpack .../54-libspatialite7_5.0.1-2_amd64.deb ...
Unpacking libspatialite7:amd64 (5.0.1-2) ...
Selecting previously unselected package libxerces-c3.2:amd64.
Preparing to unpack .../55-libxerces-c3.2_3.2.3+debian-3_amd64.deb ...
Unpacking libxerces-c3.2:amd64 (3.2.3+debian-3) ...
Selecting previously unselected package odbcinst.
Preparing to unpack .../56-odbcinst_2.3.6-0.1+b1_amd64.deb ...
Unpacking odbcinst (2.3.6-0.1+b1) ...
Selecting previously unselected package odbcinst1debian2:amd64.
Preparing to unpack .../57-odbcinst1debian2_2.3.6-0.1+b1_amd64.deb ...
Unpacking odbcinst1debian2:amd64 (2.3.6-0.1+b1) ...
Selecting previously unselected package libgdal28.
Preparing to unpack .../58-libgdal28_3.2.2+dfsg-2+deb11u1_amd64.deb ...
Unpacking libgdal28 (3.2.2+dfsg-2+deb11u1) ...
Selecting previously unselected package libxslt1.1:amd64.
Preparing to unpack .../59-libxslt1.1_1.1.34-4_amd64.deb ...
Unpacking libxslt1.1:amd64 (1.1.34-4) ...
Selecting previously unselected package proj-bin.
Preparing to unpack .../60-proj-bin_7.2.1-1_amd64.deb ...
Unpacking proj-bin (7.2.1-1) ...
Selecting previously unselected package python3-affine.
Preparing to unpack .../61-python3-affine_2.3.0-2_all.deb ...
Unpacking python3-affine (2.3.0-2) ...
Selecting previously unselected package python3-attr.
Preparing to unpack .../62-python3-attr_20.3.0-1_all.deb ...
Unpacking python3-attr (20.3.0-1) ...
Selecting previously unselected package python3-soupsieve.
Preparing to unpack .../63-python3-soupsieve_2.2.1-1_all.deb ...
Unpacking python3-soupsieve (2.2.1-1) ...
Selecting previously unselected package python3-bs4.
Preparing to unpack .../64-python3-bs4_4.9.3-1_all.deb ...
Unpacking python3-bs4 (4.9.3-1) ...
Selecting previously unselected package python3-certifi.
Preparing to unpack .../65-python3-certifi_2020.6.20-1_all.deb ...
Unpacking python3-certifi (2020.6.20-1) ...
Selecting previously unselected package python3-chardet.
Preparing to unpack .../66-python3-chardet_4.0.0-1_all.deb ...
Unpacking python3-chardet (4.0.0-1) ...
Selecting previously unselected package python3-colorama.
Preparing to unpack .../67-python3-colorama_0.4.4-1_all.deb ...
Unpacking python3-colorama (0.4.4-1) ...
Selecting previously unselected package python3-click.
Preparing to unpack .../68-python3-click_7.1.2-1_all.deb ...
Unpacking python3-click (7.1.2-1) ...
Selecting previously unselected package python3-click-plugins.
Preparing to unpack .../69-python3-click-plugins_1.1.1-3_all.deb ...
Unpacking python3-click-plugins (1.1.1-3) ...
Selecting previously unselected package python3-cligj.
Preparing to unpack .../70-python3-cligj_0.7.1-1_all.deb ...
Unpacking python3-cligj (0.7.1-1) ...
Selecting previously unselected package python3-six.
Preparing to unpack .../71-python3-six_1.16.0-2_all.deb ...
Unpacking python3-six (1.16.0-2) ...
Selecting previously unselected package python3-webencodings.
Preparing to unpack .../72-python3-webencodings_0.5.1-2_all.deb ...
Unpacking python3-webencodings (0.5.1-2) ...
Selecting previously unselected package python3-html5lib.
Preparing to unpack .../73-python3-html5lib_1.1-3_all.deb ...
Unpacking python3-html5lib (1.1-3) ...
Selecting previously unselected package python3-lxml:amd64.
Preparing to unpack .../74-python3-lxml_4.6.3+dfsg-0.1+deb11u1_amd64.deb ...
Unpacking python3-lxml:amd64 (4.6.3+dfsg-0.1+deb11u1) ...
Selecting previously unselected package python3-numpy.
Preparing to unpack .../75-python3-numpy_1%3a1.19.5-1_amd64.deb ...
Unpacking python3-numpy (1:1.19.5-1) ...
Selecting previously unselected package python3-pyparsing.
Preparing to unpack .../76-python3-pyparsing_2.4.7-1_all.deb ...
Unpacking python3-pyparsing (2.4.7-1) ...
Selecting previously unselected package python3-snuggs.
Preparing to unpack .../77-python3-snuggs_1.4.7-2_all.deb ...
Unpacking python3-snuggs (1.4.7-2) ...
Selecting previously unselected package python3-rasterio.
Preparing to unpack .../78-python3-rasterio_1.2.0-1_amd64.deb ...
Unpacking python3-rasterio (1.2.0-1) ...
Setting up liblcms2-2:amd64 (2.12~rc1-2) ...
Setting up python3-attr (20.3.0-1) ...
Setting up mysql-common (5.8+1.0.7) ...
update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Setting up proj-data (7.2.1-1) ...
Setting up libicu67:amd64 (67.1-7) ...
Setting up libogdi4.1 (4.1.0+ds-5) ...
Setting up libqhull8.0:amd64 (2020.2-3) ...
Setting up python3-colorama (0.4.4-1) ...
Setting up libcharls2:amd64 (2.2.0+dfsg-2) ...
Setting up libminizip1:amd64 (1.1-8+b1) ...
Setting up libaom0:amd64 (1.0.0.errata1-3) ...
Setting up libpq5:amd64 (13.7-0+deb11u1) ...
Setting up python3-click (7.1.2-1) ...
Setting up libepsilon1:amd64 (0.9.2+dfsg-5) ...
Setting up python3-webencodings (0.5.1-2) ...
Setting up libaec0:amd64 (1.0.4-1) ...
Setting up gdal-data (3.2.2+dfsg-2+deb11u1) ...
Setting up poppler-data (0.4.10-1) ...
Setting up libcfitsio9:amd64 (3.490-3) ...
Setting up mariadb-common (1:10.5.15-0+deb11u1) ...
update-alternatives: using /etc/mysql/mariadb.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Setting up python3-six (1.16.0-2) ...
Setting up libblas3:amd64 (3.9.0-3) ...
update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode
Setting up python3-chardet (4.0.0-1) ...
Setting up python3-pyparsing (2.4.7-1) ...
Setting up python3-certifi (2020.6.20-1) ...
Setting up libnspr4:amd64 (2:4.29-1) ...
Setting up python3-click-plugins (1.1.1-3) ...
Setting up libmariadb3:amd64 (1:10.5.15-0+deb11u1) ...
Setting up python3-html5lib (1.1-3) ...
Setting up libltdl7:amd64 (2.4.6-15) ...
Setting up libgfortran5:amd64 (10.2.1-6) ...
Setting up libhdf4-0-alt (4.2.15-3) ...
Setting up libgif7:amd64 (5.1.9-2) ...
Setting up liburiparser1:amd64 (0.9.4+dfsg-1+deb11u1) ...
Setting up libnuma1:amd64 (2.0.12-1+b1) ...
Setting up libfreexl1:amd64 (1.0.6-1) ...
Setting up libgeos-3.9.0:amd64 (3.9.0-1) ...
Setting up libfyba0:amd64 (4.1.1-7) ...
Setting up libkmlbase1:amd64 (1.3.0-9) ...
Setting up libdav1d4:amd64 (0.7.1-3) ...
Setting up libopenjp2-7:amd64 (2.4.0-3) ...
Setting up python3-affine (2.3.0-2) ...
Setting up libde265-0:amd64 (1.0.8-1) ...
Setting up libproj19:amd64 (7.2.1-1) ...
Setting up libxml2:amd64 (2.9.10+dfsg-6.7+deb11u1) ...
Setting up python3-soupsieve (2.2.1-1) ...
Setting up libsz2:amd64 (1.0.4-1) ...
Setting up libkmldom1:amd64 (1.3.0-9) ...
Setting up liblapack3:amd64 (3.9.0-3) ...
update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3 to provide /usr/lib/x86_64-linux-gnu/liblapack.so.3 (liblapack.so.3-x86_64-linux-gnu) in auto mode
Setting up libxerces-c3.2:amd64 (3.2.3+debian-3) ...
Setting up libkmlengine1:amd64 (1.3.0-9) ...
Setting up python3-cligj (0.7.1-1) ...
Setting up libarpack2:amd64 (3.8.0-1) ...
Setting up libx265-192:amd64 (3.4-2) ...
Setting up libsuperlu5:amd64 (5.2.2+dfsg1-2) ...
Setting up proj-bin (7.2.1-1) ...
Setting up libnss3:amd64 (2:3.61-1+deb11u2) ...
Setting up python3-bs4 (4.9.3-1) ...
Setting up libgeotiff5:amd64 (1.6.0-1) ...
Setting up libdap27:amd64 (3.20.7-6) ...
Setting up libgeos-c1v5:amd64 (3.9.0-1) ...
Setting up libodbc1:amd64 (2.3.6-0.1+b1) ...
Setting up python3-numpy (1:1.19.5-1) ...
Setting up libhdf5-103-1:amd64 (1.10.6+repack-4+deb11u1) ...
Setting up librttopo1:amd64 (1.1.0-2) ...
Setting up libpoppler102:amd64 (20.09.0-3.1) ...
Setting up libxslt1.1:amd64 (1.1.34-4) ...
Setting up libdapclient6v5:amd64 (3.20.7-6) ...
Setting up libhdf5-hl-100:amd64 (1.10.6+repack-4+deb11u1) ...
Setting up libspatialite7:amd64 (5.0.1-2) ...
Setting up libarmadillo10 (1:10.1.2+dfsg-6+b1) ...
Setting up libheif1:amd64 (1.11.0-1) ...
Setting up python3-lxml:amd64 (4.6.3+dfsg-0.1+deb11u1) ...
Setting up libnetcdf18:amd64 (1:4.7.4-1) ...
Setting up python3-snuggs (1.4.7-2) ...
Setting up odbcinst (2.3.6-0.1+b1) ...
Setting up odbcinst1debian2:amd64 (2.3.6-0.1+b1) ...
Setting up libgdal28 (3.2.2+dfsg-2+deb11u1) ...
Setting up python3-rasterio (1.2.0-1) ...
Processing triggers for libc-bin (2.31-13+deb11u3) ...
Processing triggers for man-db (2.9.4-2) ...
python3 process_sat.py
LC08_L1TP_025033_20201007_20201016_01_T1 1
LC08_L1TP_025033_20201007_20201016_01_T1 2
LC08_L1TP_025033_20201007_20201016_01_T1 3
Note: if you get an “ERROR 4” it is expected in earlier versions of rasterio.
Verify the results were created.
ls -lh output
total 192M
-rw-r--r-- 1 learner learner 192M May 20 17:43 result-LC08_L1TP_025033_20201007_20201016_01_T1.png
-rw-r--r-- 1 learner learner 941 May 20 17:43 result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml
Saving the Results#
We now will store the data in the bucket we created in the beginning of the episode. First we verify the environment variable and that it exists.
echo $BUCKET
essentials-learner-2022-02-14
gsutil ls -b gs://$BUCKET
gs://essentials-learner-2022-02-14/
Now copy the output data to the bucket. The -r flag recursively copies the output directory and -m copies the files in parallel. Note the locations of the -m and -r switches as they apply globally and to the cp command respectively.
gsutil -m cp -r output gs://$BUCKET
Copying file://output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml [Content-Type=application/xml]...
Copying file://output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png [Content-Type=image/png]...
==> NOTE: You are uploading one or more large file(s), which would run
significantly faster if you enable parallel composite uploads. This
feature can be enabled by editing the
"parallel_composite_upload_threshold" value in your .boto
configuration file. However, note that if you do this large files will
be uploaded as `composite objects
<https://cloud.google.com/storage/docs/composite-objects>`_,which
means that any user who downloads such objects will need to have a
compiled crcmod installed (see "gsutil help crcmod"). This is because
without a compiled crcmod, computing checksums on composite objects is
so slow that gsutil disables downloads of composite objects.
\ [2/2 files][191.6 MiB/191.6 MiB] 100% Done
Operation completed over 2 objects/191.6 MiB.
Verify that the output was uploaded.
gsutil ls gs://$BUCKET
gs://essentials-learner-2022-02-14/output/
gsutil ls -lh gs://$BUCKET/output
191.58 MiB 2022-02-14T20:36:53Z gs://essentials-learner-2022-02-14/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png
910 B 2022-02-14T20:36:51Z gs://essentials-learner-2022-02-14/output/result-LC08_L1TP_025033_20201007_20201016_01_T1.png.aux.xml
TOTAL: 2 objects, 200890195 bytes (191.58 MiB)
Viewing the Results#
You now can view the results by using the Google Cloud Web Console and navigating to “Cloud Storage”, selecting the bucket, and then the result object you wish to view (select the .png file). You will need to click the “Preview” button given the large size of the image.
Exercise
Try to find and view the results on your own
Navigate to Cloud Storage -> Buckets -> Bucket -> output folder and then click on the result object.
And press the Preview button below the object details (check the details) and you should see something similar to the following:
Cleanup#
We will now leave the resources running in order to learn more about sharing and monitoring costs and will clean up all the resources as the end of Lesson.
Danger
Do not forget to do remove the Cloud Storage Bucket and the Compute Engine Instance (Virtual Machine) when you are done with this Lesson! Running resources and stored data costs money! Cleanup when you are done!