Created a folder for basic and advanced deployment of the ETL pipeline.

CLASS · Aug 2, 2024 · 015f41b · 015f41b
1 parent 26fafb5
commit 015f41b
Show file tree

Hide file tree

Showing 13 changed files with 34 additions and 4 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,7 @@
-scripts/data/combined_data_cleaned.csv
-scripts/data/combined_data.csv
+# ignore the data files
+01-basic-deployment/scripts/data/combined_data_cleaned.csv
+01-basic-deployment/scripts/data/combined_data.csv
 
-scripts/extract_done
-scripts/transform_done
+# ignore the done files
+01-basic-deployment/scripts/extract_done
+01-basic-deployment/scripts/transform_done
diff --git a/Dockerfile → 01-basic-deployment/Dockerfile b/Dockerfile → 01-basic-deployment/Dockerfile
diff --git a/app/entrypoint_extract.sh → ...asic-deployment/app/entrypoint_extract.sh b/app/entrypoint_extract.sh → ...asic-deployment/app/entrypoint_extract.sh
diff --git a/app/entrypoint_load.sh → 01-basic-deployment/app/entrypoint_load.sh b/app/entrypoint_load.sh → 01-basic-deployment/app/entrypoint_load.sh
diff --git a/app/entrypoint_transform.sh → ...ic-deployment/app/entrypoint_transform.sh b/app/entrypoint_transform.sh → ...ic-deployment/app/entrypoint_transform.sh
diff --git a/docker-compose.yml → 01-basic-deployment/docker-compose.yml b/docker-compose.yml → 01-basic-deployment/docker-compose.yml
diff --git a/requirements.txt → 01-basic-deployment/requirements.txt b/requirements.txt → 01-basic-deployment/requirements.txt
diff --git a/scripts/.DS_Store → 01-basic-deployment/scripts/.DS_Store b/scripts/.DS_Store → 01-basic-deployment/scripts/.DS_Store
diff --git a/...ad-notebook-visualization-COMPLETED.ipynb → ...ad-notebook-visualization-COMPLETED.ipynb b/...ad-notebook-visualization-COMPLETED.ipynb → ...ad-notebook-visualization-COMPLETED.ipynb
diff --git a/scripts/extract.py → 01-basic-deployment/scripts/extract.py b/scripts/extract.py → 01-basic-deployment/scripts/extract.py
diff --git a/scripts/load.py → 01-basic-deployment/scripts/load.py b/scripts/load.py → 01-basic-deployment/scripts/load.py
diff --git a/scripts/transform.py → 01-basic-deployment/scripts/transform.py b/scripts/transform.py → 01-basic-deployment/scripts/transform.py
diff --git a/docker-guides/docker-compose.md b/docker-guides/docker-compose.md
@@ -0,0 +1,28 @@
+# Cleaning Up Docker Deployments
+
+To clean up after using `docker-compose up`, you can use the following commands to stop and remove containers, networks, volumes, and images created by Docker Compose:
+
+## Stop and Remove Containers
+This command will stop and remove all containers defined in the `docker-compose.yml` file.
+
+```sh
+docker-compose down
+```
+
+## Remove Containers, Networks, Volumes, and Images
+This command will remove containers, networks, and images created by docker-compose up. The -v flag is used to remove named volumes declared in the volumes section of the Compose file and anonymous volumes attached to containers.
+
+```sh
+docker-compose down --volumes --remove-orphans
+```
+## Remove All Unused Docker Data
+
+If you want to remove all unused Docker data, including stopped containers, networks, images, and build cache, use the following command:
+
+```sh
+docker system prune -a --volumes
+```
+
+Note:
+- -a will also remove all unused images not just dangling ones.
+- --volumes will remove all unused volumes.