-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit ca2accc
Showing
1 changed file
with
67 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Container Orchestration for Research Workflows | ||
|
|
||
| ## Overview of the Workshop | ||
| This workshop provides researchers with a comprehensive guide on containerizing and orchestrating data pipelines. Participants will gain hands-on experience with Docker, Docker Compose, and Kubernetes, and will learn to build and deploy a UI for visualizing data. The workshop focuses on practical applications of container orchestration to enhance research workflows. | ||
|
|
||
| ## Agenda | ||
|
|
||
| ### 1. Overview of the Workshop | ||
| - Introduction to the goals and structure of the workshop | ||
| - Explanation of key concepts and technologies | ||
|
|
||
| ### 2. Use Case: Containers for Research | ||
| - Benefits of using containers for research purposes | ||
| - Detailed use case: Containerizing data pipelines | ||
|
|
||
| ### 3. Introduction to Containers and Kubernetes | ||
| - Understanding the basics of containers | ||
| - Overview of Kubernetes and its components | ||
|
|
||
| ### 4. Data Processing with Jupyter Notebooks | ||
| - Review and understand the data processing Jupyter notebook | ||
|
|
||
| ### 5. Building ETL Containers | ||
| - Create containers for the Extract, Transform, and Load (ETL) processes | ||
|
|
||
| ### 6. Deploying ETL Containers | ||
| - Use Docker Compose to deploy ETL containers | ||
|
|
||
| ### 7. Building a UI Container | ||
| - Build a containerized UI application to visualize transformed data | ||
|
|
||
| ### 8. Deploying the UI Container | ||
| - Deploy the UI container using Kubernetes | ||
|
|
||
| ### 9. Customization Section | ||
| - Deploying to different cloud providers: AWS, GCP, and Azure | ||
|
|
||
|
|
||
| ## Repo Folder Structure | ||
|
|
||
| ```plaintext | ||
| Container-Orchestration-for-Research-Workflows/ | ||
| ├── README.md | ||
| ├── Dockerfile | ||
| ├── docker-compose.yml | ||
| ├── k8s/ | ||
| │ ├── deployment.yaml | ||
| │ ├── service.yaml | ||
| │ ├── ingress.yaml | ||
| │ ├── configmap.yaml | ||
| │ ├── secret.yaml | ||
| ├── notebooks/ | ||
| │ ├── data_processing_notebook.ipynb | ||
| ├── scripts/ | ||
| │ ├── extract.py | ||
| │ ├── transform.py | ||
| │ ├── load.py | ||
| │ ├── ui/ | ||
| │ │ ├── app/ | ||
| │ │ │ ├── package.json | ||
| │ │ │ ├── package-lock.json | ||
| │ │ │ ├── src/ | ||
| │ │ │ │ ├── index.js | ||
| │ │ │ │ ├── components/ | ||
| │ │ │ │ │ ├── VisualizationComponent.js | ||
| │ │ │ │ ├── public/ | ||
| │ │ │ │ │ ├── index.html |