-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
147 additions
and
48 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,68 +1,99 @@ | ||
| # Container Orchestration for Research Workflows | ||
|
|
||
| ## Overview of the Workshop | ||
| This workshop provides researchers with a comprehensive guide on containerizing and orchestrating data pipelines. Participants will gain hands-on experience with Docker, Docker Compose, and Kubernetes, and will learn to build and deploy a UI for visualizing data. The workshop focuses on practical applications of container orchestration to enhance research workflows. | ||
| # Workshop Overview | ||
|
|
||
| ## Agenda | ||
| In this workshop, you will learn how to set up and manage a containerized research workflow with Extract, Load, Transform (ELT) features and a user interface (UI) to view and analyze outputs. We will review containerization fundamentals, ELT processes, and create an intuitive UI for end users. | ||
|
|
||
| ### 1. Overview of the Workshop | ||
| - Introduction to the goals and structure of the workshop | ||
| - Explanation of key concepts and technologies | ||
| ## Course Level | ||
|
|
||
| ### 2. Use Case: Containers for Research | ||
| - Benefits of using containers for research purposes | ||
| - Detailed use case: Containerizing data pipelines | ||
| - Intermediate | ||
|
|
||
| ### 3. Introduction to Containers and Kubernetes | ||
| - Understanding the basics of containers | ||
| - Overview of Kubernetes and its components | ||
| ## Intended Audience | ||
|
|
||
| ### 4. Data Processing with Jupyter Notebooks | ||
| - Review and understand the data processing Jupyter notebook | ||
| This workshop is intended for: | ||
|
|
||
| ### 5. Building ETL Containers | ||
| - Create containers for the Extract, Transform, and Load (ETL) processes | ||
| - **Primary Audience**: Researchers and data scientists interested in streamlining their workflows. | ||
| - **Secondary Audience**: IT professionals and developers looking to implement containerized solutions for data processing and analysis. | ||
|
|
||
| ### 6. Deploying ETL Containers | ||
| - Use Docker Compose to deploy ETL containers | ||
| ## Workshop Objectives | ||
|
|
||
| ### 7. Building a UI Container | ||
| - Build a containerized UI application to visualize transformed data | ||
| By the end of this workshop, you will be able to: | ||
|
|
||
| ### 8. Deploying the UI Container | ||
| - Deploy the UI container using Kubernetes | ||
| - Understand the process to implement containers for a workshop | ||
| - Set up a containerized environment for research workflows | ||
| - Implement data processing | ||
| - Containerize a user-friendly interface for accessing output and visualization | ||
|
|
||
| ### 9. Customization Section | ||
| - Deploying to different cloud providers: AWS, GCP, and Azure | ||
| ## Delivery Method | ||
|
|
||
| This workshop is delivered through: | ||
|
|
||
| ## Repo Folder Structure | ||
| - Online via Zoom | ||
|
|
||
| ## Duration | ||
|
|
||
| - 5 Hours | ||
|
|
||
| ## Workshop Outline | ||
|
|
||
| This workshop covers the following concepts: | ||
|
|
||
| 1. **Introduction to Containerized Research Workflows** | ||
| - Overview of Docker and Kubernetes | ||
| - Benefits of containerizing research workflows | ||
| - Tools and technologies: Docker, Kubernetes, etc. | ||
|
|
||
| 2. **Setting Up the Containerized Environment** | ||
| - Creating Dockerfiles and Docker Compose files | ||
|
|
||
| 3. **Data Processing Fundamentals** | ||
| - Understanding data processing | ||
| - Implementing a containerized environment | ||
|
|
||
| 4. **Developing the User Interface** | ||
| - UI development using Flask | ||
| - Containerized UI | ||
|
|
||
| 5. **Visualizing and Analyzing Output** | ||
| - Creating visualization dashboards | ||
|
|
||
| 6. **Deploying to a Public Cloud as a Kubernetes Cluster** | ||
| - Setting up a Kubernetes cluster on a public cloud (e.g., AWS, Google Cloud, Azure) | ||
| - Deploying containerized applications to Kubernetes | ||
| - Managing and scaling applications in Kubernetes | ||
|
|
||
| ## End of Workshop Assessment | ||
|
|
||
| - Q&A session | ||
| - Feedback | ||
|
|
||
| ## Hands-On Lab Exercises | ||
|
|
||
| 1. Setting up Docker containers | ||
| 2. Deploying and managing the complete workflow | ||
| 3. Deploying the workflow to a Kubernetes cluster on a public cloud | ||
|
|
||
| By the end of this workshop, participants will have a comprehensive understanding of how to build and manage a containerized research workflow using Docker and/or Kubernetes. | ||
|
|
||
|
|
||
| ## Repo Folder Structure (Subject to change) | ||
|
|
||
| ```plaintext | ||
| Container-Orchestration-for-Research-Workflows/ | ||
| ├── README.md | ||
| ├── Dockerfile | ||
| ├── README.md | ||
| ├── README_old.md | ||
| ├── app | ||
| │ ├── entrypoint_extract.sh | ||
| │ ├── entrypoint_load.sh | ||
| │ └── entrypoint_transform.sh | ||
| ├── docker-compose.yml | ||
| ├── k8s/ | ||
| │ ├── deployment.yaml | ||
| │ ├── service.yaml | ||
| │ ├── ingress.yaml | ||
| │ ├── configmap.yaml | ||
| │ ├── secret.yaml | ||
| ├── notebooks/ | ||
| │ ├── data_processing_notebook.ipynb | ||
| ├── scripts/ | ||
| │ ├── extract.py | ||
| │ ├── transform.py | ||
| │ ├── load.py | ||
| │ ├── ui/ | ||
| │ │ ├── app/ | ||
| │ │ │ ├── package.json | ||
| │ │ │ ├── package-lock.json | ||
| │ │ │ ├── src/ | ||
| │ │ │ │ ├── index.js | ||
| │ │ │ │ ├── components/ | ||
| │ │ │ │ │ ├── VisualizationComponent.js | ||
| │ │ │ │ ├── public/ | ||
| │ │ │ │ │ ├── index.html | ||
| ├── notebooks | ||
| ├── requirements.txt | ||
| └── scripts | ||
| ├── data | ||
| │ └── fm-ad-notebook-visualization-COMPLETED.ipynb | ||
| ├── extract.py | ||
| ├── load.py | ||
| └── transform.py | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # Container Orchestration for Research Workflows | ||
|
|
||
| ## Overview of the Workshop | ||
| This workshop provides researchers with a comprehensive guide on containerizing and orchestrating data pipelines. Participants will gain hands-on experience with Docker, Docker Compose, and Kubernetes, and will learn to build and deploy a UI for visualizing data. The workshop focuses on practical applications of container orchestration to enhance research workflows. | ||
|
|
||
| ## Agenda | ||
|
|
||
| ### 1. Overview of the Workshop | ||
| - Introduction to the goals and structure of the workshop | ||
| - Explanation of key concepts and technologies | ||
|
|
||
| ### 2. Use Case: Containers for Research | ||
| - Benefits of using containers for research purposes | ||
| - Detailed use case: Containerizing data pipelines | ||
|
|
||
| ### 3. Introduction to Containers and Kubernetes | ||
| - Understanding the basics of containers | ||
| - Overview of Kubernetes and its components | ||
|
|
||
| ### 4. Data Processing with Jupyter Notebooks | ||
| - Review and understand the data processing Jupyter notebook | ||
|
|
||
| ### 5. Building ETL Containers | ||
| - Create containers for the Extract, Transform, and Load (ETL) processes | ||
|
|
||
| ### 6. Deploying ETL Containers | ||
| - Use Docker Compose to deploy ETL containers | ||
|
|
||
| ### 7. Building a UI Container | ||
| - Build a containerized UI application to visualize transformed data | ||
|
|
||
| ### 8. Deploying the UI Container | ||
| - Deploy the UI container using Kubernetes | ||
|
|
||
| ### 9. Customization Section | ||
| - Deploying to different cloud providers: AWS, GCP, and Azure | ||
|
|
||
|
|
||
| ## Repo Folder Structure | ||
|
|
||
| ```plaintext | ||
| Container-Orchestration-for-Research-Workflows/ | ||
| ├── README.md | ||
| ├── Dockerfile | ||
| ├── docker-compose.yml | ||
| ├── k8s/ | ||
| │ ├── deployment.yaml | ||
| │ ├── service.yaml | ||
| │ ├── ingress.yaml | ||
| │ ├── configmap.yaml | ||
| │ ├── secret.yaml | ||
| ├── notebooks/ | ||
| │ ├── data_processing_notebook.ipynb | ||
| ├── scripts/ | ||
| │ ├── extract.py | ||
| │ ├── transform.py | ||
| │ ├── load.py | ||
| │ ├── ui/ | ||
| │ │ ├── app/ | ||
| │ │ │ ├── package.json | ||
| │ │ │ ├── package-lock.json | ||
| │ │ │ ├── src/ | ||
| │ │ │ │ ├── index.js | ||
| │ │ │ │ ├── components/ | ||
| │ │ │ │ │ ├── VisualizationComponent.js | ||
| │ │ │ │ ├── public/ | ||
| │ │ │ │ │ ├── index.html | ||
| ``` |