diff --git a/.DS_Store b/.DS_Store index b40f4d5..310749a 100644 Binary files a/.DS_Store and b/.DS_Store differ diff --git a/README.md b/README.md index ab62bb0..257d2af 100644 --- a/README.md +++ b/README.md @@ -1,68 +1,99 @@ # Container Orchestration for Research Workflows -## Overview of the Workshop -This workshop provides researchers with a comprehensive guide on containerizing and orchestrating data pipelines. Participants will gain hands-on experience with Docker, Docker Compose, and Kubernetes, and will learn to build and deploy a UI for visualizing data. The workshop focuses on practical applications of container orchestration to enhance research workflows. +# Workshop Overview -## Agenda +In this workshop, you will learn how to set up and manage a containerized research workflow with Extract, Load, Transform (ELT) features and a user interface (UI) to view and analyze outputs. We will review containerization fundamentals, ELT processes, and create an intuitive UI for end users. -### 1. Overview of the Workshop -- Introduction to the goals and structure of the workshop -- Explanation of key concepts and technologies +## Course Level -### 2. Use Case: Containers for Research -- Benefits of using containers for research purposes -- Detailed use case: Containerizing data pipelines +- Intermediate -### 3. Introduction to Containers and Kubernetes -- Understanding the basics of containers -- Overview of Kubernetes and its components +## Intended Audience -### 4. Data Processing with Jupyter Notebooks -- Review and understand the data processing Jupyter notebook +This workshop is intended for: -### 5. Building ETL Containers -- Create containers for the Extract, Transform, and Load (ETL) processes +- **Primary Audience**: Researchers and data scientists interested in streamlining their workflows. +- **Secondary Audience**: IT professionals and developers looking to implement containerized solutions for data processing and analysis. -### 6. Deploying ETL Containers -- Use Docker Compose to deploy ETL containers +## Workshop Objectives -### 7. Building a UI Container -- Build a containerized UI application to visualize transformed data +By the end of this workshop, you will be able to: -### 8. Deploying the UI Container -- Deploy the UI container using Kubernetes +- Understand the process to implement containers for a workshop +- Set up a containerized environment for research workflows +- Implement data processing +- Containerize a user-friendly interface for accessing output and visualization -### 9. Customization Section -- Deploying to different cloud providers: AWS, GCP, and Azure +## Delivery Method +This workshop is delivered through: -## Repo Folder Structure +- Online via Zoom + +## Duration + +- 5 Hours + +## Workshop Outline + +This workshop covers the following concepts: + +1. **Introduction to Containerized Research Workflows** + - Overview of Docker and Kubernetes + - Benefits of containerizing research workflows + - Tools and technologies: Docker, Kubernetes, etc. + +2. **Setting Up the Containerized Environment** + - Creating Dockerfiles and Docker Compose files + +3. **Data Processing Fundamentals** + - Understanding data processing + - Implementing a containerized environment + +4. **Developing the User Interface** + - UI development using Flask + - Containerized UI + +5. **Visualizing and Analyzing Output** + - Creating visualization dashboards + +6. **Deploying to a Public Cloud as a Kubernetes Cluster** + - Setting up a Kubernetes cluster on a public cloud (e.g., AWS, Google Cloud, Azure) + - Deploying containerized applications to Kubernetes + - Managing and scaling applications in Kubernetes + +## End of Workshop Assessment + +- Q&A session +- Feedback + +## Hands-On Lab Exercises + +1. Setting up Docker containers +2. Deploying and managing the complete workflow +3. Deploying the workflow to a Kubernetes cluster on a public cloud + +By the end of this workshop, participants will have a comprehensive understanding of how to build and manage a containerized research workflow using Docker and/or Kubernetes. + + +## Repo Folder Structure (Subject to change) ```plaintext Container-Orchestration-for-Research-Workflows/ -├── README.md ├── Dockerfile +├── README.md +├── README_old.md +├── app +│ ├── entrypoint_extract.sh +│ ├── entrypoint_load.sh +│ └── entrypoint_transform.sh ├── docker-compose.yml -├── k8s/ -│ ├── deployment.yaml -│ ├── service.yaml -│ ├── ingress.yaml -│ ├── configmap.yaml -│ ├── secret.yaml -├── notebooks/ -│ ├── data_processing_notebook.ipynb -├── scripts/ -│ ├── extract.py -│ ├── transform.py -│ ├── load.py -│ ├── ui/ -│ │ ├── app/ -│ │ │ ├── package.json -│ │ │ ├── package-lock.json -│ │ │ ├── src/ -│ │ │ │ ├── index.js -│ │ │ │ ├── components/ -│ │ │ │ │ ├── VisualizationComponent.js -│ │ │ │ ├── public/ -│ │ │ │ │ ├── index.html +├── notebooks +├── requirements.txt +└── scripts + ├── data + │ └── fm-ad-notebook-visualization-COMPLETED.ipynb + ├── extract.py + ├── load.py + └── transform.py ``` \ No newline at end of file diff --git a/archive/README_old.md b/archive/README_old.md new file mode 100644 index 0000000..ab62bb0 --- /dev/null +++ b/archive/README_old.md @@ -0,0 +1,68 @@ +# Container Orchestration for Research Workflows + +## Overview of the Workshop +This workshop provides researchers with a comprehensive guide on containerizing and orchestrating data pipelines. Participants will gain hands-on experience with Docker, Docker Compose, and Kubernetes, and will learn to build and deploy a UI for visualizing data. The workshop focuses on practical applications of container orchestration to enhance research workflows. + +## Agenda + +### 1. Overview of the Workshop +- Introduction to the goals and structure of the workshop +- Explanation of key concepts and technologies + +### 2. Use Case: Containers for Research +- Benefits of using containers for research purposes +- Detailed use case: Containerizing data pipelines + +### 3. Introduction to Containers and Kubernetes +- Understanding the basics of containers +- Overview of Kubernetes and its components + +### 4. Data Processing with Jupyter Notebooks +- Review and understand the data processing Jupyter notebook + +### 5. Building ETL Containers +- Create containers for the Extract, Transform, and Load (ETL) processes + +### 6. Deploying ETL Containers +- Use Docker Compose to deploy ETL containers + +### 7. Building a UI Container +- Build a containerized UI application to visualize transformed data + +### 8. Deploying the UI Container +- Deploy the UI container using Kubernetes + +### 9. Customization Section +- Deploying to different cloud providers: AWS, GCP, and Azure + + +## Repo Folder Structure + +```plaintext +Container-Orchestration-for-Research-Workflows/ +├── README.md +├── Dockerfile +├── docker-compose.yml +├── k8s/ +│ ├── deployment.yaml +│ ├── service.yaml +│ ├── ingress.yaml +│ ├── configmap.yaml +│ ├── secret.yaml +├── notebooks/ +│ ├── data_processing_notebook.ipynb +├── scripts/ +│ ├── extract.py +│ ├── transform.py +│ ├── load.py +│ ├── ui/ +│ │ ├── app/ +│ │ │ ├── package.json +│ │ │ ├── package-lock.json +│ │ │ ├── src/ +│ │ │ │ ├── index.js +│ │ │ │ ├── components/ +│ │ │ │ │ ├── VisualizationComponent.js +│ │ │ │ ├── public/ +│ │ │ │ │ ├── index.html +``` \ No newline at end of file