diff --git a/README.md b/README.md new file mode 100644 index 0000000..0421597 --- /dev/null +++ b/README.md @@ -0,0 +1,67 @@ +# Container Orchestration for Research Workflows + +## Overview of the Workshop +This workshop provides researchers with a comprehensive guide on containerizing and orchestrating data pipelines. Participants will gain hands-on experience with Docker, Docker Compose, and Kubernetes, and will learn to build and deploy a UI for visualizing data. The workshop focuses on practical applications of container orchestration to enhance research workflows. + +## Agenda + +### 1. Overview of the Workshop +- Introduction to the goals and structure of the workshop +- Explanation of key concepts and technologies + +### 2. Use Case: Containers for Research +- Benefits of using containers for research purposes +- Detailed use case: Containerizing data pipelines + +### 3. Introduction to Containers and Kubernetes +- Understanding the basics of containers +- Overview of Kubernetes and its components + +### 4. Data Processing with Jupyter Notebooks +- Review and understand the data processing Jupyter notebook + +### 5. Building ETL Containers +- Create containers for the Extract, Transform, and Load (ETL) processes + +### 6. Deploying ETL Containers +- Use Docker Compose to deploy ETL containers + +### 7. Building a UI Container +- Build a containerized UI application to visualize transformed data + +### 8. Deploying the UI Container +- Deploy the UI container using Kubernetes + +### 9. Customization Section +- Deploying to different cloud providers: AWS, GCP, and Azure + + +## Repo Folder Structure + +```plaintext +Container-Orchestration-for-Research-Workflows/ +├── README.md +├── Dockerfile +├── docker-compose.yml +├── k8s/ +│ ├── deployment.yaml +│ ├── service.yaml +│ ├── ingress.yaml +│ ├── configmap.yaml +│ ├── secret.yaml +├── notebooks/ +│ ├── data_processing_notebook.ipynb +├── scripts/ +│ ├── extract.py +│ ├── transform.py +│ ├── load.py +│ ├── ui/ +│ │ ├── app/ +│ │ │ ├── package.json +│ │ │ ├── package-lock.json +│ │ │ ├── src/ +│ │ │ │ ├── index.js +│ │ │ │ ├── components/ +│ │ │ │ │ ├── VisualizationComponent.js +│ │ │ │ ├── public/ +│ │ │ │ │ ├── index.html \ No newline at end of file