README.md

# Container Orchestration for Research Workflows

# Workshop Overview

In this workshop, you will learn how to set up and manage a containerized research workflow with Extract, Load, Transform (ELT) features and a user interface (UI) to view and analyze outputs. We will review containerization fundamentals, ELT processes, and create an intuitive UI for end users.

## Course Level

- Intermediate

## Intended Audience

This workshop is intended for:

- **Primary Audience**: Researchers and data scientists interested in streamlining their workflows.
- **Secondary Audience**: IT professionals and developers looking to implement containerized solutions for data processing and analysis.

## Workshop Objectives

By the end of this workshop, you will be able to:

- Understand the process to implement containers for a workshop
- Set up a containerized environment for research workflows
- Implement data processing
- Containerize a user-friendly interface for accessing output and visualization

## Delivery Method

This workshop is delivered through:

- Online via Zoom

## Duration

- 5 Hours

## Workshop Outline

This workshop covers the following concepts:

1. **Introduction to Containerized Research Workflows**
   - Overview of Docker and Kubernetes
   - Benefits of containerizing research workflows
   - Tools and technologies: Docker, Kubernetes, etc.

2. **Setting Up the Containerized Environment**
   - Creating Dockerfiles and Docker Compose files

3. **Data Processing Fundamentals**
   - Understanding data processing
   - Implementing a containerized environment

4. **Developing the User Interface**
   - UI development using Flask
   - Containerized UI

5. **Visualizing and Analyzing Output**
   - Creating visualization dashboards

6. **Deploying to a Public Cloud as a Kubernetes Cluster**
   - Setting up a Kubernetes cluster on a public cloud (e.g., AWS, Google Cloud, Azure)
   - Deploying containerized applications to Kubernetes
   - Managing and scaling applications in Kubernetes

## End of Workshop Assessment

- Q&A session
- Feedback

## Hands-On Lab Exercises

1. Setting up Docker containers
2. Deploying and managing the complete workflow
3. Deploying the workflow to a Kubernetes cluster on a public cloud

By the end of this workshop, participants will have a comprehensive understanding of how to build and manage a containerized research workflow using Docker and/or Kubernetes.


## Repo Folder Structure (Subject to change)

```plaintext
Container-Orchestration-for-Research-Workflows/
├── Dockerfile
├── README.md
├── README_old.md
├── app
│   ├── entrypoint_extract.sh
│   ├── entrypoint_load.sh
│   └── entrypoint_transform.sh
├── docker-compose.yml
├── notebooks
├── requirements.txt
└── scripts
    ├── data
    │   └── fm-ad-notebook-visualization-COMPLETED.ipynb
    ├── extract.py
    ├── load.py
    └── transform.py
```