Container Orchestration for Research Workflows

Workshop Overview

In this workshop, you will learn how to set up and manage a containerized research workflow with Extract, Load, Transform (ELT) features and a user interface (UI) to view and analyze outputs. We will review containerization fundamentals, ELT processes, and create an intuitive UI for end users.

Course Level

• Intermediate

Intended Audience

This workshop is intended for:

• Primary Audience: Researchers and data scientists interested in streamlining their workflows.
• Secondary Audience: IT professionals and developers looking to implement containerized solutions for data processing and analysis.

Workshop Objectives

By the end of this workshop, you will be able to:

• Understand the process to implement containers for a workshop
• Set up a containerized environment for research workflows
• Implement data processing
• Containerize a user-friendly interface for accessing output and visualization

Delivery Method

This workshop is delivered through:

• Online via Zoom

Duration

• 5 Hours

Workshop Outline

This workshop covers the following concepts:

1. Introduction to Containerized Research Workflows

   • Overview of Docker and Kubernetes
   • Benefits of containerizing research workflows
   • Tools and technologies: Docker, Kubernetes, etc.

2. Setting Up the Containerized Environment

   • Creating Dockerfiles and Docker Compose files

3. Data Processing Fundamentals

   • Understanding data processing
   • Implementing a containerized environment

4. Developing the User Interface

   • UI development using Flask
   • Containerized UI

5. Visualizing and Analyzing Output

   • Creating visualization dashboards

6. Deploying to a Public Cloud as a Kubernetes Cluster

   • Setting up a Kubernetes cluster on a public cloud (e.g., AWS, Google Cloud, Azure)
   • Deploying containerized applications to Kubernetes
   • Managing and scaling applications in Kubernetes

End of Workshop Assessment

• Q&A session
• Feedback

Hands-On Lab Exercises

1. Setting up Docker containers
2. Deploying and managing the complete workflow
3. Deploying the workflow to a Kubernetes cluster on a public cloud

By the end of this workshop, participants will have a comprehensive understanding of how to build and manage a containerized research workflow using Docker and/or Kubernetes.

Repo Folder Structure (Subject to change)

Container-Orchestration-for-Research-Workflows/
├── Dockerfile
├── README.md
├── README_old.md
├── app
│   ├── entrypoint_extract.sh
│   ├── entrypoint_load.sh
│   └── entrypoint_transform.sh
├── docker-compose.yml
├── notebooks
├── requirements.txt
└── scripts
    ├── data
    │   └── fm-ad-notebook-visualization-COMPLETED.ipynb
    ├── extract.py
    ├── load.py
    └── transform.py