07-cloud-cleanup

Overview

In this lesson, we will clean up the resources we created in Google Cloud Platform (GCP) during our weather data pipeline deployment. Proper cleanup is essential to avoid unnecessary costs and maintain a tidy cloud environment. We'll cover the process of deleting Kubernetes resources, the GKE cluster, container images, and other associated resources.

By the end of this section, you will have:

1. Deleted all Kubernetes resources created during the deployment
2. Removed the GKE cluster
3. Cleaned up container images from Google Container Registry
4. Removed any other associated GCP resources

This cleanup process demonstrates responsible cloud resource management and helps you avoid unexpected charges on your GCP account.

Prerequisites

Before starting this lesson, please ensure that you have:

1. Completed the 06-cloud-deployment lesson
2. Google Cloud SDK installed and configured
3. kubectl installed and configured to work with your GKE cluster
4. Access to the Google Cloud Console

Lesson Content

5.1 Delete Kubernetes Resources

First, we'll remove all the Kubernetes resources we created:

kubectl delete cronjob data-pipeline-sequence
kubectl delete job,deployment,service --all
kubectl delete secret db-credentials

These commands will delete the cronjob, all jobs, deployments, services, and the database credentials secret we created.

5.2 Delete GKE Cluster

Now, let's delete the GKE cluster:

gcloud container clusters delete weather-cluster --zone=us-central1-a --quiet > /dev/null 2>&1 &

To check the status of the cluster deletion:

gcloud container clusters describe weather-cluster --zone=us-central1-a

If the cluster has been successfully deleted, this command should return an error indicating that the cluster doesn't exist.

5.3 Delete Container Images

Clean up the container images you pushed to Google Container Registry:

# List images
gcloud container images list

# Delete images (repeat for each image)
gcloud container images list-tags us-docker.pkg.dev/${PROJECT_ID}/my-docker-repo/IMAGE_NAME --format='get(digest)' | xargs -I {} gcloud container images delete us-docker.pkg.dev/${PROJECT_ID}/my-docker-repo/IMAGE_NAME@{} --force-delete-tags --quiet

Replace ${PROJECT_ID} with your actual project ID and IMAGE_NAME with each image name (data-pipeline-extract, data-pipeline-load, data-pipeline-transform, flask-app).

5.4 Clean Up Other Resources

Check for and delete any persistent disks that might have been created:

# List disks
gcloud compute disks list

# Delete disks if any exist
gcloud compute disks delete DISK_NAME --zone=ZONE

Replace DISK_NAME and ZONE with the appropriate values if any disks are listed.

5.5 Final Verification

After running all the cleanup commands, it's a good practice to double-check the Google Cloud Console to ensure all resources have been removed. Pay special attention to:

1. Kubernetes Engine
2. Artifact Registry
3. Compute Engine (for any lingering disks or instances)
4. VPC Network (for any created firewall rules or IP addresses)

Conclusion

In this lesson, you learned how to properly clean up the resources created during the deployment of your weather data pipeline on Google Cloud Platform. This process included deleting Kubernetes resources, removing the GKE cluster, cleaning up container images, and verifying the deletion of all associated resources.

Proper cleanup is crucial in cloud environments to avoid unnecessary costs and maintain a well-organized cloud infrastructure. The steps you've learned here can be applied to other projects and deployments, ensuring you always leave your cloud environment in a clean state after completing your work.

Key Points

• Always clean up cloud resources when they're no longer needed to avoid unnecessary costs
• Kubernetes resources should be deleted before deleting the cluster
• GKE cluster deletion may take some time; always verify its status
• Container images in Google Container Registry should be cleaned up to save storage costs
• Double-check the Google Cloud Console to ensure all resources are properly removed

Further Reading

• Google Kubernetes Engine: Deleting a cluster
• Cleaning up Container Registry images
• Google Cloud resource clean-up best practices
• Kubernetes resource management
• GCP billing and cost management