Automating ETL Workflows with CI/CD Pipelines for Machine Learning Applications

• Author(s): Antony Satya Vivek Vardhan Akisetty ; Ashish Kumar ; Murali Mohana Krishna Dandu ; Prof. (Dr) Punit Goel ; Prof. (Dr.) Arpit Jain; Er. Aman Shrivastav
• Paper ID: 1705069
• Page: 478-497
• Published Date: 30-10-2023
• Published In: Iconic Research And Engineering Journals
• Publisher: IRE Journals
• e-ISSN: 2456-8880
• Volume/Issue: Volume 7 Issue 3 September-2023

Abstract

In today's fast-paced data-driven landscape, automating Extract, Transform, Load (ETL) workflows is crucial for enhancing the efficiency of Machine Learning (ML) applications. This research explores how Continuous Integration and Continuous Deployment (CI/CD) pipelines can automate and streamline ETL processes, reducing the time and manual intervention required for data preparation and deployment. Integrating CI/CD pipelines into ETL workflows ensures that the entire data lifecycle—from extraction, transformation, loading, to model training and deployment—operates seamlessly. The automation of these processes enables rapid iteration, minimizes errors, and accelerates time-to-market for ML models. This paper investigates key strategies for leveraging modern tools such as Jenkins, Apache Airflow, and Kubernetes to build scalable and efficient automated workflows. It also examines real-world case studies where CI/CD pipelines have optimized ML workflows, leading to enhanced productivity, accuracy, and cost savings. By adopting such automation techniques, organizations can better manage large-scale data pipelines, ensure model accuracy, and reduce operational complexities in machine learning projects.

Keywords

Automated ETL workflows, CI/CD pipelines, Machine Learning applications, data lifecycle automation, Jenkins, Apache Airflow, Kubernetes, model deployment.

Citations

IRE Journals:
Antony Satya Vivek Vardhan Akisetty , Ashish Kumar , Murali Mohana Krishna Dandu , Prof. (Dr) Punit Goel , Prof. (Dr.) Arpit Jain; Er. Aman Shrivastav "Automating ETL Workflows with CI/CD Pipelines for Machine Learning Applications" Iconic Research And Engineering Journals Volume 7 Issue 3 2023 Page 478-497

IEEE:
Antony Satya Vivek Vardhan Akisetty , Ashish Kumar , Murali Mohana Krishna Dandu , Prof. (Dr) Punit Goel , Prof. (Dr.) Arpit Jain; Er. Aman Shrivastav "Automating ETL Workflows with CI/CD Pipelines for Machine Learning Applications" Iconic Research And Engineering Journals, 7(3)