ML Model Promotion Pipeline
A Git-based promotion strategy for a lead-time forecasting model across separate dev, integration, and production AWS accounts — solving training-data curation, pipeline orchestration, and validated-model promotion as one connected system.
An ML model for lead-time forecasting needed a repeatable way for training, deployment, and promotion. Three problems had to be solved: curation of high-quality training data, orchestration of model training/validation/deployment, and promotion of a validated model across separate dev, int, and prod AWS accounts.
Training data curation
Raw data in the form of Iceberg tables is drawn from a central data lake and processed using AWS Glue jobs into bronze and silver layers, then re-deposited into another S3 bucket in the data lake. AWS Athena re-exposes the data as the interface, and a feature store is built on top of it.
Model training pipeline
Implemented via SageMaker Pipelines, chosen over Step Functions because it is purpose-built for MLOps — pipeline caching, native hyperparameter tuning, model lineage, experiment tracking, feature stores, and model registry integration, reducing operational complexity. The pipeline includes a config loader, data loader, conditional retraining logic (full retrain vs. incremental fine-tune based on model age), a validation gate, and model deployment to a real-time test endpoint.
Model promotion
Feature branches merge via pull request into a shared dev branch, which auto-merges via CI into int, which auto-merges via CI into prod, with a manual approval gate between each stage. Each git stage maps to both an AWS account and an MLflow model registry stage, so a git promotion and a registry/account promotion happen together rather than as separate manual steps. Model versions carry champion/challenger aliases as they move through registry stages — a new version enters as challenger alongside the current champion and is promoted explicitly rather than silently overwriting it.