ML Inference Endpoint on AWS & Alibaba Cloud
A hands-on deep-dive into production-grade GPU model serving — deploying a handwriting-recognition (MNIST) model behind an HTTP API on two clouds at once, consumed by a small desktop client that lets you draw a digit and pick which backend answers it.
The goal was to gain hands-on experience with production-grade, GPU-backed ML model serving in both AWS and Alibaba Cloud by deploying a pretrained handwriting-recognition model (MNIST) for inference behind an HTTP API consumed by a desktop client.
Solution
On AWS the trained model is served using a SageMaker real-time inference endpoint. On Alibaba, Platform for AI (PAI)’s Elastic Algorithm Service (EAS) is used instead. Both services provision a GPU instance with a managed Deep Learning Container containing NVIDIA Triton Inference Server for model serving. On AWS, a dedicated API Gateway forwards inference requests to the SageMaker endpoint; on Alibaba PAI-EAS a public endpoint is created automatically. API authorization is realized with an API token for simplicity.
All cloud resources are created from code — AWS CloudFormation (via CDK) and Alibaba Resource Orchestration Service (via ROS CDK).
For the client side, a desktop application was built where the user draws a digit, which is converted to Triton’s data format, sent to the deployed model API via HTTP, and the returned prediction is visualized. A switch lets the user pick between the AWS and Alibaba backend.