Skip to content
Achraf Abderrazik
ENFR
All work
Academic · Team of 5Case study 05 / 05

Distributed Model Serving

Replicated inference API

A scalable inference API: a FastAPI model service replicated behind an Nginx gateway, with a versioned model registry.

Context
Distributed Systems module · 5 members
Year
2026
Deployment
Docker Compose
Valid responses
100%
Every request in the documented load test returned a valid response: 100 requests, 20 at a time.
FastAPI replicas
2
Identical copies of the model service, with Nginx as the single entry point in front of them.

01Context

A five-person team project for the Distributed Systems module: turning a trained machine-learning model into an inference service designed to scale horizontally.

02Problem

A single inference server is both a bottleneck and a single point of failure. The goal: an API that scales by adding instances, with every model version tracked.

03Architecture

  1. 01Client

    Inference requests

  2. 02Gateway

    Nginx

    Single entry point

  3. 03Service

    FastAPI · instance 1

    03Service

    FastAPI · instance 2

  4. 04Registry

    Versioned models

    Model registry

Two FastAPI replicas behind an Nginx gateway, orchestrated with Docker Compose.

04Team

Team project (5 members) focused on scalable model inference, with a FastAPI service replicated across two instances behind an Nginx gateway, Docker Compose deployment and a versioned model registry.

05Technical decisions

  • 01

    Scale out, not up

    The FastAPI service runs as identical replicas, so capacity grows by adding instances rather than a bigger server.

  • 02

    One gateway

    Nginx is the single entry point; clients never address a replica directly.

  • 03

    Versioned models

    A model registry versions every model the service can load.

  • 04

    Reproducible deployment

    Docker Compose defines the gateway and the replicas as one reproducible stack.

06Results

  • A complete serving stack, from gateway to model registry, reproducible with Docker Compose
  • Load-tested under concurrent requests, with the results documented

07Stack

  • Python
  • FastAPI
  • Nginx
  • Docker Compose
  • scikit-learn

08Source code

There’s no public repository for this project. Other projects are on GitHub.

Next case studySafe Invest AI Lead AutomationMultilingual LLM lead pipeline

Building something similar?

Start a project