Duration: 5 hours | Difficulty: Advanced | Prerequisites: Modules 03–08, basic DevOps
Learning Objectives
By the end of this module, you will be able to:
- Design HTTP APIs for LLM features with streaming, auth, and versioning
- Apply caching strategies that preserve correctness
- Implement rate limiting and fair usage controls
- Monitor latency, errors, and cost in production
- Package services with Docker and ship minimal cloud-ready artifacts
1. API Design for AI Services
FastAPI LLM Service
| Choice | Recommendation |
|---|---|
POST /v1/chat | Version in path |
| Streaming | StreamingResponse or SSE |
| Auth | API keys + rotation; OAuth for humans |
| Idempotency | Idempotency-Key header for billable ops |
Contract tips: document max_tokens defaults, timeout behavior, and error JSON shape—clients will bake in assumptions fast.
Try This! Write an OpenAPI snippet before code—forces you to name error cases.
2. Caching Strategies
Semantic Caching
Hash or embed prompts; return prior answer if similar enough. Great for FAQs; risky for personalized or regulated answers.
| Cache type | Risk |
|---|---|
| Exact key | Stale but safe logically |
| Semantic | Near-duplicate collisions |
Concept: Cache policies with TTLs and user-segment keys—never cross-tenant.
3. Rate Limiting
Token bucket or leaky bucket per user, tenant, and global. Pair with queue for burst smoothing.
| Signal | Action |
|---|---|
| 429 storm | Backoff + circuit breaker |
| Single hot key | Fair-share throttle |
4. Monitoring and Observability
Custom Metrics Collector
Track: p50/p95 latency, tokens in/out, dollars/request, error codes, tool-call success rate.
LangSmith Integration
Trace chains in dev/staging; mirror critical spans to your APM (OpenTelemetry) in prod if vendor-approved.
| Dashboard | Question it answers |
|---|---|
| Cost | Which route burns budget? |
| Errors | Provider vs app fault? |
| Quality | Eval scores trending? |
Fun Fact: The first production incident is always timeout-related—set client and server deadlines.
5. Cost Optimization
| Lever | Detail |
|---|---|
| Model routing | Small LLM for triage |
| Prompt compression | Drop boilerplate |
| Cache | Only where safe |
| Batch offline jobs | Cheaper throughput |
| Anti-pattern | Symptom |
|---|---|
Always max_tokens=4096 | You pay for unused completion headroom |
| No per-tenant budget | One customer drains the pool |
| Silent retries | 3× cost on flaky paths |
Try This! Add a temporary middleware that logs estimated USD per request—share the histogram in your next retro.
6. Deployment with Docker
Key Example: Minimal multi-stage pattern—slim base image, non-root user, env-injected secrets.
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app ./app
ENV PYTHONUNBUFFERED=1
USER nobody
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]Pair with docker compose for local API + vector DB; in cloud, map secrets to the orchestrator’s secret store—not baked into layers.
Key Takeaway
Production LLM services are normal microservices plus token economics—monitor dollars like CPU.
Practice Exercises
Exercise 1: API Gateway (Beginner)
Kong/NGINX rate limit + key auth in front of stub LLM.
Exercise 2: Semantic Cache (Intermediate)
Embeddings + Redis; tune similarity threshold.
Exercise 3: Cost Dashboard (Intermediate)
Export Prometheus metrics from FastAPI.
Exercise 4: Auto-Scaling Strategy (Advanced)
HPA on GPU nodes vs queue workers.
Exercise 5: Full Production Stack (Advanced)
Terraform + CI + canary flag for prompt version.
Mini-Project: Production LLM Service
FastAPI + streaming + Redis cache + structured logs + Dockerfile + README runbook (timeouts, rollback).
Key Takeaways
Key Takeaway
- Version APIs and prompts together.
- Cache only when correctness allows; segment by tenant.
- Rate limits protect wallets and neighbors.
- Metrics must include tokens and $ alongside latency.
- Docker + secret injection is table stakes for reproducible deploys.