"A shrunk model still has to run somewhere. Deployment is getting it onto real hardware — reliably, efficiently, and easily."
Level: Advanced · Time: ~13 min · Prerequisites: Module 33
Learning Objectives
By the end of this module, you will be able to:
- Describe how models run on devices
- Recognize runtimes and formats
- Understand tools like Foundry Local
- Follow an edge deployment flow
1. From File to Running Model
A trained, optimized model is just a file. To use it on a device you need a runtime — software that loads the model and runs inference efficiently on that hardware (CPU, GPU, or specialized NPU chip).
Explain like I'm new: The model file is like a music track; the runtime is the music player. You need the right player for your device to actually hear the song. Different devices need different (or configurable) players.
2. Formats & Runtimes
To run everywhere, models are converted to portable formats and paired with runtimes:
| Format / Runtime | Good for |
|---|---|
| ONNX / ONNX Runtime | Cross-platform, many devices |
| GGUF + llama.cpp | Running LLMs/SLMs on CPUs & laptops |
| Core ML | Apple devices |
| TensorFlow Lite | Mobile & embedded |
Key idea: The winning combo is optimized model + right format + efficient runtime for your target hardware. Get these matched and a capable model runs smoothly on modest devices; mismatch them and it's slow or won't run at all.
Concept: Modern devices often carry an NPU — a Neural Processing Unit built specifically to run AI math fast and efficiently, alongside the general-purpose CPU and the graphics-focused GPU. A runtime that knows how to target the NPU can make inference dramatically faster while sipping battery. The catch is that each chip family has its own preferred format and runtime, which is exactly why portable formats like ONNX exist — they let one model file adapt to many different accelerators instead of forcing you to rebuild it for each device.
3. Tools That Make It Easy: Foundry Local
Setting all this up used to be painful. Tools like Foundry Local (and Ollama, LM Studio) package it: download an optimized model, get a local runtime and an API, and call it like a cloud service — but it runs entirely on your machine.
Real-world use case: A developer uses Foundry Local to run a Phi SLM on their laptop, exposing a local API. Their app calls localhost exactly like it would call a cloud LLM — but there's no internet, no cost, and no data leaving the machine. Prototyping edge AI becomes as easy as cloud AI.
4. A Typical Edge Deployment Flow
- Choose a suitable (small) model
- Optimize it (quantize, Module 33)
- Convert to the target format (e.g., ONNX/GGUF)
- Bundle with a runtime for the device
- Test on real hardware (speed, memory, accuracy)
- Ship & monitor (Module 35)
Common mistake: Testing only on a beefy dev machine, then shipping to a weak device. Always test on representative hardware — a model that's snappy on your laptop may crawl on a budget phone. Measure on the real target.
When you test on that real hardware, watch three numbers together. Latency tells you if responses feel instant. Memory tells you whether the model even loads without crashing the app. Energy tells you whether all-day use will drain the battery. A model can pass one and fail another — fast but memory-hungry, or light but power-thirsty — so treat the trio as a single scorecard. Only a model that clears all three on the device your users actually own is truly "deployed."
Hands-On: Try This
Try this: Tools like Ollama or Foundry Local let you run a small model locally with one command. Read their quickstart and note the steps: download model → run → call the local API. Seeing "cloud-like API, but local" makes edge deployment concrete.
✅ Checkpoint
- What is a runtime?
- Why convert models to formats like ONNX or GGUF?
- What do tools like Foundry Local provide?
Answers: 1) Software that loads and runs a model efficiently on device hardware. 2) Portability + efficiency across different devices. 3) An easy way to run optimized models locally with a runtime and a local API.
Key Takeaway: Deploying to the edge = optimized model + portable format (ONNX/GGUF/Core ML/TFLite) + efficient runtime matched to the device. Tools like Foundry Local (or Ollama, LM Studio) package this so you run models locally with a cloud-like API — no internet, no cost, private. Always test on representative hardware.
Further Learning
Adapted from Microsoft's Edge AI for Beginners (MIT License).