AI inference is when a trained AI model makes predictions on new data (like ChatGPT generating responses to an input). It's challenging because it must be fast, reliable, and cost-efficient — requirements that often conflict with each other. Success is measured by latency (speed), throughput (efficiency), and cost.
Every day, AI applications support millions of users, giving instant, seemingly magical answers. Behind every AI application is a process that’s invisible to end users but determines everything about their product experience: inference. But what is AI inference, and why is it so important for building scalable AI applications?
The two stages of AI: Training and inference
Working with AI models involves two distinct stages:
The training stage, during which a model learns how to perform a task (like recognizing images, generating text, or making decisions).
The inference stage, where the model puts what it has learned into practice.
Think of training as the education phase: you're feeding the model massive amounts of data, adjusting its parameters through many iterations, and essentially teaching it to recognize patterns and relationships. During training, the model gradually improves its understanding by learning from its mistakes. This process is computationally intensive and can take days or weeks, depending on the model's complexity and the amount of data involved.
AI inference is the process of using a trained AI model to make predictions on new data. In this phase, the model applies what it’s learned to become useful in the real world. Unlike training, inference must be fast and efficient, as it often occurs in real-time as users interact with AI applications.
For example, when you ask an LLM-powered application like ChatGPT a question and get a reply back, that is AI inference—getting an output from a model based on a corresponding input.
AI has two stages: Training (when a model learns to perform tasks) and inference (when the model applies what it learned).In this article, we’ll break down how inference works, why it’s hard to do well, and how to learn more about inference.
What happens during AI inference?
To understand what happens during inference, let’s trace the lifecycle of a request from the end user to the model server and back.
First, the user hits an API endpoint, either directly or through a user interface. A request is sent with both the user’s input and any model parameters (such as the maximum number of tokens), plus appropriate authentication headers.
This request is then sent to the most appropriate model server. Advanced systems make intelligent routing decisions, sending a request to the best server based on geo-aware load balancing followed by LoRA-aware or KV cache-aware routing. The request may also need to be enqueued, requiring queue management with timeouts and priorities.
Once the request actually reaches a model server, it’s time for the inference runtime to take over. The model server is equipped with GPU and CPU resources for running inference and generally runs an inference framework like:
TensorRT-LLM: An open-source framework by NVIDIA with highly optimized CUDA kernels.
SGLang: An open-source framework with high extensibility and customizability.
vLLM: An open-source framework with support for a wide range of models.
Custom runtimes built on technologies like ONNX, PyTorch, Transformers, and Diffusors.
These frameworks handle the actual inference steps, from tokenization to prefill to decode.
Finally, results are sent back to the user. Depending on the model, outputs may be streamed across different protocols (SSE, WebSockets, gRPC) or sent in a single response after generation is finished. For long-running or asynchronous inference requests, results may be sent to a webhook.
The inference request lifecycle: Requests are sent to an API endpoint where they're routed to a model server, are processed on the server, and then results are sent back to the end-user.AI inference in action: Real-world applications
Here are some examples of when AI inference happens in the real world:
When you ask ChatGPT a question and it generates a response.
When Google Translate converts text between languages.
When your email filters spam.
When voice assistants process your spoken commands.
Inference powers AI apps spanning medical search, transcription, AI-powered video editing, and more.
Why AI inference is hard to build
Building production-ready AI inference systems is one of the most challenging aspects of AI development.
The complexity comes from three core challenges:
Speed requirements are unforgiving. Users expect instant responses, and moving from “decent” to “excellent” latency requires sophisticated optimizations across the entire inference stack.
Reliability is key for mission-critical applications. Users demand high availability and consistent performance for a reliable user experience.
Cost optimization becomes critical at scale. Every inference request consumes expensive compute resources, and inefficiencies compound quickly across millions of users.
What makes this particularly challenging is that these requirements often conflict with each other. Optimizing for speed might increase costs, while cost-cutting measures can hurt reliability.
Successfully building AI inference systems requires carefully balancing these tradeoffs while implementing optimizations across multiple layers. That’s where Baseten’s Inference Stack comes in.
The anatomy of an inference stack
Solving these challenges requires sophisticated optimizations across every layer of the stack, from runtime to infrastructure, from the models themselves to the hardware and numerous software layers in between, and from the GPUs to the inference runtime.
The Baseten Inference Stack bundles all of these optimizations into a single platform, combining the best open-source technologies with our own proprietary enhancements. Every model you deploy on Baseten inherits these benefits by default while remaining fully configurable.
At the runtime level, we implement techniques such as:
Custom kernels
Speculation engine
Model parallelism
Agentic tool use
At the infrastructure level, we add:
Geo-aware load balancing
SLA-aware autoscaling
Protocol flexibility
Multi-cluster management
How to measure inference success
The three pillars of inference performance—latency, throughput, and cost—each tell a critical part of the story. Here's what to measure and why it matters:
Latency: This is how fast your model responds. A key latency number when streaming model output is time to first token—the delay between a user's request and the appearance of the first generated text. Another is total generation time, and end-to-end completion time (this matters most for non-streaming uses).
Throughput: When it comes to throughput, you're essentially looking at how much your model can handle at once. The key metrics here are tokens per second, which measures your raw processing capacity, and requests per second, which is your standard API metric (though this varies quite a bit depending on your input and output lengths). There's an interesting trade-off to consider: while higher concurrency will boost your throughput, it can actually hurt your latency, so you'll need to find the right balance for your use case.
Cost: As for cost, you'll want to carefully select your hardware to meet your performance requirements. One effective strategy for reducing cost per token is batching, where you process multiple requests together rather than handling them individually.
Solving AI inference in production
Our comprehensive whitepaper breaks down the architecture, performance benchmarks, and real-world implementation strategies that teams use to scale their AI applications. Check it out to learn more about all of the different layers of AI model inference.
Subscribe to our newsletter
Stay up to date on model performance, GPUs, and more.