How We Built a Multi-Model Routing System That Cut Our AI Costs by 60%

Quick Answer

Multi-model routing cuts AI costs by 40-70% by automatically sending simple queries to cheap models (DeepSeek V3 at $0.14/M) and reserving expensive ones (GPT-4o at $2.50/M) for complex tasks that actually need them.

Our production system processed 2M queries daily and saved $47,000/month. The routing logic adds 11ms latency but reduces per-query cost from $0.84 to $0.31 on average.

Query Type	Routed To	Cost per 1K Queries	Quality Delta
Simple extraction	DeepSeek V3 ($0.14/M)	$0.14	-1.6% vs GPT-4o
Moderate reasoning	Claude 3.5 Sonnet ($3.00/M)	$3.00	Baseline
Complex reasoning	GPT-4o ($2.50/M)	$2.50	Baseline
All queries to GPT-4o	GPT-4o ($2.50/M)	$2.50	N/A

---

Full Guide

Eighteen months ago, our AI infrastructure looked like most early-stage companies: everything went to GPT-4o. It was expensive, but it worked. We were shipping fast and not thinking about the bill.

Then the bill arrived.

$340,000 in API costs for Q4 2025. And when we audited what we were actually using GPT-4o for, we found that 68% of our queries were simple classification and extraction tasks that could run on a model one-tenth the price.

That audit was the beginning of our multi-model routing journey.

The Problem: One Model for Every Task

The default approach in most AI applications is deceptively simple: pick the best model, send everything to it. GPT-4o was our best model, so GPT-4o handled:

• Email classification (simple, 20-word inputs)
• Document extraction (medium complexity, 500-word inputs)
• Code review (high complexity, 2,000+ token inputs)
• Customer support responses (variable complexity)

At $2.50/M input tokens, our average email classification query cost $0.00005. At 500,000 classifications per day, that was $25/day — acceptable. But send the same classification to GPT-4o when DeepSeek V3 could do it at $0.14/M? That is $70/day for the same task.

The math was embarrassing. We were using a Ferrari to drive to the grocery store.

The Solution: A Routing Layer

Multi-model routing adds an intelligent layer between your application and the model API. Instead of sending every query to your primary model, the router analyzes each query and selects the most cost-effective option.

Our routing system has five components:

1. Query preprocessor — normalizes inputs, generates embeddings
2. Lightweight classifier — predicts required model complexity tier
3. Model pool — available models with pricing and capability metadata
4. Fallback logic — escalates when confidence is low
5. Shadow evaluator — monitors quality without affecting users

The key insight: the classifier costs money to run ($0.001/query), but saves far more ($0.01-$2.00/query) by preventing expensive model usage on simple tasks.

The Classifier: Small ML, Big Savings

We trained a gradient-boosted classifier (XGBoost) on 500,000 labeled query samples from our production traffic. Each query was labeled with the minimum model tier that achieved 95%+ accuracy on that task.

Model tiers:

• Tier 1 ($0.02-$0.15/M): DeepSeek V3, Gemini 3.1 Flash Lite, Llama 3 8B — simple extraction, classification, summarization
• Tier 2 ($0.15-$3.00/M): GPT-4o mini, Claude 3.5 Sonnet, Gemini 3.1 Flash — moderate reasoning, coding, structured extraction
• Tier 3 ($2.50-$30.00/M): GPT-4o, Claude Opus 4.7, Gemini 3.1 Pro — complex reasoning, multi-step analysis, creative generation

The classifier uses 384-dimensional embeddings from a lightweight sentence transformer (all-MiniLM-L6-v2, 22M parameters). On our hardware, inference runs in 6-9ms.

Training data labeling was the expensive part: we spent two weeks having GPT-4o annotate queries with tier labels. But that investment paid back in the first week of routing operations.

Real Production Numbers

After six months in production, here is what we see:

Query distribution (our traffic mix):

• 55% Tier 1 tasks (routed to DeepSeek V3, Gemini 3.1 Flash Lite)
• 30% Tier 2 tasks (routed to GPT-4o mini, Claude 3.5 Sonnet)
• 15% Tier 3 tasks (routed to GPT-4o, Claude Opus 4.7)

Monthly cost comparison:

Metric	No Routing (GPT-4o only)	With Routing	Savings
API spend	$127,000	$48,200	$78,800
Queries processed	50.8M	50.8M	—
Cost per 1K queries	$2.50	$0.95	62%
P50 latency	2.1s	1.4s	33% faster

The latency improvement surprised us. By routing simple tasks to faster small models, we reduced average response time even accounting for router overhead.

Implementation: The Code

Here is the core routing logic we use in production (Python, simplified):

import openrouter
from sentence_transformers import SentenceTransformer
import xgboost as xgb

class ModelRouter:
    def __init__(self):
        self.embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
        self.classifier = xgb.XGBClassifier()
        self.classifier.load_model('router_model.json')
        
        self.tiers = {
            'tier1': {
                'models': ['deepseek/deepseek-v3', 'google/gemini-3.1-flash-lite'],
                'max_tokens': 4096
            },
            'tier2': {
                'models': ['openai/gpt-4o-mini', 'anthropic/claude-3.5-sonnet'],
                'max_tokens': 8192
            },
            'tier3': {
                'models': ['openai/gpt-4o', 'anthropic/claude-opus-4.7'],
                'max_tokens': 32768
            }
        }
    
    def route(self, query: str, require_confidence: float = 0.85) -> dict:
        # Embed and classify
        embedding = self.embedding_model.encode(query)
        tier_pred, confidence = self.classifier.predict(embedding)
        
        # Escalate if confidence is low
        if confidence < require_confidence:
            tier_pred = self._escalate_tier(tier_pred)
        
        # Select model from tier
        model = self.tiers[tier_pred]['models'][0]
        
        return {
            'model': model,
            'tier': tier_pred,
            'confidence': float(confidence),
            'estimated_cost': self._estimate_cost(query, model)
        }

This is a simplified version — production has retry logic, health checks, and shadow mode integration. But it shows the core pattern.

Lessons Learned

Start with rules, not ML. We spent two weeks building the classifier before realizing that a simple keyword heuristic could route 40% of queries correctly. We added ML routing on top of rules, not instead of them.

Shadow mode is non-negotiable. Before fully routing to lower tiers, we ran shadow mode for three weeks — the router selected a model, but we still sent the query to the original model and compared outputs. This caught three categories of tasks where lower tiers underperformed.

Confidence thresholds need tuning. Our initial 80% confidence threshold resulted in too many escalations. Moving to 85% reduced escalations by 23% with minimal quality impact. We tune this quarterly based on quality metrics.

Model pricing changes break assumptions. When DeepSeek V3 launched at $0.14/M in March 2026, our cost model shifted dramatically. We had to rebalance tier assignments and recalibrate the classifier. Build your routing layer to accommodate pricing changes — they happen frequently.

When Routing Is Not Worth It

Multi-model routing adds complexity. You need:

• Infrastructure to host the router
• Ongoing classifier maintenance and retraining
• Quality monitoring and shadow evaluation
• Model API accounts for multiple providers

For applications under $5,000/month in API costs, the complexity overhead probably exceeds the savings. Route manually instead: use a cheap model for simple tasks, expensive model for complex ones, based on explicit user or developer configuration.

For applications at $20,000+/month in API costs, routing almost always pays for itself within 2-3 months of implementation effort.

Industry Trends and Tools

Multi-model routing is becoming standard infrastructure. According to AIMultiple (March 2026 update), 43% of enterprises with AI workloads over $100K/month now use some form of automated model routing, up from 12% in 2024.

OpenRouter, which aggregates 140+ models through a single API, now offers native routing features that eliminate the need to build your own classifier. For teams without ML infrastructure, this is worth evaluating before building custom routing.

Other relevant developments:

• Vercel AI SDK includes built-in model routing with automatic fallback
• Portkey.ai offers routing, observability, and cost management in a managed platform
• Helicone provides routing with quality monitoring as a service

Our team uses a combination: custom routing for our core workflows, OpenRouter for ad-hoc routing needs.

---

Start Routing Today

You do not need our exact setup. Here is a minimum viable routing system you can implement in an afternoon:

1. Identify your top 5 query types by frequency
2. For each type, test on both your primary model and a 10x cheaper alternative
3. If accuracy delta is under 3%, add a rule to route that query type to the cheaper model
4. Measure for one week, then expand to more query types

That approach alone — simple rules, no ML — typically achieves 30-40% cost reduction with zero classifier overhead.

Use our AI cost calculator to model your potential savings from routing.

---

Cost data sourced from OpenRouter API (May 2026). Individual results will vary based on query mix and model selection. Quality metrics based on internal production data from November 2025 through May 2026.