“What LLM should we standardise on?”
It’s one of the first questions organisations ask as they begin adopting AI; it’s also, I think, becoming the wrong question.
Over the past few years, we’ve become accustomed to comparing models through benchmarks, leaderboards and feature announcements. Every few months there’s another “best” model. Better reasoning. Larger context windows. Faster inference. Lower pricing. More multimodal capabilities.
The pace of change is extraordinary and will only increase as they compete for market share, which raises an interesting architectural question…
Why are we designing systems around technology that we already know will change?
The lesson we learned from data
Earlier this year, I wrote about Data Flexibility; the principle was simple: applications shouldn’t be tightly coupled to databases, storage technologies or APIs. They should consume business information through stable semantic and business abstractions, allowing the underlying technology to evolve without disrupting the business.
The architecture was designed for change; AI demands the same thinking. We shouldn’t depend directly on GPT, Claude, Gemini or any other individual model.
Today’s best model won’t be tomorrow’s
Think about how quickly the AI landscape is evolving. New models are being released every few months, and existing models improve dramatically. Add to this API pricing changes and the continued improvement of open-source models, the question isn’t whether change will happen; it’s whether our applications are prepared for it.
It’s about much more than cost; dimensions for LLM Selection
Commercial pricing is the obvious example. A provider can change API pricing overnight and completely alter the economics of an AI solution.
| Assessment Dimension | Why it Matters |
|---|---|
| Capability | Overall reasoning quality, coding ability, writing quality and task performance. |
| Cost | API pricing, subscription costs and token economics can significantly alter the business case. |
| Latency & Performance | Response times, throughput and scalability affect user experience and operational efficiency. |
| Context & Token Efficiency | Context window size, tokeniser efficiency and reasoning efficiency influence cost, speed and solution design. |
| Specialisation | Some models excel at coding, others at reasoning, multilingual tasks, document analysis or agentic workflows. |
| Security & Privacy | Data handling, encryption, retention policies and enterprise security controls vary between providers. |
| Governance & Compliance | Regulatory certifications, auditability, explainability and industry compliance requirements continue to evolve. |
| Deployment Options | API, SaaS, VPC, on-premises or self-hosted open-weight models each have different architectural implications. |
| Licensing & Commercial Terms | Licensing models, usage rights, enterprise agreements and regional availability may change. |
| Ecosystem & Integration | SDK maturity, tool calling, MCP support, orchestration frameworks and cloud ecosystem integration differ considerably. |
| Vendor Viability | Financial stability, product roadmap, innovation pace and long-term commitment influence strategic risk. |
| Geopolitics & Sovereignty | Export controls, national AI strategies, sovereign AI requirements, procurement policies and data residency obligations may dictate which models can be used. |
Does this mean changing models is easy?
No, far from it. Different models have different APIs, prompt behaviours, context limits. Safety behaviours, to name just a few differences. Anyone who has worked across multiple models knows they aren’t interchangeable. But that’s not the objective; the objective isn’t to make changing models effortless. It’s to ensure changing models doesn’t require rebuilding every application that depends on AI. Good architecture has never eliminated change; it has always reduced the cost of change.
The next evolution of enterprise AI
I suspect mature organisations won’t think in terms of choosing a single LLM; instead, they’ll build AI platforms capable of working with many models. Business applications won’t know—or need to know—whether they’re talking to GPT, Claude, Gemini or something that hasn’t been invented yet.
An orchestration layer will make decisions based on policy, capability, governance, latency, cost and business requirements; model selection becomes an operational decision, not an architectural constraint.
Looking beyond benchmarks
The industry spends enormous energy comparing benchmark scores, and those benchmarks are valuable; they tell us which models perform best today. They do not help architects answer the question …