Most teams shipping AI products in 2026 are not training models. They’re calling APIs.
OpenAI, Anthropic, Gemini, whichever frontier model fits the use case. The model is a commodity at this point.
What isn’t a commodity is the infrastructure around it: the data layer the LLM reasons over, the evaluation harness that catches regressions, the orchestration logic that makes agentic workflows actually reliable in production.
And yet the most common hiring mistake I see: companies bringing on MLOps engineers and ML researchers to build AI products that will never touch a training run.
LLM API Vs. Custom ML: What Architecture Do You Actually Have?
The split is cleaner than most people admit. If your AI product calls an external model API, you have an LLM API architecture. If your product trains or fine-tunes weights on proprietary data, you have a custom ML architecture.
Custom ML is rare. It makes sense for companies with massive proprietary datasets, latency constraints that rule out external APIs, or competitive moats that depend on the model itself. A handful of well-capitalized labs. Some defense contractors. A few specialized verticals.
Everyone else is building on top of GPT-4o, Claude, or Gemini. Which means the problems you’re solving are not model problems. They’re data problems, evaluation problems, and orchestration problems.
The infrastructure required is completely different. MLOps tooling (experiment tracking, model registries, GPU cluster management) is mostly irrelevant. What you actually need looks a lot more like data engineering plus software engineering with an LLM shaped component in the middle.
Also Read: Cursor Vs Claude Code
What Agentic AI Actually Requires Under The Hood
Agentic AI is the current frontier for most product teams: systems where the LLM doesn’t just answer a question, it takes a sequence of actions, calls tools, reads context, and produces a result that depends on multiple steps going right.
The failure modes are almost never in the model. Frontier models are capable. They fail when the context they receive is bad, when the tools they call return unexpected data, when the orchestration logic doesn’t handle edge cases, or when there’s no mechanism to detect when the output has degraded.
MCP (Model Context Protocol) and tool-use patterns make this concrete. When an LLM can call your internal APIs, query your database, or interact with third-party services, the quality of what it produces is almost entirely determined by:
- What data it gets access to, in what format
- Whether the tool responses are clean and predictable
- Whether there’s an evaluation layer that can flag when something goes wrong
None of those are ML problems. The first is data engineering. The second is API design and integration work. The third is eval infrastructure, which most teams skip entirely and then wonder why their AI feature keeps regressing.
MCP and Tool-Use Need Evaluation Infrastructure, Not Model Monitoring
Model monitoring, in the traditional ML sense, is about catching data drift, feature skew, model decay. Useful when you own the model. Mostly irrelevant when you’re calling a third-party API.
What agentic systems need instead is behavioral evaluation: does the system produce the right output given this input, across a representative sample of real cases?
This means:
- A curated set of test cases that covers the distribution of real user inputs
- Expected outputs or rubrics for those cases, so you can measure quality automatically
- A regression suite that runs on every deploy and catches when a prompt change, tool change, or context change breaks something
- Logging that makes it possible to audit failures and add them back into the test set
Most teams I work with have none of this when they come in. They’re doing vibes-based QA: manually checking a few outputs before shipping, then triaging Slack messages when users report problems.
That’s not a process you can scale. And it’s not a data problem that more model capability will fix.
The Data Layer Determines AI Quality More Than The Model Choice
This is the insight that most teams resist, because it’s less exciting than model selection.
Switching from GPT-4o to Claude 3.5 Sonnet will not fix a retrieval problem.
If the chunks your RAG system returns are stale, out-of-context, or poorly formatted, the model will do its best with bad inputs and produce mediocre outputs. Upgrading the model gives you marginally better mediocre outputs.
The teams shipping AI products that actually work have invested in the data layer first:
- Clean, structured, up-to-date knowledge bases with clear provenance
- Retrieval that understands the query intent, not just keyword overlap
- Context assembly that gives the model exactly what it needs and nothing it doesn’t
- Pipelines that keep the data layer current as the underlying sources change
A loan readiness assessment that pulls from stale borrower data is wrong, regardless of which model scores it. An AI assistant that reasons over a poorly structured knowledge graph gives inconsistent answers, regardless of which API you call.
The model is the reasoning engine. The data layer is the fuel. Bad fuel ruins the engine’s output.
5 Signs You Need Data Engineering, Not ML Engineering
Use this as a diagnostic before your next hire or vendor conversation.
- Your AI feature calls an external model API. You don’t train weights. You need engineers who understand data pipelines and context assembly, not gradient descent.
- Your outputs are inconsistent across similar inputs. This is almost always a retrieval or context problem, not a model problem. An ML engineer won’t fix it.
- You have no systematic way to test AI quality before shipping. You need eval infrastructure. That’s a software and data engineering problem.
- Your AI feature depends on internal data that changes frequently. Keeping that data layer current and queryable is a data engineering problem.
- You’re building agentic workflows with tool-use or MCP. The reliability of those workflows depends on clean APIs, predictable tool responses, and orchestration logic. That’s engineering, not ML research.
The Mismatch Costs More Than A Bad Hire
Hiring the wrong profile for an AI product team doesn’t just slow you down by a quarter. It shapes what problems you think you’re solving.
An ML engineer looks at an underperforming AI feature and thinks about fine-tuning, training data, model selection. A data engineer looks at the same feature and asks what the model is actually receiving as input, whether the retrieval is accurate, and whether there’s a way to measure quality systematically.
The second framing gets you to the root cause faster, almost every time.
The talent market conflates “AI engineering” with “ML engineering” because the terminology is still settling. Most job descriptions are written by people who learned about ML in a different era, before LLM APIs made custom model training optional for the majority of products.
Getting this right early, before you’ve built a team around the wrong mental model, is worth more than any individual hire.
Conclusion
The AI products shipping in 2026 are built on APIs, not models.
That changes what expertise you need and what infrastructure you’re building. Eval harnesses, data pipelines, and orchestration logic are the work.
Get the mental model right first. The hiring and architecture decisions follow from there.
Add your first comment to this post