ai-architecture
teardown
05 Sept 2026
Uber's agent stack taken apart, and the part its write-up leaves out
Uber published the gateway, the token service and the MCP layer. It never published what happens when a model dies mid-request, which is the part that broke my own cascade three times.
By Purvansh Parmar
Checked 2026-09-05
Read 9 min
Entries 9
Uber has published more of its agent platform than almost anyone. There is a Go gateway in front of every model, a registry that knows which workload may host which agent, a token service that mints a fresh scoped credential for every hop, and a crawler that projects thousands of internal APIs into MCP. It is a good write-up. Read it while building the same shape at one thousandth the scale and a gap opens, because the parts Uber chose to describe are the parts that hold still, and the part that broke my own gateway is not in there at all.
How this was checkedI run the same shape at one service: an OpenAI compatible endpoint in front of three models. It broke in the way Uber's write-up never mentions. NVIDIA retired nemotron-3-nano mid-deploy and it began answering 410 Gone, which took my terminal down. gpt-oss-120b measured 0.7s to first token on one probe and 9.5s on the next, minutes apart. A congested run then starved all three models on one shared budget and turned a recoverable request into a 502, though the invocation itself lived 10.4s and still returned my own JSON. Every timeout figure below came out of fixing that.
What I compared on
- what the part actually does
- what it costs to run
- whether it still earns its place at one service instead of thousands
A Go service wrapping every model vendor behind one endpoint, so no caller ever holds a vendor SDK.
Wins
Centralising the vendor clients is what makes everything else possible. Cost attribution, audit logs and a kill switch all need one place that every model call passes through, and retrofitting that after sixty use cases have each imported their own SDK is a migration nobody funds.
Loses
It is a single point of failure wearing a platform badge. Uber runs 16 million queries a month through it at a peak of 25 queries per second, which is small enough that one Go service is the right call, and large enough that when it is down all sixty use cases are down together.
16M queries/monthpeak 25 QPS60+ use cases~30 teams
Mirroring the OpenAI HTTP and JSON contract so LangChain and LlamaIndex work against it unmodified.
Wins
This is the highest leverage decision in teh design and it costs nothing to make. Every client library, every tutorial example, and every provider worth switching to already speaks it. My own gateway repoints from NVIDIA NIM to Groq or OpenRouter with 1 environment variable for exactly this reason.
Loses
You inherit a contract you do not control, including its gaps. Reasoning models do not fit it cleanly. The knob that suppresses chain of thought is chat_template_kwargs.thinking on nemotron and reasoning_effort on gpt-oss, and strict providers reject both as unknown body params.
1 env var to repoint2 incompatible thinking knobs
Swaps sensitive values for numbered placeholders before a third party sees them, then restores them.
Wins
The placeholders carry sequential numbers, so ANONYMIZED_NAME_0 refers to the same person throughout a long prompt and the model can still reason about who did what to whom. That numbering is the detail that makes redaction survive contact with a real conversation rather than destroying it.
Loses
It only works on data you can pattern match, and it puts a lossy transform in the hot path of every single request. A name the redactor misses is a name in a third party log forever. A name it catches wrongly is a prompt the model then answers badly.
ANONYMIZED_NAME_02 passes per request
The source of truth for which agent is authorised to run on which workload.
Wins
Without it there is no answer to whether a process claiming to be the deploy agent actually is one. The registry is what lets the token service refuse a workload that is trying to host an agent it was never authorised to host, which is the whole impersonation problem in one check.
Loses
It is pure overhead until more than one team is shipping agents. At a single service the mapping is 1 constant in 1 file, and standing up a registry to hold a single row is the kind of platform work that feels productive and changes nothing about what can go wrong.
1 row at small scale
A token service minting a fresh short-lived JWT for every hop, each scoped to one audience.
Wins
The blast radius of a leaked token drops from forever to minutes. Uber measured the exchange call at a P99 below 40 milliseconds, which is close to free next to any model call it precedes, and it is the mechanism that makes the actor chain below trustworthy rather than self reported.
Loses
Every hop gains a dependency that fails closed. Forty milliseconds is nothing against a model that takes 600, but it is a second service that must be up before any agent can do anything at all, and a time to live measured in minutes quietly turns a paused workflow into an expired one.
P99 under 40msTTL in minutes
Each token carries the attested lineage, from the human who started it through every agent since.
Wins
This is the part worth stealing at any scale. Without it a downstream service sees only its immediate caller, so an action taken 4 hops deep is attributable to a robot rather than to the person who asked for it. Policy can then judge the human and the acting agent at the same time.
Loses
The chain means nothing unless every service on the path propagates it, which makes it an all or nothing migration across thousands of services. One library that drops the claim does not break anything visibly. It just makes the audit trail quietly wrong, which is worse than missing.
conceptually RFC 86931 audience per hop
A crawler that projects existing internal service APIs into MCP tools with one config change.
Wins
The insight is that the tools already exist as service APIs, so the work is projection rather than authoring. Uber made thousands of microservices agent callable without asking every owning team to hand write and then maintain a tool definition that would drift from the API within a quarter.
Loses
An API projected automatically is a tool with no affordances. Endpoint and field names written for another service are not a description a model can choose between, so a gateway exposing thousands of them has moved the problem from integration to selection, and selection is the harder one.
1 config changethousands of services
An in-house opinionated wrapper over LangGraph and LangChain, bent toward Uber's own infrastructure.
Wins
A wrapper is the honest answer when a framework is close but not aligned with your infrastructure. It gives one place to encode the gateway, the token exchange and the actor chain, so the person authoring an agent does not reimplement any of the three and cannot forget the third.
Loses
You now maintain a fork shaped object against 2 dependencies that move faster than you do. Uber has 5,000 engineers to amortise that across. A team of one writing the same wrapper is writing a second framework, and will stop updating it about four months in.
5,000 engineers2 upstream deps
Trying the next model when the current one dies, which the published architecture never describes.
Wins
It is the part that earns its keep first. My cascade runs 3 model ids in order and commits to a stream only when a visible token arrives, never when the connection opens, because reasoning models emit 39 to 94 chunks of thought before any content and a socket that opened is not a model that answered.
Loses
It is hard to size and easy to get wrong in a way that makes things worse. One shared budget across 3 attempts starved all of them during a congested run and turned a recoverable request into a 502. The per-attempt ceiling is now 8 seconds inside an 18 second cascade budget, retrying the same model only when it failed inside 1.5 seconds.
8s per attempt18s cascade budget1.5s cheap failure39-94 thinking chunks
Side by side
| Part | What it buys | Worth it at one service? |
|---|
| GenAI Gateway | One choke point for cost, audit, kill switch | Yes, it is 1 file |
| OpenAI-shaped interface | Every client library works unmodified | Yes, free |
| PII redactor | Third parties never see raw values | Only with real user data |
| Agent Registry | Workloads cannot impersonate agents | No, 1 row |
| Per-hop token exchange | Leak window of minutes, P99 under 40ms | No |
| Actor chain | Attribution 4 hops deep | Not yet, steal it later |
| MCP Gateway | Thousands of APIs projected as tools | No |
| Lang Effect | One place to encode the platform | No, it rots |
| Model fallback | A dead model stops being an outage | Build this first |
Verdict
Build model fallback first, ahead of all of it. Uber can leave it out of the write-up because a vendor outage at their scale is a conversation with an account manager, while mine is a 502 on a page someone is reading right now. The registry, the actor chain and the redactor solve problems that arrive on the day you have thousands of agents and a compliance team asking who did what. A cascade across three model ids solves the problem that arrives on a Tuesday, when one id starts answering 410 Gone and nothing else in the stack notices or cares.
Sources