LangSmith LLM Gateway is currently in private beta. Sign up here to request access.
For the last few years, AI usage was easy to ignore in a budget. At LangChain, model usage was largely confined to a few teams, usage was predictable, and monthly bills were manageable.
However, in the last year that stopped being true with a few things happening at once:
AI usage went from a few teams to the whole company
The best models got more expensive
Agents got powerful enough to easily fire off dozens of model calls to finish a single task.
As a result, AI spend became harder to understand and harder to control in real time.
The sharpest version of this showed up in engineering. One developer using coding agents heavily could generate thousands of dollars in weekly spend before anyone noticed. Our leadership needed a way to see spend as it was happening, set limits by team and user, and prevent accidental runaway usage without blocking productive work.
Implementing an LLM Gateway
With cost tracking in mind, we built LangSmith LLM Gateway straight into the product we use every day. The immediate goal was simple: prevent accidental runaway spend by coding agents to bring peace of mind to both the VP of Engineering and Head of Finance.
In LangSmith LLM Gateway, budgets can be set across several dimensions:
Organization-wide
Workspace
User
API key
We have default budgets that every employee can hit on monthly, weekly, daily, and hourly windows, with the ability to set exceptions for people working on projects that need higher usage.
We applied the Gateway everywhere in the company where it was possible to do so centrally: coding agents through Claude Code, Codex, or LangChain Deep Agents. Every eligible coding agent call at the company runs through the LLM Gateway, giving our engineering leadership a bird’s eye view of company-wide spend to the minute. And we made sure it was easy to implement, orchestrating it centrally through our MDM so each user wouldn’t have to process the setup themselves.