Production traffic is not uniform. You get a few requests that need your best model, but most are simple questions and lookups you can solve with cheaper, faster models. The most expensive mistake you can make today is sending every request to your strongest model. You need routing. Period. This is the simplest trick to improve the architecture of whatever you are building. Please, don't implement routing yourself. You don't have to. I'm currently working with TrueFoundry's Auto Routing.…

@2d ago

We cut model costs 69% by routing requests on complexity. Here's the data. We just released Auto Routing on the TrueFoundry AI Gateway. It reads each incoming request, scores how hard it is, and sends it to the model you assigned to that level of difficulty. You define three