Press Space to continue
Press Space to continue
You can’t automate what you can’t measure. This means that evals are one of the gates to diffusion of AI in the enterprise. We can test our deterministic processes through software, but most enterprises have no useful way of understanding how their non-deterministic processes are working today. Specifically the work that agents are doing for them. Evals are mission critical for enterprises adopting AI because you have no other way of knowing what’s working, what’s broken, what changed, what…
Alex Lieberman@businessbarista1d ago
Just spoke to one of the big data labeling businesses. Few interesting insights: - They predict the majority of their revenue will come from Fortune 1000 enterprises, not labs in a few years - They believe every company will want to own their intelligence, but owning intelligence does not necessarily mean using open source models - A company’s evals will become their main proprietary IP given the improvement in agent performance after properly setting up & running internal eval environments - Most enterprises haven’t graduated from coding agents and it’s largely due to not having the proper eval infrastructure to make non-Eng agents performant