Finding signal on Twitter is more difficult than it used to be. We curate the best tweets on topics like AI, startups, and product development every weekday so you can focus on what matters.
Originally published by @EXM7777 on X. Tech Twitter preserves the original source alongside this readable edition.
Before you dive in
• Learn why better prompts can't fix AI slop—discover the eval loop system that catches defective output before it ships, using Hermes agent framework.
Best for builders who want practical takeaways. 16 min read.
There's a reason some people seem to be constantly shipping the best software, writing incredible content, or generating insane images...
They adopted the eval loop, while you...
You've tried better prompts, you've switched to the more expensive model, you've written longer instructions, you've turned on memory, you've built context files the size of a novel, and the slop still comes back...
it comes back because you keep fixing the layer that was never broken
slop is not a prompt problem, it's a systems problem, the same way a factory shipping defective units doesn't have a worker problem, it has a quality-control problem, nobody is checking the output before it leaves the building
so this is the build, by the end of this read you'll have a working eval loop running inside hermes, the open-source agent, scoring every output against your standard before it ships, watching your live output after it ships, and turning every failure back into a new test so the quality floor rises on its own
we build it together, one piece at a time, and the payoff is concrete, clean output you can trust without re-reading it at midnight, a quality number you can actually look at, and slop that gets caught on the way out the door instead of by your audience
here's what you're walking away with:
the real reason better prompts, bigger models, and memory never kill slop for good, and the one layer that actually does
the two places slop hides in your work, your content output and your product output, and why the fix is identical for both
what an eval loop is in plain english, the quality layer very few people run daily, and the reason no one ever told you to build one
a quality benchmark you can stand up this week, for content and for product, exactly what to measure and what “good” looks like as a number you can read off a screen
the exact build, step by step, for wiring that whole loop into hermes out of the pieces it already gives you, skills, memory, cron, and approval buttons, so the gate runs without you
if you came here for “the 5 prompts that fix ai slop,” this is not that piece, those exist and they don't work, this is the version that does
We doomscroll, you upskill
Get the 10 tweets shaping how builders think today.
Newsletter
We doomscroll, you upskill
Get the 10 tweets shaping how builders think today.
you rewrote the prompt, three times, four times, you added examples, you added a persona, you added a “do not” list a mile long
you upgraded to the frontier model, paid 5x per token, and the output got more confident without getting less generic
you turned on memory, you built a context file, you fed it your brand voice, your past work, your style guide
and every single one of those moves bought you a few good generations, then the slop crept back in
every one of those is an input-side fix, you keep sharpening the thing that generates while ignoring the thing that should be catching, a better gun fired into the dark still hits nothing
slop is an output-side problem, it's not that the model can't produce good work, it's that you have no way to tell the good work from the bad work before it reaches someone who matters
there's no eval loop, there's no quality benchmark, there's no scoreboard, so you're tuning blind, you change a prompt and you feel like it got better, but feeling is not measurement and a feeling doesn't catch the bad run hiding in the next 50 generations
so you blame yourself, or your prompt, or your agent setup, or your context engineering, when the missing piece is an entire layer of working with ai that you were never shown, and by the end of this piece that layer is going to be running on your own machine inside hermes
a prompt is a hypothesis, the output is the result, and an eval is the only thing that closes the loop between them
without that loop you are guessing forever, you tweak the hypothesis, you eyeball one result, you declare victory, and you never find out that the same prompt produces garbage 30% of the time because you only ever looked at the one output in front of you
the model is non-deterministic, the same prompt run twice gives you two different answers, which means even a perfect prompt produces slop on some percentage of runs and you have no idea which runs until a client or a user is staring at one
so a perfect prompt isn't a quality guarantee, it's a slightly better coin flip, and you're shipping every flip
the reason everyone keeps reaching for prompts anyway is simple, the prompt is the only lever you can actually see, you can edit it, and editing it feels like control
measurement is invisible, nobody sells you a course on it, no one posts a viral thread titled “the eval suite that 10x'd my output,” so the entire conversation stays stuck on the one lever that can't solve the problem on its own
the people whose ai output is consistently clean are not better at prompting than you, they just have a second lever you don't, they measure every output against a standard before it ships, and the measuring is what makes their prompting look like magic
the two places slop lives
slop hides in exactly two places, and almost everyone is only looking at one of them
place 1, your content output
the tweets, the articles, the emails, the landing pages, the posts, anything you generate with ai and publish under your name
slop here looks like work that is technically fine and completely hollow, and it sounds like every other ai account on the timeline, correct on the outside, empty on the inside
it dies in public and you can't articulate why because each individual piece looked okay when you hit send
place 2, your product output
the ai feature you shipped, the agent, the chatbot, the support responder, the extraction pipeline, the thing your users actually touch
slop here looks like a wrong answer delivered with total confidence, a hallucinated number, a broken json payload, a tone that's off for the brand, an output that was great in the demo and quietly degraded three deploys later
it doesn't die in public, it scales in silence, every user gets a slightly worse experience and most of them never tell you, they just leave
these are the same disease with the same cure
content slop and product slop are both un-measured ai output going straight to an audience with no gate in between
the only difference is stakes and visibility, content slop embarrasses you loudly, product slop bleeds you quietly, and the loop we build in hermes grades both with the same skill, so you run one quality system across everything you generate instead of two
what an eval loop actually is
an eval loop is a repeatable test that scores your ai output against a standard, automatically, every time, before it ships and after it ships
that's it, that's the whole thing, and it is the layer almost nobody building with ai has
generate the output
score it against a benchmark you defined
catch the runs that fall below the line
fix what's failing
re-score, and let only the passing output through
software engineers have had this forever, it's called testing, you would never ship code with no tests and just hope it works in production, but that is exactly how the entire industry ships ai output right now, straight from the model to the user on a vibe and a prayer
the reason almost no one has an eval loop is demographic, the people building with ai today came from content, sales, product, founding, not from engineering, so “write tests for your output” was never in the toolkit, evals read as infrastructure for “real” engineers, and the people who need them most assume they're not allowed to want one
think of it as unit testing for the non-deterministic, you're not testing whether the code runs, you're testing whether the output is good, and you're testing it across enough cases that one bad run can't hide
an eval loop runs in three places, and the build ahead puts it in all three:
before you ship, run your new prompt or model against a saved set of cases and confirm it didn't get worse, this is regression testing, it's how you stop a change that fixes one thing and silently breaks three others
at runtime, score the output as it's generated and let conditional logic catch the failures before they reach the user, this is the guardrail
in production, score a sample of real executions continuously so you can see quality degrading the day it starts, not the week a client complains
you can stand up the first one in a spreadsheet, but running all three continuously without it turning into a second job is the whole reason we're putting this inside an agent
the moment quality becomes a number, slop stops being a feeling you keep having and becomes a bug you can fix, you can't debug a vibe, you can debug a score that dropped from 0.82 to 0.61
the benchmark, the three parts you're about to build
a benchmark has three parts, and they're the same three parts whether you're grading content or grading a product:
test cases, real inputs paired with what good output looks like (your ground truth)
metrics, how you turn an output into a score, ideally 0 to 1
a threshold, the line below which nothing ships
build those three and you have a quality gate, skip any one and you have a wish, the rest of this section is what goes inside each part, then we wire all three into hermes
for content, your test cases are your gold standard
pull 20 to 50 of your best pieces, the bangers, the posts that got bookmarked, the articles you'd put your whole name behind, this is what “good” looks like, you're not inventing a standard, you're extracting the one you already hit on your best days
for content, your metric is a rubric
a score is only as good as the rubric behind it, so encode what you actually believe makes the work good, for content i grade every piece on four criteria:
it explains how to do something specific, not a vibe, an action the reader can take tomorrow
anyone in the audience can follow it, no jargon walls, no inside-baseball
it's structured, replicable, step by step, not just inspirational
it's novel, the reader had no idea you could do this
the meta-criterion sitting on top of all four, would someone bookmark this and come back to implement it later, if the answer is no it's slop no matter how clean the prose reads
the trick is the rubric, a vague rubric (“is this good and engaging”) produces a vague score, a specific rubric (“does this contain at least one copy-paste-able template or playbook”) produces a score you can trust, the judge inherits your taste only if you actually write your taste down
for product, your test cases come from your logs
pull the actual inputs your feature sees, from your logs, from real user sessions, not the three happy-path examples you tested on launch day, the cases that break you are the weird ones and the weird ones live in your logs
for product, your metric matches the task
for each input, define what a correct output looks like, then match the metric to the task, exact match when there's one right label, a validator when the structure has to hold, semantic similarity plus a judge when the output is open-ended, the metric just has to return a number because a number is the only thing you can put a threshold on
for both, the threshold is the line you hold
0.7 is a reasonable place to start, anything below 0.7 gets reworked or killed before it ships, no exceptions, the threshold only works if you never let a 0.6 through because you liked it, the whole point is to take the late-night ego out of the decision
that's the benchmark, now we make it run by itself
building the loop inside hermes
hermes does not ship an evals button, there's no dashboard called quality where you click “turn on slop protection”
what hermes gives you instead is better, the raw pieces of an eval loop as primitives you assemble once and then own
skills it writes for itself and reuses, persistent memory that grows across sessions, built-in cron that delivers to any platform, approval buttons in slack, and a self-improvement habit baked into the core
hermes calls itself “the agent that grows with you” and that growth is exactly the loop we're building
so let's wire it, six moves
move 1, stand hermes up where it can reach you
install it and connect it to telegram, this matters more than it sounds, because the gate only works if it can interrupt you, hermes runs across 20+ channels and ships native approval buttons into slack and telegram, so the agent can do work in the background and tap you on the shoulder when a decision is yours
move 2, load your gold standard into memory
hermes has persistent memory that grows across sessions with full cross-session recall, so the 20 to 50 best pieces from your benchmark go in there once and stay, this is the part that's normally scattered across screenshots and old drafts, here it's the agent's long-term memory, queryable, the ground truth your scores get measured against
move 3, turn your rubric into a judge skill
this is the heart of it, you tell hermes once, in plain english, to create a skill that takes an output plus your rubric and returns a 0 to 1 score per criterion with a one-line reason, that's llm-as-a-judge, an agent grading your llm, and a model with a sharp rubric is a more consistent critic than you are because it has no ego in the piece and no attachment to the one sentence you're secretly proud of
the reason this lives as a skill and not a one-off prompt is that hermes skills are procedural memory, the agent writes them, keeps them, and reuses them, you encode your taste a single time and it grades every output forever, and skills compound, nous found agents with 20+ self-created skills finish similar tasks 40% faster because they stop rediscovering the process, your judge gets sharper the more it runs
move 4, make the suite a skill, not a spreadsheet
your test cases plus the metric functions become a skill hermes holds and versions, the metric library is whatever the task needs, exact match for classification, regex for extraction, json and key-and-value validators for structure, semantic similarity for generative output
your judge skill for the open-ended stuff, hermes writes the scoring code itself, you describe the task and it builds the metric, all of it sitting in one place the agent owns instead of a sheet you'll lose
move 5, gate the ship with regression testing and an approval button
this is the highest-leverage habit in the whole system and the one nobody keeps doing by hand, so we hand it to the agent, wire it so any change, a new prompt, a swapped model, a tweaked pipeline, triggers the suite, hermes re-runs every case, computes the score delta against baseline, and instead of silently shipping it pings you in slack, scores went 0.81 to 0.74, two cases regressed, approve?, and it only advances when you tap the button
you can keep it locked on that job with `/goal`, which holds the agent to a target across turns, and for anything bigger its multi-agent kanban can decompose the run, score in parallel, and schedule it, so the gate is a standing process, not a thing you remember to run
move 6, watch production on a cron and close the loop
hermes has built-in cron that delivers to any platform, so you schedule a job that samples real executions, scores them with the same judge skill, and dms you the moment the line dips, you catch degradation the day it starts instead of the week a client complains, “the eval score dropped” is a problem you can act on, “a customer seemed annoyed” is not
then the part that makes the whole thing compound, when you flag a bad output with a thumbs-down in slack, hermes writes it back into the suite skill as a new test case, that failed run becomes a permanent check, and because self-improvement is what hermes is, not a feature bolted on the side, the suite hardens every week on its own, the floor rises while you sleep
what good looks like once this is running, concretely, a content piece under 0.7 on your rubric never ships, a product change that drops any metric below baseline blocks the deploy until you approve it, and the production score line stays flat or climbs, the day it dips is the day hermes pings you, not the week the churn shows up
the part nobody wants to hear
the reason your ai output is inconsistent is not that you're bad at prompting, and it's not that the model isn't smart enough yet
it's that you're running a generation step with no quality step, you built half a system and you've been blaming the half that works
the fix isn't a better prompt, it's a missing layer, define what good looks like, turn it into a number, score every output against it, gate everything that falls below the line, and close the loop so the floor rises every week, and now that layer isn't a someday project, it's six moves inside an agent that runs on your own machine
do that and slop stops being a thing that randomly happens to you and becomes a thing you catch on the way out the door, every time, the same way a real factory catches a defect before it ever reaches a customer
the prompt was never the system
the eval loop is the system, hermes is where it runs, and now you have it
