Prefactor

Evaluate your AI Agents in real-time

SaaS

Developer Tools

Artificial Intelligence

Visit WebsiteSee on Product HuntTwitter ⧉LinkedIn ⧉

How are you handling latency? I imagine Prefactor is scanning through all of my agent traces and flagging things. Is it flagging proactively, or retrospectively? If it's proactively, I'd imagine latency and speed is a problem. I wouldn't want having to wait for Prefactor to finish scanning before my agent can continue its run.

The line about real-time control needing real-time legibility stuck with me. If someone has to scroll to find the problem, the alert may as well not exist. Refreshing to see design treated as part of the safety story.

Hey @matt_doughty — the observe → evaluate → act loop closing all inside one run is the part I keep coming back to. Most eval tooling only gets you the first two, then hands you a dashboard and a black box. I've been digging into agent-initiated payment flows lately, and "catch it live and block before it commits" is exactly the primitive you'd want on any payment-adjacent tool call, not a retro three days later.

Also really like that you open-sourced the deterministic core (no LLM in the loop by construction) as prefactor-evals — most eval startups keep that bit proprietary as the moat. Good trust signal for a category people are rightly skeptical of right now.

The live scoring plus action piece is the part that clicks for me. If an agent drifts in production, finding out three days later is already too late.

"Most agents pass their evals and fail in production" matches what we saw, and the reason it happened to us was not that our evals were too easy. It was that evals measure the average and production punishes the tail.

We run an AI that answers and places real phone calls. Our offline numbers looked fine the entire time the product felt unreliable to users, because a handful of bad calls a day outweighed a few hundred good ones. A 95% agent that fails loudly loses to a 90% agent that fails safely and says "I wasn't sure, so I didn't." Nothing in an averaged eval score expresses that, so if real-time scoring gives me a distribution and a worst-case rather than a mean, that is the part I would care about most.

The other thing that cost us, and it is a scoring-design question rather than an infrastructure one: we let a cheap model judge its own capability, and it started refusing intents it genuinely supported. Refusal rate climbed while capability stayed flat. That is close to the worst failure mode available, because it looks like caution from the inside and reads as the product being broken from the outside, and no accuracy metric flags it. Judgement had to move to a separate leg with no stake in the answer.

So the question I would ask of any evaluation layer: does the scorer share context with the agent it is scoring, or is it genuinely independent? An evaluator that sees the agent's own reasoning tends to find it persuasive, which is exactly when you need it not to be.

Congrats on the launch! It would be really interesting to see how well it integrates with the other softwares, especially the agent orchestrators as there are multiple agents running in parallel and such insights can help improve the orchestration.

Congrats Matt and team! may I ask for the original story of why do you build this?

The part I’d be most curious about is how Prefactor connects transcript quality to real-world side effects.

For agents, the scary failure is often not “bad answer in the log,” but “the system said it updated something and the external state never changed.” If an eval can check the rendered page, written row, sent message, or skipped action against the agent’s claim, that feels much closer to production trust than scoring the conversation alone.

Using this currently to secure our agents, absolutely shocking to me that people give the keys to the kingdom to their AI without considering how this all plays out. Love this product.

The evals pass then production drifts problem has a close cousin in ordinary end to end testing, where a suite goes green on a dev laptop and red on a slower CI box. Same root cause both times: the check was written against conditions that only held in one environment.

What I would want to know here is how you handle the scoring being non deterministic itself. If the judge is a model, its verdict drifts too, so a regression alert could mean the agent changed or the scorer changed. Do you pin the scorer version and replay old runs through new scorers to separate those two?

Congrats on the launch.

This is such a valuable tool. As someone who ships agents to clients, an automated testing/scoring tool would really scale my QA processes.

Congrats on the launch! How do you evaluate the AI Agents is it deterministic or using LLMs by providing the truth table?

Congrats on launching! The "did it actually do it, or just say it did" framing will ring true for anyone who has put an agent in front of real users. The 25k spans a month free tier makes it easy for a small team to give this a proper try. How do the custom spans that pull context from Linear or GitHub work in practice, do you write those checks yourself or pick from templates?

Matt, following on from the forum thread where I said the zero-token layer does most of the work: the thing I keep bumping into is that every deterministic check I own exists because something already went wrong once. Banned phrases, schema conformance, the contradicts-its-own-inputs assertion, all of it got written the morning after a bad output shipped. So my free layer's coverage is really a map of my incident history, and the failure it structurally cannot see is the one I haven't met yet.

That interacts awkwardly with scoring 100% of traffic. Full coverage on rules-someone-wrote reads as all-green on the dashboard, and all-green on 100% is a much stronger-sounding claim than all-green on a 30% sample, which is the version everyone already knows to distrust.

You told Yolanda the pattern layer asks how the agent's behaviour differs "from what you know to be the case", and that phrase is where my question lands, because that's still a declared baseline. Is there anything that flags a run as unusual without someone having said what usual is, off span shape, tool sequence, turn count? That's the piece that catches the first instance rather than the second, and it's the only part of this I've never managed to build for myself.

Congrats Matt, Simon and team, this thread is a masterclass! One angle nobody’s asked: your pitch says built for engineering teams, but at our company the agents are built and owned by ops and marketing people like me, not engineers. Once the SDK is wired up, can the business owner of an agent actually live in Prefactor day to day, reading quality scores and approving held runs without understanding what a span is? The 40-agents-no-honest-answer problem exists at non-engineering companies too, we just have nobody to hand the dashboard to.

congrats on the launch! curious as to what metrics they evalaute the other AI agents on

Real-time eval is the piece most agent stacks skip until something breaks in prod. Curious how you handle scoring when there's no ground-truth label — is it rubric-based, or are you comparing against a reference trace? Congrats on the launch.