Essay · 2026 · 7 min read
Assessing Agents Is a Research Design Problem
Evals answer whether the system performed. They don't answer whether it was any good.
The vocabulary around AI evaluation came from engineering, and it shows. Benchmarks, pass rates, red-teaming, regression suites. All of it is built to answer a question with a defensible right answer: did the system produce the correct output.
That question matters. But is also not the question most enterprise agents need answered.
When I built a set of agents for a research practice, I needed to know more than whether the agent produced valid output. I also needed to know whether the report was good — useful to the person reading it, accurate in a way that mattered, structured so a stakeholder could act on it. Those are not properties you can benchmark. They're constructs, and constructs have to be operationalized before they can be measured at all.
That's research design. It has a hundred years of methodology behind it. And it's almost entirely absent from how AI systems are currently assessed.
The construct validity problem
Here is the failure mode I see most often. A team decides to measure agent quality. They pick something countable — response time, task completion, thumbs-up rate, an LLM judging outputs against a rubric. They instrument it, they watch the number, they optimize against it.
What they haven't done is establish that the countable thing bears any relationship to the quality they actually care about.
This is construct validity, and it's the first thing you learn to worry about in research methodology. A thumbs-up measures whether the user was satisfied in the moment. It does not measure whether the output was correct, whether it will hold up when acted on, or whether the user was in any position to evaluate it.
Every metric in an eval suite is a proxy for something. The methodological question — is this proxy valid for this construct, and how would we know — mostly isn't being asked.
Sampling is doing more work than anyone admits
An eval set is a sample. Samples have properties. The properties determine what you can conclude.
Most eval sets I've encountered were assembled the way convenience samples always are: from the cases someone happened to have, plus the failures that were memorable enough to get written down. That produces a set that over-represents dramatic failures and under-represents the boring middle where the system will actually spend most of its operating life.
The fix isn't complicated, and it's standard practice in research: define the population of real tasks, sample deliberately across it, stratify by the dimensions you think matter, and be explicit about what the sample can and cannot support. None of that is novel. It's just not how eval sets typically get built.
Some things only reveal themselves longitudinally
Session-level evaluation catches session-level problems. A large category of agent failure isn't visible at that resolution.
Whether users develop appropriate trust calibration — trusting the system where it's reliable and checking it where it isn't — takes months to observe. Whether people have started routing around the system, which is the clearest signal of all, only appears if you're watching for workarounds rather than measuring the happy path.
These require longitudinal design, observation, and interviewing. They cannot be extracted from the telemetry.
What “good” means has to be decided, not discovered
For a subjective task, there is no ground truth waiting to be uncovered. Someone has to decide what quality means, write it down, and defend it.
Making it explicit is standard research practice. Define the construct. Specify what would count as evidence for and against. Establish whether independent raters agree, and if they don't, figure out whether the definition is unclear or the construct is genuinely contested. Contested constructs are fine — they just have to be handled as contested rather than averaged into a single number that hides the disagreement.
What a research-designed assessment looks like
Practically, assessing an agent well involves most of what a good study involves: start from the decision the assessment is meant to inform, not from what's measurable. Define quality as a construct with stated criteria. Sample tasks deliberately, with known coverage. Combine instrumented behavior with observation and interview, because each catches what the others miss. Watch over time, not just at launch. Pay specific attention to workarounds and disuse, which are the strongest signals available and rarely instrumented. Be explicit about validity limits.
Why this matters for who does the work
If assessing an agent is a measurement design problem, then the people trained in measurement design should be involved in it — and currently, mostly, they aren't.
Evaluation has been positioned as an engineering activity because the first generation of it was about model capability, which genuinely is. But as agents move into workflows where quality is subjective, contextual, and defined by the people using them, the assessment problem changes character. It becomes a question about human judgment, which is a research question.