Assessing Agents Is a Research Design Problem

Here is the failure mode I see most often. A team decides to measure agent quality. They pick something countable — response time, task completion, thumbs-up rate, an LLM judging outputs against a rubric. They instrument it, they watch the number, they optimize against it.

What they haven't done is establish that the countable thing bears any relationship to the quality they actually care about.

This is construct validity, and it's the first thing you learn to worry about in research methodology. A thumbs-up measures whether the user was satisfied in the moment. It does not measure whether the output was correct, whether it will hold up when acted on, or whether the user was in any position to evaluate it. Users are frequently satisfied by confident, fluent, wrong answers. That's not a flaw in the users. It's a flaw in treating satisfaction as a proxy for quality without ever testing whether the proxy holds.

Every metric in an eval suite is a proxy for something. The methodological question — is this proxy valid for this construct, and how would we know — mostly isn't being asked.

An eval set is a sample. Samples have properties. The properties determine what you can conclude.

Most eval sets I've encountered were assembled the way convenience samples always are: from the cases someone happened to have, plus the failures that were memorable enough to get written down. That produces a set that over-represents dramatic failures and under-represents the boring middle where the system will actually spend most of its operating life.

Then performance on that set gets reported as though it describes performance in general. It doesn't. It describes performance on a non-random sample with a known bias, which is a much narrower claim.

The fix isn't complicated, and it's standard practice in research: define the population of real tasks, sample deliberately across it, stratify by the dimensions you think matter, and be explicit about what the sample can and cannot support. None of that is novel. It's just not how eval sets typically get built.

Session-level evaluation catches session-level problems. A large category of agent failure isn't visible at that resolution.

Whether users develop appropriate trust calibration — trusting the system where it's reliable and checking it where it isn't — takes months to observe. Whether an agent is quietly degrading someone's own capability at the task shows up over a year, not a session. Whether people have started routing around the system, which is the clearest signal of all, only appears if you're watching for workarounds rather than measuring the happy path.

These require longitudinal design, observation, and interviewing. They cannot be extracted from telemetry, because the interesting behavior is often the behavior that stopped happening.

For a subjective task, there is no ground truth waiting to be uncovered. Someone has to decide what quality means, write it down, and defend it.

This is uncomfortable, and I think it's why teams avoid it in favor of metrics that feel objective. But an unstated definition of quality doesn't disappear — it just gets encoded implicitly in whatever got measured, without anyone having examined it.

Making it explicit is standard research practice. Define the construct. Specify what would count as evidence for and against. Establish whether independent raters agree, and if they don't, figure out whether the definition is unclear or the construct is genuinely contested. Contested constructs are fine — they just have to be handled as contested rather than averaged into a single number that hides the disagreement.

Practically, assessing an agent well involves most of what a good study involves: start from the decision the assessment is meant to inform, not from what's measurable. Define quality as a construct with stated criteria. Sample tasks deliberately, with known coverage. Combine instrumented behavior with observation and interview, because each catches what the others miss. Watch over time, not just at launch. Pay specific attention to workarounds and disuse, which are the strongest signals available and rarely instrumented. Be explicit about validity limits.

None of this is exotic. It's a research plan. The novelty is only that it's being applied to an agent rather than a feature.

If assessing an agent is a measurement design problem, then the people trained in measurement design should be involved in it — and currently, mostly, they aren't.

Evaluation has been positioned as an engineering activity because the first generation of it was about model capability, which genuinely is. But as agents move into workflows where quality is subjective, contextual, and defined by the people using them, the assessment problem changes character. It becomes a question about human judgment, which is a research question.

I don't think this is a turf argument. I think it's a gap. Organizations deploying agents are making consequential decisions based on evaluations with unexamined construct validity and convenience samples, and calling the result evidence. The methodology to do better already exists. It's just sitting in a different part of the building.