Product Evals
Turn an observation into a testable improvement.
An evaluation workflow that connects real product tasks, observed outcomes, and supporting evidence to prioritized findings and concrete acceptance conditions.
THE DESIGN QUESTION
What makes a product critique useful enough for a team to act on—and specific enough to test again?
Define success before running the task. Separate observation from inference, preserve contrary evidence, and make the next verification step explicit.
- My contribution
- Evaluation criteria, task design, structured evidence records, and the local report-validation and rendering workflow.
- Current scope
- Evaluation workflow · first-round sample
- Exploring
- Task-based evals · Evidence · Review gates
THE WORKFLOW
Define
Specify the task, constraints, and success conditions.
Observe
Run the task and retain the relevant output and evidence.
Reason
Distinguish observations, impact inferences, and causal hypotheses.
Verify
Prioritize supported findings and define how to retest them.
A CLOSER LOOK
A correct edit can still violate the task.
Archived Lovart observation · 10 October 2026 · one run, pending review
Change only the date on a poster.
The request explicitly required the original image dimensions to remain unchanged.
The date changed correctly. The dimensions changed too.
The original file was 1088 × 1440; the edited file was 1536 × 2048. The original remained available.
A small edit expanded the user’s checking work.
That impact is an inference. The underlying model or routing cause was not established by this observation.
Check both the intended change and protected properties.
Repeat the task and compare the date, exported width and height, and agreed non-target regions.
This is a dated, single-run sample—not a current product rating or reliability estimate. Formal human review and repeated samples are still pending.
UNDER THE SURFACE
Make the process legible.
Make the boundaries explicit.
Evaluation design
Task-based criteria cover product experience, AI output quality, and human–AI collaboration.
Evidence controls
Structured records link findings to evidence; untested hypotheses cannot receive a confirmed priority.
Review gates
Schema checks enforce required fields and review attribution. A report cannot be marked reviewed while its findings remain pending.
TODAY
Evaluation workflow
A ten-task set, three first-round task runs, structured evidence, and a working HTML/PDF report generator are available locally.
WHAT’S NEXT
Still exploring
Repeat key tasks, calibrate criteria with human reviewers, and extend the workflow to more product types.
Human-led, machine-assisted evaluation. The report generator validates and presents recorded findings; it does not autonomously inspect products or assign scores.