# Product Evals

Evaluation workflow · 2026-10-26

An evaluation workflow that connects real product tasks, observed outcomes, and supporting evidence to prioritized findings and concrete acceptance conditions.

## Design question

What makes a product critique useful enough for a team to act on—and specific enough to test again?

Define success before running the task. Separate observation from inference, preserve contrary evidence, and make the next verification step explicit.

## My contribution

Evaluation criteria, task design, structured evidence records, and the local report-validation and rendering workflow.

## Workflow

1. **Define** — Specify the task, constraints, and success conditions.

2. **Observe** — Run the task and retain the relevant output and evidence.

3. **Reason** — Distinguish observations, impact inferences, and causal hypotheses.

4. **Verify** — Prioritize supported findings and define how to retest them.

## Mechanisms

**Evaluation design** — Task-based criteria cover product experience, AI output quality, and human–AI collaboration.

**Evidence controls** — Structured records link findings to evidence; untested hypotheses cannot receive a confirmed priority.

**Review gates** — Schema checks enforce required fields and review attribution. A report cannot be marked reviewed while its findings remain pending.

## A correct edit can still violate the task.

Archived Lovart observation · 10 October 2026 · one run, pending review

**Task** — Change only the date on a poster.

The request explicitly required the original image dimensions to remain unchanged.

**Observed** — The date changed correctly. The dimensions changed too.

The original file was 1088 × 1440; the edited file was 1536 × 2048. The original remained available.

**Interpretation** — A small edit expanded the user’s checking work.

That impact is an inference. The underlying model or routing cause was not established by this observation.

**Retest** — Check both the intended change and protected properties.

Repeat the task and compare the date, exported width and height, and agreed non-target regions.

This is a dated, single-run sample—not a current product rating or reliability estimate. Formal human review and repeated samples are still pending.

## Current state and next steps

A ten-task set, three first-round task runs, structured evidence, and a working HTML/PDF report generator are available locally.

Repeat key tasks, calibrate criteria with human reviewers, and extend the workflow to more product types.

Human-led, machine-assisted evaluation. The report generator validates and presents recorded findings; it does not autonomously inspect products or assign scores.

https://linadong.design/lab/product-evals