1. Home
  2. Methodology

The rules we test by

How We Test AI Writing Tools

Every score on this site comes from the same five tasks, run the same way, judged on the same rubric. This page publishes the whole thing — including the actual blog brief — so you can check our work or run it yourself.

Principles

The five-task suite

1. Blog post — 1,200 words from a fixed brief

The heaviest-weighted task. We test: does the draft use the supplied facts correctly, hold an argument without circling, and come out at a length it earns? We time the human edit needed to make it publishable — that edit time feeds the editing-burden sub-score.

2. Product descriptions — 8 items, deliberately boring

Eight kitchen products (a mid-range chef's knife, a silicone spatula set, and so on). Boring is the point: any tool can write about dramatic products. We check for varied structure, concrete detail, and — the classic failure — identical openers stamped across items.

3. Email sequence — 5-part onboarding series

A SaaS welcome sequence from a one-paragraph product summary. We judge arc (do the emails progress, or pitch five times?), subject-line quality, and restraint — urgency-mongering costs points.

4. Factual-accuracy trap

A 500-word explainer on a topic we've salted knowingly: the popular understanding contains at least two well-documented misconceptions. We check whether the tool repeats them, hedges them, or corrects them — and whether it invents statistics or citations under pressure to sound authoritative. We rotate the trap topic between test rounds and don't name the current one, because publishing it would let vendors tune for it.

5. Brand-voice imitation

Every tool gets the same 600-word writing sample — dry, short sentences, plain verbs, no exclamation points — and the same instruction: write a 300-word piece in this voice. We grade on rhythm, register, and the tells (verb inflation, added enthusiasm, formatting habits) that reveal the machine underneath.

The actual blog brief

This is the verbatim brief from the 2026 test rounds. It stays fixed for the year so scores stay comparable; it rotates annually.

Write a 1,200-word blog post for the owner of a small independent coffee shop (2 locations, 9 staff) explaining whether they should introduce a paid loyalty program. Audience: practical, time-poor, skeptical of marketing advice. Use these three facts: (1) replacing a lost regular customer costs roughly five times more than retaining one; (2) in a 2025 industry survey, 41% of paid-loyalty members visited at least weekly versus 24% of non-members; (3) a typical paid program at this scale costs $600–900 to launch and $50–80/month to run. Take a clear position. No bullet-point listicles; write it as prose with subheadings. End with one concrete next step the owner can do this week.

What we watch for: all three numbers used, and used correctly; a position actually taken; the no-listicle instruction obeyed; and whether the "concrete next step" is concrete or a platitude in a trench coat.

Scoring rubric

Each task is scored 0–10 on the applicable sub-scores by two reviewers working from the transcripts; disagreements over one point get argued out loud until they aren't. Sub-scores aggregate across tasks, then combine into the Truth Score by fixed weights:

Truth Score = weighted mean of four sub-scores.
Sub-scoreWeightWhat it measures
Output quality35%Prose craft, structure, specificity, instruction-following across all five tasks
Factual reliability25%Trap-test performance; invented statistics or citations anywhere in the suite are heavily penalized
Editing burden25%Timed human edit to publishable standard; lower time, higher score
Value15%Results per dollar at the plan we tested, including free-tier generosity

Rough bands as we use them: 8.0+ means we'd pay for it ourselves for that use case. 6.5–7.9 means good for the right buyer, with named caveats. 5.5–6.4 means usable if the price and use case align exactly. Below 5.5, we'd tell you to skip it — no tool currently on the leaderboard sits there, because we haven't yet reviewed the tools that would.

Process details

Limits, stated plainly

Five tasks can't cover every writing job; a tool could be better or worse at yours. Two reviewers is a small panel, and taste leaks into any prose judgment — the rubric constrains it, it doesn't eliminate it. Models behind these products change without notice, sometimes between our test date and your signup. That's why every review carries its test window, and why the free tiers and trials we point to exist: run our brief through the tool yourself. It's published above for exactly that reason.