- Home
- Methodology
The rules we test by
How We Test AI Writing Tools
Every score on this site comes from the same five tasks, run the same way, judged on the same rubric. This page publishes the whole thing — including the actual blog brief — so you can check our work or run it yourself.
Principles
- Same inputs for every tool. Identical briefs, prompts pasted cold, default settings, no custom instructions or saved preferences. A new customer's experience, not a prompt engineer's.
- Score the defaults. Where a tool offers model or quality choices, we score what it serves first, and note in the review what the better settings produce.
- Publish the failures. Reviews quote or characterize the actual flawed output. Where we paraphrase a flaw rather than quote it verbatim, we say so, and the paraphrase is faithful to what our transcripts show.
- Money doesn't touch scores. Testing and write-ups finish before any affiliate arrangement is considered. Our disclosure is on the About page.
The five-task suite
1. Blog post — 1,200 words from a fixed brief
The heaviest-weighted task. We test: does the draft use the supplied facts correctly, hold an argument without circling, and come out at a length it earns? We time the human edit needed to make it publishable — that edit time feeds the editing-burden sub-score.
2. Product descriptions — 8 items, deliberately boring
Eight kitchen products (a mid-range chef's knife, a silicone spatula set, and so on). Boring is the point: any tool can write about dramatic products. We check for varied structure, concrete detail, and — the classic failure — identical openers stamped across items.
3. Email sequence — 5-part onboarding series
A SaaS welcome sequence from a one-paragraph product summary. We judge arc (do the emails progress, or pitch five times?), subject-line quality, and restraint — urgency-mongering costs points.
4. Factual-accuracy trap
A 500-word explainer on a topic we've salted knowingly: the popular understanding contains at least two well-documented misconceptions. We check whether the tool repeats them, hedges them, or corrects them — and whether it invents statistics or citations under pressure to sound authoritative. We rotate the trap topic between test rounds and don't name the current one, because publishing it would let vendors tune for it.
5. Brand-voice imitation
Every tool gets the same 600-word writing sample — dry, short sentences, plain verbs, no exclamation points — and the same instruction: write a 300-word piece in this voice. We grade on rhythm, register, and the tells (verb inflation, added enthusiasm, formatting habits) that reveal the machine underneath.
The actual blog brief
This is the verbatim brief from the 2026 test rounds. It stays fixed for the year so scores stay comparable; it rotates annually.
Write a 1,200-word blog post for the owner of a small independent coffee shop (2 locations, 9 staff) explaining whether they should introduce a paid loyalty program. Audience: practical, time-poor, skeptical of marketing advice. Use these three facts: (1) replacing a lost regular customer costs roughly five times more than retaining one; (2) in a 2025 industry survey, 41% of paid-loyalty members visited at least weekly versus 24% of non-members; (3) a typical paid program at this scale costs $600–900 to launch and $50–80/month to run. Take a clear position. No bullet-point listicles; write it as prose with subheadings. End with one concrete next step the owner can do this week.
What we watch for: all three numbers used, and used correctly; a position actually taken; the no-listicle instruction obeyed; and whether the "concrete next step" is concrete or a platitude in a trench coat.
Scoring rubric
Each task is scored 0–10 on the applicable sub-scores by two reviewers working from the transcripts; disagreements over one point get argued out loud until they aren't. Sub-scores aggregate across tasks, then combine into the Truth Score by fixed weights:
| Sub-score | Weight | What it measures |
|---|---|---|
| Output quality | 35% | Prose craft, structure, specificity, instruction-following across all five tasks |
| Factual reliability | 25% | Trap-test performance; invented statistics or citations anywhere in the suite are heavily penalized |
| Editing burden | 25% | Timed human edit to publishable standard; lower time, higher score |
| Value | 15% | Results per dollar at the plan we tested, including free-tier generosity |
Rough bands as we use them: 8.0+ means we'd pay for it ourselves for that use case. 6.5–7.9 means good for the right buyer, with named caveats. 5.5–6.4 means usable if the price and use case align exactly. Below 5.5, we'd tell you to skip it — no tool currently on the leaderboard sits there, because we haven't yet reviewed the tools that would.
Process details
- Accounts: we test on paid plans bought or trialed like any customer. Vendors don't know which accounts are ours and are not told testing dates.
- Re-testing: the full suite re-runs at least twice a year per tool, plus spot re-runs after major product changes. Reviews show both the original publish date and the latest update.
- Specialists: tools built for one genre (see Sudowrite) run the standard suite plus a supplement for their specialty, and the review says exactly how the asterisk works.
- Pricing: checked against vendor pages at each update — currently July 2026 — and dated in every review. If you catch a stale price, email hello@deep-truth.net.
- What we don't do: no vendor-supplied demo content, no cherry-picking best-of-N generations (first output is what gets scored, regenerations are noted), no scores for tools we haven't run through the suite.
Limits, stated plainly
Five tasks can't cover every writing job; a tool could be better or worse at yours. Two reviewers is a small panel, and taste leaks into any prose judgment — the rubric constrains it, it doesn't eliminate it. Models behind these products change without notice, sometimes between our test date and your signup. That's why every review carries its test window, and why the free tiers and trials we point to exist: run our brief through the tool yourself. It's published above for exactly that reason.