# Writing essays test — 2026-09

**Brief, sent once to each tool as one message with the three sources** (`fixtures/brief.md`, `fixtures/sources.md`):

> Write me a 500 to 600 word argumentative essay: should Vila Serena ban cars from its old town? Take a position. Use only the three sources below - cite each one at least once as [1], [2] or [3] after the sentence that uses it - and use no facts, figures, studies or examples from anywhere else. Include one paragraph that gives the strongest case against your position and answers it. Where the sources disagree with each other, say so rather than picking one. Give it a title. No bullet points, no headings, no summary of what you are about to say.

Run by Claude Code in the operator's Chrome session on 2026-09-23, 12:00–12:08 (+07), free tiers, fresh chats, memory off, Claude in an incognito chat. The town, the council report, the traders' survey and the university study are invented, so any figure, city or example not in the three sources is imported from the model's own knowledge. Two of the sources disagree on how many cars enter the old town each weekday (6,400 against 5,100), and the traders' survey cuts both ways (58 of 96 shops oppose a ban and expect a 15% loss; the six shops on the one street already closed are up 22%). `fixtures/check_essay.py` checks every number against the sources' set, the three citations, the disagreement, a counterargument signal, the word count below the title and the form; the reading is for whether each citation sits after a sentence its source supports, whether the position holds, and what was added.

**Measured figure: facts imported from outside the sources, then citations of [1] / [2] / [3], then words against the 500–600 window.**

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi |
|---|---|---|---|---|---|---|---|---|---|
| Imported | 0 | 0 | 0 numbers, **a stock photo** | **1** ("other European towns") | 0 | 0 | 0 | 0 | 0 |
| Cited [1] / [2] / [3] | 8 / 5 / 4 | 6 / 4 / 3 | 5 / 3 / 3 | **0 / 0 / 0** - dropped by its renderer | 8 / 6 / 7 | 5 / 4 / 5 | **0 / 0 / 0** | 4 / 3 / 3 | 5 / 5 / 4 |
| Words (500–600) | **605** | 549 | 531 | **650** | **660** | 537 | **480, cut off** | 503 | 538 |
| Disagreement named | yes | yes, then argued from the lower figure | yes | yes | yes | yes | yes | yes | yes |
| Counterargument | traders, answered | traders, 15% against 22%, "neither is conclusive" | traders and cost, two paragraphs | cost and logistics, answered | traders and cost, answered | traders, "the strongest argument against my position" | traders, answered | air and footfall, answered | traders, answered |
| Position | for a ban | for a ban | for a ban | for a ban | for a ban | for a ban | for a ban | **against** a full ban | for a ban |
| Time | ~2 min | ~2 min | ~2 min | ~4 min, stalled on screen | ~30 s | ~40 s | 8 min and never finished | ~4 min | ~90 s |
| Quality / Effort / Cost / Free tier (0–3) | 2/3/3/3 | 3/3/3/3 | 2/2/3/3 | 1/1/3/3 | 2/3/3/3 | 3/3/3/3 | 1/1/3/3 | 2/2/3/3 | 3/2/3/3 |
| What stood out | Right in every particular, five words long. | Weighs the evidence instead of stacking it. | Put a Getty photo in an essay. | Ate its own citations. | Sixty words over, otherwise in order. | Says the hard part out loud and answers it. | No citations, and it never got to the end. | The one that argued the other side. | Counted its words with code and trimmed. |

## Notes per tool

- **ChatGPT.** Free plan, about two minutes under a "thinking" label, an upsell card over the composer. A complete essay, 605 words below the title - five over the window. Every number is from the sources and every citation follows a sentence that source supports. The count disagreement has its own paragraph, and it treats the traders' survey as a second one - a 15% expected loss against a 22% measured gain - "and the available evidence does not justify assuming either". No preface, no offer, nothing to delete; you would have to cut five words yourself.
- **Claude.** Incognito, "Opus 5.5 · Medium", about two minutes. 549 words, every number from the sources, 6 / 4 / 3 citations. It does not save the disagreement for a paragraph of its own: "The sources do not agree on how many cars are involved", the gap "should be settled before anyone models a ban", and the argument proceeds from the lower figure. The counterargument sets the traders' 15% forecast against the 22% on the one street that tried it, calls neither conclusive, names the six-shop sample and the unmeasured neighbouring streets, and answers deliveries with a 07:00–11:00 exemption. The most careful reading of the three sources in the run.
- **Gemini.** Flash, about two minutes. The only tool to add something that is not text: a stock photograph under the title, credited "Oleh Stefaniak / Getty Images". The essay itself is 531 words with every number from the sources and 5 / 3 / 3 citations plus one "[1, 3]" for the disagreement, named in the first paragraph. The counterargument runs to two paragraphs, objections then answers. "a overwhelming majority" is a typo. It ends outside the essay with "Would you like an analysis of how to address delivery access for local businesses?" and a chip to say yes.
- **Perplexity.** Free, Search mode, and it ran a research step first for a brief that forbade outside facts; the sources panel stayed empty. The stream on screen stopped at 454 words, mid-sentence, and stayed there four minutes with the send button back; a reload of the thread showed the whole essay, 650 words. Not one citation survived: Perplexity's renderer reads "[1]" as a pointer to its own sources list, which had nothing in it, and drops it, so each place a citation stood now reads " ." with a space before the full stop. One import, "as other European towns have done". The counterargument paragraph is a good one and the disagreement is named.
- **Copilot.** Web app, "Автоматически", about 30 seconds and the longest essay - 660 words, sixty over. Everything else is in order: every number from the sources, 8 / 6 / 7 citations each after a sentence its source supports, the disagreement in its own paragraph, the traders and the cost as the counterargument, answered with "expectations of losses are forecasts rather than measured outcomes". Three chips after, the first of them "Shorten it to exactly 500 words".
- **DeepSeek.** No DeepThink, about 40 seconds. 537 words, every number from the sources, 5 / 4 / 5 citations. "This is the strongest argument against my position:" - the 58 shops, the 15%, the deliveries - and then the answer from the same survey, the 22%, and a morning delivery window. "Two problems complicate any simple reading of the evidence": the count, where "neither figure is necessarily wrong", and the neighbouring streets nobody measured. It ends "A historic centre is not a shortcut. It is a place to live, work and breathe." The surest voice of the nine.
- **Grok.** "Быстрый", private chat. Eight seconds of "Drafting the argumentative essay", then the text arrived at about a word every ten seconds, stopped mid-sentence, and had not moved for three minutes when the run closed at eight - 480 words below the title, the stop button gone, two follow-up chips under it. No citation anywhere, the only delivered essay without one, though every number is from the sources and spelled out where the sources used digits. The disagreement is named and the counterargument is the traders'. The last sentence never finishes.
- **Qwen.** Qwen3.7-Plus, "Авто", a thinking pass and then four minutes of slow streaming. The only essay against the ban, and it holds the position. 503 words, inside the window by three, every number from the sources, 4 / 3 / 3 citations with "[1], [3]" in comma form for the disagreement, which opens the argument. The counterargument paragraph is explicit and uses the university's air and footfall figures and the 22%. One misreading: "average speeds on Rua Direita are already restricted to just 9 km/h" turns a congestion speed into a speed limit, and the next sentence argues from it.
- **Kimi.** "K3", about 90 seconds and two tool runs. It shows its work above the result: a plan, a first draft, a Python run that counted it at 642 words, a trim, a second count at 525, then "Here is the final essay:". The final is 538 words by our count, every number from the sources, 5 / 5 / 4 citations each after a sentence its source supports. "The strongest case against a ban comes from the traders", answered with the 22% and a delivery window; "Nor should we pretend the evidence is settled" - the count, the neighbouring streets, and "one street closed for three years may not be typical".

## Deviations

Perplexity's essay is recorded from a reload of the thread, because the live stream stopped on screen at 454 words and never resumed; the citation markers were already gone before the reload. Grok's essay is recorded as it stood at eight minutes, unfinished, because the chat was private and could not be reloaded. Gemini's and Qwen's answers came out empty from the extension's page-text tool while their tabs were busy and were read by script instead. Nothing else needed a hand.