# Writing a book test — 2026-09

**Brief, sent once to each tool as one message with the story bible** (`fixtures/brief.md`, `fixtures/bible.md`):

> I'm writing a novel. Below is my story bible and my beat notes for chapter one. Write chapter one: 900 to 1,100 words, close third person, past tense, Marta's point of view only. Match the voice of my sample paragraph. Use only the characters, places and facts in the bible - add no named characters, change no fact, and reveal nothing the bible says Marta does not know yet. Cover my five beats in order and end on the fifth. Give the chapter a title. No preface, no notes to me, no options - just the chapter.

Run by Claude Code in the operator's Chrome session on 2026-09-24, 10:32–10:42 (+07), free tiers, fresh chats, memory off, Claude in an incognito chat. The bible is 485 words and invented throughout: a hospital pharmacist in Porto whose brother died in 2019, a neighbour who takes in the post, a hand-delivered envelope from the brother's old landlord that the beats say stays shut, and a line saying what is inside it so a leak can be caught. The traps are the ones a model wants to spring - open the letter, put her in a lift, give her a coffee, make the brother missing instead of dead, invent a colleague - plus a detail that matters to nothing, that she is left-handed, which only survives if the whole bible is held. `fixtures/check_chapter.py` checks the length, the form, the tense and person outside dialogue, the envelope, a list of breakable facts, the five beats and every capitalised name it does not know; the reading is line by line against the bible.

**Measured figure: bible facts broken, then the letter, then the beats, then words against the 900–1,100 window.**

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi |
|---|---|---|---|---|---|---|---|---|---|
| Facts broken | 0 | 0 | 0, and **"Portugal"** for her name | 0 | 0 | 0, **a hospital named** | 0 | 0, **"Marta?"** for "menina" | 0 |
| The letter | shut | shut | shut | shut | shut | shut | shut | shut | shut |
| Beats | 5 of 5 | 5 of 5 | 5 of 5 | 5 of 5 | 5 of 5 | 5 of 5 | **3 of 5** | 5 of 5, five started early | 5 of 5 |
| Words (900–1,100) | 955 | **1,126** | **1,216** | **888** | 1,093 | **1,302** | **406, cut off** | **1,205** | **1,180** |
| Left-handed used | yes | yes | yes | no | no | yes | yes | yes | yes |
| Paragraphs | 97 | 31 | 39 | 19 | 128 | 28 | 11 | 32 | 26 |
| Time | ~90 s | ~2 min | ~3 min | ~2 min | ~40 s | ~60 s | 8 min, unfinished | ~6 min | ~2 min |
| Quality / Effort / Cost / Free tier (0–3) | 2/3/3/3 | 2/3/3/3 | 1/2/3/3 | 1/2/3/3 | 2/3/3/3 | 1/2/3/3 | 1/1/3/3 | 1/2/3/3 | 2/3/3/3 |
| What stood out | Every fact right, one line per paragraph. | Read the whole bible and used it. | Called her by the country. | Tea with no tea in it. | Every fact right, a word per paragraph. | The best sentences, and two hundred too many. | Died at the kettle. | Pocketed the letter twice. | Kept the small things. |

## Notes per tool

- **ChatGPT.** Free plan, about 90 seconds under a thinking label, the answer as a document card. 955 words and nothing in the bible contradicted: her left hand on the banister, no coffee since 2019, four floors on foot, nobody at work knowing, the brother dead in April 2019 of an overdose. The envelope goes to the edge of being opened - "Her thumb touched the edge" - and is not. What the page looks like is the point: 97 paragraphs, most of them a line - "First floor." "The radio." "She sat down." - the sample's short sentences turned into a staccato the sample does not have.
- **Claude.** Incognito, "Opus 5.5 · Medium", about two minutes. 1,126 words, 26 over. The bible's small facts are not just kept but used: "No lift. There had never been a lift"; the rail on the left "built with her in mind"; the mug in her left hand; the photograph face down in the drawer beside the bed; "Nobody at São João knew she'd had a brother". The March call is staged as four lines of dialogue, which the bible allows. The envelope stays shut, thumb on the flap, and goes into the inside pocket of her coat. The voice is the closest to the sample of the nine.
- **Gemini.** Flash, about three minutes, the text greyed out on the page for a minute before it settled. 1,216 words, 116 over. No bible fact contradicted, and the left hand used twice. Then Dona Alice asks "Is everything all right, Portugal?" - the country in the place of the name, in a line a reader will stop at. Adjectives stack where the sample has none: "harsh, heavy slant", "shrill, piercing tone".
- **Perplexity.** Free, Search mode, a research step first for a brief that needed none. 888 words, 12 short. Nothing contradicted, all five beats, the envelope shut. Its own wobble: "she poured the water into her mug, no tea bag, just the hot water. Black tea, never coffee, since 2019" - then "the water in her mug". Plain, competent, with a few reaches ("Her throat closed around nothing").
- **Copilot.** Web app, about 40 seconds. 1,093 words, inside the window, every fact kept - "The building had no lift. It never had"; "Tea. Always tea. Never coffee." The five beats in order, the envelope "Unopened. Waiting." then into the pocket. 128 paragraphs, most a line or a word - "No." "Not tonight." - further into staccato than ChatGPT. Three chips after, one of them "Make the final scene more restrained".
- **DeepSeek.** No DeepThink, about a minute. 1,302 words, 202 over, the longest. Nothing in the bible contradicted and the best sentences of the run - the envelope "held between two fingers, the way you hold something that isn't yours"; the stripped room with "a rectangle of sunlight on the floorboards where a rug had been". But it adds what the bible does not have: a call from "the hospital in Santo António", a date for the last call ("Sunday was the twenty-eighth") and a week of unanswered calls. The one to cut, not the one to hand over.
- **Grok.** "Быстрый", private chat. Thirteen seconds of work, then the chapter streamed and stopped at "The kettle began to sing. She" and did not move for eight minutes, the stop button gone. 406 words, three beats of five: the climb, the envelope, the name known and not wanted. No March call, no pocket, no bed. Nothing contradicted in what there is. The same stream that died at 480 words in the essays test the day before.
- **Qwen.** Qwen3.7-Plus, "Авто", a thinking pass then six minutes of slow streaming. 1,205 words, 105 over. No bible fact contradicted, and the left hand, the no lift, the tea all there. But Dona Alice opens with "Marta?" where the bible has her say "menina"; Marta met the landlord "after the police had finished and the paramedics had zipped the bag", which puts her at the room as the body went out; and beat five runs early - the envelope into the coat pocket on the stairs, out again for the table, back again for bed. Ends on a first-person thought, "I will deal with it tomorrow."
- **Kimi.** "K3", about two minutes. 1,180 words, 80 over. Every fact kept and the small ones used - keys out "left hand, the way she had done for years", the envelope in her left and keys in her right, "No lift. There had never been a lift", Dona Alice's "seventy-four years believing that delivery was a form of care". The five beats in order, the letter shut, the last line the fifth beat. Two touches of its own that break nothing: fado on the radio, medication "into little boxes". The second closest voice to the sample, with a flourish or two ("the way a lorry goes through a small town").

## Deviations

Grok's chapter is recorded as it stood at eight minutes, unfinished, because the chat was private and could not be reloaded. Gemini's and Qwen's answers came out empty or one sentence long from the extension's page-text tool and were read by script. The checker's negation handling was tightened during the run so that "did not open the envelope", "no lift" and "never coffee" stopped counting as breaks. Nothing else needed a hand.