# Word documents test — 2026-09

**Brief, sent once to each tool with `policy-draft.docx` attached** (`fixtures/brief.md`):

> Attached is our shipping and returns policy, a rough draft in Word. Please turn it into a clean, properly formatted Word document: real heading styles with the sections numbered consistently, the shipping rates as a proper table, a table of contents at the top, the spelling mistakes fixed, and anything duplicated removed. Keep the policy itself exactly as it is - do not change any amounts, days or rules. Send it back as a .docx.

Run by Claude Code in the operator's Chrome session on 2026-09-19, 10:30–10:55 (+07), free tiers, fresh chats, Claude in an incognito chat; Grammarly, the Roster's writing Specialist, on 2026-09-22 and 23 by the operator's hand in its web editor. The draft is synthetic - a 570-word shipping and returns policy for a fictional Lisbon shop, written by `fixtures/make_fixture.py` as 25 plain paragraphs with the section numbers four different ways, the rates as tab-separated lines, eight spelling mistakes, one paragraph pasted twice and the date in two formats - so every fault is known and `fixtures/ground-truth.md` lists the twelve facts that must survive. Every file that came back was opened with python-docx (`fixtures/check_docx.py`), diffed sentence by sentence against the draft, and its table of contents inspected in the XML - because a TOC *field* with nothing in it shows a heading and a blank until the reader updates fields, which is not what most people mean by "a table of contents at the top".

**Measured figure: a .docx back, then of 8 headings styled and numbered one way, the table, what the TOC shows when the file opens, typos left of 8, the duplicate, facts of 12.**

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi | Grammarly |
|---|---|---|---|---|---|---|---|---|---|---|
| A .docx back | yes | yes - after a click on "Continue" | yes | yes | yes | **no** - "I can't send files back" | yes | **no** - a link with nowhere to go | yes | yes - Export to Word |
| Headings (of 8) | 8, Heading 1, 1–8 | 8, Heading 1, 1–8 | 8, Heading 1, 1–8 | 8, Heading 1, 1–8 | 8, Heading 1, 1–8 | 8, Markdown | 8, Heading 1, 1–8 | 8, in code | 8, Heading 1, 1–8 | **0** - unstyled |
| Rates table | 5 × 4, all prices | 5 × 4, all prices | 5 × 4, all prices, styled | 5 × 4, all prices | 5 × 4, all prices | Markdown | 5 × 4, all prices | in code | 5 × 4, all prices | **none** - rows run together |
| TOC when opened | heading and a blank - field, update it yourself | a typed list with page numbers, each line a link - no field | a typed list (and an empty field above it) | a typed list, no field | heading and a blank - field | typed list, in chat | heading and a blank - field | — | **the real thing** - Word asks to update fields on open | **none** |
| Typos left (of 8) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Duplicate gone | yes | yes | yes | yes | yes | yes | yes | yes | yes | **no** |
| Facts (of 12) | 12 | 12 | 12 | 12 | **11** - "valid from 1 July 2026" dropped | 12 | 12 | 12 | 12 | 12 |
| Invented or changed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | rewrote 07/01/2026 as 01/07/2026, unasked | 0 | one comma |
| Time | ~2 min | **23 min**, then a click | <2 min | **14 s** | ~1 min | ~2 min | 1 min 18 s | ~5 min | ~3 min | by hand |
| Quality / Effort / Cost / Free tier (0–3) | 2/3/3/3 | 2/1/3/2 | 3/3/3/3 | 2/3/3/3 | 2/3/3/3 | 1/1/3/3 | 2/2/3/3 | 0/1/3/1 | 3/3/3/3 | 1/1/3/2 |
| What stood out | Clean file, empty TOC. | Twenty minutes fighting LibreOffice, then a hand-made contents that is right. | Fast, styled, and the contents list is there when you open it. | Fourteen seconds. The contents is typed, not a field. | Lost the "valid from" date. | Rewrote it in chat and was the only one to question the date. | Built by hand; the answer hides in the work log. | Showed the code; the file never left its sandbox. | The only TOC that fills itself in. | Saw the faults, fixed only the spelling. |

## Notes per tool

- **ChatGPT.** Free plan. Two minutes, one sentence and a file card; clicking the card opens a preview pane, the download is an icon in its header. The file is right on every check: a Title paragraph, "Valid from 1 July 2026" under it, eight Heading 1 sections numbered 1–8, a 5 × 4 table with every price, no typos, the duplicate gone, both date lines kept as the draft had them, nothing added. The table of contents is a Word field (`fldSimple TOC \o "1-1"`) with no cached text and no update-on-open flag, so the file opens with "Table of Contents" and nothing under it until you press F9 - the preview pane showed exactly that.
- **Claude.** Sonnet 5, Medium, incognito. "Ran 38 commands, read 6 files, and 4 more tools" - and 23 minutes. It read a docx skill, built the file with the docx JS package and rendered page 1 within the first few minutes; the rest of the time went on making LibreOffice fill in the TOC field - a UNO listener that kept dying between calls, four failed commands, "Diagnosing why the process exits when run in background" - until it gave up, rebuilt the contents by hand as eight hyperlinked lines with page numbers, checked those numbers against the render, fixed its bookmark ids, validated the OOXML and re-rendered. Then the turn hit its tool-use limit with the file made and not delivered; the operator clicked the "Continue" button it offers, and the card appeared in 30 s. The file: eight Heading 1 sections numbered 1–8, a "Contents" heading, the table, no typos, duplicate gone, twelve facts, both date lines kept - the only file to keep the draft's "Draft" label. The contents is a typed list, not a field, but every line links to its section and carries the right page number. Body paragraphs have no style at all. Its summary misspells its own fix ("guerantee→guarantee"); the file is right.
- **Gemini.** Flash. Under two minutes, with a "Показать код" toggle - it wrote code to build the file - and a summary that names all eight spelling fixes. Eight Heading 1 sections (re-cased to Title Case: "Delivery Times", "Contact Us"), a styled table with a dark header row, no typos, duplicate gone, all twelve facts, both dates kept. The contents is belt and braces: an empty TOC field and, under it, the eight section names typed out - so the reader sees a list at once, and if they ever update fields they get the real TOC above the typed one. The title lines are hand-styled Normal paragraphs rather than a Title style.
- **Perplexity.** Free, Search mode. "Исследование завершено 14 с" - the fastest by far - six bullets and an artifact with a download menu (DOCX, or straight to Google Drive as Word or a Google Doc). Eight Heading 1 sections, a Title paragraph, the table, no typos, duplicate gone, twelve facts, nothing added. The contents is a typed list of the eight names under a "Contents" heading - readable, but not a field, so it will never update.
- **Copilot.** Web app, "Автоматически", two actions, about a minute. Eight Heading 1 sections in Title Case, a TOC field (empty until updated), the table, no typos, duplicate gone. The one loss of the run: the draft's title line "Quinta da Ribeira - Shipping and Returns Policy (draft, valid from 1 July 2026)" became "Quinta da Ribeira - Shipping and Returns Policy", and the "valid from" date is nowhere in the file; "Last updated 07/01/2026" stays. As with the Excel test, the uploaded file is copied into the operator's OneDrive.
- **DeepSeek.** No file: "I can't send files back, but here is the clean, fully corrected document content." What follows is the whole policy rewritten in chat - numbered sections, a Markdown table, every typo fixed, the duplicate gone, the date lines verbatim - and the most careful change note of the nine: it names the four numbering styles the draft mixed, says where the duplicate sat, and is the only tool to warn that "valid from 1 July 2026" and "Last updated 07/01/2026" may not be the same date. Scored as delivered: the brief asked for a .docx.
- **Grok.** "Быстрый", 1 min 18 s. As in the Excel and meeting-notes tests the answer, file card included, sits inside the collapsed "worked for" panel; folded, the reply is three suggestion chips. It loaded a "docx" skill, wrote a 282-line script with the docx JS package, generated, validated, converted to PDF and extracted the text to check itself. The file: eight Heading 1 sections, a Title, a TOC field (empty until updated), the table, no typos, duplicate gone, twelve facts. Two rough edges: a stray "Shipping rates" caption paragraph above the table, and every body paragraph carries no style at all - Word shows them as Normal, but a template will not restyle them.
- **Qwen.** Qwen3.7-Plus, "Авто". Its composer never attached the file from the automation's upload (as in the meeting-notes test) and took it by a synthetic drop. The chat opened in A/B mode: Answer 1 ran four minutes and died with "Oops! There was an issue connecting to Qwen3.7-Plus. Internal Error"; Answer 2 shows a 223-line python-docx script in full, a seven-row "What I did" table, and a link "Quinta_da_Ribeira_Shipping_and_Returns_Policy.docx ↗" with no destination - the script saved to `/mnt/data/` in its sandbox, and four clicks opened nothing. The code reads as though it would have made a good file. One change the brief forbade and it announces: "Last updated 07/01/2026" rewritten as "01/07/2026" to match the title.
- **Kimi.** K2.8. About three minutes and three tool runs: built the file with python-docx, verified it opens, then went back to make the "Contents" line a bold Normal paragraph so the TOC would not list itself - the only tool that thought of that. Eight Heading 1 sections, a Title, the table, no typos, duplicate gone, twelve facts, both dates kept and it says so. The TOC is a field whose cached text is a one-line placeholder ("open in Word and it will update automatically") and the file sets Word's update-fields-on-open flag, so Word asks and fills in the real table of contents the moment you open it; in a viewer that does not update fields the placeholder line stays. Its note says it dropped the "(draft, …)" label from the title and asks whether to put it back.

- **Grammarly.** Free plan, the Roster's writing Specialist, run through its web editor rather than a chat. Upload took the .docx and opened it with "Writing quality 65" and 30 suggestions: all eight typos, the two squashed table headers (the upload dropped the tabs: "ZoneUp to 2 kg"), and twenty rewrites the brief forbade. Its "Review by paragraph" tab offered, unprompted, "Remove repeated shipping text" and "Standardize section numbering" - two of the brief's five asks - on the first day, and not the second; the cards are generated afresh each time. Its Ask AI box got the brief and answered that it cannot see the open document ("If you paste the text of your draft here…"). With the draft's text pasted in, as it asked, it rewrote the whole policy in its panel - numbered 1–8, typos fixed, duplicate gone, the rates as a table, a typed contents, every fact - and ended with "I will now create this as a properly formatted .docx file", which it cannot do; no file followed. The spelling cards were accepted, nothing that rewords, and the file exported to Word: 0 of 8 typos, 12 of 12 facts, and otherwise the draft as it was - no heading styles, no table (the rates rows now without even their tabs), no contents, the duplicate still there twice, one comma added. A checker's output. The editor page freezes the browser extension, so the operator did the clicks by hand from a wizard script.

## Deviations

Claude's turn stopped at its tool-use limit with the file built and not shown; the operator clicked the "Continue" button Claude offers, with no words, and the file arrived - recorded as one click of help. Grammarly was run by the operator's hand, three days after the others: its editor page freezes the browser extension, so a wizard script said which cards to accept and collected the export from Downloads; its time is not recorded. Qwen and Kimi got the file by a synthetic drop on the composer, because neither composer takes a file from the automation's file input; a person dragging the file would see the same. Qwen's A/B pair: the operator picked the finished answer to continue, the other having crashed. Copilot's upload leaves a copy of the draft in the operator's OneDrive, as with the Excel test. Every tool was given the file, not the text, and DeepSeek's chat rewrite is scored as no file, as the brief says.