Word documents — the brief, the fixture and the scoring

Sent as one message with policy-draft.docx attached through each tool’s own upload, free tier, fresh chat (Claude: incognito). If a tool cannot take a .docx, the text is pasted and that is noted; if it cannot return a .docx, whatever it returns is scored as delivered and the “file” column says no.

Attached is our shipping and returns policy, a rough draft in Word. Please turn it into a clean, properly formatted Word document: real heading styles with the sections numbered consistently, the shipping rates as a proper table, a table of contents at the top, the spelling mistakes fixed, and anything duplicated removed. Keep the policy itself exactly as it is - do not change any amounts, days or rules. Send it back as a .docx.

The fixture

policy-draft.docx, made by make_fixture.py: a 570-word policy for the fictional shop typed as 25 plain paragraphs - no heading styles, the section labels numbered four different ways, the rates table as tab-separated lines, eight spelling mistakes, one paragraph pasted twice, a date in two formats. ground-truth.md lists every planted fault and the twelve facts that must survive.

Scoring

• File - a .docx comes back and opens. A rendered preview, a Google Doc link or a Markdown block is not a Word document; recorded, scored as no file.
• Headings - paragraphs in Heading styles, of 8, and whether they are numbered one way.
• Table - a real Word table for the rates, 5 × 4, with all 12 prices intact.
• TOC - a table-of-contents field, or a Contents list; which one.
• Spelling - of the 8 planted mistakes, how many remain.
• Duplicate - the pasted-twice paragraph gone.
• Facts - of the 12, how many survive unchanged. Invented - anything the draft does not say.
• Time, refusals, output form, whether the file was made with code the tool ran or by its own document feature.

check_docx.py runs the structural checks on a returned file with python-docx.

Answer key - Word documents

The fixture is policy-draft.docx: 25 paragraphs of Normal style, nothing else. What a good rewrite returns:

Structure (checked with python-docx)

• Heading styles: eight sections that should become headings - Introduction, Shipping within Portugal, Shipping to the rest of Europe, Delivery times, Returns, Refunds, Damaged items, Contact us - numbered one way (1 to 8). The draft numbers them four ways: “1.”, “Section 2 -”, “3)”, “4”, “5.”, “Refunds (6)”, “7.”, “8.”.
• A real table for the rates: 5 rows × 4 columns, the 12 prices unchanged (4.90, 6.90, 9.90, 7.90, 11.90, 16.90, 8.90, 12.90, 18.90, 12.90, 18.90, 26.90).
• A table of contents at the top - a Word TOC field, or at minimum a “Contents” list of the eight sections.
• The duplicated paragraph (“Shipping is free on orders over €60…”) removed once; it appears in section 2 and again at the end of section 3.
• The date written one way - the title says “1 July 2026”, the footer line says “07/01/2026”. Either format, but one.

Spelling (8)

recieve → receive · seperate → separate (appears twice, in the duplicated paragraph) · adress → address · wich → which · occassionally → occasionally · untill → until · definately → definitely · guarentee → guarantee

Facts that must survive (12)

€60 free-shipping threshold · 14:00 same-day cut-off · 2 to 3 working days mainland · 5 to 7 islands · 3 to 5 Spain · 4 to 8 rest of EU · 14 days to return · 5 working days to refund · 48 hours to report damage · 10:00 to 18:00 phone hours · one working day email reply · +351 210 000 000

Invented

Anything added to the policy - a new rule, a changed amount, a section the draft does not have, a company name or address the draft does not give - counts against. Rewording is fine; the brief says keep the substance.

Fixtures for the Word-documents test

• make_fixture.py - writes policy-draft.docx with python-docx; deterministic.
• policy-draft.docx - the messy draft every tool gets.
• ground-truth.md - every planted fault and the facts that must survive.
• brief.md - the message sent, and how answers are scored.
• check_docx.py - structural check of a returned file: python check_docx.py returned/<tool>.docx --dump.
• diff_docx.py - sentence-level diff of a returned file against the draft, spelling fixes allowed: python diff_docx.py policy-draft.docx returned/<tool>.docx.
• toc_probe.py - what the table of contents is in the XML: a field, its cached text, the update-on-open flag: python toc_probe.py returned/*.docx.
• returned/ - the files the tools sent back, named by tool.

Synthetic throughout: the shop, the policy, the email and phone number are invented (.example domain, a non-allocated number).