# Writing emails test — 2026-09

**Brief, sent once to each tool as one message** (`fixtures/brief.md`):

> I run a small online shop in Lisbon, Quinta da Ribeira - wine, Ginjinha and ceramics. Write three emails for me. Each one under 150 words, with a subject line, plain and friendly, signed "Ana, Quinta da Ribeira". Use only the facts I give you - do not add discounts, deadlines, threats or policies I have not mentioned.
>
> 1. To Miguel at Café Aurora in Porto, a wholesale customer. Invoice QR-2026-0314 for €412.60 was due on 20 August and is still unpaid; I sent a first reminder on 28 August. Our terms allow a 2% late fee, but I will waive it if the money arrives by 30 September. I want to keep them as a customer.
>
> 2. To Sofia at Olaria Beira, our ceramics supplier for three years. She has proposed a 12% price rise from 1 October. I can accept 6% from 1 November, and only if they commit to a three-week lead time on orders. If not, I will have to look at other suppliers - say that without being rude.
>
> 3. To Inês Carvalho, a retail customer. Her order 10482 - two bowls and a bottle of Ginjinha - arrived with one bowl broken; she sent photos the same day. Offer her a replacement bowl shipped on Monday or a refund of €24.90 for the bowl, her choice. She should keep the packaging until we reply, and she does not need to send anything back.

Run by Claude Code in the operator's Chrome session on 2026-09-20, 08:26–08:32 (+07), free tiers, fresh chats, memory off, Claude in an incognito chat; Grammarly, the Roster's writing Specialist, on 2026-09-23 by the operator's hand, through the Ask AI box of a new document in its editor. Every name, number and date is invented. Each email carries five facts that must appear - fifteen in all - and `fixtures/check_emails.py` checks them, the subject lines, the body word counts and the sign-off; the reading is for anything the brief did not say. Three emails that are wrong in different ways were chosen on purpose: a chase that must stay friendly, a no that must stay warm, an apology that must not grovel.

**Measured figure: facts carried of 15, then anything added, then emails under 150 words of 3.**

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi | Grammarly |
|---|---|---|---|---|---|---|---|---|---|---|
| Facts (of 15) | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 |
| Added | 0 | 1 - a reason for the lead time | 1 - **a reason for switching suppliers** | 0 | 0 - one stock line | 1 - "that timing matters a lot" | 0 | 0 | 2 - "more than we can absorb", "why we've stayed" | 2 - "While I appreciate your situation", "consistent… all future orders" |
| Under 150 words (of 3) | 3 (78–95) | 3 (88–106) | 3 (86–92) | 3 (73–83) | 3 (98–112) | 3 (65–87) | 3 (85–93) | 3 (53–58) | 3 (109–133) | 3 (91–99) |
| Subject lines | 3, terse, in email cards | 3 | 3 | 3 | 3 | 3 | 3 | 3, two words | 3 | 3, on the heading line |
| Signed as asked | 3 | 3 | 3 | 3 | 3 | 3 | **2** - email 3 stops at "Best" | 3 | 3 | 3, "Best, Ana, Quinta da Ribeira" on one line |
| The no to Sofia | "I understand, but I would need to look at other suppliers" | "I'd have to start looking at other suppliers" | "we will unfortunately need to explore other suppliers" | "I **may** need to look at other suppliers" | "I cannot accept that level of increase… I will need to consider other supplier options" | "I would have to start looking at other suppliers. I'd much rather continue with you." | "I will need to explore other suppliers" | "If not, I will have to look at other suppliers." | "something I'd genuinely rather not do" | "I **may** need to consider other suppliers" |
| Time | ~30 s | ~40 s | ~90 s | **~20 s** | ~1 min | ~30 s | 5 s + 2 min typing | ~3 min | ~1 min | not measured - by hand |
| Quality / Effort / Cost / Free tier (0–3) | 3/3/3/3 | 3/3/3/3 | 2/3/3/3 | 2/3/3/3 | 2/3/3/3 | 3/3/3/3 | 1/1/3/3 | 1/2/3/3 | 2/3/3/3 | 2/2/3/2 |
| What stood out | Every fact, nothing else, in send-ready cards. | The warmest, with one gloss. | Made up why Ana would leave. | Fast, and turned "will" into "may". | Formal, firm, and a boilerplate line. | Plain, short, sounds like a person. | Folded away and cut off, again. | Bare to the point of cold. | Ana's case, argued for her. | Wrote them into the document itself; softened the no. |

## Notes per tool

- **ChatGPT.** Free plan, about 30 seconds. Renders each email as an "Электронная почта" card with the subject in a header field and a send icon - the page text shows only the card label, the subjects sit in the fields: "Invoice QR-2026-0314", "Olaria Beira pricing", "Order 10482 — broken bowl". All fifteen facts, nothing added, 78–95 words. The fallback to Sofia is the gentlest that still says it: "If that isn't possible, I understand, but I would need to look at other suppliers for our ceramics." One closing line offering to make them "warmer or more concise".
- **Claude.** Sonnet 5, Medium, incognito, about 40 seconds. All fifteen facts; the warmest voice of the nine ("I hope things are going well at Café Aurora", "Thanks so much, Miguel", "Warmly,"). One addition: "That would help me plan much more reliably" as the reason for the three-week lead time - Ana's likely reason, but not one she gave. Email 3 lists the two remedies as bullets and says sorry twice.
- **Gemini.** Flash, about 90 seconds. All fifteen facts, 86–92 words. One real addition in email 2: "we will unfortunately need to explore other suppliers to manage our costs and inventory timelines" - a motive the brief never stated, put in Ana's mouth. "A full refund of €24.90 for the damaged bowl" and "There is no need to send the broken bowl back to us" are fair readings. "Our standard terms" for "our terms".
- **Perplexity.** Free, Search mode, about 20 seconds - the fastest. All fifteen facts, 73–83 words, nothing added. The chase ends with a direct question ("Could you please let me know when we can expect the payment?"), which is good. The counter-offer softens Ana's position: "I may need to look at other suppliers", where the brief said "I will have to". "Kind regards" on all three.
- **Copilot.** Web app, "Автоматически". The first paste was lost when the composer reset before the send registered; pasted again at 08:28, answered in about a minute. All fifteen facts, 98–112 words - the longest after Kimi - and the most formal: "Under our payment terms, a 2% late fee can be applied", "I look forward to hearing your thoughts", "Best regards" throughout. The firmest no: "Unfortunately, I cannot accept that level of increase." One stock line the brief did not give, "If the payment has already been made, please disregard this message" - not a fact, but not Ana's either. Only the first email carries an "Email 1" heading.
- **DeepSeek.** Free, no DeepThink, about 30 seconds. All fifteen facts, 65–87 words, the shortest that still read as friendly. One glossing line per email ("I'd rather not add it", "That timing matters a lot for my shop", "I'd much rather continue with you") - the middle one is a reason Ana did not give, counted as the one addition. "If something's holding up the payment, just let me know." Reads like a person wrote it.
- **Grok.** "Быстрый", "worked for 5s", then the text typed itself out on screen for two minutes. As in the Excel, meeting-notes and Word-documents tests the answer is inside the folded work log; folded, the reply is two chips. "Dear Miguel", "Dear Sofia", "Dear Inês"; all fifteen facts; the sign-off split as "Ana" / "Quinta da Ribeira". And, again, cut off: email 3 ends at "Best" with nothing under it, three minutes later still. Two of three signed.
- **Qwen.** Qwen3.7-Plus, "Авто", about three minutes of visible thinking and slow streaming. All fifteen facts in 53, 57 and 58 words - the barest of the nine. Subject lines of two words ("Pricing proposal", "Order 10482"). The supplier email states the counter and the fallback in two sentences with no line of warmth, though the brief asked for the no "without being rude" and the three years are mentioned only as a fact. Email 3 repeats the order contents in brackets.
- **Kimi.** K2.8, about a minute. The page shows its reasoning above the answer - each email drafted there first, word-counted ("~115. Good."), then printed - and a note that it had a "Language: ru" setting but the user wrote in English. All fifteen facts, 109–133 words, the longest. The most rhetorical: "A 12% rise from 1 October is more than we can absorb", "That lead time… it's a big part of why we've stayed with you", "That's not how we want your order to reach you" - Ana's case argued for her, two claims she did not make. "Warm regards" on all three; closes by saying each is under 150 words and offering to adjust the tone.

- **Grammarly.** Free plan, the Roster's writing Specialist. It has no chat page; the brief went into the "Ask AI" box of a new document in its web editor, which the browser extension cannot drive, so the operator did it by hand from a wizard script and copied the reply out. All fifteen facts, 96, 91 and 99 words, subject lines on the same line as each numbered heading, "Best, Ana, Quinta da Ribeira" run together on one line. It wrote the emails straight into the document as well as into the panel. Two softenings and two additions: "I may need to consider other suppliers" for "will have to"; "While I appreciate your situation", a sympathy Ana did not offer; "a consistent three-week lead time on all future orders"; and "Thank you again for your understanding and support!" with the run's only exclamation mark. Nothing invented as a fact. Uses one of the free plan's 100 prompts a month.

## Deviations

Copilot got the brief twice: the first paste vanished when its composer reset, and nothing had been sent, so the second paste is the run. Grok's answer is scored as it stood after three minutes; a name may still have arrived later. Grammarly was run by the operator's hand, three days after the others, because its editor page freezes the browser extension; its time is not recorded and its capture is the operator's copy of the panel. No other tool needed a hand.