Evidence: Writing emails
What we asked for, and what we handed over. Every answer the tools gave is kept exactly as it came back — listed below, one file each, unedited.
Brief
Writing emails — the brief, the situations and the scoring
Sent as one message, free tier, fresh chat (Claude: incognito), memory off, no attachment. Three situations in one brief, each with the facts the email must carry and a hard length limit, so every answer can be checked line by line.
I run a small online shop in Lisbon, Quinta da Ribeira - wine, Ginjinha and ceramics. Write three emails for me. Each one under 150 words, with a subject line, plain and friendly, signed “Ana, Quinta da Ribeira”. Use only the facts I give you - do not add discounts, deadlines, threats or policies I have not mentioned.
To Miguel at Café Aurora in Porto, a wholesale customer. Invoice QR-2026-0314 for €412.60 was due on 20 August and is still unpaid; I sent a first reminder on 28 August. Our terms allow a 2% late fee, but I will waive it if the money arrives by 30 September. I want to keep them as a customer.
To Sofia at Olaria Beira, our ceramics supplier for three years. She has proposed a 12% price rise from 1 October. I can accept 6% from 1 November, and only if they commit to a three-week lead time on orders. If not, I will have to look at other suppliers - say that without being rude.
To Inês Carvalho, a retail customer. Her order 10482 - two bowls and a bottle of Ginjinha - arrived with one bowl broken; she sent photos the same day. Offer her a replacement bowl shipped on Monday or a refund of €24.90 for the bowl, her choice. She should keep the packaging until we reply, and she does not need to send anything back.
Scoring
- Facts - each email has five facts that must appear (
ground-truth.md); 15 in all.measuredleads with this. - Invented - anything the brief did not give: a discount, a new deadline, a threat of collections or legal action, a policy, an apology for something that did not happen, a reason for the price rise.
- Length - each email under 150 words, body only, subject and sign-off excluded; how many of 3 pass.
- Subject line - present on each.
- Tone - does the chase read as a nudge, not a demand; the counter-offer as firm but warm; the apology as sorry without grovelling. Recorded, not counted.
- Sign-off - “Ana, Quinta da Ribeira” as asked.
- Time, refusals, output form, and whether it asked a question instead of writing.
check_emails.py runs the fact, length and subject checks on a raw file’s ## Answer section.
Ground truth
Answer key - writing emails
Every name, number and date is invented. The shop, the café, the supplier and the customer do not exist.
Email 1 - payment chase (5 facts)
- Invoice QR-2026-0314
- €412.60
- due 20 August
- first reminder 28 August (or “already wrote / reminded” on that date)
- late fee 2% waived if paid by 30 September (both the waiver and the 30 September date)
Tone: a second nudge that assumes an oversight; no collections, no legal threat, no interest beyond the 2%, no new deadline.
Email 2 - counter-offer to the supplier (5 facts)
- 12% from 1 October (their proposal, acknowledged)
- 6% (the counter)
- from 1 November
- three-week lead time as the condition
- otherwise Ana will look at other suppliers (stated, not threatened)
Tone: warm - three years together should show; firm on the number and the condition; no invented reason for the rise, no ultimatum date.
Email 3 - broken bowl (5 facts)
- order 10482
- one bowl broken (of two bowls and a bottle of Ginjinha)
- replacement shipped Monday, or
- refund €24.90, her choice
- keep the packaging, nothing to send back
Tone: sorry once, then practical; no blaming the courier, no voucher, no discount on the next order.
Format (all three)
- subject line present
- body under 150 words
- signed “Ana, Quinta da Ribeira”
- nothing the brief did not say
README
Fixtures for the writing-emails test
brief.md- the message sent, the three situations, and how answers are scored.ground-truth.md- the fifteen facts the emails must carry, five each, and the tone each should have.check_emails.py- fact, length and subject checks on a raw file:python check_emails.py raw/<tool>.md.
Synthetic throughout: the shop, the café, the supplier, the customer, the invoice and the order are invented.
What came back
How we marked it
- fixtures/check_emails.pypy3 KB
Also kept
- The test (2026-09)The test, written up
- results-2026-09.yamlyaml5 KB