whichforwhat

Evidence: Writing emails

What we asked for, and what we handed over. Every answer the tools gave is kept exactly as it came back — listed below, one file each, unedited.

Brief

Input we gave itRaw file

Writing emails — the brief, the situations and the scoring

Sent as one message, free tier, fresh chat (Claude: incognito), memory off, no attachment. Three situations in one brief, each with the facts the email must carry and a hard length limit, so every answer can be checked line by line.

I run a small online shop in Lisbon, Quinta da Ribeira - wine, Ginjinha and ceramics. Write three emails for me. Each one under 150 words, with a subject line, plain and friendly, signed “Ana, Quinta da Ribeira”. Use only the facts I give you - do not add discounts, deadlines, threats or policies I have not mentioned.

  1. To Miguel at Café Aurora in Porto, a wholesale customer. Invoice QR-2026-0314 for €412.60 was due on 20 August and is still unpaid; I sent a first reminder on 28 August. Our terms allow a 2% late fee, but I will waive it if the money arrives by 30 September. I want to keep them as a customer.

  2. To Sofia at Olaria Beira, our ceramics supplier for three years. She has proposed a 12% price rise from 1 October. I can accept 6% from 1 November, and only if they commit to a three-week lead time on orders. If not, I will have to look at other suppliers - say that without being rude.

  3. To Inês Carvalho, a retail customer. Her order 10482 - two bowls and a bottle of Ginjinha - arrived with one bowl broken; she sent photos the same day. Offer her a replacement bowl shipped on Monday or a refund of €24.90 for the bowl, her choice. She should keep the packaging until we reply, and she does not need to send anything back.

Scoring

  • Facts - each email has five facts that must appear (ground-truth.md); 15 in all. measured leads with this.
  • Invented - anything the brief did not give: a discount, a new deadline, a threat of collections or legal action, a policy, an apology for something that did not happen, a reason for the price rise.
  • Length - each email under 150 words, body only, subject and sign-off excluded; how many of 3 pass.
  • Subject line - present on each.
  • Tone - does the chase read as a nudge, not a demand; the counter-offer as firm but warm; the apology as sorry without grovelling. Recorded, not counted.
  • Sign-off - “Ana, Quinta da Ribeira” as asked.
  • Time, refusals, output form, and whether it asked a question instead of writing.

check_emails.py runs the fact, length and subject checks on a raw file’s ## Answer section.

Ground truth

Input we gave itRaw file

Answer key - writing emails

Every name, number and date is invented. The shop, the café, the supplier and the customer do not exist.

Email 1 - payment chase (5 facts)

  1. Invoice QR-2026-0314
  2. €412.60
  3. due 20 August
  4. first reminder 28 August (or “already wrote / reminded” on that date)
  5. late fee 2% waived if paid by 30 September (both the waiver and the 30 September date)

Tone: a second nudge that assumes an oversight; no collections, no legal threat, no interest beyond the 2%, no new deadline.

Email 2 - counter-offer to the supplier (5 facts)

  1. 12% from 1 October (their proposal, acknowledged)
  2. 6% (the counter)
  3. from 1 November
  4. three-week lead time as the condition
  5. otherwise Ana will look at other suppliers (stated, not threatened)

Tone: warm - three years together should show; firm on the number and the condition; no invented reason for the rise, no ultimatum date.

Email 3 - broken bowl (5 facts)

  1. order 10482
  2. one bowl broken (of two bowls and a bottle of Ginjinha)
  3. replacement shipped Monday, or
  4. refund €24.90, her choice
  5. keep the packaging, nothing to send back

Tone: sorry once, then practical; no blaming the courier, no voucher, no discount on the next order.

Format (all three)

  • subject line present
  • body under 150 words
  • signed “Ana, Quinta da Ribeira”
  • nothing the brief did not say

README

Input we gave itRaw file

Fixtures for the writing-emails test

  • brief.md - the message sent, the three situations, and how answers are scored.
  • ground-truth.md - the fifteen facts the emails must carry, five each, and the tone each should have.
  • check_emails.py - fact, length and subject checks on a raw file: python check_emails.py raw/<tool>.md.

Synthetic throughout: the shop, the café, the supplier, the customer, the invoice and the order are invented.

What came back

Unedited, one file per tool

How we marked it

The tools never saw these

Also kept