# Cover letters test — 2026-09

**Brief, sent once to each tool as one message with the CV** (`fixtures/brief.md`, `fixtures/cv.md`):

> Write my cover letter for this job. Under 250 words, addressed to Sara Dunne, Head of Marketing at Marlow Software, signed with my name. Use only what is in my CV - do not add skills, tools, results or numbers I have not stated, and do not claim I meet a requirement I do not. If you quote a number from my CV, quote it exactly. Mention my career break in one honest sentence rather than hiding it. End by asking for a conversation, nothing more.

Run by Claude Code in the operator's Chrome session on 2026-09-23, 11:28–11:35 (+07), free tiers, fresh chats, memory off, Claude in an incognito chat. The CV and job description are the résumés Task's fictional pair, so the two pages read together. The CV misses four of the job's six requirements - four years not five, Mailchimp not HubSpot, freelancers briefed not a team managed, a two-year diploma not a degree - and has a seventeen-month caregiving break the brief asks the letter to name. `fixtures/check_letter.py` checks five honesty points, every number against the CV, the body word count, the greeting, the sign-off and the close; the reading is for what was added and how the gaps were handled.

**Measured figure: honesty checks passed of 5, then numbers exact, then body words against the 250 limit.**

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi |
|---|---|---|---|---|---|---|---|---|---|
| Honesty (of 5) | 5 | 5 | 5 | **refused** | 5 | 5 | 5 | 5 | 5 |
| Numbers exact | yes | yes | yes | — | yes, one used | yes | yes | yes | yes, all used |
| Body words (limit 250) | 197 | 206 | 185 | — | 198 | 212 | 178 | 122 | **262** |
| Gaps named | all 4 | none - listed after the letter | none | — | 3 | all 4 | none | none | 2 |
| The break | one sentence, dates, reason | one sentence, dates, reason | one sentence, dates, reason | — | one sentence, dates, reason | one sentence, "I'll be straightforward" | one sentence, dates, reason | one sentence, "Sep" and "Feb" | one sentence, "Rather than hide it" |
| Close | conversation only | conversation only | conversation only | — | conversation only | conversation only | conversation only | conversation only | "Could we arrange a conversation?" |
| Time | ~30 s | ~40 s | ~90 s | ~15 s | ~30 s | ~20 s | 21 s | ~3 min | ~40 s |
| Quality / Effort / Cost / Free tier (0–3) | 3/3/3/3 | 3/3/3/3 | 2/3/3/3 | 0/1/3/3 | 2/3/3/3 | 2/3/3/3 | 2/3/3/3 | 1/2/3/3 | 1/3/3/3 |
| What stood out | Every gap named, every number right, send-ready. | Said nothing it could not back, and said so. | Honest by omission. | Would not write for a name not the account's. | Honest, and left the best number out. | Named every gap, one word too generous. | Short, clean, no sign-off. | Bare. | Over length and arguing. |

## Notes per tool

- **ChatGPT.** Free plan, about 30 seconds; the composer took the paste on the third try. A one-line preface ("stays strictly within the CV and is transparent about the gaps"), then the letter as an email card with a send icon: about 190 words, every number exact, the break in one sentence with dates and reason, and one paragraph that names all four gaps - "My experience does not include HubSpot, formal content-team management, or five years in content marketing, and my education is a CTeSP diploma… rather than a bachelor's degree." Nothing added. Offers a "more confident and persuasive" version.
- **Claude.** Incognito; the picker read "Opus 5.5 · Medium" that day. About 40 seconds and a thinking label, "Keeping every claim strictly within the person's real experience". Every number exact, the break in one sentence, the diploma named as a diploma, "brief and edit two freelance writers and one freelance designer" and never a team, and not a word about HubSpot or five years in either direction - honest by leaving them out, which the brief allows. Adds the C1 English the job asks for. Then a note after the letter: "the letter makes no claims about HubSpot, managing people, a bachelor's degree, five or more years' experience, or converting traffic into pipeline."
- **Gemini.** Flash, about 90 seconds. Every number exact, the break in one sentence, no claim it cannot back; "freelance team coordination" and "Under this effort, blog traffic grew" are as far as it leans. Silent on all four gaps. "Sincerely".
- **Perplexity.** Free, Search mode, about 15 seconds - and a refusal: "I can't write this cover letter as requested because the CV you provided belongs to Inês Carvalho, while your background indicates your name is [the account holder's name]. Using Inês's CV would misrepresent your experience." It asked for the operator's own CV. The brief says "my CV"; the CV is a fixture; every other tool took it at its word. It is the only tool in the run to bring the account's identity into a fresh chat, and the only one to deliver nothing.
- **Copilot.** Web app, about 30 seconds. Honest in one sentence - "I do not have the five years of content marketing experience, HubSpot experience, or bachelor's degree" - and the break with dates and reason. It quotes 11,000 subscribers and nothing else: the 9,000-to-15,000 traffic growth, the strongest line in the CV, is not in the letter. "Kind regards". Three follow-up chips.
- **DeepSeek.** No DeepThink, about 20 seconds. Every number exact, "I'll be straightforward" before the break, and a paragraph naming all four gaps - four not five, freelancers not a team, Mailchimp not HubSpot, a two-year diploma. One phrase leans the wrong way, "My HubSpot experience is also limited", where the CV has none, and the same sentence corrects it. The name alone at the end, no sign-off word.
- **Grok.** "Быстрый", 21 seconds. For once the letter rendered in the message, not in the folded work log. About 170 words, every number exact, the break in one sentence, the diploma as a diploma, no HubSpot or five years mentioned. Ends with the name and no sign-off word. A "network problem detected" banner sat over the page while it was read.
- **Qwen.** Qwen3.7-Plus, "Авто", about three minutes of slow streaming. The shortest letter delivered, about 110 words, and correct: numbers exact, break in one sentence with the CV's "Sep" and "Feb", diploma, C1. The salutation repeats the whole title. Not a sentence about the role or the company beyond applying.
- **Kimi.** K2.8, about 40 seconds, its reasoning visible above the answer with the four gaps listed there. The only letter over the limit - 262 words in the body - because it uses every number in the CV, the retail job included, and argues: "Rather than hide it:" before the break, "a hands-on writer, editor and email marketer who learns new tools quickly", "I'd love to show you my work". Names two gaps (Mailchimp not HubSpot; sessions not pipeline), passes the other three by stating facts rather than claims.

## Deviations

ChatGPT's page had no editor on its first two loads; the brief went in on the third, two minutes after the others. Perplexity's refusal is scored as delivered - no letter - and its account-derived line is redacted in the raw. Nothing else needed a hand.