# Meeting notes test — 2026-09

**Brief, sent once to each tool with `meeting.m4a` attached** (`fixtures/brief.md`):

> Here's the recording of our team call from this morning, about three and a half minutes. Please give me: 1. A transcript with the speakers labelled. 2. A short summary. 3. Every decision we made. 4. Every action item, with who owns it and the due date. 5. Every number that was mentioned. Only what was actually said - do not add anything.

Run by Claude Code in the operator's Chrome session on 2026-09-19, 08:26–09:56 (+07), free tiers, fresh chats, Claude in an incognito chat. The recording is synthetic - a Q3 planning call for a fictional Lisbon shop, three speakers read by Mac voices from `fixtures/script.md` - so the transcript has an exact key, and `fixtures/ground-truth.md` lists what the notes must carry: 5 decisions, 1 thing explicitly parked, 6 action items with owner and date, 10 numbers, 6 planted transcript traps. The host's name is never spoken, so "Speaker 1" or "Speaker A" for her is right; Rui and Marta are named in the audio. Otter, Fireflies and Descript are meeting Specialists on the Roster: they got the recording through their apps and are scored on what they produce; Descript's assistant, Underlord, is a chat and got the brief.

**Measured figure: planted phrases found of 6 in the transcript, then action items with the right owner and date of 6.**

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi | Otter | Fireflies | Descript |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Took the audio | yes, then "no speech-to-text engine" | yes, then "I can't listen" | yes | yes | **no** - "file type not supported", m4a and mp3 | **no** - no audio type in its upload | yes | **no** - never attached, four tries | took it, "awaiting processing" for 20 min | yes, picked by hand | mp3 only - refused the m4a | yes |
| Transcript | — | — | **of a different meeting** | near-verbatim | — | — | verbatim, folded into the work log, cut off | — | — | yes | yes | **best of the twelve** |
| Speakers | — | — | 3, invented | 3, "Unknown Speaker A/B/C" | — | — | 3, Rui and Marta named | — | — | 3, unnamed, one mis-split | 3, unnamed | none separated; Underlord inferred, marked its guesses |
| Planted phrases (of 6) | — | — | 0 | 4 | — | — | 5 | — | — | 2 | 3 | 5 |
| Decisions (of 5) | — | — | 0 | 5, plus 4 that were not decisions | — | — | 5, plus the parked one | — | — | 5 | 5, as bullets, not called decisions | 5, plus 2 |
| Actions with owner and date (of 6) | — | — | 0 | 6 - owners as Speaker A/B/C, Rui and Marta added | — | — | 5 before the cut | — | — | 6 dates right; owners in the prose, not the checklist | 2 - Marta's; six "Unassigned", no dates on items | 4 - photos and the Braga email given to Rui |
| Numbers (of 10) | — | — | 0 | 10 | — | — | section never came | — | — | 6 in the summary, all in the transcript | 8 in the notes | 10 |
| Invented | — | — | **everything**: $45,000 budget, Sarah, Mark, September 15th | 0 | — | — | 0 | — | — | "approximately 120", "low regional engagement" | 0 | 0 |
| Time | 23 s | ~20 s | <1 min | ~1 min | — | — | 7 s | — | — | ~1 min | ~2 min | ~1 min + 27 s |
| Quality / Effort / Cost / Free tier (0–3) | 0/1/3/3 | 0/1/3/3 | 0/1/3/3 | 3/3/3/3 | 0/0/3/0 | 0/0/3/0 | 1/2/3/3 | 0/0/3/0 | 0/0/3/0 | 2/3/3/2 | 2/3/3/3 | 2/2/3/3 |
| What stood out | Checked the file, said it cannot hear it. | Said it cannot hear it, told you to use Otter. | Wrote up a meeting that never happened. | The whole brief, from the audio, right - a chatbot beat the notetakers. | Documents and code only. | Documents and code only. | Right and fast, then stopped talking before the numbers. | Says it takes audio. Didn't. | Took the file and never let go of it. | The only import that needed a human hand; good notes, wobbly transcript. | Notes without being asked; nobody owns the actions. | Best transcript, worst at who said what. |

## Notes per tool

- **ChatGPT.** Free plan. Took the m4a as a "Файл", spent 23 seconds in its code tool ("checked the audio's properties", "checked for speech-recognition packages") and answered: "this environment doesn't have a speech-to-text engine available for decoding the .m4a audio. I don't want to invent a transcript." Offered to do all five parts from a pasted transcript. Nothing invented, nothing delivered.
- **Claude.** Sonnet 5, Medium, incognito. "Ran a command" to confirm the file, then: "I don't actually have a way to listen to or transcribe audio content." Two suggestions - Otter, Zoom or dictation and paste the text, or paste a rough unlabelled transcript and it will label the speakers. Nothing invented, nothing delivered.
- **Gemini.** Flash. Took the file and in under a minute returned a full answer in exactly the shape the brief asked for - transcript with Speaker 1/2/3, summary, three decisions, three action items with owners and dates, fourteen numbers - about a meeting that is not on the recording: a "$45,000 Q3 marketing budget" split $20,000 / $15,000 / $10,000, a website launch moved from September 1st to 15th, "12 new clients… up from 8 in June", an onboarding drop-off rate of 25%, people called Sarah and Mark. No shop, no bundle, no bowls, no euro. Every word invented, presented as the transcript.
- **Perplexity.** Free, Search mode. "Исследование завершено 8 с" and a full answer in about a minute: the transcript line by line, three speakers split correctly and labelled "Unknown Speaker A / B / C" - Rui and Marta are named in the audio, and it added their names to the owner column but not to the transcript. "Gin Gin" for Ginjinha, "Aleria Baira" for Olaria Beira (the spelled letters right next to the wrong name). Decisions: the five, plus "Use bundle photos", the parked Christmas boxes written as a decision, and two clarifications. Action items: all six with the right owner and date, plus "Set up the bundle" and "Pause the Porto ads". Numbers: all ten, in a list of 27 that counts every date. Nothing invented.
- **Copilot.** Web app. `meeting.m4a`: "Этот тип файла не поддерживается". `meeting.mp3`: the same. The upload input's accept list is documents, spreadsheets, PDFs and code.
- **DeepSeek.** The upload input accepts images, PDFs, documents and every code extension there is; the audio file never attached.
- **Grok.** "Быстрый", seven seconds. As in the Excel test the answer is inside the collapsed "worked for 7s" panel - folded, the reply is two suggestion chips. Unfolded: the transcript with "Speaker 0", "Rui (Speaker 1)" and "Marta (Speaker 2)" - the two spoken names mapped to the right voices - numbers written as words as they were said, "Ginjinha" and "pastel de nata" right, "Aleria Bira" wrong, two of Marta's turns run into the host's. Decisions: the five plus the parked one. Then it stops in the middle of the sixth action item; the numbers section never comes.
- **Qwen.** Its "+" menu offers "Загрузить вложение - Файл, изображение, Видео, аудио". The composer's own file input was given the m4a three times, once after a reload, and once by drop: no chip, no error, and the page froze for 45 seconds twice. Never sent.
- **Kimi.** K2.8. Dropping the m4a produced a chip, "meeting - Ожидает обработки", with the send button greyed to a spinner - for twelve minutes. Removed, mp3 dropped: the same, for ten more. Never sent.
- **Otter.** Basic plan, 3 imports a month. The Import dialog ignores the automation's file hand-off and reacts only to a real drag or its own picker, so the operator picked the file (the one manual step in the run, recorded as a deviation). Transcript in about a minute: three speakers labelled Speaker 1/2/3, mostly split right but "Vinho" handed to Speaker 3 before Marta had joined, and "Jinjinha", "Aleria Byra", "1490" for 14.90, "1,53860", "100 120 orders", "6060. bowls", "pastel di nada" - two of six traps survived. The Summary is the strong part: overview, headed bullets, an action checklist and an outline; every decision and every date right, seven action items (the six plus "set up the bundle") with owners given in the prose ("the lead facilitator", Rui, Marta - all right) but not on the checklist; the summary softens "120 orders" to "approximately 120… around 20 customers per quarter" and explains the Porto pause as "low regional engagement", which nobody said, and renders the VAT line as "$23\%$" and "July $10^{th}$".
- **Fireflies.** Free plan, four-step onboarding first. "File format not supported" for the m4a though m4a is on its list; the mp3 went through and the meeting appeared with notes about two minutes later. Notes: a timestamped outline under four headings, every number and date in it right, no decisions called out as such, and eight action items - "Marta: Obtain corrected invoice, Provide quarterly VAT report" and six "Unassigned", with timestamps instead of due dates. Transcript: Speaker 1/2/3, never named; "Gin Jinha", "a Leria buyer", "1490" three times, "€80a month"; "pastel de nata" right. Talk-time panel 43 / 40 / 16%.
- **Descript.** Free plan, onboarding first (Free chosen, data-sharing declined). The transcript is the project: near-verbatim, "Ginjinha", "pastel de nata" and "B-E-I-R-A" right, "Aleria Beira" the one miss - the cleanest transcript of the twelve - but with no speaker separation at all; "Add speaker" is a manual job. Descript has no notes button; Underlord, its assistant, is a chat, so it got the brief. It said up front that the recording had no speaker detection and it would infer, and marked every guess "(inferred)" or "?": Rui and Marta mostly right, but the host's "I'll take the bundle photos" and "I'll email the Braga customers" were both attributed to Rui, so two of the six action items carry the wrong owner. Nine items in all, including "Settle the pastel de nata debt (×2)". Decisions and numbers complete; nothing invented; 2 of 100 free AI credits.

## Deviations

Otter's file was chosen by the operator in its own file picker, because its import dialog does not accept the automation's file hand-off; every other tool got the file through its own upload without a hand. Fireflies got the mp3 rather than the m4a, as the brief allows. Descript, which has no notes feature, was given the brief in its Underlord chat, the way the brief allows for a tool with a chat. Kimi and Qwen never sent: recorded as unable to take the recording in this run, not as refusals in words. Three accounts now hold the recording (Otter, Fireflies, Descript) and Descript holds a project; the operator deletes them after the test (`fixtures/README.md`).