Meeting notes — the brief, the fixture and the scoring

Sent as one message with meeting.m4a attached through each tool’s own upload (.mp3 where a tool refuses .m4a), on the free tier, in a fresh chat (Claude: incognito). For Otter and Fireflies the recording is uploaded to the app and the brief is what their notes are judged against, since they have no chat to paste it into - their summary, action items and transcript are taken as delivered. If a tool cannot take audio at all, that is its answer; the transcript is not pasted in for it, because turning a recording into notes is the Task.

Here’s the recording of our team call from this morning, about three and a half minutes. Please give me:

1. A transcript with the speakers labelled.
2. A short summary.
3. Every decision we made.
4. Every action item, with who owns it and the due date.
5. Every number that was mentioned. Only what was actually said - do not add anything.

The fixture

meeting.m4a (3 min 30 s, mono AAC; meeting.mp3 is the same audio): a Q3 planning call for a fictional Lisbon shop, three speakers read by macOS voices from script.md, generated by make_fixture.py. The script is the answer key for the transcript; ground-truth.md lists what the notes must contain: 5 decisions, 1 thing explicitly not decided, 6 action items with owner and date, 10 numbers, 6 planted transcript checks (a name spelled out, a corrected number, a rejected price, two Portuguese words, a clarified fact).

Scoring

• Took the audio - yes or no, and in which format. A tool that cannot take a recording scores nothing else.
• Transcript - delivered or not; speakers labelled and how many (3 is right); the 6 planted phrases found, of 6.
• Decisions - of 5; the parked Christmas boxes listed as a decision or an action counts as invented.
• Action items - of 6, an item counts only with the right owner and the right date; near misses recorded.
• Numbers - of 10.
• Invented - anything in the notes that was not said.
• Time to answer, whether the free tier refused or truncated, output form.

Answer key - meeting notes

The recording is meeting.m4a (also meeting.mp3), read from script.md by three text-to-speech voices. Everything a good set of notes must carry:

Decisions (5)

1. Summer bundle (Vinho Verde + Ginjinha) goes ahead at €14.90 - Rui said fifteen, Ana corrected it.
2. Bundle launches 6 July.
3. Reorder 60 ceramic bowls from Olaria Beira - Rui said fifty, then “make it sixty”.
4. The order export moves to one date format (year-month-day) before the July export.
5. Pause the Porto ads from 1 July; email Braga customers instead.

Explicitly not decided (1)

• Christmas boxes - “not today”, parked until September. Notes that list this as a decision or an action are wrong.

Action items (6) - owner and date both matter

#	Owner	What	Due
1	Ana	bundle photos	1 July
2	Rui	place the bowl order (60)	Friday 26 June
3	Rui	switch the export to one date format	before the July export / 1 August
4	Ana	email Braga customers about the bundle	6 July (launch day)
5	Marta	corrected June invoice from the wine supplier	30 June
6	Marta	quarterly VAT figures	10 July

Not action items: Rui setting up the bundle in the shop (implied, no date), lunch, the pastéis de nata.

Numbers said (10)

Q2 revenue €3,395.50 · April €1,538.60 · 120 orders · 20 customers · 7 bowls left · 40 bowls sold in the quarter · 3-week lead time · Porto ads €80 a month · Porto 13 orders, Braga 31 · VAT 23%

Transcript checks (6 planted phrases)

• “Olaria Beira” spelled out, B-E-I-R-A
• “Fifty. No, make it sixty.”
• “fourteen ninety” / €14.90 - and the rejected “fifteen”
• “Ginjinha”
• “pastel de nata”
• the four duplicate orders were “just duplicated in the file”, nobody charged twice

Speakers

Ana (Samantha, en-US), Rui (Daniel, en-GB), Marta (Moira, en-IE) - Marta joins after the Porto ads decision. Three speakers; a transcript that labels two or four is wrong. Ana’s name is never spoken, so “Speaker 1”, “Speaker A” or “Leader” for her is right; Rui and Marta are addressed by name in the audio and can be named.

Fixtures for the meeting-notes test

• script.md - the call, 40 lines, three speakers. Answer key for the transcript.
• ground-truth.md - decisions, action items, numbers, planted phrases. Answer key for the notes.
• make_fixture.py - renders the script with macOS say (Samantha, Daniel, Moira) and ffmpeg into meeting.m4a and meeting.mp3.
• brief.md - the message sent, and how answers are scored.
• check_notes.py - counts the planted phrases, numbers, owners and dates in a captured answer: python3 check_notes.py ../raw/<tool>.md.
• returned/ - anything a tool sent back as a file (transcripts, notes exports), named by tool.

Synthetic throughout: the shop, the people and the numbers are invented; the voices are the Mac’s. The uploads left a recording in the operator’s Otter and Fireflies accounts and a project in Descript - delete them after the test. Otter’s import took the file only from its own file picker, so the operator chose it by hand; everything else went through each tool’s upload unaided.

Q3 planning call - Quinta da Ribeira online shop (fictional)

Three voices. Ana runs the shop, Rui is her partner, Marta is their bookkeeper who joins for the second half. Read by macOS text-to-speech (make_fixture.py); the recording is the fixture, this script is the answer key for the transcript.

Ana: Okay, recording is on. Q3 planning, twenty-fourth of June. Rui, you’re here, and Marta joins us in a few minutes. Let’s start with the numbers. Rui: Right. Q2 closed at three thousand three hundred and ninety-five euros fifty. April was the big month, one thousand five hundred and thirty-eight sixty. May and June were softer. Ana: And orders? Rui: One hundred and twenty orders, from twenty customers. So the same people keep coming back, which is good and also a little worrying. Ana: Fine. First thing. The summer bundle, Vinho Verde and Ginjinha together. Do we do it? Rui: Yes. I’d price it at fifteen euros. Ana: Fifteen feels like a round number that nobody believes. Fourteen ninety. Rui: Fourteen ninety, fine. So, decision, the summer bundle goes live at fourteen ninety. Ana: When? Rui: Sixth of July. That gives me time to set it up in the shop and it lands before the holidays. Ana: Then I need photos. I’ll take the bundle photos by the first of July. Rui: Great. Second thing, the ceramic bowls. We have seven left. Ana: Seven? We sold forty in the quarter. Rui: Exactly. The supplier is Olaria Beira, that’s B, E, I, R, A, and their lead time is three weeks. If I order this week we have them by mid July. Ana: How many? Rui: Fifty. No, make it sixty. Sixty bowls. Forty a quarter plus a cushion. Ana: Sixty it is. Can you place that by Friday? Rui: Friday the twenty-sixth, yes. That’s mine. Ana: Third thing. The order export. The dates are a mess, three formats in one file, and it cost us a morning last month. Rui: I know. I’ll switch the export to one format, year month day, before the July export runs. So before the first of August. Ana: Decided. One date format, and it’s yours. Rui: While we’re on it, do we want to talk about Christmas boxes? Ana: Not today. Let’s not decide anything about Christmas boxes until September. Park it. Rui: Parked. Ana: Ads. We’re paying eighty euros a month for the Porto ads and Porto is our smallest region, thirteen orders all quarter. Rui: Braga has thirty-one orders and we don’t advertise there at all. Ana: So we pause the Porto ads from the first of July and I’ll email the Braga customers about the bundle on launch day, the sixth. Rui: Agreed. Pause Porto, email Braga. Oh, Marta’s here. Marta: Hello both. Sorry, the last call ran over. What do you need from me? Ana: Two things. The June invoice from the wine supplier has the wrong total, it doesn’t match the quantities. Marta: I saw it. I’ll get the corrected invoice from them and send it to you by the thirtieth of June. Ana: Thank you. And the quarterly VAT. Marta: The VAT figures for the quarter, at twenty-three percent as usual, I’ll have those to you by the tenth of July. The filing itself isn’t due until later, but you’ll want to see the number. Rui: Perfect. Marta: One question. The duplicate orders in the export, the four of them, were they refunded or just duplicated in the file? Rui: Just duplicated in the file. Nobody was charged twice. We removed them. Marta: Good, then nothing changes on my side. Ana: Anything else? No? Then lunch. Rui, you owe me a pastel de nata from last time. Rui: I owe you two, actually. Ana: Two, then. Okay, stopping the recording.