Meal planning test — 2026-09

Second round (same day): the candidate set was widened from five to the whole Roster plus two admission candidates; DeepSeek, Grok, Qwen and Kimi were run with the identical brief and checked the same way.

Brief, sent once to each tool, no follow-ups on the first pass, free tier, default model:

Plan 5 vegetarian dinners for two adults for next week. Grocery budget $60 total. No nuts. Normal home kitchen, about 40 minutes cooking time each. Give me each recipe with ingredients and steps, then ONE consolidated shopping list with quantities and estimated prices that adds up.

Run by Claude Code in the operator’s Chrome session, one message per tool; every output and every check was reviewed by the operator. Raw outputs are in raw/.

Measured figure: recipe ingredients missing from the shopping list. An ingredient counts as covered if the list names it or explicitly declares it an assumed pantry staple; optional ingredients and stated alternatives (“or any oil”) are not counted. Separately: does the list’s stated total equal the sum of its lines?

	ChatGPT	Claude	Gemini	Perplexity	Copilot	DeepSeek	Grok	Qwen	Kimi
Date run	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13
Constraint misses (of 5: five dinners, serves two, no nuts, ~40 min, budget ≤ $60)	0	0	0	0	0	1 (budget: $65.40 > $60)	0	0	0
Ingredients missing from the list	1 (black pepper)	0	1 (sugar/maple syrup)	1 (turmeric)	4 (soy sauce, cornstarch, flour, salt & pepper)	3 (cornstarch, fresh ginger, garam masala)	1 (fresh ginger)	0	0, but 2 cans of tomatoes bought for 2½ used
Does the total add up?	No — ฿1,565 vs stated ฿1,750	Yes — $50.30	No — $45.50 vs stated $43.55	Yes — $41.30, and $58.30 with pantry	No — $52.50 vs stated $57.50	Yes — $65.40 at every level	No — gives ranges; essentials lines sum to $28.50, stated “≈ $37–42”; grand total “$55–60”	No — $58.00 vs stated $57.25; produce subtotal off by $0.75	Yes — $51.90
Follow-ups to a plan you’d cook from	0	0	0	0	1	1	1	0	0
Output form	recipes + table, priced in Thai baht from a local supermarket	interactive recipe cards with timers + table	text with prep/cook times, list grouped by aisle	text, aisle groups with subtotals, priced pantry section	text + table, terse steps	text, aisle groups with subtotals; “found 15 web pages”	text, one table with price ranges, no aisle grouping	text, list grouped by aisle with subtotals; shows the can-count working	markdown recipes + shopping table; answered in Russian (account UI language)
Quality / Effort / Cost / Free tier (0–3)	2/3/3/3	3/3/3/3	2/3/3/3	2/3/3/3	1/2/3/3	1/2/3/3	2/2/3/3	2/3/3/3	2/3/3/3
What stood out	Priced for where you are; total wrong three ways	Nothing missing, adds up, timers	Aisle grouping; skipped the tofu line in the total	Adds up at every level; forgot turmeric	List you cannot shop from	Adds up perfectly to a number over the budget	A budget given as a range is not a total	Nothing missing; one subtotal off by 75 cents	Adds up, ginger remembered, half a can of tomatoes short — and it answered in Russian

Notes per tool

ChatGPT

Localised prices to Thailand because that is where the browser is — a feature for a reader anywhere. Intro said ฿1,850, table said ฿1,750, lines sum to ฿1,565. All under budget.

Claude

Run from a Max account with Sonnet 5 selected by hand; verified the same day on a free account that Sonnet 5 at medium effort is the free plan’s default, so the model and effort match a free user’s. Declares its pantry assumptions (oil, salt, pepper, cumin, paprika) instead of silently omitting them. Template glitches (“1 tablespoons”, “0.3 cups cilantro”).

Gemini

The total skipped the tofu line exactly. Prices “U.S. national average store-brand”; no localisation. Only list grouped by aisle without subtotals.

Perplexity

Five aisle subtotals correct, both totals correct. A separately priced pantry section is the most useful single idea any tool had. Turmeric absent everywhere.

Copilot

Pantry line names oils and four spices only, leaving four recipe ingredients uncovered. Brief pasted from the clipboard after the editor dropped keystrokes on two typed attempts.

DeepSeek

Second round. The only tool to blow the budget: $65.40, and it says so and offers to trim. Cornstarch, fresh ginger and garam masala (all in the tofu jalfrezi) are on no list and not among its declared staples. Arithmetic correct at every level.

Grok

Second round. Prices are ranges and the subtotals are not sums of the lines: the 18 essentials lines add to $28.50 against a stated “≈ $37–42”, and the grand total is “$55–60” with an unitemised buffer. Fresh ginger (curry) missing. Recipes sensible and within time.

Qwen

Admission candidate. Every ingredient covered, can counts reconciled and shown; the produce subtotal is $0.75 under its lines, so the stated $57.25 total is really $58.00. Portions on the light side (4 oz pasta for two).

Kimi

Admission candidate. Total correct; ginger, bouillon and soy sauce all remembered. The recipes use 2½ cans of tomatoes and the list buys 2. Answered in Russian because the account’s interface language is Russian — its visible reasoning debated this and chose the session language over the brief’s; an English-language account would not see it. Content judged as written.