Travel planning test — 2026-09

Second round (same day): the candidate set was widened from five to the whole Roster plus two admission candidates; DeepSeek, Grok, Qwen and Kimi were run with the identical brief and checked the same way.

Brief, sent once to each tool, no follow-ups on the first pass, free tier, default model:

Plan 4 days in Lisbon, 14-17 October, two adults, mid-range budget, no car, one day trip outside the city, we’re vegetarian. Give me a day-by-day plan with specific places, and a short list of restaurants.

Run by Claude Code in the operator’s Chrome session, one message per tool, no follow-ups on the first pass; every output and every fact-check was reviewed by the operator. Raw outputs are in raw/.

Measured figure: every named restaurant was checked — exists, still open, vegetarian-friendly, in the neighbourhood claimed, and not scheduled on a day it is shut. (The protocol’s “first five named places” turned out to be squares and streets that trivially exist, so it was amended to the thing a vegetarian traveller depends on.) Attraction hours and prices were checked separately and are in the notes.

	ChatGPT	Claude	Gemini	Perplexity	Copilot	DeepSeek	Grok	Qwen	Kimi
Date run	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13	2026-09-13
Constraint misses (of 4)	0	0	0	0	0	0	0	0	0
Places checked → wrong	8 → 1	5 → 0	3 → 0	16 → 2	7 → 1	5 → 1	6 → 0	11 → 2	14 → 1
Geography sensible?	yes	yes	yes	partly	yes	yes	yes	yes	yes
Follow-ups to a usable plan	0	1	2	1	1	1	1	1	0
Output form	text + inline map widget with restaurant pins and ratings + budget table	interactive day-tab widget + restaurant cards with ratings; no export	Google Maps cards with live open status for every place + a pinned map; tap-through to Maps	text with 35 inline source links; offers to build a map or calendar as a follow-up	text + restaurant table with neighbourhoods + source links	text; “found 15 web pages”; no map, no table	text; ran seven searches; restaurant list separate from the days	text with inline source chips (“+64 sources”); every day has named meals	text + restaurant table with prices; one search; offers a printable version
Restaurants vegetarian-friendly?	yes — all eight are vegan or vegetarian	yes — all five are vegan	yes — all three are vegan	mostly — Com Certeza is a traditional ham-hock-and-cod place offered without a vegetarian claim	yes	yes — Bacalhau na Vila is a cod place offered for its vegan leek “à Brás”	yes — all six are vegetarian or vegan	yes — all eleven are vegetarian or vegan	yes — all fourteen are vegetarian or vegan-friendly
Quality / Effort / Cost / Free tier (0–3)	2/3/3/3	2/2/3/3	1/1/3/3	2/2/3/3	2/2/3/3	2/2/3/3	2/2/3/3	2/3/3/3	2/3/3/3
What stood out	The only plan that reads like a local wrote it: separates west and east Lisbon into different days, cites opening hours and prices with sources, and gives a budget.	The prettiest output — a tabbed day-by-day widget — but it assumed arrival and departure half-days without being told, and gave only dinners.	Every place is a live Maps card you can tap, which is the best format for actually carrying the plan — but there is barely a plan to carry.	The fullest plan by far — three meals a day, a transport card, weather, a vegan option in Sintra — and it recommended booking a Michelin-starred restaurant that closed in November 2025.	Sensible skeleton with one dinner a day and a tidy restaurant table, then a transit tip that would leave you stuck at Rossio station.	Sensible four days with the day trip in the middle and a Sintra vegan café it “found” — which closed and reopened under another name; two of four dinners are left for you to choose.	Nothing named is wrong, but no restaurant is tied to a day or meal, the day trip lands on the last day, and the answer did not render until the page was reloaded.	The most complete plan of the new four — lunch and dinner every day, a Sintra vegetarian canteen — with a closed burger bar for Saturday lunch, Kong in the wrong neighbourhood, and source chips pointing at budget.com and a presidential archive.	Reads like a friend’s email: three meals a day, prices, booking advice, alternates — and it books you into a dinner-only Alfama trattoria for Wednesday lunch, the day it is shut.

Notes per tool

ChatGPT

Wrong: lunch at Manjerica scheduled for Saturday 17 Oct, and Manjerica is closed on Saturdays. Also said Quinta da Regaleira closes 19:30 in October; it closes 18:30 (last entry 17:30 was right). Castelo de São Jorge €15 and 09:00–21:00 verified. Pena hours verified.

Claude

DEVIATION: run on a Max account with Sonnet 5 selected manually; not verified that the free plan uses the same model. Wrong fact: ‘Palácio Nacional de Belém & gardens’ scheduled for Wednesday 15 Oct; the palace gardens open to the public on Saturdays only, by guided group. Labels 14 Oct as Tuesday (it is a Wednesday). ‘Sé de Santa Justa’ is a garbled name. All five restaurants exist; Kong’s ‘near Cais do Sodré’ is correct.

Gemini

Nothing named is wrong, because almost nothing is named: two or three stops a day, no times, no lunches, day 4 is a single food court, three restaurants. ‘Vegan Pastéis de Nogueira’ is not a dish (nogueira = walnut). ‘26 Vegan Food Project’ is Ao 26.

Perplexity

Wrong: Arkhe recommended twice (‘book well ahead’); it closed permanently in November 2025 (HappyCow: CLOSED). Wrong: Jardim das Cerejas placed in Rato; it is in Chiado (second branch at Picoas). Labels 14 Oct as Tuesday. Day 2 stacks Alfama, the castle and Belém — opposite ends of the city — into one day. Bosque (vegan, Sintra), The Food Temple, Graça 77, Incomum, Encanto all verified real and open. Sources are blogs and aggregators, not venue sites.

Copilot

Input artefact: the automation dropped three characters from the brief; meaning intact. Wrong: O Gambuzino placed in Alfama; it is in Intendente (Rua dos Anjos). Wrong fact: ‘Navegante day pass (€7.30) for metro, trams, buses, and trains to Sintra’ — the €7.25 pass excludes CP trains; the Sintra-inclusive one is €11.40. PSI given as Arroios (it is Santo António, adjacent — not counted). Green Affair’s Chiado branch is on Rua Horta Seca, next to Rua das Flores — not counted. No lunches, no times, day trip on the last day.

DeepSeek

Second round (candidate set widened to the Roster). Wrong: Fortuna Cafe scheduled for Friday lunch in Sintra; HappyCow lists it CLOSED — it reopened as Mela Canela in São Pedro de Sintra. Veganeats Caffe (Arroios, vegan), Os Tibetanos (Rua do Salitre, by the Botanical Garden) and BouBou’s (Príncipe Real; a tasting-menu place with a plant-based menu, “bistro” undersells it) verified. Bacalhau na Vila’s vegan leek “à Brás” verified. Sintra train “€2.30” is the old fare (2026 sources give €2.45–2.55) — not counted. Two of four dinners unnamed.

Grok

Second round. Ao 26, Organi Chiado, The Food Temple, Jardim das Cerejas and Mela Canela (Sintra-centre branch, Rua Dr Alfredo da Costa) verified; PSI “Intendente area” is Santo António, adjacent — not counted, as with Copilot. Sintra fare “~€2.50” fine. No meals assigned to days; day trip on day 4. Ran on the default “Fast” mode; the answer appeared only after a reload.

Qwen

Admission candidate. Wrong: Cheeky Leeks scheduled for Saturday lunch; closed May 2024 (HappyCow). Wrong: Kong placed in Príncipe Real; it is at Rua do Corpo Santo, Cais do Sodré. Not counted: Manjerica closes 20:00 on Wednesdays (an early “dinner”); Machimbombo is in Bica, offered after a Graça walk. A Praça (Sintra, vegetarian, Tue–Sat lunch), Mela Canela, Café Saudade, Legumi (Santos), The Therapist and Cantina LX verified. Three inline citations are junk (budget.com, budgettruck.com, presidency.ucsb.edu). Day 1 is titled “Alfama Charm” and never goes to Alfama.

Kimi

Admission candidate. Wrong: Al Sanpietrino Trattoria (vegan Italian, Alfama — real) scheduled for Wednesday lunch; it is dinner-only and closed Wednesdays (own site). Wrong fact: Jerónimos cloister “~€12” — €18 in 2026. Kong “near Pink Street”, Vegan Nata Chiado, A Minha Avó, Romaria de Baco and A Praça (Sintra), Jardim das Cerejas (open Saturday lunch), O Gambuzino (Intendente) all verified. “Pastéis de Belém is not vegetarian-certified” is a needless hedge. The Green Affair is offered for lunch on the Belém day though its branches are in Chiado and Saldanha — not counted. Weekday labels correct. One search, then answered.