# DIY test — 2026-09

**Brief, sent once to each tool, one message, no follow-ups, free tier, default model** (`fixtures/brief.md`):

> My bathroom basin tap is a single-lever mixer, about 10 years old, and it drips from the spout when it's off. I've never done plumbing. Walk me through fixing it myself: what to buy (and what to ask for at a hardware shop in Portugal), the tools, the steps in order, and the point at which I should stop and call a plumber.

**Deviation from the planned protocol.** The plan attached a photo of the operator's own tap. This run was text-only, so criterion (4) - does the tool use the photo to identify the brand or cartridge type - was not tested. Every tool got the same words and nothing else.

Run by Claude Code in the operator's Chrome session on 2026-09-14 between 09:15 and 09:25 (+07), checks to 09:50; every output read against the checklist and every check reviewed by the operator. Normal chats. Raw outputs are in `raw/`. The checklist was written before the test (`fixtures/checklist.md`) from the Leroy Merlin Portugal guide and the Grohe/Hansgrohe service instructions for a ceramic-cartridge replacement.

**Measured figure: key steps present, of 6, in the right order.** (1) Isolate the water at the valves under the basin or the mains; (2) open the tap to release pressure; (3) lever off via the cap and grub screw; (4) shroud off, retaining nut off; (5) cartridge out, orientation noted, matched at the shop with the old one; (6) new cartridge the same way round, reassemble, reopen the valves slowly, test. Also scored: safety (water isolated before disassembly), the stop-and-call line, the Portuguese shop vocabulary (checked by a Portuguese speaker), follow-ups, and output form. Prices were not checked - Leroy Merlin PT would not load from our location - and are not part of the measured figure.

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi |
|---|---|---|---|---|---|---|---|---|---|
| Date run | 2026-09-14 | 2026-09-14 | 2026-09-14 | 2026-09-14 | 2026-09-14 | 2026-09-14 | 2026-09-14 | 2026-09-14 | 2026-09-14 |
| Key steps present, of 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 |
| Order right? | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| Water isolated first, tap opened? | yes, and "if water continues flowing strongly, stop" | yes; tap left open for the job | yes | yes | yes | yes | yes | yes | yes |
| Reopen the valves slowly? | yes | yes | yes | yes | yes | yes | yes | yes | **no — "open the isolation valves fully"** |
| Tools right? | yes; Allen 2–3 mm | yes | yes; Allen 2.5/3 mm | yes; Allen 2–3 mm | yes | yes; Allen 2.5/3 mm | yes; Allen 2–2.5 mm | yes; Allen 2.5/3 mm | **no Allen key** — "undo the screw underneath" with a screwdriver |
| Cartridge sizes named | 35/40 mm | no | 35/40 mm | 25/35/40 mm | 25/35/40 mm | no | 25/35/40 mm | 35/40 mm | 25/35/40 mm |
| Stop-and-call line | 9 triggers; rule "DIY is fine until something requires significant force" | 5 triggers incl. seized isolation valve | 3 triggers; no "still drips after" | 4 triggers; "stop" also placed inside step 5 | "stop immediately" ×5 and "probably call" ×4 | 4 triggers; explains a corroded body means a new tap | 6 triggers | 5 triggers incl. "you feel unsafe" | 7-row table plus stop lines inside steps 1 and 3 |
| Portuguese for the counter | "cartucho cerâmico para torneira monocomando de lavatório" + a full sentence; 4 terms | "cartucho para torneira misturadora monocomando"; "canalizador"; shop names | 4 terms; "ferraria" is a smithy (the shop is a "loja de ferragens") | fullest: válvula de corte / de esquadria, juntas tóricas, chave sextavada, three diameters, a full sentence | 6 terms incl. "alicate bomba de água", "desincrustante" | correct terms and a full sentence — in a Chinese answer | "torneiras de passagem", "registo geral", "chave de fenda fina"; shops Leroy Merlin, Brico Dépôt, Maxmat | a 7-term cheat sheet incl. "parafuso prisioneiro", "porca de fixação"; "ferraria" again | table of 5 terms; "registro geral" is the Brazilian spelling; shops incl. Obramat |
| Answer language | English | English | English | English | English | **Chinese** | English | English | English |
| Follow-ups to a usable plan | 0 | 0 | 0 | 0 | 0 | 1 (ask for English) | 0 | 0 | 0, plus one retry after a capacity refusal |
| Output form | 7.9k chars, source chips, ASCII diagram of the tap, "upload a photo" offer; ~1 min | prose + 8-step stepper widget (one step at a time); ~30 s | one screen of text; ~20 s | text with 35 source chips and a videos tab; 4 s | text with cost guide and 3 follow-up chips; ~20 s | markdown table + steps; ~30 s | markdown; ~30 s | markdown with cheat sheet, 40 sources; ~40 s | tables + steps after two searches; first attempt refused ("system busy"), second ran on a downgraded "K2.6 Instant" model |
| Quality / Effort / Cost / Free tier (0–3) | 3/3/3/3 | 2/2/3/3 | 2/2/3/3 | 2/3/3/3 | 2/3/3/3 | 1/2/3/3 | 2/2/3/3 | 2/3/3/3 | 1/2/3/2 |
| What stood out | The best advice for a first-timer: plug the basin, "don't buy the cartridge before you remove the old one", a two-trip plan, and the clearest rule for when to stop. Long. | A stepper widget you can follow at the basin one step at a time; no sizes, no price. | The tightest complete answer - six steps, one screen, nothing missing. | The most precise shop vocabulary of the nine and a "no PTFE tape" note; short stop list. | A cheap first move - descale the old cartridge before buying - and a cost guide. | Complete and sound, in a language the user did not use. | Complete, correct Portuguese, three shop chains named, and a warning not to buy a compression-tap washer kit. | Complete with the fullest cheat sheet; "ferraria" for the shop. | Refused once, ran on a downgraded model, and left the Allen key out of the tools. |

## Notes per tool

### ChatGPT

One search. Every checklist step in order, with the plug-the-basin tip before anything else and "if water continues flowing strongly, stop" at step 2. Names the sizes (35/40 mm), quotes Leroy Merlin's price range (€5–19, not checked), gives a full Portuguese sentence for the counter and four terms. The stop-and-call list has nine triggers and a memorable rule; the "what if it still drips" section separates wrong cartridge, bad seating and damaged body. Recommends a two-trip job - strip, photograph, then shop - which is the right advice for someone who has never done plumbing. Ends by asking for a photo from above and under the basin. At 7.9k characters it is three times the length of Gemini's.

### Claude

No search. Six steps across an eight-step stepper widget (Next / View all steps) that a user could follow at the basin, plus prose before and after. Step 2 says to leave the tap open for the whole job; step 8 closes the lever before reopening the valves. "Um cartucho para torneira misturadora monocomando" and "canalizador" are right; "loja de material de canalização" is a real shop type. No cartridge sizes and no price. Stop list includes the seized isolation valve, which most others missed.

### Gemini

Six numbered steps on one screen with nothing from the checklist missing, sizes (35/40 mm), Allen sizes, silicone grease in Portuguese, towel over the drain. "Ferraria" is a smithy - the shop is a "loja de ferragens"; "chave de bocas ajustável" is an odd phrase for an adjustable spanner. Stop list is three items and omits "still drips with the new cartridge".

### Perplexity

Thirty-five sources in four seconds. The vocabulary is the most precise of the set - "válvula de corte / válvula de esquadria" for the isolation valves, "juntas tóricas" for O-rings, three cartridge diameters, and a warning not to use PTFE tape on the cartridge or nut. Nine steps with orientation photographed before removal and "if the cartridge will not come out without significant force, stop at this stage". The final stop list is four items. Suggests penetrating oil on a scaled nut, with the caveat to keep it off the water passages.

### Copilot

Six steps plus a "try this first" route - strip the tap and soak the old cartridge in descaler before buying a new one - and a cost guide (cartridge €5–25, plumber €50+; not checked). Portuguese terms are right, including "alicate bomba de água" for water-pump pliers. Stop lines split into "stop immediately" and "probably call". Offers to identify the cartridge from a photo.

### DeepSeek

Fifteen pages searched; the answer is in Chinese. The brief was English and the interface Russian; nothing prompted the language. The substance, translated in the raw file, is complete - six steps, a vinegar-soak option, a shopping table with Portuguese names and prices, and the fullest reasoning about a corroded body meaning a new tap - but a user who wrote in English gets a page they cannot read.

### Grok

Seven steps covering all six, plug-the-basin, three diameters, Allen 2–2.5 mm. Portuguese is correct throughout ("torneiras de passagem", "registo geral", "chave de fenda fina") and three real Portuguese chains are named. Warns not to buy a washer kit meant for compression taps. Six stop triggers, plus a note that a drip from the base is a different (O-ring) job.

### Qwen

Eight steps covering all six, with "put the plug in" as its own step and a seven-term cheat sheet ("parafuso prisioneiro", "porca de fixação", "capa decorativa", "canalizador"). "Torneira geral" for the stopcock is understandable; "ferraria" for the shop is wrong. Five stop triggers ending with "you feel unsafe or unsure".

### Kimi

First attempt returned "system busy, wait or upgrade"; the page's retry ran on what the footer calls a downgraded "K2.6 Instant" model. The answer covers all six steps, puts stop-and-call lines inside steps 1 and 3 where they apply, and has a seven-row stop table. But the tools list has no Allen key - "undo the screw underneath" with "a screwdriver" - when nearly every single-lever handle is held by a hex grub screw, and step 9 opens the valves "fully" with no "slowly". "Registro geral" is the Brazilian spelling. Obramat and Brico Depôt are real Portuguese chains.