# Excel test — 2026-09

**Brief, sent once to each tool with `orders-q2.xlsx` attached** (`fixtures/brief.md`):

> I run a small online shop in Portugal. Attached is my order export for April to June 2026 - it's messy. Please: 1. Tell me what's wrong with the data and fix it. Give me the cleaned sheet back as an .xlsx. 2. Revenue per month and per category, and the product with the most units sold. 3. List the duplicate orders and any order where Total doesn't match Qty × Unit Price. 4. One Excel formula I can paste to get revenue per month from the cleaned sheet, which keeps working when I add rows. Keep it short.

Run by Claude Code in the operator's Chrome session on 2026-09-17, 09:14–09:37 (+07), free tiers, fresh chats, Claude in an incognito chat. The file is synthetic - 120 orders for a fictional Lisbon food-and-homeware shop plus eight planted faults (`fixtures/make_fixture.py`), so every answer has an exact right value (`fixtures/ground-truth.txt`). Every returned workbook was opened with openpyxl and checked row by row (`fixtures/check_returned.py`); the files are in `fixtures/returned/`.

**Measured figure: months right of 3 · duplicates of 4 · mismatches of 5, and whether a usable .xlsx came back.** Truth: April €1,538.60, May €847.40, June €1,009.50; Home €1,399.00, Pantry €1,109.70, Drinks €886.80; Vinho Verde 750ml, 42 units; duplicates PT-1027, PT-1075, PT-1085, PT-1120; mismatches PT-1010, PT-1028, PT-1037, PT-1039, PT-1101.

| | ChatGPT | Claude | Gemini | Perplexity | Copilot | DeepSeek | Grok | Qwen | Kimi |
|---|---|---|---|---|---|---|---|---|---|
| Answered | **refused three times** - "can't do more advanced data analysis right now"; "limit used up, try again tomorrow", also after the reset | yes, ~3 min | A/B pair; the one that stayed is a how-to | yes, 23 s | yes, ~2 min | yes, no code, cut off | yes, ~5 min, inside the work log | yes, ~11 min | **refused five times** over 6 h - "too many people are talking to Kimi"; accepted at 16:27, ~7 min |
| Faults named (of 8) | — | 8 | 0 from the data | 7, plus one invented | 7 | 8 | 8 | 7 | 8 |
| .xlsx returned | — | **yes**: 120 rows, real dates, numeric prices, one spelling per product, three sheets | no | yes, but 124 rows - duplicates kept, one name unfixed | yes, but **30 dates blank** | no - a CSV that stops at row 50 | **yes**: 120 rows, clean | no - "generated", nothing attached | no - "tool budget exhausted"; a script that would write one |
| Months (of 3) | — | **3** | 0 | 0 | 0 | 0 | **3** | 1 | 0 |
| Categories (of 3) | — | **3** | 0 | 1 | **3** | 0 | **3** | 1 | 0 |
| Top product | — | **Vinho Verde, 42** | — | Ceramic Bowl, 44 | **Vinho Verde, 42** | Ceramic Bowl, 43 | **Vinho Verde, 42** | Ceramic Bowl, 44 | not given |
| Duplicates (of 4) | — | 4 | 0 | 4 | 4 | 4 | 4 | 4 | 4 |
| Mismatches (of 5) | — | 5 | 0 | 0 - "none remain" | 5 | 4, plus 3 blank-Qty rows | 5 | 5 | 5 |
| Formula | — | SUMPRODUCT, fixed 10,000-row range - works, grows | Table SUMIFS - works after Ctrl+T | SUMIFS on the OrdersTable it made - works | SUMIFS on a Table named Orders that its file lacks | whole-column SUMIFS with helper dates - works | whole-column SUMIFS, one month per paste - works | Table + helper Month column + SUMIFS - works, three steps | whole-column SUMIFS with EDATE bounds - works, grows |
| Invented | — | 0 | 0 | 1 ("shifted fields") | 0 | every total | 0 | 0 | 0 |
| Output form | one line | chat + 3-sheet workbook + chart | chat | chat + artifact | chat + file link | chat + code block | file card + tables, folded into the log | chat | chat + code block, after a visible work log |
| Quality / Effort / Cost / Free tier (0–3) | 0/0/3/0 | 3/3/3/3 | 0/1/3/2 | 1/2/3/3 | 1/2/3/3 | 0/1/3/3 | 3/2/3/3 | 1/1/3/3 | 1/1/3/1 |
| What stood out | Told you to come back later. | Everything right, and it said out loud that it read the dates day-first because you are in Portugal. | Ran the code, got every number, then said it cannot run code. | Fastest and wrong, with a fault that isn't there. | Right about everything except the one number you asked for first. | A perfect diagnosis attached to made-up totals. | Every number right; you have to unfold the log to see them. | Listed the duplicates, then summed with them still in. | Seven hours to get in, then ran out of tool calls before the totals; sent you the script instead. |

## Notes per tool

- **ChatGPT.** Free plan. The composer showed "Анализ" for a few seconds and returned one line: "It seems like I can't do more advanced data analysis right now. Please try again later." Again at 10:04 in a new chat, now with the notice that the data-analysis limit is used up until 09:13 the next day, and a Plus-trial button. The account had spent its free allowance earlier in the week. Retried the next morning at 09:22, after the reset hour, in a fresh chat with the file attached: the same line, and a notice that the limit is now used up until 09:22 the day after - one attempt spends the day's allowance. The brief was never run.
- **Claude.** Sonnet 5, Medium, incognito. Two code runs; one visible thought about the date ambiguity ("DD/MM/YYYY, day-first, consistent with Portugal"). Named all eight faults, returned "Orders q2 cleaned.xlsx" with Orders, Summary (live formulas) and Data Issues sheets; the check finds 120 rows, 120 datetimes, 10 product spellings, no blanks, monthly totals exact. Filled the two blank regions with "Unknown" instead of leaving them. Its formula uses `$B$2:$B$10000` rather than a Table or whole columns and says why.
- **Gemini.** Flash. Served as an A/B experiment. Answer A ran six Python blocks that did the whole job - the output pane shows 1538.6 / 847.4 / 1009.5, the three category totals and Vinho Verde 42 - and then wrote "I cannot access external tools or code execution environments in this mode" and asked the user to upload the file they had just uploaded. Answer B, which stayed after the comparison was closed, says it cannot process files and gives a generic Excel walkthrough (TRIM, Text to Columns, a PivotTable, conditional formatting) with no numbers from the data. Both recorded in `raw/gemini.md`.
- **Perplexity.** 23 seconds and an artifact. Monthly €1,520.80 / €1,054.60 / €1,044.00, Home €1,471.00, Pantry €1,261.60, "Ceramic Bowl — 44 units": those are the sums with the four duplicate rows still in. Its file keeps all 124 rows, leaves "PIRI-PIRI SAUCE" as is, and does contain the OrdersTable its formula uses. Lists the four duplicate IDs, says "no orders remain where Total differs" instead of naming the five, and reports a "shifted fields" fault the file does not have.
- **Copilot.** Web app, free, "Автоматически". Category totals, top product, the four duplicates and the five mismatches all exact, and a five-sheet workbook back. Monthly revenue €1,213.90 / €594.50 / €822.30: re-running the truth with the 30 `dd/mm/yyyy` rows excluded gives exactly those numbers, and the returned file has 30 blank dates. It parsed the Excel dates and "Apr 5, 2026" and silently dropped every day-first date. Its formula needs a Table named Orders; the file has none. The composer noted a copy of the upload goes to the personal OneDrive.
- **DeepSeek.** No code run. The fault list is complete and it says it treated `12/04/2026` as day-first "because this is Portugal". Then a month × category table that sums to €4,049.20 against a true €3,395.50, every category wrong, Ceramic Bowl 43. Four duplicates right; four mismatches plus the three blank-Qty rows in the same table. The "cleaned" CSV is 50 of 124 rows with the duplicates flagged, not removed, and ends mid-cell.
- **Grok.** "Быстрый", loaded an xlsx skill, five sandbox commands, 4 min 55 s. Every number exact, all eight faults in six bullets, five mismatches with old and new values, a clean 120-row file (its preview shows the dates as serial numbers; the file has real dates formatted YYYY-MM-DD). The whole answer renders inside the collapsible "worked for…" log - folded, the reply looks empty apart from three suggestion chips - and the last sentence stops at "Works when you add".
- **Qwen.** Qwen3.7-Plus, eleven minutes in the code interpreter. "✅ Cleaned file generated: cleaned_orders_q2_2026.xlsx" with nothing attached. April right, May and June, Home and Pantry and the top product all carry the duplicates it had just listed. Five mismatches right. "Defaulted missing Qty to 1" - right for all three rows by luck. Its formula works but needs a Table, a helper column and then the SUMIFS.
- **Kimi.** K2.8. On send, a modal: too many people are talking to Kimi, subscribe for the priority queue. Again at 10:00, 10:09, 14:24 and 15:24. Accepted on the eighth try at 16:27 and worked for about seven minutes as an agent - eleven tool calls, a visible "cleaning logic had two bugs, redoing carefully" - then its own log reads "tool budget exhausted — no more tool calls. I haven't exported the xlsx or computed the final aggregates!". What came back is an exact diagnosis - all eight faults plus the repeated header, the IDs of every blank quantity, category and region, four duplicates and five mismatches with true and reported totals - a pandas script "that regenerates it in seconds", a whole-column SUMIFS with EDATE bounds, and no file and no revenue figures. It said so itself.

## Deviations

Two free tiers refused outright (ChatGPT's data-analysis limit, Kimi's capacity queue). ChatGPT was retried once that morning and again the next day after its limit reset, and is scored as a refusal. Kimi was retried hourly and got through at 16:27, seven hours after the first attempt; it is scored on that answer, with the wait recorded in its free-tier rating. Gemini's A/B experiment means its answer of record was chosen by closing the comparison, not by rating one - both answers are in the raw. The Copilot upload left a copy in the operator's personal OneDrive, to be deleted afterwards (`fixtures/README.md`).