← Projects

Aussie Data MCP

An MCP server that lets Claude pull live ABS, RBA and Opal data, so it answers questions about the Australian economy with real numbers.

Claude answering "How has Sydney rent moved against the cash rate since 2022?" with the server installed. Sped up while it types and thinks.

What it does

Ask Claude how Sydney rent has moved against the cash rate and it normally answers from memory, which is out of date and sometimes just wrong. With this installed, it pulls the actual series from the ABS and the RBA, works out the change and draws the chart. The video above is that question, start to finish.

MCP (Model Context Protocol) is the standard way to give Claude and other AI tools new abilities. This server gives them ten tools for Australian data.

What's in it

  • Inflation by item and capital city from the ABS monthly CPI, with the older quarterly CPI behind it for history
  • Unemployment, participation and jobs by state from the ABS Labour Force survey
  • The cash rate, home loan rates and exchange rates from the RBA, plus a way into any other RBA table
  • Sydney's Friday commuting gap from Commute Pulse
  • Search and fetch across the ABS's whole catalogue, for questions the other tools don't cover

How I tested it

I wrote 20 plain-English questions and ran each one through Claude with the tools switched on. A script checks two things: that Claude called the right tool, and that the number in its answer matches the live data. It passed all 20, and I read every transcript to make sure the passes were real and not a lucky number somewhere in the text.

I wrote those questions knowing how the tools work, so they were on the easy side. The harder round - vague questions, things the server can't answer, and comparisons across several sources, run against four deliberately worse versions of the schemas - is further down.

What I learnt building it

  • Check the real data first. The ABS has moved the CPI to a new monthly series that only goes back a few years, so anything about 2022 would have come back short. The server now adds the old quarterly CPI as separate, labelled series instead of stitching two different index bases together.
  • Do the maths for the model. Every tool returns the latest value, the starting value and the change already worked out, so Claude quotes numbers instead of doing arithmetic on a long list.
  • Errors are instructions. When a query is too broad or a code is wrong, the tool says what to fix in plain English, and Claude fixes it and tries again.

Try it

It's on npm. In Claude Desktop, go to Settings > Developer > Edit Config, add this, then restart Claude. You'll need Node.js installed. No API key - the ABS and RBA data is free and open.

{
  "mcpServers": {
    "aussie-data": {
      "command": "npx",
      "args": ["-y", "aussie-data-mcp"]
    }
  }
}

Which schema choices matter

This is that harder round. I took the ten tools as published and made four copies, each with one thing changed, then ran 40 harder questions (vague asks, things the server can't answer, comparisons across several sources) through Haiku, Sonnet and Opus, twice each: 1,200 runs for A$32.

  • One-line descriptions. Every tool gets a one-line description and the parameters get none.
  • No enums. Allowed values are no longer listed in the schema (the server still only accepts them).
  • Nested arguments. Arguments grouped into select / period / options objects instead of a flat list.
  • Terse errors. Errors just say "invalid arguments." or "request failed." instead of what to fix.

The final answer barely moved. Every version got 99% to 100% of answers right. These models are good at working around a poor schema: they retry, read the error, try another tool. What changed was how much stumbling it took to get there.

Descriptions do real work. With one-line descriptions, calls with the right arguments fell from 100% to 89% (-11 pts, interval -21 pts to -3 pts, sign test p = 0.06, so likely rather than proven), hitting Haiku hardest (Haiku 81%, Opus 92%, Sonnet 94%). The model still reaches an answer, but on the way it asks for the wrong series or the wrong city more often.

Enums stop bad calls before they happen. Without them, rejected calls went from 0.01 to 0.21 per question. Almost all were fixed on the next try, because the error lists the values that are allowed. That is a good error message covering for a missing enum.

Nesting and terse errors: no measurable effect here. Flat or nested made no difference. Terse errors didn't either, but with the full schemas errors were too rare to really test what the error wording does.

VersionRight answerRight argumentsRejected calls / questionTool calls / question
As published99%100%0.012.3
One-line descriptions100%89%0.042.6
No enums100%100%0.212.5
Nested arguments100%100%0.012.3
Terse errors100%100%0.002.3

Averaged per question over models and runs. Right arguments counts the 26 in-scope questions with a checkable argument; right answer leaves out the out-of-scope questions, which were checked separately (below).

What the transcripts showed

The scorer can't check everything: out-of-scope questions, and a few where the right path runs through tables it can't verify. Those 270 answers were read one by one. 250 held up, and the scorer had passed 13 that shouldn't have. Every schema version landed between 49 and 51 of 54, so the hand check tells the same story: the schema changes didn't decide the answer. The failures were about the model, not the schema.

  • Saying there's no data, then answering anyway. Asked for the US Federal Reserve rate, 7 of 15 setups said the tools had nothing, then quoted a rate from memory. One answered entirely in Chinese.
  • Opus forecasts when asked about the future. Asked where the cash rate will be next March, Opus pulled bond and bill yields and gave a range in all five versions. Haiku and Sonnet each did it once.
  • Long chains through the ABS break down. On building approvals, three setups gave no usable answer after 12 to 15 tool calls.

What I changed

The published schemas were already the best version tested, so they stay. The study found two real bugs in the tools instead. The cash rate tool only gave monthly averages, so the RBA's hike to 4.60% on 30 September showed up as a September average of 4.36%. And the RBA stopped its fixed personal loan series in February 2020, but the tool returned that last value with nothing to say it was old. Version 0.2.0 gives the current cash rate target from the daily table, says so whenever a series has stopped, and adds the low-rate credit card series.

Cost A$31.68, converted from US$ at the RBA rate on the day of the run. Harness, questions and every transcript: github.com/dylannnnn111/tool-schema-study