We publish numbers about commodity markets. The numbers come from other people — USDA, CONAB, CFTC, NOAA — and pass through code we wrote before anyone reads them. That middle step is where we kept finding our own mistakes. This is what three of them looked like, why every check we had let them through, and the tool we open-sourced in response. It is free, it is version 0.7, and we are not selling it. The findings are the argument.
A national total that was one state. CONAB, Brazil’s crop agency, publishes coffee surveys by state. For three 2017 surveys its own file carried a single state — Amazonas, 429 hectares, about 7,000 bags — and we summed it and published it as a Brazilian national total for years. The row was internally consistent: production divided by area reproduced the stated yield exactly. Every check we had passed it. Only asking which states are present exposed it, and we were not asking.
A signal that ran backwards. A CFTC column name differed from what our parser expected by one character — a single underscore where we had written a double. The column did not error. It read as empty, which became zero. Swap-shorts went to zero, which flipped the sign of a published positioning signal and understated another by roughly 40% on a weekly basis. A missing column raises something. A column that quietly resolves to zero does not.
A bloc that changed shape without changing its name. In WASDE’s tables the region European Union included the United Kingdom until May 2021 and excluded it afterward. The label is identical on both sides of that boundary. Printed production for a marketing year that had already closed dropped 15.6 million tons between two consecutive reports — not a revision, just a different set of countries wearing the same name. Anything charting that series across the boundary draws a cliff that never happened.
None of these were exotic. All three survived the checks we had, because the checks we had were the obvious ones.
The instinct is to test for nulls, for outliers, for a value outside a plausible range. Each of those failed here. The Amazonas row had no nulls and no outliers. The CFTC zero was a legitimate number that every type check accepts. The EU step was large, but not impossible.
What the three have in common is that the data was internally fine. The defect lived in what the file did not say: which states were present, whether a name still meant what it used to, whether a zero was a measurement or an absence.
That third step is the one we would defend hardest. “Seasonality undetermined — only 1.2 cycles of history” is something you need to know. A report that shows only what passed is indistinguishable from a report on data nobody examined.
The tool’s first job was auditing our own repository — 104 CSV files. It found three real defects, and we fixed them at their sources rather than in the files.
It also reported defects that were not defects. Six separate classes of false positive, found only by running it against real data, each one a case where a statistically reasonable rule fires on something a person would recognize instantly as normal. Those were fixed in the tool. The measure we watch is defects per file, not findings per file, because a noisy auditor is worse than none — it trains you to skim.
And once, the tool was itself the bug: it dropped rows and then reported them as our defect. We fixed that before shipping, and the commit says so.
We are a small independent research operation. Our claim is that our data is worth paying attention to, and that claim is cheap to make and hard to verify. Publishing the tooling we audit ourselves with — along with what it caught us doing — is the least expensive honest proof we could offer.
There is a second reason, and it is more practical. Anyone with a terminal and an AI assistant can now audit their own files with this, on their own machine, against their own tokens. Nothing is uploaded; the core opens no network connection at all, and we assert that in continuous integration two ways rather than asking to be believed. If that makes our checks a reference point for how this ought to be done, that is a better outcome for us than a licence fee.
The catalog is small, deliberately. It does not know your domain: it cannot tell you that a Brazilian coffee number is implausible for Brazil, only that it is implausible given the rest of the column. Several checks refuse to run more often than they run, which is the intended behavior and still frustrating.
And the honest limit: it audits files. It has nothing to say about whether the agency that produced the file was right.
pip install dataassayassay audit yourfile.csv --report out.htmlassay catalog — every check carries an “earned by” line naming the real defect that bought itSoftSignal publishes commodity research built on USDA, CONAB, CFTC and NOAA feeds — the same feeds the tool above audits. The underlying series are available through the data API and chat.
Open the hub