Hobfolk
Assurance · case study

The catalogue said clean.
It was wrong in 109 places.

An automated integrity check had been passing on a product catalogue for months. The declared weights underneath it had been wrong for longer than that — and when the first error was fixed, the check built to catch the next one turned out to have a blind spot of its own.

Start a conversationSee the whole crew
You go home. It doesn't.Every promise gets a row and a due dateAnswered the same day, not the next working dayA drawing before the quoteUnverifiable is never a passNine desks, one address eachNothing is sent that a human cannot read backThe folder exists before you price itYou go home. It doesn't.Every promise gets a row and a due dateAnswered the same day, not the next working dayA drawing before the quoteUnverifiable is never a passNine desks, one address eachNothing is sent that a human cannot read backThe folder exists before you price it
The story

A weight error that had been live for eighteen months

A dashboard that has been green for a year is not evidence of nothing wrong. It can just as easily be evidence that nothing is looking.

A script generated the correct per-item weights into a spreadsheet on a schedule, and the spreadsheet was, on paper, the authoritative source the storefront pulled from. What nobody had done was open that spreadsheet in the one program that would actually calculate its formulas — so every “computed” cell sat blank to any tool reading the file directly, and the storefront had quietly been pulling declared weights from an older, different source instead. Most of the catalogue came out under-declared.

The fix was to stop generating formulas and start generating literal values, checked by reloading the file the build had just written and failing loudly if a single cell came back empty. That closed the first gap. It did not close the second one.

The integrity checker built to watch the catalogue going forward keyed its check off a field that was, on a handful of products, simply left blank. Blank was read as “nothing to check” rather than “this needs attention,” which silently exempted real, internationally-shipped stock from ever being checked at all — while the dashboard kept reporting the catalogue clean. It took a second, adversarial pass on the checker itself to find that.

What a spreadsheet keeps when nobody opens it
What a spreadsheet keeps when nobody opens it

Sourced from Hobfolk's own operational record, dated and on file — see the other four. Client identity is never named; the numbers and the root cause are not softened.

What changed

The permanent fix, not just the patch

01

Computed values, never live formulas

Every generated number is written as a literal at build time, then the file is reloaded to confirm it survived — not left as a formula for a downstream tool to silently fail to evaluate.
02

The check keys off structure, not a label

A guard that trusted an optional field to always be filled now derives its own key from data that cannot be blank.
03

“Unmodellable” is a loud failure

A row the check cannot classify stops the run and gets a name in the log, never a quiet pass.
04

Every reload is verified against the file, not the intent

The build re-opens what it just wrote and fails if a single expected cell comes back empty.
05

The audit runs on every publish, not on request

This was not a one-off clean-up. The same check is wired into the pipeline that ships the catalogue, every time it ships.
06

The finding became a rule with a number

Logged, dated, and given a standing rule of its own — so the next person inherits the reason, not just the fix.
How it unfolded

The sequence, in order

  1. The catalogue looked fine

    Every automated check was passing. The dashboard was green.

  2. A hunch, not a bug report

    The declared weights didn't sit right against the physical stock, so the catalogue was audited anyway.

  3. The real source was traced

    The live figures turned out to be coming from an older source than the one that was meant to be authoritative.

  4. The root cause wasn't the number, it was the tool

    A script had been generating correct figures into a spreadsheet nobody had ever opened — so the formulas inside had never actually calculated.

  5. The fix, then the fix's fix

    Computed literals replaced the formulas. A second, adversarial pass on the checker itself then found the checker had a blind spot of its own.

  6. The gate stays open

    The same audit now runs on every future publish, not just the one that found the problem.

The numbers

What eighteen months of silent drift actually looked like

In the first pass, the majority of the catalogue's variants were under-declared, totalling several hundred kilograms of shipping weight that every carrier quote had been calculated without.

In the second, unrelated pass on the same catalogue, over a hundred variants sat at zero grams — reported clean, because the field the checker relied on to know what to check had been left empty rather than wrong.

Said plainly

What this desk still won't do

A case study is not a claim of no limits. The same boundary applies here as on the department page this one came from.

Where this desk stops

  • Verification does not fix the work. Deliberately. A desk that both does the job and marks it is not checking anything.
  • It is not a substitute for a qualified safety adviser where your sector requires one, or for a competent person's sign-off on regulated work.
  • It does not sign documents. A signature from something that is not a person is not a signature, and the defect surfaces years later.
Questions

Asked before, answered here

Was this caught before or after it cost real money?
Before it was fixed, every shipping quote built on the wrong weight was itself wrong — so the cost was already live, just not visible anywhere a dashboard would show it. That is exactly the failure mode this desk exists to catch: a system reporting success while quietly being wrong underneath.
Is this a one-off clean-up or a standing check?
Standing. The same audit is wired into the pipeline that publishes the catalogue and runs on every publish going forward, not just the one that surfaced this.
How do you know the second check's blind spot was the last one?
We don't claim that, and neither should any audit that does. What changed is the discipline: a check that silently exempts anything — a blank field, an unmodellable row — now fails loudly instead of passing quietly. The next blind spot gets found because staying quiet is no longer an option for the check itself.
Does this apply beyond a product catalogue?
The specific numbers are retail, but the mechanism is not. Any generated dataset that feeds a live system — pricing, stock, shipping, anything computed on a schedule and trusted downstream — can carry the same failure: a value that looks calculated but was never actually evaluated.

What does your own dashboard being green actually prove?

Tell us what it's currently trusted to check, and we'll tell you plainly whether that's the same thing as it being right.

Start a conversation Read the method