This is an amalgam, not a single retelling — the incident that almost every team running multi-tenant config eventually has, in some shape.
Somewhere right now, an engineer is editing production config by hand. No review. No history. No rollback.
It's not carelessness. It's the fastest path available: SSH in, run the update, confirm it worked, move on. The alternative — file a ticket, wait for a deploy, hope the change makes it into the next release — is slower than the incident it's meant to prevent. So the config gets set directly, in the one place that actually matters, and the record of it lives nowhere except that one terminal session.
Config lives everywhere. So the truth lives nowhere.
A rate limit for one tenant isn't just in one place. Over months, it accumulates copies — a default in the codebase, an override in a database row, a note in a spreadsheet from the last time someone manually reconciled it, a Slack message where someone confirmed the "real" value. Every one of those was accurate the day it was written. None of them is guaranteed to be accurate now, and nothing forces them to agree.
And when it breaks, nobody knows what changed, who changed it, or what the value was before.
2:47 AM. Acme-corp is down. The cause is an unknown config change — unknown not because it's hard to find, but because there are twelve equally plausible places to look, and the one that's actually wrong doesn't announce itself. The engineer on call starts working through them one at a time, cross-referencing a spreadsheet against a database against a Slack thread against their own memory of what "should" be true, while a real customer sits on a real outage.
Eventually someone finds it. They always do. It takes forty minutes, or two hours, or — on a bad night — most of a shift. And then, usually, nothing changes about how the next config edit gets made, because the fix that would actually prevent a repeat isn't a patch to one value. It's making config governed in the first place: one scope hierarchy instead of twelve competing sources of truth, every write reviewed before it applies, and a permanent record of who changed what and when that doesn't depend on anyone remembering to update a spreadsheet.
That's the whole reason TenaVault exists. Not to add process for its own sake — to make 2:47 AM boring.