It was just another day at work, coffee in one hand, playing around with production platform; when I noticed something off. A sneaky bug creating havoc in production.
The textbook approach? Reproduce the issue in a test environment, generate test data, verify the fix, deploy it safely, and then test again.
But let’s be real. Some days, there’s no time for the perfect process. This was one of those days.
🚨 The Problem? A Bug That Already Existed !
A pagination logic issue meant that after 100 resources, the system incorrectly marked findings as PASSED instead of FAILED. Our beloved feature-x process was throwing out false positives. Not a great look.
The issue was already live in production—meaning users were already dealing with incorrect data. So, the logical question was:
What’s the worst that could happen if I push the fix directly to prod?
⚡ The Dilemma: Do We Test? Or Do We Ship?
Ordinarily, for this case, i would have to generate more than 100 test resources in a sandbox, deploy the fix, validate results, and then push it to prod.
But here’s the kicker: That takes time. Time that we didn’t have. And since production was already in a broken state, my mindset was:
“Let’s just deploy the hotfix and observe.”
And so, we did.
🎯 The Aftermath
With the hotfix live, I sat back, refreshed the platform, and waited.
🚀 Results? The fix worked like a charm. No nasty side effects, no extra fires. Just quick action and a resolved issue.
Would this approach always work? Absolutely not. Deploying directly to production can be risky, and you should never make it a habit. But in the right circumstances—where the issue is already live, and time is critical—it can be the fastest way to restore sanity.
📌 Lessons Learned (For the Brave QA Souls Out There)
✅ Understand the risk – Know what you’re fixing and what the worst-case scenario is.
✅ Assess the impact – If production is already broken, a well-tested hotfix might be the best bet.
✅ Trust your judgment – Sometimes, QA isn’t just about testing; it’s about making decisions.
✅ Logs are your best friend – If you deploy a hotfix, monitor it like a hawk. Observability > Blind Testing
✅ Break the rules… wisely – QA is about balance. Knowing when to break protocol is a skill.
🧭 The Test I Should Have Asked First
Looking back, the decision wasn’t really “test or ship”. It was three questions I answered in about ninety seconds without writing them down:
- Is production already wrong? Yes — users were seeing false PASSED results right then. That changes the maths completely. When the current state is broken, “do nothing” is not the safe option, it’s just the invisible one.
- How big is the blast radius if the fix is wrong? A pagination boundary in a read path. Worst case: results stay wrong, the same way they were already wrong. Nothing gets deleted, nothing gets charged, nothing gets emailed to a customer.
- Can I tell within minutes whether it worked? Yes — refresh the platform and look at a dataset past 100 resources.
Three yeses is what made it defensible. Change any one of them and the answer flips.
🛑 When I Would Not Do This
The same reasoning says no far more often than it says yes:
- Anything that writes, deletes, or migrates data. A bad read is recoverable by fixing the read. A bad write is recoverable only if you have a backup and a bad afternoon.
- Anything that touches money or messaging. Billing, payments, notifications — these leave the building. You cannot un-send an email to 40,000 users.
- When production is currently fine. If there’s no active harm, there’s no time pressure, and the argument for skipping verification evaporates.
- When you can’t observe the result. If the effect only shows up in a monthly report, you’re not deploying a hotfix, you’re deploying a hypothesis.
- When you’d be the only one who knows. Which brings me to the part that actually matters.
📣 Ship Fast, Tell People Faster
The risky part of that day wasn’t skipping the sandbox. It was that the decision lived in one person’s head. A hotfix that bypasses the usual process needs, at minimum:
- A second pair of eyes before the deploy — even a two-minute “here’s what I’m about to push and why” in a channel. This is exactly the failure mode behind the case of the missing PR: work that’s correct but invisible.
- A written trail afterwards — what broke, what changed, when it went out. Future-you debugging a related issue in six months will want this.
- A rollback you’ve already identified. Not “we could revert” — the specific commit, and who can push it.
🧪 The Follow-Up That Makes It Legitimate
Skipping the test before the deploy is a loan, not a gift. It gets repaid the same week:
- Write the test you skipped. A pagination boundary bug is exactly the kind of thing that regresses. If nothing encodes “more than 100 resources must still evaluate correctly”, it will come back.
- Ask why nothing caught it. The bug was live and nobody noticed until I happened to look. That’s a monitoring gap as much as a testing one.
- Consider a feature flag for next time. A flagged fix gives you the speed of a direct deploy with an instant off switch — most of the upside, far less of the risk.
When the deploy does go sideways, the pipeline failure itself is usually the first clue. Half of them are the same handful of messages — the CI/CD error reference covers the ones I hit most.
Would you have done the same? (unless you’re a production purist who’s about to yell at me 😆)