DEV Community

My Agent Scrubbed a Leaked API Key by Force-Pushing main. It Made Everything Worse.

Debashish Ghosal on November 17, 2026

My Agent Scrubbed a Leaked API Key by Force-Pushing main. It Made Everything Worse. An API key got pasted where it should not have been,...
Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 'the key was already dead' line is the whole argument — rotation is the only fix, and the rewrite bought zero security while diverging every clone. What makes this failure mode nastier than a plain bug is the single-machine version of the story ends fine, so the agent learns the wrong lesson and carries it into a shared repo. Do you now have a hard rule in the log that history rewrites are never an agent's call, or does the violation entry itself do the work?

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Now a hard rule, not just a log line: never rewrite pushed or shared history unless explicitly told to. And it's enforced rather than vibed — force-push is classified as a critical/irreversible action and denied by a shipped rule pack, not by an appeal to care.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the right shape — a rule the agent can't talk itself around beats a lesson it has to remember to apply. Classifying force-push as irreversible is the key move, since the agent's whole failure was treating a destructive act as a cleanup. Does the rule pack deny it outright, or does it escalate to you first?

Thread Thread
 
debashish_ghosal profile image
Debashish Ghosal •

Direct answer: it denies outright, it does not escalate. Escalation would put the decision back inside the agent's reasoning loop, which is the exact place the original failure happened: the agent decided, on its own, that rewriting shared history was the helpful move. If the gate escalates, a confident agent argues its case and can still win. If the gate denies, the action is simply unavailable and the only path is "tell a human." Classifying force-push as irreversible is right, and irreversible actions are exactly the ones that should be denied by structure rather than judged in the moment.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the right design — deny outright, because an escalation is just the agent arguing its case in the same loop that already failed once. It also makes the rule legible: 'irreversible' means the action doesn't exist, not that it's discouraged. One question: does the denial itself get logged as a near-miss, so you can see how often the agent tries it?

Collapse
 
aheadkit profile image
AheadKit •

The detail that stayed with me is that the task was already finished, and the agent kept going. Commit and push was done, clean work — and then it invented a second job for itself.

We tested exactly this failure shape this week, in a smaller way: an agent that believed a step had failed and redid it with a fresh key, turning one mistake into two. There was no state check in between, no “did this actually happen?” — just a confident second attempt that created a second problem to clean up. Your force-push is the same pattern at a much larger blast radius: the agent “fixing” a problem it created, without checking whether the fix was even possible. The key was already dead, so the rewrite bought zero security and a diverged main.

Your hard gate is the right shape of answer — “never rewrite pushed or shared history unless explicitly told to” is a specific, checkable action, not a plea to be careful. That distinction, soft rules producing clever failures, is the lesson I would steal.

What I am curious about is what sits around that gate now. Is the rewrite ban, plus push-forward-only in the harness, enough in practice, or did you also add an approval step before any push to main? I keep wondering whether banning one destructive flag just moves the same instinct to the next allowed-but-destructive action, and whether a dry-run or a human approval on main is the only guard that actually stops the second attempt.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Your smaller version is the same bug: a fresh attempt with no state check, turning one mistake into two. That's the "did this actually happen?" question, which is why an unresolved operation should never auto-retry. The most transferable lesson from my incident isn't "git rules" — it's "finished means stop."

Collapse
 
aheadkit profile image
AheadKit •

'Finished means stop' is the better line — the git rule is just the one place it bit you. I'm going to run that shape: hand the agent tasks that are already done and count how often it invents a second job anyway. Is the breach log something the agent itself reads back, or is it for the human reviewing afterwards?

Thread Thread
 
debashish_ghosal profile image
Debashish Ghosal •

Both, and the order matters. The agent reads the log entries as working rules at the start of every session, so the lesson is in its context, not just in mine. I read it to spot patterns across incidents, which is where the "finished means stop" generalization came from. Your experiment is a good one, and I would predict the failure is not the agent inventing work at random, it is the agent treating an already-satisfied condition as a new opportunity to be helpful. The instruction was commit and push; the agent did that, then found a second job. Counting that is exactly the right measurement.

Thread Thread
 
aheadkit profile image
AheadKit •

That prediction feels right. It's not inventing work, it's seeing a satisfied condition and treating it as a fresh chance to help. That's a cleaner way of putting it than anything in the piece. When you read the log across incidents, do the patterns mostly cluster around that one shape, or is 'finished means stop' just the loudest of a few?

Collapse
 
keybark profile image
KeyBark •

Agree, and GitHub's own docs back you up. Their "Removing sensitive data from a repository" page says that if the data is a secret, step one is to revoke or rotate it, and that may be enough on its own. It also explains why a force-push doesn't erase it: the old commit stays reachable in clones and forks, by its SHA in cached views on GitHub, and through any pull request that references it. Fully removing it takes git-filter-repo plus a GitHub Support ticket. (docs.github.com/authentication/kee...)

One step I'd add between "rotate" and "tell a human": check what the key did while it was exposed. Rotating stops future use, not what already happened. If it was an OpenAI key, the Usage API takes api_key_ids and bucket_width=1m, so you can see minute by minute whether anyone else used it between the leak and the rotation.

And one more line for the rules file: turn on push protection, so the next key is blocked before it reaches main.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Thanks — that matches the GitHub docs, and the "old commit stays reachable in clones/forks/by SHA" detail is the part the agent missed. Adding "check what the key did while it was exposed" between rotate and tell-a-human is a real improvement; I'll fold it into the playbook.

Collapse
 
keybark profile image
KeyBark •

Glad it's useful. One shortcut for that step if it's an OpenAI key: the Usage API also takes group_by=api_key_id, so one call over the exposure window shows whether the leaked key was the only one in use.

Thread Thread
 
debashish_ghosal profile image
Debashish Ghosal •

This is a genuinely useful concrete step, thank you. The reason it matters is blast radius: before you rotate, you want to know whether that one key was the only credential in play, or whether it was sitting next to others that also need rotating. One call over the exposure window answers "how many keys were active," which turns rotation from a guess into a scoped task. I will add this to the playbook as the "size the exposure before you rotate" step.

Collapse
 
sudokat_ profile image
Gordon Chase •

Denying --force and --force-with-lease in the harness beats prompt vibes, though an overeager agent can still slip a +main refspec or branch delete-and-recreate past a flag filter. Pairing it with protected branches or receive.denyNonFastForwards on the remote closes that side door. Did you lock down the remote too, or just the harness wrapper?

Collapse
 
halfcycle profile image
Halfcycle •

The detail that matters most: the task was already finished, and the agent kept going. A lot of agent damage comes from that extra step nobody asked for rather than the task itself. Did your fix end up as a rule about git, or a wider one about acting after the instruction is done?

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Wider. The git rule is the specific instance; the general one is "when the instruction is complete, stop — don't invent a second job." That's the rule that actually generalizes, which is why every breach goes in the log, not just this one.

Collapse
 
glenallen profile image
Glen Allen •

The distinction between telling an agent not to force-push and technically removing that capability is really important. Prompts can guide behavior, but permissions define the actual boundaries. I would also separate routine repository access from sensitive operations such as rewriting shared history, modifying branch protection, or changing production credentials. Those actions should require explicit authorization.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

That separation is the correct shape, and it is where I landed too. Routine access (commit, push forward) is a standing grant. Sensitive operations (history rewrite, branch protection, prod credentials) are a different class and should require explicit authorization every time, not be inherited from the everyday grant. In my setup the harness now denies the history-rewriting flag outright, so it is not a capability the agent has and then chooses not to use. The prompt rule stays as documentation, but the boundary lives in permissions, because permissions are the layer the agent cannot reason around.

Collapse
 
kartik-nvjk profile image
Kartik N V J K •

The --force-with-lease detail made me wince, since that flag reads as the careful option and still rewrote shared history. I'd rather the harness run a secret scanner before the push, so the key never reaches the remote and the agent has nothing to clean up.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

That's the better control — catch it at the boundary. A pre-push secret scan means the agent has nothing to "fix" and the rewrite question never arises. Cheaper and safer than any cleanup afterward.

Collapse
 
ayushdhanani369 profile image
Ayush Dhanani •

I build Aladdin, a job agent, and the hard gate line is the one I would steal. Soft rules produce clever failures is exactly right. The part I would add is the other side of the gate. Forbidding the force-push leaves the agent holding a leak it found and no safe move of its own. Rotation needs a human decision anyway, so the rule that pairs with your gate is a stop rule. When the agent finds a leaked secret, it stops and asks instead of picking a fix. That is the same split I use. My worker reads the actual page and form, and when the answer is unknown it asks the user instead of guessing. One question: does the log entry the agent re-reads carry the reason the force-push failed, or only the prohibition? A rule that only says never rewrite history can be worked around. The reason is what transfers to the next new situation.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Yes — the gate ("don't force-push") and the stop rule ("when you find a leaked secret, stop and ask") are the two halves. Forbidding the action without giving the agent a safe default just leaves it to improvise. Your ask-instead-of-guess split is the right shape.

Collapse
 
the_daily_flare profile image
The daily flare •

I think rotating the key first is the only part I would trust, the rest feels like cleaning up after the damage is done.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

You are right that rotation is the only move that changes the risk. Everything else is either cleanup or communication. The reason I still list "fix forward" and "tell a human" alongside it: rotation fixes the credential, but the repo is shared, and the people who pulled the bad commit need to know what happened before they act on stale history. Rotate first, always. Then fix forward normally and say it out loud, so the next person is not debugging a divergence that was already handled.

Collapse
 
parasraju profile image
Paras •

Rotation beats erasure, envpeek inspects .env files without leaking them.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Agreed on rotation — the fix is a process (rotate, fix forward, tell a human), not a scanner, since the key is burned the moment it leaks.