Secrets Rotation: What to Do the Day an API Key Leaks | Deployxa

A secret-scanning alert says your API key is sitting in a public repo. The leak-day runbook for founders: rotate first, check the blast window, redeploy, and make rotation boring.

← Back to Dispatch Articles
Engineering Log

Secrets Rotation: What to Do the Day an API Key Leaks

A secret-scanning alert says your API key is sitting in a public repo. The leak-day runbook for founders: rotate first, check the blast window, redeploy, and make rotation boring.

The notification lands at 5:12 on a Thursday afternoon: a secret-scanning tool has found what looks like a live API key in one of your public repositories, pushed 40 minutes ago. Or it is not a tool at all — it is a friend's DM: "Not to alarm you, but I think that is your database URL in the repo you made public this morning." However the news arrives, the next five minutes inside your head run the same unhelpful loop: delete the repo? Change the password? Screenshot everything for the postmortem you have not yet admitted is happening? Close the laptop and check after dinner? Nothing about shipping a SaaS prepares you for this moment, and the quietest instinct on the list — do nothing and hope — is also the most expensive one.

Key Facts

  • Rotation Is Two Different Jobs — Only One of Them Is an Emergency: The word "rotation" covers two disciplines that behave nothing alike, and conflating them is why most founders never rotate anything.

  • API Key Leaked? What to Do First: The Leak-Day Runbook: Six steps, in order.

  • Why Deleting the Commit Is Not Enough: The cleanup instinct fails in four separate ways, and founders routinely lean on it as their entire response.

  • The Everyday Rotation Habit That Makes Leak-Day Boring: Leak-day panics for one reason above all: nobody knows where the secret lives.

Here is why the clock matters. A public repository is not a place where a secret waits politely to be discovered by a human. Scrapers and bots watch for freshly pushed credentials and can copy them within seconds of a push, long before any alert reaches you or anyone you know. The moment a key was public, you assume it was copied. The forty minutes between the push and the notification are not a grace period; they are the blast window, and every decision you make from here on is measured against that timestamp.

If you are searching "API key leaked what to do" in the middle of exactly this, the compressed answer is: act on the secret, not the repo. Rotate the credential at its provider first, measure what happened during the blast window, propagate the new value everywhere the old one lives, redeploy, and only then clean up git history and decide whether customers need to hear from you. This guide walks the leak-day runbook in that order, explains why deleting the commit is never enough, and then builds the everyday rotation habit and prevention upgrades that turn your next leak-day from a panic into a routine. None of it requires a security team.

Rotation Is Two Different Jobs — Only One of Them Is an Emergency

The word "rotation" covers two disciplines that behave nothing alike, and conflating them is why most founders never rotate anything.

Rotation on a schedule is hygiene. Like backups, it is a calendar-driven chore: in calm conditions, with time to test. Its purpose is to shrink the damage of a leak you do not yet know about. If every key rotates every ninety days, the worst possible blast window is ninety days instead of forever. You do it when nothing is wrong, which is exactly why it keeps not getting done.

Rotation on suspicion is incident response. A credential is exposed, the clock started when it went public, and the goal is to stop active damage in minutes. It has an order of operations, timestamps you must record, and decisions — revoke now or rotate with overlap — that have real consequences either way.

Treat the emergency like a chore and it pans out badly: "I will get to it this weekend," while a stranger sends emails from your domain. Treat the chore like an emergency and rotation never happens at all, which is why year-old keys named prod-key-final are still running production.

This post gives the incident runbook first and the habit second, because that is the order you will need them in: on leak-day you need a sequence, not a philosophy, and the philosophy is what makes the next leak-day survivable.

API Key Leaked? What to Do First: The Leak-Day Runbook

Six steps, in order. Steps one and two stop the damage, step three measures it, steps four through six make the fix stick. Resist the urge to start with the repository — the repo is evidence, not the problem.

Step 1: Assume it is compromised — and act on the secret, not the repo

Your instinct says delete the repository, and it is wrong for a specific reason: the repo's value to an attacker ended the moment the key was copied. Push time is exposure time — if the repo was public for forty minutes, an adversary had the credential for forty minutes, regardless of how many page views it had.

Deleting the repo now changes nothing about a key that already left it, and it destroys the evidence you will want in step three. So do the counterintuitive thing: leave the repository exactly as it is, open the provider's dashboard in another tab, and work from there.

Step 2: Revoke or rotate the credential at its provider — before anything else

Only the provider of the credential can actually kill it. Your git host cannot, your deployment platform cannot, and changing your app's password to the service does nothing — the key is a bearer credential: whoever holds it, is you. So the first real action happens in the provider's console, and you have two paths.

Immediate revoke. Kill the key now. Abuse stops instantly, and so does your own app's access to that service. This is the right call when usage patterns suggest someone is actively using the key, or the credential guards something that cannot survive another minute of exposure. Accept the outage — you are trading a controlled interruption for an uncontrolled one.

Rotate with overlap. Issue a new key, deploy it everywhere (step four), verify, then revoke the old one — ideally within minutes, not hours. This keeps your own product running while you work.

If you cannot tell whether the key is being abused, revoke first and sort out the outage second. A broken feature is recoverable; a drained cloud account or a spam campaign sent through your mail domain is a much longer week.

Step 3: Check the blast window

Now that the credential is dead or dying, measure what happened between the leak timestamp and the rotation. Two sources, in order of value:

Provider-side access logs. Many providers — mail platforms, cloud services, payment processors, database hosts — offer usage or API activity logs. Look for calls you did not make: unfamiliar IP addresses, user agents your stack never uses, endpoints your application never touches, volumes that do not match your traffic, activity at hours your product has none.

Your own application logs. Filter everything since the leak timestamp for requests carrying the old credential. Deploy platforms that keep logs per deployment make this easier: you are looking at a bounded, known window rather than grepping through months of noise.

Write down what you find with timestamps, even when the finding is "no anomalous usage observed." That note feeds step six, the eventual postmortem, and — in the worst case — any disclosure conversation. "Found and rotated within 35 minutes; provider logs show four requests from one unknown IP" is a sentence that changes how every later conversation goes.

Step 4: Update the secret everywhere it lives

This is the step founders underestimate, because a modern credential does not live in one place. Enumerate every location where the old value exists and replace it with the new one:

  • Your deployment platform's environment variables for each service that uses it
  • CI/CD pipeline secrets, build-time variables, and test fixtures that reference it
  • Cron jobs and scheduled scripts on any old server or VPS you still pay for
  • The local .env on every machine where you or a teammate runs the app
  • Third-party tools holding a copy — monitoring services, integration platforms, the automation someone built in a hurry
  • Internal docs and wikis, where someone pasted the key "temporarily" eleven months ago

Miss one location and the old key keeps calling home after you have rotated — or a stale copy resurrects itself the next time that machine deploys.

Step 5: Restart and redeploy so nothing still holds the old value

Updating a config file does not change what a running process holds in memory: servers read their environment at boot, and a container image built last week still contains whatever was baked into it. Every service that held the old value needs a restart or a redeploy — and the redeploy is your verification moment, proving the new credential works everywhere before the old one is revoked for good.

This is where health checks earn their keep. A redeploy with a health gate — new version verified before traffic switches — turns a nervous manual moment into a routine release. If the new value is wrong anywhere, you find out through a failed health check and a standby release, not through a flood of customer errors.

Step 6: Decide whether customers need a note

Not every leak is a customer event, and announcing one that was not costs trust you did not need to spend. The trigger is simple: could customer data have been touched by this credential?

A leaked send-only mail key usually cannot reach customer records — no public note, but a documented internal postmortem. A leaked database URL, an admin-level cloud key, or a credential with access to your application database probably is a customer event. If the answer is yes, write the note while the facts are fresh: what credential leaked, what it could access, the exposure window, what your logs showed, and what you changed so the class of problem shrinks. Keep it factual. Do not speculate, and do not write "no customer data was affected" unless your blast-window evidence supports it. Disclosure obligations vary by jurisdiction; when the stakes are real, get advice before you publish, not after.

Why Deleting the Commit Is Not Enough

The cleanup instinct fails in four separate ways, and founders routinely lean on it as their entire response.

Git history keeps everything. Removing a file in a new commit does not remove it from the repository's history; every prior commit still contains the secret, and anyone with the repo URL can read it with one command. History-rewrite tooling exists, but it is a hygiene step to run after rotation — never the containment itself.

Clones and forks are outside your control. The moment a public repo was forked, a copy existed that you cannot delete. Anyone who cloned it locally — a collaborator, a curious reader, a bot — holds a copy too, and rewriting your history does nothing to theirs.

Caches and logs remember. CI systems cache artifacts, pull-request previews snapshot branches, build logs can echo environment values, and code-search indexes may have catalogued the file in the minutes it was visible.

Scrapers are faster than you are. Public code hosts are watched by automated harvesters that act on credential patterns within seconds to minutes of a push. By the time you saw the alert, assume copies exist in places you will never see.

The ordering falls out naturally: rotate first, because cleanup cannot contain anything; clean history second, because a dead key in history is still future confusion. Never invert the order. Deleting the repo feels decisive because it is visible and immediate — which is exactly what makes it dangerous: it consumes the adrenaline the credential itself needed.

The Everyday Rotation Habit That Makes Leak-Day Boring

Leak-day panics for one reason above all: nobody knows where the secret lives. The panic is not the leaked key; it is the archaeological dig to find every copy before you can rotate with confidence. The habit that fixes this is small and unglamorous.

Secrets live in environment variables and config, never in code. A secret in code is in every commit, every clone, every code-review screenshot, forever. A secret in an environment variable is a value your platform injects at deploy time — swappable without touching the repository.

One place of record per secret. Each credential has exactly one authoritative location — your deployment platform's config or a secrets manager — and every other copy is a known downstream copy. When you can answer "where does this live?" in five seconds, rotation stops being archaeology.

Quarterly rotation for long-lived keys. Calendar it like backups. Ninety days caps the blast window of an unknown leak, and each routine rotation doubles as a fire drill for the process you will need on a bad day.

Short-lived credentials wherever the provider supports them. Some providers issue tokens that expire on their own — hours or days. A credential that dies of old age cannot leak for long, so prefer it wherever offered and watch your inventory of eternal keys shrink.

A written inventory. One table, one page, kept current. If this feels excessive, imagine the version of you at 5:12 on a Thursday trying to reconstruct it from memory:

Secret

Lives in

Used by

Last rotated

Mail provider send key

Platform env vars + CI secret

App email service, invoice notifications

2026-09-02

Database URL (prod)

Platform env vars

Web app, nightly report cron

2026-08-14

Cloud storage key (read/write)

Platform env vars

File upload service

2026-07-30

Maps API key (restricted)

Platform env vars

Address autocomplete

2026-09-15

Analytics ingest token

CI secret + platform env vars

Event pipeline worker

2026-08-01

(Illustrative rows — yours will have different services and dates.) Five rows is enough to start. The discipline is not the spreadsheet; it is that rotating any single credential requires no discovery work, because the "lives in" column is already true.

Prevention Upgrades Worth One Afternoon

Rotation limits how long a leaked key is dangerous. These five upgrades shrink how often keys leak at all, and each is an afternoon of work.

Secret scanning in CI. Secret-scanning tools check every push and pull request for credential patterns and fail the build on a hit. The point is not the specific tool — it is that the check runs on the machine that receives the code, every time, without depending on anyone's memory. A secret that never lands in a repository cannot leak from one.

Pre-commit hooks. The same idea moved earlier: a local hook scans staged changes before the commit is created — your last line between a git add . that swept up a .env and a push that makes it public. Hooks are bypassable, which is exactly why CI scanning runs anyway.

.env in .gitignore from commit zero. The cheapest control on this list, and the one that prevents the most leaks. New repositories get a .gitignore that excludes .env, .env.local, and every variation, on day one — before there is anything secret to protect. Guard against the "just this once" commit with the promise to remove it later; later never comes, and history keeps the promise for you, in the worst way.

Least-privilege keys. Where a provider offers scoped keys, use the narrowest scope that works: read-only where reads suffice, single-purpose where purposes can be separated. This is blast-radius design. A leaked key that can only send emails is an incident; a leaked key that can read the database, delete storage buckets, and spin up servers is a catastrophe. The scope you choose in a calm afternoon decides which of those you are having at 5:12.

Separate keys per environment. Staging and production get different credentials for every service, so a staging leak is a contained mess instead of a customer event. This also makes the inventory honest: "used by" becomes specific, and nobody has a reason to email themselves a production key "just to test locally."

A Fictional Walkthrough: The Committed .env File

Here is the runbook applied end to end. The scenario is illustrative — a fictional two-person invoicing SaaS — but every step maps to the ladder above.

At 4:32 on a Tuesday, one of the two founders pushes a large refactor to a repository that was public for demo purposes. Inside it, swept up by a careless git add .: a .env containing a production send-only key from their mail provider. At 5:12, a secret-scanning alert arrives:

SECRET SCANNING ALERT — illustrative fictional alertFinding: Mail-provider API key in public repositoryRepo: acme-invoices-demoPushed: 40 minutes agoPattern: Send-key prefix (live credential)Advice: Revoke or rotate this credential now

5:14 — She resists deleting the repo and opens the mail provider's console instead. Logs show no signs of active abuse, so she takes the overlap path: issues a new send key (step two).

5:16–5:25 — She updates the key in the platform's environment variables, the CI secret, and her cofounder's laptop .env (step four). The sweep finds one surprise the inventory had missed: a weekly "email the summary" cron on an old VPS they never decommissioned. It gets the new key too.

5:26 — She redeploys the web app with the new value; the health check passes on the first try (step five). The cron is restarted by hand.

5:31 — Old key revoked at the provider. Total exposure of the old credential: from push to revoke, fifty-nine minutes; from detection, nineteen.

5:35 — Blast window check (step three): provider logs show four send calls from one unfamiliar IP in the first twelve minutes after the push — spam attempts, all rejected by rate limits. She screenshots and timestamps the logs.

5:50 — Trust-note decision (step six): a send-only mail key cannot reach customer records, so no customer note is owed — but the incident is written up internally, including the missed cron job.

That evening — History cleanup: the repo goes private, a history rewrite removes the .env, and the residual risk is logged honestly as "unknown copies may exist; credential revoked, so copies are inert." Prevention: .gitignore fixed in every repo, pre-commit hook installed, CI scanning enabled, and the inventory table gets its missing fifth row.

Ninety minutes, start to finish, and the scary part was over in the first twenty. That is what the runbook buys: not the absence of leaks, but a response that fits inside one afternoon.

Where a Deployment Platform Helps — and Where It Cannot

A deployment platform does not rotate keys for you, but it decides how much of the runbook is friction and how much is routine. The differences that matter on leak-day are specific.

If your platform manages environment variables per deployment — as Deployxa does — then step four and step five collapse into a single loop: update the value in the dashboard, redeploy, and the new value propagates through every service that reads it, without touching code or repositories. No SSH-ing into machines, no hand-edited .env files, no "which box holds the old value?" hunt. The documentation covers the env-var workflow, and the product suite handles the surrounding mechanics — builds, health gates, domains — that the redeploy rides on.

Per-deployment logs pay off twice: during the blast-window check they give you a bounded, timestamped view of what your app actually did, and after the redeploy they confirm the new credential is the one in use. Health-gated deploys make the rotate-and-redeploy step genuinely routine — a new release goes to a standby slot, proves it is healthy with the new secret, and only then receives traffic. The step that used to be a gamble becomes a deploy.

Now the honest limits, because a security article that oversells its tools is not a security article. Git history cleanup is your job — the repo, the rewrites, the forks you cannot reach. Provider-side key management is entirely in each provider's console; your deployment platform has no revocation power over your mail provider, your database host, or your cloud account. And no platform can un-leak a secret: the exposure happened at push time, and nothing undoes it. What the right platform changes is the cost of responding — minutes instead of an evening — while the runbook, the judgment calls, and the inventory remain yours. That division of labor is not a limitation to hide; it is the actual shape of the work.

The Leak-Response Checklist

Run through this once a quarter, and after every real leak:

  • [ ] For every public repo, treat push time as exposure time — no grace periods
  • [ ] Rotate or revoke at the provider first; git cleanup is never the first move
  • [ ] Revoke immediately when abuse is suspected; use overlap rotation when it is not
  • [ ] Blast window documented from provider logs and app logs, with timestamps
  • [ ] New secret deployed to every location: platform config, CI, cron jobs, teammates' machines, third-party tools
  • [ ] Every service that held the old value restarted or redeployed, health verified
  • [ ] Customer note written only when customer data could have been touched — factual, no speculation
  • [ ] Inventory table current: every secret, where it lives, what uses it, when it last rotated
  • [ ] Long-lived keys rotate quarterly on the calendar; short-lived credentials used where providers support them
  • [ ] .env ignored from commit zero in every repo; secret scanning in CI and pre-commit hooks enabled; keys are least-privilege and separated per environment

Your Next Step: Rotate One Key on Purpose

Here is the highest-priority item on that checklist, and you can finish it tonight. Pick the one credential that would hurt most if it leaked — for most SaaS founders that is the production database URL or the primary cloud key — and add its row to the inventory table: where it lives, what uses it, when it last rotated. Then rehearse rotating it end to end, deliberately: issue the new value, propagate it everywhere the inventory says it lives, redeploy, verify, revoke the old. Do the rehearsal in a staging or non-production project — never against a production system with paying customers — and time yourself. If the whole loop takes under an hour, leak-day is now a procedure instead of a panic. If it takes all evening, that number is the finding, and shrinking it is your next improvement. A platform like Deployxa makes the propagate-and-redeploy loop minutes of work, but the decision to rehearse is yours, and it is worth making before 5:12 on some future Thursday makes it for you.

Ready to deploy with Deployxa?

Deploy your apps globally with automatic SSL and AI diagnostics.

Start Free Now