The Secret Key and the Surprise Bill

As an AWS practitioner who works with AWS day to day, I keep hearing the same headline: someone's AWS secret key leaks, and their bill explodes overnight. The organisation then wakes up to a bill of tens of thousands of dollars. The painful part is that the leak was one mistake, but the size of the disaster came from a deeper one: the key could do anything. Had it been limited to only what the app needed, the leak would have been a minor scare, not a five-figure catastrophe. The five simple habits below can help keep that from happening, followed by AWS's own recommendations, which I have summarised in plain language from its official AWS IAM security best practices guidance.

Habit 1: Give only the access that is truly needed

The technical name for this is "least privilege," but the idea is simple: give every person and every program only the access they need to do their job, and nothing more.

Think of a hotel. When you check in, you get a key card that opens your room, and only your room. It does not open every other guest's room, the manager's office, or the safe in the basement. That is deliberate. If you lose your card, the hotel has a small problem, not a total disaster, because the card could only ever do one small thing.

Now imagine the hotel handed every guest a master key that opened every door in the building. One lost card would compromise everything. That is what went wrong in the story above. The secret key was a master key when it only ever needed to open one room.

Applied to software, least privilege means: if a program only needs to read one folder of files, it does not need the power to delete everything and create servers. If it only needs to send emails, it does not need to touch the customer database. Set up this way, a leaked key or a hacked account can only cause a small, contained amount of harm. This single habit is often the difference between "we had a scare" and "we lost tens of thousands of dollars."

A practical takeaway: when setting up a new tool or bringing on a developer, it helps to ask one question. "Does this program, or this person, have more access than it actually needs?" If the answer is yes, that is a risk worth closing before anything goes wrong.

Habit 2: Keep your secrets out of the shared code

Every program has two kinds of material. There is the code, the instructions that make it work, which is usually fine to share with your team or even the world. And there are the secrets, the passwords and keys, which should never be shared with anyone.

It is common to keep the secrets in a small file that sits right next to the code (developers call it the "env" file). That is fine while it stays on your own computer. The risk comes when everything is uploaded to a shared or public location without removing that file first. It is an easy mistake to make, and a very common one. The moment it happens, the passwords are out in the open. This is how the secret key in our story escaped.

The everyday version of this is taping your house key to your front door and then posting a photo of your door online. The lock is perfectly good. The key has simply been handed to everyone.

The rule is simple: secrets and code travel separately. The secrets stay out of anything you share or upload. Many teams keep a standing instruction that tells the system "never upload the secrets file," so the mistake cannot happen by accident.

And if a secret ever does slip out, deleting it later is not enough. Once a password has been seen, even for a moment, it is safest to treat it as compromised and replace it straight away, the same way you would change the locks if you lost your keys in a public place. The old key needs to be switched off, not just hidden.

Habit 3: Keep secrets in a proper vault, not scattered around

If secrets should not live inside the code, where should they live? The answer is a purpose-built "vault", a secure service whose only job is to hold passwords and keys safely and hand them out only to the programs allowed to have them.

The big cloud providers all offer one. On Amazon's cloud, for example, these services have names like Secrets Manager and the Systems Manager Parameter Store (often shortened to "SSM"). You do not need to remember the names. What matters is the idea: instead of writing the password directly into the program, where it can be copied, leaked, or forgotten, you put it in the vault, and the program asks the vault for it at the moment it is needed.

Think of it like a hotel safe versus a sticky note. The quick approach is to write the password on a sticky note and leave it on the desk. The safer approach is to put the valuables in the room safe, which only opens for the right guest, keeps a record of every time it is opened, and lets you change the combination instantly if you are ever worried.

This approach quietly solves several problems at once. The secret is never sitting inside the code where it can leak. It can be changed in one place without hunting through dozens of files. And you can control and record exactly which programs are allowed to ask for it. It takes a little more effort than a sticky note, and that extra effort is well worth it.

Habit 4: Keep a security camera running

Even with every precaution, things can still go wrong. So the next habit is not about prevention, it is about seeing. You want a record of everything that happens on your systems, so that if something suspicious occurs, you notice quickly and can piece together what happened.

This is the digital version of a security camera and a visitor logbook. On Amazon's cloud, the service that records "who did what, and when" is called CloudTrail, and the general practice of keeping these records is called logging. Every important action, every login, every new server created, every key used, gets written down in a tamper-resistant record.

Why does this matter so much? Two reasons. First, speed. In our story, the organisation was hit with a huge bill because nobody noticed the attack for days. With good logging and alerts, the system could have raised a flag the moment hundreds of unfamiliar machines were suddenly created, turning a five-figure loss into a quick phone call. Second, understanding. After any incident, the logs are how you learn what actually happened, what was accessed, and how to make sure it does not happen again. Without them, it is very hard to know.

A useful question to ask of anyone running your systems: "If something went wrong right now, would we know, and would we be able to see what happened?" If the honest answer is no, that gap is worth fixing before you need it, not after.

Habit 5: Test in a safe space before you go live

The last habit is about where you experiment. When building and trying things out, especially with AI helping to write the code, it helps to do it in a safe, separate space, not on the live system that real customers and real money depend on.

It is worth keeping the two worlds apart. There is the "live" environment, the real thing that customers use, connected to real accounts and real data. And there is a separate "testing" or "practice" environment, a sandbox that looks the same but is walled off, uses fake data, and is connected to nothing that could cost money or expose private information. The experimenting, breaking, and fixing happens in the sandbox first. Only once something is proven safe does it move to the live system.

This is why pilots train in flight simulators before they fly a real plane full of passengers. A mistake in the simulator is a lesson. The same mistake in the air is a tragedy. The simulator exists so that mistakes can happen where they are harmless.

The AWS toolkit: services built for exactly this problem

Here is the good news, and one of the reasons I enjoy working with AWS: it already provides a purpose-built service for every one of the habits above. You do not have to invent the safety net yourself. Mostly, it is a matter of switching it on and using it well. Knowing these services exist can be the difference between an accident waiting to happen and a system that helps protect itself.

The risk	The AWS service	What it does, in plain words
Too much access	IAM, with IAM Access Analyzer	Sets exactly what each user or app is allowed to do; Access Analyzer suggests the tightest safe permissions and flags anything left too open or public
Having a key to leak at all	IAM Roles and temporary credentials (STS)	Gives an app a short-lived, self-expiring identity instead of a permanent secret key, so very often there is no lasting key to leak
Secrets sitting in the code	AWS Secrets Manager and SSM Parameter Store	Stores passwords and keys in a secure vault, hands them only to approved apps, and can rotate them automatically
Not seeing what happened	AWS CloudTrail and Amazon CloudWatch	CloudTrail records who did what and when; CloudWatch watches that record and raises alerts on unusual activity
An attack running unnoticed	Amazon GuardDuty	Continuously watches for threats, including a stolen key suddenly launching crypto-mining machines, and sounds the alarm
A bill exploding silently	AWS Budgets and Cost Anomaly Detection	Warns you the moment spending jumps above normal, turning a five-figure shock into an early phone call
A secret slipping into the repo	Secret scanning in the pipeline (for example git-secrets)	Checks the code before it is uploaded and blocks known key patterns from ever being committed

AWS's own best practices, in plain language

AWS publishes clear, official security best practices, and the ones that would have prevented the story at the top of this article are these:

1. Use permanent keys as little as possible. AWS recommends temporary credentials and IAM roles over long-term access keys. The safest secret is the one that never existed, because a short-lived credential expires on its own before anyone can misuse it.

2. Give the least privilege. Grant only the permissions a task actually needs. Start from AWS's ready-made policies, then tighten them with IAM Access Analyzer, which can build a least-privilege policy from what an app has really used.

3. Turn on multi-factor authentication (MFA). Require a second factor to log in, especially for the all-powerful root account, and use phishing-resistant methods like security keys where possible.

4. Rotate and retire. Regularly update keys, and remove unused users, roles, and permissions using AWS's "last used" information, so old forgotten access cannot come back to cause trouble.

5. Set guardrails and watch everything. Use AWS Organizations to apply rules across all accounts, and switch on CloudTrail, GuardDuty, and cost alerts from day one, not after the first incident.

None of these require specialist tools or deep expertise. They are the cloud equivalent of locking your doors and keeping the receipts, and they are worth putting in place before, not after, something goes wrong.