AI Writes Insecure Code by Default: A Security-First Guide to Secrets Management
Introduction
Ask any AI coding tool to write a script that calls an external API. The first draft will look like this:
import anthropic
api_key = "sk-ant-abc123" # Your API key here
client = anthropic.Anthropic(api_key=api_key)
Push back on security and it improves to a .env file. Push again and it suggests a "config file" or "parameter file" — a named file that holds credentials, excluded from git. This is where most AI assistance stops.
None of these are acceptable in production.
This is not a criticism of any specific tool. AI code generators optimize for producing working code quickly. Security adds friction. The path of least resistance produces vulnerable output, and most developers never push far enough to reach a secure solution.
No-code and low-code platforms reinforce the same habit. Visual automation tools encourage connecting services by pasting credentials directly into form fields — normalizing "put your key here" as the mental model for credential management. The security gap isn't in the platform's storage; it's in the habit it instills.
The result: organizations adopting AI development tools at speed are accumulating security debt they may not recognize. Every AI-generated script that hardcodes a credential, every workflow with a pasted API key, every .env file committed by accident is a liability compounding silently until a breach makes it visible.
Security-first means starting from the vault — not arriving there after an incident.
A secret is any credential that grants access to a system: database passwords, API keys, encryption keys, OAuth tokens, or cloud access credentials. When stored insecurely, a single compromised secret can give attackers complete access to your infrastructure, customer data, and financial systems.
This guide explains why the default path AI tools produce is dangerous, what the available options are ordered from worst to best, and how to implement a proper secrets strategy at any scale.
Part 1: The Problem — Why Secrets Need a Home of Their Own
The Instinct That Fails Us
Most developers' first approach to secrets is intuitive: put them in a .env file, don't commit it, and call it done. This feels secure because:
- The file is "hidden" (not visible in the IDE by default)
- It's locally scoped (only on your machine or server)
- It's separated from the code (not hardcoded in functions)
But "hidden" is not the same as "secure," and this distinction is critical.
Where .env Files Break Down
Git History is Permanent
Once a secret is committed to git—even if you delete it later—it exists in every previous commit. Git is designed to be immutable. Tools like git filter-branch or BFG Repo-Cleaner can rewrite history, but:
- They only work on systems you control
- If anyone cloned the repo before the rewrite, they have a permanent copy
- The secret remains accessible to anyone who has a copy of the repository
The Accidental Commit is Easier Than You Think
.gitignore works until it doesn't:
- New team members don't read documentation and commit their own
.envfile - Contractors unfamiliar with your project structure create a secrets file with a slightly different name that isn't caught by the ignore pattern
- Build systems or deployment scripts accidentally include ignored files in production bundles
- Merge conflicts or automation can cause ignored files to be tracked
Ignored Files Aren't Hidden from Breaches
.gitignore only prevents accidental commits. If an attacker gains access to your server, your GitHub account, or a developer's laptop, they can read ignored files just as easily as tracked ones. A security perimeter breach invalidates the entire .gitignore approach.
No Encryption, No Audit Trail, No Rotation
A .env file on disk is plaintext. There's no way to know:
- Who accessed the secret
- When it was accessed
- If it was compromised
- How to revoke access without redeploying code
The Cost of Exposure
When a secret is exposed, the attacker has:
- Immediate access to whatever system that credential protects (database, API, cloud infrastructure)
- Lateral movement potential — one exposed API key can lead to discovery of other secrets
- Data exfiltration opportunities — full access to customer databases, payment information, personal data
- Persistence options — ability to create backdoors or maintain access for future exploitation
The operational cost includes:
- Incident response and forensics
- Mandatory customer notification (required by law in most jurisdictions)
- Regulatory fines (GDPR, CCPA, HIPAA, PCI-DSS depending on data type)
- Credential rotation across all affected systems
- Loss of customer trust and business
Part 2: The Options Framework
Ordered from worst to best. Each option is a step up from the last.
Hardcoded Values (Please Don't)
What it is: Credentials written directly into source code.
# Don't do this
api_key = "sk-ant-abc123"
stripe_key = "sk_live_abc456"
db_password = "correcthorsebatterystaple"
The classic move. Fast to write, works immediately, and ensures your credentials are immortalized in git history, visible in every diff, every PR review, every contractor's laptop, and potentially every public GitHub search. Points for efficiency. Zero points for security.
Cons:
- Credentials are permanently embedded in git history — deletion does not help
- Visible in code reviews, PRs, and diffs to anyone with repo access
- Exposed in every clone, fork, and backup of the repository
- No rotation possible without a code change and redeployment
- GitHub's secret scanning bots find these within minutes of a public push
- No encryption, no audit trail, no access control
Cost: Free, in the sense that your credentials are also free — for anyone who finds them.
When to use: Never.
.env Files
Pros:
- Zero setup required
- Works offline
- Simple conceptually
Cons:
- No encryption
- No audit logging
- Git history liability
- No rotation mechanism
- Single point of failure
When to use: Local development ONLY. Never in production, never shared, never committed.
Note on encryption: EFS (Encrypting File System) can encrypt
.envfiles at rest on NTFS volumes, which protects against physical disk theft. However, EFS decrypts transparently for any process running as the authorized user — including git — so it does not address git history exposure, accidental commits, or process-level access. The core security risks of.envfiles remain.
Script Parameters (Google Apps Script)
What it is: Configuration values stored as script properties in Google Apps Script.
Pros:
- Simple to use
- Built into Apps Script
- No additional setup
Cons:
- Not encrypted
- Anyone with edit access to the script sees them
- No audit logging
- No access control beyond script edit permissions
- Not designed for secrets
Status: Not recommended for secrets under any circumstance. Acceptable only for non-sensitive configuration.
Windows Registry
Pros:
- Built into Windows OS
- Slightly more protected than plain files
- Low operational overhead
Cons:
- Windows-only (no cloud, Linux, or cross-platform support)
- No encryption by default
- No audit trail
- Not designed for secrets (general configuration store)
- Difficult to rotate without manual effort
When to use: Single-machine Windows deployments where cloud services are intentionally avoided — including local AI workflows running on-premises LLMs (Ollama, LM Studio, Hermes, etc.) that still need to secure supporting credentials such as Gmail OAuth tokens or third-party API keys. Use Windows Credential Manager rather than raw registry keys, as it provides the proper DPAPI interface. Not appropriate for multi-server or enterprise deployments — use HashiCorp Vault self-hosted instead.
Note on encryption: Encryption can be added via Windows DPAPI (
ProtectedDataclass in PowerShell/.NET), which ties encryption to the current user account. Raw registry string values have no encryption without explicit DPAPI wrapping.Note on auditing: Registry access auditing can be enabled via Group Policy (Audit Object Access) with per-key ACLs configured manually. When enabled, it logs to the Windows Security Event Log (Event IDs 4656, 4663). However, this requires non-trivial setup, produces coarser logs than a dedicated secrets manager, and is not enabled by default — it should not be considered a substitute for proper audit logging.
Google Secret Manager
What it is: Google Cloud's managed secrets vault service. Secrets are encrypted at rest and in transit, stored in Google Cloud, accessible via APIs and integrated with Google Cloud services.
Pros:
- Native integration with Google Cloud, Compute Engine, Cloud Run, and Apps Script
- Encryption included
- Detailed audit logging (CloudAudit Logs)
- IAM-based access control
- No infrastructure to manage (fully managed service)
- Versioning and secret rotation support
- Cost-effective at small scale
Cons:
- Google Cloud only (can't run on-premises or other cloud providers)
- Auto-rotation only works for Google-managed services (databases, etc.)
- Third-party API key rotation requires manual process or custom Cloud Function
- Requires Google Cloud project setup (overhead for teams not already in GCP)
How it works:
Your application authenticates to Google Cloud using a service account, requests a secret by name, receives the decrypted value, and uses it. Access is logged with timestamp, requester identity, and secret name.
Cost:
- $0.06 per 10,000 secret access operations
- $0.06 per secret version per month (storage)
- Typical small team usage: $1–$5/month
- At scale (millions of operations): $20–$60+/month
Best for: Teams already using Google Cloud, Apps Script users, organizations that trust Google as their infrastructure provider.
AWS Secrets Manager
What it is: Amazon's managed secrets vault, similar to Google Secret Manager but integrated with AWS services.
Pros:
- Native integration with AWS Lambda, EC2, RDS, and other AWS services
- Automatic rotation for AWS-managed services (RDS databases, etc.)
- Encryption via AWS KMS
- Detailed audit logging (CloudTrail)
- Cross-region replication
- Fine-grained IAM policies
Cons:
- AWS-only (can't use on-premises or with other clouds easily)
- Can become expensive at scale (per-secret pricing plus API call costs)
- Requires AWS account and IAM expertise
- Auto-rotation requires custom Lambda functions for non-AWS services
Cost:
- $0.40 per secret per month (storage) — charged per secret, not per operation
- $0.05 per 10,000 API calls
- 10 secrets = $4/month before any API calls
- At scale with many microservices (100+ secrets): $40–$500+/month
- The per-secret storage model is what drives cost up for larger teams
Best for: AWS-heavy organizations, applications already running on AWS infrastructure.
HashiCorp Vault
What it is: An open-source and enterprise secrets management platform. Vault provides a unified interface for managing secrets, encryption, and identity across multiple infrastructure environments.
Pros:
- Platform-agnostic (runs on-premises, in any cloud, or hybrid)
- Open source (free to use, full source code transparency)
- Advanced features: dynamic secrets, detailed policies, multiple authentication methods
- Works across any infrastructure (Kubernetes, VMs, cloud providers)
- Extensive audit logging
- Encryption at rest and in transit
- Vendor-independent (not locked into one cloud provider)
Cons:
- Operational overhead (you manage and maintain Vault infrastructure)
- Higher learning curve (more features = more complexity)
- Requires expertise to configure securely
- More complex initial setup compared to managed services
Options:
- Free/Open Source: Self-hosted Vault on your own infrastructure. You manage all operations, infrastructure, and updates. Requires operational expertise but zero licensing cost.
- HCP Vault (Managed): HashiCorp's hosted cloud version. Starts around $21/month. Lower operational overhead with professional support.
- Vault Enterprise: Advanced compliance and governance features. Pricing varies by scale.
Note on on-premises infrastructure: The actual infrastructure requirement is lighter than it sounds. A single VM (2 CPU, 4GB RAM) with Vault's built-in Raft storage and a TLS certificate is sufficient for small deployments. Production HA requires 3 VMs minimum. The real complexity is auto-unseal: by default, Vault starts sealed after every restart and requires manual key entry to unlock. For production, auto-unseal typically means connecting to a cloud KMS (AWS KMS, Azure Key Vault, GCP KMS) — meaning most "on-premises" Vault deployments still have a cloud dependency for this one function. A fully air-gapped deployment with no cloud dependency requires a hardware HSM, which adds significant cost ($5K–$20K+) and complexity.
Best for: Multi-cloud deployments, on-premises infrastructure, organizations needing vendor independence, enterprises with sophisticated compliance requirements.
Part 3: Implementation Strategies
Strategy 1: Google Secret Manager with Apps Script
Architecture:
Your Apps Script uses your Google Workspace authentication to access Secret Manager. It requests a secret by name, receives the decrypted value, and uses it. All access is logged.
Setup Steps:
- Create a Google Cloud Project (if you don't have one)
- Go to Google Cloud Console
- Create a new project
- Enable the Secret Manager API
- Create a Service Account (optional but recommended for better audit trails)
- In Google Cloud Console, go to Service Accounts
- Create a new service account
- Grant it "Secret Accessor" role on Secret Manager
- Where possible, use Application Default Credentials or Workload Identity Federation instead of generating a JSON key — JSON key files are high-value attack targets that must be rotated, secured, and tracked. If you must use a JSON key, treat it as a secret and store it in your vault, not on disk.
- Create Your Secrets in Secret Manager
bash
gcloud secrets create anthropic-api-key-prod \
--replication-policy="automatic"
Then paste your API key when prompted.
- Update Your Apps Script
function getSecretFromManager(secretName) {
const projectId = "your-gcp-project-id";
const url = `https://secretmanager.googleapis.com/v1/projects/${projectId}/secrets/${secretName}/versions/latest:access`;
const options = {
method: "get",
headers: {
Authorization: "Bearer " + ScriptApp.getOAuthToken()
},
muteHttpExceptions: true
};
const response = UrlFetchApp.fetch(url, options);
if (response.getResponseCode() === 200) {
const result = JSON.parse(response.getContentText());
return Utilities.newBlob(Utilities.base64Decode(result.payload.data)).getDataAsString();
} else {
throw new Error("Failed to retrieve secret: " + response.getContentText());
}
}
function useAnthropicAPI() {
const apiKey = getSecretFromManager("anthropic-api-key-prod");
const url = "https://api.anthropic.com/v1/messages";
const payload = {
model: "claude-sonnet-4-6",
messages: [{role: "user", content: "Hello"}]
};
const options = {
method: "post",
headers: {
"x-api-key": apiKey,
"content-type": "application/json"
},
payload: JSON.stringify(payload)
};
const response = UrlFetchApp.fetch(url, options);
Logger.log(response.getContentText());
}
Best Practices:
- Use different secret names for different environments:
anthropic-api-key-prodvsanthropic-api-key-dev - Rotate API keys on a schedule (quarterly or monthly recommended)
- Enable audit logging and review access regularly
- Use IAM roles to restrict who can access Secret Manager
- Store secret names in environment variables or config files rather than in code — dev, staging, and prod use different secret names, and externalizing them lets the same codebase deploy across all environments without modification
Strategy 2: HashiCorp Vault (Self-Hosted)
Architecture:
Vault runs on your infrastructure (VMs, Kubernetes, on-premises). Applications authenticate to Vault, request secrets by name, and Vault returns encrypted values. All access is audited.
Installation (Ubuntu/Debian):
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_SHA256SUMS
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_SHA256SUMS.sig
# Verify the signature and checksum before running anything
gpg --verify vault_1.15.0_SHA256SUMS.sig vault_1.15.0_SHA256SUMS
sha256sum -c vault_1.15.0_SHA256SUMS --ignore-missing
unzip vault_1.15.0_linux_amd64.zip
sudo mv vault /usr/local/bin/
vault version
Configuration (production):
# vault-config.hcl
storage "file" {
path = "/mnt/vault/data"
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_cert_file = "/mnt/vault/tls/vault.crt"
tls_key_file = "/mnt/vault/tls/vault.key"
}
seal "awskms" {
region = "us-east-1"
kms_key_id = "arn:aws:kms:..."
}
ui = true
Storing and Retrieving Secrets:
export VAULT_ADDR='https://127.0.0.1:8200'
# Use key=- to read from stdin so the secret value never appears in shell history
printf '%s' 'sk-ant-...' | vault kv put secret/anthropic-api-key-prod key=-
# Retrieve a specific field rather than printing all metadata to the terminal
vault kv get -field=key secret/anthropic-api-key-prod
Python Integration:
import os
import hvac
import anthropic
# Token injected at runtime via environment variable — never hardcoded
client = hvac.Client(
url='https://127.0.0.1:8200',
token=os.environ['VAULT_TOKEN']
)
response = client.secrets.kv.read_secret_version(path='anthropic-api-key-prod')
api_key = response['data']['data']['key']
cl = anthropic.Anthropic(api_key=api_key)
message = cl.messages.create(
model="claude-sonnet-4-6",
messages=[{"role": "user", "content": "Hello"}]
)
Node.js Integration:
// Token injected at runtime via environment variable — never hardcoded
const vault = require('node-vault')({
endpoint: 'https://127.0.0.1:8200',
token: process.env.VAULT_TOKEN
});
async function getSecret() {
const secret = await vault.read('secret/data/anthropic-api-key-prod');
return secret.data.data.key;
}
Best Practices:
- Run Vault in HA (High Availability) mode in production
- Use a proper storage backend (Consul, integrated storage, or cloud KMS)
- Enable audit logging for all secret access
- Implement access policies using Vault's policy system
- Rotate root tokens and service account credentials regularly
- Backup Vault encryption keys securely
Strategy 3: Manual Rotation Process
For third-party API keys (Anthropic, OpenAI, etc.) that don't expose rotation APIs, use this repeatable process:
Monthly API Key Rotation:
- Create new API key at the provider dashboard
- Test the new key in your development environment
- Update the secret in your vault:
bash # Write the key to a temp file, push it, then immediately delete the file gcloud secrets versions add anthropic-api-key-prod --data-file=key.txt shred -u key.txt # secure delete; use 'rm -P' on macOS - Monitor logs for successful API calls with the new key
- Revoke the old key at the provider (after confirming new key works)
- Document the rotation in your change log
Strategy 4: Pre-Commit Secret Scanning
The best time to catch a secret is before it enters git. Pre-commit hooks run locally on each developer's machine and block commits that contain patterns matching known secret formats — API keys, private keys, connection strings, and tokens.
Why this matters:
A developer who accidentally types api_key = "sk-ant-..." in a test file and commits it has already lost, even if they delete it in the next commit. The secret is in git history permanently. Pre-commit scanning catches it before git commit completes.
Tool: gitleaks (recommended)
# Install
brew install gitleaks # macOS
winget install gitleaks # Windows
apt install gitleaks # Ubuntu/Debian
# Scan the current repo manually
gitleaks detect --source . --verbose
# Install as a pre-commit hook (blocks commits automatically)
gitleaks protect --staged
Automate with pre-commit framework:
# .pre-commit-config.yaml — add to your repo root
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.0
hooks:
- id: gitleaks
pip install pre-commit
pre-commit install
Once installed, every git commit runs gitleaks automatically. If a secret pattern is detected, the commit is blocked.
GitHub built-in secret scanning
For repositories on GitHub, enable secret scanning in Settings → Security → Secret scanning. GitHub maintains a database of secret patterns from over 200 providers (AWS, Google, Anthropic, Stripe, etc.) and alerts you — or in some cases automatically notifies the provider — when a match is found in a push. This is a backstop, not a replacement for pre-commit hooks: by the time GitHub scans it, the secret is already in the repo.
Best Practices:
- Install gitleaks (or equivalent) on every developer machine as a pre-commit hook
- Add scanning to your CI/CD pipeline as a second gate
- Enable GitHub secret scanning as a third layer
- Treat any scanner alert as a confirmed compromise — rotate immediately, don't investigate first
Strategy 5: CI/CD Secrets Management
CI/CD pipelines are one of the most common places secrets are mishandled. Build logs are often retained for months, accessible to anyone with pipeline access, and occasionally exposed publicly.
The Docker build arg trap
This is one of the most common mistakes in containerized deployments:
# WRONG — build args are stored in the image layer history
ARG ANTHROPIC_API_KEY
ENV ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
docker build --build-arg ANTHROPIC_API_KEY="sk-ant-..." .
Anyone who runs docker history <image> can see the value. Instead, inject secrets at runtime:
# CORRECT — secret is never in the image
# Inject via environment variable at container start
CMD ["python", "app.py"]
docker run -e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" myimage
Or use Docker BuildKit secrets for secrets needed only during the build:
# BuildKit secret — available during build, not stored in image
RUN --mount=type=secret,id=api_key \
API_KEY=$(cat /run/secrets/api_key) && \
pip install --extra-index-url "https://user:$API_KEY@..." package
GitHub Actions
Store secrets in Settings → Secrets and variables → Actions. GitHub automatically masks these values in logs.
# .github/workflows/deploy.yml
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: python deploy.py
Never do this in a pipeline:
# WRONG — secret visible in logs, shell history, and process list
- run: echo "API_KEY=sk-ant-..." >> .env
- run: curl -H "Authorization: ${{ secrets.API_KEY }}" https://api.example.com # safe
- run: curl -H "Authorization: sk-ant-hardcoded" https://api.example.com # never
Log masking caveat:
CI/CD log masking only works for values the system knows about. If a secret is derived, transformed, or base64-encoded at runtime, the masked value and the transformed value are both visible in logs. Never rely solely on log masking — don't let secrets reach logs in the first place.
Best Practices:
- Use each platform's native secrets store (GitHub Actions Secrets, Jenkins Credentials, CircleCI Contexts)
- Never pass secrets as Docker build args
- Inject secrets as environment variables at runtime, not build time
- Audit who has access to pipeline secrets — it's often broader than intended
- Rotate pipeline secrets on the same schedule as production secrets
- Check that logs don't contain secret values after your first deployment
Part 4: Best Practices Checklist
Access Control
- [ ] Service accounts have minimal necessary permissions (principle of least privilege)
- [ ] Personal user accounts are not used for secret retrieval in production
- [ ] IAM roles or policies restrict who can create, read, or delete secrets
- [ ] API keys are rotated monthly or when suspected compromise
- [ ] Multi-factor authentication enabled on all accounts with secret access
Encryption & Storage
- [ ] Secrets are encrypted at rest in your vault
- [ ] Secrets are encrypted in transit (TLS/HTTPS)
- [ ] Encryption keys are managed separately from secrets
- [ ] No secrets are stored in git repositories
- [ ] No secrets in environment configuration files that get committed
- [ ] No secrets in application logs
Auditing & Monitoring
- [ ] All secret access is logged with timestamp, requester, and action
- [ ] Audit logs are retained for at least 90 days
- [ ] Audit logs are reviewed monthly for suspicious activity
- [ ] Alerts configured for unauthorized access attempts
- [ ] Failed authentication attempts are logged and monitored
Incident Response
- [ ] Runbook documented for secret compromise
- [ ] Process for immediately revoking compromised credentials
- [ ] Communication plan for notifying affected systems/teams
- [ ] Plan for rotating dependent credentials (cascade rotation)
- [ ] Post-incident review process documented
Infrastructure
- [ ] Secrets manager is backed up regularly
- [ ] Backups are tested for recoverability
- [ ] Backup encryption keys are managed separately
- [ ] Infrastructure as Code includes secrets management setup
- [ ] Documentation includes disaster recovery procedures
Prevention & Detection
- [ ] Pre-commit secret scanning installed on all developer machines (gitleaks or equivalent)
- [ ] Secret scanning enabled in CI/CD pipeline as a second gate
- [ ] GitHub/GitLab secret scanning enabled as a third layer
- [ ] Secrets inventory documented — every secret mapped to its consuming service
- [ ] No secrets passed as Docker build args
- [ ] No secrets embedded in URLs, webhook endpoints, or query strings
- [ ] No API keys exposed in frontend JavaScript or mobile app bundles
CI/CD
- [ ] Pipeline secrets stored in platform-native secrets store (not in pipeline YAML)
- [ ] Pipeline secret access restricted to required jobs and environments
- [ ] Build logs reviewed to confirm no secret values appear
- [ ] Docker images scanned for accidentally embedded secrets before push
Part 5: Making the Migration
From .env Files to Secrets Manager:
Phase 1: Inventory (Week 1)
- Identify all secrets in your codebase (API keys, database passwords, tokens)
- Categorize by environment (dev, staging, prod)
- Document current access patterns
Phase 2: Planning (Week 2)
- Choose your secrets management solution based on your infrastructure
- Design the naming convention for secrets
- Plan the authentication mechanism for your applications
- Get buy-in from your team
Phase 3: Implementation (Weeks 3-4)
- Set up your chosen vault
- Create all secrets in the vault
- Update your code to fetch from the vault instead of
.envfiles - Test thoroughly in development and staging
Phase 4: Rollout (Week 5+)
- Deploy changes to production
- Monitor for errors and performance impact
- Remove old
.envfiles from servers - Update deployment documentation
- Train team on new process
Part 6: Common Mistakes to Avoid
- Storing the vault key or credentials in code — This defeats the purpose. The vault credential itself must be injected at runtime, not stored in your code.
- Using the same credentials across environments — Use different API keys, database credentials, and service accounts for dev, staging, and production.
- Not rotating credentials — A compromised key is only dangerous for as long as it's valid. Regular rotation limits the exposure window.
- Excessive logging of secret values — Never log the actual secret. Log that a secret was accessed, by whom, and when — but not the secret itself.
- Sharing vault access too broadly — Limit who can view or rotate secrets to only those who actually need it.
- Assuming the vault itself is secure — Vault security depends on the underlying infrastructure. Lock down the servers hosting it.
- Skipping audit logs — Audit logs are your forensics trail if a compromise happens. Don't skip them or let them accumulate unreviewed.
Part 7: Advanced Considerations
The Secret Zero Problem
Every secrets management system has a bootstrapping problem: to retrieve a secret from a vault, your application needs a credential to authenticate to the vault in the first place. Where does that credential come from? If it's in a .env file, you've solved nothing.
This is called the Secret Zero problem — the first secret that unlocks everything else.
The wrong approaches:
- Hardcoding the Vault token in code
- Storing it in a .env file on the server
- Passing it as a build arg
The right approaches:
Cloud-native identity (recommended for cloud workloads)
Cloud platforms provide identity to workloads automatically via instance metadata. Your application proves who it is by virtue of where it's running, not by presenting a credential.
- AWS: IAM roles attached to EC2 instances or Lambda functions. The SDK fetches a short-lived token from the instance metadata service automatically. No credential to manage.
- Google Cloud: Service account attached to the Compute Engine instance or Cloud Run service. Applications use Application Default Credentials (ADC) — no key file needed.
- Azure: Managed Identity assigned to a VM or App Service.
In each case, the cloud provider is the authority on identity. Your application doesn't hold a secret — it holds an identity, and the platform vouches for it.
OIDC federation (for CI/CD and cross-cloud)
GitHub Actions, GitLab CI, and other platforms can issue OIDC tokens proving a job's identity. Your vault trusts the OIDC issuer and grants access based on claims (which repo, which branch, which environment). No static credential is ever stored.
# GitHub Actions — request an OIDC token to authenticate to AWS
permissions:
id-token: write
contents: read
jobs:
deploy:
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/github-actions-role
aws-region: us-east-1
# AWS SDK now works with temporary credentials — no key stored anywhere
The practical takeaway: If you're on a cloud platform, use its native identity mechanism. If you're not, OIDC federation is the next best option. Static Vault tokens stored on disk are a last resort.
Short-Lived Credentials
The document so far treats credentials as long-lived values that need to be rotated periodically. A more powerful model is to use credentials that expire automatically — minutes or hours after issuance.
Why this matters:
A credential that expires in 15 minutes is fundamentally different from one valid for a year. If it's stolen:
- An attacker has a narrow window to use it before it's worthless
- You don't need to know exactly when the compromise happened to bound the damage
- Rotation is automatic — no runbook, no human intervention
How it works in practice:
AWS STS (Security Token Service)
Instead of a long-lived IAM access key, applications request temporary credentials from STS:
import boto3
sts = boto3.client('sts')
response = sts.assume_role(
RoleArn='arn:aws:iam::123456789:role/my-app-role',
RoleSessionName='my-app-session',
DurationSeconds=900 # 15 minutes
)
# credentials expire in 15 minutes and are useless after that
credentials = response['Credentials']
Google Cloud short-lived tokens
Application Default Credentials automatically manages short-lived OAuth tokens. You never see them or manage them — the SDK handles refresh transparently.
HashiCorp Vault dynamic secrets
Vault's most powerful feature: instead of storing a static database password, Vault creates a unique database credential on demand with a TTL, then revokes it when the TTL expires. The credential never existed before the request and ceases to exist after use.
# Vault creates a unique, temporary Postgres credential
vault read database/creds/my-role
# Key Value
# lease_duration 1h
# username v-token-my-role-abc123
# password A1a-xyz789...
# This credential is automatically revoked after 1 hour
The direction the industry is moving: Long-lived static API keys are a legacy pattern. Short-lived, auto-expiring credentials with cloud-native identity are the target architecture.
Kubernetes Secrets
Kubernetes has a built-in Secret object type. It is widely misunderstood.
The critical misconception: base64 is not encryption.
kubectl create secret generic api-key --from-literal=key=sk-ant-abc123
kubectl get secret api-key -o jsonpath='{.data.key}' | base64 --decode
# outputs: sk-ant-abc123
Anyone with kubectl get secret permission retrieves the plaintext value immediately. base64 is encoding, not encryption — it exists to handle binary data in YAML, not to protect secrets.
By default, Kubernetes stores Secrets unencrypted in etcd. Anyone with etcd access (common in self-managed clusters) has all your secrets in plaintext.
What to do instead:
Enable etcd encryption at rest
# kube-apiserver encryption config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
Use Vault with the Vault Agent Sidecar
The Vault Agent runs as a sidecar container in each pod, authenticates to Vault using Kubernetes service account tokens, retrieves secrets, and writes them to a shared volume that the main container reads. Secrets are never stored in etcd.
Use External Secrets Operator
Syncs secrets from Google Secret Manager, AWS Secrets Manager, or Vault into Kubernetes Secrets automatically, with rotation support.
The rule: Never treat Kubernetes Secrets as secure by default. Treat them as plaintext unless you have explicitly enabled etcd encryption at rest or are using an external secrets operator.
Secrets in Frontend Code
If a secret ends up in JavaScript that runs in a browser — it's public. Browser DevTools, view-source, curl, and automated scrapers all have equal access to your frontend bundle.
The request that keeps coming up: "We need to call the Anthropic API directly from the browser."
Why it can't work:
Any API key bundled into frontend JavaScript is visible to every visitor. It will be found — by accident, by a curious developer, or by an automated scanner. Rate limits and billing are immediately at risk.
The correct pattern: backend proxy
Your frontend calls your own backend. Your backend holds the API key and calls the AI provider.
Browser → your-api.com/generate → [your server with the key] → Anthropic API
Your server can:
- Authenticate the user before forwarding the request
- Rate-limit per user
- Log usage
- Rotate the key without touching the frontend
// Frontend — calls your backend, never holds a key
const response = await fetch('/api/generate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prompt: userInput })
});
# Your backend — holds the key, calls Anthropic
@app.route('/api/generate', methods=['POST'])
def generate():
api_key = get_secret('anthropic-api-key-prod') # from your vault
client = anthropic.Anthropic(api_key=api_key)
# ... call Anthropic, return result
Mobile apps have the same problem. A key embedded in an iOS or Android binary can be extracted with standard reverse engineering tools. The same backend proxy pattern applies.
Secrets in URLs
API keys, tokens, and passwords embedded in URLs are one of the most common and least visible leaks in production systems.
Where URL secrets end up:
- Web server access logs — every request URL is logged by default, including query strings
- Browser history — any user who visits a URL with a token in it has it stored locally
- HTTP Referer headers — if a page with a secret URL links to another site, the destination server receives the secret in the Referer header
- Shared links and screenshots — users copy-paste URLs without realizing they contain credentials
- CDN and proxy logs — intermediate infrastructure logs full URLs
Common patterns to avoid:
# All of these leak the secret in logs and history
https://api.example.com/webhook?api_key=sk-ant-abc123
https://user:[email protected]:5432/mydb
https://callback.example.com/notify?token=eyJhbGci...
The correct pattern:
Secrets belong in headers, not URLs:
# WRONG
requests.get(f"https://api.example.com/data?key={api_key}")
# CORRECT
requests.get(
"https://api.example.com/data",
headers={"Authorization": f"Bearer {api_key}"}
)
For webhooks that must include a token in the URL (some platforms require this), treat the entire webhook URL as a secret — store it in your vault, rotate it, and never log it.
Secrets Sprawl
As a codebase and team grow, secrets accumulate across systems without any central inventory. Sprawl creates two concrete problems:
Problem 1: You can't rotate what you can't find.
When a secret is compromised, you need to know every system that uses it within minutes. Without an inventory, rotation becomes a guessing game — and the systems you miss remain compromised.
Problem 2: Orphaned secrets accumulate.
Services get decommissioned, but their credentials remain valid. Old API keys for retired systems stay active indefinitely, creating unnecessary attack surface.
Building a secrets inventory:
At minimum, maintain a record of:
| Secret Name | Service | Environment | Owner | Last Rotated | Consuming Systems |
|---|---|---|---|---|---|
| anthropic-api-key-prod | AI assistant | production | platform-team | 2026-06-01 | email-assistant, web-api |
| stripe-webhook-secret | Payments | production | payments-team | 2026-05-15 | billing-service |
This doesn't need to be elaborate — a shared spreadsheet works for small teams. The point is that rotation becomes a lookup, not a search.
Vault helps here: Secret Manager and Vault both support metadata and labels on secrets. Tag each secret with the consuming service and owner at creation time.
Preventing Secrets in Logs
Part 6 mentions not logging secret values, but the implementation requires deliberate choices in how you structure logging.
Why it happens accidentally:
# Logs the entire request including headers (which may contain Authorization: Bearer sk-ant-...)
logger.debug(f"Request: {request.headers}")
# Logs the entire env dict (which may contain API keys)
logger.info(f"Config: {os.environ}")
# Exception tracebacks sometimes include variable values
try:
call_api(api_key=secret)
except Exception as e:
logger.exception(e) # may include api_key in the traceback
Patterns to prevent it:
Allowlist logging for sensitive objects:
SAFE_HEADERS = {'Content-Type', 'Accept', 'User-Agent'}
def log_request(request):
safe = {k: v for k, v in request.headers.items() if k in SAFE_HEADERS}
logger.debug(f"Request headers: {safe}")
Use a secret wrapper class that redacts on str():
class Secret:
def __init__(self, value):
self._value = value
def get(self):
return self._value
def __repr__(self):
return "Secret(***)"
def __str__(self):
return "***"
api_key = Secret(get_secret('anthropic-api-key-prod'))
logger.info(f"Using key: {api_key}") # logs: "Using key: ***"
actual_key = api_key.get() # only when you actually need the value
Structured logging with field exclusions:
If you use a structured logging library (structlog, python-json-logger), configure it to scrub known sensitive field names:
REDACTED_FIELDS = {'api_key', 'password', 'token', 'secret', 'authorization'}
def scrub(event_dict):
for key in list(event_dict.keys()):
if key.lower() in REDACTED_FIELDS:
event_dict[key] = '***'
return event_dict
Add log scanning to your checklist: After any deployment, search your log aggregator for known secret prefixes (sk-ant-, sk_live_, AKIA for AWS keys) to confirm nothing leaked.
Quick Reference: Tool Comparison Matrix
| Requirement | Hardcoded | .env |
Registry | Secret Manager | AWS Secrets | Vault |
|---|---|---|---|---|---|---|
| Encryption at Rest | ❌ | ❌ | ⚠️ | ✅ | ✅ | ✅ |
| Audit Logging | ❌ | ❌ | ⚠️ | ✅ | ✅ | ✅ |
| Auto-Rotation | ❌ | ❌ | ❌ | ⚠️ | ✅ | ⚠️ |
| On-Premises Support | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| Multi-Cloud | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Setup Complexity | ⭐ | ⭐ | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐ |
| Operational Overhead | ⭐ | ⭐ | ⭐⭐ | ⭐⭐ | ⭐⭐ | ⭐⭐⭐⭐ |
| Production Ready | ❌ | ❌ | ⚠️ | ✅ | ✅ | ✅ |
Legend: ✅ Full support | ⚠️ Partial/caveats | ❌ Not supported | ⭐ = simplest → ⭐⭐⭐⭐ = most complex
Conclusion
Secrets management is foundational to application security. Moving from .env files to a proper vault takes an afternoon to set up but prevents breaches that cost thousands or millions of dollars.
Start with whichever solution matches your infrastructure:
- Google Cloud shop? Use Google Secret Manager
- AWS shop? Use AWS Secrets Manager
- On-premises or multi-cloud? Use HashiCorp Vault self-hosted
- Local AI deployment on Windows (Ollama, Hermes, LM Studio)? Use Windows Credential Manager with DPAPI — you're intentionally off-cloud and the machine is the trust boundary
The specific tool matters less than implementing one at all. What matters is that secrets never live in code, never live in .env files in production, and never come out of an AI code generator without being moved to a vault first.