You Probably Don't Need Vault: How Deleting a Cache Made Our Key Storage More Secure

A deep dive into envelope encryption for a BYOK product, and the counterintuitive story of how removing a component collapsed the most dangerous window in the whole design.

---

Disclaimer: The code snippets in this article are real, lightly trimmed excerpts from pCraft's Go backend, with comments condensed and error handling elided for readability, not pseudocode. You'll still want to adapt the patterns to your own threat model, deployment shape, and crypto review.

---

Introduction

What does "we encrypt your API keys at rest" actually have to mean before it's true?

Not the marketing version: the version an attacker who has just walked out with a copy of your production database would agree with. That's the bar I set for pCraft, and clearing it took me somewhere I didn't expect: I made the system materially more secure by deleting a component I had spent real effort building. No new cryptography, no hardening sprint. I removed a network hop and a cache, and the most dangerous window in the whole design collapsed from sixty seconds to a few microseconds.

This is the story of that decision: the threat model that justified it, the envelope-encryption scheme underneath it, and the uncomfortable question it forced me to answer honestly. Do you actually need HashiCorp Vault, or do you just feel like you're supposed to?

It's a long one. Grab a coffee.

What You'll Learn

• Why "encrypted at rest" only means something if a stolen database backup is genuinely useless
• How envelope encryption splits the problem across two keys, and how to bind ciphertext to its owner so it can't be spliced
• Why a colocated secrets server buys you ceremony, not isolation, and how to tell the difference
• How removing a network round-trip let me delete a cache that a security review had flagged as critical
• How to version and rotate a key-encryption key without re-encrypting a single row
• The interface seam that turned "rip out Vault" into a one-day change instead of a rewrite

---

The setup: pCraft holds keys that are literally money

pCraft is, in one line, Postman for LLMs: a workbench for building, versioning, and running multi-step prompt chains across Anthropic, OpenAI, and Google, side by side. The core business decision is BYOK: bring your own key. You paste in your own provider API key, and pCraft runs your chains against your provider account. I don't resell tokens, I don't sit on a markup, and I don't lose money on my power users the way a flat-rate wrapper does.

That model is great for unit economics and terrible for your blood pressure, because it means my database is full of other people's provider keys. And an LLM provider key is not like a hashed password. It's a bearer credential with a credit card behind it. If it leaks, the attacker doesn't need to crack anything; they just start spending. The blast radius of a key leak is "how much is the victim's monthly spend limit," and for a serious user that number has a lot of zeros.

So the storage design isn't a checkbox feature. It's the load-bearing wall. If I get this wrong, the entire product is a liability instead of a tool.

The architectural invariant I wrote down, and have defended in every review since, reads like this:

User LLM API keys exist in plaintext only in process memory for the duration of one LLM round-trip, and at rest only as AES-256-GCM ciphertext under an envelope scheme whose key-encryption key never appears in the database.

Every word of that sentence is a constraint I have to actually honor in code. The interesting part of this post is the phrase "for the duration of one LLM round-trip", because for a while, that sentence was a lie, and the thing making it a lie was a performance optimization I'd added on purpose.

---

Envelope encryption, quickly

Before the plot, the mechanism. If you already know envelope encryption cold, skim to the next section; if you don't, this is the whole foundation.

Naively, "encrypt the keys at rest" means: pick one secret, AES-encrypt every key with it, store the ciphertext. The problem is that one secret. It has to be available to the running app to decrypt anything, it has to be the same for every row, and rotating it means re-encrypting every ciphertext you've ever stored. It's a single point of failure that's also a single point of operational pain.

Envelope encryption splits the job across two layers, the way a physical envelope separates the letter from the address:

• Every secret gets its own data-encryption key (DEK), a fresh 32 bytes from the CSPRNG, unique per stored key. The DEK encrypts the actual plaintext (the user's API key) with AES-256-GCM.
• A single long-lived key-encryption key (KEK) encrypts the DEKs. The KEK lives outside the database, in pCraft's case in the process environment.
• You store the ciphertext, the per-row wrapped (encrypted) DEK, and the nonce. You never store the KEK.

The payoff is the property I care about: the database alone is worthless. A stolen backup gives an attacker the ciphertext and the wrapped DEKs, but the wrapped DEKs are themselves encrypted under a KEK that only exists in the application's runtime environment. To decrypt anything, you need both a database row and the running process's environment, two separate compromises, not one. Rotation gets cheap too: to retire a KEK you re-wrap the small DEKs, not the whole corpus (and, as you'll see, in pCraft you don't even do that eagerly).

Here's the actual wrap path in pCraft. Two Seal calls, two layers:

func (l *LocalKEK) Encrypt(_ context.Context, bindingContext, plaintext []byte) (Wrapped, error) {
	dek := make([]byte, 32)
	if _, err := io.ReadFull(rand.Reader, dek); err != nil {
		return Wrapped{}, fmt.Errorf("keywrap: dek: %w", err)
	}
	defer zero(dek) // the plaintext DEK is gone the instant this function returns

	// Layer 1: the per-row DEK encrypts the user's key.
	ciphertext, nonce, err := sealGCM(dek, plaintext, bindingContext)
	if err != nil {
		return Wrapped{}, err
	}

	// Layer 2: the KEK wraps the DEK. Same binding context as AAD.
	wrappedCT, wrappedNonce, err := sealGCM(l.current[:], dek, bindingContext)
	if err != nil {
		return Wrapped{}, err
	}
	wrapped := fmt.Sprintf("%s%d:%s", wrappedDEKPrefix, l.version,
		base64.StdEncoding.EncodeToString(append(wrappedNonce, wrappedCT...)))

	return Wrapped{
		WrappedDEK: []byte(wrapped),
		Ciphertext: ciphertext,
		Nonce:      nonce,
		KEKVersion: l.version,
	}, nil
}

Four values come out: WrappedDEK, Ciphertext, Nonce, and KEKVersion. All four go into the Postgres row. None of them is sensitive in isolation. That's the whole point.

Hold onto that bindingContext argument threaded through both sealGCM calls. It's the cleverest part of the design and I'll come back to it. First, the villain of the story.

---

Why I was running Vault at all

In the original architecture, the KEK didn't live in an environment variable. It lived in HashiCorp Vault, and I used Vault's Transit secrets engine to do the DEK wrapping. Transit is "encryption as a service": you hand Vault a blob, it wraps or unwraps it with a master key that never leaves Vault, and hands you back the result. It's a genuinely good product, and reaching for it felt like the responsible, grown-up choice. Real companies run Vault. Storing a master key in an env var felt like something you'd get flamed for in a code review.

But Transit has a property that matters enormously for this story: it's over the network. Every unwrap is an HTTP round-trip to the Vault server. On the same box that's cheap; across a cluster it's a millisecond or three. Individually, nothing. In aggregate, a problem, because pCraft runs chains.

A single run might be a five-step chain, and every step reuses the same provider key, which means every step needs the same DEK unwrapped. Do that naively and one chain execution is five Vault round-trips just to keep decrypting the same 32 bytes over and over. Multiply by concurrent runs and you're hammering Vault to re-derive material you already had a moment ago.

So I did the obvious thing. I added a cache.

// A small in-process DEK cache (60-second TTL, keyed by SHA256 of wrapped_dek)
// amortizes the unwrap cost across multi-step chains that reuse the same key.
const dekCacheTTL = 60 * time.Second

type dekCacheEntry struct {
	dek      []byte
	expireAt time.Time
}

Unwrap a DEK once, keep the plaintext DEK in a map keyed by sha256(wrapped_dek), and for the next sixty seconds every step in that chain, and every concurrent run using the same key, hits memory instead of the network. Textbook. I even did the hygiene right: a background sweeper zeroed expired entries, put() zeroed the bytes it overwrote, and Close() wiped the whole cache on graceful shutdown so a clean pod rotation wouldn't leave plaintext lying around.

I was pretty pleased with it. Then I had the codebase security-reviewed, and the reviewer put their finger on exactly the thing I'd talked myself out of worrying about.

---

The finding that named the price

The review's headline finding, rated critical, was blunt:

The DEK cache holds plaintext data-encryption keys in memory, against the architectural invariant. It changes the blast radius of any process-memory exposure from per-key to per-tenant-per-window.

Read that carefully, because it's the crux. My invariant promised plaintext key material lived in memory "only for the duration of one LLM round-trip." The cache blew straight through that promise. It was holding plaintext DEKs, the keys that decrypt the user keys, in a map for up to sixty seconds (five minutes, in the first version, before an earlier fix shortened it). And a DEK held in memory is one AES operation away from the user's actual API key.

Now play out the compromise. Suppose an attacker gets a read of process memory. This is not exotic. A pprof endpoint accidentally left exposed. A panic that dumps goroutine locals into a log aggregator. A container core dump. A /proc/<pid>/mem snapshot on a shared host. None of those is remote code execution; several are ordinary operational mishaps. With the cache in place, any one of them hands the attacker every DEK that passed through the cache in the last window, which, on a busy node, is a large fraction of the currently active tenants. One heap dump, many victims.

Without the cache, that same heap dump catches only whatever single key happens to be mid-flight in a request goroutine at that exact instant, on its way to being zeroed. The difference between those two worlds is the difference between "an incident" and "a catastrophe."

Here's the part that stung: the cache was defensible. I could zero the entries, I could shorten the TTL, I could mlock the pages to keep them out of swap. Each of those makes the window smaller. But every one of them is mitigation, shrinking a hole rather than closing it. The cache existed for one reason: to amortize a network round-trip. It was a workaround for a latency problem I had chosen to have by putting the KEK behind a network service. The security cost was a direct, quantifiable consequence of that choice.

Which reframes the whole question. The right move isn't "how do I make the cache safer." It's "why is there a network round-trip to amortize in the first place?"

---

Delete the network, and the cache deletes itself

Around the same time, I was scaling pCraft's whole deployment down to something a solo founder can actually operate: from a Kubernetes topology I'd overbuilt (and never actually deployed) to a boring, cheap platform-as-a-service setup. Vault was one of the last things standing between me and a clean boot. So I asked the question directly: on a single-app deployment, what is Vault actually buying me?

Transit's core operation, the only Vault feature this system truly depends on, is AES-256-GCM key-wrapping with a master key Vault holds. That's it. That's a dozen lines of Go against the standard library. The reason to pay for a whole Vault server to do it is isolation: the master key lives on separate, hardened infrastructure, with an audit log and unseal ceremony around it.

But my Vault wasn't on separate infrastructure. In the Kubernetes design it ran on the same cluster; in the MVP it would run on the same box as the app it protected. And colocated Vault doesn't buy you isolation. An attacker with root on that machine gets the KEK path either way: a token sitting in the app's environment to talk to Vault on localhost is not meaningfully harder to steal than the KEK itself sitting in that same environment. What colocated Vault does still buy you is audit logging and rotation ceremony. Real things! Valuable at compliance time. Worth approximately nothing when your user count is zero and your threat model is "don't lose the keys," not "prove to an auditor who touched them."

So I replaced Vault Transit with a local KEK: 32 bytes, base64-encoded, generated once with openssl rand -base64 32, delivered to the app as an encrypted environment variable. The KEK now lives in process memory for the lifetime of the process, which it has to, because it's needed for every wrap and unwrap, and the wrapping happens with crypto/aes and crypto/cipher in-process.

And here's the payoff, the thing that made this worth writing about. A local AES-GCM unwrap of 32 bytes takes on the order of a microsecond. There is nothing to amortize. The entire reason the cache existed, the network round-trip, is gone. So the cache goes with it. The doc comment on the new implementation says it plainly:

// LocalKEK envelope-encrypts secrets under an in-process 32-byte KEK.
//
// Unlike the Vault implementation there is no DEK cache: unwrapping is a
// local AES operation (~µs), so every Decrypt unwraps fresh and zeroes the
// DEK before returning, a strictly shorter plaintext-DEK lifetime than the
// 60s cache the Vault client needs to amortize network round-trips.

Look at what happened to the invariant. Before: a plaintext DEK could live in a cache for up to sixty seconds, reachable by any memory-exposure event in that window. After: a plaintext DEK exists only inside a single Decrypt call, wrapped in defer zero(dek), and is scrubbed before the function returns. Its lifetime dropped from sixty seconds to microseconds. The blast radius of a heap dump dropped from per-tenant-per-window back to per-key-in-flight.

I made the system more secure by deleting code. Fewer moving parts, no cache to reason about, no sweeper goroutine, no TTL to tune, no mlock to consider, and a strictly better security posture on the exact axis the reviewer flagged as critical. This is the rarest and most satisfying kind of change: the one where the security win and the simplicity win point in the same direction, because they had the same root cause. The complexity and the exposure were both children of the network hop. Remove the parent, and both children vanish.

That's the counterintuitive lesson worth internalizing: a cache is a liability you accept to pay down a cost you chose to incur. When you can remove the cost instead of caching around it, you often remove a whole class of bug with it. Security bugs love caches, because a cache's entire job is to keep sensitive things around longer than they'd otherwise live.

---

"But an env var is less secure than Vault." Is it?

This is the objection everyone raises, and it deserves a straight answer instead of hand-waving, because sometimes it's right.

The honest comparison is not "env-file KEK versus Vault." It's "env-file KEK versus the Vault I was actually going to run." And the Vault I was actually going to run was colocated with the app. Here's the threat model laid out plainly:

The two rows where Vault genuinely wins, audit logging and rotation ceremony, are compliance properties, not confidentiality properties. They matter enormously the day an enterprise customer sends you a security questionnaire, and approximately not at all the day you have zero users and are trying to ship. And the one row where the local KEK wins outright is the one that was flagged critical.

The trap I nearly fell into is a specific and common one: reaching for a heavyweight security tool because it signals rigor, without checking whether the deployment shape lets it deliver the property the tool is famous for. Vault on separate, hardened infrastructure with a KMS-backed auto-unseal is a real security boundary. Vault on the same box as the app, unsealing itself from a key on that same box, is ceremony: it looks like a boundary and isn't one. Ceremony has a cost (operational surface, a KMS dependency whose only job is unsealing Vault, backups of Vault's own storage) and, in that shape, no matching benefit. Cargo-culting the tool without cargo-culting its deployment topology gets you the bill without the protection.

None of this means Vault is bad. It means Vault is a control that pays off under a threat model I don't have yet. Which is why I didn't delete it: I made it swappable. More on that below.

---

The parts I kept, because they were never the problem

Removing Vault didn't touch the actual cryptographic design, because Vault was never doing the interesting cryptographic work; it was just holding one key. The scheme underneath is where the real security lives, and it's worth walking through, because these are the parts that survive regardless of where the KEK lives.

The binding context: AEAD as an anti-splicing weapon

Remember that bindingContext I told you to hold onto. Here's where it earns its keep. It's constructed from the tenant ID and the key's own ID:

func BindingContext(tenantID, keyID string) []byte {
	return []byte(tenantID + ":" + keyID)
}

This value is fed as the associated data (AAD) to both GCM layers, the DEK-encrypts-plaintext layer and the KEK-wraps-DEK layer. AAD in an AEAD cipher isn't encrypted; it's authenticated. The Open operation cryptographically refuses to return plaintext unless the exact same AAD is supplied at decrypt time. And critically, the binding context is never stored; it's recomputed from the row's own identity every time you decrypt.

Think about what that defends against. Suppose an attacker has database write access and tries a splicing attack. Say Mallory (tenant M, key k_M) wants Victoria's key. She copies Victoria's wrapped_dek, ciphertext, and nonce into her own row and asks pCraft to smoke-test it. Without AAD binding, envelope encryption doesn't inherently stop this: the DEK is the DEK, and the KEK would happily unwrap it. With it, the moment pCraft tries to decrypt Mallory's row, it recomputes the binding context from that row's identity, M:k_M, but the ciphertext was sealed under V:k_V. The tags don't match. Open returns an error instead of plaintext. The key stays sealed.

This is defense that survives even a total row-level-security bypass. pCraft already enforces tenant isolation at the Postgres RLS layer, but the AAD binding means that even if an attacker defeated RLS entirely, they still couldn't unwrap another tenant's key, because the cryptography itself is tenant-aware. Two independent walls, and the crypto wall doesn't depend on the database wall holding.

There's a subtle implementation consequence worth flagging for anyone building this: the key's UUID has to be minted before the INSERT, not generated by the database, because the ID is part of the binding context that gets baked into the ciphertext. You can't bind to an identity you don't have yet. So the service does keyID := uuid.New() up front and threads it into both the encryption and the InsertProviderKeyWithID call. A small thing that a DEFAULT gen_random_uuid() column would quietly break.

The wrapped_dek prefix: ciphertext that names its own decryptor

The other design choice that made the whole migration painless is that every wrapped DEK is self-describing. It's not just a base64 blob; it carries a prefix:

local:v2:iVBORw0KGgoAAAANSUhEUg...

local says which cipher implementation can unwrap it (the local KEK, versus vault: for a Transit-wrapped row). v2 says which KEK version. The parser splits it back apart on read:

func parseWrappedDEK(s string) (version int, blob []byte, err error) {
	rest, ok := strings.CutPrefix(s, wrappedDEKPrefix) // "local:v"
	if !ok {
		return 0, nil, errors.New("keywrap: wrapped DEK is not local-format")
	}
	vStr, b64, ok := strings.Cut(rest, ":")
	// ... parse version, base64-decode the blob
}

The ciphertext tells the system how to decrypt it. This is what let Vault and the local KEK coexist in the same column during the transition: old rows wear vault:v1:, new rows wear local:v1:, and each decrypts through the right path with zero data migration. It's a tiny protocol, and it's the reason "rip out Vault" was a one-day change instead of a re-encrypt-everything project.

Rotation without re-encrypting a single row

The same prefix makes KEK rotation nearly free, and this is where the "never store the KEK" discipline pays a second dividend. To rotate:

1. Generate a new KEK, set it as PCRAFT_KEK, bump PCRAFT_KEK_VERSION to 2.
2. Move the old KEK to PCRAFT_KEK_V1 so old rows stay readable.

New writes wrap under v2. Old rows keep their local:v1: prefix and decrypt under the retained v1 KEK. Nothing gets re-encrypted eagerly; rows lazily rewrap to v2 the next time they're written. The Decrypt path just selects the right KEK by the version it parsed out of the row:

kek, ok := l.kekFor(version)
if !ok {
	return nil, fmt.Errorf("keywrap: no KEK for version %d (set PCRAFT_KEK_V%d to decrypt pre-rotation rows)", version, version)
}

Notice the error message names the exact environment variable you forgot to set. Which brings me to the last principle.

Fail at boot, not at 3 a.m.

The failure mode I most wanted to avoid is the one where a KEK misconfiguration lies dormant and only surfaces when some user tries to run a chain against a pre-rotation key at an unpredictable hour. So the validation is front-loaded to startup. The boot path scans the environment for PCRAFT_KEK_V<N> variables and refuses to boot if any retained key claims a version greater than or equal to the current one:

if v >= currentVersion {
	return nil, fmt.Errorf("keywrap: %s (v%d) must be older than PCRAFT_KEK_VERSION (%d)", name, v, currentVersion)
}

And the constructor rejects any key that doesn't base64-decode to exactly 32 bytes. A fat-fingered rotation doesn't corrupt data or half-work: the process crash-loops immediately with a message that tells you which variable is wrong. A latent, data-dependent production failure becomes a loud, immediate, un-missable boot failure. That's a trade I'll take every single time: I would much rather a bad config stop the deploy than pass the deploy and fail a user.

---

The seam that made "rip out Vault" a one-day job

The reason all of this was a calm, one-day change instead of a heart-in-throat rewrite is a single interface, defined before I knew I'd need it. Both implementations satisfy keywrap.Cipher:

type Cipher interface {
	Encrypt(ctx context.Context, bindingContext, plaintext []byte) (Wrapped, error)
	Decrypt(ctx context.Context, bindingContext []byte, w Wrapped) ([]byte, error)
	Close() error
}

The Wrapped struct and the four Postgres columns were provider-agnostic from day one. So swapping Vault for the local KEK meant writing one new implementation of a three-method interface and flipping which one gets constructed at boot, chosen by whether PCRAFT_KEK is set:

func newCipher(ctx context.Context, cfg config) (keywrap.Cipher, error) {
	if cfg.KEK != "" {
		prev, err := previousKEKsFromEnv(cfg.KEKVersion)
		if err != nil {
			return nil, err
		}
		return keywrap.NewLocalKEK(cfg.KEK, cfg.KEKVersion, prev)
	}
	// ... else fall through to Vault (static token in dev, K8s auth in a cluster)
}

Crucially, I didn't delete the Vault implementation. It's still compiled, still tested against a real Vault container in CI, still selectable by leaving PCRAFT_KEK unset. The dev environment still exercises it daily. This matters because the whole argument for removing Vault was "not at this scale, not this deployment shape", which is an argument that reverses the day an enterprise customer with a compliance department shows up. When that day comes, re-admitting Vault (or a cloud-KMS-backed KEK) is an environment-variable change per deployment, and kek_version plus lazy rewrap migrates rows without downtime. I removed the dependency, not the option. Deleting a capability and disabling a capability look similar in a diff and are worlds apart in what they cost your future self.

---

So, do you need Vault?

Here's the framing I wish I'd started with. A managed secrets service earns its operational cost when at least one of these is true:

• The KEK genuinely lives on separate infrastructure from the app it protects, so compromising the app doesn't hand over the key. (Colocated Vault fails this; it's the whole point above.)
• You need an audit trail of every unwrap for compliance, forensics, or dual-control.
• You're rotating often enough that ceremony, approvals, and automation around key changes are worth standing up.
• Multiple services share one KEK and you want a central authority rather than the same secret copied into N environments.

If none of those is true today, a KEK in an encrypted environment variable is not the reckless choice; it's the proportionate one. It gives you the exact same "database dump is worthless" property, a smaller attack surface, one fewer thing to operate, and, in my case, a strictly shorter plaintext lifetime because it let me delete the cache that the network hop required. And because the whole thing sits behind an interface, "upgrade to Vault later" stays a cheap, boring change instead of a rewrite.

The meta-lesson is about proportion. Security is not a linear scale where more machinery is always more secure. Machinery has a threat model it's good against and a cost it always charges. Running a control whose benefit your deployment can't realize means you pay the cost (operational surface, new dependencies, more code to get wrong) with none of the protection. The reviewer's critical finding and my simplest possible fix pointed the same direction precisely because the complexity and the vulnerability shared a root cause. When that happens, it's not a trade-off. It's a gift. Take it.

---

Ten things I'd tell you to take away

1. The bar for "encrypted at rest" is "a stolen DB backup is useless." Envelope encryption clears it: the KEK lives outside the database, so ciphertext plus wrapped DEKs decrypt to nothing on their own.
2. A cache is a liability you accept to pay down a cost you chose. Before you cache sensitive material, ask whether you can delete the cost instead. Removing the network hop removed the cache, and the cache was the vulnerability.
3. Compare against the tool you'll actually deploy, not its brochure. Colocated Vault and an env-var KEK have an identical posture against a stolen backup and against host root. The differences are audit and ceremony: compliance properties, not confidentiality ones.
4. Bind ciphertext to its owner with AEAD associated data. A tenant:key binding context on both envelope layers, recomputed at decrypt and never stored, makes cross-tenant splicing attacks fail cryptographically, even if your row-level security is bypassed.
5. Make ciphertext self-describing. A local:v2: prefix that names the cipher and KEK version lets multiple implementations share one column and makes migrations zero-downtime.
6. Rotate KEKs without re-encrypting rows. Version the KEK, retain old versions for decrypt, let rows lazily rewrap on write. Never store the KEK anywhere near the data.
7. Mint identities before you insert when the row's ID is part of its binding context. A database-generated default silently breaks the crypto binding.
8. Fail at boot, not at runtime. Validate key material at startup (right length, versions consistent) and crash-loop with a message that names the misconfigured variable. Loud and early beats silent and data-dependent.
9. Remove dependencies, not options. Keep the heavyweight implementation compiled, tested, and selectable behind an interface. "Not at this scale" reverses the day you land an enterprise deal.
10. When the security fix and the simplicity fix are the same fix, you've found a shared root cause. Those are the best changes you'll ever ship. Go looking for them.

---

If you're building on top of your own model provider accounts and want a workbench that treats your keys like the money they are, that's exactly what I'm building at pcraft.io: BYOK, envelope-encrypted, no markup on your tokens. And if you want to argue with any of the crypto choices above, or you've found a hole in the binding-context reasoning, I want to hear it.

Build it. Encrypt it properly. Then delete the part you didn't need.

---

Have questions about this implementation? Email me at fcostoyaprograms@gmail.com.