I already wrote about the AWS incident that pushed us onto Kubernetes. That post was about the infrastructure side: the account lockout, the blast radius, the move to a self-managed k3s cluster.

This one is about the code, because moving our Trackpac API off Lambda didn’t just change where it ran. It changed how it had to behave, and that surfaced a bunch of things that had been quietly “fine” for years.

Lambda’s favourite trick: never sticking around

A Lambda execution environment isn’t necessarily thrown away the moment a request finishes. AWS keeps it warm and reuses it for the next invocation when it can, so global state does sometimes survive between calls. What doesn’t happen is concurrency within that environment: it handles exactly one invocation at a time, and if a burst of traffic comes in, Lambda spins up more environments rather than queuing everything behind one shared process. Combine that with environments being disposable by design, and code gets a lot less time, and a lot less concurrent pressure, to reveal whatever is wrong with it. For genuinely spiky workloads that’s a real advantage: you don’t pay for idle compute, and no single environment has to survive the kind of sustained, concurrent load a long-running pod does. It’s a good tool for the job it’s good at.

The API itself is a FastAPI service. FastAPI happily runs both sync and async route handlers side by side, which is exactly the kind of flexibility that lets sync code sit there unnoticed for a long time. When the Trackpac API was originally built, writing everything synchronously was a completely reasonable call given the constraints: one invocation per environment at a time. Blocking calls, blocking DB queries, requests instead of an async HTTP client, none of that cost much when nothing was competing with it for the same process.

Move that same code into a container that’s supposed to stay up for weeks under real concurrent traffic, and those same decisions look different. We already had CloudWatch metrics and logs on the old stack, so we knew there were rough edges somewhere. What we hadn’t seen yet was exactly which ones would turn into an incident once a single process had to carry all of it at once, for weeks on end.

The rewrite

The biggest single change was rewriting the synchronous parts of the API to run properly async: psycopg3 (the psycopg[binary] package) instead of the old sync driver, httpx instead of requests, most of the route handlers touched in one sitting.

On a long-running server, a stack of blocking, synchronous route handlers is a liability in a way it never was inside a disposable function. Each one ties up a worker thread for its full duration. FastAPI only hands out so many threads, depending on how the server is configured. Enough slow, blocking calls in flight at once and other work needing that thread pool queues up behind them: worker starvation. That’s a much harder failure mode to hit on Lambda, where concurrency mostly comes from spinning up more execution environments rather than sharing one process across everything happening at once.

Predictably, a rewrite that size didn’t land clean the first time. A string of “fix: async await function_x” and “fix: properly await path_y” commits followed within days, because a missing await in Python doesn’t crash, it silently hands you a coroutine object instead of the value you wanted. Quiet bugs are the worst kind… luckily we get notified when they happen!

The readiness probe that kept getting queued out

Kubernetes hits a simple health endpoint to decide whether a pod is healthy enough to receive traffic. Ours looked harmless:

@app.get("/health")
def health():
    return {"status": "ok"}

The endpoint itself was fine, trivially fast in isolation. The problem was everything else. All those blocking synchronous calls elsewhere in the app were saturating FastAPI’s thread pool. This endpoint, still sync at the time, had to wait in the same queue to get a worker turn, and under load it sometimes didn’t get one in time. A failed readiness check removes the pod from traffic. A failed liveness check eventually restarts it. We had managed to make being busy look exactly like being broken.

Making the handler async fixed the symptom directly. It now runs straight on the event loop instead of waiting on the same thread pool as the blocking work:

@app.get("/health")
async def health():
    return {"status": "ok"}

But the real fix was upstream: clearing out the blocking calls that were starving the workers in the first place. Lambda never needed a readiness probe to begin with. There was no long-lived pod to ask “are you okay”, so this particular failure mode didn’t exist before we migrated.

Connections that used to be somebody else’s problem

A Lambda invocation either opens what it needs fresh, or on a warm environment reuses what a previous invocation already had open. Either way, that individual environment isn’t under much concurrent pressure, since it handles one invocation at a time rather than juggling hundreds of requests inside one process. Connection limits were rarely our problem for that reason.

A long-running API keeps connections open on purpose, for performance, which also means it can quietly accumulate more of them than intended. We increased the Postgres pool size to match real concurrent usage instead of the older, more conservative sizing. This is also where that warm-reuse idea from earlier actually paid off. The client used to validate incoming JWTs against our auth provider’s public signing keys was being constructed fresh on every single request. That client already has its own JWKS cache, but rebuilding it on every request threw the cache away before it could do anything useful. We moved it to a single module-level instance, built once and reused for the life of the process, so the caching it already knew how to do could actually happen.

Memory that finally had a chance to hurt

This one didn’t come from old code, it’s a bug we introduced while fixing a different problem. We’ve stored device frame history in S3-compatible object storage since 2023 (we wrote about the original setup on Trackpac’s Medium; these days that’s Ceph RGW inside the cluster rather than AWS S3 itself). Fetching that frame data uses boto3, which is still a synchronous library. aioboto3 is on the list of things to look at, but we haven’t needed it yet.

Under concurrent load, a burst of large frame history requests meant a burst of blocking boto3 calls, all competing for the same limited thread pool. That’s the exact worker starvation problem again, except this time each stuck request was also holding a chunk of frame data in memory while it waited. Enough of those piling up at once was a genuine memory spike. The kubelet doesn’t wait around for good intentions, it OOMKilled the pod.

The fix was a small in-memory LRU frame cache so repeat requests for the same device data don’t have to hit object storage again. If it ever outgrows an in-process cache, moving it to something like Redis or Valkey is the obvious next step. For the current traffic levels, in-memory is still plenty.

The part that actually matters: it got faster

This kind of migration doesn’t usually pay you back directly, but this one did. Running warm, taking Lambda cold starts out of the request path, doing IO properly asynchronously instead of blocking workers, and fixing a handful of N+1 queries with proper eager loading along the way added up to a real, noticeable drop in latency. None of this is from a formal benchmark, just what we’ve watched change in our own metrics, but roughly:

Call	Before	After
Organisation GET	~120ms	~40ms
Frame history	1-3s	~300-600ms miss, ~100-200ms hit

Call it 50-90% faster depending on the endpoint.

Why serverless is so good at hiding this stuff

None of these were exotic: a missing await, a client rebuilt on every call, blocking calls on a hot path. Ordinary mistakes. What kept them invisible for years had less to do with the code and more to do with the platform. An execution environment that handles one request at a time and can disappear between invocations has fewer opportunities to accumulate the state, connections, or backlog that turns a small mistake into an incident.

A pod that’s expected to stay up for weeks under continuous concurrent load doesn’t get that grace. Every one of these had been sitting there the whole time. Kubernetes didn’t introduce the bugs, it just gave them enough rope.