This is the short version. I wrote the full story on the Zepto engineering blog: Everything was fine until Kubernetes said no more CPU.
We had a Python service on the hot path. Busy, slow, expensive. So we rewrote it in Go and cut latency by about 95 percent. Dashboards went green, we felt clever, we moved on.
It worked beautifully, till it didn’t. Traffic grew, latencies spiked, and every CPU graph we trusted stayed perfectly calm. That gap, between the graphs looking healthy and the service very much not, is where we spent the next few days. Here is what we learned.
Pod-level CPU averages lie. We were autoscaling on the average CPU across a whole pod. But a pod is not one container. Ours had the app asking for around 800m sitting next to sidecars like Vault asking for 20m. Average those together and the pod looks relaxed even while the app container is pinned and gasping.
CPU usage lies. Throttling doesn’t. The problem was never how much CPU we used. It was how often Kubernetes refused to give us any. Under the hood that refusal is the CFS quota: you get a slice of CPU time per period, and once it is spent everything waits, no matter how idle the node looks from outside.
Go was fighting a machine that didn’t exist. Go sets its thread count from the cores it sees on the host, not the CPU limit on your container. So the runtime happily scheduled goroutines across a dozen “cores” that Kubernetes was throttling down to half of one. Lots of threads, almost no CPU time, work piling up while the usage graph stayed flat.
The fixes were unglamorous, which is usually a good sign. Right-size requests and limits per container instead of per pod. Pin the Go runtime to the actual CPU limit so it stops fighting an imaginary machine. Put CFS throttled time on the dashboard as a first-look metric, not a last resort. And tighten circuit breaking at the edge with Envoy so a throttled dependency fails fast instead of dragging its callers down. Incidents from this class of problem dropped by around 40 percent once throttling was something we could actually see.
The real lesson was not about Go or quotas. It was about trusting the wrong signal. Utilization feels like health because it usually is, but in a world of limits a system can be starved and idle at the same time, and the graph you built your intuition on will keep telling you everything is fine.
The full write-up has the graphs, the dashboards, and the gory details.