What they're testing
Whether you understand process lifecycle under an orchestrator, and why handling SIGTERM alone isn't enough.
The short answer~30 seconds
Because removing the pod from the endpoint list and sending SIGTERM happen IN PARALLEL, not in sequence. For a few hundred milliseconds the load balancer still routes to a process that has started shutting down. The fix: on SIGTERM, fail the readiness probe IMMEDIATELY, sleep a few seconds so every routing component catches up, and only then stop accepting new connections and drain in-flight requests within a bounded window.
The long answer
That sleep is the most counter-intuitive part and the one most often omitted. It looks like a hack, but it reflects reality: endpoint changes propagate asynchronously through several layers — kube-proxy on each node, the ingress controller, sometimes the cloud provider's load balancer. There's no way to know they've all caught up, so you wait long enough that the remaining probability is small. A preStop hook with sleep 5 is the usual form.
The second part is the deadline. The orchestrator gives you terminationGracePeriodSeconds (30 by default) and then SIGKILLs. So sleep plus drain must be LESS than that number, or you're killed mid-request — exactly what you were avoiding. Systems with long requests (uploads, exports) must raise the grace period accordingly, and that's a decision to state rather than leave at the default.
There's another 502 source often confused with this one: keep-alive. If the process closes an idle keep-alive connection at the moment the load balancer dispatches a request onto it, that request dies although both sides behaved correctly. The mitigation is setting the server's idle timeout HIGHER than the load balancer's, so the balancer is always the side that closes. This produces scattered 502s even without deploys, and is very hard to find if you don't know to look.
The shutdown order, expressed as configuration
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5"] # để endpoint lan truyền xong
terminationGracePeriodSeconds: 45 # > sleep + thời gian drain
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 2 # phát hiện nhanh khi /readyz đỏ
livenessProbe:
httpGet: { path: /healthz, port: 8080 } # KHÔNG kiểm tra dependency ở đâyWhat they'll ask next
?How do readiness and liveness differ?
Readiness decides whether traffic arrives; liveness decides whether the container is restarted. The classic trap is checking the database in liveness: one database hiccup restarts every pod at once, turning a small incident into a total one. Dependencies belong in readiness.
?What about queue workers?
Same principle, different mechanism: on SIGTERM stop accepting new jobs, finish the current one, then exit. A job killed mid-flight must return to the queue — which means acking after processing, not on receipt.
These lose points
- Catching SIGTERM and calling
process.exit()immediately. You just cut off the requests in flight. - Not knowing the endpoint update and SIGTERM are concurrent. That's the root of the whole thing.