Kubernetes Notes from a Payroll Platform on EKS

These are condensed notes from running a payroll-shaped workload on AWS EKS. No heroics — just the incidents that forced better defaults.

OOMKilled cascade

One PDF-heavy service leaked under large documents. Pods restarted, neighbors thrashed, the namespace felt haunted.

Change: treat the gap between request and limit as a signal. When usage hugs the limit, you are already in the blast radius. Profile the burst path; do not only "raise the limit and pray."

CoreDNS friction

Random multi-second timeouts. Not every request — enough to poison trust.

Change: scale CoreDNS, NodeLocal DNSCache, tighten ndots so every call does not search the galaxy. DNS is part of your SLO whether you named it or not.

Rolling update with a silent contract break

A serialization change passed CI and failed the fleet.

Change: canary + automatic rollback, contract tests between services, shadow traffic before full promote.

Kubernetes gives levers. Strategy is still on you.