Kubernetes Container Memory: Requests, Limits, and OOMKilled

Prerequisites: kubernetes

Kubernetes gives a container two different memory numbers:

resources:
  requests:
    memory: 512Mi
  limits:
    memory: 1Gi

The request influences scheduling. The scheduler uses it when deciding whether a node has room for the Pod. The limit is enforced at runtime through Linux control groups. They are not a minimum and maximum allocation handed to the process at startup.

The Kubernetes resource-management documentation describes the distinction: requests participate in placement, while exceeding an enforced memory limit can activate the kernel out-of-memory path and terminate the process.

What OOMKilled proves

Inspect the current and previous container state:

kubectl -n logging get pod loki-0 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="loki")].lastState}'
 
kubectl -n logging describe pod loki-0

A terminated state with reason OOMKilled and exit code 137 proves the container crossed an enforced memory boundary and was killed. It does not identify which internal operation consumed the memory.

That distinction separates symptom from cause:

Raising a limit can stop the symptom while leaving the cause unexplained. It becomes a sound change only when measured recovery and steady-state behavior justify the new budget.

Why requests still matter

A 512Mi request does not reserve a private block of 512 MiB for Loki. It tells the scheduler to account for that much capacity when placing the Pod. The process can use less, and it can use more until it reaches the limit or the node experiences broader pressure.

For a stateful service, the request should represent a realistic normal operating floor. A request far below normal use lets Kubernetes pack too much onto the node. A request equal to a rare recovery peak can waste scarce homelab capacity. Devata therefore uses a 512 MiB request and a 1 GiB limit after observing about 181 MiB at final steady state and roughly 400 MiB during the recovered backlog path.

Application ceilings belong below container limits

An application-level memory threshold and a container limit solve different problems:

application threshold: begin shedding, flushing, or slowing work
container limit: kernel-enforced last boundary

If an application waits until 4 GB before applying replay backpressure but its container is limited to 384 MiB, the application never reaches its own safety behavior. The kernel kills it first.

The useful relationship is:

steady state < application recovery ceiling plus overhead < container limit < node capacity

The middle term is not simple arithmetic. Measure it. Compression, concurrency, garbage collection, caches, and storage latency all affect peak usage.

Measure more than the current point

Useful signals include:

container_memory_working_set_bytes{
  namespace="logging",
  pod="loki-0",
  container="loki"
}

kube_pod_container_status_restarts_total{
  namespace="logging",
  pod="loki-0",
  container="loki"
}

kube_pod_container_status_last_terminated_reason{
  namespace="logging",
  pod="loki-0",
  container="loki",
  reason="OOMKilled"
}

Current memory alone can look normal between crashes. Correlate the time series with restart state and application logs from the seconds before termination.