Kubernetes Cost Optimization¶

Overview¶

Kubernetes cost is mostly node cost: you pay for the machines, and the cluster decides how well they are packed. This guide covers the levers that matter, from the Kubernetes, Karpenter and OpenCost documentation. It gives no prices; use your cloud's pricing pages and the per-cloud guides: AWS, Azure, GCP, and FinOps principles.

The Kubernetes docs put the goal plainly: ensure all schedulable pods run "whilst keeping the cost of the cluster as low as possible", and treat overall node utilization as a proxy for how cost-effective the cluster is. They add that correctly setting Pod resource requests is as important to cost-effectiveness as optimizing node utilization.

📖 Kubernetes - Node autoscaling - provisioning, consolidation, and the cost framing.

flowchart TB
    A[1. See the spend: allocate cost by namespace, team, workload] --> B[2. Right-size requests]
    B --> C[3. Scale pods to demand: HPA]
    C --> D[4. Scale nodes to pods: Cluster Autoscaler or Karpenter]
    D --> E[5. Pick cheaper capacity and schedule off-hours]
    E --> F[6. Govern: quotas, defaults, review]
    F --> A

Cost work is a loop: measure, shrink requests, scale pods, scale nodes, then govern.

1. See the spend¶

You cannot cut what you cannot attribute. OpenCost describes itself as "a vendor-neutral open source project for measuring and allocating cloud infrastructure and container costs", with real-time monitoring, showback and chargeback, and optional cloud billing integration for more accurate results.

• Allocate by namespace, label, team and workload.
• Standardize labels (team, service, environment) and enforce them at admission. Unlabelled spend is shared overhead nobody owns.
• Watch idle cost: capacity you pay for but no request uses.

Kubecost is a commercial product built around the same allocation ideas. This guide did not fetch Kubecost's documentation, so check its current features and licensing on its own site before comparing.

📖 OpenCost - Prometheus - "Prometheus is a prerequisite for OpenCost installation." 📖 OpenCost - "OpenCost is a Cloud Native Computing Foundation incubating project."

2. Right-size requests and limits¶

The scheduler places Pods using requests. The kubelet enforces limits. Per the docs, the kubelet also reserves at least the request amount for that container.

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    memory: "512Mi"

Facts to rely on, from the Kubernetes docs: - CPU limits are enforced by throttling; memory limits by OOM kills, reactively. - If you set a limit but no request, Kubernetes uses the limit as the request. - QoS classes: Guaranteed (requests equal limits for CPU and memory in every container), Burstable, BestEffort. Under node pressure BestEffort is evicted first.

How to size: observe real usage over a representative period (including peaks), set requests near typical usage with headroom, and revisit regularly. Check live usage with kubectl top pod -n NAMESPACE (needs Metrics Server).

Vertical Pod Autoscaler (VPA)¶

VPA adjusts requests to match actual usage. It is a custom resource that "must be installed separately", unlike HPA. Recommendations live in .status.recommendation (target, lower bound, upper bound). The updateMode values include Off (recommendations only), Initial (set requests only at Pod creation), Recreate, InPlaceOrRecreate and InPlace. Starting with Off is a safe way to get right-sizing data without evictions. Eviction-based updates recreate Pods and respect PodDisruptionBudgets.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"

Per the docs, Recreate evicts Pods whose requests differ significantly from the recommendation and the VPA admission controller applies the new requests to the replacement. InPlaceOrRecreate resizes without a restart when possible and otherwise falls back to eviction. InPlace is alpha (VPA 1.7.0, Kubernetes 1.33 or later, feature gates on). Auto is deprecated since VPA 1.4.0. Do not run VPA and HPA on the same metric for the same workload without reading the docs on how they interact (the Kubernetes VPA page has no guidance on this).

📖 Kubernetes - Resource management for Pods and containers - requests, limits, enforcement. 📖 Kubernetes - Pod QoS classes - class definitions. 📖 Kubernetes - Vertical Pod autoscaling - update modes and limitations.

3. Scale Pods to demand (HPA)¶

The HorizontalPodAutoscaler runs a control loop (default sync period 15 seconds) and uses the ratio of current to desired metric:

desiredReplicas = ceil[ currentReplicas * ( currentMetricValue / desiredMetricValue ) ]

For CPU utilization targets, the controller calculates utilization as a percentage of the container requests. If a container has no request, CPU utilization is undefined and the autoscaler takes no action for that metric. The metrics.k8s.io API normally comes from Metrics Server, which you install separately.

Cost link: inflated requests make HPA scale out later or on misleading percentages, so fix requests first.

kubectl autoscale deployment my-app --cpu=50% --min=2 --max=10

(--cpu takes a percentage of requested CPU or a quantity such as 500m; --max is required. Older kubectl releases used --cpu-percent, which the current reference does not list.)

📖 kubectl autoscale - flags and examples.

📖 Kubernetes - Horizontal Pod Autoscaling - algorithm, requests and metrics.

4. Scale nodes to Pods¶

Node autoscalers add nodes when Pods cannot be scheduled and consolidate underused nodes. The Kubernetes docs name Cluster Autoscaler and Karpenter as the two node autoscalers sponsored by SIG Autoscaling. They do not look at real Pod usage, only at requests and constraints, which is why the docs recommend pairing them with workload autoscaling and right-sized requests.

Karpenter consolidation¶

Karpenter's disruption settings (NodePool spec.disruption):

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
    - nodes: "10%"
    - nodes: "0"
      schedule: "0 9 * * mon-fri"
      duration: 8h
      reasons:
      - "Underutilized"

Budgets cap how many nodes can be disrupted at once (default one budget of nodes: 10%). A schedule is a cron expression in UTC (no time zones) and must be paired with duration. The example above blocks underutilized-node consolidation for 8 hours on weekday mornings; adapt it to your own traffic. (The cron value is an illustration, not taken from the docs.)

Protect work that must not be interrupted with the karpenter.sh/do-not-disrupt pod annotation ("true" or a duration like "30m"). Per the docs it blocks consolidation, and drift only when the NodeClaim has no terminationGracePeriod; it does not block expiration, interruption, manual deletion or node repair. Overusing it prevents consolidation and raises cost.

Consolidation evicts Pods, so set PodDisruptionBudgets and enough replicas, or savings turn into outages.

📖 Karpenter - Disruption - consolidation, expiration, drift, budgets.

Cluster Autoscaler¶

Per the Cluster Autoscaler FAQ, a node becomes a scale-down candidate when the sum of its pods' CPU and memory requests is below --scale-down-utilization-threshold (default 0.5), its pods can move elsewhere, and it lacks the cluster-autoscaler.kubernetes.io/scale-down-disabled annotation. A node unneeded for --scale-down-unneeded-time (default 10m) is removed. It measures requests, not real usage.

📖 Cluster Autoscaler FAQ - scale-down conditions and defaults.

5. Cheaper capacity and schedules¶

These are options to evaluate with your cloud's current documentation; this guide has not fetched the per-cloud pages for them. - Interruptible capacity (AWS Spot, Azure Spot, GCP Spot VMs) for fault-tolerant, restartable work. Needs graceful shutdown handling and diversified instance types. - Committed discounts (Savings Plans, reservations, committed use) for the steady baseline. See the per-cloud guides above. - Architecture choice: ARM-based instances where your images support them. - Off-hours shutdown of dev and test: scale Deployments to zero or scale node groups down on a schedule. - Namespace and cluster sprawl: each idle cluster still pays for its control plane and baseline nodes. Delete what nobody uses.

6. Govern with quotas and defaults¶

ResourceQuota caps aggregate consumption in a namespace so one team cannot take more than its share. A quota on cpu or memory requires every new Pod to specify requests or limits, so pair it with a LimitRange that supplies defaults.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi

kubectl describe resourcequota compute-quota -n team-a

📖 Kubernetes - Resource Quotas - quota names, behavior, example.

Other cost drivers outside the Pods¶

• Load balancers: one per Service of type LoadBalancer adds up; consolidate behind an Ingress or Gateway.
• Persistent volumes: unattached volumes and old snapshots keep billing.
• Data transfer: cross-zone and egress traffic is billed by the cloud. Check your provider's pricing page.
• Logs and metrics: high-cardinality metrics and debug logs are a hidden cost. See observability basics.
• Control plane fees for managed clusters: see each cloud's pricing page.

Quick wins checklist¶

1. Install cost allocation (OpenCost or your cloud's equivalent) and label by team.
2. Run VPA in Off mode and compare recommendations to current requests.
3. Fix workloads with no requests.
4. Turn on node consolidation with budgets and PDBs.
5. Scale non-production down outside working hours.
6. Add ResourceQuota and LimitRange to shared namespaces.
7. Review the biggest unlabeled or idle line items monthly.

Related¶

• FinOps principles
• Kubernetes troubleshooting
• Kubernetes and cloud-native interview guide
• FinOps topic index