I was on a call last month with an engineering director who told me, with a straight face, that their Kubernetes cost allocation was "solved."
They had Kubecost. They had dashboards. Every namespace had an owner. Every pod had resource requests. On paper, perfect.
Their AWS bill was $1.2M/month. Their Kubecost allocation summed to $740K.
Almost half a million dollars a month was falling into a black hole labeled "cluster overhead" — and nobody was asking where it went.
This isn't unusual. This is the norm. And KubeCon EU happening this week in Amsterdam is the perfect reminder that while the Kubernetes scheduling layer keeps getting more sophisticated (DRA graduating to GA, NVIDIA donating their GPU driver to CNCF, Karpenter doing Spot-to-Spot consolidation), the cost layer is still stuck in 2022.
Every Kubernetes cost tool tells you the same three lies. Not because they're bad tools — because the problem is harder than anyone admits.
Most allocation tools calculate cost based on resource requests. Pod requests 4 CPU cores on a node that costs $X/hour, tool divides accordingly.
Problem: nobody requests what they actually use.
I've seen teams where average CPU utilization is 12% of requested. Memory is slightly better — maybe 35%. That means 65-88% of what you're "allocated" is wasted capacity that the tool dutifully attributes to your team's cost center.
Your engineering leads see their dashboard showing $180K/month and think "that's the cost of running our services." The real cost of their actual compute consumption is closer to $40K. The other $140K is the tax they're paying for requesting 8x what they need because someone got paged once when a pod got OOMKilled and said "just double the memory request."
System daemonsets. Kube-system pods. Istio sidecars. Monitoring agents. CoreDNS. The ingress controller.
These run on every node, consume real resources, and belong to... nobody? Everybody? The platform team?
Most tools either ignore them entirely (hello, that $460K gap) or split them evenly across namespaces. Neither is right.
An evenly-split model charges your 2-pod dev namespace the same platform overhead as the 200-pod production namespace that's actually generating the load those shared services exist to handle. That's not allocation — that's a participation trophy for infrastructure.
The teams doing this right are splitting shared costs proportionally by actual resource consumption, not headcount or namespace count. It's more work. It's also the difference between useful data and expensive fiction.
Here's where it gets really ugly, especially with GPUs.
AWS doesn't bill you for CPU cores or GB of memory. It bills you for instances. A c6i.8xlarge costs the same whether you're using 2% or 95% of it. Karpenter helps (it really does — bin-packing, consolidation, right-sizing node groups), but you still pay for the whole node.
GPU instances make this 10x worse. A p5.48xlarge costs roughly $98/hour. If you're running inference workloads that use 3 of the 8 H100 GPUs, you're paying $98/hour for $37/hour of actual GPU use. And here's the fun part: some GPU instance types (P5, G6) report a GPU count of zero to the AWS API. Your cost tool might not even see the GPUs exist.
So you've got a $70K/month GPU cluster where the allocation tool says "I see compute nodes with zero GPUs costing a lot of money" and shrugs.
After looking at this problem across dozens of environments, here's what separates teams who have real cost attribution from teams who have expensive dashboards.
Stop allocating on requests. Start allocating on actual usage with a request-based ceiling.
If a pod requests 8GB of memory and uses 2GB, the cost should reflect something closer to 2GB plus a fair share of the overhead of reserving 8GB. Not the full 8GB.
This is harder to implement, but Prometheus + custom allocation logic gets you there. Some teams run a nightly job that pulls container_memory_working_set_bytes and container_cpu_usage_seconds_total, maps to node costs, and produces a daily cost-per-namespace report that's actually grounded in reality.
Yes, it's custom. Yes, it's worth it. The alternative is lying to your finance team with vendor-provided numbers.
Create a "platform tax" that's calculated proportionally and shown explicitly.
Every team should see two numbers: their direct workload cost and their share of platform overhead. The overhead ratio should be calculated monthly based on actual shared resource consumption relative to each team's footprint.
When the platform tax is visible, something magical happens: platform teams start optimizing it. I've seen teams cut Istio sidecar resource consumption by 40% once the cost was visible and attributed back to the platform team's budget.
For GPU workloads, instance-level billing is useless. You need workload-level time tracking.
How long did this training job run? On how many GPUs? At what utilization? That's your cost unit.
Teams building ML platforms are starting to instrument this: GPU-seconds per job, with idle time between jobs attributed to the scheduling layer (which is, again, the platform team's problem to optimize).
If more than 15% of your cluster cost is in an unattributed overhead bucket, your allocation is broken. Full stop.
That overhead is real money going to real infrastructure. Break it down: what percentage is system pods? What percentage is unused capacity from poor bin-packing? What percentage is the gap between requests and usage?
Each of those has a different owner and a different fix. Lumping them together into "overhead" is how half a million dollars a month disappears.
Here's what I keep saying and what KubeCon EU is proving this week: your platform team is your FinOps team, whether they know it or not.
The people setting default resource requests, choosing node instance types, configuring autoscalers, and deciding whether to use Spot — they're making 70-80% of your cloud cost decisions. If your FinOps practice doesn't have a direct line to platform engineering, you're optimizing at the margins while the big decisions happen without cost context.
The teams winning at K8s cost aren't the ones with the fanciest dashboards. They're the ones where cost data shows up in the deploy pipeline. Where resource requests get validated against actual usage before hitting production. Where the platform team has a cost efficiency target that's just as real as their uptime SLA.
Kubernetes cost attribution isn't a tooling problem. It's a methodology problem.
The tools will get better — OpenCost is doing genuinely good work, Kubecost keeps improving, and the FOCUS spec will eventually standardize some of this. But even the best tool can't fix fundamentally wrong assumptions about what "cost" means in a shared, overprovisioned, multi-tenant cluster.
Start with the truth: you don't know what your Kubernetes workloads actually cost. That's not a failure — it's the starting point for actually figuring it out.
Want to know where your Kubernetes spend is actually going? Our free Cloud Waste Audit includes a full cluster cost breakdown — actual utilization vs. requests, unattributed overhead, and the three highest-impact optimizations. 24 hours, zero engineering time.
