Scale before the spike: Predictive autoscaling for GPU workloads on Kubernetes

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

The 3 AM Call

We got paged one Tuesday morning. A critical production service had crashed under traffic—not gradually degraded, but crashed. Hundreds of pending pods. Users were seeing 15–20% error rates. The incident postmortem was brutal: reactive autoscaling had fired, but it was already too late.

The timeline looked like this:

06:00 – Traffic spike arrives

06:05 – HPA threshold crossed, scales up Deployment replicas

06:15 – New pods begin scheduling

06:45 – First GPU nodes finish provisioning, pods actually run

By 06:45, the spike was over. Customers had already hit errors. The system had tried to scale, but the physics of infrastructure didn’t cooperate.

The root cause wasn’t a bug—it was a mismatch between workload requirements and provisioning speed. Scaling CPU-only services takes minutes. Scaling GPU nodes takes 3–5x longer: firmware loads, drivers initialize, CUDA gets ready. Reactive HPA, by definition, waits for demand to appear before ordering capacity. For GPU workloads, that’s reactionary in the worst sense.

We realized that night: we needed to see the spike coming before it arrived.

The Insight: Prediction Changes Everything

We already had all the data we needed —Prometheus was collecting CPU, memory, latency, RPS, and NVIDIA GPU utilization continuously. A week of history sat in storage. The question wasn’t whether we could predict demand; it was whether we could predict it well enough to matter.

We decided to test a hypothesis: what if a Kubernetes controller running every 60 seconds could look at the past hour of metrics and forecast demand …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://www.cncf.io/blog/2026/08/28/scale-before-the-spike-predictive-autoscaling-for-gpu-workloads-on-kubernetes/
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.