[AINews] Megakernels are so dead and so back

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

Part of our Inference Engineering Masterclass pod yesterday involved a spicy discussion about Megakernels:megakernels are deadwhy are megakernels useful? you spend two months writing a kernel to save time on launch overhead and poor inter-kernel overlap. you had PDL but then people said it wasn't perfect, that you could still get some marginal gains due to straggler CTAs and therefore- wait, sorry, I forgot, Rubin fixes that (kernel two needs 10 CTAs and kernel one has seven finished and three straggling, kernel two launches seven of its CTAs). given a long enough timeline, it all evens out. no serious inference provider is using a 67k loc hand-fused forward pass kernel in production, and the teams doing that are doing so out of pure research. dead.The full discussion, for those who care to listen through:Ali: A fused kernel can’t save you. Like here with tensor parallelism, half the matrix is on one GPU and the other half is on another, and if I need the entire matrix in order to do like a nonlinear operation in the next step, which is, for instance, like if I’m doing attention, I need the softmax, or I need to do like exponentiation, I need to have the entire row. So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. So I have to make them communicate with each other, even if I had a fused kernel, because of the nonlinearities within each one. Also with like mega kernels, like honestly, I’m very bearish. It was a good research direction, and it seems like intuitively, …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://www.latent.space/p/ainews-megakernels-are-so-dead-and
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.