Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

This post is a collaboration between AWS, NVIDIA and Heidi. Reducing automatic speech recognition (ASR) inference costs on Amazon Elastic Compute Cloud (Amazon EC2) becomes critical when GPU utilization per request is low but latency requirements are strict. A single ASR inference request typically uses only 15–20 percent of a GPU’s compute capacity, yet the default time-slicing behavior in NVIDIA CUDA® forces sequential access, leaving 80 percent of the hardware idle. Heidi Health is an AI Care Partner that processes over 2.4 million clinical consultations per week across 190 countries. To sustain sub-second transcription latency at peak traffic, this inefficiency forces the company to run 16 GPU instances. In a previous post, you learned how to fine-tune a Nemotron speech model, NVIDIA Parakeet TDT 0.6B V2 for clinical speech recognition. In this post, we focus on what comes after fine-tuning: serving that model efficiently. We demonstrate how NVIDIA CUDA Multi-Process Service (MPS), combined with NVIDIA Triton Inference Server on Amazon EC2 GPU instances, reduces GPU infrastructure requirements by 75 percent (from 16 instances to 4). This setup maintains sub-second latency at 92.1 requests per second (RPS) per GPU. Solution overview This section covers the following: The GPU utilization challenge. The three available sharing mechanisms. Model-level optimizations with ONNX and TensorRT. Request scheduling with Triton. How these components integrate on Amazon EC2. The GPU utilization problem A single ASR inference request on the Parakeet TDT 0.6B V2 …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://aws.amazon.com/blogs/machine-learning/reduce-asr-inference-costs-by-75-with-nvidia-mps-on-amazon-ec2/
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.