Running Whisper on GPU in Rust with ONNX and DirectML
Local AI is experiencing a golden age. Running multi-billion parameter language models, vision decoders, and automatic speech recognition (ASR) pipelines locally on consumer hardware has shifted from an exotic science experiment to a standard feature in modern desktop software.
If you are developing for Windows in Rust, however, there is a painful rite of passage awaiting you the moment you attempt to turn on GPU acceleration: the C++ wrapper and CUDA build nightmare.
Whether you are binding to whisper.cpp, llama.cpp, or stable-diffusion.cpp, the story is always the same. What begins as a clean cargo build rapidly spirals into a labyrinth of external CMake scripts, MSVC toolchain conflicts, Clang environment flags (LIBCLANG_PATH), and gigabytes of NVIDIA CUDA SDKs. And even if you conquer your local build environment, you face an insurmountable distribution wall: your end-users cannot be expected to install 3 GB of CUDA runtimes, and anyone rocking an AMD Radeon or Intel Arc GPU is abruptly locked out.
It does not have to be this way.
By bypassing the fragile C++ FFI layers and coupling ONNX Runtime (ort) with Microsoft's DirectML execution provider, we can achieve high-performance, vendor-agnostic DirectX 12 GPU acceleration in pure Rust. Here is how it works, the architectural pipeline for models like Whisper, and the hard-earned production lessons you only uncover after shipping.
1. The Desktop AI Dilemma: The C++ Toolchain Trap
When building native desktop applications - such as audio/video recorders, local voice assistants, media transcribers, or any others - shipping AI inference requires three non-negotiables:
- Zero-friction installation: The user runs an installer or unzips a binary; the feature must just work.
- Deterministic compilation: The continuous integration (CI) pipeline and developer machines shouldn't break when a path variable moves.
- Universal hardware acceleration: Hardware is diverse. Alienating AMD, Intel, and Qualcomm users is unacceptable.
The dominant approach in the open-source ecosystem has been writing Rust FFI wrappers around popular C/C++ inference engines (such as whisper-rs wrapping whisper.cpp). While these libraries are heroic technical feats, packaging them on Windows exposes the fundamental friction between the Rust and C++ ecosystems.
The Build Matrix from Hell
To compile a native C++ ML engine with CUDA support on Windows, your build instructions often resemble this:
1. Install Visual Studio Build Tools with C++ and Clang components.
2. Install CMake 3.26+ and verify it is in your system PATH.
3. Download and install the NVIDIA CUDA Toolkit (3.5 GB).
4. Run `where.exe clang` and configure `LIBCLANG_PATH` pointing to the exact MSVC Clang bin.
5. In your shell, restart environment variables and invoke `cargo build --features cuda`.
If the user has MinGW/MSYS2 in their PATH, CMake might pick up gcc instead of cl.exe. If LIBCLANG_PATH points to an x86 directory instead of x64, bindgen explodes with hundreds of missing type definitions.
Worse, CUDA is proprietary to NVIDIA. If your desktop app has users running AMD Radeon graphics, Intel Iris Xe / Arc, or modern Qualcomm Snapdragon X Elite chips, CUDA is useless. You are either forced to maintain separate OpenCL / Vulkan / ROCm backends through convoluted C++ build flags, or condemn those users to slow CPU threads that pin all cores and drain laptop batteries.
2. Enter DirectML: True Vendor-Agnostic GPU Acceleration
Microsoft introduced DirectML as a low-level, high-performance DirectX 12 library specifically designed for machine learning. Instead of writing custom compute shaders for every linear algebra operation or binding to vendor-specific APIs, DirectML abstracts tensor operations across the entire DirectX 12 ecosystem.
graph TD
A["Rust Application Logic"] --> B["ort (ONNX Runtime Rust Bindings)"]
B --> C["DirectML Execution Provider"]
C --> D["Direct3D 12 Compute"]
D --> E["NVIDIA GeForce"]
D --> F["AMD Radeon"]
D --> G["Intel Arc / Iris"]Why DirectML Wins on Windows
- Universal Hardware Support: Any GPU with DirectX 12 Feature Level 11_0 or higher is supported out of the box. This includes NVIDIA, AMD, Intel, and Qualcomm Adreno/NPUs.
- Zero Driver/SDK Installs for Users: DirectML communicates directly with standard Windows WDDM display drivers. End-users do not need to install CUDA, cuDNN, or developer toolkits.
- Trivial Redistribution: You don't need a sprawling C++ compiler toolchain. You simply bundle two DLLs alongside your Rust application binary:
onnxruntime.dllandDirectML.dll.
3. The Rust Setup: ort + DirectML
In the Rust ecosystem, the ort crate (v2.0+) provides safe, ergonomic, and blazingly fast bindings to ONNX Runtime.
Cargo.toml Dependencies
[dependencies]
ort = { version = "2.0.0-rc.13", features = ["directml"] }
ndarray = "0.17"
tokenizers = "0.23"
anyhow = "1.0"
tracing = "0.1"
When building with the directml feature, ort links against the ONNX Runtime DirectML binaries. In production, you place onnxruntime.dll and DirectML.dll right next to your final executable in target/release/ or your installer package.
The Dual-GPU Trap: Adapter Enumeration
One of the most insidious bugs encountered when deploying DirectML on Windows laptops is the default adapter selection.
Most modern laptops have two GPUs: 1. An integrated GPU (iGPU) built into the CPU (e.g., Intel UHD Graphics or AMD Radeon 780M). 2. A discrete GPU (dGPU) designed for heavy compute (e.g., NVIDIA RTX 4070).
By default, DirectML queries the DXGI adapter list and selects device index 0. On many Windows laptops, device 0 is the power-saving integrated GPU! If you initialize your session blindly, your app will run on the weak iGPU, causing slow inference and heating up the CPU die, while the high-powered discrete GPU sits completely idle.
To prevent this, explicitly configure the DirectMLExecutionProvider in Rust:
use anyhow::{anyhow, Result};
use ort::execution_providers::DirectMLExecutionProvider;
use ort::session::Session;
use std::path::Path;
use tracing::info;
pub fn create_directml_session(model_path: &Path, preferred_device_id: i32) -> Result<Session> {
info!("Initializing DirectML Execution Provider on GPU device {}", preferred_device_id);
// Explicitly target the discrete adapter ID (0 or 1 depending on DXGI enum)
let directml_ep = DirectMLExecutionProvider::default()
.with_device_id(preferred_device_id);
let session = Session::builder()?
.with_execution_providers([directml_ep])?
.commit_from_file(model_path)
.map_err(|e| anyhow!("Failed to load ONNX model via DirectML: {:?}", e))?;
info!("DirectML session successfully initialized on GPU");
Ok(session)
}
4. The End-to-End Whisper Pipeline in Rust
Whisper is an encoder-decoder transformer architecture. Unlike standard text embeddings, audio models require strict mathematical preprocessing before any tensor reaches the GPU.
graph TD
A["Raw Audio: MP4 / WAV / MKV"] -->|"Resampled via ffmpeg-next / hound"| B["16 kHz Mono Float32 PCM"]
B -->|"FFT: Hann window, 400 frame, 160 hop"| C["Log-Mel Spectrogram (80 bins)"]
C -->|"30-second fixed window"| D["ndarray::Array3 [1, 80, 3000]"]
D --> E["ort::Session (DirectML GPU)"]
E --> F["Token IDs"]
F -->|"tokenizers BPE Decode"| G["Final Transcript Text"]Step 1: Preprocessing & Mel-Spectrogram Extraction
Whisper expects audio formatted strictly as 16,000 Hz, single-channel (mono), 32-bit floating-point PCM between [-1.0, 1.0].
From these raw samples, we compute an 80-channel (or 128-channel for Whisper large-v3) log-mel spectrogram using:
- A window size of 400 samples (25 ms) with a Hann window function.
- A hop length of 160 samples (10 ms).
- Triangular mel filterbank matrices mapping the FFT bins onto auditory frequencies.
- Logarithmic scaling: \(S = \log_{10}(\max(\text{mel_spectrum}, 10^{-5}))\).
The result is shaped into an ndarray::Array3<f32> with dimensions [batch_size, n_mels, n_frames]—specifically [1, 80, 3000] for a 30-second audio slice.
Step 2: Feeding DirectML Tensors in Rust
With the ort crate, converting an ndarray slice into an ONNX GPU tensor and executing the model is clean and expressive:
use anyhow::Result;
use ndarray::Array3;
use ort::session::Session;
use ort::value::Tensor;
pub fn run_whisper_inference(
session: &mut Session,
mel_spectrogram: &Array3<f32>,
) -> Result<Vec<i64>> {
// 1. Create a zero-copy tensor view from the contiguous ndarray buffer
let input_tensor = Tensor::from_array(session.allocator(), mel_spectrogram)?;
// 2. Dispatch inference through DirectML to the DirectX 12 hardware queue
let outputs = session.run(ort::inputs![
"input_features" => input_tensor
]?)?;
// 3. Extract the generated token sequences from the output graph
let output_tensor = outputs["sequences"].extract_tensor::<i64>()?;
let token_ids: Vec<i64> = output_tensor.view().iter().copied().collect();
Ok(token_ids)
}
Step 3: Decoding with Pure Rust Tokenizers
Instead of shelling out to Python or linking C++ text decoders, we use Hugging Face’s official tokenizers crate in Rust:
use tokenizers::Tokenizer;
pub fn decode_tokens(tokenizer: &Tokenizer, tokens: &[u32]) -> String {
tokenizer
.decode(tokens, true)
.unwrap_or_else(|_| String::from("[Decode Error]"))
}
5. Battle-Hardened Production Gotchas
When you transition from a prototype to a real desktop application, you quickly encounter quirks that benchmark tutorials never mention.
1. The Direct3D 12 Shader Warmup Hitch
DirectML does not use pre-compiled machine-code binaries for every possible GPU architecture like CUDA does with PTX/SASS. Instead, DirectML compiles HLSL compute shaders into DirectX 12 Pipeline State Objects (PSOs) at runtime, tailored specifically to your exact GPU driver and tensor input shapes.
- The Problem: The very first time
session.run()is called, DirectML pauses to compile all operator shaders. On mid-range hardware, this first-run warmup can take 2 to 5 seconds. If this happens on the UI thread when the user clicks "Start Transcribing", the application appears frozen. - The Production Fix: Pre-warm the session during startup or model loading on a background thread with a synthetic dummy tensor of zeros:
pub fn warmup_session(session: &mut Session) -> Result<()> {
let dummy_mel = ndarray::Array3::<f32>::zeros((1, 80, 3000));
let dummy_tensor = Tensor::from_array(session.allocator(), &dummy_mel)?;
// Executes shader compilation in the background before the user touches anything
let _ = session.run(ort::inputs!["input_features" => dummy_tensor]?);
Ok(())
}
Once the PSOs are compiled and cached in DirectX memory, all subsequent inference calls run at full hardware speed with zero hitching.
2. Fixed vs. Dynamic Tensor Shapes
DirectML optimizes shaders aggressively around fixed buffer strides. If you pass variable-length audio chunks (e.g. 3.4 seconds, then 11.2 seconds, then 24.1 seconds), DirectML may be forced to recompile or switch shader pipelines constantly.
Always pad your mel-spectrogram arrays to the constant 30-second window (3000 frames). Memory allocation on modern GPUs is cheap; shader recompilation churn is not.
3. The Anti-Hallucination Shield: Silence & Voice Activity Detection (VAD)
Whisper’s autoregressive decoder has a notorious Achilles' heel: ambient silence.
When fed audio containing long pauses, room reverb, or microphone hiss with no actual speech, the model’s attention mechanisms wander. It begins hallucinating phrases like "Thank you for watching", "Please subscribe", or getting trapped in infinite repetition loops:
[00:01:14] ...
[00:01:17] Thank you.
[00:01:20] Thank you.
[00:01:23] Thank you.
[00:01:26] Thank you very much.
In a production desktop recorder or transcription tool, you cannot feed raw, unsegmented audio streams directly into Whisper. You need an energy-based Voice Activity Detection (VAD) stage prior to inference:
/// Scans audio samples to detect regions with speech activity above an RMS threshold
pub fn detect_speech_regions(samples: &[f32]) -> Vec<(usize, usize)> {
const SAMPLE_RATE: usize = 16000;
const FRAME_MS: usize = 30;
const FRAME_SIZE: usize = SAMPLE_RATE * FRAME_MS / 1000; // 480 samples
const ENERGY_THRESHOLD: f32 = 0.008;
const MIN_SPEECH_DURATION_MS: usize = 250;
const SILENCE_PADDING_MS: usize = 300;
let mut regions = Vec::new();
let mut in_speech = false;
let mut speech_start = 0;
let mut silence_frames = 0;
for (i, frame) in samples.chunks(FRAME_SIZE).enumerate() {
let rms = (frame.iter().map(|s| s * s).sum::<f32>() / frame.len() as f32).sqrt();
if rms >= ENERGY_THRESHOLD {
if !in_speech {
in_speech = true;
speech_start = i * FRAME_SIZE;
}
silence_frames = 0;
} else if in_speech {
silence_frames += 1;
// End segment after sustained silence
if silence_frames * FRAME_MS > SILENCE_PADDING_MS {
in_speech = false;
let speech_end = (i * FRAME_SIZE).min(samples.len());
if (speech_end - speech_start) * 1000 / SAMPLE_RATE >= MIN_SPEECH_DURATION_MS {
regions.push((speech_start, speech_end));
}
}
}
}
regions
}
By filtering out silence intervals before calling the ONNX session:
- Hallucinations drop to zero: Whisper only ever receives actual vocal harmonics.
- Inference time plummets: A 60-minute conference recording with 20 minutes of silence runs in 33% less GPU time.
- Multi-Track Dialogue Alignment: In a multi-party call (e.g. microphone track vs remote system audio), each track's speech regions can be processed independently and chronologically merged into a natural dialogue transcript:
6. The Verdict
| Metric | whisper.cpp / whisper-rs (CUDA) |
ort + DirectML (Rust) |
|---|---|---|
| Toolchain Dependencies | MSVC + CMake + Clang + CUDA SDK | Standard cargo build |
| GPU Vendor Support | NVIDIA GeForce / RTX only | NVIDIA, AMD, Intel, Qualcomm |
| End-User Prerequisites | Matching NVIDIA Display Drivers & CUDA DLLs | Standard Windows DirectX 12 drivers |
| Redistribution Size | Varies; complex runtime DLL matrix | Just onnxruntime.dll + DirectML.dll |
| Relative Throughput | Baseline (1.0x) | ~0.85x – 0.95x of native CUDA |
| Developer Sanity | Low (fragile C++ FFI & environment flags) | High (pure Rust crate ecosystem) |
Is native CUDA slightly faster on a top-tier RTX 4090? Yes, by approximately 5% to 15% in raw compute throughput.
But in desktop engineering, portability and deployment simplicity trumps microscopic benchmark wins every day of the week. DirectML lets you ship a single Rust binary that boots instantly, accelerates seamlessly on whatever GPU the user possesses, and never breaks because an environmental variable was missing on a CI worker.
If you are shipping local AI models in Rust on Windows today, step out of the CUDA build trap. DirectML is the pragmatic path forward.
