Microsoft launches Windows ML Runtime API with experimental GGUF and ONNX support

At its Windows and Surface event on October 8, Microsoft outlined the latest updates to Windows ML, its local AI inference framework for Windows 11 that lets developers run models on-device for text generation, speech recognition, and similar applications while reducing dependence on hardware-specific optimizations.

The company said it is working with the llama.cpp and GGUF open-source communities to add experimental native support for GGUF and ONNX model inference in Windows 11. Through an experimental integration with llama.cpp, developers can import GGUF models from Hugging Face and run them locally. Microsoft also said it contributed performance optimization code to llama.cpp together with Nvidia.

GGUF is a model file format commonly used for running large language models locally, while ONNX is an open neural network exchange format for porting models between different frameworks and hardware. The new native inference path for Windows, formally called the Windows ML Runtime API, can run both ONNX and GGUF models. Microsoft described it as offering higher performance, deeper system integration, and finer-grained control.

On top of Windows ML Runtime API, Microsoft also introduced Text Generation API and Speech Recognition API, enabling developers to build text generation and speech recognition features. According to Microsoft, ONNX Runtime API remains fully supported; future Windows-focused optimizations will prioritize Windows ML Runtime API. Both runtimes are included in the current version, and developers can choose when to migrate based on their familiarity.