appLocal Models
Inference engine written in C that runs very large mixture-of-experts open models on hardware you already own, keeping active experts in memory and streaming the rest from disk. Supports CPU, CUDA and Vulkan.
Every tool tagged โinferenceโ, grouped by category.
Inference engine written in C that runs very large mixture-of-experts open models on hardware you already own, keeping active experts in memory and streaming the rest from disk. Supports CPU, CUDA and Vulkan.
Minimal Python framework for building custom AI inference servers, with full control over batching, streaming, routing and scaling for models, agents and RAG pipelines.
Open-source inference engine that tunes itself to your hardware to run open models locally on Mac, Windows and Linux.