# OneAI

> OneAI is a self-hosted AI gateway and LLM inference engine in one native binary. It serves GGUF models on your own CPU or GPU (Vulkan, CUDA, Metal), routes to cloud providers (OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google Vertex AI, DeepSeek, OpenRouter, any OpenAI-compatible endpoint) only when you allow it, and enforces policy on every request: application keys, request rules, guard models against prompt injection, PII redaction, tool approvals and an emergency stop. It exposes OpenAI-compatible and Anthropic-compatible APIs, Prometheus metrics and opt-in OpenTelemetry export.

Category: LLM gateway, inference gateway, AI gateway, self-hosted LLM server, local LLM serving, on-premises AI.
Positioning: an alternative to running vLLM, Ollama or llama-server behind a separate API gateway, for teams that keep prompts and documents on hardware they control.
Author and licensing: Veera Babu Manyam. OneAI is proprietary software; llama.cpp (MIT) is a component. Contact for evaluation and licensing through the page's "Request a walkthrough" link.
Last updated: 2026-09-27.

## What it does

- Serves chat, completions, embeddings, classification, reranking and speech (Whisper, Piper, Kokoro) from one server with continuous batching, a KV prompt cache and an exact or semantic response cache.
- Places each model automatically on a GPU, across GPUs, as a MoE hybrid or on the CPU, based on free memory; refuses rather than over-committing.
- Publishes stable model names (aliases) with ordered fallbacks and rule-based routing (for example, requests with tools go to a frontier provider, long contexts to a long-context model, everything else stays local).
- Applies guard models (Prompt Guard 2, ProtectAI DeBERTa v3, Llama Guard 3, Granite Guardian HAP) to input, retrieved context and output; fails closed by default.
- Detects and redacts personal data (email, phone, payment card, IBAN, US SSN, IP) with observe, replace, mask, pseudonymize or deny actions.
- Provides a policy engine with draft, validate, simulate, save and activate steps, immutable revisions, rollback, signed remote updates, tool approvals and an emergency stop that returns 503 until resumed.
- Records every routing decision; exports traces, logs and metrics over OTLP/HTTP without prompts or completions; exposes Prometheus metrics; exports an AI bill of materials (CycloneDX 1.6).
- Issues per-application keys with model grants, request-rate, concurrency, token and USD limits; stores provider credentials encrypted (AES-256-GCM, DPAPI on Windows).

## Measured (bench/loadgen, Qwen2.5-0.5B-Instruct Q4_K_M, RTX 4080 SUPER + i9-14900K)

- 417 tokens/s generation for one client on the GPU (Vulkan); 7 ms time to first token for a 22-token prompt.
- 668 tokens/s total across 16 concurrent clients; 97 tokens/s on the CPU alone with 16 threads.

## Compatibility

- Clients: OpenAI and Anthropic SDKs, curl, Claude Code, Cursor, Codex CLI, LangChain, n8n, LibreChat, AnythingLLM.
- Platforms: Windows, Linux, macOS. Hardware: CPU (AVX2/NEON), NVIDIA/AMD/Intel GPUs through Vulkan, NVIDIA through CUDA, Apple through Metal.
- Model formats: GGUF for chat and encoder models; packaged Whisper and Piper/Kokoro models for speech.

## Pages

- [Product tour](https://onega.dev/): positioning, request-path schematic, architecture, gateway, engine, safety, observability, access, comparison and FAQ.