# AI Gateway > Track every token. Cap every budget. Route across a 26-provider, 182-model catalog behind one OpenAI-compatible endpoint. Apinizer governs your LLM, MCP, and agent traffic on the same runtime that already runs your REST APIs — same audit, same identity, same operators on call. *AI Gateway — agentic plane* ## Govern every AI request. Tokens, cost, and risk — one gateway. [Request a demo](https://calendly.com/apinizer/15min) · [Read the docs](https://apinizer.com/developers/docs) **Highlights** - **Catalog** — 26 providers · 182 models - **Standards** — OpenAI · MCP · A2A - **Modalities** — Chat · Embed · Audio · Image --- ## Capabilities ### Token economics and cost control Every prompt, response, and embedding is counted, attributed, and capped. Budgets live on the gateway — in tokens or in dollars — so the team that owns the workload owns the bill, and finance gets a line item instead of a surprise. - Live token tracking — input, output, cached, and total — on every request - Ceilings in tokens or dollars across six scope tiers, crossed with provider and model - Threshold alarms at 50, 80, 90, and 100 percent before anything hard-stops - Automatic fall-back to a cheaper model or a cached answer when a budget tips - Cost attributed back to the project that ran the workload, exportable for finance ### Cost-aware multi-LLM routing Write the application once against an OpenAI-compatible endpoint; the gateway decides which model answers — by cost, by latency, by load, or by your own rules. Bodies are rebuilt in each provider's native format, so a fall-back chain can cross vendor families without the client noticing. - One OpenAI-compatible endpoint — a drop-in for the openai and langchain_openai SDKs - Seven routing algorithms, including least-cost, least-latency, and cheaper-model downgrade - Cross-family fall-back with two-way translation, and billing that never double-counts a retry - A 26-provider, 182-model catalog you extend on screen — a new provider needs no release - Real per-chunk streaming, with usage collected when the stream ends ### Exact and semantic response cache Skip the token bill on repeat prompts. An identical prompt is answered from the Hazelcast cluster the REST gateway already runs; a re-worded one is answered from a real vector layer scored against your own VectorDB. The cache stores the masked response, so a hit can never replay what the PII policy stripped. - Exact match on Hazelcast, semantic match on pgvector, Qdrant, or Redis - Your own similarity threshold and top-K, tuned per policy - Masked-first — the semantic layer never stores the raw pre-mask body - Hit rate, tokens saved, and latency saved reported per cache bucket - A scheduled purge keeps the vector layer from filling with orphaned entries ### Prompt templates and decorators on the gateway The client sends a template name and a few parameters; the gateway assembles the real prompt. Application code stops carrying the prompt, and a prompt change ships through APIops instead of a release. - A template catalog with parameters and defaults, scoped built-in, platform, or project - Gateway-side expansion, closed to template and JSON-structural injection - Nine responsible-AI decorator presets, from privacy to regulatory compliance - Assembly order enforced — template, decorator, RAG, then guards and cache - Templates are exportable artifacts: wizard, REST, or APIops promotion ### RAG on your own VectorDB — nothing leaves the cluster Ground answers in your documents without shipping them anywhere. Upload to a knowledge base and Apinizer chunks, embeds, and indexes into a VectorDB you run. Retrieval is bounded by project and knowledge base, so one tenant's question can never reach another tenant's chunks. - Knowledge bases from PDF, Excel, and document upload — chunked, embedded, indexed - Four injection modes, with top-K, similarity threshold, and an explicit no-match behavior - Multi-tenant isolation enforced at query time, and fail-closed when scope cannot be resolved - A groundedness check scores whether the answer is actually supported by what was retrieved - GDPR / KVKK erasure — delete the source document, drop its vectors ### Prompt firewall — injection, jailbreak, and off-topic Block the patterns that put regulated AI projects on hold. The guards run inline, so the bad request never reaches the model — and encoded payloads are decoded before the guards read them, which means an attack wrapped in base64 gets inspected instead of forwarded blind. - Encoded content decoded before scanning — base64, hex, URL, unicode, HTML entity - Injection and jailbreak detection with signature packs mapped to OWASP LLM Top 10 and MITRE ATLAS - Content safety on 14 MLCommons AILuminate hazard categories, extended from a catalog screen - Context-integrity checks catch forged roles and control tokens before the model sees them - Run any guard inline, async, or in shadow mode while you tune it ### PII and data-loss control that survives streaming Turkish identity data is a first-class citizen, not a regex you maintain. TCKN, IBAN, and phone numbers are checksum-validated before anything is masked, and masking holds across SSE chunk boundaries and on tool results. Where the law asks for pseudonymization rather than redaction, the real value comes back for authorized roles only. - Twelve checksum-validated identifier types, with TCKN, IBAN-TR, and phone-TR built in - Six masking strategies, from redaction to synthetic values generated independently of the original - Reversible pseudonymization — restored in the response or through an unmask API, deny-by-default and audited - Versioned DLP rule packs covering the 18 HIPAA Safe Harbor identifiers and PCI DSS cardholder data - Embedded files identified by signature bytes, with an external guardrail engine available behind any of it ### AI observability — every prompt, every token, every model The Analytics Engine ingests AI traffic next to REST traffic. One query answers cost-by-team, latency-by-provider, error-rate-by-model, and which guard tripped last night — and when you already run Grafana, AI traffic lands in the dashboards your SRE team already watches. - Token spend and latency by user, project, team, provider, and model - Time-to-first-token and time-per-output-token, not just total latency - Live trace with a request DAG, a timeline, and replay of a captured request - Outbound MCP tool and A2A skill names on the traffic record — no trace required - Native OTLP, Prometheus, and a Grafana dashboard, plus anomaly alerts on spend and latency ### Retention profiles, asset inventory, and shadow-AI discovery Know what AI is running in the installation, and decide how much of it is ever written down. A retention profile sets how much of an exchange persists — the full record, counters and decisions without the bodies, or nothing at all — as an installation default you override per proxy. - Three retention profiles — standard, no-payload, no-persist — with the stricter setting winning - An inventory of every AI asset: proxies, providers, models, knowledge bases, vector databases - A shadow-AI report naming the models in use that nobody put in the catalog - An endpoints panel per environment, with a one-click Test Console - Topic categories and prompt templates defined through APIops, like every other asset ### MCP and A2A as first-class proxies MCP and Agent2Agent are proxy types alongside REST, SOAP, and AI — same policy chain, same deploy history, same rollback. Agents authenticate through the same policies as every other client, and every call lands in the same audit record. - Publish the REST and SOAP APIs you already run as MCP tools, or pass through to external servers - Publish a proxy as a discoverable agent, or front an external agent and rewrite its card - Default-deny tool access per credential, role, or organization, with per-tool call quotas - Tool schemas pinned by hash — a silent change is flagged, not quietly trusted - Standard auth — JWT, OIDC, OAuth 2.0, API key, mTLS — with no separate agent scheme to maintain ### One gateway, one audit, one runtime AI Gateway is not a side-car. It is a layer of policies on the same gateway that runs your REST, gRPC, WebSocket, SOAP, and GraphQL traffic. There is no second control plane to learn, no second pager rotation, and no second invoice. - One runtime for REST, gRPC, WebSocket, SOAP, GraphQL, MCP, A2A, and AI - One identity surface — OAuth 2.0, OIDC, JWT, mTLS, SAML — for people, agents, and partners - One permission model — System, Project, Team — plus dedicated AI roles - Provider keys stay on the LLM connection; applications authenticate with their own credentials - Kubernetes-native and air-gap friendly — no data leaves the cluster --- ## Use cases ### Stop the AI bill from running away Token budgets per project. Response cache on the hot path. Cheap-model fall-back when the budget tips. Cost attribution by team and project. The AI line item stops being a surprise, and finance gets a cost report instead of a Slack message. - Token and USD ceilings on credential, organization, proxy, and project scopes - Exact and semantic response cache with hit-rate and savings reports - Cheaper-model downgrade and least-cost routing off the catalog price - Cost attribution back to the project that ran the workload - Monthly cost exports for finance ### Production-grade agents under one governance plane Agents authenticate like users. Agent-to-agent messages are audited at the persistence layer. Tool access is scoped per identity, tool schemas are pinned against silent drift, and runaway loops cap themselves. The platform team owns one control plane for human and agent traffic — not two. - Agent identities provisioned in Identity Manager, authenticated by the standard policy chain - Per-tool call quotas, argument constraints, and default-deny tool RBAC - Agent-to-agent task lifecycle with streaming relay, cancel, and audit - Tool-catalog drift detection — a changed schema is flagged, not silently trusted - Publish the REST and SOAP APIs you already run as MCP tools - Outbound MCP tool and A2A skill names on the traffic record — what a flow called, and how often, without turning on trace ### PII, secrets, and prompt injection — handled at the edge Run regulated AI projects without a binder of risk acceptances. Inline prompt firewall, checksum-validated Turkish PII masking, and credential blocking in both directions — including inside streaming responses and tool results. Where the law asks for pseudonymization rather than redaction, the real value comes back for authorized roles only, and the restore is audited. Audit trail on every prompt and every response. Same auditor view as the REST estate — one report, two surfaces. - Inline prompt firewall — injection, jailbreak, and content safety, with encoded payloads decoded first - TCKN, IBAN, and Turkish phone masking validated by checksum, holding across streaming chunks and tool results - Reversible pseudonymization with deny-by-default restore — GDPR Article 4(5), not a one-way redaction - Retention profiles that keep prompt and response bodies out of persistence entirely - Audit trail at the persistence layer — every prompt, response, and decision - Air-gap-friendly deployment — your data never leaves your cluster ### One endpoint, many providers — no client rewrites Application code stays on the OpenAI-compatible surface. The gateway decides who answers — frontier, open-weights, or a self-hosted model — by cost, by latency, by load, or by your own routing rules. Bodies are rebuilt in each provider's native format, so a fall-back chain can cross from one vendor family to another without the client noticing. - OpenAI-compatible request and response shape, including `GET /v1/models` discovery - A 26-provider, 182-model built-in catalog, extended on screen without waiting for a release - Least-cost and least-latency routing off the live catalog and health probes - Load-balanced pools and rule-based conditional routing - Cross-family fall-back chains with two-way format translation - Cheaper-model downgrade and per-leg cost ceilings when the budget tips ### Chat, embeddings, audio, and image — one path Same identity, same audit, same observability for every modality. Embeddings get the same masking as chat. Image generation gets the same quota window as text. Audio transcripts land in the same Analytics Engine as request logs. - Chat completions — streaming and function calling - Responses API — unary and streaming - Embeddings — client-facing, priced on input tokens, reused by RAG and semantic cache - Audio — speech-to-text transcription and text-to-speech - Image — generation across supported providers --- ## What ships in the box ### AI traffic types - Chat completions — unary and streaming - Responses API — unary and streaming - Embeddings — client-facing, shared with RAG and semantic cache - Audio — speech-to-text and text-to-speech - Image — generation - Function calling, tool use, and unary agentic loops - Agent-to-Agent (A2A) messages and tasks - Model Context Protocol (MCP) tool calls - Model discovery — OpenAI-compatible `GET /v1/models` ### Cost & token governance - Live token tracking — input, output, cached, total - Ceilings per window — minute, hour, day, month - Ceilings per owner tier — credential, organization, proxy, project - Ceilings per provider and per model, measured in tokens or dollars - Threshold alarms at 50 / 80 / 90 / 100 percent, with monthly reset - Auto fall-back to cached answer or cheaper model on budget tip - Cost exports back to the project that ran the workload ### Security & guardrails - Encoded payloads decoded before scanning — base64, hex, URL, unicode, HTML entity - Prompt injection, jailbreak, and context-integrity guards, scoped per message role - Signature packs mapped to OWASP LLM Top 10 and MITRE ATLAS, with drift warnings - Topic guard plus content safety on 14 AILuminate hazard categories - Twelve checksum-validated PII types and six masking strategies, streaming-safe - Reversible pseudonymization with deny-by-default restore and a full audit trail - Versioned DLP rule packs — HIPAA Safe Harbor identifiers and PCI DSS cardholder data - External guardrail engines through one adapter, on the guards, DLP, and PII masking - Default-deny tool RBAC per identity, and an audit trail at the persistence layer ### RAG & knowledge - Knowledge bases with upload, PDF and Excel extraction, chunking, and indexing - pgvector, Qdrant, or Redis as the vector store — self-hosted - Four RAG injection modes with top-K and similarity threshold - Project and knowledge-base isolation enforced at query time - Fail-closed when the project scope cannot be resolved - Groundedness check on RAG answers — flag or block what the retrieved context does not support - GDPR / KVKK erasure on indexed documents ### Operability - Same gateway runtime as REST, SOAP, gRPC, WebSocket, MCP, and A2A - Same identity, same audit, same RBAC across API and AI - Explicit deploy with history and rollback; no pod restart for a prompt change - Three-tier permission model (System / Project / Team) plus dedicated AI roles - Live cost, latency, and reliability dashboards - OTLP, Prometheus, and Grafana export for existing SRE tooling - Retention profiles — standard, no-payload, no-persist — as an installation default with per-proxy override - AI asset inventory plus a shadow-AI report for off-catalog model usage - Kubernetes-native, air-gap-friendly deployment --- ## Resources - [AI Gateway docs](https://apinizer.com/developers/docs) — Configure providers, set token budgets, write prompt firewall policies, and observe AI traffic alongside REST. - [Cost & token playbook](https://apinizer.com/developers/docs/ai-gateway/cost-control) — Patterns for project budgets, cost attribution, response cache tuning, and cheap-model fall-back chains. - [Provider quickstarts](https://apinizer.com/developers/docs/ai-gateway/providers) — Drop-in recipes for OpenAI, Anthropic, Bedrock, Azure OpenAI, Gemini, and self-hosted Llama or vLLM. - [Prompt firewall reference](https://apinizer.com/developers/docs/ai-gateway/firewall) — The guard catalog — injection, jailbreak, PII, credentials, off-topic, tool-use — with policy snippets. - [AI observability guide](https://apinizer.com/developers/docs/ai-gateway/observability) — Cost, latency, reliability, and firewall dashboards in the Analytics Engine — one query for API and AI. - [Architecture overview](https://apinizer.com/products) — How the AI plane shares one runtime with the API Gateway, Identity Manager, Cache, and Analytics Engine. - [Migration from a side-car gateway](https://apinizer.com/developers/docs/ai-gateway/migration) — A short field guide for teams running a dedicated AI gateway today — what to keep, what to retire, what to consolidate. --- ## Next step *Govern every AI request* **Bring tokens, agents, and risk under one control plane.** A 30-minute walkthrough of the Apinizer AI Gateway — token budgets, multi-LLM routing, exact and semantic cache, prompt firewall, reversible PII pseudonymization, RAG on your own VectorDB, MCP and A2A governance, retention profiles, and AI observability — on a Kubernetes of your choice. [Book a Demo](https://calendly.com/apinizer/15min) · [Read the docs](https://apinizer.com/developers/docs) --- ## Links - Products: https://apinizer.com/products - AI Gateway: https://apinizer.com/products/ai-gateway - Solutions: https://apinizer.com/solutions - Pricing: https://apinizer.com/pricing - Developers: https://apinizer.com/developers - Documentation: https://docs.apinizer.com/index-en - Blog: https://apinizer.com/blog - Contact: https://apinizer.com/company/contact © 2026 Apinizer. All rights reserved.