AI Gateway — agentic plane

NEW

Govern every AI request. Tokens, cost, and risk — one gateway.

Track every token. Cap every budget. Route across a 26-provider, 182-model catalog behind one OpenAI-compatible endpoint. Apinizer governs your LLM, MCP, and agent traffic on the same runtime that already runs your REST APIs — same audit, same identity, same operators on call.

Request a demoRead the docs
  • Catalog26 providers · 182 models
  • StandardsOpenAI · MCP · A2A
  • ModalitiesChat · Embed · Audio · Image

Token economics and cost control

Every prompt, response, and embedding is counted, attributed, and capped. Budgets live on the gateway — in tokens or in dollars — so the team that owns the workload owns the bill, and finance gets a line item instead of a surprise.

  • Live token tracking — input, output, cached, and total — on every request
  • Ceilings in tokens or dollars across six scope tiers, crossed with provider and model
  • Threshold alarms at 50, 80, 90, and 100 percent before anything hard-stops
  • Automatic fall-back to a cheaper model or a cached answer when a budget tips
  • Cost attributed back to the project that ran the workload, exportable for finance
Tracked per request
input · output · cached · total tokens
Quota windows
minute · hour · day · month
Quota scopes
global · anonymous · credential · org · proxy · tenant
Measured in
tokens or USD
Enforcement
hard cap · threshold alarm · graceful fallback
Reporting
cost by project · model · team · time range

Cost-aware multi-LLM routing

Write the application once against an OpenAI-compatible endpoint; the gateway decides which model answers — by cost, by latency, by load, or by your own rules. Bodies are rebuilt in each provider's native format, so a fall-back chain can cross vendor families without the client noticing.

  • One OpenAI-compatible endpoint — a drop-in for the openai and langchain_openai SDKs
  • Seven routing algorithms, including least-cost, least-latency, and cheaper-model downgrade
  • Cross-family fall-back with two-way translation, and billing that never double-counts a retry
  • A 26-provider, 182-model catalog you extend on screen — a new provider needs no release
  • Real per-chunk streaming, with usage collected when the stream ends
  • Native adaptersOpenAI · Anthropic · Gemini / Vertex · Bedrock · vLLM
  • Catalog26 providers · 182 models · pricing you edit in the UI
  • Provider definitiona catalog record, not code — add one without a release
  • Endpointschat/completions · responses · embeddings · audio · images · models
  • Lifecyclea model past its sunset date is rejected and dropped from discovery

Exact and semantic response cache

Skip the token bill on repeat prompts. An identical prompt is answered from the Hazelcast cluster the REST gateway already runs; a re-worded one is answered from a real vector layer scored against your own VectorDB. The cache stores the masked response, so a hit can never replay what the PII policy stripped.

  • Exact match on Hazelcast, semantic match on pgvector, Qdrant, or Redis
  • Your own similarity threshold and top-K, tuned per policy
  • Masked-first — the semantic layer never stores the raw pre-mask body
  • Hit rate, tokens saved, and latency saved reported per cache bucket
  • A scheduled purge keeps the vector layer from filling with orphaned entries
Exact match
SHA-256 request key on Hazelcast
Semantic match
embedding similarity · threshold + top-K
Vector stores
pgvector · Qdrant · Redis (RediSearch KNN)
Safety
masked body cached · proxy-scoped isolation
Housekeeping
TTL · redeploy invalidation · orphan purge
Reports
hit rate · tokens saved · latency saved

Prompt templates and decorators on the gateway

The client sends a template name and a few parameters; the gateway assembles the real prompt. Application code stops carrying the prompt, and a prompt change ships through APIops instead of a release.

  • A template catalog with parameters and defaults, scoped built-in, platform, or project
  • Gateway-side expansion, closed to template and JSON-structural injection
  • Nine responsible-AI decorator presets, from privacy to regulatory compliance
  • Assembly order enforced — template, decorator, RAG, then guards and cache
  • Templates are exportable artifacts: wizard, REST, or APIops promotion
json
// The client sends this…{ "model": "gpt-4o-mini",  "template": "support-summary",  "template_params": { "tone": "formal", "ticket": "INC-4417" } }// …the gateway expands the catalog template, prepends the// privacy + regulatory-compliance decorators, injects RAG// context, then runs the guards and the cache — in that order.

RAG on your own VectorDB — nothing leaves the cluster

Ground answers in your documents without shipping them anywhere. Upload to a knowledge base and Apinizer chunks, embeds, and indexes into a VectorDB you run. Retrieval is bounded by project and knowledge base, so one tenant's question can never reach another tenant's chunks.

  • Knowledge bases from PDF, Excel, and document upload — chunked, embedded, indexed
  • Four injection modes, with top-K, similarity threshold, and an explicit no-match behavior
  • Multi-tenant isolation enforced at query time, and fail-closed when scope cannot be resolved
  • A groundedness check scores whether the answer is actually supported by what was retrieved
  • GDPR / KVKK erasure — delete the source document, drop its vectors
Sources
document upload · PDF and Excel extraction · chunk + embed
Stores
pgvector · Qdrant · Redis (RediSearch KNN)
Injection
prepend user · prepend system · system suffix · template
Tuning
top-K · similarity threshold · no-match behaviour
Isolation
project + knowledge-base filter, fail-closed
Grounding
judge-model groundedness score · flag or block

Prompt firewall — injection, jailbreak, and off-topic

Block the patterns that put regulated AI projects on hold. The guards run inline, so the bad request never reaches the model — and encoded payloads are decoded before the guards read them, which means an attack wrapped in base64 gets inspected instead of forwarded blind.

  • Encoded content decoded before scanning — base64, hex, URL, unicode, HTML entity
  • Injection and jailbreak detection with signature packs mapped to OWASP LLM Top 10 and MITRE ATLAS
  • Content safety on 14 MLCommons AILuminate hazard categories, extended from a catalog screen
  • Context-integrity checks catch forged roles and control tokens before the model sees them
  • Run any guard inline, async, or in shadow mode while you tune it
Pre-scan decode
base64 · hex · URL · unicode · HTML entity
Inline guards
injection · jailbreak · context integrity
Content safety
14 AILuminate hazard categories · own catalog
Signature packs
OWASP LLM Top 10 · MITRE ATLAS · drift channel
Scan scope
system · user · assistant · tool, per guard
Latency modes
inline · async · shadow

PII and data-loss control that survives streaming

Turkish identity data is a first-class citizen, not a regex you maintain. TCKN, IBAN, and phone numbers are checksum-validated before anything is masked, and masking holds across SSE chunk boundaries and on tool results. Where the law asks for pseudonymization rather than redaction, the real value comes back for authorized roles only.

  • Twelve checksum-validated identifier types, with TCKN, IBAN-TR, and phone-TR built in
  • Six masking strategies, from redaction to synthetic values generated independently of the original
  • Reversible pseudonymization — restored in the response or through an unmask API, deny-by-default and audited
  • Versioned DLP rule packs covering the 18 HIPAA Safe Harbor identifiers and PCI DSS cardholder data
  • Embedded files identified by signature bytes, with an external guardrail engine available behind any of it
Turkish types
TCKN · IBAN-TR · phone-TR · driver's licence
International
email · card · IP · URL · passport · SSN · IBAN · wallet
Mask modes
fixed · keep-first · keep-last · placeholder · synthetic
Reversible
in-response restore · unmask API · deny-by-default RBAC
Rule packs
HIPAA Safe Harbor · PCI DSS · file signatures
Also covers
SSE chunk boundaries · MCP and A2A tool results

AI observability — every prompt, every token, every model

The Analytics Engine ingests AI traffic next to REST traffic. One query answers cost-by-team, latency-by-provider, error-rate-by-model, and which guard tripped last night — and when you already run Grafana, AI traffic lands in the dashboards your SRE team already watches.

  • Token spend and latency by user, project, team, provider, and model
  • Time-to-first-token and time-per-output-token, not just total latency
  • Live trace with a request DAG, a timeline, and replay of a captured request
  • Outbound MCP tool and A2A skill names on the traffic record — no trace required
  • Native OTLP, Prometheus, and a Grafana dashboard, plus anomaly alerts on spend and latency
  • Cost dashboardstokens · dollars · by project, team, model
  • Latency dashboardsTTFT · TPOT · p50 · p99 · by provider
  • Tracinglive trace · request DAG · timeline · replay
  • Securityguardrail hits · masking counts · injection attempts
  • OutboundMCP tool and A2A skill names · real target address
  • ExportOTLP GenAI semconv · Prometheus · Grafana dashboard

Retention profiles, asset inventory, and shadow-AI discovery

Know what AI is running in the installation, and decide how much of it is ever written down. A retention profile sets how much of an exchange persists — the full record, counters and decisions without the bodies, or nothing at all — as an installation default you override per proxy.

  • Three retention profiles — standard, no-payload, no-persist — with the stricter setting winning
  • An inventory of every AI asset: proxies, providers, models, knowledge bases, vector databases
  • A shadow-AI report naming the models in use that nobody put in the catalog
  • An endpoints panel per environment, with a one-click Test Console
  • Topic categories and prompt templates defined through APIops, like every other asset
Retention profiles
standard · no payload · no persist
Scope
installation default · per-proxy override · stricter wins
Inventory
proxies · providers · models · knowledge bases · vector DBs
Shadow AI
off-catalog model and provider usage · catalog drift
Quick test
per-environment endpoint panel · one-click Test Console
APIops
topic categories · prompt templates · prompt decorators

MCP and A2A as first-class proxies

MCP and Agent2Agent are proxy types alongside REST, SOAP, and AI — same policy chain, same deploy history, same rollback. Agents authenticate through the same policies as every other client, and every call lands in the same audit record.

  • Publish the REST and SOAP APIs you already run as MCP tools, or pass through to external servers
  • Publish a proxy as a discoverable agent, or front an external agent and rewrite its card
  • Default-deny tool access per credential, role, or organization, with per-tool call quotas
  • Tool schemas pinned by hash — a silent change is flagged, not quietly trusted
  • Standard auth — JWT, OIDC, OAuth 2.0, API key, mTLS — with no separate agent scheme to maintain
MCP modes
expose REST / SOAP APIs as tools · passthrough
A2A modes
agent-expose · passthrough
MCP versions
2024-11-05 · 2025-03-26 · 2025-06-18 (default) · 2025-11-25 · 2026-07-28
Access
default-deny ACL · per-tool RBAC · output filter
Governance
schema drift pin · argument constraints · call quota
Outbound
mTLS fail-closed · OAuth 2.0 · SSRF-guarded probes

One gateway, one audit, one runtime

AI Gateway is not a side-car. It is a layer of policies on the same gateway that runs your REST, gRPC, WebSocket, SOAP, and GraphQL traffic. There is no second control plane to learn, no second pager rotation, and no second invoice.

  • One runtime for REST, gRPC, WebSocket, SOAP, GraphQL, MCP, A2A, and AI
  • One identity surface — OAuth 2.0, OIDC, JWT, mTLS, SAML — for people, agents, and partners
  • One permission model — System, Project, Team — plus dedicated AI roles
  • Provider keys stay on the LLM connection; applications authenticate with their own credentials
  • Kubernetes-native and air-gap friendly — no data leaves the cluster
  • Runtimeone gateway process · API + AI
  • Identityhumans · agents · partners on one surface
  • Auditpersistence-layer, immutable, replayable
  • RBACSystem / Project / Team — everywhere
  • Deployhot — prompts, routes, models
  • PostureKubernetes-native · air-gap friendly

Use cases

Where teams put it to work.

Stop the AI bill from running away

Token budgets per project. Response cache on the hot path. Cheap-model fall-back when the budget tips. Cost attribution by team and project. The AI line item stops being a surprise, and finance gets a cost report instead of a Slack message.

  • Token and USD ceilings on credential, organization, proxy, and project scopes
  • Exact and semantic response cache with hit-rate and savings reports
  • Cheaper-model downgrade and least-cost routing off the catalog price
  • Cost attribution back to the project that ran the workload
  • Monthly cost exports for finance

In the box

What's included

The capabilities below are part of the standard install — no add-on SKUs and no separate licenses.

AI traffic types

  • Chat completions — unary and streaming
  • Responses API — unary and streaming
  • Embeddings — client-facing, shared with RAG and semantic cache
  • Audio — speech-to-text and text-to-speech
  • Image — generation
  • Function calling, tool use, and unary agentic loops
  • Agent-to-Agent (A2A) messages and tasks
  • Model Context Protocol (MCP) tool calls
  • Model discovery — OpenAI-compatible `GET /v1/models`

Cost & token governance

  • Live token tracking — input, output, cached, total
  • Ceilings per window — minute, hour, day, month
  • Ceilings per owner tier — credential, organization, proxy, project
  • Ceilings per provider and per model, measured in tokens or dollars
  • Threshold alarms at 50 / 80 / 90 / 100 percent, with monthly reset
  • Auto fall-back to cached answer or cheaper model on budget tip
  • Cost exports back to the project that ran the workload

Security & guardrails

  • Encoded payloads decoded before scanning — base64, hex, URL, unicode, HTML entity
  • Prompt injection, jailbreak, and context-integrity guards, scoped per message role
  • Signature packs mapped to OWASP LLM Top 10 and MITRE ATLAS, with drift warnings
  • Topic guard plus content safety on 14 AILuminate hazard categories
  • Twelve checksum-validated PII types and six masking strategies, streaming-safe
  • Reversible pseudonymization with deny-by-default restore and a full audit trail
  • Versioned DLP rule packs — HIPAA Safe Harbor identifiers and PCI DSS cardholder data
  • External guardrail engines through one adapter, on the guards, DLP, and PII masking
  • Default-deny tool RBAC per identity, and an audit trail at the persistence layer

RAG & knowledge

  • Knowledge bases with upload, PDF and Excel extraction, chunking, and indexing
  • pgvector, Qdrant, or Redis as the vector store — self-hosted
  • Four RAG injection modes with top-K and similarity threshold
  • Project and knowledge-base isolation enforced at query time
  • Fail-closed when the project scope cannot be resolved
  • Groundedness check on RAG answers — flag or block what the retrieved context does not support
  • GDPR / KVKK erasure on indexed documents

Operability

  • Same gateway runtime as REST, SOAP, gRPC, WebSocket, MCP, and A2A
  • Same identity, same audit, same RBAC across API and AI
  • Explicit deploy with history and rollback; no pod restart for a prompt change
  • Three-tier permission model (System / Project / Team) plus dedicated AI roles
  • Live cost, latency, and reliability dashboards
  • OTLP, Prometheus, and Grafana export for existing SRE tooling
  • Retention profiles — standard, no-payload, no-persist — as an installation default with per-proxy override
  • AI asset inventory plus a shadow-AI report for off-catalog model usage
  • Kubernetes-native, air-gap-friendly deployment

Resources

Keep going

AI Gateway docs

Configure providers, set token budgets, write prompt firewall policies, and observe AI traffic alongside REST.

Read the docs

Cost & token playbook

Patterns for project budgets, cost attribution, response cache tuning, and cheap-model fall-back chains.

See the playbook

Provider quickstarts

Drop-in recipes for OpenAI, Anthropic, Bedrock, Azure OpenAI, Gemini, and self-hosted Llama or vLLM.

Browse providers

Prompt firewall reference

The guard catalog — injection, jailbreak, PII, credentials, off-topic, tool-use — with policy snippets.

Read the catalog

AI observability guide

Cost, latency, reliability, and firewall dashboards in the Analytics Engine — one query for API and AI.

Open the guide

Architecture overview

How the AI plane shares one runtime with the API Gateway, Identity Manager, Cache, and Analytics Engine.

See the suite

Migration from a side-car gateway

A short field guide for teams running a dedicated AI gateway today — what to keep, what to retire, what to consolidate.

Read the guide

Govern every AI request

Bring tokens, agents, and risk under one control plane.

A 30-minute walkthrough of the Apinizer AI Gateway — token budgets, multi-LLM routing, exact and semantic cache, prompt firewall, reversible PII pseudonymization, RAG on your own VectorDB, MCP and A2A governance, retention profiles, and AI observability — on a Kubernetes of your choice.

Request a demoRead the docs