AISIX AI Gateway

io.github.api7Published by GitHub account api7v1.3.0Updated Oct 24, 2026

Self-hosted AI gateway that governs MCP servers, LLM traffic, and A2A agents behind one endpoint

Overview

AI-generated overview

Self-hosted AI gateway that puts one OpenAI-compatible API in front of LLM providers and fronts registered MCP servers, A2A agents and REST APIs as tools.

What you can do
  • Proxy LLM traffic through OpenAI-compatible and Anthropic Messages routes, translating to OpenAI, Anthropic, Bedrock, Vertex or Azure OpenAI upstreams
  • Route and fail over across models with round-robin, consistent-hash, failover, least-cost, least-latency, least-busy, ensemble and semantic strategies
  • Front registered upstream MCP servers at one endpoint with gateway-held credentials, per-server tool namespaces and per-caller access, and expose a REST API's OpenAPI description as MCP tools
  • Front A2A agents over JSON-RPC with agent cards rewritten to the gateway
  • Enforce caller API keys or OIDC/JWT tokens, rate and concurrency limits, content guardrails, and exact-match response…
When to use it
Use it when several teams or applications need one governed endpoint for many LLM providers, or when you want to consolidate multiple MCP servers and A2A agents behind a single authenticated gateway you operate yourself. It suits platform teams that prefer declarative configuration over a hosted control plane.
Requirements
Runs as a local container image (ghcr.io/api7/aisix) or a static binary in your own infrastructure. Needs a config file and a resources.yaml declaring provider keys, models, caller keys, MCP servers and agents; provider credentials are supplied through environment variable references such as OPENAI_API_KEY, and caller keys through a variable such as CALLER_API_KEY. etcd is optional for multi-replica setups; Redis is optional for shared rate limits and caching.
Before you install
The gateway holds upstream provider credentials and caller API keys, so protect the resources file and environment. The metrics/status listener on port 9090 is unauthenticated and should stay on loopback or a private network. It proxies live AI traffic and can spend money on upstream providers; budgets and spend caps are only in the commercial editions. Configuration changes are applied by rewriting resources.yaml and sending SIGHUP, and plaintext credentials in etcd are not encrypted at rest.

Installation

In SourceWeft

  1. Open AISIX AI Gateway in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

AISIX OSS

The open-source, Rust-native AI gateway for LLMs and AI agents

One OpenAI-compatible API in front of every model. Route, govern, secure, cache, and observe all your LLM and AI-agent traffic from a single control point — shipped as one static binary with low per-request overhead. Run it in your infrastructure for free, forever.

Built by the original creators of Apache APISIX.

[License: Apache 2.0] [Built with Rust] [Docs] [Discord] [Website]

Start free · Documentation · Quickstart · AISIX Enterprise and AISIX Cloud · Roadmap


[AISIX OSS architecture — one OpenAI- or Anthropic-compatible API in front of OpenAI, Anthropic, Gemini/Vertex, Bedrock, Azure OpenAI, and DeepSeek, with API key auth, rate and token limits, guardrails, caching, routing and failover, and observability in between]

AISIX OSS is the open-source product built around the Rust-native AISIX gateway. The gateway puts a single, OpenAI-compatible API in front of every LLM provider — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, DeepSeek, and any OpenAI-compatible endpoint. It gives platform teams one place to route, govern, secure, and observe LLM traffic, with first-class SSE streaming and low gateway overhead.

It runs as a single static binary — low cold-start, lock-free config reads, and hot configuration reloads with no restarts: declare resources in one resources.yaml and reload on SIGHUP, or point the gateway at etcd for a multi-replica cluster. Run the gateway in your infrastructure, or connect it to AISIX Enterprise or AISIX Cloud for centralized management with team governance, budgets, audit, and a dashboard.

AISIX OSS (this repo) is the open-source product. It runs without a control plane using declarative configuration or etcd. The same gateway binary serves as the data plane in the commercial products: customers host the control plane with AISIX Enterprise, while API7 hosts it with AISIX Cloud. In all three products, the gateway runs in your environment and calls providers directly; live AI traffic does not pass through either control plane. The proxy API is identical throughout. Talk to us about AISIX Enterprise or AISIX Cloud →

⚡ Quickstart

One container. No control plane, no database, no configuration store — the gateway reads every dynamic resource from one declarative resources.yaml.

yaml
# config.yamlresources_file: /etc/aisix/resources.yamlproxy:  addr: "0.0.0.0:3000"admin:  enabled: false          # a declarative gateway needs no admin listenerobservability:  metrics:    prometheus:      enabled: true      addr: "0.0.0.0:9090"
yaml
# resources.yaml_format_version: "1"
provider_keys:  - display_name: openai-main    provider: openai    api_key: ${OPENAI_API_KEY}        # interpolated from the environment
models:  - display_name: my-model    provider: openai    model_name: gpt-4o-mini    provider_key: openai-main
api_keys:  - display_name: local-dev    key_env: CALLER_API_KEY           # hashed at load; the plaintext is never stored    allowed_models: ["my-model"]
bash
export OPENAI_API_KEY="YOUR_PROVIDER_KEY"export CALLER_API_KEY="YOUR_CALLER_KEY"
docker run -d --name aisix \  -v "$(pwd)/config.yaml:/etc/aisix/config.yaml:ro" \  -v "$(pwd)/resources.yaml:/etc/aisix/resources.yaml:ro" \  -e OPENAI_API_KEY -e CALLER_API_KEY \  -p 3000:3000 -p 127.0.0.1:9090:9090 \  ghcr.io/api7/aisix:latest        # proxy → :3000, metrics + status → :9090#                                  ^ the metrics/status listener is unauthenticated;#                                    keep it on loopback or a private network

Then call the gateway exactly like OpenAI:

bash
curl http://localhost:3000/v1/chat/completions \  -H "Authorization: Bearer $CALLER_API_KEY" \  -H "Content-Type: application/json" \  -d '{"model":"my-model","messages":[{"role":"user","content":"hello"}]}'

Edit resources.yaml and send SIGHUP (docker kill -s HUP aisix) to apply changes with no restart — an invalid file is rejected whole and the last good configuration keeps serving. Check a file before booting with aisix validate --resources resources.yaml.

Full walkthrough: the Gateway Quickstart · every field: the resources file reference. For a multi-replica cluster, point the gateway at etcd instead — resources_file and etcd are mutually exclusive.

✨ Why AISIX

  • One API, every model. Speak the OpenAI or Anthropic wire format in; the gateway translates to whichever provider each model points at. Point an OpenAI or Claude SDK at one base_url and switch models without changing code.
  • A real gateway, in Rust. Single static binary, low cold-start, lock-free config reads on the hot path, native streaming.
  • Open source, free forever. Apache-2.0 licensed and built to run in your infrastructure. Choose AISIX Enterprise or AISIX Cloud when you want centralized management through a control plane and dashboard.
  • Production controls built in. Routing & failover, rate limits, guardrails, caching, and observability ship in the box. (Budgets and spend caps are AISIX Enterprise and AISIX Cloud features — the gateway enforces the control plane's decisions.)

🧩 Features — available today

Covered by 183 end-to-end scenario files (496 cases) that run against real gateway processes.

  • OpenAI-compatible proxy (:3000) — chat/completions, completions, responses, embeddings, rerank, images/{generations,edits}, audio/{speech,transcriptions,translations}, videos (submit → poll → fetch), files, batches, fine_tuning/jobs, realtime, GET /v1/models, plus a root-level /passthrough/:provider/* escape hatch. Native SSE streaming, tool/function calling, JSON mode, vision/multimodal input, and reasoning-content support.
  • Anthropic Messages API — POST /v1/messages as a first-class route, working against any configured upstream: requests and responses (including streaming) are translated both ways when a model points at a non-Anthropic provider.
  • Routing & failover — virtual/routing models with six strategies: round_robin (smooth weighted round-robin), consistent_hash (session affinity keyed by header / cookie / API key / client IP), failover, plus metric-based least_cost, least_latency, and least_busy. Per-target priority tiers (active/backup pools), retry budgets, cooldowns, tag-conditional targets, and per-attempt timeouts.
  • Ensemble models — fan one request out to a panel of models concurrently, then have a judge model synthesize a single answer, with a minimum-successful-responses threshold.
  • Semantic routing — one virtual model that dispatches by the meaning of each request: it embeds the prompt, scores it against per-route example utterances, and routes to the best match (or a default). See the semantic routing docs.
  • Rate limiting & concurrency — RPS/RPM/RPH/RPD + TPM/TPD + concurrency caps, AND-combined across caller keys, models, and policy scopes (api_key / model / team / member / team_member). Counters are per-process by default, or shared across replicas with the Redis backend.
  • Guardrails — content-policy enforcement on input and output, in-process or through a provider: keyword/regex, built-in PII detection and redaction, Presidio, Lakera, OpenAI Moderation, AWS Bedrock Guardrails, Azure AI Content Safety (Prompt Shield + text moderation), and two Alibaba Cloud services. A block returns 422 content_filter; monitor mode records what would have happened without blocking.
  • Caching — exact-match response cache with per-policy TTL and model/key scope matchers; memory and Redis backends; cost-saved telemetry on every hit. Separately, automatic prompt caching can be enabled per direct Anthropic model to inject cache breakpoints, so callers get provider-side prompt discounts without changing their requests.
  • MCP gateway — front registered upstream MCP servers at /mcp with gateway-held credentials, per-server tool namespaces, and per-caller access. It serves every Streamable HTTP revision from 2025-03-26 through stateless 2026-07-28 without downstream sessions. Upstreams use initialize by default or server/discover with protocol_version: "2026-07-28". CI runs the official MCP suite's applicable tools-only protocol scenarios. Also exposes a REST API as MCP tools from its OpenAPI description.
  • A2A agent gateway — front A2A (Agent-to-Agent) agents at /a2a/:agent, serving each agent's card with URLs rewritten to the gateway, over JSON-RPC 2.0.
  • Inbound authentication — caller API keys (SHA-256 hashed, model allowlists, expiry, rotation), or OIDC/JWT bearer tokens validated against registered providers (Entra ID, Okta, Google Workspace, or any OIDC issuer) with JWKS caching.
  • Observability — Prometheus /metrics, structured per-request access logs, usage events, OTLP/GenAI span export (Langfuse, Honeycomb, Grafana Cloud, or any OTLP receiver), plus dedicated Datadog and Aliyun SLS log exporters and object-storage (S3/GCS/Azure Blob) telemetry.
  • Declarative configuration — one resources.yaml carries all ten resource collections (provider keys, models, caller keys, guardrails, MCP servers, A2A agents, cache policies, observability exporters, rate-limit policies, OIDC providers), validated against the same JSON Schemas the gateway uses at runtime. aisix validate checks a file offline; SIGHUP reloads it atomically.
  • Operational endpoints — /livez and /readyz on the proxy listener; /status/config, /status/ready, /status/models, and Prometheus /metrics on a dedicated metrics listener (:9090). The admin listener (:3001) additionally serves a read-only resource surface, OpenAPI 3 with a Scalar UI, and a playground. Resources are managed declaratively — through the resources_file (reloaded on SIGHUP) or direct etcd writes — not through the admin listener; its former write endpoints were removed.

🔌 Supported providers

AISIX dispatches through five native adapter families — distinct wire-protocol bridges, not one generic relabel. Whatever the upstream protocol, the client-facing API stays OpenAI-shaped.

Adapter familyReachesWire shape · auth
openaiOpenAI + any OpenAI-compatible vendor — DeepSeek, Groq, Mistral, Together, Fireworks, Perplexity, vLLM, Ollama, or self-hosted OpenAI-compatible endpointsOpenAI chat completions · Bearer
anthropicAnthropic ClaudeAnthropic Messages · x-api-key
bedrockAWS Bedrock — Anthropic, Meta Llama, Mistral, Cohere, Amazon Titan/Nova, AI21Bedrock Converse + /invoke · SigV4
vertexGoogle Vertex AI (Gemini)Vertex :generateContent · OAuth2
azure-openaiAzure OpenAIAzure deployments · api-key / Entra ID

Plus specialized handling for vendor quirks (e.g. DeepSeek reasoning content) and dedicated rerank / embeddings vendors (Cohere, Jina). Details in adapter protocol families.

☁️ Choose an AISIX Product

Same gateway binary, same proxy API — in every form the gateway runs in your environment. AISIX Enterprise adds a customer-hosted commercial control plane. AISIX Cloud adds an API7-hosted commercial control plane.

The AISIX Enterprise and AISIX Cloud dashboard — overview metrics, multi-provider models, guardrails, budgets, and observability exporters across all your gateways.

▶ Try the live dashboard demo — aisix-demo.api7.ai

AISIX OSS (this repo)AISIX EnterpriseAISIX Cloud
LicenseApache-2.0CommercialCommercial
Control planeNoneCustomer-hostedAPI7-hosted
ConfigurationDeclarative resources.yaml, or etcd for a clusterDashboard + AISIX Cloud Admin APIDashboard + AISIX Cloud Admin API
TenancySingle instance / namespaceOrganization-scoped environments, teams, and membersOrganization-scoped environments, teams, and members
Provider keys${VAR} references in resources.yaml; plaintext credentials in etcdEncrypted at rest; operator-triggered in-place rotationEncrypted at rest; operator-triggered in-place rotation
Budgets— (rate and token limits only)Per key, provider, environment, organization, and teamPer key, provider, environment, organization, and team
RBAC and auditLocal admin key; no organization layerOrganization roles, invitations, and audit historyOrganization roles, invitations, and audit history
OperationsYou operate the gatewayYou operate the control plane and gatewaysAPI7 operates the control plane; you operate gateways

→ Want centralized governance, budgets, and a dashboard? Talk to API7 about AISIX Enterprise or AISIX Cloud, or book a demo.

🏗️ Architecture

A single Cargo workspace; the aisix-server crate builds one binary named aisix that wires the crates together.

text
crates/├── aisix-core           Config, snapshot, resource model, resources.yaml source, errors├── aisix-etcd           Config provider + watch supervisor├── aisix-gateway        Hub & bridge, SSE parser, provider trait├── aisix-proxy          /v1/*, /mcp, /a2a handlers, routing, middleware├── aisix-admin          Read-only resource surface + playground + OpenAPI├── aisix-provider-*     openai · anthropic · azure-openai · bedrock · vertex├── aisix-mcp            MCP gateway — server registry, tool ACL, transports├── aisix-a2a            A2A agent gateway — agent cards, JSON-RPC bridge├── aisix-ratelimit      fixed-window + token accounting + concurrency (local | redis)├── aisix-cache          memory + redis backends├── aisix-redis          shared Redis connection for cache + rate limits├── aisix-guardrails     pre/post content-policy hooks├── aisix-obs            tracing, metrics, access log, exporters└── aisix-server         the `aisix` binary — bootstrap + CLI

🗺️ Roadmap

Highlights on the roadmap; tracked live in issues:

  • Semantic (embedding-similarity) response caching
  • More observability sinks — Langsmith, Helicone, Slack alerts
  • Prompt templates managed as gateway resources
  • Llama-Guard as a guardrail provider

Shipped since this list was last written: the MCP gateway, the A2A agent gateway, OIDC/JWT inbound auth, Redis-backed distributed rate limiting, and the Lakera, Presidio, PII, and OpenAI Moderation guardrails — see Features above.

🛠️ Development

Prerequisites: the Rust toolchain pinned in rust-toolchain.toml. Docker is only needed for the tests that exercise etcd, Redis, or provider emulators.

bash
cargo check --workspacecargo fmt --checkcargo clippy --workspace -- -D warningscargo test --workspace
# Coverage (matches the CI gate)cargo llvm-cov --workspace --lcov --output-path lcov.info
# Run locally against a resources.yaml (no etcd needed). Copy the Quickstart's two files# and change resources_file to the local path, e.g. resources_file: ./resources.yamlcargo run -p aisix-server --bin aisix -- --config config.local.yaml
# Check a resources file without starting a listenercargo run -p aisix-server --bin aisix -- validate --resources resources.yaml

💬 Community

If AISIX is useful to you, a ⭐ helps other engineers find it.

📄 License

Apache 2.0.

Source: README.md at commit ad58900

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

2
  1. v1.3.0LatestSep 21, 2026
  2. v0.12.0Sep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts