# LLM Gateway

[LLM Gateway](https://llmgateway.io) is an open-source AI gateway that routes one OpenAI-compatible endpoint to hundreds of models across many providers — with automatic provider selection, fallback, usage analytics, and cost tracking. Use the hosted gateway at `api.llmgateway.io` or point the adapter at your own self-hosted deployment.

## Installation

<!-- ::start:tabs variant="package-manager" mode="install" -->

react: @tanstack/ai-llmgateway
vue: @tanstack/ai-llmgateway
solid: @tanstack/ai-llmgateway
svelte: @tanstack/ai-llmgateway
preact: @tanstack/ai-llmgateway
angular: @tanstack/ai-llmgateway
vanilla: @tanstack/ai-llmgateway
octane: @tanstack/ai-llmgateway

<!-- ::end:tabs -->

## Basic Usage

```typescript
import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("gpt-5.6-terra"),
  messages: [{ role: "user", content: "Hello!" }],
});
```

`llmGatewayText` reads your API key from the `LLM_GATEWAY_API_KEY` environment variable. Use `createLLMGatewayText` to pass it explicitly.

## Configuration

```typescript
import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText(
  "gpt-5.6-terra",
  process.env.LLM_GATEWAY_API_KEY!,
  {
    baseURL: "https://api.llmgateway.io/v1", // Optional — set for self-hosted deployments
  },
);
```

LLM Gateway is open source and self-hostable; point `baseURL` at your own deployment to keep the same adapter surface.

Need a header on each request, or a log of each model call? This adapter supports [`wrapFetch`](../advanced/middleware#change-the-http-requests-of-a-call).

## Available Models

Any model listed at [llmgateway.io/models](https://llmgateway.io/models) works — pass its id as the model name. A bare model id lets the gateway route to the best available provider; prefix it with `provider/` to pin routing to a specific provider:

```text
model: "gpt-5.6-terra"          // gateway picks the provider
model: "claude-sonnet-5"        // gateway picks the provider
model: "moonshot/kimi-k3"       // always routed to Moonshot
model: "fireworks/kimi-k3"      // always routed to Fireworks
```

A curated set of flagship models (see `LLMGATEWAY_CHAT_MODELS`) additionally carries per-model type metadata — input modalities and provider options — with editor autocomplete. Uncurated ids still work and fall back to text-only input with the generic options.

`adapter.inputModalities` lists the input kinds of the selected model. It is `undefined` for a model that this package does not know. See [Check what a model accepts](../advanced/extend-adapter#check-what-a-model-accepts).

## Example: Chat Completion

```typescript
import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
  });

  return toServerSentEventsResponse(stream);
}
```

## Example: With Tools

```typescript
import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";
import { z } from "zod";

const getWeatherDef = toolDefinition({
  name: "get_weather",
  description: "Get the current weather",
  inputSchema: z.object({
    location: z.string(),
  }),
});

const getWeather = getWeatherDef.server(async ({ location }) => {
  return { temperature: 72, conditions: "sunny" };
});

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    tools: [getWeather],
  });

  return toServerSentEventsResponse(stream);
}
```

## Model Options

The gateway accepts the standard Chat Completions parameters and forwards them to the routed provider (parameters a provider doesn't support are stripped server-side). Sampling parameters live in `modelOptions`:

```typescript
import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Hello!" }],
  modelOptions: {
    temperature: 0.7,
    max_completion_tokens: 4096,
  },
  reasoning: "high",
});
```

### Reasoning

`reasoning` goes out as `reasoning_effort`. The types take the levels of each model, and no `budgetTokens`:

- `kimi-k3`, `glm-5.2`, `deepseek-v4-pro`, `qwen3.7-max`: `off` to `max`.
- `gpt-5.6-terra`: `off`, `low` to `max`. `gpt-5.5`, `gpt-5.4-mini`: `off`, `low` to `xhigh`.
- `claude-opus-5`, `claude-sonnet-5`, `claude-haiku-4-5`, `minimax-m2.5`: `low` to `max`.
- `gemini-pro-latest`, `grok-4-5`: `low` to `high`. `gemini-3.6-flash`: `minimal` to `high`.

Reasoning models stream their thinking as `reasoning_content` deltas, which the adapter surfaces as AG-UI `REASONING_*` events.

## Summarization

```typescript
import { summarize } from "@tanstack/ai";
import { llmGatewaySummarize } from "@tanstack/ai-llmgateway";

const result = await summarize({
  adapter: llmGatewaySummarize("gpt-5.4-mini"),
  text: "Long article text...",
  stream: false,
});

console.log(result.summary);
```

## Environment Variables

Set your API key in environment variables:

```bash
LLM_GATEWAY_API_KEY=llmgtwy_your-api-key
```

Get an API key from the [LLM Gateway dashboard](https://llmgateway.io).