pangolin/docs/ai-gateway-provider-selection.md

# AI Gateway Provider Selection

How the AI gateway picks which attached provider handles a request when an
inference resource has more than one AI provider.

**Code:**

- Route → capability binding: `server/routers/aiGateway/createAiGatewayRouter.ts`
- Request pipeline: `server/routers/aiGateway/pipeline.ts` (`selectProvider`)
- Model discovery: `server/routers/aiGateway/v1Models.ts` and
  `server/lib/aiModelDiscovery.ts`
- Tie-break scoring: `server/lib/aiProviderSelection.ts`
- Allow/block matching: `server/lib/aiModelKeyMatch.ts`
- Model catalog: `server/lib/aiModelCatalog.ts`
- Default capabilities per provider type: `server/lib/aiProviderDefaults.ts`

Overlapping model allows are permitted at save time. Selection happens at
request time. If the algorithm cannot confidently pick one provider, the
gateway returns `403` with an ambiguous-provider error.

## Selection Pipeline

Every gateway request runs through these steps in order. Each step narrows
the candidate set. Later steps only run when more than one provider remains.

```
1. Capability filter
2. Allow / block lists
3. Most specific allow pattern
4. Catalog ownership
5. Provider class preference
6. Ambiguous → error
```

### 1. Capability Filter

The incoming path selects a capability before any provider logic runs.

| Path | Capability |
|------|------------|
| `POST /v1/chat/completions` | `openai_chat` |
| `POST /v1/responses` | `openai_responses` |
| `POST /v1/messages` | `anthropic_messages` |
| `GET /v1/models`, `GET /v1/models/{id}` | `v1_models` |
| Gemini / Vertex / Bedrock routes | their respective capability ids |

Only attached providers that advertise that capability stay in the candidate
set. Default capabilities do not overlap for native OpenAI vs Anthropic:

| Provider type | Default capabilities |
|---------------|----------------------|
| `openai` | `openai_chat`, `openai_responses` |
| `anthropic` | `anthropic_messages`, `v1_models` |
| `openRouter` | `openai_chat` |
| `vercelAiGateway` | `openai_chat`, `openai_responses` |
| `microsoftFoundry` | `openai_chat`, `openai_responses`, `anthropic_messages`, `v1_models` |
| `custom` | whatever was configured |

### 2. Allow / Block Lists

For each remaining provider, the gateway resolves the effective allow and
block patterns:

- **`inherit`**: use the provider's own model lists
- **`select`**: use the resource-selected subset of those lists

A candidate is kept only if `isAllowedByLists(requestedModel, allows, blocks)`
passes:

1. At least one allow pattern must match
2. No block pattern may match

Patterns support `*` and `?` globs (`gpt-*`, `claude-3-5-sonnet-?`).

### 3. Most Specific Allow Pattern

Among providers that allow the model, keep those whose matching allow
pattern is most specific:

1. Exact keys beat patterns
2. Fewer wildcard characters win
3. Longer literal length wins

Example: `gpt-4o` beats `gpt-*` beats `*`.

### 4. Catalog Ownership

When specificity is tied (common with multiple `*` allows), score each
provider against the known model catalog:

| Score | Meaning |
|------:|---------|
| 2 | Typed provider whose catalog contains the model (`openai` → openai catalog, `anthropic` → anthropic, etc.) |
| 1 | Aggregator or custom (`openRouter`, `vercelAiGateway`, `custom`) and the model exists somewhere in the catalog |
| 0 | No ownership signal (typed catalog miss, or unknown model on aggregator/custom) |

Model id lookup tries the raw id, then a stripped `vendor/model` form
(e.g. `openai/gpt-4o` → also try `gpt-4o`).

Typed providers map to catalog providers as:

| Provider type | Catalog |
|---------------|---------|
| `openai` | `openai` |
| `anthropic` | `anthropic` |
| `googleGemini` | `gemini` |
| `vertexAi` | `vertex` |
| `bedrock` | `bedrock` |
| `microsoftFoundry` | `azure` |
| `openRouter` / `vercelAiGateway` / `custom` | none (aggregator/custom path) |

### 5. Provider Class Preference

If catalog ownership is still tied, prefer:

| Rank | Class |
|-----:|-------|
| 2 | Native typed provider (`openai`, `anthropic`, `googleGemini`, …) |
| 1 | Aggregator (`openRouter`, `vercelAiGateway`) |
| 0 | `custom` |

### 6. Ambiguous Error

If more than one distinct provider remains after all steps, the gateway
rejects the request:

```
Model "<id>" is ambiguous across multiple AI providers on this resource
```

Typical remaining ties: two OpenAI-type providers both with `*`, or two
customs advertising the same capability for an unknown model.

## Model Discovery Is Not Selection

`GET /v1/models` and `GET /v1/models/{id}` (`v1_models`) skip steps 3-6
entirely. There is no requested model to disambiguate on, so the gateway does
not pick one provider - it returns the **union** of what every attached
provider advertising `v1_models` would accept, deduplicated by model id
(lowest `providerId` wins a collision).

Discovery is answered from the gateway's own view of the allow/block lists,
never proxied upstream. Providers that expose no `/v1/models` endpoint of their
own still get a working listing, and a model an allow/block list forbids is
never advertised.

Each provider's candidate ids come from two places:

| Source | Contributes |
|--------|-------------|
| Exact (non-wildcard) allow entries | the model key itself |
| The model catalog for the provider's type | every catalog id matching an allow pattern |

Both sources are then filtered through the same
`isAllowedByLists(id, allows, blocks)` check step 2 applies, so a block pattern
hides a model from discovery exactly as it would reject it at request time.

The catalog source is what makes a wildcard allow such as `claude-*`
enumerable. Provider types with no catalog mapping (`openRouter`,
`vercelAiGateway`, `custom`) have nothing to expand against, so a wildcard
allow on those types lists nothing - **add exact allow entries to make their
models discoverable.**

### Where each field comes from

Token limits and capability flags can't be derived from an allow/block list.
They come from the model catalog (`server/lib/aiModelCatalog.ts`), which the
Fossorial API builds from LiteLLM:

| Field | Source |
|-------|--------|
| `max_input_tokens` | catalog `limits.input` |
| `max_tokens` | catalog `limits.output` |
| `capabilities` | catalog flags, mapped to the Models API shape by `capabilitiesFromCatalog` |
| `display_name` | the configured model row's name, else the model id |
| `created_at` | the configured model row's timestamp, else the epoch |

A model the catalog doesn't know (an exact allow entry for a fine-tune, say)
reports `null` for all three metadata fields. The Models API declares them
nullable, so that is a valid answer rather than a broken one.

The catalog's flags are coarser than the Models API describes: it carries a
single `reasoning` flag with no way to distinguish adaptive from
`budget_tokens`-style thinking, and nothing at all for batch, citations, code
execution, PDF input, or context management. Anything it reports as unknown
(`null`) is surfaced as unsupported rather than invented, so `capabilities`
understates rather than overstates what a model can do.

The gateway does **not** query the provider's own `/v1/models`. Discovery is
answered entirely from local state.

Results are ordered newest-first with the id as tie-break, and paginated with
Anthropic's `limit` / `after_id` / `before_id` semantics (default 20, max
1000).

## Examples

Assume each provider below is attached and enabled on the same inference
resource.

### Example A: OpenAI + Anthropic, Both `*`

| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenAI | `*` | `openai_chat`, `openai_responses` |
| Anthropic | `*` | `anthropic_messages` |

**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`

1. Capability → only OpenAI remains
2. Allow → OpenAI matches `*`
3. Result → **OpenAI**

Anthropic never reaches pattern or catalog scoring. Capability alone decides.

**Request:** `POST /v1/messages` with `model: "claude-3-5-sonnet-latest"`

1. Capability → only Anthropic remains
2. Result → **Anthropic**

### Example B: OpenAI + OpenRouter, Both `*`

| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenAI | `*` | `openai_chat`, … |
| OpenRouter | `*` | `openai_chat` |

**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`

1. Capability → both remain (`openai_chat`)
2. Allow → both match `*`
3. Specificity → tie (`*` vs `*`)
4. Catalog → OpenAI scores `2` (owns `gpt-4o`); OpenRouter scores `1`
5. Result → **OpenAI**

### Example C: OpenRouter Only Serving a Claude Model Over OpenAI Chat

| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenRouter | `*` | `openai_chat` |

**Request:** `POST /v1/chat/completions` with `model: "anthropic/claude-3.5-sonnet"`

1. Capability → OpenRouter remains
2. Only one candidate → **OpenRouter**

No tie-breaking needed.

### Example D: OpenAI (`gpt-*`) + OpenRouter (`*`)

| Provider | Allow |
|----------|-------|
| OpenAI | `gpt-*` |
| OpenRouter | `*` |

**Request:** `model: "gpt-4o"` on `openai_chat`

1. Capability → both
2. Allow → both match
3. Specificity → OpenAI's `gpt-*` beats OpenRouter's `*`
4. Result → **OpenAI**

Catalog scoring is not needed because specificity already unique'd the set.

### Example E: OpenAI + Anthropic With Overlapping Custom Capabilities

Someone grants Anthropic `openai_chat` as well (non-default).

| Provider | Allow | Capabilities |
|----------|-------|--------------|
| OpenAI | `*` | `openai_chat`, … |
| Anthropic | `*` | `anthropic_messages`, `openai_chat` |

**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`

1. Capability → both remain
2. Allow → both match `*`
3. Specificity → tie
4. Catalog → OpenAI `2`, Anthropic `0` (`gpt-4o` is not in the anthropic catalog)
5. Result → **OpenAI**

### Example F: Two Aggregators, Known Model

| Provider | Allow |
|----------|-------|
| OpenRouter | `*` |
| Vercel AI Gateway | `*` |

**Request:** `model: "gpt-4o"` on `openai_chat`

1. Capability → both
2. Allow / specificity → tie
3. Catalog → both score `1` (known model, no typed owner in the set)
4. Class → both aggregators (rank `1`) → still tied
5. Result → **ambiguous error**

Attach a native OpenAI provider (or narrow one aggregator's allow list) to
make this determinable.

### Example G: Two OpenAI Providers, Both `*`

| Provider | Type | Allow |
|----------|------|-------|
| OpenAI Prod | `openai` | `*` |
| OpenAI Staging | `openai` | `*` |

**Request:** `model: "gpt-4o"`

1–5 all leave both candidates (same capability, same specificity, same
catalog ownership, same class).

Result → **ambiguous error**

Disambiguate with different allow patterns, disable one attachment, or
split across resources.

### Example H: Unknown Model Across Native + Aggregator

| Provider | Allow |
|----------|-------|
| OpenAI | `*` |
| OpenRouter | `*` |

**Request:** `model: "my-fine-tune-v3"` (not in catalog)

1. Capability → both
2. Allow / specificity → tie
3. Catalog → both score `0` (typed miss + unknown aggregator model)
4. Class → OpenAI (`2`) beats OpenRouter (`1`)
5. Result → **OpenAI**

## Practical Guidance

- Native OpenAI + Anthropic with `*` is safe. Different default APIs never
  collide.
- OpenAI + OpenRouter with `*` is usually fine for catalog-known OpenAI
  models. Native wins.
- Prefer specific allow patterns (`gpt-4o`, `gpt-*`) when two providers share
  a capability.
- Two providers of the same type both using `*` will stay ambiguous. Narrow
  at least one allow list.
- Custom providers only win ties when no stronger native/aggregator signal
  remains.

## Related Behavior

- **Saving providers on a resource does not reject overlapping allows.**
  Collisions are resolved (or rejected) per request.
- Budgets, auth, and upstream URL / target routing run after a single
  provider has been selected.