mirror of
https://github.com/fosrl/pangolin.git
synced 2026-08-12 15:30:53 +02:00
improve model provider selection algorithm
This commit is contained in:
@@ -0,0 +1,285 @@
|
||||
# AI Gateway Provider Selection
|
||||
|
||||
How the AI gateway picks which attached provider handles a request when an
|
||||
inference resource has more than one AI provider.
|
||||
|
||||
**Code:**
|
||||
|
||||
- Route → capability binding: `server/routers/aiGateway/createAiGatewayRouter.ts`
|
||||
- Request pipeline: `server/routers/aiGateway/pipeline.ts` (`selectProvider`)
|
||||
- Tie-break scoring: `server/lib/aiProviderSelection.ts`
|
||||
- Allow/block matching: `server/lib/aiModelKeyMatch.ts`
|
||||
- Model catalog: `server/lib/aiModelCatalog.ts`
|
||||
- Default capabilities per provider type: `server/lib/aiProviderDefaults.ts`
|
||||
|
||||
Overlapping model allows are permitted at save time. Selection happens at
|
||||
request time. If the algorithm cannot confidently pick one provider, the
|
||||
gateway returns `403` with an ambiguous-provider error.
|
||||
|
||||
## Selection Pipeline
|
||||
|
||||
Every gateway request runs through these steps in order. Each step narrows
|
||||
the candidate set. Later steps only run when more than one provider remains.
|
||||
|
||||
```
|
||||
1. Capability filter
|
||||
2. Allow / block lists
|
||||
3. Most specific allow pattern
|
||||
4. Catalog ownership
|
||||
5. Provider class preference
|
||||
6. Ambiguous → error
|
||||
```
|
||||
|
||||
### 1. Capability Filter
|
||||
|
||||
The incoming path selects a capability before any provider logic runs.
|
||||
|
||||
| Path | Capability |
|
||||
|------|------------|
|
||||
| `POST /v1/chat/completions` | `openai_chat` |
|
||||
| `POST /v1/responses` | `openai_responses` |
|
||||
| `POST /v1/messages` | `anthropic_messages` |
|
||||
| Gemini / Vertex / Bedrock routes | their respective capability ids |
|
||||
|
||||
Only attached providers that advertise that capability stay in the candidate
|
||||
set. Default capabilities do not overlap for native OpenAI vs Anthropic:
|
||||
|
||||
| Provider type | Default capabilities |
|
||||
|---------------|----------------------|
|
||||
| `openai` | `openai_chat`, `openai_responses` |
|
||||
| `anthropic` | `anthropic_messages` |
|
||||
| `openRouter` | `openai_chat` |
|
||||
| `vercelAiGateway` | `openai_chat`, `openai_responses` |
|
||||
| `microsoftFoundry` | `openai_chat`, `openai_responses`, `anthropic_messages` |
|
||||
| `custom` | whatever was configured |
|
||||
|
||||
### 2. Allow / Block Lists
|
||||
|
||||
For each remaining provider, the gateway resolves the effective allow and
|
||||
block patterns:
|
||||
|
||||
- **`inherit`**: use the provider's own model lists
|
||||
- **`select`**: use the resource-selected subset of those lists
|
||||
|
||||
A candidate is kept only if `isAllowedByLists(requestedModel, allows, blocks)`
|
||||
passes:
|
||||
|
||||
1. At least one allow pattern must match
|
||||
2. No block pattern may match
|
||||
|
||||
Patterns support `*` and `?` globs (`gpt-*`, `claude-3-5-sonnet-?`).
|
||||
|
||||
### 3. Most Specific Allow Pattern
|
||||
|
||||
Among providers that allow the model, keep those whose matching allow
|
||||
pattern is most specific:
|
||||
|
||||
1. Exact keys beat patterns
|
||||
2. Fewer wildcard characters win
|
||||
3. Longer literal length wins
|
||||
|
||||
Example: `gpt-4o` beats `gpt-*` beats `*`.
|
||||
|
||||
### 4. Catalog Ownership
|
||||
|
||||
When specificity is tied (common with multiple `*` allows), score each
|
||||
provider against the known model catalog:
|
||||
|
||||
| Score | Meaning |
|
||||
|------:|---------|
|
||||
| 2 | Typed provider whose catalog contains the model (`openai` → openai catalog, `anthropic` → anthropic, etc.) |
|
||||
| 1 | Aggregator or custom (`openRouter`, `vercelAiGateway`, `custom`) and the model exists somewhere in the catalog |
|
||||
| 0 | No ownership signal (typed catalog miss, or unknown model on aggregator/custom) |
|
||||
|
||||
Model id lookup tries the raw id, then a stripped `vendor/model` form
|
||||
(e.g. `openai/gpt-4o` → also try `gpt-4o`).
|
||||
|
||||
Typed providers map to catalog providers as:
|
||||
|
||||
| Provider type | Catalog |
|
||||
|---------------|---------|
|
||||
| `openai` | `openai` |
|
||||
| `anthropic` | `anthropic` |
|
||||
| `googleGemini` | `gemini` |
|
||||
| `vertexAi` | `vertex` |
|
||||
| `bedrock` | `bedrock` |
|
||||
| `microsoftFoundry` | `azure` |
|
||||
| `openRouter` / `vercelAiGateway` / `custom` | none (aggregator/custom path) |
|
||||
|
||||
### 5. Provider Class Preference
|
||||
|
||||
If catalog ownership is still tied, prefer:
|
||||
|
||||
| Rank | Class |
|
||||
|-----:|-------|
|
||||
| 2 | Native typed provider (`openai`, `anthropic`, `googleGemini`, …) |
|
||||
| 1 | Aggregator (`openRouter`, `vercelAiGateway`) |
|
||||
| 0 | `custom` |
|
||||
|
||||
### 6. Ambiguous Error
|
||||
|
||||
If more than one distinct provider remains after all steps, the gateway
|
||||
rejects the request:
|
||||
|
||||
```
|
||||
Model "<id>" is ambiguous across multiple AI providers on this resource
|
||||
```
|
||||
|
||||
Typical remaining ties: two OpenAI-type providers both with `*`, or two
|
||||
customs advertising the same capability for an unknown model.
|
||||
|
||||
## Examples
|
||||
|
||||
Assume each provider below is attached and enabled on the same inference
|
||||
resource.
|
||||
|
||||
### Example A: OpenAI + Anthropic, Both `*`
|
||||
|
||||
| Provider | Allow | Capabilities |
|
||||
|----------|-------|--------------|
|
||||
| OpenAI | `*` | `openai_chat`, `openai_responses` |
|
||||
| Anthropic | `*` | `anthropic_messages` |
|
||||
|
||||
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
|
||||
|
||||
1. Capability → only OpenAI remains
|
||||
2. Allow → OpenAI matches `*`
|
||||
3. Result → **OpenAI**
|
||||
|
||||
Anthropic never reaches pattern or catalog scoring. Capability alone decides.
|
||||
|
||||
**Request:** `POST /v1/messages` with `model: "claude-3-5-sonnet-latest"`
|
||||
|
||||
1. Capability → only Anthropic remains
|
||||
2. Result → **Anthropic**
|
||||
|
||||
### Example B: OpenAI + OpenRouter, Both `*`
|
||||
|
||||
| Provider | Allow | Capabilities |
|
||||
|----------|-------|--------------|
|
||||
| OpenAI | `*` | `openai_chat`, … |
|
||||
| OpenRouter | `*` | `openai_chat` |
|
||||
|
||||
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
|
||||
|
||||
1. Capability → both remain (`openai_chat`)
|
||||
2. Allow → both match `*`
|
||||
3. Specificity → tie (`*` vs `*`)
|
||||
4. Catalog → OpenAI scores `2` (owns `gpt-4o`); OpenRouter scores `1`
|
||||
5. Result → **OpenAI**
|
||||
|
||||
### Example C: OpenRouter Only Serving a Claude Model Over OpenAI Chat
|
||||
|
||||
| Provider | Allow | Capabilities |
|
||||
|----------|-------|--------------|
|
||||
| OpenRouter | `*` | `openai_chat` |
|
||||
|
||||
**Request:** `POST /v1/chat/completions` with `model: "anthropic/claude-3.5-sonnet"`
|
||||
|
||||
1. Capability → OpenRouter remains
|
||||
2. Only one candidate → **OpenRouter**
|
||||
|
||||
No tie-breaking needed.
|
||||
|
||||
### Example D: OpenAI (`gpt-*`) + OpenRouter (`*`)
|
||||
|
||||
| Provider | Allow |
|
||||
|----------|-------|
|
||||
| OpenAI | `gpt-*` |
|
||||
| OpenRouter | `*` |
|
||||
|
||||
**Request:** `model: "gpt-4o"` on `openai_chat`
|
||||
|
||||
1. Capability → both
|
||||
2. Allow → both match
|
||||
3. Specificity → OpenAI's `gpt-*` beats OpenRouter's `*`
|
||||
4. Result → **OpenAI**
|
||||
|
||||
Catalog scoring is not needed because specificity already unique'd the set.
|
||||
|
||||
### Example E: OpenAI + Anthropic With Overlapping Custom Capabilities
|
||||
|
||||
Someone grants Anthropic `openai_chat` as well (non-default).
|
||||
|
||||
| Provider | Allow | Capabilities |
|
||||
|----------|-------|--------------|
|
||||
| OpenAI | `*` | `openai_chat`, … |
|
||||
| Anthropic | `*` | `anthropic_messages`, `openai_chat` |
|
||||
|
||||
**Request:** `POST /v1/chat/completions` with `model: "gpt-4o"`
|
||||
|
||||
1. Capability → both remain
|
||||
2. Allow → both match `*`
|
||||
3. Specificity → tie
|
||||
4. Catalog → OpenAI `2`, Anthropic `0` (`gpt-4o` is not in the anthropic catalog)
|
||||
5. Result → **OpenAI**
|
||||
|
||||
### Example F: Two Aggregators, Known Model
|
||||
|
||||
| Provider | Allow |
|
||||
|----------|-------|
|
||||
| OpenRouter | `*` |
|
||||
| Vercel AI Gateway | `*` |
|
||||
|
||||
**Request:** `model: "gpt-4o"` on `openai_chat`
|
||||
|
||||
1. Capability → both
|
||||
2. Allow / specificity → tie
|
||||
3. Catalog → both score `1` (known model, no typed owner in the set)
|
||||
4. Class → both aggregators (rank `1`) → still tied
|
||||
5. Result → **ambiguous error**
|
||||
|
||||
Attach a native OpenAI provider (or narrow one aggregator's allow list) to
|
||||
make this determinable.
|
||||
|
||||
### Example G: Two OpenAI Providers, Both `*`
|
||||
|
||||
| Provider | Type | Allow |
|
||||
|----------|------|-------|
|
||||
| OpenAI Prod | `openai` | `*` |
|
||||
| OpenAI Staging | `openai` | `*` |
|
||||
|
||||
**Request:** `model: "gpt-4o"`
|
||||
|
||||
1–5 all leave both candidates (same capability, same specificity, same
|
||||
catalog ownership, same class).
|
||||
|
||||
Result → **ambiguous error**
|
||||
|
||||
Disambiguate with different allow patterns, disable one attachment, or
|
||||
split across resources.
|
||||
|
||||
### Example H: Unknown Model Across Native + Aggregator
|
||||
|
||||
| Provider | Allow |
|
||||
|----------|-------|
|
||||
| OpenAI | `*` |
|
||||
| OpenRouter | `*` |
|
||||
|
||||
**Request:** `model: "my-fine-tune-v3"` (not in catalog)
|
||||
|
||||
1. Capability → both
|
||||
2. Allow / specificity → tie
|
||||
3. Catalog → both score `0` (typed miss + unknown aggregator model)
|
||||
4. Class → OpenAI (`2`) beats OpenRouter (`1`)
|
||||
5. Result → **OpenAI**
|
||||
|
||||
## Practical Guidance
|
||||
|
||||
- Native OpenAI + Anthropic with `*` is safe. Different default APIs never
|
||||
collide.
|
||||
- OpenAI + OpenRouter with `*` is usually fine for catalog-known OpenAI
|
||||
models. Native wins.
|
||||
- Prefer specific allow patterns (`gpt-4o`, `gpt-*`) when two providers share
|
||||
a capability.
|
||||
- Two providers of the same type both using `*` will stay ambiguous. Narrow
|
||||
at least one allow list.
|
||||
- Custom providers only win ties when no stronger native/aggregator signal
|
||||
remains.
|
||||
|
||||
## Related Behavior
|
||||
|
||||
- **Saving providers on a resource does not reject overlapping allows.**
|
||||
Collisions are resolved (or rejected) per request.
|
||||
- Budgets, auth, and upstream URL / target routing run after a single
|
||||
provider has been selected.
|
||||
@@ -10,7 +10,6 @@ import {
|
||||
type Transaction
|
||||
} from "@server/db";
|
||||
import { z } from "zod";
|
||||
import { modelKeysConflict } from "@server/lib/aiModelKeyMatch";
|
||||
|
||||
type DbOrTrx = Transaction | typeof db;
|
||||
|
||||
@@ -100,142 +99,6 @@ function normalizeAttachments(
|
||||
);
|
||||
}
|
||||
|
||||
type EffectiveAllowRow = {
|
||||
providerId: number;
|
||||
modelKey: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* Ensure effective allow modelKeys do not conflict across attached providers.
|
||||
* inherit uses provider allows; select uses resource-selected allows (or the
|
||||
* optional override map). Block patterns are ignored for overlap checks.
|
||||
*/
|
||||
export async function assertNoOverlappingModelKeys(
|
||||
attachments: ResourceAiProviderAttachment[],
|
||||
options: {
|
||||
trx?: DbOrTrx;
|
||||
resourceId?: number;
|
||||
siteResourceId?: number;
|
||||
selectedAllowsByProvider?: Map<number, string[]>;
|
||||
} = {}
|
||||
): Promise<InferenceFieldsError | null> {
|
||||
const trx = options.trx ?? db;
|
||||
|
||||
const activeAttachments = attachments.filter((a) => a.enabled);
|
||||
|
||||
if (activeAttachments.length < 2) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const inheritProviderIds = activeAttachments
|
||||
.filter((a) => a.accessMode === "inherit")
|
||||
.map((a) => a.providerId);
|
||||
const selectProviderIds = activeAttachments
|
||||
.filter((a) => a.accessMode === "select")
|
||||
.map((a) => a.providerId);
|
||||
|
||||
const effectiveAllows: EffectiveAllowRow[] = [];
|
||||
|
||||
if (inheritProviderIds.length > 0) {
|
||||
const providerAllows = await trx
|
||||
.select({
|
||||
providerId: aiModels.providerId,
|
||||
modelKey: aiModels.modelKey
|
||||
})
|
||||
.from(aiModels)
|
||||
.where(
|
||||
and(
|
||||
inArray(aiModels.providerId, inheritProviderIds),
|
||||
eq(aiModels.enabled, true),
|
||||
eq(aiModels.listType, "allow")
|
||||
)
|
||||
);
|
||||
effectiveAllows.push(...providerAllows);
|
||||
}
|
||||
|
||||
if (selectProviderIds.length > 0) {
|
||||
if (options.selectedAllowsByProvider) {
|
||||
for (const providerId of selectProviderIds) {
|
||||
const keys =
|
||||
options.selectedAllowsByProvider.get(providerId) ?? [];
|
||||
for (const modelKey of keys) {
|
||||
effectiveAllows.push({ providerId, modelKey });
|
||||
}
|
||||
}
|
||||
} else if (options.resourceId !== undefined) {
|
||||
const rows = await trx
|
||||
.select({
|
||||
providerId: aiModels.providerId,
|
||||
modelKey: aiModels.modelKey
|
||||
})
|
||||
.from(resourceAiModels)
|
||||
.innerJoin(
|
||||
aiModels,
|
||||
eq(resourceAiModels.modelId, aiModels.modelId)
|
||||
)
|
||||
.where(
|
||||
and(
|
||||
eq(resourceAiModels.resourceId, options.resourceId),
|
||||
inArray(aiModels.providerId, selectProviderIds),
|
||||
eq(resourceAiModels.listType, "allow"),
|
||||
eq(aiModels.enabled, true)
|
||||
)
|
||||
);
|
||||
effectiveAllows.push(...rows);
|
||||
} else if (options.siteResourceId !== undefined) {
|
||||
const rows = await trx
|
||||
.select({
|
||||
providerId: aiModels.providerId,
|
||||
modelKey: aiModels.modelKey
|
||||
})
|
||||
.from(siteResourceAiModels)
|
||||
.innerJoin(
|
||||
aiModels,
|
||||
eq(siteResourceAiModels.modelId, aiModels.modelId)
|
||||
)
|
||||
.where(
|
||||
and(
|
||||
eq(
|
||||
siteResourceAiModels.siteResourceId,
|
||||
options.siteResourceId
|
||||
),
|
||||
inArray(aiModels.providerId, selectProviderIds),
|
||||
eq(siteResourceAiModels.listType, "allow"),
|
||||
eq(aiModels.enabled, true)
|
||||
)
|
||||
);
|
||||
effectiveAllows.push(...rows);
|
||||
}
|
||||
}
|
||||
|
||||
const conflictPairs: string[] = [];
|
||||
for (let i = 0; i < effectiveAllows.length; i++) {
|
||||
for (let j = i + 1; j < effectiveAllows.length; j++) {
|
||||
const left = effectiveAllows[i];
|
||||
const right = effectiveAllows[j];
|
||||
if (left.providerId === right.providerId) {
|
||||
continue;
|
||||
}
|
||||
if (!modelKeysConflict(left.modelKey, right.modelKey)) {
|
||||
continue;
|
||||
}
|
||||
const pair = [left.modelKey, right.modelKey].sort().join(" vs ");
|
||||
if (!conflictPairs.includes(pair)) {
|
||||
conflictPairs.push(pair);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (conflictPairs.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
conflictPairs.sort();
|
||||
return {
|
||||
error: `Model keys must be unique across providers on a resource. Overlapping keys: ${conflictPairs.join(", ")}`
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate provider attachments for an org.
|
||||
*/
|
||||
@@ -243,8 +106,6 @@ export async function resolveProviderAttachments(input: {
|
||||
orgId: string;
|
||||
attachments: ResourceAiProviderInput[];
|
||||
requireAtLeastOne: boolean;
|
||||
resourceId?: number;
|
||||
siteResourceId?: number;
|
||||
}): Promise<ResourceAiProviderAttachment[] | InferenceFieldsError> {
|
||||
const attachments = normalizeAttachments(input.attachments);
|
||||
|
||||
@@ -286,14 +147,6 @@ export async function resolveProviderAttachments(input: {
|
||||
};
|
||||
}
|
||||
|
||||
const overlapError = await assertNoOverlappingModelKeys(attachments, {
|
||||
resourceId: input.resourceId,
|
||||
siteResourceId: input.siteResourceId
|
||||
});
|
||||
if (overlapError) {
|
||||
return overlapError;
|
||||
}
|
||||
|
||||
return attachments;
|
||||
}
|
||||
|
||||
@@ -830,7 +683,6 @@ async function assertModelEntriesValid(input: {
|
||||
const catalogRows = await db
|
||||
.select({
|
||||
modelId: aiModels.modelId,
|
||||
modelKey: aiModels.modelKey,
|
||||
listType: aiModels.listType,
|
||||
providerId: aiModels.providerId,
|
||||
enabled: aiModels.enabled
|
||||
@@ -850,7 +702,6 @@ async function assertModelEntriesValid(input: {
|
||||
}
|
||||
|
||||
const catalogById = new Map(catalogRows.map((row) => [row.modelId, row]));
|
||||
const selectedAllowsByProvider = new Map<number, string[]>();
|
||||
for (const entry of input.modelEntries) {
|
||||
const catalog = catalogById.get(entry.modelId);
|
||||
if (!catalog) {
|
||||
@@ -862,18 +713,6 @@ async function assertModelEntriesValid(input: {
|
||||
if (!catalog.enabled) {
|
||||
return `Model ${entry.modelId} is disabled on its provider`;
|
||||
}
|
||||
if (entry.listType === "allow") {
|
||||
const keys = selectedAllowsByProvider.get(catalog.providerId) ?? [];
|
||||
keys.push(catalog.modelKey);
|
||||
selectedAllowsByProvider.set(catalog.providerId, keys);
|
||||
}
|
||||
}
|
||||
|
||||
const overlapError = await assertNoOverlappingModelKeys(input.attachments, {
|
||||
selectedAllowsByProvider
|
||||
});
|
||||
if (overlapError) {
|
||||
return overlapError.error;
|
||||
}
|
||||
|
||||
return null;
|
||||
|
||||
@@ -61,29 +61,6 @@ export function compareModelKeySpecificity(a: string, b: string): number {
|
||||
return literalLength(b) - literalLength(a);
|
||||
}
|
||||
|
||||
/**
|
||||
* Attach-time conflict check. Detects identical keys and exact-vs-pattern
|
||||
* matches. Does not attempt full glob intersection.
|
||||
*/
|
||||
export function modelKeysConflict(a: string, b: string): boolean {
|
||||
if (a === b) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const aIsPattern = isModelKeyPattern(a);
|
||||
const bIsPattern = isModelKeyPattern(b);
|
||||
|
||||
if (aIsPattern === bIsPattern) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (aIsPattern) {
|
||||
return modelKeyMatches(a, b);
|
||||
}
|
||||
|
||||
return modelKeyMatches(b, a);
|
||||
}
|
||||
|
||||
/**
|
||||
* Provider-layer policy: empty allowlist denies all. Blocklist only applies
|
||||
* after an allow match.
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import {
|
||||
aiModelCatalog,
|
||||
getCatalogProviderForType,
|
||||
type CatalogProvider
|
||||
} from "@server/lib/aiModelCatalog";
|
||||
import type { AiProviderType } from "@server/lib/aiProviderDefaults";
|
||||
|
||||
function stripVendorPrefix(modelId: string): string | null {
|
||||
const idx = modelId.indexOf("/");
|
||||
if (idx === -1 || idx === modelId.length - 1) {
|
||||
return null;
|
||||
}
|
||||
return modelId.slice(idx + 1);
|
||||
}
|
||||
|
||||
function modelKeysToTry(modelId: string): string[] {
|
||||
const keys = [modelId];
|
||||
const stripped = stripVendorPrefix(modelId);
|
||||
if (stripped) {
|
||||
keys.push(stripped);
|
||||
}
|
||||
return keys;
|
||||
}
|
||||
|
||||
function catalogOwnsModel(
|
||||
catalogProvider: CatalogProvider,
|
||||
modelId: string
|
||||
): boolean {
|
||||
for (const key of modelKeysToTry(modelId)) {
|
||||
if (aiModelCatalog.get(catalogProvider, key)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function modelKnownInAnyCatalog(modelId: string): boolean {
|
||||
for (const key of modelKeysToTry(modelId)) {
|
||||
if (aiModelCatalog.listByKey(key).length > 0) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* How strongly a provider "owns" a requested model id via the known catalog.
|
||||
*
|
||||
* 2 - Typed provider whose catalog contains the model
|
||||
* 1 - Aggregator/custom that can proxy a catalog-known model
|
||||
* 0 - No ownership signal (typed miss, or unknown model on aggregator/custom)
|
||||
*/
|
||||
export function catalogOwnershipScore(
|
||||
type: AiProviderType,
|
||||
modelId: string
|
||||
): number {
|
||||
const catalogProvider = getCatalogProviderForType(type);
|
||||
if (catalogProvider != null) {
|
||||
return catalogOwnsModel(catalogProvider, modelId) ? 2 : 0;
|
||||
}
|
||||
return modelKnownInAnyCatalog(modelId) ? 1 : 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Prefer native vendor providers over aggregators over custom when catalog
|
||||
* ownership is tied.
|
||||
*
|
||||
* 2 - Native typed provider (openai, anthropic, gemini, ...)
|
||||
* 1 - Aggregator gateway (openRouter, vercelAiGateway)
|
||||
* 0 - Custom
|
||||
*/
|
||||
export function providerClassRank(type: AiProviderType): number {
|
||||
if (type === "custom") {
|
||||
return 0;
|
||||
}
|
||||
if (type === "openRouter" || type === "vercelAiGateway") {
|
||||
return 1;
|
||||
}
|
||||
return 2;
|
||||
}
|
||||
|
||||
export function keepBestScored<T>(
|
||||
items: T[],
|
||||
scoreFn: (item: T) => number
|
||||
): T[] {
|
||||
if (items.length <= 1) {
|
||||
return items;
|
||||
}
|
||||
let best = Number.NEGATIVE_INFINITY;
|
||||
for (const item of items) {
|
||||
const score = scoreFn(item);
|
||||
if (score > best) {
|
||||
best = score;
|
||||
}
|
||||
}
|
||||
return items.filter((item) => scoreFn(item) === best);
|
||||
}
|
||||
@@ -50,6 +50,11 @@ import {
|
||||
isAllowedByLists,
|
||||
mostSpecificMatchingAllow
|
||||
} from "@server/lib/aiModelKeyMatch";
|
||||
import {
|
||||
catalogOwnershipScore,
|
||||
keepBestScored,
|
||||
providerClassRank
|
||||
} from "@server/lib/aiProviderSelection";
|
||||
import { aiGatewayUpstreamFetch } from "@server/lib/aiGatewayUpstreamFetch";
|
||||
import { getModelPricing, calculateAiCost } from "@server/lib/aiModelPricing";
|
||||
import {
|
||||
@@ -502,17 +507,28 @@ async function selectProvider(
|
||||
};
|
||||
}
|
||||
|
||||
// 1) Prefer the most specific allow pattern that matched the request.
|
||||
candidates.sort((a, b) =>
|
||||
compareModelKeySpecificity(a.modelKey, b.modelKey)
|
||||
);
|
||||
|
||||
const bestSpecificity = candidates[0].modelKey;
|
||||
const topCandidates = candidates.filter(
|
||||
let remaining = candidates.filter(
|
||||
(c) => compareModelKeySpecificity(c.modelKey, bestSpecificity) === 0
|
||||
);
|
||||
|
||||
// 2) Prefer providers whose catalog owns this model id. Aggregators only
|
||||
// score when the model is known somewhere in the catalog.
|
||||
remaining = keepBestScored(remaining, (c) =>
|
||||
catalogOwnershipScore(c.provider.type as AiProviderType, requestedModel)
|
||||
);
|
||||
|
||||
// 3) Prefer native typed providers over aggregators over custom.
|
||||
remaining = keepBestScored(remaining, (c) =>
|
||||
providerClassRank(c.provider.type as AiProviderType)
|
||||
);
|
||||
|
||||
const uniqueProviders = new Map<number, AiProvider>();
|
||||
for (const candidate of topCandidates) {
|
||||
for (const candidate of remaining) {
|
||||
uniqueProviders.set(candidate.provider.providerId, candidate.provider);
|
||||
}
|
||||
|
||||
@@ -523,7 +539,7 @@ async function selectProvider(
|
||||
return {
|
||||
ok: false,
|
||||
status: HttpCode.FORBIDDEN,
|
||||
message: `Model "${requestedModel}" is ambiguous across multiple AI providers on this resource`
|
||||
message: `Model "${requestedModel}" is ambiguous across multiple AI providers on this resource. Ask your administrator to configure a more specific allow pattern for this model.`
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -131,8 +131,7 @@ export async function addAiProviderToResource(
|
||||
const attachments = await resolveProviderAttachments({
|
||||
orgId: resource.orgId,
|
||||
attachments: nextAttachments,
|
||||
requireAtLeastOne: true,
|
||||
resourceId
|
||||
requireAtLeastOne: true
|
||||
});
|
||||
if (isInferenceFieldsError(attachments)) {
|
||||
return next(
|
||||
|
||||
@@ -134,8 +134,7 @@ export async function removeAiProviderFromResource(
|
||||
const attachments = await resolveProviderAttachments({
|
||||
orgId: resource.orgId,
|
||||
attachments: remaining,
|
||||
requireAtLeastOne: false,
|
||||
resourceId
|
||||
requireAtLeastOne: false
|
||||
});
|
||||
if (isInferenceFieldsError(attachments)) {
|
||||
return next(
|
||||
|
||||
@@ -113,8 +113,7 @@ export async function setResourceAiProviders(
|
||||
const attachments = await resolveProviderAttachments({
|
||||
orgId: resource.orgId,
|
||||
attachments: providers,
|
||||
requireAtLeastOne: false,
|
||||
resourceId
|
||||
requireAtLeastOne: false
|
||||
});
|
||||
if (isInferenceFieldsError(attachments)) {
|
||||
return next(
|
||||
|
||||
@@ -131,8 +131,7 @@ export async function addAiProviderToSiteResource(
|
||||
const attachments = await resolveProviderAttachments({
|
||||
orgId: siteResource.orgId,
|
||||
attachments: nextAttachments,
|
||||
requireAtLeastOne: true,
|
||||
siteResourceId
|
||||
requireAtLeastOne: true
|
||||
});
|
||||
if (isInferenceFieldsError(attachments)) {
|
||||
return next(
|
||||
|
||||
@@ -133,8 +133,7 @@ export async function removeAiProviderFromSiteResource(
|
||||
const attachments = await resolveProviderAttachments({
|
||||
orgId: siteResource.orgId,
|
||||
attachments: remaining,
|
||||
requireAtLeastOne: false,
|
||||
siteResourceId
|
||||
requireAtLeastOne: false
|
||||
});
|
||||
if (isInferenceFieldsError(attachments)) {
|
||||
return next(
|
||||
|
||||
@@ -115,8 +115,7 @@ export async function setSiteResourceAiProviders(
|
||||
const attachments = await resolveProviderAttachments({
|
||||
orgId: siteResource.orgId,
|
||||
attachments: providers,
|
||||
requireAtLeastOne: false,
|
||||
siteResourceId
|
||||
requireAtLeastOne: false
|
||||
});
|
||||
if (isInferenceFieldsError(attachments)) {
|
||||
return next(
|
||||
|
||||
Reference in New Issue
Block a user