Skip to content

Configure the Upstream Provider

This page covers everything about pointing the guardrails server at your LLM provider: choosing a provider (OpenAI, Azure OpenAI, OpenAI-compatible proxies, local Ollama), forwarding HTTP headers and query parameters, and routing requests to different upstreams with a multi-model config file.

For installing and starting the server, see Run the server.

Configuring the upstream provider

There are two base_url values in play — one on the client, one on the server — and they serve opposite roles:

Side Setting Points at
Client OpenAI(base_url=...) The guardrails server
Server OPENAI_BASE_URL env var (or models config) The real LLM provider

The full request path is:

Your app  ──►  Guardrails server (OPENAI_BASE_URL → real provider)  ──►  LLM
              ↑ applies guardrails here

The server defaults to https://api.openai.com/v1 when OPENAI_BASE_URL is not set and no matching model config entry exists.

API key forwarding: the Authorization: Bearer <key> header sent by your client is forwarded as-is to the upstream provider. You can therefore pass the real API key from the client and leave OPENAI_API_KEY unset on the server, or set OPENAI_API_KEY on the server and use a placeholder on the client.

OpenAI (default)

flowchart LR
    App["Your App\nOpenAI(base_url=localhost:8000)"]
    GS["Guardrails Server\n:8000"]
    OAI["OpenAI API\napi.openai.com/v1"]

    App -->|"POST /v1/chat/completions\nAuthorization: Bearer sk-..."| GS
    GS -->|"guardrails applied\nchat.completions.create()"| OAI
    OAI -->|"ChatCompletion"| GS
    GS -->|"200 JSON"| App
# OPENAI_BASE_URL not needed — server targets api.openai.com/v1 by default
OPENAI_API_KEY=sk-...   # set on the server, or pass from the client
mend-guardrails-server --policy-dir ./policies

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="any",  # forwarded to OpenAI — or set OPENAI_API_KEY on the server
)

OpenAI-compatible proxy

If your infrastructure routes LLM traffic through an OpenAI-compatible proxy that requires custom headers or query parameters, configure them via environment variables (simple sidecar) or a models config file (per-model routing):

flowchart LR
    App["Your App\nOpenAI(base_url=localhost:8000)"]
    GS["Guardrails Server\n:8000\nmodels.json → proxy config"]
    Proxy["Internal LLM Proxy\nmyproxy.com/openai\nx-my-custom-header + api-version"]
    LLM["Upstream LLM"]

    App -->|"POST /v1/chat/completions\nmodel: gpt-4o"| GS
    GS -->|"guardrails applied\ndefault_headers + default_query injected"| Proxy
    Proxy -->|"forwards to LLM"| LLM
    LLM -->|"ChatCompletion"| Proxy
    Proxy -->|"response"| GS
    GS -->|"200 JSON"| App
[
  {
    "name": "gpt-4o",
    "provider": "openai",
    "parameters": {
      "base_url": "https://api.myproxy.com/openai",
      "default_headers": { "x-my-custom-header": "myValue" },
      "default_query":   { "api-version": "2025-01-01" }
    }
  }
]

mend-guardrails-server \
  --models-config ./models.json \
  --policy-dir ./policies

Azure OpenAI

Single deployment

A single Azure deployment can be reached without a models config file by pointing OPENAI_BASE_URL at the deployment URL:

flowchart LR
    App["Your App\nOpenAI(base_url=localhost:8000)"]
    GS["Guardrails Server\n:8000\nOPENAI_BASE_URL → Azure endpoint"]
    AZ["Azure OpenAI\nresource.openai.azure.com\n/openai/deployments/my-dep"]

    App -->|"POST /v1/chat/completions"| GS
    GS -->|"guardrails applied\nMendGuardrailsAsyncOpenAI"| AZ
    AZ -->|"ChatCompletion"| GS
    GS -->|"200 JSON"| App
OPENAI_BASE_URL=https://<resource>.openai.azure.com/openai/deployments/<deployment> \
OPENAI_API_KEY=<azure-key> \
mend-guardrails-server --policy-dir ./policies

Multiple deployments

Using the models config file is the recommended way to connect to multiple Azure OpenAI endpoints, because it maps each Azure deployment to a model name and supports multiple deployments in the same server instance:

flowchart LR
    App["Your App\nOpenAI(base_url=localhost:8000)"]

    subgraph GS["Guardrails Server  :8000"]
        Router["model router\nmodels.json"]
        C1["client — gpt-4o\nMendGuardrailsAsyncAzureOpenAI"]
        C2["client — gpt-4o-mini\nMendGuardrailsAsyncAzureOpenAI"]
        C3["client — gpt-35-turbo\nMendGuardrailsAsyncAzureOpenAI"]
        Router --> C1
        Router --> C2
        Router --> C3
    end

    AZ1["Azure OpenAI\nresource-a.openai.azure.com\ndeployment: gpt4o-prod"]
    AZ2["Azure OpenAI\nresource-b.openai.azure.com\ndeployment: mini-prod"]
    AZ3["Azure OpenAI\nresource-a.openai.azure.com\ndeployment: gpt35-prod"]

    App -->|"model: gpt-4o"| Router
    App -->|"model: gpt-4o-mini"| Router
    App -->|"model: gpt-35-turbo"| Router

    C1 -->|"guardrails applied"| AZ1
    C2 -->|"guardrails applied"| AZ2
    C3 -->|"guardrails applied"| AZ3
[
  {
    "name": "gpt-4o",
    "provider": "azure",
    "parameters": {
      "azure_endpoint":   "https://resource-a.openai.azure.com",
      "azure_deployment": "gpt4o-prod",
      "api_version":      "2024-02-01",
      "api_key":          "azure-key-a"
    }
  },
  {
    "name": "gpt-4o-mini",
    "provider": "azure",
    "parameters": {
      "azure_endpoint":   "https://resource-b.openai.azure.com",
      "azure_deployment": "mini-prod",
      "api_version":      "2024-02-01",
      "api_key":          "azure-key-b"
    }
  },
  {
    "name": "gpt-35-turbo",
    "provider": "azure",
    "parameters": {
      "azure_endpoint":   "https://resource-a.openai.azure.com",
      "azure_deployment": "gpt35-prod",
      "api_version":      "2024-02-01",
      "api_key":          "azure-key-a"
    }
  }
]

The client selects the deployment by setting model to the name from the config file:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="any")

# Routes to resource-a / gpt4o-prod
client.chat.completions.create(model="gpt-4o", messages=[...])

# Routes to resource-b / mini-prod
client.chat.completions.create(model="gpt-4o-mini", messages=[...])

Azure-specific parameters can also be supplied (or overridden) via environment variables: AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, AZURE_OPENAI_DEPLOYMENT, AZURE_OPENAI_API_VERSION.

Native Azure deployment URL

The server also accepts requests on the Azure-native deployment path so that clients using the Azure OpenAI SDK — or any tool that constructs the URL directly — can point at the guardrail server:

POST /openai/deployments/{deployment}/chat/completions

The {deployment} path segment is used as the model name when no model field is present in the request body. If both are provided, the body's model field wins.

Using azure_deployment — model name derived from the path

Pass azure_deployment to the AzureOpenAI constructor. The SDK builds the URL as /openai/deployments/<azure_deployment>/chat/completions; the guardrail server reads the deployment name from the path and uses it as the model identifier — no model field is required in the request body.

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="http://localhost:8000",   # guardrail server, not Azure
    azure_deployment="gpt4o-prod",            # becomes the model name
    api_key="any",
    api_version="2024-02-01",
)

# SDK posts to /openai/deployments/gpt4o-prod/chat/completions.
# Server extracts "gpt4o-prod" as the model, applies guardrails, then
# forwards the request to the upstream configured for that model name.
response = client.chat.completions.create(
    messages=[{"role": "user", "content": "Hello"}],
    # model= is optional here; the deployment path supplies it
)
print(response.choices[0].message.content)

Using both model and azure_deployment

When model is present in the request body it takes precedence over the path segment. This lets the same deployment URL serve different logical models:

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="http://localhost:8000",
    azure_deployment="gpt4o-prod",
    api_key="any",
    api_version="2024-02-01",
)

# model= in the body overrides the deployment path → routes to "gpt-4o"
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Local Ollama

flowchart LR
    App["Your App\nOpenAI(base_url=localhost:8000)"]
    GS["Guardrails Server\n:8000\nOPENAI_BASE_URL=localhost:11434/v1"]
    Ollama["Ollama\nlocalhost:11434"]

    App -->|"POST /v1/chat/completions\nmodel: llama3"| GS
    GS -->|"guardrails applied"| Ollama
    Ollama -->|"ChatCompletion"| GS
    GS -->|"200 JSON"| App
OPENAI_BASE_URL=http://localhost:11434/v1 \
OPENAI_API_KEY=ollama \
mend-guardrails-server --policy-dir ./policies

from openai import OpenAI

# Client still talks to the guardrail server on port 8000
client = OpenAI(base_url="http://localhost:8000/v1", api_key="ollama")
response = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"guardrails": {"config_id": "default"}},
)

Forwarding HTTP headers and query parameters

Inbound headers and query parameters can be carried through to the upstream LLM using two methods:

Static injection (server-configured)

Use this when the upstream always needs the same headers or query parameters regardless of who is calling the guardrail server.

Set one or both environment variables:

Variable What it does
MEND_GUARDRAILS_UPSTREAM_HEADERS JSON object — added as default_headers to every upstream call.
MEND_GUARDRAILS_UPSTREAM_QUERY JSON object — added as default_query to every upstream call. 
flowchart LR
    App["Your App"]
    GS["Guardrails Server\nUPSTREAM_HEADERS={x-tenant:acme}\nUPSTREAM_QUERY={api-version:2025-01-01}"]
    Proxy["Upstream / Proxy\nmyproxy.com/openai"]

    App -->|"POST /v1/chat/completions\n(no special headers needed)"| GS
    GS -->|"x-tenant: acme\napi-version=2025-01-01\n(injected by server)"| Proxy
OPENAI_BASE_URL=https://api.myproxy.com/openai \
MEND_GUARDRAILS_UPSTREAM_HEADERS='{"x-tenant": "acme"}' \
MEND_GUARDRAILS_UPSTREAM_QUERY='{"api-version": "2025-01-01"}' \
mend-guardrails-server --policy-dir ./policies

The caller does not need to know about these headers — they are invisible to the application and always present on the upstream call.

Per-request forwarding (caller-controlled)

Use this when the header or query parameter value differs per request and the calling application already sends it. The server maintains an explicit allowlist; only named headers/params are forwarded. Anything not on the list is silently dropped.

Variable What it does
MEND_GUARDRAILS_FORWARD_HEADERS Comma-separated header names to forward from the inbound request to the upstream call (extra_headers). Matching is case-insensitive.
MEND_GUARDRAILS_FORWARD_QUERY_PARAMS Comma-separated query param names to forward from the inbound request URL to the upstream call (extra_query). 
flowchart LR
    App["Your App\nsends x-request-id + api-version"]
    GS["Guardrails Server\nFORWARD_HEADERS=x-request-id\nFORWARD_QUERY_PARAMS=api-version"]
    Proxy["Upstream / Proxy\nmyproxy.com/openai"]
    Dropped["x-internal-token\n(not on allowlist — dropped)"]

    App -->|"x-request-id: abc123\napi-version=2025-01-01\nx-internal-token: secret"| GS
    GS -->|"x-request-id: abc123\napi-version=2025-01-01\n(forwarded)"| Proxy
    GS -. dropped .-> Dropped
OPENAI_BASE_URL=https://api.myproxy.com/openai \
MEND_GUARDRAILS_FORWARD_HEADERS="x-request-id,x-tenant-id" \
MEND_GUARDRAILS_FORWARD_QUERY_PARAMS="api-version" \
mend-guardrails-server --policy-dir ./policies

The caller sends the values as normal HTTP headers or query parameters on the request to the guardrail server:

curl -X POST "http://localhost:8000/v1/chat/completions?api-version=2025-01-01" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "x-request-id: abc123" \
  -H "x-tenant-id: acme" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello!"}],
       "guardrails": {"config_id": "default"}}'

Security note: the allowlist is intentional. Internal headers (trace IDs, session tokens, routing metadata) that the caller sends to the guardrail server are not forwarded to the external LLM API unless you explicitly name them.

Combining both mechanisms

Both mechanisms can be used together. Static values (Mechanism 1) are merged with forwarded values (Mechanism 2) on every call. Per-request values take precedence when the same header or param name appears in both.

OPENAI_BASE_URL=https://api.myproxy.com/openai \
MEND_GUARDRAILS_UPSTREAM_HEADERS='{"x-tenant": "acme"}' \
MEND_GUARDRAILS_UPSTREAM_QUERY='{"api-version": "2025-01-01"}' \
MEND_GUARDRAILS_FORWARD_HEADERS="x-request-id" \
mend-guardrails-server --policy-dir ./policies

Every upstream call will carry x-tenant: acme and api-version=2025-01-01 from the static config, and additionally x-request-id from whichever value the caller sends on that specific request.

Multi-model configuration

When your upstream LLM infrastructure uses different endpoints, API keys, or provider types per model, you can define all configurations in a single JSON file and let the server route each request to the right upstream automatically.

Set the path via --models-config (or MEND_GUARDRAILS_MODELS_CONFIG):

mend-guardrails-server \
  --models-config ./models.json \
  --policy-dir ./policies \
  --default-config default

The file is a JSON array where each object represents one model endpoint:

[
  {
    "name": "gpt-4o",
    "provider": "openai",
    "parameters": {
      "base_url": "https://api.myproxy.com/openai",
      "api_key": "sk-proxy-key",
      "default_headers": { "x-tenant": "acme" },
      "default_query":  { "api-version": "2025-01-01" }
    }
  },
  {
    "name": "my-azure-deployment",
    "provider": "azure",
    "parameters": {
      "azure_endpoint":    "https://my-resource.openai.azure.com",
      "azure_deployment":  "my-deployment",
      "api_version":       "2024-02-01",
      "api_key":           "azure-key"
    }
  }
]

The server matches incoming requests by the model field and constructs the appropriate SDK client (MendGuardrailsAsyncOpenAI for "provider": "openai", MendGuardrailsAsyncAzureOpenAI for "provider": "azure").

Schema

Each entry in the array is validated against the following fields. Unknown top-level keys are rejected (the server refuses to start on an invalid file).

Field Type Required Default Description
name string Yes — Routing key matched against the request's model field.
provider string enum: "openai" | "azure" No "openai" Upstream provider type and SDK client used.
model string | null No null Optional override for the model value forwarded upstream. When null, the request's model is forwarded verbatim.
parameters object No {} Provider-specific settings (see below).
default boolean No false Marks this entry as the fallback for unmatched requests. At most one entry may set true.

The parameters object accepts arbitrary keys. Keys matching an SDK constructor argument are forwarded to the client; all others are applied as per-request defaults (see Parameters split: client vs request).

Parameter Type Applies to Description
base_url string openai Base URL of the OpenAI-compatible upstream.
api_key string openai, azure API key forwarded to the upstream provider.
default_headers object (string → string) openai, azure Headers added to every upstream request.
default_query object (string → string) openai, azure Query parameters added to every upstream request.
timeout number openai, azure Request timeout in seconds.
max_retries integer openai, azure Maximum number of upstream retries.
azure_endpoint string azure Azure OpenAI resource endpoint.
azure_deployment string azure Azure deployment name.
api_version string azure Azure API version.
(any other key) any openai, azure Applied as a per-request default (e.g. seed, logit_bias).

Default model entry

Add "default": true to one entry to make it the fallback for any request whose model value does not match any named entry in the file. Only one entry may have "default": true; the server will refuse to start if more than one entry is marked.

[
  {
    "name": "gpt-4o",
    "provider": "openai",
    "parameters": { "base_url": "https://api.myproxy.com/openai" }
  },
  {
    "name": "proxy-catch-all",
    "provider": "openai",
    "default": true,
    "parameters": {
      "base_url": "https://api.myproxy.com/openai",
      "default_headers": { "x-tenant": "acme" }
    }
  }
]

Resolution order for an incoming request:

  1. Exact name match — the entry whose name equals the request's model field.
  2. Default entry — the entry with "default": true, when no exact match exists.
  3. Global env-var fallback — OPENAI_BASE_URL, OPENAI_API_KEY, etc., when neither an exact match nor a default entry is configured.

Model name override

name is the routing key that callers send as model. By default that same value is forwarded upstream. Add an optional model field to send a different model name to the provider — useful when the route name differs from the provider's model name (e.g. an alias in front of an Azure AI Foundry agent endpoint):

{
  "name": "agent-1",
  "provider": "openai",
  "model": "gpt-4o",
  "parameters": {
    "base_url": "https://<foundry>/api/projects/<project>/agents/<agent>/endpoint/protocols/openai/",
    "default_query": { "api-version": "v1" }
  }
}

A request with "model": "agent-1" routes to this endpoint but forwards "model": "gpt-4o" upstream. Omit model to forward the request's value unchanged.

Parameter precedence

Parameters are applied in the following order (later entries win):

  1. Values from the model config file parameters object.
  2. Azure-specific environment variable overrides (AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, etc.) — for azure provider only.
  3. OPENAI_BASE_URL — for openai provider, applied only when base_url is not already set in the model config.

Parameters split: client vs request

Parameters whose names match OpenAI/Azure SDK constructor arguments (base_url, api_key, default_headers, default_query, timeout, max_retries, azure_endpoint, azure_deployment, api_version) are forwarded to the SDK client constructor. Any other parameters in parameters are applied as per-request defaults (e.g. seed, logit_bias).

Requests with no matching model config

Requests whose model value does not match any entry in the config file fall through to the global defaults: OPENAI_BASE_URL, OPENAI_API_KEY, etc. No error is raised for backward-compatibility.