Networking and connectivity

Use these settings to change where SDKs connect, tune incoming request behavior, control which external and private services the data plane can reach, and optionally run the Braintrust Gateway in your own infrastructure.

Client routing

Customize the webapp URL

The SDKs guide users to https://www.braintrust.dev (or the BRAINTRUST_APP_URL variable) to view their experiments. In some advanced configurations, you can reverse proxy traffic to the BRAINTRUST_APP_URL from the SDKs while pointing users to a different URL. To do this, you can set the BRAINTRUST_APP_PUBLIC_URL environment variable to the URL of your webapp. By default, this variable is set to the value of BRAINTRUST_APP_URL, but you can customize it as you wish. This variable is only used to display information, so even its destination does not need to be accessible from the SDK.

AWS
GCP / Azure

Set it through the braintrust_api_extra_env_vars passthrough map (Terraform module v6.0.0 or later):

braintrust_api_extra_env_vars = {
  BRAINTRUST_APP_PUBLIC_URL = "https://braintrust.example.com"
}

Deployments still served by the API Lambda (before module v6.0.0, or v6 with enable_ecs_api = false) set the same variable through service_extra_env_vars.APIHandler instead.

Add it to api.extraEnvVars in your values.yaml:

api:
  extraEnvVars:
    - name: BRAINTRUST_APP_PUBLIC_URL
      value: "https://braintrust.example.com"

Constrain SDKs to the data plane

If you’re self-hosting the data plane, you can also constrain the SDKs to only communicate with your data plane. Normally, they communicate with the control plane to:

Get your data plane’s URL
Register and retrieve metadata (e.g. about experiments)
Print URLs to the webapp

The data plane can proxy the endpoints that the SDKs use to communicate with the control plane, allowing your SDKs to only communicate with the data plane directly. Set the BRAINTRUST_APP_URL environment variable to the URL of your data plane and BRAINTRUST_APP_PUBLIC_URL to https://www.braintrust.dev (or the URL of your webapp).

AWS
GCP / Azure

braintrust_api_extra_env_vars = {
  BRAINTRUST_APP_URL        = "https://dataplane.example.com"
  BRAINTRUST_APP_PUBLIC_URL = "https://www.braintrust.dev"
}

api:
  extraEnvVars:
    - name: BRAINTRUST_APP_URL
      value: "https://dataplane.example.com"
    - name: BRAINTRUST_APP_PUBLIC_URL
      value: "https://www.braintrust.dev"

Inbound traffic

Set HTTPS on the API load balancer

On AWS with the ECS API, an internal Application Load Balancer (ALB) fronts the API services. By default, the ALB serves plain HTTP on port 80 using its AWS-assigned DNS name. To serve HTTPS on a custom domain instead, set both braintrust_api_alb_certificate_arn and braintrust_api_alb_custom_domain (available in Terraform module v6.0.0 or later):

braintrust_api_alb_certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/abc123"
braintrust_api_alb_custom_domain   = "braintrust.internal.example.com"

When both are set, the ALB serves HTTPS on port 443, plain HTTP is disabled, and all API URLs use https://<braintrust_api_alb_custom_domain>. The certificate must cover the custom domain, and the domain must resolve to the ALB.

These two variables must both be set or both be null. Setting only one fails at plan time.

Set the HTTP keep-alive timeout

When the API server runs behind a load balancer, you may need to configure the HTTP keep-alive timeout to prevent connection resets. Load balancers typically have an idle timeout for connections, and if the API server’s keep-alive timeout is shorter than the load balancer’s timeout, the API server closes the connection while the load balancer still considers it open. When the load balancer tries to reuse that backend connection, it encounters a closed socket, resulting in connection reset errors and 502 responses. The API server exposes the following environment variable to configure the keep-alive timeout:

TS_API_KEEP_ALIVE_TIMEOUT_SECONDS: The HTTP keep-alive timeout in seconds. Default: 65

The default value of 65 seconds is designed to work with most load balancers, including AWS Application Load Balancer (which has a default idle timeout of 60 seconds). However, if your load balancer has a longer idle timeout, you should set this value to match or exceed your load balancer’s timeout. For example, to match an AWS ALB configured with a 300-second idle timeout:

AWS
GCP / Azure

Set it through the braintrust_api_extra_env_vars passthrough map (Terraform module v6.0.0 or later). This applies to the ECS API services:

braintrust_api_extra_env_vars = {
  TS_API_KEEP_ALIVE_TIMEOUT_SECONDS = "300"
}

Add it to api.extraEnvVars in your values.yaml:

api:
  extraEnvVars:
    - name: TS_API_KEEP_ALIVE_TIMEOUT_SECONDS
      value: "300"

Set the CloudFront origin timeout

On AWS, requests are served through CloudFront, which closes a connection and returns 504 Gateway Timeout if the origin takes too long to respond. Long-running scorers or tools invoked through /function/invoke can exceed the default 60-second origin read timeout. Raise it with the cloudfront_origin_read_timeout Terraform variable (available in Terraform module v5.3.0 or later):

cloudfront_origin_read_timeout = 120

The value must be between 1 and 180 seconds. CloudFront caps the origin read timeout at 180 seconds, and values above 60 seconds can require an AWS Support request to raise the account quota.

Set inbound request rate limits

The API server can rate-limit log ingestion, SQL queries, and function invocation. Configure each surface separately with its own environment variables for limits, window length, and enforcement. All three surfaces behave the same way in these respects:

Windows: Each limit uses a fixed window that starts when the first matching request is counted, not on a clock boundary. The counter resets after the configured number of seconds, and the next matching request starts a new window. Rejected requests still count toward the limit.
Enforcement: With enforcement disabled, requests over a limit are allowed and the API server logs a warning. With enforcement enabled, they fail with HTTP 429 and a Retry-After header, and the response body reports the configured limit, the window length, and the requests consumed. A limit of 0 is a real zero-request limit, so with enforcement enabled every matching request is rejected.
Replicas: Each limit applies across all API server replicas combined.
Restarts: Rate limit configuration is read once at process start. Restart or redeploy the API services after changing any of these variables.

None of these variables has a dedicated Terraform variable or Helm value, so pass them through your deployment’s environment variable map, as shown in the examples below. Variables that take <id>=<max_requests> pairs accept a comma-separated list of pairs.

Limit log ingestion

Log ingestion limits apply per organization and per project, and both are disabled by default. A project limit replaces the organization limit rather than adding to it, so a project with its own entry ignores the organization limit entirely. With no limit configured, ingestion is uncapped. Window and enforcement

RATELIMIT_API_LOGS_ORG_WINDOW_SECS: Window length in seconds. Default 60. Despite the name, this also sets the window for project-based limits.
RATELIMIT_API_LOGS_ORG_ENFORCE: Return HTTP 429 when a limit is exceeded. Default false (log a warning and allow the request). Despite the name, this also governs the enforcement of project-based limits.

Organization limits

RATELIMIT_API_LOGS_ORG: Per-organization limits, as <org_id>=<max_requests> pairs. Find an organization’s ID in the organization switcher.

Project limits

Project-scoped limits require data plane v2.2.1 or later.

RATELIMIT_API_LOGS_PROJECT: Per-project limits, as <project_id>=<max_requests> pairs. Find a project’s ID under Settings > General.
RATELIMIT_API_LOGS_PROJECT_DEFAULT: Limit for every project without an entry in RATELIMIT_API_LOGS_PROJECT. Set it only if you want every project capped.

AWS
GCP / Azure

Set these variables through the braintrust_api_extra_env_vars passthrough map (Terraform module v6.0.0 or later). For example:

braintrust_api_extra_env_vars = {
  RATELIMIT_API_LOGS_ORG             = "<org_id>=1000"
  RATELIMIT_API_LOGS_PROJECT         = "<project_id>=500"
  RATELIMIT_API_LOGS_ORG_WINDOW_SECS = "60"
  RATELIMIT_API_LOGS_ORG_ENFORCE     = "true"
}

Deployments still served by the API Lambda (before module v6.0.0, or v6 with enable_ecs_api = false) set the same variables through service_extra_env_vars.APIHandler instead.

Add these variables to api.extraEnvVars in the values.yaml you pass to the Helm chart. For example:

api:
  extraEnvVars:
    - name: RATELIMIT_API_LOGS_ORG
      value: "<org_id>=1000"
    - name: RATELIMIT_API_LOGS_PROJECT
      value: "<project_id>=500"
    - name: RATELIMIT_API_LOGS_ORG_WINDOW_SECS
      value: "60"
    - name: RATELIMIT_API_LOGS_ORG_ENFORCE
      value: "true"

Setting a log ingestion limit to 0 with enforcement on rejects every matching logging request and stores nothing. Any spans the client does not retry are lost.

A request counts as one unit against whichever limit applies, regardless of how many spans it carries. If any organization or project referenced in the request exceeds its limit, the entire payload is rejected.

Limit SQL queries

SQL query limits apply per organization and per project, and both are disabled by default. A project limit adds to the organization limit rather than replacing it, so it can only tighten the effective limit. Organizations without their own entry fall back to RATELIMIT_BTQL_DEFAULT, and with no default configured, queries are uncapped. Window and enforcement

RATELIMIT_BTQL_WINDOW_SECS: Window length in seconds. Default 60.
RATELIMIT_BTQL_ENFORCE: Return HTTP 429 when a limit is exceeded. Default false (log a warning and allow the query).

Organization limits

RATELIMIT_BTQL_ORG: Per-organization limits, as <org_id>=<max_queries> pairs. Find an organization’s ID in the organization switcher.
RATELIMIT_BTQL_DEFAULT: Limit for every organization without an entry in RATELIMIT_BTQL_ORG.
RATELIMIT_BTQL_DEFAULT_FUNCTIONS: Separate default for queries against prompts and functions. Defaults to 20 times RATELIMIT_BTQL_DEFAULT, including when set to -1.

Project limits

Project-scoped limits require data plane v2.2.1 or later.

RATELIMIT_BTQL_PROJECT: Per-project limits, as <project_id>=<max_queries> pairs. Setting one above the organization limit has no effect. With no organization limit configured, the project limit is the only one that applies. Find a project’s ID under Settings > General.

AWS
GCP / Azure

Set these variables through the braintrust_api_extra_env_vars passthrough map (Terraform module v6.0.0 or later). For example:

braintrust_api_extra_env_vars = {
  RATELIMIT_BTQL_ORG         = "<org_id>=1000"
  RATELIMIT_BTQL_PROJECT     = "<project_id>=100"
  RATELIMIT_BTQL_WINDOW_SECS = "60"
  RATELIMIT_BTQL_ENFORCE     = "true"
}

Deployments still served by the API Lambda (before module v6.0.0, or v6 with enable_ecs_api = false) set the same variables through service_extra_env_vars.APIHandler instead.

Add these variables to api.extraEnvVars in the values.yaml you pass to the Helm chart. For example:

api:
  extraEnvVars:
    - name: RATELIMIT_BTQL_ORG
      value: "<org_id>=1000"
    - name: RATELIMIT_BTQL_PROJECT
      value: "<project_id>=100"
    - name: RATELIMIT_BTQL_WINDOW_SECS
      value: "60"
    - name: RATELIMIT_BTQL_ENFORCE
      value: "true"

A query counts once against the organization limit and once against each project it reads, so a query spanning several projects must stay under every limit it touches.

These limits apply to queries from the API, the SDKs, and the MCP server. Queries issued from the Braintrust UI are exempt.

Limit function invocation

Invocation limits cover a project’s prompts, scorers, tools, and other custom code functions. They are project-scoped, and there is no organization-scoped limit. Two independent limits apply: a per-project limit that is disabled by default, and a per-function cap that is on by default with a fixed 10-second window and always returns HTTP 429. Window and enforcement

RATELIMIT_INVOKE_WINDOW_SECS: Window length in seconds for the project limits. Default 10. It does not affect the per-function cap.
RATELIMIT_INVOKE_ENFORCE: Return HTTP 429 when a project limit is exceeded. Default false (log a warning and allow the invocation). It does not affect the per-function cap.

Project limits

Project-scoped limits require data plane v2.2.1 or later.

RATELIMIT_INVOKE_PROJECT: Per-project limits, as <project_id>=<max_invocations> pairs. The count covers every function in the project, across all API keys. Find a project’s ID under Settings > General.
RATELIMIT_INVOKE_PROJECT_DEFAULT: Limit for every project without an entry in RATELIMIT_INVOKE_PROJECT. Set it only if you want every project capped.

Per-function limit

INVOKE_RATE_LIMIT_PER_10S: Maximum invocations per function, per API key, in a 10-second window. Default 10000. The count is per function, not per project.
ENABLE_INVOKE_RATE_LIMIT: Whether invocation rate limiting runs at all. Default true. Setting it to false turns off the per-function cap and the project limits.

AWS
GCP / Azure

Set these variables through the braintrust_api_extra_env_vars passthrough map (Terraform module v6.0.0 or later). For example:

braintrust_api_extra_env_vars = {
  RATELIMIT_INVOKE_PROJECT     = "<project_id>=100"
  RATELIMIT_INVOKE_WINDOW_SECS = "10"
  RATELIMIT_INVOKE_ENFORCE     = "true"
  INVOKE_RATE_LIMIT_PER_10S    = "5000"
}

Deployments still served by the API Lambda (before module v6.0.0, or v6 with enable_ecs_api = false) set the same variables through service_extra_env_vars.APIHandler instead.

Add these variables to api.extraEnvVars in the values.yaml you pass to the Helm chart. For example:

api:
  extraEnvVars:
    - name: RATELIMIT_INVOKE_PROJECT
      value: "<project_id>=100"
    - name: RATELIMIT_INVOKE_WINDOW_SECS
      value: "10"
    - name: RATELIMIT_INVOKE_ENFORCE
      value: "true"
    - name: INVOKE_RATE_LIMIT_PER_10S
      value: "5000"

Preprocessors are exempt from invocation rate limits.

Outbound traffic

Secure outbound requests

The data plane makes outbound requests both to Braintrust and to URLs you or your users supply, such as webhooks, remote scorers, and integrations. Allow traffic to Braintrust through your firewall If you restrict outbound network traffic, allow the data plane to reach Braintrust at:

www.braintrust.dev
braintrust.dev
gateway.braintrust.dev

gateway.braintrust.dev is the Braintrust-hosted Gateway. Two kinds of request go there:

Built-in model requests. Braintrust serves built-in models itself, so requests for them always go to the hosted Gateway. Self-hosted organizations have built-in models off by default and opt in explicitly, and Topics requires them.
LLM calls from user-authored code, such as custom scorers and tools, on AWS deployments that run the API on ECS (enable_ecs_api = true) with Terraform module v6.4.0 or earlier. Module v6.5.0 and later sends these to the deployment’s own AI proxy instead, so that traffic stays in your AWS account.

This is firewall guidance. The data plane does not enforce a destination allowlist itself. Block requests to internal addresses To stop user-supplied URLs from reaching private or reserved IP addresses (server-side request forgery), configure URL validation. See Configure URL security. Trust a private certificate authority If the internal services your custom scorers and tools call present certificates signed by a private or enterprise certificate authority, supply the CA bundle so those requests validate. See Configure a custom CA bundle.

Set outbound request rate limits

The Braintrust API server can rate-limit the outbound requests it makes to external domains, such as BRAINTRUST_APP_URL. Rate limiting prevents unintentionally overloading an external domain, which might otherwise block the API server’s IP in response. It is disabled by default. When enabled, requests are counted per API auth token per destination domain within a rolling window.

OUTBOUND_RATE_LIMIT_MAX_REQUESTS: The maximum number of requests per window. Default 0, which disables rate limiting. Set a value greater than 0 to enable it.
OUTBOUND_RATE_LIMIT_WINDOW_MINUTES: The window length in minutes before the count resets. Default 1.

AWS
GCP / Azure

Use the dedicated variables (Terraform module v1.0.0 or later):

outbound_rate_limit_max_requests   = 100
outbound_rate_limit_window_minutes = 1

Set the environment variables through api.extraEnvVars:

api:
  extraEnvVars:
    - name: OUTBOUND_RATE_LIMIT_MAX_REQUESTS
      value: "100"
    - name: OUTBOUND_RATE_LIMIT_WINDOW_MINUTES
      value: "1"

Connect to internal resources over VPC

On AWS, to connect Braintrust’s VPC to other internal resources (like an LLM Gateway), use one of the following approaches:

Create a VPC Endpoint Service for your internal resource, then create a VPC Interface Endpoint inside the Braintrust “Quarantine” VPC.
Set up VPC peering with the Braintrust “Quarantine” VPC.

Braintrust Gateway

The Braintrust Gateway gives your applications a single OpenAI-compatible API for reaching any supported model provider (OpenAI, Anthropic, Google, AWS, and others), using provider keys you manage centrally in Braintrust. At request time it adds completions caching, provider failover, streaming, rate limiting, usage tracking, and trace logging. Braintrust hosts a Gateway at gateway.braintrust.dev. Self-hosted deployments can optionally run their own instance in the data plane, alongside the API and Brainstore, so Gateway requests are served from your own infrastructure.

Beta — This feature is subject to change.

How the Gateway works

Running the Gateway yourself keeps LLM traffic on your own infrastructure: requests egress directly from your VPC to providers, completions are cached (encrypted) in the data plane’s Redis, and internal features like Topics route through it too. When the Gateway is disabled, the data plane does not fall back to the Braintrust-hosted Gateway. The API service calls providers directly. On AWS, user-authored code such as custom scorers and tools runs in an isolated quarantine environment that cannot reach your main VPC directly. Terraform module v6.5.0 and later sends its LLM calls to the deployment’s own AI proxy, which forwards them to the Gateway once enable_ai_gateway is set, so those calls stay in your AWS account.

Even with the Gateway enabled, two paths still leave your data plane:

Built-in models are served by Braintrust, so requests for them reach the Braintrust-hosted Gateway at gateway.braintrust.dev rather than your own. Self-hosted organizations have built-in models disabled by default and opt in explicitly.
Operational telemetry (status, metrics, usage, and optional logs and traces) can still be sent to Braintrust’s control plane, the same as the rest of the data plane. See Telemetry and data retention. This is monitoring traffic, not LLM traffic.

Enable the Gateway

The Gateway is disabled by default. It reuses the Redis instance and Brainstore license key the data plane already provisions, so no additional secrets are required.

Prerequisite - Before you deploy the Gateway service in your data plane, contact Braintrust to enable the Gateway for your organization. Until your organization is enabled, Gateway-dependent features stay hidden in the Braintrust UI, and requests continue to use the AI proxy.

AWS
GCP / Azure

Requires Terraform module v6.5.0 or later, which pins the Gateway image to the data plane version the module ships and keeps quarantine LLM calls in your account. The Gateway runs as an ECS Fargate service behind an internal load balancer.Three variables control the Gateway, so you can provision the infrastructure before routing traffic to it:

create_ai_gateway creates the private Gateway infrastructure (an internal ALB and the Gateway ECS service).
enable_ai_gateway sets GATEWAY_URL on the APIHandler, AI Proxy, and API ECS service so internal data plane traffic routes through the Gateway. It requires create_ai_gateway.
use_private_ai_gateway_origin points the public CloudFront /v1/proxy endpoint at the private Gateway (a VPC origin on its internal load balancer). It requires create_ai_gateway.

Existing deploymentOn an existing deployment, roll the variables out in separate applies so each change is verifiable before the next:

Provision the infrastructure without changing traffic and apply:
create_ai_gateway = true enable_ai_gateway = false
Route internal traffic through the Gateway. Once the Gateway is healthy, set enable_ai_gateway = true and apply:
enable_ai_gateway = true
Public /v1/proxy traffic still uses its existing origin.
Cut over the public endpoint and apply:
use_private_ai_gateway_origin = true
CloudFront then routes /v1/proxy and /v1/proxy/* to the private Gateway.

New deploymentOn a new deployment, you can set create_ai_gateway and enable_ai_gateway together in the first apply, then cut over the public endpoint with use_private_ai_gateway_origin once the Gateway is healthy.Configure task sizing and autoscaling with ai_gateway_cpu, ai_gateway_memory, ai_gateway_min_capacity, and ai_gateway_max_capacity. To authorize additional security groups to reach the internal ALB, set ai_gateway_authorized_security_groups. The API and Brainstore security groups are authorized automatically.The module pins the Gateway image, the same way it manages the API and Brainstore images, so upgrading the module is how you move to a newer Gateway. Leave ai_gateway_version_override unset unless Braintrust instructs you to pin a specific tag.

Requires Helm chart 6.6.0 or later. The chart pins the Gateway image to the data plane version it ships.

The Gateway is disabled by default. To deploy it, add the following to your values.yaml and apply with helm upgrade:
aiGateway: enabled: true
Verify the Gateway pods are running before routing traffic to them.
Enabling routing before the Gateway is ready causes API requests to fail.
Once the Gateway pods are running and ready, set aiGateway.useGateway: true to route API traffic through the Gateway, then apply again with helm upgrade:
aiGateway: enabled: true useGateway: true

Configuration options:

Field	Default	Description
`aiGateway.enabled`	`false`	Deploy the Gateway service.
`aiGateway.useGateway`	`false`	Route API requests through the Gateway. Requires `enabled: true`.
`aiGateway.replicas`	`2`	Number of Gateway pod replicas.
`aiGateway.region`	`""`	Optional cloud region identifier (for example, `us-east-1`). Sets `GATEWAY_REGION` when provided.
`aiGateway.braintrustApiUrl`	`""`	Override the API URL the Gateway uses. Defaults to the in-cluster API service URL.
`aiGateway.resources`	`cpu: 2`, `memory: 4Gi`	Gateway pod resource requests and limits.

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Networking and connectivity

Client routing

Customize the webapp URL

Constrain SDKs to the data plane

Inbound traffic

Set HTTPS on the API load balancer

Set the HTTP keep-alive timeout

Set the CloudFront origin timeout

Set inbound request rate limits

Limit log ingestion

Limit SQL queries

Limit function invocation

Outbound traffic

Secure outbound requests

Set outbound request rate limits

Connect to internal resources over VPC

Braintrust Gateway

How the Gateway works

Enable the Gateway

​Client routing

​Customize the webapp URL

​Constrain SDKs to the data plane

​Inbound traffic

​Set HTTPS on the API load balancer

​Set the HTTP keep-alive timeout

​Set the CloudFront origin timeout

​Set inbound request rate limits

​Limit log ingestion

​Limit SQL queries

​Limit function invocation

​Outbound traffic

​Secure outbound requests

​Set outbound request rate limits

​Connect to internal resources over VPC

​Braintrust Gateway

​How the Gateway works

​Enable the Gateway

Client routing

Customize the webapp URL

Constrain SDKs to the data plane

Inbound traffic

Set HTTPS on the API load balancer

Set the HTTP keep-alive timeout

Set the CloudFront origin timeout

Set inbound request rate limits

Limit log ingestion

Limit SQL queries

Limit function invocation

Outbound traffic

Secure outbound requests

Set outbound request rate limits

Connect to internal resources over VPC

Braintrust Gateway

How the Gateway works

Enable the Gateway