SRE / Platform

Enterprise Deployment Guide

Select one supported identity and state model before writing deployment manifests. These modes have different compatibility and scaling constraints; their environment variables are not interchangeable.

Synced with v1.25.1 ·

Choose a Deployment Mode

Workspace MCP supports several authentication paths, but they do not all compose. Treat this table as a configuration boundary.

ModeIdentity and credentialsScaling constraint
OAuth 2.1 statelessThe MCP server authenticates clients and holds encrypted upstream Google tokens in Valkey.Supports multiple replicas when Valkey is available.
Trusted gatewayA proxy supplies a verified assertion; Google grants use the local credential directory.Use one replica; credential sharing alone does not make transport sessions portable.
Gateway + DWDThe gateway identifies the user and a delegated service account impersonates that user.No grant store; transport sessions still require affinity unless separately validated.
Stateful OAuth 2.1 + GCSPer-user Google grants are stored in GCS with optional CMEK enforcement.GCS shares grants, not MCP transport sessions; replicas are not interchangeable by default.
Incompatible settings TRUST_GATEWAY_IDENTITY=true cannot be combined with MCP_ENABLE_OAUTH21=true. Stateless mode requires OAuth 2.1. Service-account mode also cannot be combined with OAuth 2.1.

OAuth 2.1 Stateless Deployment

This is the supported multi-replica mode. Set the public URL explicitly, enable stateless HTTP, and use Valkey for FastMCP client registrations and encrypted upstream token state. Do not configure GCS: stateless mode bypasses it.

Build the required dependency

The current repository Dockerfile does not install the optional Valkey dependency. Add it to your image build; otherwise the server warns and falls back to its default storage.

# The repository Dockerfile includes disk and OpenTelemetry extras.
# Add the Valkey extra before using the multi-replica examples below:
RUN uv sync --frozen --no-dev --extra disk --extra valkey --extra otel

Valkey

WORKSPACE_MCP_OAUTH_PROXY_STORAGE_BACKEND=valkey
WORKSPACE_MCP_OAUTH_PROXY_VALKEY_HOST=valkey.internal.corp.example.com
WORKSPACE_MCP_OAUTH_PROXY_VALKEY_PORT=6380
WORKSPACE_MCP_OAUTH_PROXY_VALKEY_USE_TLS=true
WORKSPACE_MCP_OAUTH_PROXY_VALKEY_USERNAME=workspace-mcp
WORKSPACE_MCP_OAUTH_PROXY_VALKEY_PASSWORD=<from-secret-manager>

# Optional for higher-latency remote or TLS endpoints
WORKSPACE_MCP_OAUTH_PROXY_VALKEY_REQUEST_TIMEOUT_MS=5000
WORKSPACE_MCP_OAUTH_PROXY_VALKEY_CONNECTION_TIMEOUT_MS=10000

Keep FASTMCP_SERVER_AUTH_GOOGLE_JWT_SIGNING_KEY stable across replicas and deployments. It participates in encrypting stored OAuth state.

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: workspace-mcp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: workspace-mcp
  template:
    metadata:
      labels:
        app: workspace-mcp
    spec:
      serviceAccountName: workspace-mcp
      containers:
        - name: workspace-mcp
          # Use an immutable tag or digest. The image must include the valkey extra.
          image: your-registry/workspace-mcp:1.25.0
          ports:
            - name: http
              containerPort: 8000
          env:
            - name: MCP_ENABLE_OAUTH21
              value: "true"
            - name: WORKSPACE_EXTERNAL_URL
              value: "https://mcp.corp.example.com"
            - name: WORKSPACE_MCP_STATELESS_MODE
              value: "true"
            - name: WORKSPACE_MCP_OAUTH_PROXY_STORAGE_BACKEND
              value: "valkey"
            - name: WORKSPACE_MCP_OAUTH_PROXY_VALKEY_HOST
              value: "valkey.internal.corp.example.com"
            - name: WORKSPACE_MCP_OAUTH_PROXY_VALKEY_PORT
              value: "6380"
            - name: WORKSPACE_MCP_OAUTH_PROXY_VALKEY_USE_TLS
              value: "true"
          envFrom:
            - secretRef:
                # GOOGLE_OAUTH_CLIENT_ID, GOOGLE_OAUTH_CLIENT_SECRET,
                # FASTMCP_SERVER_AUTH_GOOGLE_JWT_SIGNING_KEY, and Valkey password.
                name: workspace-mcp-secrets
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 5
            failureThreshold: 12
          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
apiVersion: v1
kind: Service
metadata:
  name: workspace-mcp
spec:
  type: ClusterIP
  selector:
    app: workspace-mcp
  ports:
    - name: http
      port: 8000
      targetPort: http
Probe scope /health confirms that the HTTP process is responding. It does not test Valkey, GCS, Google APIs, or existing credentials. Monitor those dependencies separately and alert on Valkey fallback warnings.

Cloud Run

The public URL must match the Google OAuth callback registration. This example allows requests through the Cloud Run IAM layer because Workspace MCP performs protocol-level OAuth 2.1 authentication; ingress still restricts internet traffic to the load balancer. If policy requires IAP, use trusted-gateway mode.

gcloud run deploy workspace-mcp \
  --image=your-registry/workspace-mcp:1.25.0 \
  --port=8000 \
  --set-env-vars="MCP_ENABLE_OAUTH21=true" \
  --set-env-vars="WORKSPACE_EXTERNAL_URL=https://mcp.corp.example.com" \
  --set-env-vars="WORKSPACE_MCP_STATELESS_MODE=true" \
  --set-env-vars="WORKSPACE_MCP_OAUTH_PROXY_STORAGE_BACKEND=valkey" \
  --set-env-vars="WORKSPACE_MCP_OAUTH_PROXY_VALKEY_HOST=10.0.0.5" \
  --set-env-vars="WORKSPACE_MCP_OAUTH_PROXY_VALKEY_PORT=6379" \
  --set-secrets="GOOGLE_OAUTH_CLIENT_ID=mcp-client-id:latest" \
  --set-secrets="GOOGLE_OAUTH_CLIENT_SECRET=mcp-client-secret:latest" \
  --set-secrets="FASTMCP_SERVER_AUTH_GOOGLE_JWT_SIGNING_KEY=mcp-jwt-key:latest" \
  --set-secrets="WORKSPACE_MCP_OAUTH_PROXY_VALKEY_PASSWORD=valkey-password:latest" \
  --service-account=workspace-mcp@your-project.iam.gserviceaccount.com \
  --vpc-connector=workspace-mcp \
  --min-instances=1 \
  --max-instances=10 \
  --ingress=internal-and-cloud-load-balancing \
  --allow-unauthenticated

Configure VPC routing, firewall rules, TLS, and the connector for your Valkey service. Configure an HTTP startup probe for /health explicitly in Cloud Run YAML or Terraform if you want that path used.

Trusted-Gateway Identity

In this mode an MCP-aware proxy authenticates each request and injects a signed JWT assertion. Workspace MCP verifies the assertion before selecting that user's Google grant.

# Trusted-gateway mode: do not enable OAuth 2.1 or stateless mode.
TRUST_GATEWAY_IDENTITY=true
WORKSPACE_MCP_HOST=0.0.0.0
WORKSPACE_EXTERNAL_URL=https://mcp.corp.example.com
GATEWAY_IDENTITY_JWKS_URL=https://authenticate.corp.example.com/.well-known/pomerium/jwks.json
GATEWAY_IDENTITY_AUDIENCE=workspace-mcp.corp.example.com

# Cloudflare Access overrides
# GATEWAY_IDENTITY_HEADER=cf-access-jwt-assertion
# GATEWAY_IDENTITY_ALGORITHMS=RS256
  • Leave MCP_ENABLE_OAUTH21 and WORKSPACE_MCP_STATELESS_MODE unset.
  • Set WORKSPACE_MCP_HOST=0.0.0.0 in a container; non-OAuth HTTP otherwise defaults to loopback.
  • Expose the service only through the proxy, which must overwrite the configured identity header.
  • Persist WORKSPACE_MCP_CREDENTIALS_DIR; the current GCS backend cannot be used in gateway mode.

See the trusted-gateway reference for provider-specific headers and algorithms.

Domain-Wide Delegation

DWD removes per-user Google consent, but it does not authenticate callers to the MCP endpoint. For user-facing HTTP deployments, combine it with trusted-gateway identity so the verified gateway principal becomes the impersonation subject.

# DWD can be combined with trusted-gateway identity.
GOOGLE_SERVICE_ACCOUNT_KEY_FILE=/secrets/service-account.json
[email protected]
DWD_ALLOWED_DOMAINS=corp.example.com,subsidiary.example.com
  • Set exactly one of GOOGLE_SERVICE_ACCOUNT_KEY_FILE and GOOGLE_SERVICE_ACCOUNT_KEY_JSON.
  • USER_GOOGLE_EMAIL is required at startup and supplies the fallback subject.
  • Set DWD_ALLOWED_DOMAINS to restrict request-selected subjects.
  • Authorize only the required Workspace scopes in the Admin console.

GCS Credential Storage

GCS stores per-user Google grants for stateful OAuth 2.1 deployments and uses generation preconditions to reject conflicting writes. It is not the state store for stateless OAuth proxy tokens or MCP transport sessions.

# GCS is supported for stateful OAuth 2.1 deployments.
# Do not combine it with WORKSPACE_MCP_STATELESS_MODE=true.
MCP_ENABLE_OAUTH21=true
WORKSPACE_MCP_CREDENTIAL_STORE_BACKEND=gcs
WORKSPACE_MCP_GCS_BUCKET=workspace-mcp-credentials
WORKSPACE_MCP_GCS_PREFIX=production
WORKSPACE_MCP_GCS_REQUIRE_CMEK=true
  • Build the image with --extra gcs; the repository Dockerfile does not include it.
  • Grant the runtime identity roles/storage.objectUser on the bucket.
  • For CMEK enforcement, also grant storage.buckets.get, for example through roles/storage.bucketViewer.
  • The startup check verifies the bucket's default KMS key; KMS permissions and rotation remain infrastructure responsibilities.

Operations and Observability

Tracing

The repository image includes the OpenTelemetry extra. Tracing remains off until an OTLP endpoint is configured. Both gRPC and HTTP/protobuf are supported.

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.corp.example.com:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=workspace-mcp-prod
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

# Optional PII: adds user.email to active spans
# WORKSPACE_MCP_OTEL_USER_EMAIL=true

Logging

Console logs are human-readable text written to stdout or stderr; they are not JSON. Set WORKSPACE_MCP_LOG_LEVEL to CRITICAL, ERROR, WARNING, INFO (the default), or DEBUG; invalid values fall back to INFO. Stateless mode disables file logging. Otherwise, WORKSPACE_MCP_LOG_DIR enables a detailed local debug log. Treat DEBUG output, detailed file logs, and optional user.email span attributes as sensitive because they can include user text.

Health and failure detection

GET /health returns process metadata only. Pair it with checks for Valkey connectivity, OAuth failures, Google API error rates, latency, and storage fallback warnings.

Permission Controls

# Limit scopes and exposed tools by service.
WORKSPACE_MCP_PERMISSIONS="gmail:send drive:full calendar:full"

# Optionally narrow that selection to core-tier tools.
WORKSPACE_MCP_TOOL_TIER=core

# Names must match registered MCP tools exactly.
WORKSPACE_MCP_DISABLED_TOOLS=send_gmail_message,create_drive_file

Permission levels determine requested Google scopes and eligible tools. A tier can narrow tools within those services. The disabled list wins over both, but an unmatched name only warns; verify startup output after changing it.

Release Checklist

  • The auth mode matches the compatibility table, and incompatible variables are absent.
  • The pinned image contains every optional backend dependency configured at runtime.
  • WORKSPACE_EXTERNAL_URL and the Google callback use the final HTTPS hostname.
  • OAuth secrets, signing keys, Valkey credentials, and service-account material come from a secret manager.
  • Valkey survives instance replacement and is reachable from every stateless replica.
  • Gateway deployments cannot bypass the proxy, and incoming identity headers are overwritten.
  • DWD domains and Admin-console scopes are restricted to intended users and services.
  • Permission and blocklist settings have been checked against registered tools.
  • Dependency monitoring supplements the process-only /health endpoint.
  • Restore, rotation, rollout, and rollback procedures have been exercised.