Part 1 named agentgateway as the gateway candidate and listed what a proof of concept had to show. It had to accept tokens from both Keycloak and SPIRE, use mTLS with SPIFFE certificates to reach the MCP servers, and work with the OAuth discovery of real MCP clients. This part is that proof of concept, on our non-production Talos cluster, with one MCP server behind it.

All three held up. The third took two wrong turns first.

The setup #

client ──JWT──> HAProxy ingress (mcp-dev.bsdserver.nl, internal allow-list)
       ──> agentgateway ──mTLS (X.509-SVIDs)──> ghostunnel ──loopback──> mcp-grafana

Everything lives in one namespace, mcp-dev, deployed by Flux. The namespace carries the spiffe.io/enabled=true label from part 3, so every pod in it gets a SPIFFE ID derived from its service account. Each workload has its own service account, and none of them mounts a Kubernetes API token.

The test server is mcp-grafana 2.3.0. It listens on 127.0.0.1 only, so nothing outside its own pod can reach it.

The sidecar is ghostunnel 1.11.5. It is the pod’s only network entry. It takes its certificate from the SPIRE agent and accepts exactly one client identity:

args:
  - server
  - --listen=0.0.0.0:8443
  - --target=127.0.0.1:8000
  - --use-workload-api-addr=unix:///spiffe-workload-api/spire-agent.sock
  - --allow-uri=spiffe://nonprod.bsdserver.lan/ns/mcp-dev/sa/mcp-gateway
  - --status=http://0.0.0.0:6060

The gateway is agentgateway 1.7.0. We run it standalone, from a config file, not through its Kubernetes Gateway API controller. For one gateway with a handful of routes, a config file in Git is easier to read and review.

Network policies start at default deny. The MCP server pods accept traffic only from the gateway, on the mTLS port. The policy and the certificate check overlap on purpose: either one alone would stop a stray pod.

Two small things broke on the way. Neither is specific to MCP, but both cost time:

  • ghostunnel’s --status listener speaks TLS unless the address starts with http://. The kubelet’s readiness probe then got “HTTP request to an HTTPS server”.
  • mcp-grafana protects against DNS rebinding by checking the Host header. By default only loopback names are allowed, so every request from the gateway got a 403. The fix is --allowed-hosts with the Service name, never *.

Workloads: a route for JWT-SVIDs #

The gateway gets its own SPIFFE identity from the SPIRE agent, the same way the backend does:

config:
  spiffe:
    endpoint: unix:///spiffe-workload-api/spire-agent.sock

The workload route validates JWT-SVIDs with audience mcp-gateway and applies rules written in CEL. Anything a rule doesn’t allow is denied:

- name: grafana-workload
  matches:
  - path: {exact: /workload/grafana/mcp}
  policies:
    jwtAuth:
      mode: strict
      providers:
      - issuer: https://spire-oidc-nonprod.bsdserver.nl
        audiences: [mcp-gateway]
        jwks:
          url: http://spire-spiffe-oidc-discovery-provider.spire-server.svc.cluster.local/keys
    mcpAuthorization:
      rules:
      - 'jwt.sub == "spiffe://nonprod.bsdserver.lan/ns/mcp-dev/sa/spiffe-smoke" && mcp.tool.name == "list_datasources"'
  backends:
  - mcp:
      targets:
      - name: grafana
        mcp:
          host: grafana-mcp.mcp-dev.svc.cluster.local
          port: 8443
          path: /mcp
        policies:
          backendTLS:
            spiffe: {}
            subjectAltNames:
            - spiffe://nonprod.bsdserver.lan/ns/mcp-dev/sa/grafana-mcp

Two details stand out. The JWKS comes from the in-cluster Service, because the ingress allow-list from part 3 doesn’t admit pod addresses. And backendTLS with subjectAltNames means the gateway checks the backend’s identity as well. mTLS here works in both directions.

We tested from a pod running under the spiffe-smoke service account, through the real ingress:

TestResult
No token401, with a WWW-Authenticate header
JWT-SVID with audience vault401
Valid JWT-SVID, initialize200
tools/listOnly list_datasources; the rest of mcp-grafana’s tools are filtered out
tools/call on another toolDenied by the gateway
A pod with the gateway’s labels but a different SPIFFE ID, calling the backend directlyRefused by ghostunnel: principal not allowed
Plain HTTP to the backend portRefused

The impostor test matters most. The network policy lets that pod through, because it matches on labels and the impostor copied them. The certificate check is what stops it.

People: Keycloak OAuth #

MCP clients discover how to sign in. The gateway answers an unauthenticated request with a 401 and a link to its protected-resource metadata (RFC 9728). That document names an authorization server. The client then fetches the authorization server’s own metadata (RFC 8414) and runs an OAuth flow with PKCE.

agentgateway’s mcpAuthentication policy handles the gateway side. One limitation shaped the config: it takes one issuer per route. So people and workloads get separate paths, /grafana/mcp and /workload/grafana/mcp, in front of the same backend.

The Keycloak side, in Terraform #

Our rule is that new Keycloak clients are first tried in a disposable test realm. I first proposed creating that realm with Keycloak’s admin CLI inside the pod. That was rejected straight away, and rightly: we don’t change infrastructure by hand, and a test realm is no exception.

The Keycloak Terraform module we use needed two additions before it could manage a second, disposable realm:

  • Test users with generated passwords. The module now creates a random password and writes it to Vault, so no password ends up in Git.
  • A switch to skip the module’s own Terraform client. That client exists once, in the main realm. A second unit would otherwise try to create it again.

The realm test-mcp then holds:

  • a bearer-only client mcp-gateway, which only serves as the token audience;
  • a client scope mcp that puts mcp-gateway in the audience;
  • a public client mcp-cli with PKCE (S256), the device grant and loopback redirect URIs, which every CLI uses;
  • one test user, mcp-tester.

The gateway route:

- name: grafana-people
  matches:
  - path: {exact: /grafana/mcp}
  - path: {exact: /.well-known/oauth-protected-resource/grafana/mcp}
  policies:
    mcpAuthentication:
      mode: strict
      issuer: https://keycloak.bsdserver.nl/realms/test-mcp
      audiences: [mcp-gateway]
      jwks:
        url: https://keycloak.bsdserver.nl/realms/test-mcp/protocol/openid-connect/certs
      resourceMetadata:
        resource: https://mcp-dev.bsdserver.nl/grafana/mcp
        scopesSupported: [openid, mcp]
        bearerMethodsSupported: [header]
    mcpAuthorization:
      rules:
      - 'jwt.azp == "mcp-cli" && jwt.preferred_username == "mcp-tester" && mcp.tool.name in ["list_datasources", "search_dashboards"]'

The wrong turn: the Keycloak adapter #

agentgateway has a provider: keycloak option for this policy. The documentation presents it as the way to make Keycloak work with MCP clients, so the first version used it. Discovery looked fine, and Codex started the sign-in. Then it stopped:

Authorization server issuer mismatch:
expected https://mcp-dev.bsdserver.nl/grafana/mcp,
received https://keycloak.bsdserver.nl/realms/test-mcp

The adapter serves its own copy of the authorization server metadata, with the gateway’s URL as issuer. Keycloak, meanwhile, puts its real issuer in the iss parameter of the authorization response (RFC 9207). That parameter exists to stop mix-up attacks, and Codex checks it strictly. The two values disagree, so Codex refuses.

My first fix was to make Keycloak stop sending iss. That needed a third change to the Terraform module, and it made things worse:

Authorization server response missing required issuer:
expected https://mcp-dev.bsdserver.nl/grafana/mcp

Keycloak’s metadata says it supports iss in responses. A strict client then requires the parameter. Removing it can’t work, and it would have weakened a security check to work around a proxy.

The real fix was to remove the adapter. Keycloak 26.7 serves RFC 8414 metadata itself, at both well-known locations, so the gateway doesn’t need to imitate it. Without the adapter, the protected-resource metadata names Keycloak directly. The issuer the client expects, the issuer in the metadata and the iss in the response are then all the same. We reverted the Keycloak change, and the module setting became tri-state so that “not managed” is the default again.

There is a cost. The adapter also answered dynamic client registration with our pre-registered client. Without it, clients have to be told which client ID to use. That’s one line of configuration:

claude mcp add --transport http --client-id mcp-cli grafana-dev https://mcp-dev.bsdserver.nl/grafana/mcp
# ~/.codex/config.toml
[mcp_servers.grafana-dev]
url = "https://mcp-dev.bsdserver.nl/grafana/mcp"

[mcp_servers.grafana-dev.oauth]
client_id = "mcp-cli"

Neither file holds a secret. mcp-cli is a public client, and the tokens live in each tool’s own credential store.

What we saw #

Claude Code signed in through the browser as mcp-tester. The gateway log shows tools/list returning exactly list_datasources and search_dashboards, out of mcp-grafana’s full tool set. Both tool calls were authorized and forwarded to the backend over mTLS.

Both calls then failed with a 500, which we expected. In this proof of concept mcp-grafana has no network path to Grafana and no credentials for it. The gateway’s part worked. The backend’s part comes in phase 2.

Codex completed the sign-in and stored its token once the adapter was gone. We haven’t yet run a tool call through Codex. That is the one piece of this proof of concept still to be shown.

Shortcuts to remove #

A proof of concept takes shortcuts, and these have to go before real MCP servers move in:

  • Session encryption is off. agentgateway encrypts MCP session state with a key. That key has to come from Vault, so we turned the feature off for now.
  • Backends are stateless, which follows from the previous point.
  • mcp-grafana has no egress and no credentials, as described above.

Verdict #

agentgateway stays. It validates tokens from both issuers, it filters tool lists and tool calls per caller, and it speaks SPIFFE mTLS to the backends with certificates it gets from SPIRE. The two points that need care are that one route can only have one issuer, and that the Keycloak adapter is better left out.

The next part moves real MCP servers in, each with its own Vault role for its credentials.