Part 4 ended with a gateway in front of one MCP server that had no credentials and no way out of its namespace. This part fills in the rest. Six MCP servers now run behind that gateway: Grafana, GitLab, n8n, Vault, Kubernetes and listmonk. Each gets its own credentials from Vault by proving who it is with SPIFFE. Claude Code and Codex point at the gateway and keep no tokens in their config files.

The goal we set for this phase was short: all six usable from both clients, and no secret left in any client config. Most of it worked the first time. The parts that didn’t are below.

One pattern for credentials #

Every server needs a credential for the system behind it: an API token for GitLab, a service-account token for Grafana, an API key for n8n. In phase 1 those tokens lived in the AI clients’ config files. Now they live in Vault, and the pod fetches them itself when it starts.

Each server pod has two init containers:

  1. spiffe-helper asks the SPIRE agent for a JWT-SVID with the audience vault, writes it to a file and exits.
  2. Vault Agent logs in on the auth/spiffe mount from part 3, renders the server’s secrets into an env file, deletes the JWT and exits (exit_after_auth).

The main container sources that env file and starts the server:

command:
  - /bin/sh
  - -c
  - . /run/creds/env && exec /app/mcp-grafana -t streamable-http --address=127.0.0.1:8000

/run/creds is an emptyDir with medium: Memory. No Kubernetes Secret is involved, nothing touches the node’s disk, and the Vault token lives 60 seconds and is never written anywhere. Each Vault role is bound to exactly one service account and can read only that server’s own paths. The GitLab server can’t read the n8n key, and the reverse.

A Vault Agent config for one server is short:

exit_after_auth = true

auto_auth {
  method "jwt" {
    mount_path = "auth/spiffe"
    config = {
      role                     = "mcp-dev-gitlab"
      path                     = "/run/creds/vault.jwt"
      remove_jwt_after_reading = true
    }
  }
}

template {
  source               = "/etc/mcp-creds/gitlab.env.tmpl"
  destination          = "/run/creds/env"
  perms                = "0400"
  error_on_missing_key = true
}

Everything around it is Terraform: the Vault roles, the policies, and the secrets where we can generate them. Where an upstream system has to issue the credential, Terraform issues and rotates it too. The Grafana token comes from a small new module that creates a Viewer service account, gives its token 90 days, and replaces the token after 60. The GitLab token is a group access token with rotation turned on, written straight into Vault by the same apply.

The six servers #

ServerCredentialIts own limit
GrafanaViewer service-account token, rotated by TerraformViewer role
GitLabgroup access token, Developer, rotated by Terraformno delete tools
n8nexisting n8n API keynetwork policy: arcturus only
Vaultno token in the pod (see below)can list mounts and secret paths, never read a value
Kubernetesthe pod’s own service-account tokenread-only, ClusterRole view
listmonkexisting API userlistmonk role: no campaign send, no settings

Two servers don’t follow the pattern exactly.

The Vault MCP server talks to Vault itself, so it needs a token for as long as it runs. Instead of rendering one once, the pod runs spiffe-helper and Vault Agent as sidecars. The agent keeps a fresh login and runs an API proxy on 127.0.0.1:8200 that adds the token to every request. The server is pointed at the proxy and never holds a token itself. It does insist on VAULT_TOKEN being set, so it gets a placeholder value that the proxy replaces.

The Kubernetes MCP server doesn’t need Vault at all. A projected service-account token, mounted only into that one container, is already short-lived and rotated by the kubelet. Its service account has the built-in view ClusterRole, which can’t read Secrets, and the server runs with read_only = true.

Getting to the servers #

Each server listens on 127.0.0.1 only. The single way into the pod is ghostunnel on port 8443, which accepts a client certificate from SPIRE and only the one with the gateway’s SPIFFE ID. The gateway reaches every server over mTLS with its own certificate. A network policy also allows ingress only from the gateway pod, so there are two locks: the network path and the certificate.

GitLab’s and n8n’s MCP servers refuse to run over HTTP without a bearer token of their own. Those tokens are generated in Vault. The gateway renders them to a file at startup and sends them with backendAuth, which reloads the file when it changes. They exist only between the gateway and the server, and no client ever sees them.

People now sign in on our real Keycloak realm instead of the test realm from part 4. Login there is limited to the platform-admins group, and the gateway rule on every route is the same:

mcpAuthorization:
  rules:
  - 'jwt.azp == "mcp-cli" && "platform-admins" in jwt.groups'

What didn’t work the first time #

Sessions. We turned on agentgateway’s encrypted session state, the shortcut from part 4, with its key from Vault. Our smoke test then failed on every call after initialize with a 400. Stateful sessions mean the client has to send back the Mcp-Session-Id header from the first response. Real clients do. Our curl script didn’t.

n8n. Its MCP server has SSRF protection that refuses private IP addresses. Our n8n is on a private address, so every call failed with “Private IP addresses not allowed”. We set WEBHOOK_SECURITY_MODE=permissive and leave the boundary to the network policy, which lets that pod reach one host on one port.

Host headers. The MCP SDKs guard against DNS rebinding by checking the Host header. Behind the gateway that header is the Kubernetes Service name, not localhost. Each server needed it on its allow-list in its own way: --allowed-hosts, MCP_ALLOWED_HOSTS, or disable_localhost_protection. We listed exact names, never *.

The Kubernetes API. The API server runs on the host network, so a normal network policy with an IP block doesn’t match it. Cilium needs toEntities: kube-apiserver.

listmonk. This one took the most work. The listmonk MCP server is only published on PyPI and only runs over stdio. We expected to need a stdio-to-HTTP bridge, but it’s built on the MCP SDK’s FastMCP, which can serve streamable HTTP by itself. So we built a small image with an entry point of about a hundred lines and left upstream’s code untouched.

Writing that entry point turned up a bug that only shows over HTTP. The SDK runs the server’s startup and shutdown hook once per HTTP session, not once per process. listmonk-mcp opens its API client in that hook, keeps it in a global variable and closes it on exit. Over HTTP, the first session to end would close the client for everyone else. Our entry point replaces that hook with the SDK’s empty default and opens one client per process instead. A test covers it. Any FastMCP server you move from stdio to HTTP is worth checking for the same thing.

The firewall. The listmonk pod then timed out on every request. Hubble showed the packets leaving the node, with no reply. Our firewall keeps the management cluster, where listmonk runs, closed to non-production by default and drops silently, which looks just like a dead service. We opened one rule, from non-production to the management ingress on port 443, through the firewall’s own repository and deploy pipeline.

How we tested it #

Two ways.

As a workload. A short-lived pod with its own SPIFFE identity, spiffe-smoke, asked SPIRE for a JWT-SVID and called every server through the gateway. Its rule allows exactly one tool per server. Every server returned real data for that tool: Grafana’s datasources, GitLab’s projects, n8n’s health, Vault’s mounts, the cluster’s namespaces and listmonk’s health. A second tool on every server was refused by the gateway with a 400.

As a person. We signed in from Claude Code and from Codex through the browser and called a read-only tool on each of the six, from both. All six returned real data under our own login in both clients. That also closes the one item part 4 left open: a tool call from Codex through the gateway.

Then we cleaned up the client configs. The old entries in Claude Code and Codex carried a GitLab personal access token, a Grafana key, an n8n key, a Vault token and a listmonk password. They’re gone. Each server is now one entry like this one:

{
  "type": "http",
  "url": "https://mcp-dev.bsdserver.nl/gitlab/mcp",
  "oauth": { "clientId": "mcp-cli" }
}

Neither config file contains a token for any of the six servers any more.

Still open #

  • Rotation. The secrets from the old client configs sat in plain text in config files on a laptop. Removing them from the files doesn’t make them safe again. They get rotated.
  • One credential still isn’t short-lived. n8n and listmonk only accept static API keys. Those keys now live in Vault and nowhere else, but they don’t expire on their own.

Next #

This is still the dev stage, on our non-production cluster. The next part moves the platform to production: its own SPIRE trust domain, its own Vault mount, and the other AI clients in the house.