<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>SPIFFE | A DevOps Blog</title><link>https://blog.bsdserver.nl/tag/spiffe/</link><description>Hands-on DevOps and homelab content: Kubernetes clusters, self-hosted services, CI/CD pipelines and infrastructure as code, by Patrick de Ruiter of We Build Your Cloud.</description><language>en</language><image><url>https://blog.bsdserver.nl/images/app-icon.svg</url><title>A DevOps Blog</title><link>https://blog.bsdserver.nl/</link></image><lastBuildDate>Fri, 20 Nov 2026 08:00:00 +0100</lastBuildDate><atom:link href="https://blog.bsdserver.nl/tag/spiffe/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title>Six MCP Servers, and No Secret in Any Client</title><link>https://blog.bsdserver.nl/six-mcp-servers-and-no-secret-in-any-client/</link><guid isPermaLink="true">https://blog.bsdserver.nl/six-mcp-servers-and-no-secret-in-any-client/</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 20 Nov 2026 08:00:00 +0100</pubDate><description>Phase 2 of our MCP platform. Six MCP servers on Talos get their credentials from Vault with their SPIFFE identity, and the AI clients keep no tokens at all.</description><category>MCP</category><category>SPIFFE</category><category>Vault</category><category>Kubernetes</category><category>Security</category><media:content url="https://blog.bsdserver.nl/content/images/2026/11/six-mcp-servers-no-client-secret.png" medium="image"/><content:encoded><![CDATA[<p>Part 4 ended with a gateway in front of one MCP server that had no credentials and no way out of its namespace. This part fills in the rest. Six MCP servers now run behind that gateway: Grafana, GitLab, n8n, Vault, Kubernetes and listmonk. Each gets its own credentials from Vault by proving who it is with SPIFFE. Claude Code and Codex point at the gateway and keep no tokens in their config files.</p>
<p>The goal we set for this phase was short: all six usable from both clients, and no secret left in any client config. Most of it worked the first time. The parts that didn&rsquo;t are below.</p>
<h2 id="one-pattern-for-credentials">One pattern for credentials <a class="anchor" data-pagefind-ignore href="#one-pattern-for-credentials" aria-label="Link to this section">#</a></h2>
<p>Every server needs a credential for the system behind it: an API token for GitLab, a service-account token for Grafana, an API key for n8n. In phase 1 those tokens lived in the AI clients&rsquo; config files. Now they live in Vault, and the pod fetches them itself when it starts.</p>
<p>Each server pod has two init containers:</p>
<ol>
<li><strong>spiffe-helper</strong> asks the SPIRE agent for a JWT-SVID with the audience <code>vault</code>, writes it to a file and exits.</li>
<li><strong>Vault Agent</strong> logs in on the <code>auth/spiffe</code> mount from part 3, renders the server&rsquo;s secrets into an env file, deletes the JWT and exits (<code>exit_after_auth</code>).</li>
</ol>
<p>The main container sources that env file and starts the server:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">command</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">/bin/sh</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- -<span class="l">c</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">. /run/creds/env &amp;&amp; exec /app/mcp-grafana -t streamable-http --address=127.0.0.1:8000</span><span class="w">
</span></span></span></code></pre></div><p><code>/run/creds</code> is an <code>emptyDir</code> with <code>medium: Memory</code>. No Kubernetes Secret is involved, nothing touches the node&rsquo;s disk, and the Vault token lives 60 seconds and is never written anywhere. Each Vault role is bound to exactly one service account and can read only that server&rsquo;s own paths. The GitLab server can&rsquo;t read the n8n key, and the reverse.</p>
<p>A Vault Agent config for one server is short:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="n">exit_after_auth</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">auto_auth</span> {
</span></span><span class="line"><span class="cl">  <span class="k">method</span> <span class="s2">&#34;jwt&#34;</span> {
</span></span><span class="line"><span class="cl"><span class="n">    mount_path</span> <span class="o">=</span> <span class="s2">&#34;auth/spiffe&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">    config</span> <span class="o">=</span> {
</span></span><span class="line"><span class="cl"><span class="n">      role</span>                     <span class="o">=</span> <span class="s2">&#34;mcp-dev-gitlab&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      path</span>                     <span class="o">=</span> <span class="s2">&#34;/run/creds/vault.jwt&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      remove_jwt_after_reading</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">template</span> {
</span></span><span class="line"><span class="cl"><span class="n">  source</span>               <span class="o">=</span> <span class="s2">&#34;/etc/mcp-creds/gitlab.env.tmpl&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  destination</span>          <span class="o">=</span> <span class="s2">&#34;/run/creds/env&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  perms</span>                <span class="o">=</span> <span class="s2">&#34;0400&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  error_on_missing_key</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><p>Everything around it is Terraform: the Vault roles, the policies, and the secrets where we can generate them. Where an upstream system has to issue the credential, Terraform issues and rotates it too. The Grafana token comes from a small new module that creates a Viewer service account, gives its token 90 days, and replaces the token after 60. The GitLab token is a group access token with rotation turned on, written straight into Vault by the same apply.</p>
<h2 id="the-six-servers">The six servers <a class="anchor" data-pagefind-ignore href="#the-six-servers" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Server</th>
					<th>Credential</th>
					<th>Its own limit</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Grafana</td>
					<td>Viewer service-account token, rotated by Terraform</td>
					<td>Viewer role</td>
			</tr>
			<tr>
					<td>GitLab</td>
					<td>group access token, Developer, rotated by Terraform</td>
					<td>no delete tools</td>
			</tr>
			<tr>
					<td>n8n</td>
					<td>existing n8n API key</td>
					<td>network policy: arcturus only</td>
			</tr>
			<tr>
					<td>Vault</td>
					<td>no token in the pod (see below)</td>
					<td>can list mounts and secret paths, never read a value</td>
			</tr>
			<tr>
					<td>Kubernetes</td>
					<td>the pod&rsquo;s own service-account token</td>
					<td>read-only, ClusterRole <code>view</code></td>
			</tr>
			<tr>
					<td>listmonk</td>
					<td>existing API user</td>
					<td>listmonk role: no campaign send, no settings</td>
			</tr>
	</tbody>
</table>
<p>Two servers don&rsquo;t follow the pattern exactly.</p>
<p><strong>The Vault MCP server</strong> talks to Vault itself, so it needs a token for as long as it runs. Instead of rendering one once, the pod runs spiffe-helper and Vault Agent as sidecars. The agent keeps a fresh login and runs an API proxy on <code>127.0.0.1:8200</code> that adds the token to every request. The server is pointed at the proxy and never holds a token itself. It does insist on <code>VAULT_TOKEN</code> being set, so it gets a placeholder value that the proxy replaces.</p>
<p><strong>The Kubernetes MCP server</strong> doesn&rsquo;t need Vault at all. A projected service-account token, mounted only into that one container, is already short-lived and rotated by the kubelet. Its service account has the built-in <code>view</code> ClusterRole, which can&rsquo;t read Secrets, and the server runs with <code>read_only = true</code>.</p>
<h2 id="getting-to-the-servers">Getting to the servers <a class="anchor" data-pagefind-ignore href="#getting-to-the-servers" aria-label="Link to this section">#</a></h2>
<p>Each server listens on <code>127.0.0.1</code> only. The single way into the pod is <a href="https://github.com/ghostunnel/ghostunnel" rel="noopener">ghostunnel</a> on port 8443, which accepts a client certificate from SPIRE and only the one with the gateway&rsquo;s SPIFFE ID. The gateway reaches every server over mTLS with its own certificate. A network policy also allows ingress only from the gateway pod, so there are two locks: the network path and the certificate.</p>
<p>GitLab&rsquo;s and n8n&rsquo;s MCP servers refuse to run over HTTP without a bearer token of their own. Those tokens are generated in Vault. The gateway renders them to a file at startup and sends them with <code>backendAuth</code>, which reloads the file when it changes. They exist only between the gateway and the server, and no client ever sees them.</p>
<p>People now sign in on our real Keycloak realm instead of the test realm from part 4. Login there is limited to the platform-admins group, and the gateway rule on every route is the same:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">mcpAuthorization</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s1">&#39;jwt.azp == &#34;mcp-cli&#34; &amp;&amp; &#34;platform-admins&#34; in jwt.groups&#39;</span><span class="w">
</span></span></span></code></pre></div><h2 id="what-didnt-work-the-first-time">What didn&rsquo;t work the first time <a class="anchor" data-pagefind-ignore href="#what-didnt-work-the-first-time" aria-label="Link to this section">#</a></h2>
<p><strong>Sessions.</strong> We turned on agentgateway&rsquo;s encrypted session state, the shortcut from part 4, with its key from Vault. Our smoke test then failed on every call after <code>initialize</code> with a 400. Stateful sessions mean the client has to send back the <code>Mcp-Session-Id</code> header from the first response. Real clients do. Our curl script didn&rsquo;t.</p>
<p><strong>n8n.</strong> Its MCP server has SSRF protection that refuses private IP addresses. Our n8n is on a private address, so every call failed with &ldquo;Private IP addresses not allowed&rdquo;. We set <code>WEBHOOK_SECURITY_MODE=permissive</code> and leave the boundary to the network policy, which lets that pod reach one host on one port.</p>
<p><strong>Host headers.</strong> The MCP SDKs guard against DNS rebinding by checking the <code>Host</code> header. Behind the gateway that header is the Kubernetes Service name, not <code>localhost</code>. Each server needed it on its allow-list in its own way: <code>--allowed-hosts</code>, <code>MCP_ALLOWED_HOSTS</code>, or <code>disable_localhost_protection</code>. We listed exact names, never <code>*</code>.</p>
<p><strong>The Kubernetes API.</strong> The API server runs on the host network, so a normal network policy with an IP block doesn&rsquo;t match it. Cilium needs <code>toEntities: kube-apiserver</code>.</p>
<p><strong>listmonk.</strong> This one took the most work. The listmonk MCP server is only published on PyPI and only runs over stdio. We expected to need a stdio-to-HTTP bridge, but it&rsquo;s built on the MCP SDK&rsquo;s FastMCP, which can serve streamable HTTP by itself. So we built a small image with an entry point of about a hundred lines and left upstream&rsquo;s code untouched.</p>
<p>Writing that entry point turned up a bug that only shows over HTTP. The SDK runs the server&rsquo;s startup and shutdown hook once per HTTP session, not once per process. listmonk-mcp opens its API client in that hook, keeps it in a global variable and closes it on exit. Over HTTP, the first session to end would close the client for everyone else. Our entry point replaces that hook with the SDK&rsquo;s empty default and opens one client per process instead. A test covers it. Any FastMCP server you move from stdio to HTTP is worth checking for the same thing.</p>
<p><strong>The firewall.</strong> The listmonk pod then timed out on every request. Hubble showed the packets leaving the node, with no reply. Our firewall keeps the management cluster, where listmonk runs, closed to non-production by default and drops silently, which looks just like a dead service. We opened one rule, from non-production to the management ingress on port 443, through the firewall&rsquo;s own repository and deploy pipeline.</p>
<h2 id="how-we-tested-it">How we tested it <a class="anchor" data-pagefind-ignore href="#how-we-tested-it" aria-label="Link to this section">#</a></h2>
<p>Two ways.</p>
<p><strong>As a workload.</strong> A short-lived pod with its own SPIFFE identity, <code>spiffe-smoke</code>, asked SPIRE for a JWT-SVID and called every server through the gateway. Its rule allows exactly one tool per server. Every server returned real data for that tool: Grafana&rsquo;s datasources, GitLab&rsquo;s projects, n8n&rsquo;s health, Vault&rsquo;s mounts, the cluster&rsquo;s namespaces and listmonk&rsquo;s health. A second tool on every server was refused by the gateway with a 400.</p>
<p><strong>As a person.</strong> We signed in from Claude Code and from Codex through the browser and called a read-only tool on each of the six, from both. All six returned real data under our own login in both clients. That also closes the one item part 4 left open: a tool call from Codex through the gateway.</p>
<p>Then we cleaned up the client configs. The old entries in Claude Code and Codex carried a GitLab personal access token, a Grafana key, an n8n key, a Vault token and a listmonk password. They&rsquo;re gone. Each server is now one entry like this one:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;http&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;url&#34;</span><span class="p">:</span> <span class="s2">&#34;https://mcp-dev.bsdserver.nl/gitlab/mcp&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;oauth&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;clientId&#34;</span><span class="p">:</span> <span class="s2">&#34;mcp-cli&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>Neither config file contains a token for any of the six servers any more.</p>
<h2 id="still-open">Still open <a class="anchor" data-pagefind-ignore href="#still-open" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Rotation.</strong> The secrets from the old client configs sat in plain text in config files on a laptop. Removing them from the files doesn&rsquo;t make them safe again. They get rotated.</li>
<li><strong>One credential still isn&rsquo;t short-lived.</strong> n8n and listmonk only accept static API keys. Those keys now live in Vault and nowhere else, but they don&rsquo;t expire on their own.</li>
</ul>
<h2 id="next">Next <a class="anchor" data-pagefind-ignore href="#next" aria-label="Link to this section">#</a></h2>
<p>This is still the dev stage, on our non-production cluster. The next part moves the platform to production: its own SPIRE trust domain, its own Vault mount, and the other AI clients in the house.</p>
]]></content:encoded></item><item><title>An MCP Gateway With Keycloak and SPIFFE: the Proof of Concept</title><link>https://blog.bsdserver.nl/an-mcp-gateway-with-keycloak-and-spiffe/</link><guid isPermaLink="true">https://blog.bsdserver.nl/an-mcp-gateway-with-keycloak-and-spiffe/</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Thu, 19 Nov 2026 08:05:00 +0100</pubDate><description>agentgateway in front of an MCP server on Talos. SPIRE JWT-SVIDs for workloads, Keycloak OAuth for people, mTLS to the backend, and one issuer mistake.</description><category>MCP</category><category>SPIFFE</category><category>mTLS</category><category>Kubernetes</category><category>Security</category><media:content url="https://blog.bsdserver.nl/content/images/2026/11/mcp-gateway-poc.png" medium="image"/><content:encoded><![CDATA[<p>Part 1 named <a href="https://agentgateway.dev/" rel="noopener">agentgateway</a> as the gateway candidate and listed what a proof of concept had to show. It had to accept tokens from both Keycloak and SPIRE, use mTLS with SPIFFE certificates to reach the MCP servers, and work with the OAuth discovery of real MCP clients. This part is that proof of concept, on our non-production Talos cluster, with one MCP server behind it.</p>
<p>All three held up. The third took two wrong turns first.</p>
<h2 id="the-setup">The setup <a class="anchor" data-pagefind-ignore href="#the-setup" aria-label="Link to this section">#</a></h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">client ──JWT──&gt; HAProxy ingress (mcp-dev.bsdserver.nl, internal allow-list)
</span></span><span class="line"><span class="cl">       ──&gt; agentgateway ──mTLS (X.509-SVIDs)──&gt; ghostunnel ──loopback──&gt; mcp-grafana
</span></span></code></pre></div><p>Everything lives in one namespace, <code>mcp-dev</code>, deployed by Flux. The namespace carries the <code>spiffe.io/enabled=true</code> label from part 3, so every pod in it gets a SPIFFE ID derived from its service account. Each workload has its own service account, and none of them mounts a Kubernetes API token.</p>
<p><strong>The test server</strong> is <a href="https://github.com/grafana/mcp-grafana" rel="noopener">mcp-grafana</a> 2.3.0. It listens on <code>127.0.0.1</code> only, so nothing outside its own pod can reach it.</p>
<p><strong>The sidecar</strong> is <a href="https://github.com/ghostunnel/ghostunnel" rel="noopener">ghostunnel</a> 1.11.5. It is the pod&rsquo;s only network entry. It takes its certificate from the SPIRE agent and accepts exactly one client identity:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">args</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">server</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- --<span class="l">listen=0.0.0.0:8443</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- --<span class="l">target=127.0.0.1:8000</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- --<span class="l">use-workload-api-addr=unix:///spiffe-workload-api/spire-agent.sock</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- --<span class="l">allow-uri=spiffe://nonprod.bsdserver.lan/ns/mcp-dev/sa/mcp-gateway</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- --<span class="l">status=http://0.0.0.0:6060</span><span class="w">
</span></span></span></code></pre></div><p><strong>The gateway</strong> is agentgateway 1.7.0. We run it standalone, from a config file, not through its Kubernetes Gateway API controller. For one gateway with a handful of routes, a config file in Git is easier to read and review.</p>
<p><strong>Network policies</strong> start at default deny. The MCP server pods accept traffic only from the gateway, on the mTLS port. The policy and the certificate check overlap on purpose: either one alone would stop a stray pod.</p>
<p>Two small things broke on the way. Neither is specific to MCP, but both cost time:</p>
<ul>
<li>ghostunnel&rsquo;s <code>--status</code> listener speaks TLS unless the address starts with <code>http://</code>. The kubelet&rsquo;s readiness probe then got &ldquo;HTTP request to an HTTPS server&rdquo;.</li>
<li>mcp-grafana protects against DNS rebinding by checking the <code>Host</code> header. By default only loopback names are allowed, so every request from the gateway got a 403. The fix is <code>--allowed-hosts</code> with the Service name, never <code>*</code>.</li>
</ul>
<h2 id="workloads-a-route-for-jwt-svids">Workloads: a route for JWT-SVIDs <a class="anchor" data-pagefind-ignore href="#workloads-a-route-for-jwt-svids" aria-label="Link to this section">#</a></h2>
<p>The gateway gets its own SPIFFE identity from the SPIRE agent, the same way the backend does:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">config</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">spiffe</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">endpoint</span><span class="p">:</span><span class="w"> </span><span class="l">unix:///spiffe-workload-api/spire-agent.sock</span><span class="w">
</span></span></span></code></pre></div><p>The workload route validates JWT-SVIDs with audience <code>mcp-gateway</code> and applies rules written in CEL. Anything a rule doesn&rsquo;t allow is denied:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">grafana-workload</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">matches</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span>{<span class="nt">exact</span><span class="p">:</span><span class="w"> </span><span class="l">/workload/grafana/mcp}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">policies</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">jwtAuth</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l">strict</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">providers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">issuer</span><span class="p">:</span><span class="w"> </span><span class="l">https://spire-oidc-nonprod.bsdserver.nl</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">audiences</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">mcp-gateway]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">jwks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">http://spire-spiffe-oidc-discovery-provider.spire-server.svc.cluster.local/keys</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mcpAuthorization</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;jwt.sub == &#34;spiffe://nonprod.bsdserver.lan/ns/mcp-dev/sa/spiffe-smoke&#34; &amp;&amp; mcp.tool.name == &#34;list_datasources&#34;&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">backends</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">mcp</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">grafana</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mcp</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="l">grafana-mcp.mcp-dev.svc.cluster.local</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">8443</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/mcp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">policies</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">backendTLS</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">spiffe</span><span class="p">:</span><span class="w"> </span>{}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">subjectAltNames</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span>- <span class="l">spiffe://nonprod.bsdserver.lan/ns/mcp-dev/sa/grafana-mcp</span><span class="w">
</span></span></span></code></pre></div><p>Two details stand out. The JWKS comes from the in-cluster Service, because the ingress allow-list from part 3 doesn&rsquo;t admit pod addresses. And <code>backendTLS</code> with <code>subjectAltNames</code> means the gateway checks the backend&rsquo;s identity as well. mTLS here works in both directions.</p>
<p>We tested from a pod running under the <code>spiffe-smoke</code> service account, through the real ingress:</p>
<table>
	<thead>
			<tr>
					<th>Test</th>
					<th>Result</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>No token</td>
					<td>401, with a <code>WWW-Authenticate</code> header</td>
			</tr>
			<tr>
					<td>JWT-SVID with audience <code>vault</code></td>
					<td>401</td>
			</tr>
			<tr>
					<td>Valid JWT-SVID, <code>initialize</code></td>
					<td>200</td>
			</tr>
			<tr>
					<td><code>tools/list</code></td>
					<td>Only <code>list_datasources</code>; the rest of mcp-grafana&rsquo;s tools are filtered out</td>
			</tr>
			<tr>
					<td><code>tools/call</code> on another tool</td>
					<td>Denied by the gateway</td>
			</tr>
			<tr>
					<td>A pod with the gateway&rsquo;s labels but a different SPIFFE ID, calling the backend directly</td>
					<td>Refused by ghostunnel: principal not allowed</td>
			</tr>
			<tr>
					<td>Plain HTTP to the backend port</td>
					<td>Refused</td>
			</tr>
	</tbody>
</table>
<p>The impostor test matters most. The network policy lets that pod through, because it matches on labels and the impostor copied them. The certificate check is what stops it.</p>
<h2 id="people-keycloak-oauth">People: Keycloak OAuth <a class="anchor" data-pagefind-ignore href="#people-keycloak-oauth" aria-label="Link to this section">#</a></h2>
<p>MCP clients discover how to sign in. The gateway answers an unauthenticated request with a 401 and a link to its protected-resource metadata (RFC 9728). That document names an authorization server. The client then fetches the authorization server&rsquo;s own metadata (RFC 8414) and runs an OAuth flow with PKCE.</p>
<p>agentgateway&rsquo;s <code>mcpAuthentication</code> policy handles the gateway side. One limitation shaped the config: it takes one issuer per route. So people and workloads get separate paths, <code>/grafana/mcp</code> and <code>/workload/grafana/mcp</code>, in front of the same backend.</p>
<h3 id="the-keycloak-side-in-terraform">The Keycloak side, in Terraform <a class="anchor" data-pagefind-ignore href="#the-keycloak-side-in-terraform" aria-label="Link to this section">#</a></h3>
<p>Our rule is that new Keycloak clients are first tried in a disposable test realm. I first proposed creating that realm with Keycloak&rsquo;s admin CLI inside the pod. That was rejected straight away, and rightly: we don&rsquo;t change infrastructure by hand, and a test realm is no exception.</p>
<p>The Keycloak Terraform module we use needed two additions before it could manage a second, disposable realm:</p>
<ul>
<li><strong>Test users with generated passwords.</strong> The module now creates a random password and writes it to Vault, so no password ends up in Git.</li>
<li><strong>A switch to skip the module&rsquo;s own Terraform client.</strong> That client exists once, in the main realm. A second unit would otherwise try to create it again.</li>
</ul>
<p>The realm <code>test-mcp</code> then holds:</p>
<ul>
<li>a bearer-only client <code>mcp-gateway</code>, which only serves as the token audience;</li>
<li>a client scope <code>mcp</code> that puts <code>mcp-gateway</code> in the audience;</li>
<li>a public client <code>mcp-cli</code> with PKCE (S256), the device grant and loopback redirect URIs, which every CLI uses;</li>
<li>one test user, <code>mcp-tester</code>.</li>
</ul>
<p>The gateway route:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">grafana-people</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">matches</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span>{<span class="nt">exact</span><span class="p">:</span><span class="w"> </span><span class="l">/grafana/mcp}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span>{<span class="nt">exact</span><span class="p">:</span><span class="w"> </span><span class="l">/.well-known/oauth-protected-resource/grafana/mcp}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">policies</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mcpAuthentication</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l">strict</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">issuer</span><span class="p">:</span><span class="w"> </span><span class="l">https://keycloak.bsdserver.nl/realms/test-mcp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">audiences</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">mcp-gateway]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">jwks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">https://keycloak.bsdserver.nl/realms/test-mcp/protocol/openid-connect/certs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">resourceMetadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">resource</span><span class="p">:</span><span class="w"> </span><span class="l">https://mcp-dev.bsdserver.nl/grafana/mcp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">scopesSupported</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">openid, mcp]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">bearerMethodsSupported</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">header]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mcpAuthorization</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;jwt.azp == &#34;mcp-cli&#34; &amp;&amp; jwt.preferred_username == &#34;mcp-tester&#34; &amp;&amp; mcp.tool.name in [&#34;list_datasources&#34;, &#34;search_dashboards&#34;]&#39;</span><span class="w">
</span></span></span></code></pre></div><h3 id="the-wrong-turn-the-keycloak-adapter">The wrong turn: the Keycloak adapter <a class="anchor" data-pagefind-ignore href="#the-wrong-turn-the-keycloak-adapter" aria-label="Link to this section">#</a></h3>
<p>agentgateway has a <code>provider: keycloak</code> option for this policy. The documentation presents it as the way to make Keycloak work with MCP clients, so the first version used it. Discovery looked fine, and Codex started the sign-in. Then it stopped:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Authorization server issuer mismatch:
</span></span><span class="line"><span class="cl">expected https://mcp-dev.bsdserver.nl/grafana/mcp,
</span></span><span class="line"><span class="cl">received https://keycloak.bsdserver.nl/realms/test-mcp
</span></span></code></pre></div><p>The adapter serves its own copy of the authorization server metadata, with the gateway&rsquo;s URL as <code>issuer</code>. Keycloak, meanwhile, puts its real issuer in the <code>iss</code> parameter of the authorization response (RFC 9207). That parameter exists to stop mix-up attacks, and Codex checks it strictly. The two values disagree, so Codex refuses.</p>
<p>My first fix was to make Keycloak stop sending <code>iss</code>. That needed a third change to the Terraform module, and it made things worse:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Authorization server response missing required issuer:
</span></span><span class="line"><span class="cl">expected https://mcp-dev.bsdserver.nl/grafana/mcp
</span></span></code></pre></div><p>Keycloak&rsquo;s metadata says it supports <code>iss</code> in responses. A strict client then requires the parameter. Removing it can&rsquo;t work, and it would have weakened a security check to work around a proxy.</p>
<p>The real fix was to remove the adapter. Keycloak 26.7 serves RFC 8414 metadata itself, at both well-known locations, so the gateway doesn&rsquo;t need to imitate it. Without the adapter, the protected-resource metadata names Keycloak directly. The issuer the client expects, the issuer in the metadata and the <code>iss</code> in the response are then all the same. We reverted the Keycloak change, and the module setting became tri-state so that &ldquo;not managed&rdquo; is the default again.</p>
<p>There is a cost. The adapter also answered dynamic client registration with our pre-registered client. Without it, clients have to be told which client ID to use. That&rsquo;s one line of configuration:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">claude mcp add --transport http --client-id mcp-cli grafana-dev https://mcp-dev.bsdserver.nl/grafana/mcp
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="c"># ~/.codex/config.toml</span>
</span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">mcp_servers</span><span class="p">.</span><span class="nx">grafana-dev</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">url</span> <span class="p">=</span> <span class="s2">&#34;https://mcp-dev.bsdserver.nl/grafana/mcp&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">mcp_servers</span><span class="p">.</span><span class="nx">grafana-dev</span><span class="p">.</span><span class="nx">oauth</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">client_id</span> <span class="p">=</span> <span class="s2">&#34;mcp-cli&#34;</span>
</span></span></code></pre></div><p>Neither file holds a secret. <code>mcp-cli</code> is a public client, and the tokens live in each tool&rsquo;s own credential store.</p>
<h3 id="what-we-saw">What we saw <a class="anchor" data-pagefind-ignore href="#what-we-saw" aria-label="Link to this section">#</a></h3>
<p><strong>Claude Code</strong> signed in through the browser as <code>mcp-tester</code>. The gateway log shows <code>tools/list</code> returning exactly <code>list_datasources</code> and <code>search_dashboards</code>, out of mcp-grafana&rsquo;s full tool set. Both tool calls were authorized and forwarded to the backend over mTLS.</p>
<p>Both calls then failed with a 500, which we expected. In this proof of concept mcp-grafana has no network path to Grafana and no credentials for it. The gateway&rsquo;s part worked. The backend&rsquo;s part comes in phase 2.</p>
<p><strong>Codex</strong> completed the sign-in and stored its token once the adapter was gone. We haven&rsquo;t yet run a tool call through Codex. That is the one piece of this proof of concept still to be shown.</p>
<h2 id="shortcuts-to-remove">Shortcuts to remove <a class="anchor" data-pagefind-ignore href="#shortcuts-to-remove" aria-label="Link to this section">#</a></h2>
<p>A proof of concept takes shortcuts, and these have to go before real MCP servers move in:</p>
<ul>
<li><strong>Session encryption is off.</strong> agentgateway encrypts MCP session state with a key. That key has to come from Vault, so we turned the feature off for now.</li>
<li><strong>Backends are stateless,</strong> which follows from the previous point.</li>
<li><strong>mcp-grafana has no egress and no credentials,</strong> as described above.</li>
</ul>
<h2 id="verdict">Verdict <a class="anchor" data-pagefind-ignore href="#verdict" aria-label="Link to this section">#</a></h2>
<p>agentgateway stays. It validates tokens from both issuers, it filters tool lists and tool calls per caller, and it speaks SPIFFE mTLS to the backends with certificates it gets from SPIRE. The two points that need care are that one route can only have one issuer, and that the Keycloak adapter is better left out.</p>
<p>The next part moves real MCP servers in, each with its own Vault role for its credentials.</p>
]]></content:encoded></item><item><title>SPIRE on Talos, and a Vault Login Without a Secret</title><link>https://blog.bsdserver.nl/spire-on-talos-and-a-vault-login-without-a-secret/</link><guid isPermaLink="true">https://blog.bsdserver.nl/spire-on-talos-and-a-vault-login-without-a-secret/</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Thu, 19 Nov 2026 08:00:00 +0100</pubDate><description>We installed SPIRE on a Talos cluster with Flux and let pods log in to Vault Community with a JWT-SVID. No Vault token, no secret in the pod.</description><category>SPIFFE</category><category>Kubernetes</category><category>Security</category><category>MCP</category><category>Homelab</category><media:content url="https://blog.bsdserver.nl/content/images/2026/11/spire-talos-vault-login.png" medium="image"/><content:encoded><![CDATA[<p>The design in part 1 rests on one assumption: a pod can prove who it is without anyone handing it a secret first. That&rsquo;s what SPIRE does. This part covers the first real step of the build. We installed SPIRE on our non-production Talos cluster, then let a pod log in to Vault with nothing but the identity SPIRE gave it.</p>
<p>Both work. Getting there took two small detours, which are below.</p>
<h2 id="what-we-installed">What we installed <a class="anchor" data-pagefind-ignore href="#what-we-installed" aria-label="Link to this section">#</a></h2>
<p>SPIRE comes as a Helm chart from the SPIFFE project. We pinned chart 0.31.0, which ships SPIRE 1.16.0, plus the separate <code>spire-crds</code> chart. Flux installs both from our GitOps repository, like every other controller on the cluster.</p>
<p>The values that matter:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">global</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">spire</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">trustDomain</span><span class="p">:</span><span class="w"> </span><span class="l">nonprod.bsdserver.lan</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">jwtIssuer</span><span class="p">:</span><span class="w"> </span><span class="l">https://spire-oidc-nonprod.bsdserver.nl</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">recommendations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spire-server</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">dataStore</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">sql</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">databaseType</span><span class="p">:</span><span class="w"> </span><span class="l">sqlite3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">caTTL</span><span class="p">:</span><span class="w"> </span><span class="l">24h</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">defaultX509SvidTTL</span><span class="p">:</span><span class="w"> </span><span class="l">1h</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">defaultJwtSvidTTL</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">controllerManager</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">identities</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">clusterSPIFFEIDs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">default</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">namespaceSelector</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">spiffe.io/enabled</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;true&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spire-agent</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">kubeletAddress</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l">hostip</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spiffe-oidc-discovery-provider</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span></code></pre></div><p>A few choices in there are worth explaining.</p>
<p><strong>The version is exact.</strong> We normally let Flux follow a minor range. This chart&rsquo;s minor releases have changed value names before, so it gets a fixed version and an upgrade is a deliberate change.</p>
<p><strong>Identities are opt-in.</strong> The chart&rsquo;s default <code>ClusterSPIFFEID</code> gives every pod an identity of the form <code>spiffe://&lt;trust domain&gt;/ns/&lt;namespace&gt;/sa/&lt;service account&gt;</code>. We limited it to namespaces with the label <code>spiffe.io/enabled=true</code>. A pod in any other namespace gets nothing from SPIRE, even though the agent runs on every node.</p>
<p><strong>Short lifetimes.</strong> X.509 certificates last an hour and JWTs five minutes. The agent renews them on its own, so short lifetimes cost nothing.</p>
<p><strong>SQLite is fine here.</strong> This cluster has a single node. The production SPIRE server will use PostgreSQL. That&rsquo;s for a later part.</p>
<p><strong>The agent doesn&rsquo;t need host networking.</strong> It has to reach the kubelet to check which pod is asking for an identity. With <code>kubeletAddress.mode: hostip</code> it uses the node&rsquo;s IP from the downward API instead of <code>127.0.0.1</code>, so the agent pod stays off the host network.</p>
<h3 id="pod-security">Pod security <a class="anchor" data-pagefind-ignore href="#pod-security" aria-label="Link to this section">#</a></h3>
<p>Talos enforces Pod Security Admission. The SPIRE agent and the CSI driver need host paths, so their namespace, <code>spire-system</code>, runs at the <code>privileged</code> level. The server runs in its own namespace, <code>spire-server</code>, at <code>restricted</code>. We create both namespaces ourselves, with those labels, and tell the chart not to create them.</p>
<h3 id="how-a-pod-gets-its-identity">How a pod gets its identity <a class="anchor" data-pagefind-ignore href="#how-a-pod-gets-its-identity" aria-label="Link to this section">#</a></h3>
<p>Workloads don&rsquo;t talk to the agent over a host path. The chart includes the SPIFFE CSI driver, which mounts the agent&rsquo;s Workload API socket into the pod as a read-only volume:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">spiffe-workload-api</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">csi</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">driver</span><span class="p">:</span><span class="w"> </span><span class="l">csi.spiffe.io</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span></code></pre></div><p>The pod needs no privileges for this. It asks the socket for a certificate or a JWT. The agent looks up the calling process, checks with the kubelet which pod and service account it belongs to, and answers only if a registration matches.</p>
<h2 id="the-oidc-discovery-endpoint">The OIDC discovery endpoint <a class="anchor" data-pagefind-ignore href="#the-oidc-discovery-endpoint" aria-label="Link to this section">#</a></h2>
<p>Anything that verifies a JWT-SVID needs SPIRE&rsquo;s public keys. The chart&rsquo;s OIDC discovery provider publishes them as a standard <code>/.well-known/openid-configuration</code> and a JWKS. That makes SPIRE look like any other OpenID Connect issuer, which is exactly what Vault&rsquo;s JWT auth method expects.</p>
<p>We exposed it on <code>spire-oidc-nonprod.bsdserver.nl</code> through our ingress, with a Let&rsquo;s Encrypt certificate and an allow-list for the LAN, the cluster networks and our NetBird mesh. The public certificate means Vault can fetch the keys without a custom CA bundle. The endpoint only serves public keys, but it still isn&rsquo;t reachable from the internet.</p>
<p>One thing caught us later. Pods inside the cluster can&rsquo;t use that hostname, because the ingress allow-list doesn&rsquo;t include pod addresses. Anything in-cluster that needs the keys reads them from the provider&rsquo;s Service directly. That comes up again in the next part.</p>
<h2 id="vault-login-with-a-jwt-svid">Vault login with a JWT-SVID <a class="anchor" data-pagefind-ignore href="#vault-login-with-a-jwt-svid" aria-label="Link to this section">#</a></h2>
<p>Vault Enterprise has a native SPIFFE auth method. We run Vault Community, which doesn&rsquo;t. It does have the JWT auth method, and a JWT-SVID is a normal signed JWT. So the plan was a JWT auth mount that trusts SPIRE&rsquo;s discovery endpoint.</p>
<p>Our Vault is configured with Terraform through a shared module. That module already managed one JWT mount, for GitLab CI, but it couldn&rsquo;t create a second one. So the first change was to the module: a new <code>jwt_auth_backends</code> input, released as v2.6.0, with tests for the new input.</p>
<p>The unit that uses it is short:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="n">inputs</span> <span class="o">=</span> {
</span></span><span class="line"><span class="cl"><span class="n">  jwt_auth_backends</span> <span class="o">=</span> {
</span></span><span class="line"><span class="cl"><span class="n">    spiffe</span> <span class="o">=</span> {
</span></span><span class="line"><span class="cl"><span class="n">      description</span>        <span class="o">=</span> <span class="s2">&#34;SPIFFE JWT-SVID login, trust domain nonprod.bsdserver.lan&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      oidc_discovery_url</span> <span class="o">=</span> <span class="s2">&#34;https://spire-oidc-nonprod.bsdserver.nl&#34;</span>
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">  jwt_roles</span> <span class="o">=</span> {
</span></span><span class="line"><span class="cl"><span class="n">    spiffe-smoke</span> <span class="o">=</span> {
</span></span><span class="line"><span class="cl"><span class="n">      backend</span>         <span class="o">=</span> <span class="s2">&#34;spiffe&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      bound_audiences</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;vault&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">      bound_claims</span> <span class="o">=</span> {
</span></span><span class="line"><span class="cl"><span class="n">        sub</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;spiffe://nonprod.bsdserver.lan/ns/mcp-dev/sa/spiffe-smoke&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">      }
</span></span><span class="line"><span class="cl"><span class="n">      user_claim</span>     <span class="o">=</span> <span class="s2">&#34;sub&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      token_policies</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;default&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">      token_ttl</span>      <span class="o">=</span> <span class="m">60</span>
</span></span><span class="line"><span class="cl"><span class="n">      token_max_ttl</span>  <span class="o">=</span> <span class="m">60</span>
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><p>Each role is bound to exactly one SPIFFE ID and to the audience <code>vault</code>. A workload that wants to log in asks SPIRE for a JWT-SVID with that audience and hands it to Vault. Vault checks the signature against SPIRE&rsquo;s keys, the issuer, the audience and the <code>sub</code> claim. If all of it matches, the workload gets a Vault token with that role&rsquo;s policy.</p>
<p><strong>One mount per trust domain.</strong> Vault&rsquo;s JWT method trusts one issuer per mount. This mount belongs to the non-production trust domain. Production will get its own mount rather than a second issuer here, so a non-production identity can never be mapped to a production policy by accident.</p>
<h3 id="the-first-detour-one-grant-was-too-narrow">The first detour: one grant was too narrow <a class="anchor" data-pagefind-ignore href="#the-first-detour-one-grant-was-too-narrow" aria-label="Link to this section">#</a></h3>
<p>Our Terraform runs with an AppRole whose policy lists exactly what it may touch. We granted it the new mount&rsquo;s paths, including <code>sys/mounts/auth/spiffe</code>. The first apply created the mount and then failed with a 403.</p>
<p>The Vault provider reads the mount&rsquo;s tuning at <code>sys/mounts/auth/spiffe/tune</code>, one level deeper than the grant. Nothing used the mount yet, so recreating it was harmless. The fix was a grant that covers the subpath. It&rsquo;s the kind of failure you only find by applying.</p>
<h2 id="testing-it">Testing it <a class="anchor" data-pagefind-ignore href="#testing-it" aria-label="Link to this section">#</a></h2>
<p>We tested on a throwaway Vault in dev mode first, so we could try things that should fail. With a real JWT-SVID from a test pod:</p>
<table>
	<thead>
			<tr>
					<th>Attempt</th>
					<th>Result</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Correct role, audience <code>vault</code></td>
					<td>Logged in, <code>default</code> policy</td>
			</tr>
			<tr>
					<td>JWT-SVID with a different audience</td>
					<td>Refused</td>
			</tr>
			<tr>
					<td>The role of another workload</td>
					<td>Refused: <code>sub</code> doesn&rsquo;t match the bound claim</td>
			</tr>
			<tr>
					<td>Payload changed to another SPIFFE ID, signature kept</td>
					<td>Refused: signature invalid</td>
			</tr>
	</tbody>
</table>
<p>Then on the production Vault, from a pod running under the <code>spiffe-smoke</code> service account:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="nv">T</span><span class="o">=</span><span class="k">$(</span>spire-agent api fetch jwt -audience vault <span class="se">\
</span></span></span><span class="line"><span class="cl">      -socketPath /spiffe-workload-api/spire-agent.sock <span class="p">|</span> awk <span class="s1">&#39;/token\(/{getline; print $1; exit}&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">vault write auth/spiffe/login <span class="nv">role</span><span class="o">=</span>spiffe-smoke <span class="nv">jwt</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$T</span><span class="s2">&#34;</span>
</span></span></code></pre></div><p>The login returned a token with only the <code>default</code> policy, valid for 60 seconds. A read under <code>secret/</code> with that token returned 403, as intended. The pod never held a Vault credential. Its identity came from the platform, and the token it got from Vault expires a minute later.</p>
<h2 id="what-this-doesnt-solve-yet">What this doesn&rsquo;t solve yet <a class="anchor" data-pagefind-ignore href="#what-this-doesnt-solve-yet" aria-label="Link to this section">#</a></h2>
<p>SPIRE uses its own CA on this cluster. In production we want its signing key to come from Vault&rsquo;s PKI engine. SPIRE then needs one credential to talk to Vault. So the number of secrets goes from many to one, not to zero. We&rsquo;d rather be honest about that than pretend otherwise.</p>
<p>The other open part is real policy. <code>spiffe-smoke</code> exists only to prove that the login path works. The MCP servers get their own roles in phase 2, each bound to one service account and allowed to read only its own credentials.</p>
<h2 id="next">Next <a class="anchor" data-pagefind-ignore href="#next" aria-label="Link to this section">#</a></h2>
<p>With identities in place, the next part puts a gateway in front of an MCP server. Workloads use these JWT-SVIDs, people sign in with Keycloak, and every MCP server accepts only the gateway. It&rsquo;s also where we got an OAuth detail wrong before getting it right.</p>
]]></content:encoded></item><item><title>Running a SPIRE Agent on macOS by Building It Yourself</title><link>https://blog.bsdserver.nl/running-a-spire-agent-on-macos/</link><guid isPermaLink="true">https://blog.bsdserver.nl/running-a-spire-agent-on-macos/</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Wed, 18 Nov 2026 16:05:00 +0100</pubDate><description>SPIRE ships no macOS build. We compiled spire-agent for Apple Silicon and got a working X.509-SVID in a local end-to-end test.</description><category>SPIFFE</category><category>Security</category><category>MCP</category><category>mTLS</category><media:content url="https://blog.bsdserver.nl/content/images/2026/11/spire-agent-macos.png" medium="image"/><content:encoded><![CDATA[<p>SPIRE 1.16.0 comes with release builds for Linux and Windows. There is no macOS download. For a design where a Mac may need a SPIFFE identity, that looked like a dead end until we tried the obvious thing: SPIRE is written in Go, so build it.</p>
<p>It works. Here&rsquo;s the build, what makes it work on macOS, and a local end-to-end test that issues a real certificate.</p>
<h2 id="build-the-agent">Build the agent <a class="anchor" data-pagefind-ignore href="#build-the-agent" aria-label="Link to this section">#</a></h2>
<p>You need Go and the SPIRE source at a release tag. We used Go 1.27 on Apple Silicon.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">git clone --depth <span class="m">1</span> --branch v1.16.0 https://github.com/spiffe/spire.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> spire
</span></span><span class="line"><span class="cl"><span class="nv">CGO_ENABLED</span><span class="o">=</span><span class="m">0</span> go build -o out/spire-agent ./cmd/spire-agent
</span></span><span class="line"><span class="cl">file out/spire-agent
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">out/spire-agent: Mach-O 64-bit executable arm64
</span></span></code></pre></div><p>The agent needs no cgo. The binary is large (about 110 MB) because it carries every plugin.</p>
<h2 id="why-it-actually-works-on-macos">Why it actually works on macOS <a class="anchor" data-pagefind-ignore href="#why-it-actually-works-on-macos" aria-label="Link to this section">#</a></h2>
<p>A binary that compiles isn&rsquo;t the same as one that works. The part that matters is the Workload API: when a process asks the agent for an identity, the agent has to find out which process is asking. On Linux it reads the peer credentials of the Unix socket.</p>
<p>SPIRE has a separate implementation for the BSD family, and macOS is part of it. The peer tracker files carry this build constraint:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="cp">//go:build darwin || freebsd || netbsd || openbsd</span><span class="w">
</span></span></span></code></pre></div><p>On macOS the agent asks the socket for the caller&rsquo;s PID (<code>LOCAL_PEERPID</code>) and uses kqueue to notice when that process exits, so a PID can&rsquo;t be reused to steal an identity. The fallback that returns &ldquo;unsupported platform&rdquo; is only built on other systems.</p>
<p>The <code>unix</code> workload attestor then turns the caller into selectors: <code>uid</code>, <code>user</code>, <code>gid</code>, <code>group</code>, and with <code>discover_workload_path = true</code> also the binary&rsquo;s <code>path</code> and <code>sha256</code>. That last pair lets you tie an identity to one specific executable, not just to a user.</p>
<h2 id="a-local-end-to-end-test">A local end-to-end test <a class="anchor" data-pagefind-ignore href="#a-local-end-to-end-test" aria-label="Link to this section">#</a></h2>
<p>To prove it, we ran a throwaway SPIRE server and the agent on the same Mac. The server isn&rsquo;t the point here, so we built it locally too. Its SQLite datastore needs cgo, which the agent doesn&rsquo;t:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="nv">CGO_ENABLED</span><span class="o">=</span><span class="m">1</span> go build -o out/spire-server ./cmd/spire-server
</span></span></code></pre></div><p>Without cgo the server starts and immediately stops with:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">datastore-sql: sqlite3 is not a supported dialect when CGO is not enabled
</span></span></code></pre></div><p>Keep the sockets on a short path. macOS limits Unix socket paths to about 104 characters, and a deep working directory breaks the agent with a confusing <code>connect: invalid argument</code>. We put the sockets in a fresh directory under <code>/tmp</code>.</p>
<p><code>server.conf</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">server</span> {
</span></span><span class="line"><span class="cl"><span class="n">  bind_address</span> <span class="o">=</span> <span class="s2">&#34;127.0.0.1&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  bind_port</span>    <span class="o">=</span> <span class="s2">&#34;18081&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  socket_path</span>  <span class="o">=</span> <span class="s2">&#34;/tmp/spire-demo/api.sock&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  trust_domain</span> <span class="o">=</span> <span class="s2">&#34;poc.test&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  data_dir</span>     <span class="o">=</span> <span class="s2">&#34;./server-data&#34;</span>
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">plugins</span> {
</span></span><span class="line"><span class="cl">  <span class="k">DataStore</span> <span class="s2">&#34;sql&#34;</span> {
</span></span><span class="line"><span class="cl">    <span class="k">plugin_data</span> {
</span></span><span class="line"><span class="cl"><span class="n">      database_type</span>     <span class="o">=</span> <span class="s2">&#34;sqlite3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      connection_string</span> <span class="o">=</span> <span class="s2">&#34;./server-data/datastore.sqlite3&#34;</span>
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">  <span class="k">KeyManager</span> <span class="s2">&#34;memory&#34;</span> { <span class="k">plugin_data</span> {} }
</span></span><span class="line"><span class="cl">  <span class="k">NodeAttestor</span> <span class="s2">&#34;join_token&#34;</span> { <span class="k">plugin_data</span> {} }
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><p>Start the server, create a join token for the Mac and register an identity for your own user ID:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">mkdir -p /tmp/spire-demo
</span></span><span class="line"><span class="cl">./out/spire-server run -config server.conf <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">./out/spire-server token generate <span class="se">\
</span></span></span><span class="line"><span class="cl">  -socketPath /tmp/spire-demo/api.sock <span class="se">\
</span></span></span><span class="line"><span class="cl">  -spiffeID spiffe://poc.test/node/mac
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">./out/spire-server entry create <span class="se">\
</span></span></span><span class="line"><span class="cl">  -socketPath /tmp/spire-demo/api.sock <span class="se">\
</span></span></span><span class="line"><span class="cl">  -parentID spiffe://poc.test/node/mac <span class="se">\
</span></span></span><span class="line"><span class="cl">  -spiffeID spiffe://poc.test/workstation/<span class="k">$(</span>id -un<span class="k">)</span>/claude-code <span class="se">\
</span></span></span><span class="line"><span class="cl">  -selector unix:uid:<span class="k">$(</span>id -u<span class="k">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">./out/spire-server bundle show <span class="se">\
</span></span></span><span class="line"><span class="cl">  -socketPath /tmp/spire-demo/api.sock &gt; bundle.pem
</span></span></code></pre></div><p><code>agent.conf</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">agent</span> {
</span></span><span class="line"><span class="cl"><span class="n">  data_dir</span>          <span class="o">=</span> <span class="s2">&#34;./agent-data&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  server_address</span>    <span class="o">=</span> <span class="s2">&#34;127.0.0.1&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  server_port</span>       <span class="o">=</span> <span class="s2">&#34;18081&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  socket_path</span>       <span class="o">=</span> <span class="s2">&#34;/tmp/spire-demo/agent.sock&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  trust_bundle_path</span> <span class="o">=</span> <span class="s2">&#34;bundle.pem&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  trust_domain</span>      <span class="o">=</span> <span class="s2">&#34;poc.test&#34;</span>
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">plugins</span> {
</span></span><span class="line"><span class="cl">  <span class="k">NodeAttestor</span> <span class="s2">&#34;join_token&#34;</span> { <span class="k">plugin_data</span> {} }
</span></span><span class="line"><span class="cl">  <span class="k">KeyManager</span> <span class="s2">&#34;memory&#34;</span> { <span class="k">plugin_data</span> {} }
</span></span><span class="line"><span class="cl">  <span class="k">WorkloadAttestor</span> <span class="s2">&#34;unix&#34;</span> {
</span></span><span class="line"><span class="cl"><span class="n">    plugin_data { discover_workload_path</span> <span class="o">=</span> <span class="kt">true</span> }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><p>Start the agent with the token, then ask it for an identity as your normal user:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">./out/spire-agent run -config agent.conf -joinToken &lt;token&gt; <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">./out/spire-agent api fetch x509 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -socketPath /tmp/spire-demo/agent.sock -write .
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Received 1 svid after 312.761333ms
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">SPIFFE ID:          spiffe://poc.test/workstation/pderuiter/claude-code
</span></span><span class="line"><span class="cl">SVID Valid After:   2026-11-18 15:21:39 +0000 UTC
</span></span><span class="line"><span class="cl">SVID Valid Until:   2026-11-18 16:21:49 +0000 UTC
</span></span></code></pre></div><p>The certificate carries the SPIFFE ID as a URI SAN and is valid for one hour. The agent rotates it before it expires.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">openssl x509 -in svid.0.pem -noout -ext subjectAltName
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">X509v3 Subject Alternative Name:
</span></span><span class="line"><span class="cl">    URI:spiffe://poc.test/workstation/pderuiter/claude-code
</span></span></code></pre></div><p>If your first fetch logs <code>No identity issued</code>, give it a few seconds. The agent syncs registration entries from the server on an interval, and a fetch that arrives before the first sync finds nothing.</p>
<h2 id="what-this-doesnt-give-you-yet">What this doesn&rsquo;t give you yet <a class="anchor" data-pagefind-ignore href="#what-this-doesnt-give-you-yet" aria-label="Link to this section">#</a></h2>
<p>This test proves the agent runs on macOS. It isn&rsquo;t a production setup:</p>
<ul>
<li><strong>Node attestation.</strong> A join token is single-use. If the agent loses its state, it needs a new token. A device certificate (the <code>x509pop</code> attestor) lets the Mac attest again by itself, at the cost of a key in a file. There&rsquo;s no attestor for the Secure Enclave.</li>
<li><strong>The identity is &ldquo;processes of this user on this Mac&rdquo;.</strong> Add the <code>path</code> or <code>sha256</code> selectors if you want to narrow it to one binary, and accept that every update changes the hash.</li>
<li><strong>Run it as a service.</strong> In practice that means a launchd agent and a socket path every client agrees on.</li>
<li><strong>Build it in CI and sign it.</strong> You&rsquo;re running a security component you compiled yourself, so treat it like any other artifact you ship.</li>
</ul>
<h2 id="do-you-even-need-it">Do you even need it? <a class="anchor" data-pagefind-ignore href="#do-you-even-need-it" aria-label="Link to this section">#</a></h2>
<p>In our MCP design, mostly not. The interactive AI tools on a Mac can sign in to the gateway with Keycloak OAuth, which works on every operating system. The macOS agent becomes useful when an unattended workload runs on the Mac and needs its own identity, the same way our Kubernetes and Linux workloads get theirs.</p>
<p>It&rsquo;s good to know the option is a <code>go build</code> away and not a missing platform.</p>
]]></content:encoded></item><item><title>Designing a Self-Hosted MCP Gateway for Every AI Client</title><link>https://blog.bsdserver.nl/designing-a-self-hosted-mcp-gateway-for-every-ai-client/</link><guid isPermaLink="true">https://blog.bsdserver.nl/designing-a-self-hosted-mcp-gateway-for-every-ai-client/</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Wed, 18 Nov 2026 16:00:00 +0100</pubDate><description>Our MCP servers move to Kubernetes behind one gateway. Keycloak OAuth for people, SPIFFE for workloads, and no public endpoint.</description><category>MCP</category><category>SPIFFE</category><category>Security</category><category>Kubernetes</category><category>Homelab</category><media:content url="https://blog.bsdserver.nl/content/images/2026/11/mcp-gateway-every-ai-client.png" medium="image"/><content:encoded><![CDATA[<p>When I counted, Claude Code on my Mac had 25 MCP servers configured. Some ran as local processes started with <code>npx</code> or <code>uvx</code>. Two ran as local Docker containers, and both failed because Docker wasn&rsquo;t on the PATH. Six ran as containers on one of our Docker hosts behind Traefik. Of those six, two were protected by an IP allow-list, one of those also required a bearer token, and four had no authentication at all. The tokens for the local ones sat in a JSON file in my home directory.</p>
<p>That works for one person with one tool. It falls apart the moment a second tool wants the same servers. This post is the design for fixing that, before any of it is built. The rest of the series follows the build.</p>
<h2 id="the-requirement-that-changed-the-design">The requirement that changed the design <a class="anchor" data-pagefind-ignore href="#the-requirement-that-changed-the-design" aria-label="Link to this section">#</a></h2>
<p>My first draft was built around Claude Code on a Mac. That&rsquo;s the wrong starting point, and it took a few rounds to see it. The MCP servers have to work for every AI client we run:</p>
<ul>
<li>Claude Code and Codex, on macOS, Linux and Windows</li>
<li>Hermes Agent</li>
<li>our self-hosted AI stack: open-webui, its pipelines, and n8n</li>
<li>Home Assistant</li>
</ul>
<p>Once you list them, the question stops being &ldquo;how does Claude Code authenticate&rdquo; and becomes &ldquo;what can each of these clients actually do&rdquo;.</p>
<h2 id="what-the-clients-support">What the clients support <a class="anchor" data-pagefind-ignore href="#what-the-clients-support" aria-label="Link to this section">#</a></h2>
<p>I checked each client&rsquo;s documentation instead of assuming.</p>
<table>
	<thead>
			<tr>
					<th>Client</th>
					<th>Remote MCP auth options</th>
					<th>Notes</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Claude Code</td>
					<td>OAuth, static headers, <code>headersHelper</code> (a command that prints fresh headers)</td>
					<td>Client certificates are documented only for the OAuth sign-in requests</td>
			</tr>
			<tr>
					<td>Codex</td>
					<td>OAuth via <code>codex mcp login</code>, bearer token from an environment variable</td>
					<td>Streamable HTTP only, no SSE</td>
			</tr>
			<tr>
					<td>Hermes Agent</td>
					<td>Static headers, <code>auth: oauth</code> (OAuth 2.1 with PKCE)</td>
					<td></td>
			</tr>
			<tr>
					<td>Home Assistant</td>
					<td>OAuth with application credentials</td>
					<td>Its client docs mention SSE</td>
			</tr>
			<tr>
					<td>open-webui, n8n</td>
					<td>Whatever we configure</td>
					<td>They run as workloads we control</td>
			</tr>
	</tbody>
</table>
<p>Two things stand out. OAuth with a short-lived bearer token is the only method every interactive client supports, which matches what the MCP specification itself defines. And mutual TLS from the client is not something you can rely on.</p>
<h2 id="phones-and-tablets-left-out-on-purpose">Phones and tablets: left out on purpose <a class="anchor" data-pagefind-ignore href="#phones-and-tablets-left-out-on-purpose" aria-label="Link to this section">#</a></h2>
<p>Claude&rsquo;s mobile apps, claude.ai and the desktop app&rsquo;s connectors don&rsquo;t call your MCP server from your device. Anthropic&rsquo;s documentation says the call comes from Anthropic&rsquo;s cloud, and servers behind a VPN or a firewall won&rsquo;t connect. Supporting them would mean a public endpoint.</p>
<p>We decided against that. I can drive my desktop Claude session from my phone, and that desktop session reaches the internal gateway. The exposure footprint stays at zero public endpoints, which is worth more than native mobile support.</p>
<h2 id="the-design">The design <a class="anchor" data-pagefind-ignore href="#the-design" aria-label="Link to this section">#</a></h2>
<p>One internal gateway is the only way in. It listens on port 443 on the LAN and our NetBird mesh, and nowhere else.</p>
<p><strong>People</strong> sign in with Keycloak through OAuth 2.1 (authorization code with PKCE). The client handles the browser flow, and Keycloak issues short-lived tokens. Devices without a browser can use the device authorization grant.</p>
<p><strong>Workloads</strong> such as open-webui, n8n and agents running on servers get a SPIFFE identity from SPIRE. They present a JWT-SVID, or use mTLS with an X.509-SVID.</p>
<p>The gateway accepts tokens from exactly two issuers, Keycloak and SPIRE. Both are signed JWTs, so the gateway validates them the same way. It then decides per caller and per tool what is allowed:</p>
<ul>
<li>the Vault and Kubernetes MCP servers: only me and named workloads</li>
<li>Home Assistant: only the tools it needs</li>
<li>write tools: people only, unless a workload is explicitly granted them</li>
</ul>
<p>Behind the gateway, every MCP server pod accepts only the gateway&rsquo;s SPIFFE identity over mTLS. Even inside the cluster, nothing else can call them.</p>
<h2 id="what-spiffe-solves-and-what-it-doesnt">What SPIFFE solves, and what it doesn&rsquo;t <a class="anchor" data-pagefind-ignore href="#what-spiffe-solves-and-what-it-doesnt" aria-label="Link to this section">#</a></h2>
<p>SPIFFE was my first idea for the whole problem, so it&rsquo;s worth being precise about where it fits.</p>
<p>It fits the workloads well. A pod gets an identity from the platform itself, with no secret to distribute. The MCP servers use that identity to log in to Vault (through Vault&rsquo;s standard JWT auth method, because native SPIFFE auth is an Enterprise feature) and receive short-lived tokens with a narrow policy.</p>
<p>It fits people badly. SPIFFE identifies workloads, and the AI clients mostly can&rsquo;t present a client certificate anyway. Keycloak already knows who the people are.</p>
<p>And it can&rsquo;t remove credentials that the upstream API insists on:</p>
<table>
	<thead>
			<tr>
					<th>Upstream</th>
					<th>Best we can do</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Kubernetes</td>
					<td>Vault&rsquo;s Kubernetes secrets engine mints a service-account token per request, valid for minutes. No static kubeconfig.</td>
			</tr>
			<tr>
					<td>Vault</td>
					<td>No standing token at all</td>
			</tr>
			<tr>
					<td>GitLab, Grafana</td>
					<td>Access tokens with an expiry, rotated by Terraform</td>
			</tr>
			<tr>
					<td>n8n, listmonk</td>
					<td>Static keys in Vault, fetched at runtime by an attested pod, rotated on a schedule</td>
			</tr>
	</tbody>
</table>
<p>So SPIFFE turns &ldquo;a token in every config file&rdquo; into &ldquo;a few upstream keys in Vault, only readable by the pod that needs them&rdquo;. That&rsquo;s a large reduction, not zero.</p>
<h2 id="why-a-gateway-instead-of-auth-in-every-server">Why a gateway instead of auth in every server <a class="anchor" data-pagefind-ignore href="#why-a-gateway-instead-of-auth-in-every-server" aria-label="Link to this section">#</a></h2>
<p>We run more than a dozen MCP servers from many different authors. Some support bearer tokens, some support nothing, none support SPIFFE. Patching each one would mean maintaining a dozen forks.</p>
<p>A gateway moves authentication, authorization, audit logging and rate limiting into one place. The MCP servers stay stock, and adding a server means adding a route and a policy. The cost is a single point of failure for all AI tooling, so the production gateway will run with two replicas.</p>
<p>Our first candidate is <a href="https://agentgateway.dev/" rel="noopener">agentgateway</a>, an MCP-aware proxy that validates JWTs and can apply rules to claims. A proof of concept has to confirm three things before we commit:</p>
<ul>
<li>it accepts tokens from both issuers</li>
<li>it talks mTLS with SPIFFE certificates to the backends</li>
<li>Keycloak&rsquo;s metadata satisfies the MCP clients, which the agentgateway docs themselves warn about</li>
</ul>
<p>If it falls short, Envoy with an external authorization service is the fallback.</p>
<h2 id="what-comes-next">What comes next <a class="anchor" data-pagefind-ignore href="#what-comes-next" aria-label="Link to this section">#</a></h2>
<p>The build runs in phases, and each one gets its own post:</p>
<ol>
<li>SPIRE on our non-production Talos cluster, plus the Vault login for SPIFFE identities.</li>
<li>The gateway proof of concept with Keycloak, Claude Code and Codex.</li>
<li>Six MCP servers moved off my Mac into their own namespace.</li>
<li>Promotion to production, and moving the MCP servers that still run on Docker.</li>
</ol>
<p>One piece is already done. SPIRE ships no macOS build, so I built and tested the agent myself. That&rsquo;s the next part.</p>
]]></content:encoded></item></channel></rss>