<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>HAProxy in production: load balancing, security and automation | A DevOps Blog</title><link>https://blog.bsdserver.nl/series/haproxy/</link><description>Hands-on DevOps and homelab content: Kubernetes clusters, self-hosted services, CI/CD pipelines and infrastructure as code, by Patrick de Ruiter of We Build Your Cloud.</description><language>en</language><image><url>https://blog.bsdserver.nl/images/app-icon.svg</url><title>A DevOps Blog</title><link>https://blog.bsdserver.nl/</link></image><lastBuildDate>Wed, 18 Nov 2026 09:00:00 +0100</lastBuildDate><atom:link href="https://blog.bsdserver.nl/series/haproxy/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title>HAProxy TCP Load Balancing: A Complete Guide to Layer 4 Load Balancing</title><link>https://blog.bsdserver.nl/haproxy-tcp-load-balancing-a-complete-guide-to-layer-4-load-balancing/</link><guid isPermaLink="false">693b721e79eb54000160a5dc</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:38:38 +0000</pubDate><description>HAProxy is one of the most powerful and reliable load balancers available.</description><category>HAProxy</category><category>Load balancing</category><category>TCP</category><category>High availability</category><category>Networking</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-tcp-loadbalancing.png" medium="image"/><content:encoded><![CDATA[<p>HAProxy is one of the most powerful and reliable load balancers available. It is best known for HTTP, but databases, Redis, message queues and TLS services speak their own protocols over TCP, and they need load balancing and failover too.</p>
<p>This guide covers HAProxy in TCP mode: algorithms, protocol-aware health checks, connection limits, failover patterns, TLS passthrough, client IP preservation and logging, ending with a complete PostgreSQL and Redis configuration.</p>
<h2 id="why-tcp-load-balancing">Why TCP Load Balancing? <a class="anchor" data-pagefind-ignore href="#why-tcp-load-balancing" aria-label="Link to this section">#</a></h2>
<p>Layer 4 load balancing works at the transport layer: it routes on IP addresses and ports and forwards the bytes without reading them.</p>
<ul>
<li><strong>Protocol agnostic</strong>: Works with any TCP-based protocol (databases, message queues, custom protocols)</li>
<li><strong>Lower latency</strong>: No content inspection means faster routing decisions</li>
<li><strong>TLS passthrough</strong>: End-to-end encryption without termination at the load balancer</li>
<li><strong>Simplicity</strong>: Fewer moving parts compared to Layer 7 configuration</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> for non-HTTP protocols, and for HTTPS when the backends must terminate TLS themselves.</li>
<li><strong>Skip it</strong> for plain HTTP(S) services where you want path routing, header changes or HTTP-level logs: use <code>mode http</code>.</li>
</ul>
<table>
	<thead>
			<tr>
					<th></th>
					<th><code>mode tcp</code> (Layer 4)</th>
					<th><code>mode http</code> (Layer 7)</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Routes on</td>
					<td>IP, port, TLS SNI</td>
					<td>Host, path, headers, cookies</td>
			</tr>
			<tr>
					<td>Sees content</td>
					<td>No</td>
					<td>Yes</td>
			</tr>
			<tr>
					<td>Protocols</td>
					<td>Any TCP</td>
					<td>HTTP/1.x, HTTP/2, HTTP/3</td>
			</tr>
			<tr>
					<td>TLS</td>
					<td>Passthrough or termination</td>
					<td>Termination</td>
			</tr>
	</tbody>
</table>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Frontend</td>
					<td>HAProxy</td>
					<td>Listens on a port (<code>bind *:5432</code>) and picks a backend.</td>
			</tr>
			<tr>
					<td>Backend</td>
					<td>HAProxy</td>
					<td>Holds the server list, the algorithm and the health check.</td>
			</tr>
			<tr>
					<td>Servers</td>
					<td>Database, Redis or app hosts</td>
					<td>Receive the forwarded TCP connections.</td>
			</tr>
			<tr>
					<td>Stats socket</td>
					<td>HAProxy</td>
					<td>Runtime control: drain, disable, show state.</td>
			</tr>
	</tbody>
</table>
<p>The flow: client connects to HAProxy → frontend picks a backend → algorithm picks a healthy server → HAProxy opens a connection to it and copies bytes both ways until one side closes.</p>
<h2 id="installation">Installation <a class="anchor" data-pagefind-ignore href="#installation" aria-label="Link to this section">#</a></h2>
<p>On Debian/Ubuntu, use the official HAProxy packages from <a href="https://haproxy.debian.net" rel="noopener">haproxy.debian.net</a> instead of the distribution package, which lags several branches behind:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Ubuntu 24.04: official PPA. 3.3 is the newest branch built for 24.04; it is a</span>
</span></span><span class="line"><span class="cl"><span class="c1"># short-lived, non-LTS branch. HAProxy 3.4 LTS is available for Ubuntu 26.04</span>
</span></span><span class="line"><span class="cl"><span class="c1"># (ppa:vbernat/haproxy-3.4) and for Debian 13 (below).</span>
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install --no-install-recommends software-properties-common
</span></span><span class="line"><span class="cl">add-apt-repository ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">apt install <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Debian 13: official haproxy.debian.net repository (3.4 LTS)</span>
</span></span><span class="line"><span class="cl">curl https://haproxy.debian.net/haproxy-archive-keyring.gpg <span class="se">\
</span></span></span><span class="line"><span class="cl">  --create-dirs --output /etc/apt/keyrings/haproxy-archive-keyring.gpg
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> deb <span class="s2">&#34;[signed-by=/etc/apt/keyrings/haproxy-archive-keyring.gpg]&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  https://haproxy.debian.net trixie-backports-3.4 main <span class="se">\
</span></span></span><span class="line"><span class="cl">  &gt; /etc/apt/sources.list.d/haproxy.list
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install <span class="nv">haproxy</span><span class="o">=</span>3.4.<span class="se">\*</span>
</span></span></code></pre></div><p>On RHEL/CentOS there is no official HAProxy package repository, so you get the distribution version:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">yum install haproxy
</span></span><span class="line"><span class="cl"><span class="c1"># RHEL/Rocky/Alma 9 AppStream ships the 2.8 LTS branch; check what you got</span>
</span></span><span class="line"><span class="cl">haproxy -v
</span></span></code></pre></div><p>The RHEL unit file does not create <code>/run/haproxy</code>, so the <code>stats socket /run/haproxy/admin.sock</code> line used throughout this post stops HAProxy from starting. Let systemd create the directory:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">mkdir -p /etc/systemd/system/haproxy.service.d
</span></span><span class="line"><span class="cl"><span class="nb">printf</span> <span class="s1">&#39;[Service]\nRuntimeDirectory=haproxy\n&#39;</span> &gt; /etc/systemd/system/haproxy.service.d/runtime-directory.conf
</span></span><span class="line"><span class="cl">systemctl daemon-reload
</span></span></code></pre></div><h2 id="basic-tcp-load-balancer-configuration">Basic TCP Load Balancer Configuration <a class="anchor" data-pagefind-ignore href="#basic-tcp-load-balancer-configuration" aria-label="Link to this section">#</a></h2>
<p>A minimal load balancer for three MySQL servers. What matters for TCP:</p>
<ul>
<li><code>mode tcp</code> and <code>option tcplog</code> in <code>defaults</code>: every proxy works at Layer 4 and logs connections, not requests.</li>
<li><code>timeout client</code> / <code>timeout server</code>: idle time before HAProxy closes the connection. Too low cuts off idle database sessions.</li>
<li><code>retries 3</code>: connection attempts to a server before giving up.</li>
<li><code>option mysql-check</code>: a health check that logs in to MySQL instead of only opening the port.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    # Performance tuning
    maxconn 50000
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    timeout connect 10s
    timeout client  30s
    timeout server  30s
    retries 3

frontend mysql_frontend
    bind *:3306
    default_backend mysql_servers

backend mysql_servers
    balance roundrobin
    option mysql-check user haproxy
    server mysql1 192.168.1.10:3306 check
    server mysql2 192.168.1.11:3306 check
    server mysql3 192.168.1.12:3306 check
</code></pre><h2 id="load-balancing-algorithms">Load Balancing Algorithms <a class="anchor" data-pagefind-ignore href="#load-balancing-algorithms" aria-label="Link to this section">#</a></h2>
<p>The algorithm decides which server gets the next connection.</p>
<table>
	<thead>
			<tr>
					<th>Algorithm</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>roundrobin</code></td>
					<td>Short connections of similar cost; supports weights</td>
			</tr>
			<tr>
					<td><code>leastconn</code></td>
					<td>Long-lived connections such as database sessions</td>
			</tr>
			<tr>
					<td><code>source</code></td>
					<td>Clients that must stay on the same server</td>
			</tr>
			<tr>
					<td><code>first</code></td>
					<td>Filling servers one by one, for example so spare servers can sleep</td>
			</tr>
	</tbody>
</table>
<h3 id="round-robin">Round Robin <a class="anchor" data-pagefind-ignore href="#round-robin" aria-label="Link to this section">#</a></h3>
<p>Distributes connections in turn, in proportion to <code>weight</code>:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance roundrobin
    server s1 192.168.1.10:5432 check weight 1
    server s2 192.168.1.11:5432 check weight 2  # Gets 2x traffic
    server s3 192.168.1.12:5432 check weight 1
</code></pre><h3 id="least-connections">Least Connections <a class="anchor" data-pagefind-ignore href="#least-connections" aria-label="Link to this section">#</a></h3>
<p>Sends traffic to the server with fewest active connections:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance leastconn
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check
    server s3 192.168.1.12:5432 check
</code></pre><h3 id="source-ip-hash">Source IP Hash <a class="anchor" data-pagefind-ignore href="#source-ip-hash" aria-label="Link to this section">#</a></h3>
<p>Ensures the same client always connects to the same server (session persistence). <code>hash-type consistent</code> moves only a few clients when a server is added or removed.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance source
    hash-type consistent
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check
    server s3 192.168.1.12:5432 check
</code></pre><h3 id="first-available">First Available <a class="anchor" data-pagefind-ignore href="#first-available" aria-label="Link to this section">#</a></h3>
<p>Uses the first server with available connection slots. Requires <code>maxconn</code> on the servers: the next server only gets traffic when the first is full.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance first
    server s1 192.168.1.10:5432 check maxconn 100
    server s2 192.168.1.11:5432 check maxconn 100
</code></pre><h2 id="health-checks">Health Checks <a class="anchor" data-pagefind-ignore href="#health-checks" aria-label="Link to this section">#</a></h2>
<p>HAProxy only sends traffic to servers that pass their check. An open port is not a working service, so use a protocol check where one exists.</p>
<table>
	<thead>
			<tr>
					<th>Check</th>
					<th>Proves</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>option tcp-check</code> (no steps)</td>
					<td>The port accepts connections</td>
			</tr>
			<tr>
					<td><code>mysql-check</code>, <code>pgsql-check</code></td>
					<td>The database answers its handshake</td>
			</tr>
			<tr>
					<td><code>tcp-check send</code> / <code>expect</code></td>
					<td>The service gives the right answer to a request you define</td>
			</tr>
	</tbody>
</table>
<h3 id="basic-tcp-check">Basic TCP Check <a class="anchor" data-pagefind-ignore href="#basic-tcp-check" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    option tcp-check
    server s1 192.168.1.10:5432 check
</code></pre><h3 id="protocol-specific-health-checks">Protocol-Specific Health Checks <a class="anchor" data-pagefind-ignore href="#protocol-specific-health-checks" aria-label="Link to this section">#</a></h3>
<h4 id="mysqlmariadb">MySQL/MariaDB <a class="anchor" data-pagefind-ignore href="#mysqlmariadb" aria-label="Link to this section">#</a></h4>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend mysql_servers
    option mysql-check user haproxy
    server mysql1 192.168.1.10:3306 check
</code></pre><p>The check logs in as <code>haproxy</code> without a password. Create the HAProxy user in MySQL:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">USER</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="o">@</span><span class="s1">&#39;%&#39;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">FLUSH</span><span class="w"> </span><span class="k">PRIVILEGES</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><h4 id="postgresql">PostgreSQL <a class="anchor" data-pagefind-ignore href="#postgresql" aria-label="Link to this section">#</a></h4>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend postgres_servers
    option pgsql-check user haproxy
    server pg1 192.168.1.10:5432 check
</code></pre><h4 id="redis">Redis <a class="anchor" data-pagefind-ignore href="#redis" aria-label="Link to this section">#</a></h4>
<p>Sends <code>PING</code> and expects <code>+PONG</code>.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend redis_servers
    option tcp-check
    tcp-check connect
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    server redis1 192.168.1.10:6379 check
</code></pre><h4 id="custom-tcp-check">Custom TCP Check <a class="anchor" data-pagefind-ignore href="#custom-tcp-check" aria-label="Link to this section">#</a></h4>
<p>Sends a raw HTTP request and expects a <code>200</code> status line. <code>tcp-check connect port</code> lets the check use another port than the traffic, for a health endpoint on a separate port.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend custom_servers
    option tcp-check
    tcp-check connect port 8080
    tcp-check send &#34;GET /health HTTP/1.1\r\nHost: localhost\r\n\r\n&#34;
    tcp-check expect rstring ^HTTP/1\.[01]\ 200
    server app1 192.168.1.10:8080 check
</code></pre><h3 id="health-check-parameters">Health Check Parameters <a class="anchor" data-pagefind-ignore href="#health-check-parameters" aria-label="Link to this section">#</a></h3>
<p><code>default-server</code> sets the values for all servers in the backend; a value on a <code>server</code> line overrides it.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    default-server inter 3s fall 3 rise 2
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check inter 5s fall 2 rise 3
</code></pre><ul>
<li><code>inter</code>: Interval between checks (default: 2s)</li>
<li><code>fall</code>: Number of failed checks before marking server down</li>
<li><code>rise</code>: Number of successful checks before marking server up</li>
</ul>
<h2 id="connection-management">Connection Management <a class="anchor" data-pagefind-ignore href="#connection-management" aria-label="Link to this section">#</a></h2>
<p>Limits protect the backends: HAProxy queues or rejects connections instead of overloading a server.</p>
<h3 id="maximum-connections">Maximum Connections <a class="anchor" data-pagefind-ignore href="#maximum-connections" aria-label="Link to this section">#</a></h3>
<p><code>maxconn</code> works at three levels:</p>
<ul>
<li><code>global</code>: the whole process. Above it, HAProxy stops accepting.</li>
<li><code>frontend</code>: per listener.</li>
<li><code>server</code>: per backend server. Extra connections wait in the backend queue.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    maxconn 50000

frontend db_frontend
    bind *:5432
    maxconn 10000
    default_backend db_servers

backend db_servers
    server db1 192.168.1.10:5432 check maxconn 500
    server db2 192.168.1.11:5432 check maxconn 500
</code></pre><h3 id="connection-queuing">Connection Queuing <a class="anchor" data-pagefind-ignore href="#connection-queuing" aria-label="Link to this section">#</a></h3>
<p><code>maxqueue 50</code> caps the queue for a server; beyond that, connections go to another server or fail instead of waiting.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    fullconn 1000
    server s1 192.168.1.10:5432 check maxconn 100 maxqueue 50
</code></pre><h3 id="connection-timeouts">Connection Timeouts <a class="anchor" data-pagefind-ignore href="#connection-timeouts" aria-label="Link to this section">#</a></h3>
<p>Each timeout covers one phase of a connection; the comments say which.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">defaults
    timeout connect 5s      # Time to establish connection to server
    timeout client  60s     # Inactivity timeout on client side
    timeout server  60s     # Inactivity timeout on server side
    timeout tunnel  1h      # Timeout for tunneled connections (WebSocket, etc.)
    timeout client-fin 30s  # Timeout for client FIN
    timeout server-fin 30s  # Timeout for server FIN
</code></pre><p>Database clients and pools keep connections open and idle. Raise the client and server timeouts, or HAProxy closes them:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">defaults
    timeout client  3600s
    timeout server  3600s
</code></pre><h2 id="high-availability-patterns">High Availability Patterns <a class="anchor" data-pagefind-ignore href="#high-availability-patterns" aria-label="Link to this section">#</a></h2>
<h3 id="active-passive-with-backup-servers">Active-Passive with Backup Servers <a class="anchor" data-pagefind-ignore href="#active-passive-with-backup-servers" aria-label="Link to this section">#</a></h3>
<p>Servers marked <code>backup</code> only get traffic when all normal servers are down.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend db_servers
    balance roundrobin
    server primary1 192.168.1.10:5432 check
    server primary2 192.168.1.11:5432 check
    server backup1  192.168.1.20:5432 check backup
    server backup2  192.168.1.21:5432 check backup
</code></pre><h3 id="readwrite-splitting">Read/Write Splitting <a class="anchor" data-pagefind-ignore href="#readwrite-splitting" aria-label="Link to this section">#</a></h3>
<p>For database clusters with a primary and replicas. One frontend listens on two ports; <code>dst_port</code> sends writes to the primary and reads to the replicas. The application chooses the port.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mysql_frontend
    bind *:3306
    bind *:3307
    
    # Port 3306 -&gt; Write (Primary)
    # Port 3307 -&gt; Read (Replicas)
    use_backend mysql_write if { dst_port 3306 }
    use_backend mysql_read if { dst_port 3307 }

backend mysql_write
    option mysql-check user haproxy
    server primary 192.168.1.10:3306 check

backend mysql_read
    balance leastconn
    option mysql-check user haproxy
    server replica1 192.168.1.11:3306 check
    server replica2 192.168.1.12:3306 check
    server primary  192.168.1.10:3306 check backup  # Fallback to primary
</code></pre><h3 id="graceful-server-maintenance">Graceful Server Maintenance <a class="anchor" data-pagefind-ignore href="#graceful-server-maintenance" aria-label="Link to this section">#</a></h3>
<p>Take a server out without cutting active connections. <code>weight 0</code> in the configuration stops new connections; the stats socket does the same at runtime without a reload.</p>
<table>
	<thead>
			<tr>
					<th>Command</th>
					<th>Effect</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>disable server</code></td>
					<td>Maintenance mode: no new connections, existing ones continue</td>
			</tr>
			<tr>
					<td><code>set server ... state drain</code></td>
					<td>No new load-balanced connections, health checks keep running</td>
			</tr>
			<tr>
					<td><code>enable server</code></td>
					<td>Back in rotation</td>
			</tr>
	</tbody>
</table>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check weight 0  # No new connections (drain)
</code></pre><p>Use the stats socket to manage servers dynamically:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Disable a server (existing connections preserved)</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;disable server servers/s1&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Set server to drain mode</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;set server servers/s1 state drain&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Re-enable a server</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;enable server servers/s1&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check server status</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show servers state&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span></code></pre></div><h2 id="tls-passthrough">TLS Passthrough <a class="anchor" data-pagefind-ignore href="#tls-passthrough" aria-label="Link to this section">#</a></h2>
<p>For end-to-end encryption without terminating TLS at the load balancer. HAProxy forwards the encrypted stream; certificates live on the backends.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend tls_frontend
    bind *:443
    mode tcp
    option tcplog
    default_backend tls_servers

backend tls_servers
    mode tcp
    balance roundrobin
    server s1 192.168.1.10:443 check
    server s2 192.168.1.11:443 check
</code></pre><h3 id="sni-based-routing">SNI-Based Routing <a class="anchor" data-pagefind-ignore href="#sni-based-routing" aria-label="Link to this section">#</a></h3>
<p>Route to different backends based on the TLS SNI header, the hostname the client sends unencrypted in its TLS hello.</p>
<ul>
<li><code>tcp-request inspect-delay 5s</code>: wait up to 5 seconds for the hello before deciding.</li>
<li><code>req_ssl_hello_type 1</code>: accept once a ClientHello arrived.</li>
<li><code>req_ssl_sni</code>: the requested hostname.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend tls_frontend
    bind *:443
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    
    use_backend app1_servers if { req_ssl_sni -i app1.example.com }
    use_backend app2_servers if { req_ssl_sni -i app2.example.com }
    default_backend default_servers

backend app1_servers
    mode tcp
    server app1 192.168.1.10:443 check

backend app2_servers
    mode tcp
    server app2 192.168.1.20:443 check

backend default_servers
    mode tcp
    server default1 192.168.1.30:443 check
</code></pre><h2 id="client-ip-preservation">Client IP Preservation <a class="anchor" data-pagefind-ignore href="#client-ip-preservation" aria-label="Link to this section">#</a></h2>
<h3 id="proxy-protocol">PROXY Protocol <a class="anchor" data-pagefind-ignore href="#proxy-protocol" aria-label="Link to this section">#</a></h3>
<p>In TCP mode the backend sees HAProxy&rsquo;s IP, not the client&rsquo;s, and there are no headers to add <code>X-Forwarded-For</code> to. The PROXY protocol passes client information through TCP load balancers by sending a short header before the data:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend tcp_frontend
    bind *:5432
    default_backend tcp_servers

backend tcp_servers
    server s1 192.168.1.10:5432 check send-proxy-v2
</code></pre><p>The backend server must support PROXY protocol, or it reads the header as garbage and drops the connection. For HAProxy as the backend:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend backend_frontend
    bind *:5432 accept-proxy
    # ...
</code></pre><h2 id="logging">Logging <a class="anchor" data-pagefind-ignore href="#logging" aria-label="Link to this section">#</a></h2>
<h3 id="tcp-specific-logging">TCP-Specific Logging <a class="anchor" data-pagefind-ignore href="#tcp-specific-logging" aria-label="Link to this section">#</a></h3>
<p><code>option tcplog</code> logs one line per connection. The custom <code>log-format</code> adds timings and connection counts.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    log 127.0.0.1:514 local0 info

defaults
    mode tcp
    option tcplog
    log global

# Custom TCP log format
frontend tcp_frontend
    bind *:5432
    log-format &#34;%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq&#34;
</code></pre><p>Log format variables for TCP:</p>
<ul>
<li><code>%ci</code>: Client IP</li>
<li><code>%cp</code>: Client port</li>
<li><code>%ft</code>: Frontend name</li>
<li><code>%b</code>: Backend name</li>
<li><code>%s</code>: Server name</li>
<li><code>%Tw</code>: Time waiting in queue (ms)</li>
<li><code>%Tc</code>: Time to connect to server (ms)</li>
<li><code>%Tt</code>: Total time (ms)</li>
<li><code>%B</code>: Bytes read from server</li>
<li><code>%ts</code>: Termination state</li>
</ul>
<p>HAProxy sends logs to syslog on UDP 514. Configure rsyslog to receive them and write <code>local0</code> to a file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/rsyslog.d/haproxy.conf</span>
</span></span><span class="line"><span class="cl"><span class="nv">$ModLoad</span> imudp
</span></span><span class="line"><span class="cl"><span class="nv">$UDPServerRun</span> <span class="m">514</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">local0.* /var/log/haproxy.log
</span></span></code></pre></div><h2 id="statistics-and-monitoring">Statistics and Monitoring <a class="anchor" data-pagefind-ignore href="#statistics-and-monitoring" aria-label="Link to this section">#</a></h2>
<p>The stats page is HTTP even when all your proxies are TCP, so it sets <code>mode http</code> itself.</p>
<h3 id="stats-page">Stats Page <a class="anchor" data-pagefind-ignore href="#stats-page" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST
</code></pre><h3 id="prometheus-metrics">Prometheus Metrics <a class="anchor" data-pagefind-ignore href="#prometheus-metrics" aria-label="Link to this section">#</a></h3>
<p>See <a href="https://blog.bsdserver.nl/haproxy-monitoring-with-prometheus-complete-observability-guide/">HAProxy Monitoring with Prometheus</a> for scraping and alerting.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend stats
    bind *:8404
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
</code></pre><h2 id="complete-production-example">Complete Production Example <a class="anchor" data-pagefind-ignore href="#complete-production-example" aria-label="Link to this section">#</a></h2>
<p>A complete configuration for a PostgreSQL cluster and Redis:</p>
<table>
	<thead>
			<tr>
					<th>Frontend</th>
					<th>Port</th>
					<th>Backend</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>pg_write</code></td>
					<td>5432</td>
					<td>Primary, standby as backup; over 5000 connections are rejected</td>
			</tr>
			<tr>
					<td><code>pg_read</code></td>
					<td>5433</td>
					<td>Standbys with <code>leastconn</code>, primary as backup</td>
			</tr>
			<tr>
					<td><code>redis</code></td>
					<td>6379</td>
					<td>Only the server reporting <code>role:master</code> passes the check</td>
			</tr>
			<tr>
					<td><code>stats</code></td>
					<td>8404</td>
					<td>Stats page and <code>/metrics</code></td>
			</tr>
	</tbody>
</table>
<p>Settings worth knowing:</p>
<ul>
<li><code>option redispatch</code>: retry on another server if the chosen one fails.</li>
<li><code>option log-health-checks</code>: log health check state changes.</li>
<li><code>pgsql-check</code> checks that PostgreSQL answers, not whether it is the primary.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    maxconn 100000
    nbthread 4
    cpu-map auto:1/1-4 0-3
    
    # Security
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    option  log-health-checks
    
    timeout connect 5s
    timeout client  3600s
    timeout server  3600s
    timeout check   3s
    
    retries 3
    option  redispatch
    
    default-server inter 3s fall 3 rise 2

# Stats frontend
listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST
    http-request use-service prometheus-exporter if { path /metrics }

# PostgreSQL Primary (Read/Write)
frontend pg_write
    bind *:5432
    default_backend pg_primary
    
    # ACL for connection limiting
    acl too_many_conns fe_conn gt 5000
    tcp-request connection reject if too_many_conns

backend pg_primary
    option pgsql-check user haproxy
    server pg-primary 192.168.1.10:5432 check
    server pg-standby1 192.168.1.11:5432 check backup

# PostgreSQL Replicas (Read-Only)
frontend pg_read
    bind *:5433
    default_backend pg_replicas

backend pg_replicas
    balance leastconn
    option pgsql-check user haproxy
    server pg-standby1 192.168.1.11:5432 check
    server pg-standby2 192.168.1.12:5432 check
    server pg-primary  192.168.1.10:5432 check backup  # Fallback

# Redis Cluster
frontend redis
    bind *:6379
    default_backend redis_servers

backend redis_servers
    balance first
    option tcp-check
    tcp-check connect
    tcp-check send &#34;PING\r\n&#34;
    tcp-check expect string +PONG
    tcp-check send &#34;INFO replication\r\n&#34;
    tcp-check expect string role:master
    tcp-check send &#34;QUIT\r\n&#34;
    tcp-check expect string +OK
    server redis1 192.168.1.20:6379 check
    server redis2 192.168.1.21:6379 check
    server redis3 192.168.1.22:6379 check
</code></pre><h2 id="validation-and-testing">Validation and Testing <a class="anchor" data-pagefind-ignore href="#validation-and-testing" aria-label="Link to this section">#</a></h2>
<h3 id="configuration-validation">Configuration Validation <a class="anchor" data-pagefind-ignore href="#configuration-validation" aria-label="Link to this section">#</a></h3>
<p>Check the syntax before every reload.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">haproxy -c -f /etc/haproxy/haproxy.cfg
</span></span></code></pre></div><h3 id="load-testing">Load Testing <a class="anchor" data-pagefind-ignore href="#load-testing" aria-label="Link to this section">#</a></h3>
<p>Open many connections at once, then run a real database benchmark through HAProxy.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Test TCP connection throughput</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> i in <span class="o">{</span>1..100<span class="o">}</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">  nc -z -w1 haproxy.example.com <span class="m">5432</span> <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span><span class="line"><span class="cl"><span class="nb">wait</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># PostgreSQL specific (initialise the pgbench tables once first)</span>
</span></span><span class="line"><span class="cl">pgbench -i -h haproxy.example.com -p <span class="m">5432</span> mydb
</span></span><span class="line"><span class="cl">pgbench -h haproxy.example.com -p <span class="m">5432</span> -c <span class="m">50</span> -j <span class="m">4</span> -T <span class="m">60</span> mydb
</span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Connections dropped after idle time</td>
					<td>Increase <code>timeout server</code> and <code>timeout client</code></td>
			</tr>
			<tr>
					<td>Server marked DOWN</td>
					<td>Check health check configuration and network connectivity</td>
			</tr>
			<tr>
					<td>Connection refused</td>
					<td>Verify backend servers are listening and firewall rules</td>
			</tr>
			<tr>
					<td>Slow connections</td>
					<td>Check <code>balance</code> algorithm and server capacity</td>
			</tr>
			<tr>
					<td>HAProxy does not start on RHEL</td>
					<td>Missing <code>/run/haproxy</code>; see the <code>RuntimeDirectory</code> fix above</td>
			</tr>
	</tbody>
</table>
<h3 id="debug-mode">Debug Mode <a class="anchor" data-pagefind-ignore href="#debug-mode" aria-label="Link to this section">#</a></h3>
<p><code>haproxy -d</code> runs in the foreground and prints every event. The socket commands show process info, per-server stats and the last protocol errors.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Run HAProxy in debug mode</span>
</span></span><span class="line"><span class="cl">haproxy -d -f /etc/haproxy/haproxy.cfg
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check runtime status</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show info&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show stat&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show errors&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span></code></pre></div><h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-tcp-lb" rel="noopener">pderuiter/blog-haproxy-tcp-lb</a>. It creates an HAProxy VM on Proxmox and configures the complete production example from this post, PostgreSQL read/write and read-only frontends and the Redis master check included; set your database and Redis servers in <code>group_vars</code>, run <code>terraform apply</code>, then <code>ansible-playbook site.yml</code>.</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>Next in the series: DNS load balancing with HAProxy, to spread DNS queries across multiple resolvers.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<p>Key takeaways:</p>
<ul>
<li>Use <code>mode tcp</code> for Layer 4 load balancing</li>
<li>Choose the right balancing algorithm for your use case</li>
<li>Implement proper health checks for each protocol</li>
<li>Configure appropriate timeouts for your workload</li>
<li>Use the stats socket for runtime management</li>
<li>Enable logging for troubleshooting and monitoring</li>
</ul>
<p>TCP load balancing is the building block for highly available infrastructure, whether you&rsquo;re running databases, message queues, or custom services.</p>
]]></content:encoded></item><item><title>HAProxy with Let's Encrypt: Automated HTTPS Certificate Management</title><link>https://blog.bsdserver.nl/haproxy-with-lets-encrypt-automated-https-certificate-management/</link><guid isPermaLink="false">693b734379eb54000160a5f2</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:43:31 +0000</pubDate><description>HTTPS is no longer optional - it's a requirement for modern web applications.</description><category>HAProxy</category><category>Let's Encrypt</category><category>HTTPS</category><category>TLS</category><category>Certificates</category><category>Security</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-letsencrypt-certificates.png" medium="image"/><content:encoded><![CDATA[<p>HTTPS is no longer optional - it&rsquo;s a requirement for modern web applications. Let&rsquo;s Encrypt certificates are free but expire after 90 days, so renewing and installing them by hand does not scale.</p>
<p>This guide connects HAProxy to certbot: HAProxy terminates TLS and routes the ACME challenge, certbot obtains and renews the certificates, and a deploy hook loads renewed certificates into HAProxy, without a restart where possible.</p>
<h2 id="why-lets-encrypt-with-haproxy">Why Let&rsquo;s Encrypt with HAProxy? <a class="anchor" data-pagefind-ignore href="#why-lets-encrypt-with-haproxy" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Free, trusted certificates</strong> for every domain behind the load balancer</li>
<li><strong>One place for TLS</strong>: backends speak plain HTTP, certificates live only on HAProxy</li>
<li><strong>Automatic renewal</strong> with a systemd timer or cron</li>
<li><strong>No downtime</strong>: the runtime API swaps certificates in a running HAProxy</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> for public sites where HAProxy is the TLS endpoint and is reachable from the internet (HTTP-01) or your DNS has an API (DNS-01).</li>
<li><strong>Skip it</strong> for internal services without public DNS (use an internal CA), or when backends must terminate TLS themselves (TLS passthrough).</li>
</ul>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy</td>
					<td>Load balancer</td>
					<td>Terminates TLS with the PEM files in <code>/etc/haproxy/certs/</code>; sends <code>/.well-known/acme-challenge/</code> on port 80 to the challenge server.</td>
			</tr>
			<tr>
					<td>Challenge server</td>
					<td>Load balancer, <code>127.0.0.1:8080</code> (8888 in Method 1)</td>
					<td>Serves the webroot where certbot writes challenge files.</td>
			</tr>
			<tr>
					<td>Certbot</td>
					<td>Load balancer</td>
					<td>Requests and renews certificates in <code>/etc/letsencrypt/live/&lt;name&gt;/</code>.</td>
			</tr>
			<tr>
					<td>Deploy hook</td>
					<td>Load balancer</td>
					<td>Combines <code>fullchain.pem</code> and <code>privkey.pem</code>, loads the result into HAProxy.</td>
			</tr>
	</tbody>
</table>
<p>The flow: certbot writes a challenge file → Let&rsquo;s Encrypt fetches it over HTTP → HAProxy routes it to the challenge server → certificate issued → deploy hook builds the PEM → runtime API updates HAProxy.</p>
<h3 id="choosing-a-method">Choosing a method <a class="anchor" data-pagefind-ignore href="#choosing-a-method" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Method</th>
					<th>HAProxy downtime</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>1. HTTP-01 webroot</td>
					<td>None</td>
					<td>The basic setup behind HAProxy</td>
			</tr>
			<tr>
					<td>2. Standalone with hooks</td>
					<td>Seconds per renewal</td>
					<td>Simple setups where a short stop is acceptable</td>
			</tr>
			<tr>
					<td>3. Runtime API</td>
					<td>None, no reload</td>
					<td>Production (recommended)</td>
			</tr>
			<tr>
					<td>4. DNS-01</td>
					<td>None</td>
					<td>Wildcard certificates, servers not reachable on port 80</td>
			</tr>
	</tbody>
</table>
<h2 id="prerequisites">Prerequisites <a class="anchor" data-pagefind-ignore href="#prerequisites" aria-label="Link to this section">#</a></h2>
<ul>
<li>HAProxy from the official packages (examples checked on 3.4 LTS and on 3.3, the newest branch for Ubuntu 24.04)</li>
<li>Certbot installed</li>
<li>Domain(s) pointing to your HAProxy server</li>
<li>Port 80 and 443 accessible from the internet</li>
</ul>
<h2 id="installation">Installation <a class="anchor" data-pagefind-ignore href="#installation" aria-label="Link to this section">#</a></h2>
<h3 id="install-haproxy">Install HAProxy <a class="anchor" data-pagefind-ignore href="#install-haproxy" aria-label="Link to this section">#</a></h3>
<p>The runtime certificate API used in Method 3 needs HAProxy 2.4 or later. Use the official packages from <a href="https://haproxy.debian.net" rel="noopener">haproxy.debian.net</a> rather than the distribution package:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Ubuntu 24.04: official PPA. 3.3 is the newest branch built for 24.04; it is a</span>
</span></span><span class="line"><span class="cl"><span class="c1"># short-lived, non-LTS branch. HAProxy 3.4 LTS is available for Ubuntu 26.04</span>
</span></span><span class="line"><span class="cl"><span class="c1"># (ppa:vbernat/haproxy-3.4) and for Debian 13 (below).</span>
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install --no-install-recommends software-properties-common
</span></span><span class="line"><span class="cl">add-apt-repository ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">apt install <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Debian 13: official haproxy.debian.net repository (3.4 LTS)</span>
</span></span><span class="line"><span class="cl">curl https://haproxy.debian.net/haproxy-archive-keyring.gpg <span class="se">\
</span></span></span><span class="line"><span class="cl">  --create-dirs --output /etc/apt/keyrings/haproxy-archive-keyring.gpg
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> deb <span class="s2">&#34;[signed-by=/etc/apt/keyrings/haproxy-archive-keyring.gpg]&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  https://haproxy.debian.net trixie-backports-3.4 main <span class="se">\
</span></span></span><span class="line"><span class="cl">  &gt; /etc/apt/sources.list.d/haproxy.list
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install <span class="nv">haproxy</span><span class="o">=</span>3.4.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># RHEL/CentOS (no official repository: AppStream ships 2.8)</span>
</span></span><span class="line"><span class="cl">yum install haproxy
</span></span><span class="line"><span class="cl"><span class="c1"># RHEL only: the unit does not create /run/haproxy, which the admin socket below needs</span>
</span></span><span class="line"><span class="cl">mkdir -p /etc/systemd/system/haproxy.service.d
</span></span><span class="line"><span class="cl"><span class="nb">printf</span> <span class="s1">&#39;[Service]\nRuntimeDirectory=haproxy\n&#39;</span> &gt; /etc/systemd/system/haproxy.service.d/runtime-directory.conf
</span></span><span class="line"><span class="cl">systemctl daemon-reload
</span></span></code></pre></div><h3 id="install-certbot">Install Certbot <a class="anchor" data-pagefind-ignore href="#install-certbot" aria-label="Link to this section">#</a></h3>
<p>Certbot is the Let&rsquo;s Encrypt client. Pick one source; the snap gets updates first.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Debian/Ubuntu</span>
</span></span><span class="line"><span class="cl">apt install certbot
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># RHEL/CentOS (certbot comes from EPEL)</span>
</span></span><span class="line"><span class="cl">yum install epel-release
</span></span><span class="line"><span class="cl">yum install certbot
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Or using snap (recommended by Let&#39;s Encrypt)</span>
</span></span><span class="line"><span class="cl">snap install --classic certbot
</span></span><span class="line"><span class="cl">ln -s /snap/bin/certbot /usr/bin/certbot
</span></span></code></pre></div><h2 id="haproxy-certificate-format">HAProxy Certificate Format <a class="anchor" data-pagefind-ignore href="#haproxy-certificate-format" aria-label="Link to this section">#</a></h2>
<p>HAProxy requires certificates in a specific format: the certificate chain and private key combined in a single PEM file.</p>
<pre tabindex="0"><code>-----BEGIN CERTIFICATE-----
[Your domain certificate]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[Intermediate certificate]
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
[Private key]
-----END PRIVATE KEY-----
</code></pre><p>Let&rsquo;s Encrypt provides separate files (<code>fullchain.pem</code>, <code>privkey.pem</code>), so every method below combines them. Repeat this after each renewal, or HAProxy keeps serving the old certificate.</p>
<h2 id="method-1-http-01-challenge-with-haproxy">Method 1: HTTP-01 Challenge with HAProxy <a class="anchor" data-pagefind-ignore href="#method-1-http-01-challenge-with-haproxy" aria-label="Link to this section">#</a></h2>
<p>Let&rsquo;s Encrypt checks that you control a domain by fetching a token from <code>http://&lt;domain&gt;/.well-known/acme-challenge/</code> on port 80. HAProxy already owns port 80, so it routes that path to a small local web server and redirects everything else to HTTPS.</p>
<h3 id="haproxy-configuration">HAProxy Configuration <a class="anchor" data-pagefind-ignore href="#haproxy-configuration" aria-label="Link to this section">#</a></h3>
<ul>
<li><code>acl is_acme_challenge</code>: matches the challenge path; <code>use_backend</code> sends it to <code>127.0.0.1:8888</code>.</li>
<li><code>redirect scheme https unless is_acme_challenge</code>: all other HTTP goes to HTTPS. The challenge must stay on HTTP.</li>
<li><code>crt /etc/haproxy/certs/</code>: loads every PEM in the directory; HAProxy picks one by SNI.</li>
<li><code>Strict-Transport-Security</code>: tells browsers to use HTTPS only. <code>preload</code> and <code>includeSubDomains</code> are hard to undo, so add them only when every subdomain has HTTPS.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    user haproxy
    group haproxy
    daemon
    
    # SSL/TLS settings
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    tune.ssl.default-dh-param 2048

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5s
    timeout client  50s
    timeout server  50s

# HTTP Frontend - handles Let&#39;s Encrypt challenges and redirects
frontend http_frontend
    bind *:80
    
    # Let&#39;s Encrypt ACME challenge
    acl is_acme_challenge path_beg /.well-known/acme-challenge/
    use_backend letsencrypt_backend if is_acme_challenge
    
    # Redirect all other HTTP to HTTPS
    http-request redirect scheme https unless is_acme_challenge

# HTTPS Frontend
frontend https_frontend
    bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    
    # HSTS header
    http-response set-header Strict-Transport-Security &#34;max-age=31536000; includeSubDomains; preload&#34;
    
    # Default backend
    default_backend web_servers

# Backend for Let&#39;s Encrypt challenges
backend letsencrypt_backend
    server certbot 127.0.0.1:8888

# Your web servers
backend web_servers
    balance roundrobin
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check
</code></pre><h3 id="initial-certificate-request">Initial Certificate Request <a class="anchor" data-pagefind-ignore href="#initial-certificate-request" aria-label="Link to this section">#</a></h3>
<p>HAProxy does not start with an empty certificate directory, so the first certificate comes from certbot&rsquo;s standalone mode: certbot runs its own web server on port 80 while HAProxy is stopped.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Stop HAProxy temporarily</span>
</span></span><span class="line"><span class="cl">systemctl stop haproxy
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Get initial certificate</span>
</span></span><span class="line"><span class="cl">certbot certonly --standalone <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d example.com <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d www.example.com <span class="se">\
</span></span></span><span class="line"><span class="cl">  --email admin@example.com <span class="se">\
</span></span></span><span class="line"><span class="cl">  --agree-tos <span class="se">\
</span></span></span><span class="line"><span class="cl">  --non-interactive
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Combine certificate files for HAProxy</span>
</span></span><span class="line"><span class="cl">mkdir -p /etc/haproxy/certs
</span></span><span class="line"><span class="cl"><span class="nv">BASE_DIR</span><span class="o">=</span>/etc/letsencrypt/live/example.com
</span></span><span class="line"><span class="cl">cat <span class="nv">$BASE_DIR</span>/fullchain.pem <span class="nv">$BASE_DIR</span>/privkey.pem &gt; /etc/haproxy/certs/example.com.pem
</span></span><span class="line"><span class="cl">chmod <span class="m">600</span> /etc/haproxy/certs/example.com.pem
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Start HAProxy</span>
</span></span><span class="line"><span class="cl">systemctl start haproxy
</span></span></code></pre></div><h3 id="renewal-with-http-01-challenge">Renewal with HTTP-01 Challenge <a class="anchor" data-pagefind-ignore href="#renewal-with-http-01-challenge" aria-label="Link to this section">#</a></h3>
<p>Once HAProxy is running, use webroot mode for renewals: certbot writes the token into a directory, and a web server on port 8888 serves it to HAProxy&rsquo;s ACME backend.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Create directory for challenges</span>
</span></span><span class="line"><span class="cl">mkdir -p /var/www/letsencrypt/.well-known/acme-challenge
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Use a simple HTTP server for challenges</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Option 1: Python</span>
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> /var/www/letsencrypt <span class="o">&amp;&amp;</span> python3 -m http.server <span class="m">8888</span> <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Option 2: socat (install socat package)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Create a script to serve files</span>
</span></span></code></pre></div><p>Then point certbot at the webroot. Method 3 replaces the background Python process with a systemd service.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># For a certificate that already exists, add --force-renewal (otherwise certbot asks interactively)</span>
</span></span><span class="line"><span class="cl">certbot certonly --webroot <span class="se">\
</span></span></span><span class="line"><span class="cl">  -w /var/www/letsencrypt <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d example.com <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d www.example.com
</span></span></code></pre></div><h2 id="method-2-using-certbot-standalone-with-haproxy">Method 2: Using Certbot Standalone with HAProxy <a class="anchor" data-pagefind-ignore href="#method-2-using-certbot-standalone-with-haproxy" aria-label="Link to this section">#</a></h2>
<p>This method temporarily stops HAProxy during renewal, so certbot&rsquo;s standalone server can use port 80. Simple, but the site is down for a few seconds per renewal.</p>
<h3 id="pre-and-post-hooks">Pre and Post Hooks <a class="anchor" data-pagefind-ignore href="#pre-and-post-hooks" aria-label="Link to this section">#</a></h3>
<p>Certbot runs <code>pre-hook</code> before and <code>post-hook</code> after each renewal attempt.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/letsencrypt/cli.ini</span>
</span></span><span class="line"><span class="cl"><span class="c1"># cli.ini has no sections: these hooks apply to every certbot run.</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Only use this for certificates obtained with --standalone; webroot</span>
</span></span><span class="line"><span class="cl"><span class="c1"># renewals fail while HAProxy is stopped.</span>
</span></span><span class="line"><span class="cl">pre-hook <span class="o">=</span> systemctl stop haproxy
</span></span><span class="line"><span class="cl">post-hook <span class="o">=</span> /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh
</span></span></code></pre></div><h3 id="deployment-script">Deployment Script <a class="anchor" data-pagefind-ignore href="#deployment-script" aria-label="Link to this section">#</a></h3>
<p>Combines the renewed files for each domain and brings HAProxy back. <code>RENEWED_DOMAINS</code> is set by certbot.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -e
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_CERT_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/haproxy/certs&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Process each renewed domain</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> domain in <span class="nv">$RENEWED_DOMAINS</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Find the certificate directory</span>
</span></span><span class="line"><span class="cl">    <span class="nv">CERT_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/letsencrypt/live/</span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="o">[</span> -d <span class="s2">&#34;</span><span class="nv">$CERT_DIR</span><span class="s2">&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Combine cert and key for HAProxy</span>
</span></span><span class="line"><span class="cl">        cat <span class="s2">&#34;</span><span class="nv">$CERT_DIR</span><span class="s2">/fullchain.pem&#34;</span> <span class="s2">&#34;</span><span class="nv">$CERT_DIR</span><span class="s2">/privkey.pem&#34;</span> &gt; <span class="s2">&#34;</span><span class="nv">$HAPROXY_CERT_DIR</span><span class="s2">/</span><span class="nv">$domain</span><span class="s2">.pem&#34;</span>
</span></span><span class="line"><span class="cl">        chmod <span class="m">600</span> <span class="s2">&#34;</span><span class="nv">$HAPROXY_CERT_DIR</span><span class="s2">/</span><span class="nv">$domain</span><span class="s2">.pem&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="nb">echo</span> <span class="s2">&#34;Updated certificate for </span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">fi</span>
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Reload HAProxy</span>
</span></span><span class="line"><span class="cl">systemctl reload haproxy <span class="o">||</span> systemctl start haproxy
</span></span></code></pre></div><p>Make it executable:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">chmod +x /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh
</span></span></code></pre></div><h2 id="method-3-zero-downtime-renewal-recommended">Method 3: Zero-Downtime Renewal (Recommended) <a class="anchor" data-pagefind-ignore href="#method-3-zero-downtime-renewal-recommended" aria-label="Link to this section">#</a></h2>
<p>This approach uses HAProxy&rsquo;s runtime API to reload certificates without restart or reload. Existing connections are untouched.</p>
<h3 id="haproxy-configuration-1">HAProxy Configuration <a class="anchor" data-pagefind-ignore href="#haproxy-configuration-1" aria-label="Link to this section">#</a></h3>
<p>Changes from Method 1:</p>
<ul>
<li><code>expose-fd listeners</code> on the stats socket: allows seamless reloads for the fallback path.</li>
<li><code>strict-sni</code>: refuse TLS handshakes for names without a certificate instead of serving a wrong one.</li>
<li>The ACME backend moves to <code>127.0.0.1:8080</code>.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    # Enable runtime API for certificate management
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log global
    mode http
    option httplog
    timeout connect 5s
    timeout client  50s
    timeout server  50s

# HTTP Frontend
frontend http_frontend
    bind *:80
    
    # ACME challenge path
    acl is_acme path_beg /.well-known/acme-challenge/
    use_backend acme_backend if is_acme
    
    # Redirect to HTTPS
    http-request redirect scheme https code 301 unless is_acme

# HTTPS Frontend
frontend https_frontend
    bind *:443 ssl crt /etc/haproxy/certs/ strict-sni alpn h2,http/1.1
    
    http-response set-header Strict-Transport-Security &#34;max-age=63072000&#34;
    
    default_backend web_servers

# ACME challenge backend
backend acme_backend
    server acme 127.0.0.1:8080

backend web_servers
    balance roundrobin
    option httpchk GET /health
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check
</code></pre><h3 id="zero-downtime-certificate-update-script">Zero-Downtime Certificate Update Script <a class="anchor" data-pagefind-ignore href="#zero-downtime-certificate-update-script" aria-label="Link to this section">#</a></h3>
<p>For a certificate HAProxy already has loaded, the script swaps it in memory with <code>set ssl cert</code> and <code>commit ssl cert</code>. A new certificate is not known to the running process yet, so it falls back to a reload.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /usr/local/bin/haproxy-cert-update.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -e
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_CERT_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/haproxy/certs&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_SOCKET</span><span class="o">=</span><span class="s2">&#34;/run/haproxy/admin.sock&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">update_certificate<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="nb">local</span> <span class="nv">domain</span><span class="o">=</span><span class="nv">$1</span>
</span></span><span class="line"><span class="cl">    <span class="nb">local</span> <span class="nv">cert_dir</span><span class="o">=</span><span class="s2">&#34;/etc/letsencrypt/live/</span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="nb">local</span> <span class="nv">combined_cert</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$HAPROXY_CERT_DIR</span><span class="s2">/</span><span class="nv">$domain</span><span class="s2">.pem&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Combine certificates</span>
</span></span><span class="line"><span class="cl">    cat <span class="s2">&#34;</span><span class="nv">$cert_dir</span><span class="s2">/fullchain.pem&#34;</span> <span class="s2">&#34;</span><span class="nv">$cert_dir</span><span class="s2">/privkey.pem&#34;</span> &gt; <span class="s2">&#34;</span><span class="nv">$combined_cert</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    chmod <span class="m">600</span> <span class="s2">&#34;</span><span class="nv">$combined_cert</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Check if HAProxy is running with this certificate</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="nb">echo</span> <span class="s2">&#34;show ssl cert&#34;</span> <span class="p">|</span> socat stdio <span class="s2">&#34;</span><span class="nv">$HAPROXY_SOCKET</span><span class="s2">&#34;</span> <span class="p">|</span> grep -q <span class="s2">&#34;</span><span class="nv">$combined_cert</span><span class="s2">&#34;</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Update certificate in running HAProxy: the payload must be sent in the</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># same session as &#34;set ssl cert&#34;, the commit in a second session</span>
</span></span><span class="line"><span class="cl">        <span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;set ssl cert </span><span class="nv">$combined_cert</span><span class="s2"> &lt;&lt;&#34;</span><span class="p">;</span> cat <span class="s2">&#34;</span><span class="nv">$combined_cert</span><span class="s2">&#34;</span><span class="p">;</span> <span class="nb">echo</span> <span class="s2">&#34;&#34;</span><span class="p">;</span> <span class="o">}</span> <span class="p">|</span> socat stdio <span class="s2">&#34;</span><span class="nv">$HAPROXY_SOCKET</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="nb">echo</span> <span class="s2">&#34;commit ssl cert </span><span class="nv">$combined_cert</span><span class="s2">&#34;</span> <span class="p">|</span> socat stdio <span class="s2">&#34;</span><span class="nv">$HAPROXY_SOCKET</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="nb">echo</span> <span class="s2">&#34;Certificate for </span><span class="nv">$domain</span><span class="s2"> updated in-place&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Certificate not yet loaded, reload HAProxy</span>
</span></span><span class="line"><span class="cl">        systemctl reload haproxy
</span></span><span class="line"><span class="cl">        <span class="nb">echo</span> <span class="s2">&#34;HAProxy reloaded for new certificate </span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">fi</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Process the renewed certificate (certbot sets RENEWED_LINEAGE to its live directory)</span>
</span></span><span class="line"><span class="cl">update_certificate <span class="s2">&#34;</span><span class="k">$(</span>basename <span class="s2">&#34;</span><span class="nv">$RENEWED_LINEAGE</span><span class="s2">&#34;</span><span class="k">)</span><span class="s2">&#34;</span>
</span></span></code></pre></div><h3 id="haproxy-24-runtime-certificate-api">HAProxy 2.4+ Runtime Certificate API <a class="anchor" data-pagefind-ignore href="#haproxy-24-runtime-certificate-api" aria-label="Link to this section">#</a></h3>
<p>The same commands by hand, to inspect or update a certificate. The update only changes memory; the file on disk is what HAProxy loads at the next start.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># List certificates</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show ssl cert&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Show certificate details</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show ssl cert /etc/haproxy/certs/example.com.pem&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Update certificate in-place (HAProxy 2.4+): payload in the same session, then commit</span>
</span></span><span class="line"><span class="cl"><span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;set ssl cert /etc/haproxy/certs/example.com.pem &lt;&lt;&#34;</span><span class="p">;</span> cat /etc/haproxy/certs/example.com.pem<span class="p">;</span> <span class="nb">echo</span> <span class="s2">&#34;&#34;</span><span class="p">;</span> <span class="o">}</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;commit ssl cert /etc/haproxy/certs/example.com.pem&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span></code></pre></div><h2 id="method-4-dns-01-challenge-wildcard-certificates">Method 4: DNS-01 Challenge (Wildcard Certificates) <a class="anchor" data-pagefind-ignore href="#method-4-dns-01-challenge-wildcard-certificates" aria-label="Link to this section">#</a></h2>
<p>For wildcard certificates or when HTTP challenges aren&rsquo;t possible. Certbot proves control by creating a TXT record through your DNS provider&rsquo;s API; port 80 is not involved.</p>
<h3 id="with-cloudflare-dns">With Cloudflare DNS <a class="anchor" data-pagefind-ignore href="#with-cloudflare-dns" aria-label="Link to this section">#</a></h3>
<p>Use an API token limited to editing DNS for the zone, not your global API key.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Install Cloudflare DNS plugin</span>
</span></span><span class="line"><span class="cl">apt install python3-certbot-dns-cloudflare
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Or, if certbot itself was installed with pip in /opt/certbot</span>
</span></span><span class="line"><span class="cl"><span class="c1"># (Ubuntu 24.04 and Debian 12 refuse a system-wide pip install)</span>
</span></span><span class="line"><span class="cl">/opt/certbot/bin/pip install certbot-dns-cloudflare
</span></span></code></pre></div><p>Create credentials file (readable by root only):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># /etc/letsencrypt/cloudflare.ini</span>
</span></span><span class="line"><span class="cl"><span class="na">dns_cloudflare_api_token</span> <span class="o">=</span> <span class="s">your-api-token-here</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">chmod <span class="m">600</span> /etc/letsencrypt/cloudflare.ini
</span></span></code></pre></div><p>Request wildcard certificate:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">certbot certonly <span class="se">\
</span></span></span><span class="line"><span class="cl">  --dns-cloudflare <span class="se">\
</span></span></span><span class="line"><span class="cl">  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d example.com <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;*.example.com&#39;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  --email admin@example.com <span class="se">\
</span></span></span><span class="line"><span class="cl">  --agree-tos
</span></span></code></pre></div><h3 id="with-route53">With Route53 <a class="anchor" data-pagefind-ignore href="#with-route53" aria-label="Link to this section">#</a></h3>
<p>The plugin uses standard AWS credentials; an IAM role or <code>~/.aws/credentials</code> is better than exported keys.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">apt install python3-certbot-dns-route53
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Configure AWS credentials</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span>your-key
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span>your-secret
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">certbot certonly <span class="se">\
</span></span></span><span class="line"><span class="cl">  --dns-route53 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d example.com <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;*.example.com&#39;</span>
</span></span></code></pre></div><h2 id="automated-renewal-setup">Automated Renewal Setup <a class="anchor" data-pagefind-ignore href="#automated-renewal-setup" aria-label="Link to this section">#</a></h2>
<p>Certbot only renews certificates that are close to expiry, so running it twice a day is cheap. The <code>--deploy-hook</code> runs only when a certificate was actually renewed.</p>
<table>
	<thead>
			<tr>
					<th>Option</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Systemd timer</td>
					<td>Most systems; <code>Persistent=true</code> catches up on missed runs</td>
			</tr>
			<tr>
					<td>Cron</td>
					<td>Systems without systemd</td>
			</tr>
	</tbody>
</table>
<p>Packaged certbot often ships its own timer (<code>certbot.timer</code>) or cron entry. Use one scheduler, not both.</p>
<h3 id="systemd-timer">Systemd Timer <a class="anchor" data-pagefind-ignore href="#systemd-timer" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># /etc/systemd/system/certbot-renewal.service</span>
</span></span><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Certbot Renewal</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network-online.target</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/bin/certbot renew --deploy-hook /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># /etc/systemd/system/certbot-renewal.timer</span>
</span></span><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Run certbot renewal twice daily</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Timer]</span>
</span></span><span class="line"><span class="cl"><span class="na">OnCalendar</span><span class="o">=</span><span class="s">*-*-* 00,12:00:00</span>
</span></span><span class="line"><span class="cl"><span class="na">RandomizedDelaySec</span><span class="o">=</span><span class="s">43200</span>
</span></span><span class="line"><span class="cl"><span class="na">Persistent</span><span class="o">=</span><span class="s">true</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">timers.target</span>
</span></span></code></pre></div><p>Enable the timer:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">systemctl daemon-reload
</span></span><span class="line"><span class="cl">systemctl <span class="nb">enable</span> certbot-renewal.timer
</span></span><span class="line"><span class="cl">systemctl start certbot-renewal.timer
</span></span></code></pre></div><h3 id="cron-job-alternative">Cron Job Alternative <a class="anchor" data-pagefind-ignore href="#cron-job-alternative" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/cron.d/certbot</span>
</span></span><span class="line"><span class="cl"><span class="m">0</span> 0,12 * * * root certbot renew --deploy-hook /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh
</span></span></code></pre></div><h2 id="multi-domain-configuration">Multi-Domain Configuration <a class="anchor" data-pagefind-ignore href="#multi-domain-configuration" aria-label="Link to this section">#</a></h2>
<p>One HAProxy can serve many domains, each with its own certificate.</p>
<h3 id="separate-certificates-per-domain">Separate Certificates per Domain <a class="anchor" data-pagefind-ignore href="#separate-certificates-per-domain" aria-label="Link to this section">#</a></h3>
<p>List each PEM on the bind line and route on the SNI name.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend https_frontend
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem crt /etc/haproxy/certs/example.org.pem
    
    # Route based on SNI
    use_backend example_com_servers if { ssl_fc_sni example.com }
    use_backend example_org_servers if { ssl_fc_sni example.org }
</code></pre><h3 id="certificate-directory">Certificate Directory <a class="anchor" data-pagefind-ignore href="#certificate-directory" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend https_frontend
    # Load all .pem files from directory
    bind *:443 ssl crt /etc/haproxy/certs/
</code></pre><p>HAProxy automatically matches certificates based on SNI. New files are picked up at the next reload.</p>
<h3 id="request-multiple-certificates">Request Multiple Certificates <a class="anchor" data-pagefind-ignore href="#request-multiple-certificates" aria-label="Link to this section">#</a></h3>
<p>Each line in <code>DOMAINS</code> becomes one certificate; the first name is the certificate name and the file name in <code>/etc/letsencrypt/live/</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /usr/local/bin/request-certs.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">DOMAINS</span><span class="o">=(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;example.com www.example.com&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;api.example.com&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;app.example.com&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">for</span> domain_set in <span class="s2">&#34;</span><span class="si">${</span><span class="nv">DOMAINS</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    <span class="nv">primary</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="nv">$domain_set</span> <span class="p">|</span> awk <span class="s1">&#39;{print $1}&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    certbot certonly <span class="se">\
</span></span></span><span class="line"><span class="cl">        --webroot <span class="se">\
</span></span></span><span class="line"><span class="cl">        -w /var/www/letsencrypt <span class="se">\
</span></span></span><span class="line"><span class="cl">        <span class="k">$(for</span> d in <span class="nv">$domain_set</span><span class="p">;</span> <span class="k">do</span> <span class="nb">echo</span> <span class="s2">&#34;-d </span><span class="nv">$d</span><span class="s2">&#34;</span><span class="p">;</span> <span class="k">done)</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">        --cert-name <span class="nv">$primary</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">        --non-interactive
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span></code></pre></div><h2 id="complete-production-setup">Complete Production Setup <a class="anchor" data-pagefind-ignore href="#complete-production-setup" aria-label="Link to this section">#</a></h2>
<p>Method 3 with host routing and security headers.</p>
<h3 id="haproxy-configuration-2">HAProxy Configuration <a class="anchor" data-pagefind-ignore href="#haproxy-configuration-2" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Part</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>global</code> ssl settings</td>
					<td>TLS 1.2 minimum, modern ciphers, for client (<code>bind</code>) and backend (<code>server</code>) sides</td>
			</tr>
			<tr>
					<td><code>listen stats</code></td>
					<td>Stats page and <code>/metrics</code> on 8404; change <code>admin:changeme</code></td>
			</tr>
			<tr>
					<td><code>http_frontend</code></td>
					<td>ACME challenges to <code>acme_backend</code>, everything else 301 to HTTPS</td>
			</tr>
			<tr>
					<td><code>https_frontend</code></td>
					<td>All certificates from <code>/etc/haproxy/certs/</code>, security headers, <code>X-Forwarded-Proto</code>, routing by Host</td>
			</tr>
			<tr>
					<td><code>app_servers</code></td>
					<td>Sticky sessions with the <code>SERVERID</code> cookie</td>
			</tr>
	</tbody>
</table>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    maxconn 50000
    nbthread 4
    
    # Modern SSL settings
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
    tune.ssl.default-dh-param 2048

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    option http-server-close
    timeout connect 5s
    timeout client  50s
    timeout server  50s
    timeout http-request 10s
    timeout http-keep-alive 10s

# Stats
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:changeme
    http-request use-service prometheus-exporter if { path /metrics }

# HTTP Frontend
frontend http_frontend
    bind *:80
    
    # ACME challenges
    acl is_acme path_beg /.well-known/acme-challenge/
    use_backend acme_backend if is_acme
    
    # Redirect to HTTPS with 301
    http-request redirect scheme https code 301 unless is_acme

# HTTPS Frontend
frontend https_frontend
    bind *:443 ssl crt /etc/haproxy/certs/ strict-sni alpn h2,http/1.1
    
    # Security headers
    http-response set-header Strict-Transport-Security &#34;max-age=63072000; includeSubDomains; preload&#34;
    http-response set-header X-Content-Type-Options &#34;nosniff&#34;
    http-response set-header X-Frame-Options &#34;SAMEORIGIN&#34;
    http-response set-header X-XSS-Protection &#34;1; mode=block&#34;
    
    # Log SSL info
    http-request capture req.hdr(Host) len 50
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-SSL-Client-Verify %[ssl_c_verify]
    http-request set-header X-SSL-Client-DN %{+Q}[ssl_c_s_dn]
    
    # Route based on host
    use_backend api_servers if { hdr(host) -i api.example.com }
    use_backend app_servers if { hdr(host) -i app.example.com }
    default_backend web_servers

# ACME challenge backend
backend acme_backend
    server acme 127.0.0.1:8080

# Web servers
backend web_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server web1 192.168.1.10:8080 check inter 5s
    server web2 192.168.1.11:8080 check inter 5s

# API servers
backend api_servers
    balance leastconn
    option httpchk GET /api/health
    http-check expect status 200
    server api1 192.168.1.20:8080 check inter 5s
    server api2 192.168.1.21:8080 check inter 5s

# App servers
backend app_servers
    balance roundrobin
    option httpchk GET /health
    cookie SERVERID insert indirect nocache
    server app1 192.168.1.30:8080 check cookie app1
    server app2 192.168.1.31:8080 check cookie app2
</code></pre><h3 id="complete-deployment-script">Complete Deployment Script <a class="anchor" data-pagefind-ignore href="#complete-deployment-script" aria-label="Link to this section">#</a></h3>
<p>Sets up the challenge server as a systemd service on port 8080 and requests a certificate for each domain given as argument:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /usr/local/bin/setup-haproxy-letsencrypt.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -e
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Run this with HAProxy already up (it needs at least one certificate in</span>
</span></span><span class="line"><span class="cl"><span class="c1"># /etc/haproxy/certs, see the initial request above) and the deploy hook below in place.</span>
</span></span><span class="line"><span class="cl"><span class="nv">DOMAINS</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">EMAIL</span><span class="o">=</span><span class="s2">&#34;admin@example.com&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_CERT_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/haproxy/certs&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">WEBROOT</span><span class="o">=</span><span class="s2">&#34;/var/www/letsencrypt&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create directories</span>
</span></span><span class="line"><span class="cl">mkdir -p <span class="nv">$HAPROXY_CERT_DIR</span>
</span></span><span class="line"><span class="cl">mkdir -p <span class="nv">$WEBROOT</span>/.well-known/acme-challenge
</span></span><span class="line"><span class="cl">chown -R www-data:www-data <span class="nv">$WEBROOT</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create ACME challenge server</span>
</span></span><span class="line"><span class="cl">cat &gt; /etc/systemd/system/acme-challenge.service <span class="s">&lt;&lt; &#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">[Unit]
</span></span></span><span class="line"><span class="cl"><span class="s">Description=ACME Challenge Server
</span></span></span><span class="line"><span class="cl"><span class="s">After=network.target
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[Service]
</span></span></span><span class="line"><span class="cl"><span class="s">Type=simple
</span></span></span><span class="line"><span class="cl"><span class="s">User=www-data
</span></span></span><span class="line"><span class="cl"><span class="s">WorkingDirectory=/var/www/letsencrypt
</span></span></span><span class="line"><span class="cl"><span class="s">ExecStart=/usr/bin/python3 -m http.server 8080
</span></span></span><span class="line"><span class="cl"><span class="s">Restart=always
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[Install]
</span></span></span><span class="line"><span class="cl"><span class="s">WantedBy=multi-user.target
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">systemctl daemon-reload
</span></span><span class="line"><span class="cl">systemctl <span class="nb">enable</span> acme-challenge
</span></span><span class="line"><span class="cl">systemctl start acme-challenge
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Request certificates</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> domain in <span class="nv">$DOMAINS</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;Requesting certificate for </span><span class="nv">$domain</span><span class="s2">...&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    certbot certonly <span class="se">\
</span></span></span><span class="line"><span class="cl">        --webroot <span class="se">\
</span></span></span><span class="line"><span class="cl">        -w <span class="nv">$WEBROOT</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">        -d <span class="nv">$domain</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">        --email <span class="nv">$EMAIL</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">        --agree-tos <span class="se">\
</span></span></span><span class="line"><span class="cl">        --non-interactive <span class="se">\
</span></span></span><span class="line"><span class="cl">        --deploy-hook /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Create HAProxy certificate</span>
</span></span><span class="line"><span class="cl">    cat /etc/letsencrypt/live/<span class="nv">$domain</span>/fullchain.pem <span class="se">\
</span></span></span><span class="line"><span class="cl">        /etc/letsencrypt/live/<span class="nv">$domain</span>/privkey.pem <span class="se">\
</span></span></span><span class="line"><span class="cl">        &gt; <span class="nv">$HAPROXY_CERT_DIR</span>/<span class="nv">$domain</span>.pem
</span></span><span class="line"><span class="cl">    chmod <span class="m">600</span> <span class="nv">$HAPROXY_CERT_DIR</span>/<span class="nv">$domain</span>.pem
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Reload HAProxy</span>
</span></span><span class="line"><span class="cl">systemctl reload haproxy
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Setup complete!&#34;</span>
</span></span></code></pre></div><h3 id="deploy-hook-script">Deploy Hook Script <a class="anchor" data-pagefind-ignore href="#deploy-hook-script" aria-label="Link to this section">#</a></h3>
<p>The production version of the hook: runtime update first, reload as fallback, everything logged to <code>/var/log/certbot-haproxy.log</code>. It uses <code>RENEWED_LINEAGE</code>, the live directory of the renewed certificate.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -e
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_CERT_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/haproxy/certs&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_SOCKET</span><span class="o">=</span><span class="s2">&#34;/run/haproxy/admin.sock&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">LOG_FILE</span><span class="o">=</span><span class="s2">&#34;/var/log/certbot-haproxy.log&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">log<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;</span><span class="k">$(</span>date <span class="s1">&#39;+%Y-%m-%d %H:%M:%S&#39;</span><span class="k">)</span><span class="s2"> - </span><span class="nv">$1</span><span class="s2">&#34;</span> &gt;&gt; <span class="nv">$LOG_FILE</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">update_haproxy_cert<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">    <span class="nb">local</span> <span class="nv">domain</span><span class="o">=</span><span class="nv">$1</span>
</span></span><span class="line"><span class="cl">    <span class="nb">local</span> <span class="nv">cert_path</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$HAPROXY_CERT_DIR</span><span class="s2">/</span><span class="nv">$domain</span><span class="s2">.pem&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="nb">local</span> <span class="nv">le_cert</span><span class="o">=</span><span class="s2">&#34;/etc/letsencrypt/live/</span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Combine certificates</span>
</span></span><span class="line"><span class="cl">    cat <span class="s2">&#34;</span><span class="nv">$le_cert</span><span class="s2">/fullchain.pem&#34;</span> <span class="s2">&#34;</span><span class="nv">$le_cert</span><span class="s2">/privkey.pem&#34;</span> &gt; <span class="s2">&#34;</span><span class="nv">$cert_path</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    chmod <span class="m">600</span> <span class="s2">&#34;</span><span class="nv">$cert_path</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Try runtime update first (HAProxy 2.4+)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="o">[</span> -S <span class="s2">&#34;</span><span class="nv">$HAPROXY_SOCKET</span><span class="s2">&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="nb">echo</span> <span class="s2">&#34;show ssl cert </span><span class="nv">$cert_path</span><span class="s2">&#34;</span> <span class="p">|</span> socat stdio <span class="s2">&#34;</span><span class="nv">$HAPROXY_SOCKET</span><span class="s2">&#34;</span> 2&gt;/dev/null <span class="p">|</span> grep -q <span class="s2">&#34;</span><span class="nv">$cert_path</span><span class="s2">&#34;</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># Certificate exists, update in-place: payload first, commit in a</span>
</span></span><span class="line"><span class="cl">            <span class="c1"># separate session (HAProxy 3.2 and later ignore a commit sent after the payload)</span>
</span></span><span class="line"><span class="cl">            <span class="o">{</span>
</span></span><span class="line"><span class="cl">                <span class="nb">echo</span> <span class="s2">&#34;set ssl cert </span><span class="nv">$cert_path</span><span class="s2"> &lt;&lt;&#34;</span>
</span></span><span class="line"><span class="cl">                cat <span class="s2">&#34;</span><span class="nv">$cert_path</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">                <span class="nb">echo</span> <span class="s2">&#34;&#34;</span>
</span></span><span class="line"><span class="cl">            <span class="o">}</span> <span class="p">|</span> socat stdio <span class="s2">&#34;</span><span class="nv">$HAPROXY_SOCKET</span><span class="s2">&#34;</span> 2&gt;/dev/null
</span></span><span class="line"><span class="cl">            
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="nb">echo</span> <span class="s2">&#34;commit ssl cert </span><span class="nv">$cert_path</span><span class="s2">&#34;</span> <span class="p">|</span> socat stdio <span class="s2">&#34;</span><span class="nv">$HAPROXY_SOCKET</span><span class="s2">&#34;</span> 2&gt;/dev/null <span class="p">|</span> grep -q <span class="s2">&#34;Success&#34;</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">                log <span class="s2">&#34;Updated certificate for </span><span class="nv">$domain</span><span class="s2"> via runtime API&#34;</span>
</span></span><span class="line"><span class="cl">                <span class="k">return</span> <span class="m">0</span>
</span></span><span class="line"><span class="cl">            <span class="k">fi</span>
</span></span><span class="line"><span class="cl">        <span class="k">fi</span>
</span></span><span class="line"><span class="cl">    <span class="k">fi</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Fallback to reload</span>
</span></span><span class="line"><span class="cl">    systemctl reload haproxy
</span></span><span class="line"><span class="cl">    log <span class="s2">&#34;Reloaded HAProxy for </span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Process the renewed certificate (RENEWED_LINEAGE is /etc/letsencrypt/live/&lt;name&gt;;</span>
</span></span><span class="line"><span class="cl"><span class="c1"># RENEWED_DOMAINS lists every name on it, which may not have a live directory of its own)</span>
</span></span><span class="line"><span class="cl"><span class="nv">domain</span><span class="o">=</span><span class="k">$(</span>basename <span class="s2">&#34;</span><span class="nv">$RENEWED_LINEAGE</span><span class="s2">&#34;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">log <span class="s2">&#34;Processing renewal for </span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">update_haproxy_cert <span class="s2">&#34;</span><span class="nv">$domain</span><span class="s2">&#34;</span>
</span></span></code></pre></div><h2 id="testing-and-verification">Testing and Verification <a class="anchor" data-pagefind-ignore href="#testing-and-verification" aria-label="Link to this section">#</a></h2>
<p>Check what clients actually get, then test the renewal path before it runs for real.</p>
<h3 id="test-certificate">Test Certificate <a class="anchor" data-pagefind-ignore href="#test-certificate" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check certificate</span>
</span></span><span class="line"><span class="cl">openssl s_client -connect example.com:443 -servername example.com &lt; /dev/null 2&gt;/dev/null <span class="p">|</span> openssl x509 -text -noout <span class="p">|</span> head -20
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Verify chain (s_client checks the served chain against the system trust store)</span>
</span></span><span class="line"><span class="cl">openssl s_client -connect example.com:443 -servername example.com -verify_return_error &lt; /dev/null 2&gt;<span class="p">&amp;</span><span class="m">1</span> <span class="p">|</span> grep <span class="s2">&#34;Verify return code&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check expiration</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="p">|</span> openssl s_client -connect example.com:443 -servername example.com 2&gt;/dev/null <span class="p">|</span> openssl x509 -noout -dates
</span></span></code></pre></div><h3 id="ssl-labs-test">SSL Labs Test <a class="anchor" data-pagefind-ignore href="#ssl-labs-test" aria-label="Link to this section">#</a></h3>
<p>A full external grade of your TLS configuration.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -s <span class="s2">&#34;https://api.ssllabs.com/api/v3/analyze?host=example.com&amp;publish=off&amp;all=done&#34;</span>
</span></span></code></pre></div><h3 id="test-renewal">Test Renewal <a class="anchor" data-pagefind-ignore href="#test-renewal" aria-label="Link to this section">#</a></h3>
<p>Runs the whole renewal against the staging server without replacing certificates.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">certbot renew --dry-run
</span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<ol>
<li><strong>Permission denied on socket</strong>:</li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">chmod <span class="m">660</span> /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">chown haproxy:haproxy /run/haproxy/admin.sock
</span></span></code></pre></div><ol start="2">
<li><strong>Certificate not found</strong>:</li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check certificate directory</span>
</span></span><span class="line"><span class="cl">ls -la /etc/haproxy/certs/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Verify HAProxy can load them (it reads certificates as root before</span>
</span></span><span class="line"><span class="cl"><span class="c1"># dropping privileges, so root-only 600 files are fine)</span>
</span></span><span class="line"><span class="cl">haproxy -c -f /etc/haproxy/haproxy.cfg
</span></span></code></pre></div><ol start="3">
<li><strong>ACME challenge fails</strong>:</li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Verify challenge server is running</span>
</span></span><span class="line"><span class="cl">curl http://localhost:8080/.well-known/acme-challenge/test
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check HAProxy routing</span>
</span></span><span class="line"><span class="cl">curl -H <span class="s2">&#34;Host: example.com&#34;</span> http://localhost/.well-known/acme-challenge/test
</span></span></code></pre></div><ol start="4">
<li><strong>SNI not matching</strong>:</li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Test specific SNI</span>
</span></span><span class="line"><span class="cl">curl -v --resolve example.com:443:127.0.0.1 https://example.com/
</span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Private keys.</strong> The combined PEM files contain the key: keep them mode 600, as the scripts do.</li>
<li><strong>DNS API credentials.</strong> <code>cloudflare.ini</code> or AWS keys can change your DNS; restrict their scope and file permissions.</li>
<li><strong>Stats page.</strong> Change <code>stats auth admin:changeme</code> and firewall port 8404.</li>
<li><strong>HSTS preload.</strong> Once preloaded, browsers refuse HTTP for the domain and its subdomains for a long time; enable it deliberately.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-letsencrypt" rel="noopener">pderuiter/blog-haproxy-letsencrypt</a>. It creates an HAProxy VM on Proxmox, requests the certificates with certbot over HTTP-01 and installs the complete production setup from this post, with the runtime API deploy hook and the renewal timer; point your domains at the VM, run <code>terraform apply</code>, then <code>ansible-playbook site.yml</code> (try it against the Let&rsquo;s Encrypt staging server first).</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>Next in the series: debugging HTTP headers in HAProxy, for troubleshooting web application issues.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<p>Key takeaways for HAProxy with Let&rsquo;s Encrypt:</p>
<ul>
<li>Combine fullchain.pem and privkey.pem for HAProxy certificates</li>
<li>Use HTTP-01 challenges for standard domains, DNS-01 for wildcards</li>
<li>Implement zero-downtime certificate updates with HAProxy&rsquo;s runtime API</li>
<li>Set up automated renewal with systemd timers or cron</li>
<li>Use proper TLS settings for security (TLS 1.2+, strong ciphers)</li>
<li>Always enable HSTS for production sites</li>
</ul>
<p>With this setup, certificates renew and load themselves without manual work.</p>
]]></content:encoded></item><item><title>HAProxy HTTP Headers Debugging: A Complete Troubleshooting Guide</title><link>https://blog.bsdserver.nl/haproxy-http-headers-debugging-a-complete-troubleshooting-guide/</link><guid isPermaLink="false">693b73ab79eb54000160a5fe</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:45:15 +0000</pubDate><description>When applications misbehave behind a load balancer, HTTP headers are often the culprit.</description><category>HAProxy</category><category>HTTP</category><category>Debugging</category><category>HTTP headers</category><category>Troubleshooting</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-http-headers-debugging.png" medium="image"/><content:encoded><![CDATA[<p>When applications misbehave behind a load balancer, HTTP headers are often the culprit. Redirect loops, wrong client IPs in logs, lost sessions and blocked browser requests usually trace back to a header that is missing, wrong or doubled.</p>
<p>This guide shows how to see the headers HAProxy receives and sends, and how to set, remove and route on them. It ends with a complete debug configuration you can drop on a test load balancer.</p>
<h2 id="why-header-debugging-matters">Why Header Debugging Matters <a class="anchor" data-pagefind-ignore href="#why-header-debugging-matters" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Header</th>
					<th>Carries</th>
					<th>Typical symptom when wrong</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>X-Forwarded-For</code></td>
					<td>Client IP</td>
					<td>The app logs the load balancer&rsquo;s IP</td>
			</tr>
			<tr>
					<td><code>X-Forwarded-Proto</code></td>
					<td><code>http</code> or <code>https</code></td>
					<td>Redirect loops, <code>http://</code> links behind TLS</td>
			</tr>
			<tr>
					<td><code>Host</code></td>
					<td>The requested site</td>
					<td>Wrong virtual host or backend</td>
			</tr>
			<tr>
					<td><code>Cookie</code> / <code>Set-Cookie</code></td>
					<td>Sessions, persistence</td>
					<td>Users logged out, sticky sessions lost</td>
			</tr>
			<tr>
					<td><code>Cache-Control</code>, <code>ETag</code></td>
					<td>Caching</td>
					<td>Stale or uncached content</td>
			</tr>
			<tr>
					<td>CORS, CSP, HSTS</td>
					<td>Browser security policies</td>
					<td>The browser blocks requests</td>
			</tr>
			<tr>
					<td><code>Authorization</code></td>
					<td>Credentials</td>
					<td>Unexpected 401s</td>
			</tr>
	</tbody>
</table>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> while troubleshooting, and in staging: captures, debug endpoints and <code>haproxy -d</code> show exactly what passes through.</li>
<li><strong>Skip it</strong> in production beyond a few captured headers. Debug endpoints and full header logs expose cookies and tokens.</li>
</ul>
<h2 id="how-it-works">How it works <a class="anchor" data-pagefind-ignore href="#how-it-works" aria-label="Link to this section">#</a></h2>
<p>HAProxy processes headers at fixed points. Knowing which rule runs where explains most &ldquo;my header is not there&rdquo; problems.</p>
<table>
	<thead>
			<tr>
					<th>Directive</th>
					<th>Runs on</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>capture request/response header</code></td>
					<td>Requests and responses, frontend only</td>
					<td>Writing chosen headers to the log</td>
			</tr>
			<tr>
					<td><code>http-request</code></td>
					<td>Requests, before they go to a backend</td>
					<td>Setting, removing and routing on request headers; returning answers directly</td>
			</tr>
			<tr>
					<td><code>http-response</code></td>
					<td>Responses from the servers</td>
					<td>Security headers, removing <code>Server</code></td>
			</tr>
			<tr>
					<td><code>http-after-response</code></td>
					<td>Every response, also HAProxy&rsquo;s own (<code>http-request return</code>, errors)</td>
					<td>Headers that must be on every answer</td>
			</tr>
	</tbody>
</table>
<p>The flow: client → frontend (<code>capture</code>, <code>http-request</code>) → backend → server → <code>http-response</code> → <code>http-after-response</code> → client.</p>
<p>The debugging tools in this post:</p>
<table>
	<thead>
			<tr>
					<th>Tool</th>
					<th>Shows</th>
					<th>Use it when</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Header captures in the log</td>
					<td>Chosen headers per request</td>
					<td>Always on, also in production</td>
			</tr>
			<tr>
					<td><code>haproxy -d</code></td>
					<td>All traffic, live</td>
					<td>In a lab only: the service is stopped</td>
			</tr>
			<tr>
					<td>Debug backend (netcat, Python)</td>
					<td>Headers as the server receives them</td>
					<td>Checking what HAProxy adds or changes</td>
			</tr>
			<tr>
					<td><code>http-request return</code> endpoint</td>
					<td>Headers as HAProxy sees them</td>
					<td>A quick check with curl</td>
			</tr>
			<tr>
					<td>Lua service</td>
					<td>Every request header</td>
					<td>You do not know which header matters</td>
			</tr>
	</tbody>
</table>
<h2 id="logging-headers">Logging Headers <a class="anchor" data-pagefind-ignore href="#logging-headers" aria-label="Link to this section">#</a></h2>
<h3 id="basic-http-logging">Basic HTTP Logging <a class="anchor" data-pagefind-ignore href="#basic-http-logging" aria-label="Link to this section">#</a></h3>
<p><code>option httplog</code> turns on the HTTP log format: one line per request with timers, status and bytes.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">defaults
    mode http
    option httplog
    log global
</code></pre><h3 id="detailed-custom-logging">Detailed Custom Logging <a class="anchor" data-pagefind-ignore href="#detailed-custom-logging" aria-label="Link to this section">#</a></h3>
<p><code>capture request header</code> and <code>capture response header</code> add chosen headers to each log line, cut at <code>len</code> characters. Captures only work in a frontend.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Capture specific headers for logging
    capture request header Host len 50
    capture request header User-Agent len 100
    capture request header X-Forwarded-For len 50
    capture request header Authorization len 100
    capture request header Content-Type len 50
    
    capture response header Content-Type len 50
    capture response header Location len 200
    capture response header Set-Cookie len 100
    
    default_backend web_servers
</code></pre><h3 id="custom-log-format-with-headers">Custom Log Format with Headers <a class="anchor" data-pagefind-ignore href="#custom-log-format-with-headers" aria-label="Link to this section">#</a></h3>
<p>A custom <code>log-format</code> decides where the captured headers appear in the line.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Define captures
    capture request header Host len 50
    capture request header X-Request-ID len 50
    capture response header X-Response-Time len 20
    
    # Custom log format
    log-format &#34;%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r&#34;
    
    default_backend web_servers
</code></pre><p>Log format variables:</p>
<ul>
<li><code>%hr</code>: Captured request headers</li>
<li><code>%hs</code>: Captured response headers</li>
<li><code>%{+Q}r</code>: Quoted HTTP request line</li>
<li><code>%ST</code>: HTTP status code</li>
<li><code>%TR</code>: Request receive time</li>
<li><code>%Ta</code>: Total active time</li>
</ul>
<h3 id="logging-all-headers">Logging All Headers <a class="anchor" data-pagefind-ignore href="#logging-all-headers" aria-label="Link to this section">#</a></h3>
<p><code>req.hdrs</code> is the complete request header block. Capture it on a separate debug frontend, not on production traffic: lines get long and include cookies and credentials.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend debug_frontend
    bind *:8080
    mode http
    
    # Log all request headers
    http-request capture req.hdrs len 1024
    
    # Log format with all headers
    log-format &#34;%ci [%t] %r headers:{%[capture.req.hdr(0)]}&#34;
    
    default_backend web_servers
</code></pre><h2 id="inspecting-headers-in-real-time">Inspecting Headers in Real-Time <a class="anchor" data-pagefind-ignore href="#inspecting-headers-in-real-time" aria-label="Link to this section">#</a></h2>
<h3 id="using-the-stats-socket">Using the Stats Socket <a class="anchor" data-pagefind-ignore href="#using-the-stats-socket" aria-label="Link to this section">#</a></h3>
<p><code>show sess</code> lists the sessions HAProxy holds right now. Use it for hanging or slow requests.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Show current sessions with header info</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show sess&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Show detailed session info</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show sess &lt;session_id&gt;&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span></code></pre></div><h3 id="debug-mode">Debug Mode <a class="anchor" data-pagefind-ignore href="#debug-mode" aria-label="Link to this section">#</a></h3>
<p>Run HAProxy in debug mode to see all traffic. It runs in the foreground, so the service has to stop first: use this in a lab only.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Stop HAProxy service</span>
</span></span><span class="line"><span class="cl">systemctl stop haproxy
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Run in foreground with debug</span>
</span></span><span class="line"><span class="cl">haproxy -d -f /etc/haproxy/haproxy.cfg
</span></span></code></pre></div><p>This outputs all headers to stdout.</p>
<h3 id="divert-traffic-for-debugging">Divert Traffic for Debugging <a class="anchor" data-pagefind-ignore href="#divert-traffic-for-debugging" aria-label="Link to this section">#</a></h3>
<p>Send part of the traffic to a small debug server that prints what it receives. That shows the request exactly as a backend gets it, after all HAProxy rules.</p>
<ul>
<li><code>use_backend debug_backend if { path_beg /api }</code>: only <code>/api</code> goes to the debug server.</li>
<li><code>X-Mirror true</code>: a marker header, so you can see it arrive.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Send /api traffic to a debug backend instead of the real servers
    # (this diverts the requests; HAProxy does not copy them)
    http-request set-header X-Mirror true
    use_backend debug_backend if { path_beg /api }
    
    default_backend web_servers

backend debug_backend
    server debug 127.0.0.1:9999
</code></pre><p>Run a simple debug server on port 9999. netcat prints the raw request; the Python version prints path and headers and also answers <code>POST</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Using netcat (prints the request, answers with a fixed 200 OK)</span>
</span></span><span class="line"><span class="cl"><span class="k">while</span> true<span class="p">;</span> <span class="k">do</span> <span class="nb">printf</span> <span class="s1">&#39;HTTP/1.1 200 OK\r\nContent-Length: 3\r\nConnection: close\r\n\r\nOK\n&#39;</span> <span class="p">|</span> nc -l -p <span class="m">9999</span> -q 1<span class="p">;</span> <span class="k">done</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Using Python</span>
</span></span><span class="line"><span class="cl">python3 -c <span class="s2">&#34;
</span></span></span><span class="line"><span class="cl"><span class="s2">import http.server
</span></span></span><span class="line"><span class="cl"><span class="s2">class Handler(http.server.BaseHTTPRequestHandler):
</span></span></span><span class="line"><span class="cl"><span class="s2">    def do_GET(self):
</span></span></span><span class="line"><span class="cl"><span class="s2">        print(f&#39;Path: {self.path}&#39;)
</span></span></span><span class="line"><span class="cl"><span class="s2">        print(&#39;Headers:&#39;)
</span></span></span><span class="line"><span class="cl"><span class="s2">        for h, v in self.headers.items():
</span></span></span><span class="line"><span class="cl"><span class="s2">            print(f&#39;  {h}: {v}&#39;)
</span></span></span><span class="line"><span class="cl"><span class="s2">        self.send_response(200)
</span></span></span><span class="line"><span class="cl"><span class="s2">        self.end_headers()
</span></span></span><span class="line"><span class="cl"><span class="s2">        self.wfile.write(b&#39;OK&#39;)
</span></span></span><span class="line"><span class="cl"><span class="s2">    do_POST = do_GET
</span></span></span><span class="line"><span class="cl"><span class="s2">http.server.HTTPServer((&#39;&#39;, 9999), Handler).serve_forever()
</span></span></span><span class="line"><span class="cl"><span class="s2">&#34;</span>
</span></span></code></pre></div><h2 id="adding-and-modifying-request-headers">Adding and Modifying Request Headers <a class="anchor" data-pagefind-ignore href="#adding-and-modifying-request-headers" aria-label="Link to this section">#</a></h2>
<p><code>set-header</code> replaces any existing header with that name. The value can be fixed text or a sample in <code>%[...]</code>.</p>
<h3 id="setting-headers">Setting Headers <a class="anchor" data-pagefind-ignore href="#setting-headers" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Add static header
    http-request set-header X-Forwarded-Proto http
    
    # Add header from variable
    http-request set-header X-Client-IP %[src]
    
    # Add header with the name of the HAProxy node
    http-request set-header X-Proxy-Node %[hostname]
    
    # Add timestamp
    http-request set-header X-Request-Start %[date()]
    
    default_backend web_servers
</code></pre><h3 id="common-forwarding-headers">Common Forwarding Headers <a class="anchor" data-pagefind-ignore href="#common-forwarding-headers" aria-label="Link to this section">#</a></h3>
<p>These tell the server who the client is and how it connected. The server only sees HAProxy&rsquo;s address otherwise.</p>
<ul>
<li><code>X-Forwarded-For</code>: overwritten with the client IP. The commented lines append instead, for when another proxy sits in front of HAProxy.</li>
<li><code>X-Forwarded-Proto</code>: fixed per frontend, <code>http</code> on port 80 and <code>https</code> on 443. Apps use it to build correct links and redirects.</li>
<li><code>X-Forwarded-Host</code> and <code>X-Forwarded-Port</code>: the original host and port, for apps that build absolute URLs.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # X-Forwarded-For (client IP)
    http-request set-header X-Forwarded-For %[src]
    
    # Or append to existing (use instead of the line above)
    # http-request set-header X-Forwarded-For %[req.hdr(X-Forwarded-For)],%[src] if { req.hdr(X-Forwarded-For) -m found }
    # http-request set-header X-Forwarded-For %[src] unless { req.hdr(X-Forwarded-For) -m found }
    
    # X-Real-IP
    http-request set-header X-Real-IP %[src]
    
    # X-Forwarded-Proto
    http-request set-header X-Forwarded-Proto http
    
    # X-Forwarded-Host
    http-request set-header X-Forwarded-Host %[req.hdr(Host)]
    
    # X-Forwarded-Port
    http-request set-header X-Forwarded-Port %[dst_port]
    
    default_backend web_servers

frontend https_frontend
    bind *:443 ssl crt /etc/haproxy/certs/
    
    http-request set-header X-Forwarded-For %[src]
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Host %[req.hdr(Host)]
    http-request set-header X-Forwarded-Port 443
    
    default_backend web_servers
</code></pre><h3 id="adding-request-id-for-tracing">Adding Request ID for Tracing <a class="anchor" data-pagefind-ignore href="#adding-request-id-for-tracing" aria-label="Link to this section">#</a></h3>
<p>A request ID follows one request through HAProxy, the server and its logs.</p>
<ul>
<li><code>unique-id-format</code>: builds the ID from client and frontend address and port, a timestamp, a request counter and the process ID, in hex (<code>%{+X}o</code>).</li>
<li><code>unique-id-header X-Request-ID</code>: sends it to the server.</li>
<li>The <code>uuid()</code> alternative keeps an ID the client or an upstream proxy already set.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Generate unique request ID (unique-id-header adds it to every request)
    unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
    unique-id-header X-Request-ID
    
    # Or use UUID format (instead of the two lines above, which always add a header)
    # http-request set-header X-Request-ID %[uuid()] unless { req.hdr(X-Request-ID) -m found }
    
    default_backend web_servers
</code></pre><h3 id="conditional-header-setting">Conditional Header Setting <a class="anchor" data-pagefind-ignore href="#conditional-header-setting" aria-label="Link to this section">#</a></h3>
<p>Add <code>if</code> or <code>unless</code> with an ACL to set a header only for some requests: by path, client network or user agent.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Set header based on path
    http-request set-header X-API-Version v2 if { path_beg /api/v2 }
    http-request set-header X-API-Version v1 if { path_beg /api/v1 }
    
    # Set header based on client IP
    acl internal_net src 192.168.0.0/16 10.0.0.0/8
    http-request set-header X-Internal-Request true if internal_net
    
    # Set header based on user agent
    acl is_mobile req.hdr(User-Agent) -m sub -i mobile android iphone
    http-request set-header X-Device-Type mobile if is_mobile
    http-request set-header X-Device-Type desktop unless is_mobile
    
    default_backend web_servers
</code></pre><h2 id="modifying-response-headers">Modifying Response Headers <a class="anchor" data-pagefind-ignore href="#modifying-response-headers" aria-label="Link to this section">#</a></h2>
<h3 id="adding-security-headers">Adding Security Headers <a class="anchor" data-pagefind-ignore href="#adding-security-headers" aria-label="Link to this section">#</a></h3>
<p>Set these once on the load balancer instead of in every application.</p>
<table>
	<thead>
			<tr>
					<th>Header</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>Strict-Transport-Security</code></td>
					<td>Browsers use HTTPS only, here for two years and all subdomains</td>
			</tr>
			<tr>
					<td><code>Content-Security-Policy</code></td>
					<td>Where scripts, styles and other content may come from</td>
			</tr>
			<tr>
					<td><code>X-Content-Type-Options nosniff</code></td>
					<td>Browsers trust the <code>Content-Type</code> instead of guessing</td>
			</tr>
			<tr>
					<td><code>X-Frame-Options SAMEORIGIN</code></td>
					<td>Only your own site may show the page in a frame</td>
			</tr>
			<tr>
					<td><code>X-XSS-Protection</code></td>
					<td>For old browsers only; current browsers ignore it</td>
			</tr>
			<tr>
					<td><code>Referrer-Policy</code></td>
					<td>How much of the URL goes to other sites</td>
			</tr>
			<tr>
					<td><code>Permissions-Policy</code></td>
					<td>Turns off browser features such as camera and location</td>
			</tr>
	</tbody>
</table>
<p>Test the CSP with your application before you enable it: a policy that is too strict breaks pages.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:443 ssl crt /etc/haproxy/certs/
    
    # HSTS
    http-response set-header Strict-Transport-Security &#34;max-age=63072000; includeSubDomains; preload&#34;
    
    # Content Security Policy
    http-response set-header Content-Security-Policy &#34;default-src &#39;self&#39;; script-src &#39;self&#39; &#39;unsafe-inline&#39;; style-src &#39;self&#39; &#39;unsafe-inline&#39;&#34;
    
    # X-Content-Type-Options
    http-response set-header X-Content-Type-Options nosniff
    
    # X-Frame-Options
    http-response set-header X-Frame-Options SAMEORIGIN
    
    # X-XSS-Protection
    http-response set-header X-XSS-Protection &#34;1; mode=block&#34;
    
    # Referrer Policy
    http-response set-header Referrer-Policy strict-origin-when-cross-origin
    
    # Permissions Policy
    http-response set-header Permissions-Policy &#34;geolocation=(), microphone=(), camera=()&#34;
    
    default_backend web_servers
</code></pre><h3 id="cors-headers">CORS Headers <a class="anchor" data-pagefind-ignore href="#cors-headers" aria-label="Link to this section">#</a></h3>
<p>Browsers send a preflight <code>OPTIONS</code> request before cross-origin API calls. HAProxy answers it directly with a 204 and adds the CORS headers.</p>
<ul>
<li><code>set-var(txn.cors_origin)</code>: saves <code>Origin</code>, because response rules cannot read request headers.</li>
<li><code>http-after-response</code>: also adds the headers to HAProxy&rsquo;s own 204, which <code>http-response</code> would skip.</li>
<li><code>Access-Control-Allow-Origin &quot;*&quot;</code>: any site may call the API. Use the next example to allow only some.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Handle CORS preflight
    # (store Origin in a variable: request headers are not available in response rules)
    http-request set-var(txn.cors_origin) req.hdr(Origin)
    acl is_options method OPTIONS
    acl is_cors_request var(txn.cors_origin) -m found
    
    # Return 204 for preflight
    http-request return status 204 if is_options is_cors_request
    
    # Add CORS headers to responses (http-after-response also covers the 204 above)
    http-after-response set-header Access-Control-Allow-Origin &#34;*&#34; if is_cors_request
    http-after-response set-header Access-Control-Allow-Methods &#34;GET, POST, PUT, DELETE, OPTIONS&#34; if is_cors_request
    http-after-response set-header Access-Control-Allow-Headers &#34;Content-Type, Authorization, X-Requested-With&#34; if is_cors_request
    http-after-response set-header Access-Control-Max-Age &#34;86400&#34; if is_cors_request
    
    default_backend web_servers
</code></pre><h3 id="conditional-cors-by-origin">Conditional CORS by Origin <a class="anchor" data-pagefind-ignore href="#conditional-cors-by-origin" aria-label="Link to this section">#</a></h3>
<p>Only listed origins get the CORS headers, with their own origin echoed back. <code>Allow-Credentials true</code> lets the browser send cookies, which is not allowed together with <code>*</code>.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Define allowed origins (stored in a variable so response rules can use it)
    http-request set-var(txn.origin) req.hdr(Origin)
    acl allowed_origin var(txn.origin) -m str -i https://example.com https://app.example.com
    
    # Echo back the allowed origin
    http-response set-header Access-Control-Allow-Origin %[var(txn.origin)] if allowed_origin
    http-response set-header Access-Control-Allow-Credentials true if allowed_origin
    
    default_backend web_servers
</code></pre><h2 id="deleting-headers">Deleting Headers <a class="anchor" data-pagefind-ignore href="#deleting-headers" aria-label="Link to this section">#</a></h2>
<h3 id="remove-sensitive-headers">Remove Sensitive Headers <a class="anchor" data-pagefind-ignore href="#remove-sensitive-headers" aria-label="Link to this section">#</a></h3>
<p>Clients can send <code>X-Forwarded-For</code> and friends themselves. Delete them first, then set your own trusted values, so the server never trusts a forged client IP.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Remove potentially dangerous headers from requests
    http-request del-header X-Forwarded-For
    http-request del-header X-Real-IP
    http-request del-header X-Forwarded-Proto
    http-request del-header X-Forwarded-Host
    
    # Now add our own trusted values
    http-request set-header X-Forwarded-For %[src]
    http-request set-header X-Forwarded-Proto http
    
    default_backend web_servers
</code></pre><h3 id="remove-headers-from-responses">Remove Headers from Responses <a class="anchor" data-pagefind-ignore href="#remove-headers-from-responses" aria-label="Link to this section">#</a></h3>
<p>Version headers tell attackers which software to target; debug headers leak internals. Remove them in the backend.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend web_servers
    # Remove server version headers
    http-response del-header Server
    http-response del-header X-Powered-By
    http-response del-header X-AspNet-Version
    http-response del-header X-AspNetMvc-Version
    
    # Remove debug headers
    http-response del-header X-Debug-Token
    http-response del-header X-Debug-Token-Link
    
    server web1 192.168.1.10:8080 check
</code></pre><h2 id="header-based-routing">Header-Based Routing <a class="anchor" data-pagefind-ignore href="#header-based-routing" aria-label="Link to this section">#</a></h2>
<p>ACLs on headers pick the backend. The first <code>use_backend</code> that matches wins; <code>default_backend</code> takes the rest.</p>
<table>
	<thead>
			<tr>
					<th>Route by</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>Host</code></td>
					<td>Several sites or services on one IP</td>
			</tr>
			<tr>
					<td>A custom header (<code>X-Version</code>)</td>
					<td>API versions, testing a new backend with selected clients</td>
			</tr>
			<tr>
					<td><code>Accept</code></td>
					<td>Separate backends per response format</td>
			</tr>
	</tbody>
</table>
<h3 id="route-by-host-header">Route by Host Header <a class="anchor" data-pagefind-ignore href="#route-by-host-header" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # ACLs based on Host header
    acl host_api hdr(host) -i api.example.com
    acl host_admin hdr(host) -i admin.example.com
    acl host_www hdr(host) -i www.example.com example.com
    
    use_backend api_servers if host_api
    use_backend admin_servers if host_admin
    use_backend www_servers if host_www
    
    default_backend www_servers
</code></pre><h3 id="route-by-custom-header">Route by Custom Header <a class="anchor" data-pagefind-ignore href="#route-by-custom-header" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Route based on X-Version header
    acl version_v2 req.hdr(X-Version) -i v2
    acl version_v1 req.hdr(X-Version) -i v1
    
    use_backend api_v2 if version_v2
    use_backend api_v1 if version_v1
    
    default_backend api_v2
</code></pre><h3 id="route-by-accept-header">Route by Accept Header <a class="anchor" data-pagefind-ignore href="#route-by-accept-header" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    acl accepts_json req.hdr(Accept) -m sub application/json
    acl accepts_xml req.hdr(Accept) -m sub application/xml
    
    use_backend json_api if accepts_json
    use_backend xml_api if accepts_xml
    
    default_backend json_api
</code></pre><h2 id="debugging-specific-issues">Debugging Specific Issues <a class="anchor" data-pagefind-ignore href="#debugging-specific-issues" aria-label="Link to this section">#</a></h2>
<h3 id="session-persistence-problems">Session Persistence Problems <a class="anchor" data-pagefind-ignore href="#session-persistence-problems" aria-label="Link to this section">#</a></h3>
<p>Users that bounce between servers lose their session. <code>cookie SERVERID insert</code> makes HAProxy set a cookie naming the server, so the next request goes to the same one.</p>
<ul>
<li><code>indirect</code>: the cookie is not passed on to the server.</li>
<li><code>nocache</code>: marks responses with the cookie as not cacheable, so a shared cache does not hand it to other users.</li>
<li>The captures log the cookies in both directions: is the cookie set, and does the browser send it back?</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend web_servers
    balance roundrobin
    cookie SERVERID insert indirect nocache
    
    server web1 192.168.1.10:8080 cookie s1 check
    server web2 192.168.1.11:8080 cookie s2 check

frontend http_frontend
    bind *:80
    
    # Log cookie info (capture only works in a frontend)
    capture cookie SERVERID len 32
    
    # Capture cookies for debugging
    capture request header Cookie len 200
    capture response header Set-Cookie len 200
    
    # Log format showing cookies
    log-format &#34;%ci [%t] %r Cookie:{%[capture.req.hdr(0)]} Set-Cookie:{%[capture.res.hdr(0)]}&#34;
    
    default_backend web_servers
</code></pre><h3 id="authentication-header-issues">Authentication Header Issues <a class="anchor" data-pagefind-ignore href="#authentication-header-issues" aria-label="Link to this section">#</a></h3>
<p>For 401 problems: does the <code>Authorization</code> header arrive, and what kind is it? The debug headers <code>X-Auth-Debug</code> and <code>X-Auth-Type</code> tell the server (and your debug backend) without logging the token itself.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Capture auth header (be careful with logging sensitive data!)
    capture request header Authorization len 100
    
    # Check if auth header exists
    acl has_auth req.hdr(Authorization) -m found
    acl valid_bearer req.hdr(Authorization) -m beg Bearer
    
    # Add debug header
    http-request set-header X-Auth-Debug has_auth if has_auth
    http-request set-header X-Auth-Debug missing unless has_auth
    http-request set-header X-Auth-Type bearer if valid_bearer
    http-request set-header X-Auth-Type other if has_auth !valid_bearer
    
    default_backend web_servers
</code></pre><h3 id="content-type-mismatches">Content-Type Mismatches <a class="anchor" data-pagefind-ignore href="#content-type-mismatches" aria-label="Link to this section">#</a></h3>
<p>Copies the request and response <code>Content-Type</code> into extra headers and the log. Useful when a client and a server disagree about JSON, form data or encoding.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Capture content types
    capture request header Content-Type len 50
    capture response header Content-Type len 50
    
    # Add debug info
    http-request set-header X-Request-Content-Type %[req.hdr(Content-Type)]
    http-response set-header X-Response-Content-Type %[res.hdr(Content-Type)]
    
    default_backend web_servers
</code></pre><h3 id="caching-issues">Caching Issues <a class="anchor" data-pagefind-ignore href="#caching-issues" aria-label="Link to this section">#</a></h3>
<p>Captures the cache headers in both directions. The backend rules turn caching off completely; use that only while you debug, as it puts the full load on the servers.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Capture cache headers
    capture response header Cache-Control len 100
    capture response header ETag len 50
    capture request header If-None-Match len 50
    capture request header If-Modified-Since len 50
    
    default_backend web_servers

backend web_servers
    # Force no caching for debugging
    http-response set-header Cache-Control &#34;no-store, no-cache, must-revalidate&#34;
    http-response del-header ETag
    http-response del-header Last-Modified
    
    server web1 192.168.1.10:8080 check
</code></pre><h2 id="creating-a-debug-endpoint">Creating a Debug Endpoint <a class="anchor" data-pagefind-ignore href="#creating-a-debug-endpoint" aria-label="Link to this section">#</a></h2>
<p><code>http-request return</code> answers from HAProxy itself, without a backend. <code>lf-string</code> fills in samples like <code>%[src]</code> per request. Requests to the path get the debug answer; everything else goes to the servers as usual.</p>
<h3 id="echo-all-headers">Echo All Headers <a class="anchor" data-pagefind-ignore href="#echo-all-headers" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    # Debug endpoint that returns request info
    acl is_debug path /debug/headers
    
    http-request return status 200 content-type &#34;text/plain&#34; lf-string &#34;Client IP: %[src]\nMethod: %[method]\nPath: %[path]\nQuery: %[query]\nHost: %[req.hdr(Host)]\nUser-Agent: %[req.hdr(User-Agent)]\nAccept: %[req.hdr(Accept)]\nContent-Type: %[req.hdr(Content-Type)]\nAuthorization: %[req.hdr(Authorization)]\nX-Forwarded-For: %[req.hdr(X-Forwarded-For)]\nCookies: %[req.hdr(Cookie)]&#34; if is_debug
    
    default_backend web_servers
</code></pre><h3 id="json-debug-response">JSON Debug Response <a class="anchor" data-pagefind-ignore href="#json-debug-response" aria-label="Link to this section">#</a></h3>
<p>The same as JSON, for scripts and monitoring tools.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend http_frontend
    bind *:80
    
    acl is_debug path /debug/info
    
    http-request return status 200 content-type &#34;application/json&#34; lf-string &#39;{&#34;client_ip&#34;:&#34;%[src]&#34;,&#34;method&#34;:&#34;%[method]&#34;,&#34;path&#34;:&#34;%[path]&#34;,&#34;host&#34;:&#34;%[req.hdr(Host)]&#34;,&#34;user_agent&#34;:&#34;%[req.hdr(User-Agent)]&#34;,&#34;timestamp&#34;:&#34;%[date]&#34;}&#39; if is_debug
    
    default_backend web_servers
</code></pre><h2 id="lua-scripting-for-advanced-debugging">Lua Scripting for Advanced Debugging <a class="anchor" data-pagefind-ignore href="#lua-scripting-for-advanced-debugging" aria-label="Link to this section">#</a></h2>
<p>The debug endpoints above only show headers you name. A small Lua service prints every header, whatever it is called.</p>
<h3 id="install-lua-support">Install Lua Support <a class="anchor" data-pagefind-ignore href="#install-lua-support" aria-label="Link to this section">#</a></h3>
<p>Debian and Ubuntu build HAProxy with Lua included, there is no separate package. Check it with:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">haproxy -vv <span class="p">|</span> grep -i lua
</span></span></code></pre></div><h3 id="header-dump-lua-script">Header Dump Lua Script <a class="anchor" data-pagefind-ignore href="#header-dump-lua-script" aria-label="Link to this section">#</a></h3>
<p><code>core.register_service</code> defines an HTTP service HAProxy can answer with. It loops over all request headers, adds connection info, and returns it as plain text.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/haproxy/lua/debug-headers.lua</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">core.register_service</span><span class="p">(</span><span class="s2">&#34;debug_headers&#34;</span><span class="p">,</span> <span class="s2">&#34;http&#34;</span><span class="p">,</span> <span class="kr">function</span><span class="p">(</span><span class="n">applet</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="kd">local</span> <span class="n">response</span> <span class="o">=</span> <span class="s2">&#34;Request Headers:</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;================</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1">-- Get all request headers</span>
</span></span><span class="line"><span class="cl">    <span class="kd">local</span> <span class="n">headers</span> <span class="o">=</span> <span class="n">applet.headers</span>
</span></span><span class="line"><span class="cl">    <span class="kr">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">values</span> <span class="kr">in</span> <span class="n">pairs</span><span class="p">(</span><span class="n">headers</span><span class="p">)</span> <span class="kr">do</span>
</span></span><span class="line"><span class="cl">        <span class="kr">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">value</span> <span class="kr">in</span> <span class="n">pairs</span><span class="p">(</span><span class="n">values</span><span class="p">)</span> <span class="kr">do</span>
</span></span><span class="line"><span class="cl">            <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="n">name</span> <span class="o">..</span> <span class="s2">&#34;: &#34;</span> <span class="o">..</span> <span class="n">value</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="kr">end</span>
</span></span><span class="line"><span class="cl">    <span class="kr">end</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">Connection Info:</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;================</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;Client IP: &#34;</span> <span class="o">..</span> <span class="n">applet.sf</span><span class="p">:</span><span class="n">src</span><span class="p">()</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;Client Port: &#34;</span> <span class="o">..</span> <span class="n">applet.sf</span><span class="p">:</span><span class="n">src_port</span><span class="p">()</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;Server IP: &#34;</span> <span class="o">..</span> <span class="n">applet.sf</span><span class="p">:</span><span class="n">dst</span><span class="p">()</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;Server Port: &#34;</span> <span class="o">..</span> <span class="n">applet.sf</span><span class="p">:</span><span class="n">dst_port</span><span class="p">()</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;Method: &#34;</span> <span class="o">..</span> <span class="n">applet.method</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;Path: &#34;</span> <span class="o">..</span> <span class="n">applet.path</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="n">response</span> <span class="o">..</span> <span class="s2">&#34;Query: &#34;</span> <span class="o">..</span> <span class="p">(</span><span class="n">applet.qs</span> <span class="ow">or</span> <span class="s2">&#34;&#34;</span><span class="p">)</span> <span class="o">..</span> <span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="n">applet</span><span class="p">:</span><span class="n">set_status</span><span class="p">(</span><span class="mi">200</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">applet</span><span class="p">:</span><span class="n">add_header</span><span class="p">(</span><span class="s2">&#34;Content-Type&#34;</span><span class="p">,</span> <span class="s2">&#34;text/plain&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">applet</span><span class="p">:</span><span class="n">add_header</span><span class="p">(</span><span class="s2">&#34;Content-Length&#34;</span><span class="p">,</span> <span class="n">string.len</span><span class="p">(</span><span class="n">response</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">applet</span><span class="p">:</span><span class="n">start_response</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">applet</span><span class="p">:</span><span class="n">send</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="kr">end</span><span class="p">)</span>
</span></span></code></pre></div><h3 id="use-in-haproxy-config">Use in HAProxy Config <a class="anchor" data-pagefind-ignore href="#use-in-haproxy-config" aria-label="Link to this section">#</a></h3>
<p><code>lua-load</code> loads the script at startup; <code>use-service lua.debug_headers</code> answers <code>/debug/headers</code> with it.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    # HAProxy 3.1+: also set &#34;tune.lua.bool-sample-conversion normal&#34; before lua-load
    lua-load /etc/haproxy/lua/debug-headers.lua

frontend http_frontend
    bind *:80
    
    acl is_debug path /debug/headers
    http-request use-service lua.debug_headers if is_debug
    
    default_backend web_servers
</code></pre><h2 id="complete-debug-configuration">Complete Debug Configuration <a class="anchor" data-pagefind-ignore href="#complete-debug-configuration" aria-label="Link to this section">#</a></h2>
<p>Everything above in one file, for a test or staging load balancer.</p>
<table>
	<thead>
			<tr>
					<th>Part</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>log ... debug</code>, captures, <code>log-format</code></td>
					<td>Logs chosen headers per request; <code>Authorization</code> cut to 20 characters, enough to see the type</td>
			</tr>
			<tr>
					<td><code>unique-id-*</code></td>
					<td>Request ID to the server and back in <code>X-Request-ID</code></td>
			</tr>
			<tr>
					<td><code>del-header</code> + <code>set-header</code></td>
					<td>Trusted forwarding headers only</td>
			</tr>
			<tr>
					<td><code>/debug/headers</code>, <code>/debug/json</code></td>
					<td>Debug endpoints</td>
			</tr>
			<tr>
					<td>Backend <code>http-response</code> rules</td>
					<td>Hide <code>Server</code>/<code>X-Powered-By</code>, add the server name, request ID and response time</td>
			</tr>
	</tbody>
</table>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0 debug
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    user haproxy
    group haproxy
    daemon
    
    # Enable Lua for advanced debugging
    # lua-load /etc/haproxy/lua/debug-headers.lua

defaults
    log global
    mode http
    option httplog
    option dontlognull
    # no &#34;option forwardfor&#34;: X-Forwarded-For is set explicitly below
    timeout connect 5s
    timeout client  50s
    timeout server  50s

# Stats with header info
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 5s
    http-request use-service prometheus-exporter if { path /metrics }

frontend http_frontend
    bind *:80
    
    # Generate unique request ID
    unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
    unique-id-header X-Request-ID
    
    # Capture headers for logging
    capture request header Host len 50
    capture request header User-Agent len 100
    capture request header X-Forwarded-For len 50
    capture request header Authorization len 20
    capture request header Content-Type len 50
    capture request header Cookie len 100
    capture request header X-Request-ID len 50
    
    capture response header Content-Type len 50
    capture response header Location len 200
    capture response header Set-Cookie len 100
    capture response header Cache-Control len 50
    
    # Detailed log format
    log-format &#34;%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq {%hrl} {%hsl} %{+Q}r&#34;
    
    # Remove potentially spoofed headers
    http-request del-header X-Forwarded-For
    http-request del-header X-Real-IP
    http-request del-header X-Forwarded-Proto
    
    # Add trusted forwarding headers
    http-request set-header X-Forwarded-For %[src]
    http-request set-header X-Real-IP %[src]
    http-request set-header X-Forwarded-Proto http
    http-request set-header X-Forwarded-Host %[req.hdr(Host)]
    http-request set-header X-Forwarded-Port %[dst_port]
    
    # Debug endpoints
    acl is_debug_headers path /debug/headers
    acl is_debug_json path /debug/json
    
    http-request return status 200 content-type &#34;text/plain&#34; lf-string &#34;Client: %[src]:%[src_port]\nHost: %[req.hdr(Host)]\nMethod: %[method]\nPath: %[path]\nQuery: %[query]\nProtocol: %[req.ver]\nUser-Agent: %[req.hdr(User-Agent)]\nAccept: %[req.hdr(Accept)]\nContent-Type: %[req.hdr(Content-Type)]\nX-Request-ID: %[unique-id]\nCookies: %[req.hdr(Cookie)]&#34; if is_debug_headers
    
    http-request return status 200 content-type &#34;application/json&#34; lf-string &#39;{&#34;client_ip&#34;:&#34;%[src]&#34;,&#34;host&#34;:&#34;%[req.hdr(Host)]&#34;,&#34;method&#34;:&#34;%[method]&#34;,&#34;path&#34;:&#34;%[path]&#34;,&#34;request_id&#34;:&#34;%[unique-id]&#34;}&#39; if is_debug_json
    
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    
    # Remove server identification from responses
    http-response del-header Server
    http-response del-header X-Powered-By
    
    # Add debug headers to response
    http-response set-header X-Backend-Server %s
    http-response set-header X-Request-ID %[unique-id]
    http-response set-header X-Response-Time %Ta
    
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check
</code></pre><h2 id="quick-debugging-commands">Quick Debugging Commands <a class="anchor" data-pagefind-ignore href="#quick-debugging-commands" aria-label="Link to this section">#</a></h2>
<p>Start with <code>curl -v</code> (headers in both directions), then the debug endpoint, then the logs. Sending your own <code>X-Forwarded-For</code> shows whether HAProxy passes a forged client IP on.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Watch HAProxy logs in real-time</span>
</span></span><span class="line"><span class="cl">tail -f /var/log/haproxy.log <span class="p">|</span> grep http_frontend
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test with curl and show headers</span>
</span></span><span class="line"><span class="cl">curl -v http://example.com/
</span></span><span class="line"><span class="cl">curl -H <span class="s2">&#34;X-Custom: test&#34;</span> http://example.com/debug/headers
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test from specific IP (if accessible)</span>
</span></span><span class="line"><span class="cl">curl -H <span class="s2">&#34;X-Forwarded-For: 1.2.3.4&#34;</span> http://example.com/debug/headers
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check HAProxy session info</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show sess&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Show backend status</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show stat&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock <span class="p">|</span> cut -d<span class="s1">&#39;,&#39;</span> -f1,2,18
</span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Logs hold secrets.</strong> Captured <code>Authorization</code> and <code>Cookie</code> headers end up in log files. Keep captures short (like <code>len 20</code>), or leave them out in production.</li>
<li><strong>Debug endpoints are public</strong> on the frontend they are on. Restrict them to your network with an ACL, or remove them after debugging.</li>
<li><strong>Never trust client forwarding headers.</strong> Delete them before you set your own.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-http-headers" rel="noopener">pderuiter/blog-haproxy-http-headers</a>. It creates an HAProxy VM on Proxmox with the complete debug configuration from this post, header capture, request IDs, debug endpoints and the optional Lua header dump; run <code>terraform apply</code>, then <code>ansible-playbook site.yml</code> (add <code>-e @../tests/demo-vars.yml</code> for two demo backends that echo the headers they receive).</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>Next in the series: <a href="https://blog.bsdserver.nl/haproxy-mtls-securing-api-access-with-mutual-tls-authentication/">mTLS for securing API access</a>, building on the TLS knowledge from previous posts.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<ul>
<li>Use <code>capture request/response header</code> for logging</li>
<li>Create debug endpoints with <code>http-request return</code></li>
<li>Remove untrusted headers before adding your own</li>
<li>Use <code>http-request/response set-header</code> for manipulation</li>
<li>Add X-Request-ID for tracing across services</li>
<li>Use Lua for advanced header inspection</li>
<li>Always secure sensitive header logging in production</li>
</ul>
]]></content:encoded></item><item><title>HAProxy mTLS: Securing API Access with Mutual TLS Authentication</title><link>https://blog.bsdserver.nl/haproxy-mtls-securing-api-access-with-mutual-tls-authentication/</link><guid isPermaLink="false">693b755d79eb54000160a60f</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:52:29 +0000</pubDate><description>Mutual TLS (mTLS) provides strong authentication by requiring both client and server to present certificates.</description><category>HAProxy</category><category>mTLS</category><category>Security</category><category>TLS</category><category>Certificates</category><category>API</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-mtls.png" medium="image"/><content:encoded><![CDATA[<p>Mutual TLS (mTLS) provides strong authentication by requiring both client and server to present certificates. API keys and passwords can leak and be replayed; a client certificate proves identity during the TLS handshake, before any request reaches your application.</p>
<p>This guide builds a small internal CA, configures HAProxy to require client certificates, passes the certificate identity to the backends, authorizes by CN, OU or fingerprint, and revokes certificates with a CRL.</p>
<h2 id="what-is-mtls">What is mTLS? <a class="anchor" data-pagefind-ignore href="#what-is-mtls" aria-label="Link to this section">#</a></h2>
<p>In standard TLS, only the server presents a certificate. With mTLS:</p>
<ol>
<li>Server presents its certificate to the client</li>
<li>Client verifies the server certificate</li>
<li>Client presents its certificate to the server</li>
<li>Server verifies the client certificate</li>
</ol>
<p>Both parties are authenticated. HAProxy does the client check, so the backends do not need any TLS code.</p>
<h2 id="why-mtls">Why mTLS? <a class="anchor" data-pagefind-ignore href="#why-mtls" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Service-to-service communication</strong>: Microservices authenticating each other</li>
<li><strong>API security</strong>: Authenticating API clients without passwords</li>
<li><strong>Zero-trust networks</strong>: Verifying identity at every connection</li>
<li><strong>IoT devices</strong>: Authenticating embedded devices</li>
<li><strong>B2B integrations</strong>: Partner API access with certificate-based auth</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> for machine clients you control or issue certificates to: services, devices, partners.</li>
<li><strong>Skip it</strong> for public websites and end users in browsers. Installing certificates on user devices is painful; use OAuth or sessions instead.</li>
</ul>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Internal CA</td>
					<td>A locked-down host (here: the HAProxy host)</td>
					<td>Signs server and client certificates, publishes the CRL.</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td>Load balancer</td>
					<td>Terminates TLS, checks the client certificate against the CA and CRL, applies ACLs, adds identity headers.</td>
			</tr>
			<tr>
					<td>Backends</td>
					<td>App servers</td>
					<td>Receive plain HTTP plus <code>X-Client-*</code> headers.</td>
			</tr>
			<tr>
					<td>Clients</td>
					<td>Services, devices, partners</td>
					<td>Present a certificate signed by the CA.</td>
			</tr>
	</tbody>
</table>
<p>The flow: client connects with its certificate → HAProxy verifies it against <code>ca-file</code> and <code>crl-file</code> → ACLs on CN/OU allow or deny → request goes to a backend with the client identity in headers.</p>
<h2 id="certificate-infrastructure-setup">Certificate Infrastructure Setup <a class="anchor" data-pagefind-ignore href="#certificate-infrastructure-setup" aria-label="Link to this section">#</a></h2>
<h3 id="create-a-certificate-authority-ca">Create a Certificate Authority (CA) <a class="anchor" data-pagefind-ignore href="#create-a-certificate-authority-ca" aria-label="Link to this section">#</a></h3>
<p>Your own CA decides who is a valid client: HAProxy trusts every certificate it signs. Use a private CA, never a public one, or anyone with a public certificate gets in.</p>
<p>What the <code>openssl.cnf</code> sections do:</p>
<table>
	<thead>
			<tr>
					<th>Section</th>
					<th>Use</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>CA_default</code></td>
					<td>Paths, 365-day default validity, 30-day CRL validity. <code>copy_extensions = copy</code> keeps the SAN from the CSR.</td>
			</tr>
			<tr>
					<td><code>policy_strict</code></td>
					<td>Only <code>commonName</code> is required in requests.</td>
			</tr>
			<tr>
					<td><code>v3_ca</code></td>
					<td>Extensions for the CA certificate itself.</td>
			</tr>
			<tr>
					<td><code>usr_cert</code></td>
					<td>Client certificates: <code>extendedKeyUsage = clientAuth</code>.</td>
			</tr>
			<tr>
					<td><code>server_cert</code></td>
					<td>Server certificates: <code>extendedKeyUsage = serverAuth</code>.</td>
			</tr>
	</tbody>
</table>
<p>The CA key is encrypted with a passphrase (<code>-aes256</code>) and the CA certificate is valid for 10 years.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Create CA directory structure</span>
</span></span><span class="line"><span class="cl">mkdir -p /etc/haproxy/ca/<span class="o">{</span>certs,crl,newcerts,private,csr<span class="o">}</span>
</span></span><span class="line"><span class="cl">chmod <span class="m">700</span> /etc/haproxy/ca/private
</span></span><span class="line"><span class="cl">touch /etc/haproxy/ca/index.txt
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="m">1000</span> &gt; /etc/haproxy/ca/serial
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create CA configuration</span>
</span></span><span class="line"><span class="cl">cat &gt; /etc/haproxy/ca/openssl.cnf <span class="s">&lt;&lt; &#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">[ca]
</span></span></span><span class="line"><span class="cl"><span class="s">default_ca = CA_default
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[CA_default]
</span></span></span><span class="line"><span class="cl"><span class="s">dir               = /etc/haproxy/ca
</span></span></span><span class="line"><span class="cl"><span class="s">certs             = $dir/certs
</span></span></span><span class="line"><span class="cl"><span class="s">crl_dir           = $dir/crl
</span></span></span><span class="line"><span class="cl"><span class="s">new_certs_dir     = $dir/newcerts
</span></span></span><span class="line"><span class="cl"><span class="s">database          = $dir/index.txt
</span></span></span><span class="line"><span class="cl"><span class="s">serial            = $dir/serial
</span></span></span><span class="line"><span class="cl"><span class="s">private_key       = $dir/private/ca.key
</span></span></span><span class="line"><span class="cl"><span class="s">certificate       = $dir/certs/ca.crt
</span></span></span><span class="line"><span class="cl"><span class="s">default_md        = sha256
</span></span></span><span class="line"><span class="cl"><span class="s">default_days      = 365
</span></span></span><span class="line"><span class="cl"><span class="s">default_crl_days  = 30
</span></span></span><span class="line"><span class="cl"><span class="s">copy_extensions   = copy
</span></span></span><span class="line"><span class="cl"><span class="s">preserve          = no
</span></span></span><span class="line"><span class="cl"><span class="s">policy            = policy_strict
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[policy_strict]
</span></span></span><span class="line"><span class="cl"><span class="s">countryName             = optional
</span></span></span><span class="line"><span class="cl"><span class="s">stateOrProvinceName     = optional
</span></span></span><span class="line"><span class="cl"><span class="s">localityName            = optional
</span></span></span><span class="line"><span class="cl"><span class="s">organizationName        = optional
</span></span></span><span class="line"><span class="cl"><span class="s">organizationalUnitName  = optional
</span></span></span><span class="line"><span class="cl"><span class="s">commonName              = supplied
</span></span></span><span class="line"><span class="cl"><span class="s">emailAddress            = optional
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[req]
</span></span></span><span class="line"><span class="cl"><span class="s">default_bits        = 4096
</span></span></span><span class="line"><span class="cl"><span class="s">distinguished_name  = req_distinguished_name
</span></span></span><span class="line"><span class="cl"><span class="s">string_mask         = utf8only
</span></span></span><span class="line"><span class="cl"><span class="s">default_md          = sha256
</span></span></span><span class="line"><span class="cl"><span class="s">x509_extensions     = v3_ca
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[req_distinguished_name]
</span></span></span><span class="line"><span class="cl"><span class="s">countryName                     = Country Name (2 letter code)
</span></span></span><span class="line"><span class="cl"><span class="s">stateOrProvinceName             = State or Province Name
</span></span></span><span class="line"><span class="cl"><span class="s">localityName                    = Locality Name
</span></span></span><span class="line"><span class="cl"><span class="s">organizationName                = Organization Name
</span></span></span><span class="line"><span class="cl"><span class="s">organizationalUnitName          = Organizational Unit Name
</span></span></span><span class="line"><span class="cl"><span class="s">commonName                      = Common Name
</span></span></span><span class="line"><span class="cl"><span class="s">emailAddress                    = Email Address
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[v3_ca]
</span></span></span><span class="line"><span class="cl"><span class="s">subjectKeyIdentifier = hash
</span></span></span><span class="line"><span class="cl"><span class="s">authorityKeyIdentifier = keyid:always,issuer
</span></span></span><span class="line"><span class="cl"><span class="s">basicConstraints = critical, CA:true
</span></span></span><span class="line"><span class="cl"><span class="s">keyUsage = critical, digitalSignature, cRLSign, keyCertSign
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[usr_cert]
</span></span></span><span class="line"><span class="cl"><span class="s">basicConstraints = CA:FALSE
</span></span></span><span class="line"><span class="cl"><span class="s">nsCertType = client, email
</span></span></span><span class="line"><span class="cl"><span class="s">nsComment = &#34;Client Certificate&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">subjectKeyIdentifier = hash
</span></span></span><span class="line"><span class="cl"><span class="s">authorityKeyIdentifier = keyid,issuer
</span></span></span><span class="line"><span class="cl"><span class="s">keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
</span></span></span><span class="line"><span class="cl"><span class="s">extendedKeyUsage = clientAuth, emailProtection
</span></span></span><span class="line"><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="cl"><span class="s">[server_cert]
</span></span></span><span class="line"><span class="cl"><span class="s">basicConstraints = CA:FALSE
</span></span></span><span class="line"><span class="cl"><span class="s">nsCertType = server
</span></span></span><span class="line"><span class="cl"><span class="s">nsComment = &#34;Server Certificate&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">subjectKeyIdentifier = hash
</span></span></span><span class="line"><span class="cl"><span class="s">authorityKeyIdentifier = keyid,issuer:always
</span></span></span><span class="line"><span class="cl"><span class="s">keyUsage = critical, digitalSignature, keyEncipherment
</span></span></span><span class="line"><span class="cl"><span class="s">extendedKeyUsage = serverAuth
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Generate CA private key</span>
</span></span><span class="line"><span class="cl">openssl genrsa -aes256 -out /etc/haproxy/ca/private/ca.key <span class="m">4096</span>
</span></span><span class="line"><span class="cl">chmod <span class="m">400</span> /etc/haproxy/ca/private/ca.key
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Generate CA certificate</span>
</span></span><span class="line"><span class="cl">openssl req -config /etc/haproxy/ca/openssl.cnf <span class="se">\
</span></span></span><span class="line"><span class="cl">  -key /etc/haproxy/ca/private/ca.key <span class="se">\
</span></span></span><span class="line"><span class="cl">  -new -x509 -days <span class="m">3650</span> -sha256 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -extensions v3_ca <span class="se">\
</span></span></span><span class="line"><span class="cl">  -out /etc/haproxy/ca/certs/ca.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  -subj <span class="s2">&#34;/C=US/ST=State/L=City/O=MyOrg/CN=MyOrg CA&#34;</span>
</span></span></code></pre></div><h3 id="generate-server-certificate">Generate Server Certificate <a class="anchor" data-pagefind-ignore href="#generate-server-certificate" aria-label="Link to this section">#</a></h3>
<p>HAProxy presents this certificate to clients on the mTLS port. Clients verify it against the same CA, so they know they talk to the real API. The bind line loads certificate and key from one combined PEM file.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Generate server key</span>
</span></span><span class="line"><span class="cl">mkdir -p /etc/haproxy/certs
</span></span><span class="line"><span class="cl">openssl genrsa -out /etc/haproxy/certs/server.key <span class="m">2048</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Generate server CSR (clients such as Go and Python require the SAN)</span>
</span></span><span class="line"><span class="cl">openssl req -new <span class="se">\
</span></span></span><span class="line"><span class="cl">  -key /etc/haproxy/certs/server.key <span class="se">\
</span></span></span><span class="line"><span class="cl">  -out /etc/haproxy/ca/csr/server.csr <span class="se">\
</span></span></span><span class="line"><span class="cl">  -subj <span class="s2">&#34;/CN=api.example.com&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -addext <span class="s2">&#34;subjectAltName=DNS:api.example.com&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Sign server certificate with CA</span>
</span></span><span class="line"><span class="cl">openssl ca -config /etc/haproxy/ca/openssl.cnf <span class="se">\
</span></span></span><span class="line"><span class="cl">  -extensions server_cert <span class="se">\
</span></span></span><span class="line"><span class="cl">  -days <span class="m">365</span> -notext -md sha256 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -in /etc/haproxy/ca/csr/server.csr <span class="se">\
</span></span></span><span class="line"><span class="cl">  -out /etc/haproxy/certs/server.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  -batch
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create combined PEM for HAProxy</span>
</span></span><span class="line"><span class="cl">cat /etc/haproxy/certs/server.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">    /etc/haproxy/certs/server.key <span class="se">\
</span></span></span><span class="line"><span class="cl">    &gt; /etc/haproxy/certs/server.pem
</span></span><span class="line"><span class="cl">chmod <span class="m">600</span> /etc/haproxy/certs/server.pem
</span></span></code></pre></div><h3 id="generate-client-certificate">Generate Client Certificate <a class="anchor" data-pagefind-ignore href="#generate-client-certificate" aria-label="Link to this section">#</a></h3>
<p>Every client gets its own certificate, so you can authorize and revoke them one by one. The script takes a name (becomes the CN) and an optional OU (default <code>internal</code>); both are used in the ACLs later.</p>
<table>
	<thead>
			<tr>
					<th>Output</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>.key</code> + <code>.crt</code></td>
					<td>curl, Python, Go, Node.js</td>
			</tr>
			<tr>
					<td><code>.pem</code></td>
					<td>Tools that want certificate and key in one file</td>
			</tr>
			<tr>
					<td><code>.p12</code></td>
					<td>Browsers and Java; password <code>changeme</code>, change it</td>
			</tr>
	</tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /usr/local/bin/generate-client-cert.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">CLIENT_NAME</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">CLIENT_OU</span><span class="o">=</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">2</span><span class="k">:-</span><span class="nv">internal</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="o">[</span> -z <span class="s2">&#34;</span><span class="nv">$CLIENT_NAME</span><span class="s2">&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;Usage: </span><span class="nv">$0</span><span class="s2"> &lt;client-name&gt; [organizational-unit]&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="nb">exit</span> <span class="m">1</span>
</span></span><span class="line"><span class="cl"><span class="k">fi</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">CA_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/haproxy/ca&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">OUTPUT_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/haproxy/clients/</span><span class="nv">$CLIENT_NAME</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">mkdir -p <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Generate client key</span>
</span></span><span class="line"><span class="cl">openssl genrsa -out <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.key&#34;</span> <span class="m">2048</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Generate client CSR</span>
</span></span><span class="line"><span class="cl">openssl req -new <span class="se">\
</span></span></span><span class="line"><span class="cl">  -key <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.key&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -out <span class="s2">&#34;</span><span class="nv">$CA_DIR</span><span class="s2">/csr/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.csr&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -subj <span class="s2">&#34;/OU=</span><span class="nv">$CLIENT_OU</span><span class="s2">/CN=</span><span class="nv">$CLIENT_NAME</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Sign client certificate</span>
</span></span><span class="line"><span class="cl">openssl ca -config <span class="s2">&#34;</span><span class="nv">$CA_DIR</span><span class="s2">/openssl.cnf&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -extensions usr_cert <span class="se">\
</span></span></span><span class="line"><span class="cl">  -days <span class="m">365</span> -notext -md sha256 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -in <span class="s2">&#34;</span><span class="nv">$CA_DIR</span><span class="s2">/csr/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.csr&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -out <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.crt&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -batch
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create combined PEM (for applications that need it)</span>
</span></span><span class="line"><span class="cl">cat <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.crt&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.key&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    &gt; <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.pem&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create PKCS12 bundle (for browsers and some clients)</span>
</span></span><span class="line"><span class="cl">openssl pkcs12 -export <span class="se">\
</span></span></span><span class="line"><span class="cl">  -out <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.p12&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -inkey <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.key&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -in <span class="s2">&#34;</span><span class="nv">$OUTPUT_DIR</span><span class="s2">/</span><span class="nv">$CLIENT_NAME</span><span class="s2">.crt&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -certfile <span class="s2">&#34;</span><span class="nv">$CA_DIR</span><span class="s2">/certs/ca.crt&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -passout pass:changeme
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Client certificate generated in </span><span class="nv">$OUTPUT_DIR</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Files: </span><span class="nv">$CLIENT_NAME</span><span class="s2">.key, </span><span class="nv">$CLIENT_NAME</span><span class="s2">.crt, </span><span class="nv">$CLIENT_NAME</span><span class="s2">.pem, </span><span class="nv">$CLIENT_NAME</span><span class="s2">.p12&#34;</span>
</span></span></code></pre></div><h2 id="basic-mtls-configuration">Basic mTLS Configuration <a class="anchor" data-pagefind-ignore href="#basic-mtls-configuration" aria-label="Link to this section">#</a></h2>
<p>Two bind options switch mTLS on:</p>
<ul>
<li><code>ca-file</code>: the CA that client certificates must be signed by.</li>
<li><code>verify</code>: <code>required</code> rejects the handshake without a valid certificate; <code>optional</code> lets it through and you decide in ACLs.</li>
</ul>
<table>
	<thead>
			<tr>
					<th>Mode</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>verify required</code></td>
					<td>Endpoints only certificate holders may reach</td>
			</tr>
			<tr>
					<td><code>verify optional</code></td>
					<td>One endpoint serving both anonymous and authenticated clients</td>
			</tr>
	</tbody>
</table>
<h3 id="required-mtls">Required mTLS <a class="anchor" data-pagefind-ignore href="#required-mtls" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    user haproxy
    group haproxy
    daemon
    
    # SSL/TLS settings
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log global
    mode http
    option httplog
    timeout connect 5s
    timeout client  50s
    timeout server  50s

frontend mtls_api
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt verify required
    
    # Client certificate is required - connections without valid cert are rejected
    default_backend api_servers

backend api_servers
    server api1 192.168.1.10:8080 check
    server api2 192.168.1.11:8080 check
</code></pre><h3 id="optional-mtls-certificate-if-available">Optional mTLS (Certificate if Available) <a class="anchor" data-pagefind-ignore href="#optional-mtls-certificate-if-available" aria-label="Link to this section">#</a></h3>
<p>Clients with a valid certificate go to <code>authenticated_api</code>, clients without one to <code>public_api</code>, and an invalid certificate gets a 403.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mtls_optional
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt verify optional
    
    # Check if client provided valid certificate
    # (ssl_c_verify is also 0 when no certificate was sent, so combine it with ssl_fc_has_crt)
    acl client_cert_valid ssl_c_verify 0
    acl client_cert_provided ssl_fc_has_crt
    
    # Reject invalid certificates
    http-request deny if client_cert_provided !client_cert_valid
    
    # Route based on certificate status
    use_backend authenticated_api if client_cert_provided client_cert_valid
    use_backend public_api unless client_cert_provided

backend authenticated_api
    server api1 192.168.1.10:8080 check

backend public_api
    server public1 192.168.1.20:8080 check
</code></pre><h2 id="passing-client-certificate-info-to-backend">Passing Client Certificate Info to Backend <a class="anchor" data-pagefind-ignore href="#passing-client-certificate-info-to-backend" aria-label="Link to this section">#</a></h2>
<p>The backend sees plain HTTP from HAProxy, not the certificate. Add headers so the application knows which client called.</p>
<h3 id="certificate-details-as-headers">Certificate Details as Headers <a class="anchor" data-pagefind-ignore href="#certificate-details-as-headers" aria-label="Link to this section">#</a></h3>
<ul>
<li><code>ssl_c_s_dn(cn)</code>: the client name, the usual identity for application logic.</li>
<li><code>ssl_c_serial</code> / <code>ssl_c_sha1</code>: unique per certificate, useful for audit logs.</li>
<li><code>%{+Q}</code>: quotes the value, since DNs contain commas and spaces.</li>
</ul>
<p><code>set-header</code> overwrites any <code>X-Client-*</code> header a client sends itself, so clients cannot fake them.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mtls_api
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt verify required
    
    # Client certificate subject (DN)
    http-request set-header X-Client-DN %{+Q}[ssl_c_s_dn]
    
    # Client certificate common name
    http-request set-header X-Client-CN %{+Q}[ssl_c_s_dn(cn)]
    
    # Certificate serial number
    http-request set-header X-Client-Serial %[ssl_c_serial,hex]
    
    # Certificate fingerprint (SHA1)
    http-request set-header X-Client-Fingerprint %[ssl_c_sha1,hex]
    
    # Certificate validity dates
    http-request set-header X-Client-NotBefore %[ssl_c_notbefore]
    http-request set-header X-Client-NotAfter %[ssl_c_notafter]
    
    # Issuer DN
    http-request set-header X-Client-Issuer %{+Q}[ssl_c_i_dn]
    
    # Verification status (0 = success)
    http-request set-header X-Client-Verify %[ssl_c_verify]
    
    default_backend api_servers
</code></pre><h3 id="full-certificate-as-header-base64">Full Certificate as Header (Base64) <a class="anchor" data-pagefind-ignore href="#full-certificate-as-header-base64" aria-label="Link to this section">#</a></h3>
<p>Use this when the application must parse the certificate itself, for example to read custom extensions.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mtls_api
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt verify required
    
    # Pass entire certificate (DER format, base64 encoded)
    http-request set-header X-Client-Cert %[ssl_c_der,base64]
    
    default_backend api_servers
</code></pre><h2 id="certificate-based-authorization">Certificate-Based Authorization <a class="anchor" data-pagefind-ignore href="#certificate-based-authorization" aria-label="Link to this section">#</a></h2>
<p>A valid certificate proves who the client is, not what it may do. ACLs on certificate fields add authorization.</p>
<table>
	<thead>
			<tr>
					<th>Match on</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>CN</td>
					<td>Individual services or clients</td>
			</tr>
			<tr>
					<td>OU</td>
					<td>Groups of clients (internal, partner, admin)</td>
			</tr>
			<tr>
					<td>Fingerprint</td>
					<td>An explicit allowlist of exact certificates</td>
			</tr>
	</tbody>
</table>
<h3 id="route-by-client-cn">Route by Client CN <a class="anchor" data-pagefind-ignore href="#route-by-client-cn" aria-label="Link to this section">#</a></h3>
<p>Each named client gets its own backend; everyone else goes to <code>api_backend</code>.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mtls_api
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt verify required
    
    # ACLs based on client certificate CN
    acl cn_admin ssl_c_s_dn(cn) -i admin-service
    acl cn_api ssl_c_s_dn(cn) -i api-client
    acl cn_monitoring ssl_c_s_dn(cn) -i monitoring
    
    # Route to different backends
    use_backend admin_backend if cn_admin
    use_backend monitoring_backend if cn_monitoring
    default_backend api_backend
</code></pre><h3 id="restrict-access-by-certificate-ou">Restrict Access by Certificate OU <a class="anchor" data-pagefind-ignore href="#restrict-access-by-certificate-ou" aria-label="Link to this section">#</a></h3>
<p>Paths are locked to a group: <code>/admin</code> only for the <code>administrators</code> OU, <code>/api/partner</code> only for <code>partner-api</code>.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mtls_api
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt verify required
    
    # Check organizational unit
    acl ou_internal ssl_c_s_dn(ou) -i internal-services
    acl ou_partner ssl_c_s_dn(ou) -i partner-api
    acl ou_admin ssl_c_s_dn(ou) -i administrators
    
    # Restrict admin endpoints
    acl admin_path path_beg /admin
    http-request deny if admin_path !ou_admin
    
    # Partner access restrictions
    acl partner_path path_beg /api/partner
    http-request deny if partner_path !ou_partner
    
    default_backend api_servers
</code></pre><h3 id="whitelist-specific-certificates">Whitelist Specific Certificates <a class="anchor" data-pagefind-ignore href="#whitelist-specific-certificates" aria-label="Link to this section">#</a></h3>
<p>The strictest option: only certificates whose SHA1 fingerprint is in the file get through, even if the CA signed others.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mtls_api
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt verify required
    
    # Define allowed certificate fingerprints
    acl allowed_cert ssl_c_sha1,hex -f /etc/haproxy/allowed-certs.txt
    
    # Reject certificates not in whitelist
    http-request deny unless allowed_cert
    
    default_backend api_servers
</code></pre><p>Create the whitelist file. A renewed certificate has a new fingerprint, so update the file when you reissue.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/haproxy/allowed-certs.txt</span>
</span></span><span class="line"><span class="cl"><span class="c1"># One SHA1 fingerprint per line, uppercase hex without colons:</span>
</span></span><span class="line"><span class="cl"><span class="c1">#   openssl x509 -in client.crt -noout -fingerprint -sha1 | cut -d= -f2 | tr -d :</span>
</span></span><span class="line"><span class="cl">A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0
</span></span><span class="line"><span class="cl">9T8S7R6Q5P4O3N2M1L0K9J8I7H6G5F4E3D2C1B0A
</span></span></code></pre></div><h2 id="certificate-revocation">Certificate Revocation <a class="anchor" data-pagefind-ignore href="#certificate-revocation" aria-label="Link to this section">#</a></h2>
<p>A stolen or retired client certificate stays valid until it expires. A CRL lists revoked serial numbers; HAProxy rejects them during the handshake.</p>
<h3 id="using-crl-certificate-revocation-list">Using CRL (Certificate Revocation List) <a class="anchor" data-pagefind-ignore href="#using-crl-certificate-revocation-list" aria-label="Link to this section">#</a></h3>
<p>Revoke the certificate, regenerate the CRL, then load it with <code>crl-file</code> on the bind line.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Generate initial CRL</span>
</span></span><span class="line"><span class="cl">openssl ca -config /etc/haproxy/ca/openssl.cnf <span class="se">\
</span></span></span><span class="line"><span class="cl">  -gencrl -out /etc/haproxy/ca/crl/ca.crl
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Revoke a certificate</span>
</span></span><span class="line"><span class="cl">openssl ca -config /etc/haproxy/ca/openssl.cnf <span class="se">\
</span></span></span><span class="line"><span class="cl">  -revoke /etc/haproxy/clients/bad-client/bad-client.crt
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Regenerate CRL</span>
</span></span><span class="line"><span class="cl">openssl ca -config /etc/haproxy/ca/openssl.cnf <span class="se">\
</span></span></span><span class="line"><span class="cl">  -gencrl -out /etc/haproxy/ca/crl/ca.crl
</span></span></code></pre></div><p>HAProxy reads the CRL at startup, so reload after each change.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mtls_api
    # HAProxy has no line continuation: keep the bind on one line
    bind *:443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt crl-file /etc/haproxy/ca/crl/ca.crl verify required
    
    default_backend api_servers
</code></pre><h3 id="automated-crl-updates">Automated CRL Updates <a class="anchor" data-pagefind-ignore href="#automated-crl-updates" aria-label="Link to this section">#</a></h3>
<p>A CRL expires after <code>default_crl_days</code> (30). An expired CRL makes HAProxy reject every client, so regenerate it on a schedule.</p>
<p>The CA key is encrypted, and cron cannot type its passphrase. Store the passphrase in a file that only root can read, so the script can pass it to openssl with <code>-passin</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Same passphrase as the one protecting ca.key</span>
</span></span><span class="line"><span class="cl">install -m <span class="m">0400</span> /dev/null /etc/haproxy/ca/private/ca.pass
</span></span><span class="line"><span class="cl"><span class="nb">read</span> -rsp <span class="s1">&#39;CA key passphrase: &#39;</span> P <span class="o">&amp;&amp;</span> <span class="nb">printf</span> <span class="s1">&#39;%s\n&#39;</span> <span class="s2">&#34;</span><span class="nv">$P</span><span class="s2">&#34;</span> &gt; /etc/haproxy/ca/private/ca.pass<span class="p">;</span> <span class="nb">unset</span> P<span class="p">;</span> <span class="nb">echo</span>
</span></span></code></pre></div><p>Anyone with root on this machine can now sign certificates with your CA, so keep the CA host locked down.</p>
<p>The script writes to a temporary file first, so a failed run never leaves HAProxy with a broken CRL.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /usr/local/bin/update-crl.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -euo pipefail
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">CA_DIR</span><span class="o">=</span><span class="s2">&#34;/etc/haproxy/ca&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">CRL_FILE</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$CA_DIR</span><span class="s2">/crl/ca.crl&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">PASS_FILE</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$CA_DIR</span><span class="s2">/private/ca.pass&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Regenerate CRL; if openssl fails the script stops here and HAProxy keeps the current CRL</span>
</span></span><span class="line"><span class="cl">openssl ca -config <span class="s2">&#34;</span><span class="nv">$CA_DIR</span><span class="s2">/openssl.cnf&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -gencrl -passin <span class="s2">&#34;file:</span><span class="nv">$PASS_FILE</span><span class="s2">&#34;</span> -out <span class="s2">&#34;</span><span class="nv">$CRL_FILE</span><span class="s2">.new&#34;</span>
</span></span><span class="line"><span class="cl">mv <span class="s2">&#34;</span><span class="nv">$CRL_FILE</span><span class="s2">.new&#34;</span> <span class="s2">&#34;</span><span class="nv">$CRL_FILE</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Reload HAProxy</span>
</span></span><span class="line"><span class="cl">systemctl reload haproxy
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">chmod <span class="m">700</span> /usr/local/bin/update-crl.sh
</span></span></code></pre></div><p>Schedule with cron, in root&rsquo;s crontab (<code>crontab -e</code> as root):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Update CRL every hour</span>
</span></span><span class="line"><span class="cl"><span class="m">0</span> * * * * /usr/local/bin/update-crl.sh
</span></span></code></pre></div><h2 id="complete-production-configuration">Complete Production Configuration <a class="anchor" data-pagefind-ignore href="#complete-production-configuration" aria-label="Link to this section">#</a></h2>
<p>Everything combined. The layout:</p>
<table>
	<thead>
			<tr>
					<th>Frontend / listen</th>
					<th>Port</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>stats</code></td>
					<td>8404</td>
					<td>Stats page and Prometheus <code>/metrics</code></td>
			</tr>
			<tr>
					<td><code>public_https</code></td>
					<td>443</td>
					<td>Public site with a public certificate; <code>/api</code> is redirected to 8443</td>
			</tr>
			<tr>
					<td><code>mtls_api</code></td>
					<td>8443</td>
					<td>mTLS with CRL, identity headers, CN/OU authorization</td>
			</tr>
	</tbody>
</table>
<p>The authorization rules in <code>mtls_api</code>:</p>
<ul>
<li><code>/api/admin</code> only for CN <code>admin-service</code>.</li>
<li>CNs containing <code>readonly</code> cannot use POST, PUT, DELETE or PATCH.</li>
<li>Only OU <code>internal</code> or <code>partner</code> gets in at all; partners go to <code>partner_api</code>.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    maxconn 50000
    nbthread 4
    
    # Modern TLS settings
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    tune.ssl.default-dh-param 2048

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    timeout connect 5s
    timeout client  50s
    timeout server  50s

# Stats
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    http-request use-service prometheus-exporter if { path /metrics }

# Public HTTPS (no client cert required)
frontend public_https
    # public.pem: your publicly trusted certificate (for example Let&#39;s Encrypt), not one from the internal CA
    bind *:443 ssl crt /etc/haproxy/certs/public.pem alpn h2,http/1.1
    
    # Route to mTLS frontend for API paths
    acl api_path path_beg /api
    use_backend mtls_redirect if api_path
    
    default_backend public_web

# mTLS API endpoint
frontend mtls_api
    bind *:8443 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/ca/certs/ca.crt crl-file /etc/haproxy/ca/crl/ca.crl verify required alpn h2,http/1.1
    
    # Log client certificate info
    capture request header X-Client-CN len 50
    log-format &#34;%ci:%cp [%tr] %ft %b/%s %ST %B %TR/%Tw/%Tc/%Tr/%Ta client_cn:%{+Q}[ssl_c_s_dn(cn)] %r&#34;
    
    # Add client info headers
    http-request set-header X-Client-CN %{+Q}[ssl_c_s_dn(cn)]
    http-request set-header X-Client-DN %{+Q}[ssl_c_s_dn]
    http-request set-header X-Client-Serial %[ssl_c_serial,hex]
    http-request set-header X-Client-Verify %[ssl_c_verify]
    http-request set-header X-Client-Fingerprint %[ssl_c_sha1,hex]
    
    # Certificate-based authorization
    acl cn_admin ssl_c_s_dn(cn) -i admin-service
    acl cn_readonly ssl_c_s_dn(cn) -m sub readonly
    acl ou_internal ssl_c_s_dn(ou) -i internal
    acl ou_partner ssl_c_s_dn(ou) -i partner
    
    # Path-based ACLs
    acl admin_path path_beg /api/admin
    acl write_method method POST PUT DELETE PATCH
    
    # Authorization rules
    http-request deny if admin_path !cn_admin
    http-request deny if write_method cn_readonly
    http-request deny if !ou_internal !ou_partner
    
    # Route based on certificate
    use_backend admin_api if cn_admin
    use_backend partner_api if ou_partner
    default_backend internal_api

# Redirect to mTLS port
backend mtls_redirect
    http-request redirect code 301 location https://%[req.hdr(Host)]:8443%[capture.req.uri]

backend public_web
    balance roundrobin
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check

backend internal_api
    balance leastconn
    option httpchk GET /health
    http-check expect status 200
    server api1 192.168.1.20:8080 check
    server api2 192.168.1.21:8080 check

backend admin_api
    option httpchk GET /health
    server admin1 192.168.1.30:8080 check

backend partner_api
    balance roundrobin
    option httpchk GET /health
    server partner1 192.168.1.40:8080 check
    server partner2 192.168.1.41:8080 check
</code></pre><h2 id="testing-mtls">Testing mTLS <a class="anchor" data-pagefind-ignore href="#testing-mtls" aria-label="Link to this section">#</a></h2>
<p>Test both sides: a request with a certificate must work, one without must fail.</p>
<h3 id="test-with-curl">Test with curl <a class="anchor" data-pagefind-ignore href="#test-with-curl" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Test with client certificate</span>
</span></span><span class="line"><span class="cl">curl -v <span class="se">\
</span></span></span><span class="line"><span class="cl">  --cert /etc/haproxy/clients/api-client/api-client.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  --key /etc/haproxy/clients/api-client/api-client.key <span class="se">\
</span></span></span><span class="line"><span class="cl">  --cacert /etc/haproxy/ca/certs/ca.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  https://api.example.com:8443/api/status
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test with combined PEM</span>
</span></span><span class="line"><span class="cl">curl -v <span class="se">\
</span></span></span><span class="line"><span class="cl">  --cert /etc/haproxy/clients/api-client/api-client.pem <span class="se">\
</span></span></span><span class="line"><span class="cl">  --cacert /etc/haproxy/ca/certs/ca.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  https://api.example.com:8443/api/status
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test without certificate (should fail)</span>
</span></span><span class="line"><span class="cl">curl -v <span class="se">\
</span></span></span><span class="line"><span class="cl">  --cacert /etc/haproxy/ca/certs/ca.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  https://api.example.com:8443/api/status
</span></span></code></pre></div><h3 id="test-with-openssl">Test with openssl <a class="anchor" data-pagefind-ignore href="#test-with-openssl" aria-label="Link to this section">#</a></h3>
<p><code>s_client</code> shows the handshake: the server certificate, whether HAProxy asked for a client certificate, and the verify result.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Connect and show certificate exchange</span>
</span></span><span class="line"><span class="cl">openssl s_client <span class="se">\
</span></span></span><span class="line"><span class="cl">  -connect api.example.com:8443 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -cert /etc/haproxy/clients/api-client/api-client.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  -key /etc/haproxy/clients/api-client/api-client.key <span class="se">\
</span></span></span><span class="line"><span class="cl">  -CAfile /etc/haproxy/ca/certs/ca.crt <span class="se">\
</span></span></span><span class="line"><span class="cl">  -verify_return_error
</span></span></code></pre></div><h2 id="client-configuration-examples">Client Configuration Examples <a class="anchor" data-pagefind-ignore href="#client-configuration-examples" aria-label="Link to this section">#</a></h2>
<p>Every client needs the same three files: its certificate, its key, and the CA certificate to verify HAProxy.</p>
<h3 id="python-requests">Python requests <a class="anchor" data-pagefind-ignore href="#python-requests" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">requests</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s1">&#39;https://api.example.com:8443/api/data&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">cert</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;/path/to/client.crt&#39;</span><span class="p">,</span> <span class="s1">&#39;/path/to/client.key&#39;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="n">verify</span><span class="o">=</span><span class="s1">&#39;/path/to/ca.crt&#39;</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span></code></pre></div><h3 id="nodejs">Node.js <a class="anchor" data-pagefind-ignore href="#nodejs" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">https</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">&#39;https&#39;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">fs</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">&#39;fs&#39;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">options</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nx">hostname</span><span class="o">:</span> <span class="s1">&#39;api.example.com&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">port</span><span class="o">:</span> <span class="mi">8443</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">path</span><span class="o">:</span> <span class="s1">&#39;/api/data&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">method</span><span class="o">:</span> <span class="s1">&#39;GET&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">cert</span><span class="o">:</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s1">&#39;/path/to/client.crt&#39;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">key</span><span class="o">:</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s1">&#39;/path/to/client.key&#39;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">ca</span><span class="o">:</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s1">&#39;/path/to/ca.crt&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">req</span> <span class="o">=</span> <span class="nx">https</span><span class="p">.</span><span class="nx">request</span><span class="p">(</span><span class="nx">options</span><span class="p">,</span> <span class="p">(</span><span class="nx">res</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// Handle response
</span></span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span></code></pre></div><h3 id="go">Go <a class="anchor" data-pagefind-ignore href="#go" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;crypto/tls&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;crypto/x509&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;io/ioutil&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="s">&#34;net/http&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">cert</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">tls</span><span class="p">.</span><span class="nf">LoadX509KeyPair</span><span class="p">(</span><span class="s">&#34;client.crt&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;client.key&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">caCert</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ioutil</span><span class="p">.</span><span class="nf">ReadFile</span><span class="p">(</span><span class="s">&#34;ca.crt&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">caCertPool</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">x509</span><span class="p">.</span><span class="nf">NewCertPool</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">caCertPool</span><span class="p">.</span><span class="nf">AppendCertsFromPEM</span><span class="p">(</span><span class="nx">caCert</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">client</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">http</span><span class="p">.</span><span class="nx">Client</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nx">Transport</span><span class="p">:</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">http</span><span class="p">.</span><span class="nx">Transport</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nx">TLSClientConfig</span><span class="p">:</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">tls</span><span class="p">.</span><span class="nx">Config</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">Certificates</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nx">tls</span><span class="p">.</span><span class="nx">Certificate</span><span class="p">{</span><span class="nx">cert</span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nx">RootCAs</span><span class="p">:</span><span class="w">      </span><span class="nx">caCertPool</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="s">&#34;https://api.example.com:8443/api/data&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">// Handle response</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="common-errors">Common Errors <a class="anchor" data-pagefind-ignore href="#common-errors" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Error</th>
					<th>Cause</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>tlsv1 alert unknown ca</code></td>
					<td>Client certificate not signed by the CA in <code>ca-file</code></td>
			</tr>
			<tr>
					<td><code>SSL certificate verify failed</code></td>
					<td>CA certificate mismatch</td>
			</tr>
			<tr>
					<td><code>tlsv13 alert certificate required</code> (TLS 1.2: handshake failure)</td>
					<td>Client didn&rsquo;t provide certificate when required</td>
			</tr>
			<tr>
					<td><code>ssl/tls alert certificate revoked</code></td>
					<td>Certificate is in CRL</td>
			</tr>
	</tbody>
</table>
<h3 id="debug-commands">Debug Commands <a class="anchor" data-pagefind-ignore href="#debug-commands" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Verify certificate chain</span>
</span></span><span class="line"><span class="cl">openssl verify -CAfile ca.crt client.crt
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check certificate details</span>
</span></span><span class="line"><span class="cl">openssl x509 -in client.crt -text -noout
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check CRL contents</span>
</span></span><span class="line"><span class="cl">openssl crl -in ca.crl -text -noout
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test SSL connection with debug</span>
</span></span><span class="line"><span class="cl">openssl s_client -connect api.example.com:8443 -debug
</span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Protect the CA key.</strong> Whoever has it can mint valid client certificates. Keep it encrypted and the CA host locked down.</li>
<li><strong>Short client lifetimes.</strong> 365 days here; shorter limits the damage of a leaked key.</li>
<li><strong>Keep the CRL fresh.</strong> Run the hourly update; an expired CRL locks everyone out.</li>
<li><strong>Do not trust client headers.</strong> Always overwrite <code>X-Client-*</code> in HAProxy, as above, and make backends reachable only through HAProxy.</li>
<li><strong>Change the PKCS12 password</strong> from <code>changeme</code> in the client script.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-mtls" rel="noopener">pderuiter/blog-haproxy-mtls</a>. It creates an HAProxy VM on Proxmox, builds the internal CA with server and client certificates, and configures the complete production configuration from this post with certificate-based authorization and the hourly CRL update; put the CA passphrase in <code>vault.yml</code>, run <code>terraform apply</code>, then <code>ansible-playbook site.yml</code>.</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>Next in the series: the HAProxy Data Plane API for dynamic configuration management.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<p>Key points for HAProxy mTLS:</p>
<ul>
<li>Use <code>verify required</code> for mandatory client certificates</li>
<li>Use <code>verify optional</code> for mixed authentication</li>
<li>Pass certificate details to backends via headers</li>
<li>Implement authorization based on CN, OU, or fingerprint</li>
<li>Maintain CRL for certificate revocation</li>
<li>Generate unique certificates for each client/service</li>
<li>Test thoroughly with curl and openssl</li>
</ul>
<p>mTLS gives strong mutual authentication without passwords, a good fit for service-to-service communication and API access.</p>
]]></content:encoded></item><item><title>Securing HAProxy with Coraza-SPOA: A Step-by-Step Guide</title><link>https://blog.bsdserver.nl/securing-haproxy-with-coraza-spoa-a-step-by-step-guide/</link><guid isPermaLink="false">693b50e879eb54000160a470</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Thu, 11 Dec 2025 23:16:56 +0000</pubDate><description>Coraza-SPOA offers a powerful way to add Web Application Firewall (WAF) capabilities to HAProxy, enhancing security for your web applications.</description><category>HAProxy</category><category>Security</category><category>WAF</category><category>Coraza</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-coraza-waf.png" medium="image"/><content:encoded><![CDATA[<p>Coraza-SPOA offers a powerful way to add Web Application Firewall (WAF) capabilities to HAProxy. HAProxy routes and balances requests but does not look for SQL injection, XSS or path traversal in them; a WAF does.</p>
<p>This guide compiles Coraza-SPOA, loads the OWASP Core Rule Set, runs it as a systemd service, and connects HAProxy to it so that malicious requests get a 403 before they reach your application.</p>
<h2 id="why-coraza-spoa">Why Coraza-SPOA? <a class="anchor" data-pagefind-ignore href="#why-coraza-spoa" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>OWASP Core Rule Set</strong>: a maintained, widely used rule set against common web attacks</li>
<li><strong>ModSecurity-compatible rules</strong>: Coraza reads the SecLang rule language</li>
<li><strong>Separate process</strong>: the WAF runs outside HAProxy; HAProxy only asks for a verdict</li>
<li><strong>Open source</strong>: no commercial HAProxy edition needed</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> in front of web applications you cannot fully trust or patch quickly: CMSs, legacy apps, third-party software.</li>
<li><strong>Skip it</strong> for pure APIs with strict input validation, or traffic that is not HTTP. Every request takes an extra round trip to the agent, and the CRS needs tuning against false positives.</li>
</ul>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy</td>
					<td>Load balancer</td>
					<td>Receives requests; the SPOE filter sends each one to the agent and acts on the verdict.</td>
			</tr>
			<tr>
					<td>Coraza-SPOA</td>
					<td>Same host, <code>127.0.0.1:9000</code></td>
					<td>Evaluates the request against the rules and returns an action.</td>
			</tr>
			<tr>
					<td>Core Rule Set</td>
					<td><code>/etc/coraza-spoa/rules/</code></td>
					<td>The attack-detection rules Coraza loads.</td>
			</tr>
			<tr>
					<td>Backend</td>
					<td>App servers</td>
					<td>Only receives requests Coraza allowed. Here: a static test response.</td>
			</tr>
	</tbody>
</table>
<p>The flow: request arrives → SPOE sends method, path, headers and body to Coraza-SPOA → Coraza runs the rules → HAProxy reads <code>txn.coraza.action</code> → <code>deny</code> gives a 403, otherwise the request goes to the backend.</p>
<h2 id="step-1-clone-and-compile-coraza-spoa">Step 1: Clone and Compile Coraza-SPOA <a class="anchor" data-pagefind-ignore href="#step-1-clone-and-compile-coraza-spoa" aria-label="Link to this section">#</a></h2>
<p>There are no packages, so you build the binary from source. Make sure you have Go installed before proceeding. The project builds with <a href="https://magefile.org/" rel="noopener">Mage</a> through <code>go run</code>, so nothing else needs to be installed.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Clone the Coraza-SPOA repo</span>
</span></span><span class="line"><span class="cl">git clone https://github.com/corazawaf/coraza-spoa.git
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Change into the directory and compile the binary (needs Go 1.25+;</span>
</span></span><span class="line"><span class="cl"><span class="c1"># the result is build/coraza-spoa)</span>
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> coraza-spoa
</span></span><span class="line"><span class="cl">go run mage.go build
</span></span></code></pre></div><h2 id="step-2-install-the-binary-and-set-permissions">Step 2: Install the Binary and Set Permissions <a class="anchor" data-pagefind-ignore href="#step-2-install-the-binary-and-set-permissions" aria-label="Link to this section">#</a></h2>
<p>Copy the binary to the path the bundled systemd unit expects, owned by root so the service user cannot replace it.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Place the binary in /usr/bin (the path contrib/coraza-spoa.service starts;</span>
</span></span><span class="line"><span class="cl"><span class="c1"># that unit also makes /usr/local inaccessible)</span>
</span></span><span class="line"><span class="cl">cp -p build/coraza-spoa /usr/bin/coraza-spoa
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Set ownership and permissions</span>
</span></span><span class="line"><span class="cl">chown root:root /usr/bin/coraza-spoa
</span></span><span class="line"><span class="cl">chmod <span class="m">755</span> /usr/bin/coraza-spoa
</span></span></code></pre></div><h2 id="step-3-create-directory-structure-for-logs-and-configuration">Step 3: Create Directory Structure for Logs and Configuration <a class="anchor" data-pagefind-ignore href="#step-3-create-directory-structure-for-logs-and-configuration" aria-label="Link to this section">#</a></h2>
<p>Coraza-SPOA reads its configuration from <code>/etc/coraza-spoa</code> and writes logs to <code>/var/log/coraza-spoa</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Create directories</span>
</span></span><span class="line"><span class="cl">mkdir -p /etc/coraza-spoa
</span></span><span class="line"><span class="cl">mkdir -p /var/log/coraza-spoa/audit
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create necessary log files</span>
</span></span><span class="line"><span class="cl">touch /var/log/coraza-spoa/server.log
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Repeat for all required log files</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> logfile in error.log debug.log audit/audit.log<span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    touch /var/log/coraza-spoa/<span class="nv">$logfile</span>
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span></code></pre></div><h2 id="step-4-add-user-and-set-permissions">Step 4: Add User and Set Permissions <a class="anchor" data-pagefind-ignore href="#step-4-add-user-and-set-permissions" aria-label="Link to this section">#</a></h2>
<p>For security, Coraza-SPOA runs under its own user: a system account without a home directory or password, so a compromise of the WAF does not give a login.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Add a system group and user</span>
</span></span><span class="line"><span class="cl">addgroup --quiet --system coraza-spoa
</span></span><span class="line"><span class="cl">adduser --quiet --system --ingroup coraza-spoa --no-create-home --home /nonexistent --disabled-password coraza-spoa
</span></span></code></pre></div><p>There is no need to <code>chown</code> the log directory. The systemd unit from Step 7 has <code>LogsDirectory=coraza-spoa</code>, so every time the service starts, systemd makes <code>/var/log/coraza-spoa</code> and the files in it owned by <code>coraza-spoa:coraza-spoa</code> with mode 0755. A different owner or group set by hand would be undone at the next start.</p>
<h2 id="step-5-configure-coraza-spoa">Step 5: Configure Coraza-SPOA <a class="anchor" data-pagefind-ignore href="#step-5-configure-coraza-spoa" aria-label="Link to this section">#</a></h2>
<p>This file tells the agent where to listen and which rules to load.</p>
<ul>
<li><code>bind: 127.0.0.1:9000</code>: the port HAProxy connects to. Localhost only.</li>
<li><code>applications</code>: a named rule set. HAProxy picks it per request (<code>app=str(haproxy_waf)</code> in Step 9); <code>default_application</code> is used otherwise.</li>
<li><code>directives</code>: the rules, loaded in order: Coraza base settings, CRS setup, CRS plugins, then the CRS rules.</li>
<li><code>response_check: false</code>: only requests are inspected.</li>
<li><code>transaction_ttl_ms</code>: how long the agent keeps a transaction&rsquo;s state.</li>
</ul>
<p>Edit <code>/etc/coraza-spoa/config.yaml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># Bind address</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">bind</span><span class="p">:</span><span class="w"> </span><span class="m">127.0.0.1</span><span class="p">:</span><span class="m">9000</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">log_level</span><span class="p">:</span><span class="w"> </span><span class="l">info</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">log_file</span><span class="p">:</span><span class="w"> </span><span class="l">/var/log/coraza-spoa/server.log</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">log_format</span><span class="p">:</span><span class="w"> </span><span class="l">console</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">default_application</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_waf</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">applications</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_waf</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">directives</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      Include /etc/coraza-spoa/coraza.conf
</span></span></span><span class="line"><span class="cl"><span class="sd">      Include /etc/coraza-spoa/crs-setup.conf
</span></span></span><span class="line"><span class="cl"><span class="sd">      Include /etc/coraza-spoa/plugins/*-config.conf
</span></span></span><span class="line"><span class="cl"><span class="sd">      Include /etc/coraza-spoa/rules/*.conf</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">response_check</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">transaction_ttl_ms</span><span class="p">:</span><span class="w"> </span><span class="m">60000</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">log_level</span><span class="p">:</span><span class="w"> </span><span class="l">info</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">log_file</span><span class="p">:</span><span class="w"> </span><span class="l">/var/log/coraza-spoa/server.log</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">log_format</span><span class="p">:</span><span class="w"> </span><span class="l">console</span><span class="w">
</span></span></span></code></pre></div><h2 id="step-6-configure-coraza-rules">Step 6: Configure Coraza Rules <a class="anchor" data-pagefind-ignore href="#step-6-configure-coraza-rules" aria-label="Link to this section">#</a></h2>
<p>Create <code>/etc/coraza-spoa/coraza.conf</code> from the Coraza recommended configuration and add the OWASP Core Rule Set.</p>
<table>
	<thead>
			<tr>
					<th>Mode</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>SecRuleEngine DetectionOnly</code></td>
					<td>A new site: log matches, block nothing, tune false positives first</td>
			</tr>
			<tr>
					<td><code>SecRuleEngine On</code></td>
					<td>Blocking mode, once the rules fit your application (the <code>sed</code> below sets this)</td>
			</tr>
	</tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Coraza base configuration; switch DetectionOnly to On for blocking mode</span>
</span></span><span class="line"><span class="cl">wget -O /etc/coraza-spoa/coraza.conf <span class="se">\
</span></span></span><span class="line"><span class="cl">  https://raw.githubusercontent.com/corazawaf/coraza/v3.8.1/coraza.conf-recommended
</span></span><span class="line"><span class="cl">sed -i <span class="s1">&#39;s/^SecRuleEngine DetectionOnly/SecRuleEngine On/&#39;</span> /etc/coraza-spoa/coraza.conf
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># OWASP Core Rule Set v4 (the version bundled with current coraza-spoa)</span>
</span></span><span class="line"><span class="cl">git clone --depth <span class="m">1</span> --branch v4.25.0 https://github.com/coreruleset/coreruleset.git
</span></span><span class="line"><span class="cl">cp -p coreruleset/crs-setup.conf.example /etc/coraza-spoa/crs-setup.conf
</span></span><span class="line"><span class="cl">cp -Rp coreruleset/rules /etc/coraza-spoa/
</span></span></code></pre></div><h2 id="step-7-system-configuration">Step 7: System Configuration <a class="anchor" data-pagefind-ignore href="#step-7-system-configuration" aria-label="Link to this section">#</a></h2>
<p>The repository ships a systemd unit and a logrotate file in <code>contrib/</code>. The unit runs the agent as <code>coraza-spoa</code> and creates the log directory; logrotate keeps the logs from filling the disk.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Copy log rotation configuration and systemd unit files</span>
</span></span><span class="line"><span class="cl">cp -Rp contrib/coraza-spoa.logrotate /etc/logrotate.d/coraza-spoa
</span></span><span class="line"><span class="cl">cp -Rp contrib/coraza-spoa.service /lib/systemd/system/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Reload systemd and enable the Coraza-SPOA agent</span>
</span></span><span class="line"><span class="cl">systemctl daemon-reload
</span></span><span class="line"><span class="cl">systemctl <span class="nb">enable</span> coraza-spoa.service
</span></span><span class="line"><span class="cl">systemctl start coraza-spoa.service
</span></span></code></pre></div><h2 id="step-8-configure-haproxy">Step 8: Configure HAProxy <a class="anchor" data-pagefind-ignore href="#step-8-configure-haproxy" aria-label="Link to this section">#</a></h2>
<p>If HAProxy is not installed yet, install it from the official HAProxy packages rather than the distribution&rsquo;s older <code>haproxy</code> package:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Official HAProxy packages for Ubuntu 24.04: one PPA per branch</span>
</span></span><span class="line"><span class="cl">sudo add-apt-repository -y ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">sudo apt update
</span></span><span class="line"><span class="cl">sudo apt install -y <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">haproxy -v
</span></span></code></pre></div><p>3.3 is the newest branch the PPA builds for Ubuntu 24.04, but it is a short-lived stable branch, maintained until about Q1 2027. The newest LTS branch, 3.4, is built for Ubuntu 26.04 (<code>ppa:vbernat/haproxy-3.4</code>) and for Debian 13 on <a href="https://haproxy.debian.net/" rel="noopener">haproxy.debian.net</a>, which lists the repository lines per release and branch. For a long-lived Ubuntu 24.04 install, use the 3.2 LTS branch instead (<code>ppa:vbernat/haproxy-3.2</code>, maintained until 2030).</p>
<p>HAProxy configuration consists of three main steps:</p>
<ol>
<li>Create the frontend.</li>
<li>Create backends for Coraza-SPOA and static content.</li>
<li>Configure the SPOE engine.</li>
</ol>
<p>What the frontend does:</p>
<ul>
<li><code>filter spoe engine coraza</code>: sends every request to the agent, using the SPOE file from Step 9.</li>
<li>First <code>deny</code>: Coraza&rsquo;s verdict is <code>deny</code> → 403.</li>
<li>Second <code>deny</code>: the agent could not be reached or timed out → 500. The WAF fails closed: no verdict, no access.</li>
<li><code>backend coraza-spoa</code>: the agent itself, in <code>mode tcp</code> because SPOP is not HTTP.</li>
</ul>
<p>Add the following frontend and backend configurations in <code>/etc/haproxy/haproxy.cfg</code>:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend static_test
    mode http
    bind *:80
    filter spoe engine coraza config /etc/haproxy/coraza.cfg
    # Act on the verdict Coraza returns in txn.coraza.action
    http-request deny deny_status 403 if { var(txn.coraza.action) -m str deny }
    http-request deny deny_status 500 if { var(txn.coraza.error) -m int gt 0 }
    use_backend bsdserver_static

backend bsdserver_static
    mode http
    http-request return status 200 content-type &#34;text/plain&#34; string &#34;Welcome!\n&#34;

backend coraza-spoa
    mode tcp
    balance roundrobin
    server s1 127.0.0.1:9000
</code></pre><h2 id="step-9-spoe-configuration-for-haproxy">Step 9: SPOE Configuration for HAProxy <a class="anchor" data-pagefind-ignore href="#step-9-spoe-configuration-for-haproxy" aria-label="Link to this section">#</a></h2>
<p>SPOE (Stream Processing Offload Engine) is how HAProxy hands request data to an external agent. This file says what to send and where the answer goes.</p>
<ul>
<li><code>option var-prefix coraza</code>: the agent&rsquo;s answers become <code>txn.coraza.*</code> variables, such as <code>txn.coraza.action</code>.</li>
<li><code>option set-on-error error</code>: sets <code>txn.coraza.error</code> when the agent fails, which the frontend turns into a 500.</li>
<li><code>timeout processing 500ms</code>: how long HAProxy waits for a verdict.</li>
<li><code>args</code>: the request parts Coraza inspects, including headers and body.</li>
<li><code>event on-frontend-http-request</code>: send the message for every request in the frontend.</li>
</ul>
<p>Edit <code>/etc/haproxy/coraza.cfg</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-cfg" data-lang="cfg"><span class="line"><span class="cl"><span class="k">[coraza]</span>
</span></span><span class="line"><span class="cl"><span class="na">spoe-agent coraza-agent</span>
</span></span><span class="line"><span class="cl">    <span class="na">messages coraza-req</span>
</span></span><span class="line"><span class="cl">    <span class="na">option var-prefix coraza</span>
</span></span><span class="line"><span class="cl">    <span class="na">option set-on-error error</span>
</span></span><span class="line"><span class="cl">    <span class="na">timeout processing 500ms</span>
</span></span><span class="line"><span class="cl">    <span class="na">use-backend coraza-spoa</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="na">spoe-message coraza-req</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Arguments are required to be in this order</span>
</span></span><span class="line"><span class="cl">    <span class="na">args app</span><span class="o">=</span><span class="s">str(haproxy_waf) src-ip=src src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
</span></span></span><span class="line"><span class="cl"><span class="s">    event on-frontend-http-request</span>
</span></span></code></pre></div><h2 id="step-10-validate-and-start-haproxy">Step 10: Validate and Start HAProxy <a class="anchor" data-pagefind-ignore href="#step-10-validate-and-start-haproxy" aria-label="Link to this section">#</a></h2>
<p>Check the configuration before you restart HAProxy.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">haproxy -c -V -f /etc/haproxy/haproxy.cfg
</span></span></code></pre></div><p>If there are no errors, enable and restart HAProxy:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># Enable HAProxy</span>
</span></span><span class="line"><span class="cl">systemctl <span class="nb">enable</span> haproxy.service
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Restart HAProxy (the Ubuntu/Debian package already started it with the</span>
</span></span><span class="line"><span class="cl"><span class="c1"># default configuration, so &#34;start&#34; would not load the new one)</span>
</span></span><span class="line"><span class="cl">systemctl restart haproxy.service
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Verify status</span>
</span></span><span class="line"><span class="cl">systemctl status haproxy.service
</span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Every request gets a 500</td>
					<td>HAProxy cannot reach the agent: check <code>systemctl status coraza-spoa</code> and that it listens on <code>127.0.0.1:9000</code></td>
			</tr>
			<tr>
					<td>Legitimate requests get a 403</td>
					<td>A CRS false positive: check the Coraza logs for the rule ID; run in <code>DetectionOnly</code> while tuning</td>
			</tr>
			<tr>
					<td>Agent does not start</td>
					<td>A rule file fails to load: check the agent&rsquo;s log in <code>/var/log/coraza-spoa/</code></td>
			</tr>
	</tbody>
</table>
<h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Fail closed.</strong> The <code>txn.coraza.error</code> rule blocks requests when the WAF is down. Remove it only if availability matters more than protection.</li>
<li><strong>Agent on localhost.</strong> Keep <code>bind: 127.0.0.1:9000</code>; anyone who can talk to the agent can see request data.</li>
<li><strong>Keep the CRS current.</strong> Update the rules with new CRS releases.</li>
<li><strong>A WAF is an extra layer.</strong> It does not replace patching and input validation in the application.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-coraza" rel="noopener">pderuiter/blog-haproxy-coraza</a>. It creates an Ubuntu 24.04 VM, installs HAProxy from the official packages, builds Coraza-SPOA with the Core Rule Set and wires it into HAProxy; <code>terraform apply</code> followed by <code>ansible-playbook site.yml</code> gives you the blocking WAF from this guide.</p>
<h2 id="conclusion">Conclusion <a class="anchor" data-pagefind-ignore href="#conclusion" aria-label="Link to this section">#</a></h2>
<p>You&rsquo;ve now set up Coraza-SPOA integrated with HAProxy: a WAF that checks every request against the Core Rule Set before it reaches your application. Coraza can inspect both incoming and outgoing traffic; this setup checks requests (response inspection needs <code>response_check: true</code> and a <code>coraza-res</code> SPOE message).</p>
<p>For more information and additional configurations, visit:</p>
<ul>
<li><a href="https://github.com/corazawaf/coraza-spoa" rel="noopener">Coraza-SPOA GitHub</a></li>
<li><a href="https://github.com/haproxy/haproxy/blob/master/doc/SPOE.txt" rel="noopener">HAProxy Documentation for SPOE</a></li>
</ul>
<p>Start in detection mode on a real site, tune the false positives, then switch to blocking.</p>
]]></content:encoded></item><item><title>HAProxy DNS Load Balancing: High Availability for DNS Infrastructure</title><link>https://blog.bsdserver.nl/haproxy-dns-load-balancing-high-availability-for-dns-infrastructure/</link><guid isPermaLink="false">693b727a79eb54000160a5e9</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:40:11 +0000</pubDate><description>DNS is critical infrastructure - when DNS fails, nothing works.</description><category>HAProxy</category><category>DNS</category><category>Load balancing</category><category>High availability</category><category>Networking</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-dns-loadbalancing.png" medium="image"/><content:encoded><![CDATA[<p>DNS is critical infrastructure - when DNS fails, nothing works. Clients usually have two resolver addresses at most, and a dead server behind one of them means slow lookups everywhere.</p>
<p>This guide puts a load balancer in front of resolvers and authoritative servers. HAProxy handles DNS over TCP and DNS over TLS; dnsdist handles DNS over UDP next to it, because HAProxy cannot proxy UDP. keepalived makes the load balancer itself redundant.</p>
<h2 id="why-load-balance-dns">Why Load Balance DNS? <a class="anchor" data-pagefind-ignore href="#why-load-balance-dns" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>High availability</strong>: If one DNS server fails, queries go to the healthy ones</li>
<li><strong>Performance</strong>: Spread queries over several servers</li>
<li><strong>Maintenance flexibility</strong>: Take a server offline without clients noticing</li>
<li><strong>Both protocols</strong>: UDP and TCP on one address</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when many clients depend on a few DNS servers, or when you want one stable address in front of a changing set of servers.</li>
<li><strong>Skip it</strong> when clients already list two or three resolvers and can live with a short timeout on failure. The load balancer pair is two more machines to run.</li>
</ul>
<h2 id="dns-protocol-considerations">DNS Protocol Considerations <a class="anchor" data-pagefind-ignore href="#dns-protocol-considerations" aria-label="Link to this section">#</a></h2>
<p>DNS primarily uses UDP on port 53, but can fall back to TCP for:</p>
<ul>
<li>Responses larger than 512 bytes (or EDNS0 buffer size)</li>
<li>Zone transfers (AXFR/IXFR)</li>
<li>DNS over TCP enforcement</li>
</ul>
<p>HAProxy only handles the TCP side. The community edition has no UDP proxy mode: <code>bind ... udp</code> and <code>mode udp</code> are rejected as unknown keywords, also in HAProxy 3.4. So the work is split:</p>
<table>
	<thead>
			<tr>
					<th>Traffic</th>
					<th>Handled by</th>
					<th>Why</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>UDP port 53</td>
					<td><a href="https://www.dnsdist.org/" rel="noopener">dnsdist</a></td>
					<td>A DNS-aware load balancer from PowerDNS; HAProxy has no UDP proxy</td>
			</tr>
			<tr>
					<td>TCP port 53</td>
					<td>HAProxy</td>
					<td>Large responses, zone transfers</td>
			</tr>
			<tr>
					<td>TCP port 853 (DoT)</td>
					<td>HAProxy</td>
					<td>TLS termination or passthrough</td>
			</tr>
	</tbody>
</table>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>dnsdist</td>
					<td>Both load balancers</td>
					<td>Balances UDP queries, health-checks the servers, rate limits per client.</td>
			</tr>
			<tr>
					<td>nftables</td>
					<td>Both load balancers</td>
					<td>Redirects UDP port 53 to dnsdist on port 5353, because TCP port 53 belongs to HAProxy.</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td>Both load balancers</td>
					<td>Balances DNS over TCP and DoT.</td>
			</tr>
			<tr>
					<td>keepalived</td>
					<td>Both load balancers</td>
					<td>Moves the virtual IPs to the backup when HAProxy or dnsdist stops.</td>
			</tr>
			<tr>
					<td>DNS servers</td>
					<td>Separate machines</td>
					<td>Resolvers (<code>192.168.1.10-12</code>) and authoritative servers (<code>192.168.1.20-22</code>).</td>
			</tr>
	</tbody>
</table>
<p>The flow: client → virtual IP → UDP: nftables → dnsdist, TCP: HAProxy → healthy DNS server.</p>
<h2 id="basic-udp-dns-load-balancing">Basic UDP DNS Load Balancing <a class="anchor" data-pagefind-ignore href="#basic-udp-dns-load-balancing" aria-label="Link to this section">#</a></h2>
<p>dnsdist runs on the same machine as HAProxy, so clients use one address for both protocols.</p>
<p>Install dnsdist from the official PowerDNS repository (dnsdist 2.1 on Ubuntu 24.04):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">install -d /etc/apt/keyrings
</span></span><span class="line"><span class="cl">curl https://repo.powerdns.com/FD380FBB-pub.asc -o /etc/apt/keyrings/dnsdist-21-pub.asc
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;deb [signed-by=/etc/apt/keyrings/dnsdist-21-pub.asc] http://repo.powerdns.com/ubuntu noble-dnsdist-21 main&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  &gt; /etc/apt/sources.list.d/pdns.list
</span></span><span class="line"><span class="cl">cat &gt; /etc/apt/preferences.d/dnsdist <span class="s">&lt;&lt; EOF
</span></span></span><span class="line"><span class="cl"><span class="s">Package: dnsdist*
</span></span></span><span class="line"><span class="cl"><span class="s">Pin: origin repo.powerdns.com
</span></span></span><span class="line"><span class="cl"><span class="s">Pin-Priority: 600
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install dnsdist nftables
</span></span></code></pre></div><p>On Debian 13 use <code>http://repo.powerdns.com/debian trixie-dnsdist-21 main</code> instead.</p>
<p>dnsdist always listens on UDP and TCP together, and TCP port 53 belongs to HAProxy. So dnsdist listens on port 5353, and an nftables rule hands it the UDP traffic for port 53.</p>
<ul>
<li><code>addLocal</code>: the listen address.</li>
<li><code>newServer</code>: one line per DNS server.</li>
<li><code>setServerPolicy(roundrobin)</code>: queries in turn over all healthy servers.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- UDP port 53 is redirected here by nftables; HAProxy owns TCP port 53</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- dnsdist only answers private (RFC1918) clients by default; widen with setACL() if needed</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.10:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;dns1&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.11:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;dns2&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.12:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;dns3&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">setServerPolicy</span><span class="p">(</span><span class="n">roundrobin</span><span class="p">)</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/nftables.conf (add below the existing tables)</span>
</span></span><span class="line"><span class="cl">table ip dns_udp <span class="o">{</span>
</span></span><span class="line"><span class="cl">    chain prerouting <span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="nb">type</span> nat hook prerouting priority dstnat<span class="p">;</span> policy accept<span class="p">;</span>
</span></span><span class="line"><span class="cl">        udp dport <span class="m">53</span> redirect to :5353
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><p>Load both and start dnsdist:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">dnsdist --check-config
</span></span><span class="line"><span class="cl">systemctl <span class="nb">enable</span> --now nftables
</span></span><span class="line"><span class="cl">systemctl restart dnsdist
</span></span></code></pre></div><p><strong>Warning before you enable <code>nftables.service</code>:</strong> the stock <code>/etc/nftables.conf</code> on Debian and Ubuntu starts with <code>flush ruleset</code>. Loading it removes every other nftables rule on the host, including the iptables-nft rules that Docker, ufw and firewalld create.</p>
<ul>
<li>Run this on dedicated load balancer machines, or leave <code>flush ruleset</code> out and load the <code>dns_udp</code> table from its own file.</li>
<li>The companion repo below adds the table to <code>/etc/nftables.conf</code> in the same way, so it is meant for dedicated load balancer VMs and its README warns about this.</li>
</ul>
<p>What dnsdist does without extra configuration:</p>
<ul>
<li><strong>Health checks</strong>: every server once a second, with a query for <code>a.root-servers.net</code>. Any answer except SERVFAIL counts as up.</li>
<li><strong>Failover</strong>: failing servers leave the rotation until they answer again.</li>
<li><strong>Client addresses</strong>: kept, so per-client rate limits (below) keep working.</li>
</ul>
<h2 id="tcp-dns-load-balancing">TCP DNS Load Balancing <a class="anchor" data-pagefind-ignore href="#tcp-dns-load-balancing" aria-label="Link to this section">#</a></h2>
<p>HAProxy takes TCP port 53 in <code>mode tcp</code>: it passes the connection through without parsing DNS.</p>
<ul>
<li><code>option tcp-check</code> with <code>check</code>: a server is up when it accepts a TCP connection.</li>
<li><code>timeout client</code> / <code>timeout server 30s</code>: room for zone transfers and slow large answers.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_tcp
    bind *:53
    mode tcp
    option tcplog
    timeout client 30s
    default_backend dns_servers_tcp

backend dns_servers_tcp
    mode tcp
    balance roundrobin
    timeout server 30s
    timeout connect 5s
    option tcp-check
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
    server dns3 192.168.1.12:53 check
</code></pre><h2 id="combined-udp-and-tcp-configuration">Combined UDP and TCP Configuration <a class="anchor" data-pagefind-ignore href="#combined-udp-and-tcp-configuration" aria-label="Link to this section">#</a></h2>
<p>A production DNS load balancer should handle both protocols. dnsdist from the previous section takes UDP; HAProxy takes TCP, with health checks and statistics.</p>
<ul>
<li><code>defaults dns</code>: short timeouts (DNS answers fast) and <code>default-server inter 3s fall 3 rise 2</code>: check every 3 seconds, down after 3 failures, up after 2 successes.</li>
<li><code>listen stats</code>: stats page on <code>:8404/stats</code>, Prometheus metrics on <code>:8404/metrics</code>.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    maxconn 100000
    nbthread 4

defaults dns
    log global
    timeout connect 1s
    timeout client  5s
    timeout server  5s
    default-server inter 3s fall 3 rise 2

# UDP DNS is handled by dnsdist (see above)

# TCP DNS
frontend dns_tcp_frontend
    bind *:53
    mode tcp
    option tcplog
    default_backend dns_tcp_backend

backend dns_tcp_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
    server dns3 192.168.1.12:53 check

# Stats interface
listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 5s
    http-request use-service prometheus-exporter if { path /metrics }
</code></pre><h2 id="dns-health-checks">DNS Health Checks <a class="anchor" data-pagefind-ignore href="#dns-health-checks" aria-label="Link to this section">#</a></h2>
<p>Pick the check by how much you need to know about the server:</p>
<table>
	<thead>
			<tr>
					<th>Check</th>
					<th>Proves</th>
					<th>Cost</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>TCP connect</td>
					<td>The port accepts connections</td>
					<td>Lowest</td>
			</tr>
			<tr>
					<td>DNS query in <code>tcp-check</code></td>
					<td>The server answers a real query</td>
					<td>Hand-built packet</td>
			</tr>
			<tr>
					<td>External script</td>
					<td>Anything a script can test</td>
					<td>A process per check; needs <code>insecure-fork-wanted</code></td>
			</tr>
	</tbody>
</table>
<h3 id="basic-tcp-health-check">Basic TCP Health Check <a class="anchor" data-pagefind-ignore href="#basic-tcp-health-check" aria-label="Link to this section">#</a></h3>
<p>The simplest check verifies the TCP port is responding. It does not notice a server that accepts connections but cannot answer.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_tcp_backend
    mode tcp
    option tcp-check
    server dns1 192.168.1.10:53 check inter 2s fall 3 rise 2
</code></pre><h3 id="dns-specific-health-check">DNS-Specific Health Check <a class="anchor" data-pagefind-ignore href="#dns-specific-health-check" aria-label="Link to this section">#</a></h3>
<p>For more thorough checking, verify the server can resolve queries. The check sends a query for <code>health.local</code> and expects a response with no error.</p>
<ul>
<li>Every server needs a <code>health.local</code> A record: an NXDOMAIN answer does not match and marks the server down.</li>
<li>The 2-byte length prefix comes first because DNS over TCP requires it.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_tcp_backend
    mode tcp
    option tcp-check
    
    # DNS query for &#34;health.local&#34; A record
    # This is a raw DNS query packet (DNS over TCP needs the 2-byte length prefix)
    tcp-check connect port 53
    tcp-check send-binary 001e  # Length: 30-byte query follows
    tcp-check send-binary 0000  # Transaction ID
    tcp-check send-binary 0100  # Flags: Standard query
    tcp-check send-binary 0001  # Questions: 1
    tcp-check send-binary 0000  # Answer RRs: 0
    tcp-check send-binary 0000  # Authority RRs: 0
    tcp-check send-binary 0000  # Additional RRs: 0
    tcp-check send-binary 066865616c7468  # &#34;health&#34;
    tcp-check send-binary 056c6f63616c    # &#34;local&#34;
    tcp-check send-binary 00              # End of name
    tcp-check send-binary 0001  # Type: A
    tcp-check send-binary 0001  # Class: IN
    tcp-check expect rbinary ^[0-9A-Fa-f]{8}8[15]80  # Response flags: answer, no error (AA bit optional)
    
    server dns1 192.168.1.10:53 check
</code></pre><h3 id="external-health-check-script">External Health Check Script <a class="anchor" data-pagefind-ignore href="#external-health-check-script" aria-label="Link to this section">#</a></h3>
<p>For complex health checks, use an external script. This needs <code>external-check</code> and <code>insecure-fork-wanted</code> in the <code>global</code> section, otherwise HAProxy refuses the configuration.</p>
<p><code>insecure-fork-wanted</code> lets HAProxy start processes, which it blocks by default for safety. Only use it when the other checks are not enough.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    external-check
    insecure-fork-wanted

backend dns_tcp_backend
    mode tcp
    option external-check
    external-check path &#34;/usr/bin:/bin&#34;
    external-check command /etc/haproxy/check-dns.sh
    server dns1 192.168.1.10:53 check inter 5s
</code></pre><p>Create the health check script. HAProxy passes the server address and port; exit code 0 means up. The script resolves <code>google.com</code>, so it suits resolvers; for authoritative servers, query a zone they serve.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /etc/haproxy/check-dns.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Arguments from HAProxy:</span>
</span></span><span class="line"><span class="cl"><span class="c1"># $1 = proxy address, $2 = proxy port (NOT_USED for a backend)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># $3 = server address</span>
</span></span><span class="line"><span class="cl"><span class="c1"># $4 = server port</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">SERVER</span><span class="o">=</span><span class="nv">$3</span>
</span></span><span class="line"><span class="cl"><span class="nv">PORT</span><span class="o">=</span><span class="nv">$4</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Try to resolve a known domain</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> dig @<span class="si">${</span><span class="nv">SERVER</span><span class="si">}</span> -p <span class="si">${</span><span class="nv">PORT</span><span class="si">}</span> +time<span class="o">=</span><span class="m">2</span> +tries<span class="o">=</span><span class="m">1</span> google.com A &gt; /dev/null 2&gt;<span class="p">&amp;</span>1<span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    <span class="nb">exit</span> <span class="m">0</span>  <span class="c1"># Success</span>
</span></span><span class="line"><span class="cl"><span class="k">else</span>
</span></span><span class="line"><span class="cl">    <span class="nb">exit</span> <span class="m">1</span>  <span class="c1"># Failure</span>
</span></span><span class="line"><span class="cl"><span class="k">fi</span>
</span></span></code></pre></div><p>Make it executable:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">chmod +x /etc/haproxy/check-dns.sh
</span></span></code></pre></div><h2 id="load-balancing-algorithms-for-dns">Load Balancing Algorithms for DNS <a class="anchor" data-pagefind-ignore href="#load-balancing-algorithms-for-dns" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Algorithm</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>roundrobin</code></td>
					<td>The default: equal servers, even spread</td>
			</tr>
			<tr>
					<td><code>leastconn</code></td>
					<td>TCP connections of uneven length, like zone transfers</td>
			</tr>
			<tr>
					<td><code>source</code></td>
					<td>Keeping each client on one resolver, so its cache is reused</td>
			</tr>
			<tr>
					<td><code>first</code></td>
					<td>Active-passive: everything to one server until it fails</td>
			</tr>
	</tbody>
</table>
<h3 id="round-robin">Round Robin <a class="anchor" data-pagefind-ignore href="#round-robin" aria-label="Link to this section">#</a></h3>
<p>Default algorithm, distributes queries evenly:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_servers
    balance roundrobin
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
</code></pre><h3 id="least-connections-tcp-only">Least Connections (TCP only) <a class="anchor" data-pagefind-ignore href="#least-connections-tcp-only" aria-label="Link to this section">#</a></h3>
<p>Best for DNS servers with varying query complexity:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_tcp_backend
    mode tcp
    balance leastconn
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
</code></pre><h3 id="source-ip-hash">Source IP Hash <a class="anchor" data-pagefind-ignore href="#source-ip-hash" aria-label="Link to this section">#</a></h3>
<p>Ensures clients consistently reach the same server (useful for caching). <code>hash-type consistent</code> keeps most clients on their server when one is added or removed.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_servers
    balance source
    hash-type consistent
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
</code></pre><h3 id="first-available">First Available <a class="anchor" data-pagefind-ignore href="#first-available" aria-label="Link to this section">#</a></h3>
<p>Send all traffic to the first available server (active-passive). The <code>backup</code> server only gets traffic when the first is down.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_servers
    balance first
    server dns-primary   192.168.1.10:53 check
    server dns-secondary 192.168.1.11:53 check backup
</code></pre><h2 id="recursive-vs-authoritative-dns">Recursive vs Authoritative DNS <a class="anchor" data-pagefind-ignore href="#recursive-vs-authoritative-dns" aria-label="Link to this section">#</a></h2>
<p>The two server types need different settings:</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Recursive resolvers</th>
					<th>Authoritative servers</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Clients</td>
					<td>Internal only</td>
					<td>Anyone on the internet</td>
			</tr>
			<tr>
					<td>Address</td>
					<td><code>10.0.0.1</code> / <code>10.0.0.53</code></td>
					<td><code>203.0.113.53</code></td>
			</tr>
			<tr>
					<td>Fallback</td>
					<td>Public DNS when all internal resolvers are down</td>
					<td>None: only your servers have your zones</td>
			</tr>
			<tr>
					<td>dnsdist ACL</td>
					<td>Default (private clients only)</td>
					<td>Open to all</td>
			</tr>
	</tbody>
</table>
<h3 id="load-balancing-recursive-resolvers">Load Balancing Recursive Resolvers <a class="anchor" data-pagefind-ignore href="#load-balancing-recursive-resolvers" aria-label="Link to this section">#</a></h3>
<p>For caching resolvers serving internal clients, HAProxy takes TCP. The public resolvers are <code>backup</code> servers: they only get traffic when both internal resolvers are down.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_resolver
    bind 10.0.0.1:53
    mode tcp
    default_backend resolvers

backend resolvers
    mode tcp
    balance roundrobin
    # Internal resolvers
    server resolver1 192.168.1.10:53 check weight 100
    server resolver2 192.168.1.11:53 check weight 100
    # Public fallback
    server cloudflare 1.1.1.1:53 check weight 50 backup
    server google     8.8.8.8:53 check weight 50 backup
</code></pre><p>And dnsdist takes UDP (with the nftables redirect from above), using a pool for the internal resolvers and one for the public fallback.</p>
<ul>
<li><code>pool</code>: groups servers. Rules pick a pool per query.</li>
<li><code>addAction</code> rules run in order: the first that matches decides. <code>PoolAvailableRule('internal')</code> matches while at least one internal resolver is up.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal resolvers</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.10:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver1&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.11:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver2&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="c1">-- Public fallback</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;1.1.1.1:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;cloudflare&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;8.8.8.8:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;google&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">setServerPolicy</span><span class="p">(</span><span class="n">roundrobin</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal resolvers while at least one is up, public DNS otherwise</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">PoolAvailableRule</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">AllRule</span><span class="p">(),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;fallback&#39;</span><span class="p">))</span>
</span></span></code></pre></div><h3 id="load-balancing-authoritative-servers">Load Balancing Authoritative Servers <a class="anchor" data-pagefind-ignore href="#load-balancing-authoritative-servers" aria-label="Link to this section">#</a></h3>
<p>For authoritative servers serving external queries. They listen on their own public address, separate from the internal resolvers.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_authoritative
    bind 203.0.113.53:53
    mode tcp
    default_backend auth_servers

backend auth_servers
    mode tcp
    balance roundrobin
    option tcp-check
    server auth1 192.168.1.20:53 check
    server auth2 192.168.1.21:53 check
    server auth3 192.168.1.22:53 check
</code></pre><p>The UDP side in dnsdist; an authoritative server answers anyone, so open the ACL:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">setACL</span><span class="p">({</span> <span class="s1">&#39;0.0.0.0/0&#39;</span><span class="p">,</span> <span class="s1">&#39;::/0&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.20:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth1&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.21:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth2&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.22:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth3&#39;</span> <span class="p">})</span>
</span></span></code></pre></div><h2 id="dns-over-tls-dot-load-balancing">DNS over TLS (DoT) Load Balancing <a class="anchor" data-pagefind-ignore href="#dns-over-tls-dot-load-balancing" aria-label="Link to this section">#</a></h2>
<p>For encrypted DNS traffic on port 853, pick one of two modes:</p>
<table>
	<thead>
			<tr>
					<th>Mode</th>
					<th>Where TLS ends</th>
					<th>Use it when</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Termination</td>
					<td>HAProxy, with <code>dns.example.com.pem</code></td>
					<td>The DNS servers only speak plain DNS. One certificate on the load balancer.</td>
			</tr>
			<tr>
					<td>Passthrough</td>
					<td>The DNS servers</td>
					<td>The servers already run DoT, or the load balancer should not see the queries.</td>
			</tr>
	</tbody>
</table>
<p>In passthrough mode, <code>check-ssl verify none</code> health-checks over TLS without verifying the server certificate.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># TLS Termination at HAProxy
frontend dot_frontend
    bind *:853 ssl crt /etc/haproxy/certs/dns.example.com.pem
    mode tcp
    option tcplog
    default_backend dns_tcp_backend

backend dns_tcp_backend
    mode tcp
    balance roundrobin
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check

# TLS Passthrough (use either termination or passthrough on port 853, not both)
frontend dot_passthrough
    bind *:853
    mode tcp
    option tcplog
    default_backend dot_servers

backend dot_servers
    mode tcp
    balance roundrobin
    # check-ssl: health check over TLS, client traffic is passed through untouched
    server dot1 192.168.1.10:853 check check-ssl verify none
    server dot2 192.168.1.11:853 check check-ssl verify none
</code></pre><h2 id="rate-limiting-dns-queries">Rate Limiting DNS Queries <a class="anchor" data-pagefind-ignore href="#rate-limiting-dns-queries" aria-label="Link to this section">#</a></h2>
<p>Protect against DNS amplification attacks. Open DNS servers can be abused to send large answers to a spoofed victim address. For UDP, rate limit in dnsdist:</p>
<ul>
<li><code>MaxQPSIPRule(100)</code> with <code>DropAction()</code>: drops queries above 100 per second per client address.</li>
<li><code>TCAction()</code> for <code>ANY</code>: the client must retry over TCP, which needs a real handshake and so cannot be spoofed.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Drop clients that send more than 100 queries per second</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">MaxQPSIPRule</span><span class="p">(</span><span class="mi">100</span><span class="p">),</span> <span class="n">DropAction</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Answer ANY queries with the TC bit, so the client has to retry over TCP (HAProxy)</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">QTypeRule</span><span class="p">(</span><span class="n">DNSQType.ANY</span><span class="p">),</span> <span class="n">TCAction</span><span class="p">())</span>
</span></span></code></pre></div><p>For TCP with more options. The stick table tracks each client address:</p>
<ul>
<li><code>conn_rate(10s)</code> above 50: new connections are rejected.</li>
<li><code>bytes_out_rate(10s)</code> above 100000: the connection is rejected, to cap how much one client can pull.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_tcp
    bind *:53
    mode tcp
    option tcplog
    
    # Rate limiting table
    stick-table type ip size 100k expire 60s store conn_rate(10s),bytes_out_rate(10s)
    
    # Limit connections per IP
    tcp-request connection track-sc0 src
    tcp-request connection reject if { sc0_conn_rate gt 50 }
    
    # Limit bandwidth (prevent amplification)
    tcp-request content reject if { sc0_bytes_out_rate gt 100000 }
    
    default_backend dns_tcp_backend
</code></pre><h2 id="high-availability-haproxy-setup">High Availability HAProxy Setup <a class="anchor" data-pagefind-ignore href="#high-availability-haproxy-setup" aria-label="Link to this section">#</a></h2>
<p>For the load balancer itself, use keepalived for failover. Both nodes run HAProxy and dnsdist; the virtual IPs live on one node at a time.</p>
<p>The backup node must be able to bind the virtual IP before it owns it, so allow that first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s1">&#39;net.ipv4.ip_nonlocal_bind = 1&#39;</span> &gt; /etc/sysctl.d/90-nonlocal-bind.conf
</span></span><span class="line"><span class="cl">sysctl --system
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">apt install keepalived
</span></span></code></pre></div><p>Configure keepalived on the primary. Both virtual IPs go in <code>virtual_ipaddress</code>: the internal resolver address and the authoritative address. An address that is not listed is never brought up on the backup, so it has no failover.</p>
<ul>
<li><code>vrrp_script check_haproxy</code>: <code>killall -0</code> succeeds only while both <code>haproxy</code> and <code>dnsdist</code> run. While it succeeds, the node&rsquo;s priority goes up by <code>weight 2</code>.</li>
<li><code>priority 101</code> (primary) vs <code>100</code> (backup): with both checks passing, 103 beats 102. When the check fails on the primary, 101 loses to 102 and the IPs move.</li>
<li><code>virtual_router_id 53</code>: must match on both nodes and be unique in the network.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/keepalived/keepalived.conf (primary)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">global_defs <span class="o">{</span>
</span></span><span class="line"><span class="cl">    router_id HAPROXY_DNS_1
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_script check_haproxy <span class="o">{</span>
</span></span><span class="line"><span class="cl">    script <span class="s2">&#34;/usr/bin/killall -0 haproxy dnsdist&#34;</span>
</span></span><span class="line"><span class="cl">    interval <span class="m">2</span>
</span></span><span class="line"><span class="cl">    weight <span class="m">2</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_instance VI_DNS <span class="o">{</span>
</span></span><span class="line"><span class="cl">    state MASTER
</span></span><span class="line"><span class="cl">    interface eth0
</span></span><span class="line"><span class="cl">    virtual_router_id <span class="m">53</span>
</span></span><span class="line"><span class="cl">    priority <span class="m">101</span>
</span></span><span class="line"><span class="cl">    advert_int <span class="m">1</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    authentication <span class="o">{</span>
</span></span><span class="line"><span class="cl">        auth_type PASS
</span></span><span class="line"><span class="cl">        auth_pass secret123
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    virtual_ipaddress <span class="o">{</span>
</span></span><span class="line"><span class="cl">        10.0.0.53/24
</span></span><span class="line"><span class="cl">        <span class="c1"># the authoritative address; add &#34;dev eth1&#34; if it lives on another NIC</span>
</span></span><span class="line"><span class="cl">        203.0.113.53/24
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    track_script <span class="o">{</span>
</span></span><span class="line"><span class="cl">        check_haproxy
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><p>Configure the backup. Only <code>router_id</code>, <code>state</code> and <code>priority</code> differ:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/keepalived/keepalived.conf (backup)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">global_defs <span class="o">{</span>
</span></span><span class="line"><span class="cl">    router_id HAPROXY_DNS_2
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_script check_haproxy <span class="o">{</span>
</span></span><span class="line"><span class="cl">    script <span class="s2">&#34;/usr/bin/killall -0 haproxy dnsdist&#34;</span>
</span></span><span class="line"><span class="cl">    interval <span class="m">2</span>
</span></span><span class="line"><span class="cl">    weight <span class="m">2</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_instance VI_DNS <span class="o">{</span>
</span></span><span class="line"><span class="cl">    state BACKUP
</span></span><span class="line"><span class="cl">    interface eth0
</span></span><span class="line"><span class="cl">    virtual_router_id <span class="m">53</span>
</span></span><span class="line"><span class="cl">    priority <span class="m">100</span>
</span></span><span class="line"><span class="cl">    advert_int <span class="m">1</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    authentication <span class="o">{</span>
</span></span><span class="line"><span class="cl">        auth_type PASS
</span></span><span class="line"><span class="cl">        auth_pass secret123
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    virtual_ipaddress <span class="o">{</span>
</span></span><span class="line"><span class="cl">        10.0.0.53/24
</span></span><span class="line"><span class="cl">        <span class="c1"># the authoritative address; add &#34;dev eth1&#34; if it lives on another NIC</span>
</span></span><span class="line"><span class="cl">        203.0.113.53/24
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    track_script <span class="o">{</span>
</span></span><span class="line"><span class="cl">        check_haproxy
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><h2 id="complete-production-configuration">Complete Production Configuration <a class="anchor" data-pagefind-ignore href="#complete-production-configuration" aria-label="Link to this section">#</a></h2>
<p>Here&rsquo;s a complete configuration for a production DNS load balancer. It combines the sections above:</p>
<table>
	<thead>
			<tr>
					<th>Frontend</th>
					<th>Address</th>
					<th>Backend</th>
					<th>Rate limit</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>internal_dns_tcp</code></td>
					<td><code>10.0.0.53:53</code></td>
					<td>Internal resolvers, public DNS as <code>backup</code></td>
					<td>30 connections / 10s per client</td>
			</tr>
			<tr>
					<td><code>authoritative_dns_tcp</code></td>
					<td><code>203.0.113.53:53</code></td>
					<td>Authoritative servers, <code>timeout server 30s</code> for zone transfers</td>
					<td>20 connections / 10s per client</td>
			</tr>
			<tr>
					<td><code>dot_frontend</code></td>
					<td><code>*:853</code>, TLS</td>
					<td>Internal resolvers</td>
					<td>None</td>
			</tr>
	</tbody>
</table>
<p>Other settings:</p>
<ul>
<li><code>nbthread 4</code> with <code>cpu-map</code>: four threads, each pinned to one CPU.</li>
<li><code>retries 2</code>: retry a failed connection to a server twice.</li>
<li><code>stats auth admin:changeme</code>: replace this password.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0 info
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    # Performance
    maxconn 100000
    nbthread 4
    cpu-map auto:1/1-4 0-3
    
    # Security
    tune.ssl.default-dh-param 2048

defaults dns_defaults
    log global
    timeout connect 1s
    timeout client  5s
    timeout server  5s
    default-server inter 3s fall 3 rise 2
    retries 2

# Stats and monitoring
listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 5s
    stats auth admin:changeme
    http-request use-service prometheus-exporter if { path /metrics }

# ====================
# Internal Resolvers
# ====================

# UDP DNS for internal and external clients is handled by dnsdist (below)

# TCP DNS for internal clients
frontend internal_dns_tcp
    bind 10.0.0.53:53
    mode tcp
    option tcplog
    default_backend internal_resolvers_tcp
    
    # Connection limiting
    stick-table type ip size 50k expire 60s store conn_rate(10s)
    tcp-request connection track-sc0 src
    tcp-request connection reject if { sc0_conn_rate gt 30 }

backend internal_resolvers_tcp
    mode tcp
    balance leastconn
    option tcp-check
    server resolver1 192.168.1.10:53 check
    server resolver2 192.168.1.11:53 check
    server resolver3 192.168.1.12:53 check
    server cloudflare 1.1.1.1:53 check backup
    server quad9      9.9.9.9:53 check backup

# ====================
# Authoritative Servers
# ====================

# TCP DNS for zone transfers and large responses
frontend authoritative_dns_tcp
    bind 203.0.113.53:53
    mode tcp
    option tcplog
    default_backend auth_servers_tcp
    
    # Connection limiting
    stick-table type ip size 100k expire 60s store conn_rate(10s),bytes_out_rate(10s)
    tcp-request connection track-sc0 src
    tcp-request connection reject if { sc0_conn_rate gt 20 }

backend auth_servers_tcp
    mode tcp
    balance roundrobin
    option tcp-check
    timeout server 30s  # Longer timeout for zone transfers
    server auth1 192.168.1.20:53 check
    server auth2 192.168.1.21:53 check
    server auth3 192.168.1.22:53 check

# ====================
# DNS over TLS (DoT)
# ====================

frontend dot_frontend
    bind *:853 ssl crt /etc/haproxy/certs/dns.example.com.pem alpn dns
    mode tcp
    option tcplog
    timeout client 30s
    default_backend internal_resolvers_tcp
    
    # TLS client validation (optional)
    # bind *:853 ssl crt /etc/haproxy/certs/dns.example.com.pem ca-file /etc/haproxy/certs/ca.pem verify optional
</code></pre><p>And the UDP side, with one dnsdist listener per virtual IP. nftables sends each virtual IP to its own port, and the rules pick the pool by port:</p>
<ul>
<li>Port 5354 (authoritative): 50 queries per second per client, then the <code>auth</code> pool.</li>
<li>Port 5353 (internal): 100 queries per second per client, then the <code>internal</code> pool, or <code>fallback</code> when all internal resolvers are down.</li>
<li><code>controlSocket</code> and <code>setKey</code>: the console for <code>dnsdist -c</code>, local only.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- nftables redirects UDP 10.0.0.53:53 to 5353 and UDP 203.0.113.53:53 to 5354</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>   <span class="c1">-- internal resolvers</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5354&#39;</span><span class="p">)</span>   <span class="c1">-- authoritative servers</span>
</span></span><span class="line"><span class="cl"><span class="n">setACL</span><span class="p">({</span> <span class="s1">&#39;0.0.0.0/0&#39;</span><span class="p">,</span> <span class="s1">&#39;::/0&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal resolvers, public DNS as fallback</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.10:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver1&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.11:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver2&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.12:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver3&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;1.1.1.1:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;cloudflare&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;9.9.9.9:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;quad9&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Authoritative servers</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.20:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth1&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;auth&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.21:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth2&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;auth&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.22:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth3&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;auth&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">setServerPolicy</span><span class="p">(</span><span class="n">roundrobin</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Authoritative listener: strict rate limiting, authoritative servers only</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">AndRule</span><span class="p">({</span> <span class="n">DSTPortRule</span><span class="p">(</span><span class="mi">5354</span><span class="p">),</span> <span class="n">MaxQPSIPRule</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="p">}),</span> <span class="n">DropAction</span><span class="p">())</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">DSTPortRule</span><span class="p">(</span><span class="mi">5354</span><span class="p">),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;auth&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal listener: rate limiting, resolvers with public fallback</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">MaxQPSIPRule</span><span class="p">(</span><span class="mi">100</span><span class="p">),</span> <span class="n">DropAction</span><span class="p">())</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">PoolAvailableRule</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">AllRule</span><span class="p">(),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;fallback&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Console for dnsdist -c (generate a key with: head -c 32 /dev/urandom | base64)</span>
</span></span><span class="line"><span class="cl"><span class="n">controlSocket</span><span class="p">(</span><span class="s1">&#39;127.0.0.1:5199&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">setKey</span><span class="p">(</span><span class="s1">&#39;replace-with-your-base64-key&#39;</span><span class="p">)</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/nftables.conf (add below the existing tables)</span>
</span></span><span class="line"><span class="cl">table ip dns_udp <span class="o">{</span>
</span></span><span class="line"><span class="cl">    chain prerouting <span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="nb">type</span> nat hook prerouting priority dstnat<span class="p">;</span> policy accept<span class="p">;</span>
</span></span><span class="line"><span class="cl">        ip daddr 10.0.0.53 udp dport <span class="m">53</span> redirect to :5353
</span></span><span class="line"><span class="cl">        ip daddr 203.0.113.53 udp dport <span class="m">53</span> redirect to :5354
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><h2 id="monitoring-and-logging">Monitoring and Logging <a class="anchor" data-pagefind-ignore href="#monitoring-and-logging" aria-label="Link to this section">#</a></h2>
<h3 id="dns-query-logging">DNS Query Logging <a class="anchor" data-pagefind-ignore href="#dns-query-logging" aria-label="Link to this section">#</a></h3>
<p><code>option tcplog</code> logs one line per connection, not per query. The custom <code>log-format</code> adds the client, the chosen server, the timers (<code>%Tw/%Tc/%Tt</code>), bytes sent and the connection counts.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_tcp
    bind *:53
    mode tcp
    option tcplog
    log-format &#34;%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc&#34;
    default_backend dns_servers
</code></pre><h3 id="prometheus-metrics">Prometheus Metrics <a class="anchor" data-pagefind-ignore href="#prometheus-metrics" aria-label="Link to this section">#</a></h3>
<p>Enable the built-in Prometheus exporter:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">listen stats
    bind *:8404
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
</code></pre><p>Key metrics to monitor:</p>
<ul>
<li><code>haproxy_server_status</code>: Backend server health</li>
<li><code>haproxy_frontend_current_sessions</code>: Active connections</li>
<li><code>haproxy_frontend_bytes_in_total</code>: Incoming traffic</li>
<li><code>haproxy_frontend_bytes_out_total</code>: Outgoing traffic (watch for amplification)</li>
</ul>
<p>These cover HAProxy, so TCP and DoT only. dnsdist handles most queries (UDP); check it with <code>showServers()</code> (below).</p>
<h2 id="testing-your-configuration">Testing Your Configuration <a class="anchor" data-pagefind-ignore href="#testing-your-configuration" aria-label="Link to this section">#</a></h2>
<h3 id="validate-configuration">Validate Configuration <a class="anchor" data-pagefind-ignore href="#validate-configuration" aria-label="Link to this section">#</a></h3>
<p>Check the syntax before every reload.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">haproxy -c -f /etc/haproxy/haproxy.cfg
</span></span></code></pre></div><h3 id="test-dns-resolution">Test DNS Resolution <a class="anchor" data-pagefind-ignore href="#test-dns-resolution" aria-label="Link to this section">#</a></h3>
<p>Query the virtual IP over both protocols, then each server directly. If a direct query works but the virtual IP does not, the problem is in the load balancer.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># UDP query (dnsdist)</span>
</span></span><span class="line"><span class="cl">dig @10.0.0.53 google.com A
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># TCP query (HAProxy)</span>
</span></span><span class="line"><span class="cl">dig @10.0.0.53 +tcp google.com A
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test all servers</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> server in 192.168.1.10 192.168.1.11 192.168.1.12<span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;Testing </span><span class="nv">$server</span><span class="s2">:&#34;</span>
</span></span><span class="line"><span class="cl">    dig @<span class="nv">$server</span> +short google.com A
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span></code></pre></div><h3 id="load-testing">Load Testing <a class="anchor" data-pagefind-ignore href="#load-testing" aria-label="Link to this section">#</a></h3>
<p><code>dnsperf</code> sends queries from a file: <code>-c 100</code> clients, at most <code>-Q 10000</code> queries per second. Run it once per protocol. Keep the rate limits in mind: from one test machine you hit them quickly.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Install dnsperf</span>
</span></span><span class="line"><span class="cl">apt install dnsperf
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create query file</span>
</span></span><span class="line"><span class="cl">cat &gt; queries.txt <span class="s">&lt;&lt; EOF
</span></span></span><span class="line"><span class="cl"><span class="s">google.com A
</span></span></span><span class="line"><span class="cl"><span class="s">example.com A
</span></span></span><span class="line"><span class="cl"><span class="s">github.com A
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Run load test (UDP, through dnsdist)</span>
</span></span><span class="line"><span class="cl">dnsperf -s 10.0.0.53 -d queries.txt -c <span class="m">100</span> -Q <span class="m">10000</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># The same over TCP, through HAProxy</span>
</span></span><span class="line"><span class="cl">dnsperf -m tcp -s 10.0.0.53 -d queries.txt -c <span class="m">100</span> -Q <span class="m">10000</span>
</span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Check</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>UDP not working</td>
					<td>UDP goes through dnsdist, not HAProxy. Check that dnsdist listens (<code>ss -ulpn | grep 5353</code>), that the nftables redirect is loaded (<code>nft list table ip dns_udp</code>), that <code>setACL()</code> allows your clients, and that <code>showServers()</code> lists the backends as up.</td>
			</tr>
			<tr>
					<td>Health checks failing</td>
					<td>Verify DNS servers are responding, check firewall rules. With the DNS query check: does <code>health.local</code> exist on every server?</td>
			</tr>
			<tr>
					<td>High latency</td>
					<td>Check <code>timeout</code> settings and server response times.</td>
			</tr>
			<tr>
					<td>Connection refused</td>
					<td>Verify HAProxy is binding correctly (<code>ss -tulpn | grep 53</code>).</td>
			</tr>
			<tr>
					<td>Other firewall rules gone</td>
					<td><code>flush ruleset</code> in <code>/etc/nftables.conf</code> (see the warning above).</td>
			</tr>
	</tbody>
</table>
<h3 id="debug-commands">Debug Commands <a class="anchor" data-pagefind-ignore href="#debug-commands" aria-label="Link to this section">#</a></h3>
<p>The admin socket shows what HAProxy sees; <code>dnsdist -c</code> does the same for the UDP side.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check HAProxy status</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show info&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check backend servers</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show servers state&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check statistics</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show stat&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Watch connections in real-time</span>
</span></span><span class="line"><span class="cl">watch -n <span class="m">1</span> <span class="s1">&#39;echo &#34;show stat&#34; | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,5,18&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># dnsdist (UDP) backends and their state</span>
</span></span><span class="line"><span class="cl">dnsdist -c -e <span class="s1">&#39;showServers()&#39;</span>
</span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Change the example secrets</strong>: <code>auth_pass secret123</code> in keepalived, <code>stats auth admin:changeme</code> in HAProxy, and the dnsdist console key. VRRP <code>PASS</code> authentication is sent in plain text; it only guards against mistakes.</li>
<li><strong>Keep resolvers internal.</strong> The production dnsdist config sets <code>setACL({ '0.0.0.0/0', '::/0' })</code> for the authoritative side, and that ACL also covers the internal listener. Make sure only internal clients can reach <code>10.0.0.53</code> and port 5353, or you run an open resolver.</li>
<li><strong>Keep the rate limits.</strong> They are what stops the load balancer from being used for amplification attacks.</li>
<li><strong>Avoid external checks where you can.</strong> <code>insecure-fork-wanted</code> lifts a safety guard in HAProxy.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-dns-lb" rel="noopener">pderuiter/blog-haproxy-dns-lb</a>. It creates two load balancer VMs on Proxmox and configures HAProxy for TCP and DoT, dnsdist with the nftables redirect for UDP, and keepalived for the virtual IPs, as in the complete production configuration from this post; run <code>terraform apply</code>, then <code>ansible-playbook site.yml</code>.</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>Next in the series: <a href="https://blog.bsdserver.nl/haproxy-with-lets-encrypt-automated-https-certificate-management/">HAProxy with Let&rsquo;s Encrypt</a> for automatic HTTPS certificate management.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<ul>
<li>HAProxy cannot proxy UDP (also not in 3.4): use dnsdist for DNS over UDP</li>
<li>Cover both UDP (dnsdist) and TCP (HAProxy) on the same address for complete DNS coverage</li>
<li>Implement proper health checks for DNS servers</li>
<li>Use rate limiting to protect against abuse</li>
<li>Deploy keepalived for HAProxy and dnsdist high availability</li>
</ul>
]]></content:encoded></item><item><title>HAProxy and Consul: Dynamic Service Discovery for Load Balancing</title><link>https://blog.bsdserver.nl/haproxy-and-consul-dynamic-service-discovery-for-load-balancing/</link><guid isPermaLink="false">693b765b79eb54000160a628</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:56:43 +0000</pubDate><description>In dynamic environments where services scale up and down automatically, static HAProxy configurations become a management burden.</description><category>HAProxy</category><category>Consul</category><category>Service discovery</category><category>DevOps</category><category>Microservices</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-and-consul-service-discovery.png" medium="image"/><content:encoded><![CDATA[<p>In dynamic environments where services scale up and down automatically, static HAProxy configurations become a management burden. Every instance you add or remove means editing <code>server</code> lines and reloading HAProxy.</p>
<p>Consul keeps a live list of which instances run and which are healthy. The HAProxy Data Plane API reads that list and keeps one HAProxy backend per service up to date. The server lists maintain themselves.</p>
<h2 id="why-consul--haproxy">Why Consul + HAProxy? <a class="anchor" data-pagefind-ignore href="#why-consul--haproxy" aria-label="Link to this section">#</a></h2>
<p>The combination provides:</p>
<ul>
<li><strong>Automatic service registration</strong> - Services announce themselves to Consul</li>
<li><strong>Health-aware routing</strong> - Only healthy instances receive traffic</li>
<li><strong>Dynamic scaling</strong> - New instances are automatically added to load balancing</li>
<li><strong>No downtime updates</strong> - Changes are applied with seamless reloads</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when backends change often: autoscaling, frequent deployments, several teams deploying their own services.</li>
<li><strong>Skip it</strong> for a few static backends. A Consul cluster is three more machines to run; a static <code>haproxy.cfg</code> from Ansible is simpler.</li>
</ul>
<h3 id="data-plane-api-instead-of-consul-template">Data Plane API instead of consul-template <a class="anchor" data-pagefind-ignore href="#data-plane-api-instead-of-consul-template" aria-label="Link to this section">#</a></h3>
<p>This post uses the Data Plane API&rsquo;s built-in Consul service discovery. Earlier versions used consul-template to render <code>haproxy.cfg</code>; that still works, but you write and maintain the template yourself.</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>consul-template</th>
					<th>Data Plane API</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Backends</td>
					<td>Whatever your template writes</td>
					<td>One per service, generated</td>
			</tr>
			<tr>
					<td>Select instances by</td>
					<td>Service, tag, datacenter, metadata, KV</td>
					<td>Service name only (allowlist, denylist, regexp)</td>
			</tr>
			<tr>
					<td>Server weights</td>
					<td>From Consul or metadata</td>
					<td>Fixed (128)</td>
			</tr>
			<tr>
					<td>Routing changes</td>
					<td>Edit the template</td>
					<td>Runtime API (maps, frontends), no file editing</td>
			</tr>
	</tbody>
</table>
<p>The trade-off: fewer moving parts, less template freedom.</p>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<pre tabindex="0"><code>                    ┌─────────────────────────────────┐
                    │           Consul Cluster        │
                    │  ┌───────┐ ┌───────┐ ┌───────┐ │
                    │  │Server1│ │Server2│ │Server3│ │
                    │  └───────┘ └───────┘ └───────┘ │
                    └────────────────┬────────────────┘
                                     │
              ┌──────────────────────┼──────────────────────┐
              │                      │                      │
              ▼                      ▼                      ▼
      ┌───────────────┐      ┌───────────────┐      ┌───────────────┐
      │   Service A   │      │   Service B   │      │ Load balancer │
      │ + Consul Agent│      │ + Consul Agent│      │ + Consul Agent│
      └───────────────┘      └───────────────┘      │ + Data Plane  │
              ▲                      ▲              │   API         │
              │                      │              │ + HAProxy     │
              └──────────────────────┴──────────────┴───────────────┘
                                                            ▲
                                                            │
                                                      Clients/Users
</code></pre><table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Consul servers</td>
					<td>3 dedicated machines</td>
					<td>Store the catalog: services, addresses, health. Three so the cluster survives losing one.</td>
			</tr>
			<tr>
					<td>Consul agent</td>
					<td>Every service node and the load balancer</td>
					<td>Registers local services, runs their health checks, reports to the servers.</td>
			</tr>
			<tr>
					<td>Data Plane API</td>
					<td>Load balancer</td>
					<td>Polls Consul, creates a <code>consul-backend-…-&lt;service&gt;</code> backend per service, adds and removes servers, reloads HAProxy.</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td>Load balancer</td>
					<td>Balances traffic. The frontend picks a backend by Host header from a map file.</td>
			</tr>
	</tbody>
</table>
<p>The flow: an instance starts → its agent registers it → the check passes → the Data Plane API sees it at its next poll (every few seconds) → adds a server to the service&rsquo;s backend → HAProxy reloads seamlessly. Stopping or failing an instance runs the same chain in reverse.</p>
<h2 id="setting-up-consul">Setting Up Consul <a class="anchor" data-pagefind-ignore href="#setting-up-consul" aria-label="Link to this section">#</a></h2>
<h3 id="consul-server-installation">Consul Server Installation <a class="anchor" data-pagefind-ignore href="#consul-server-installation" aria-label="Link to this section">#</a></h3>
<p>One binary for servers and agents; the configuration decides the role. Install it on all servers and service nodes.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Download Consul</span>
</span></span><span class="line"><span class="cl"><span class="nv">CONSUL_VERSION</span><span class="o">=</span><span class="s2">&#34;2.1.0&#34;</span>
</span></span><span class="line"><span class="cl">wget https://releases.hashicorp.com/consul/<span class="si">${</span><span class="nv">CONSUL_VERSION</span><span class="si">}</span>/consul_<span class="si">${</span><span class="nv">CONSUL_VERSION</span><span class="si">}</span>_linux_amd64.zip
</span></span><span class="line"><span class="cl">unzip consul_<span class="si">${</span><span class="nv">CONSUL_VERSION</span><span class="si">}</span>_linux_amd64.zip
</span></span><span class="line"><span class="cl">sudo mv consul /usr/local/bin/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create directories</span>
</span></span><span class="line"><span class="cl">sudo mkdir -p /etc/consul.d /var/lib/consul
</span></span><span class="line"><span class="cl">sudo useradd --system --home /var/lib/consul --shell /bin/false consul
</span></span><span class="line"><span class="cl">sudo chown -R consul:consul /etc/consul.d /var/lib/consul
</span></span></code></pre></div><h3 id="consul-server-configuration">Consul Server Configuration <a class="anchor" data-pagefind-ignore href="#consul-server-configuration" aria-label="Link to this section">#</a></h3>
<p>What the key settings do:</p>
<ul>
<li><code>bootstrap_expect = 3</code>: wait for three servers before electing a leader.</li>
<li><code>retry_join</code>: the other servers; a restarted server finds them again.</li>
<li><code>bind_addr</code>: picks the private address for cluster traffic, so one file fits all servers.</li>
<li><code>client_addr = &quot;0.0.0.0&quot;</code>: API and UI on all interfaces. Firewall port 8500.</li>
<li><code>acl</code> with <code>default_policy = &quot;deny&quot;</code>: no access without a token. Enable it from day one.</li>
<li><code>connect</code> and the gRPC ports: only for Consul Connect (see Next Steps).</li>
</ul>
<p>Create <code>/etc/consul.d/consul.hcl</code> (unique <code>node_name</code> per server):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="n">datacenter</span> <span class="o">=</span> <span class="s2">&#34;dc1&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">node_name</span> <span class="o">=</span> <span class="s2">&#34;consul-server-1&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">data_dir</span> <span class="o">=</span> <span class="s2">&#34;/var/lib/consul&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">log_level</span> <span class="o">=</span> <span class="s2">&#34;INFO&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">server</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl"><span class="n">bootstrap_expect</span> <span class="o">=</span> <span class="m">3</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">bind_addr</span> <span class="o">=</span> <span class="s2">&#34;{{ GetPrivateInterfaces | include \&#34;network\&#34; \&#34;10.0.0.0/8\&#34; | attr \&#34;address\&#34; }}&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">client_addr</span> <span class="o">=</span> <span class="s2">&#34;0.0.0.0&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">ui_config</span> {
</span></span><span class="line"><span class="cl"><span class="n">  enabled</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">connect</span> {
</span></span><span class="line"><span class="cl"><span class="n">  enabled</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">addresses</span> {
</span></span><span class="line"><span class="cl"><span class="n">  http</span> <span class="o">=</span> <span class="s2">&#34;0.0.0.0&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  grpc</span> <span class="o">=</span> <span class="s2">&#34;0.0.0.0&#34;</span>
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">ports</span> {
</span></span><span class="line"><span class="cl"><span class="n">  http</span> <span class="o">=</span> <span class="m">8500</span>
</span></span><span class="line"><span class="cl"><span class="n">  grpc</span> <span class="o">=</span> <span class="m">8502</span>
</span></span><span class="line"><span class="cl"><span class="n">  grpc_tls</span> <span class="o">=</span> <span class="m">8503</span>
</span></span><span class="line"><span class="cl">}
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">retry_join</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;consul-server-1.example.com&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;consul-server-2.example.com&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;consul-server-3.example.com&#34;</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">acl</span> {
</span></span><span class="line"><span class="cl"><span class="n">  enabled</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl"><span class="n">  default_policy</span> <span class="o">=</span> <span class="s2">&#34;deny&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  enable_token_persistence</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><h3 id="consul-agent-configuration-for-service-nodes">Consul Agent Configuration (for service nodes) <a class="anchor" data-pagefind-ignore href="#consul-agent-configuration-for-service-nodes" aria-label="Link to this section">#</a></h3>
<p>Same as the server, minus the server parts. <code>client_addr = &quot;127.0.0.1&quot;</code> keeps the agent&rsquo;s API local. The load balancer gets an agent too, so the Data Plane API only ever talks to <code>127.0.0.1:8500</code>.</p>
<p>Create <code>/etc/consul.d/consul.hcl</code> on service nodes:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="n">datacenter</span> <span class="o">=</span> <span class="s2">&#34;dc1&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">node_name</span> <span class="o">=</span> <span class="s2">&#34;service-node-1&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">data_dir</span> <span class="o">=</span> <span class="s2">&#34;/var/lib/consul&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">log_level</span> <span class="o">=</span> <span class="s2">&#34;INFO&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">server</span> <span class="o">=</span> <span class="kt">false</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">bind_addr</span> <span class="o">=</span> <span class="s2">&#34;{{ GetPrivateInterfaces | include \&#34;network\&#34; \&#34;10.0.0.0/8\&#34; | attr \&#34;address\&#34; }}&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">client_addr</span> <span class="o">=</span> <span class="s2">&#34;127.0.0.1&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">retry_join</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;consul-server-1.example.com&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;consul-server-2.example.com&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;consul-server-3.example.com&#34;</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span><span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Must match the servers, otherwise the agent cannot use tokens at all
</span></span></span><span class="line"><span class="cl"><span class="k">acl</span> {
</span></span><span class="line"><span class="cl"><span class="n">  enabled</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl"><span class="n">  default_policy</span> <span class="o">=</span> <span class="s2">&#34;deny&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  enable_token_persistence</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><h3 id="systemd-service">Systemd Service <a class="anchor" data-pagefind-ignore href="#systemd-service" aria-label="Link to this section">#</a></h3>
<p>One unit for servers and agents. <code>Type=notify</code> makes units that start after Consul, like the Data Plane API, wait until it is ready.</p>
<p>Create <code>/etc/systemd/system/consul.service</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Consul Service Discovery Agent</span>
</span></span><span class="line"><span class="cl"><span class="na">Documentation</span><span class="o">=</span><span class="s">https://www.consul.io/</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network-online.target</span>
</span></span><span class="line"><span class="cl"><span class="na">Wants</span><span class="o">=</span><span class="s">network-online.target</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">notify</span>
</span></span><span class="line"><span class="cl"><span class="na">User</span><span class="o">=</span><span class="s">consul</span>
</span></span><span class="line"><span class="cl"><span class="na">Group</span><span class="o">=</span><span class="s">consul</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/consul agent -config-dir=/etc/consul.d/</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecReload</span><span class="o">=</span><span class="s">/bin/kill -HUP $MAINPID</span>
</span></span><span class="line"><span class="cl"><span class="na">KillMode</span><span class="o">=</span><span class="s">process</span>
</span></span><span class="line"><span class="cl"><span class="na">KillSignal</span><span class="o">=</span><span class="s">SIGTERM</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
</span></span><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl"><span class="na">LimitNOFILE</span><span class="o">=</span><span class="s">65536</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
</span></span></code></pre></div><p>Run <code>systemctl enable --now consul</code>. <code>consul members</code> shows the cluster once three servers have joined.</p>
<h3 id="bootstrapping-acls">Bootstrapping ACLs <a class="anchor" data-pagefind-ignore href="#bootstrapping-acls" aria-label="Link to this section">#</a></h3>
<p>With <code>default_policy = &quot;deny&quot;</code> nothing works until tokens exist: agents log <code>Coordinate update blocked by ACLs</code> and <code>Service registration blocked by ACLs</code>, and the services never reach the catalog.</p>
<p>You create three tokens:</p>
<ul>
<li><strong>Bootstrap token</strong>: full access, administration only.</li>
<li><strong>Agent token</strong>: lets agents register nodes and services. Goes on every agent.</li>
<li><strong>Data Plane API token</strong>: read-only. The API only reads services, nodes and health.</li>
</ul>
<p>Once the servers have elected a leader:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Run once, on one server. Keep the SecretID safe: it is the management token.</span>
</span></span><span class="line"><span class="cl">consul acl bootstrap
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">CONSUL_HTTP_TOKEN</span><span class="o">=</span><span class="s2">&#34;&lt;bootstrap SecretID&gt;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">cat &gt; agent-policy.hcl <span class="s">&lt;&lt;&#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">node_prefix &#34;&#34; {
</span></span></span><span class="line"><span class="cl"><span class="s">  policy = &#34;write&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">}
</span></span></span><span class="line"><span class="cl"><span class="s">service_prefix &#34;&#34; {
</span></span></span><span class="line"><span class="cl"><span class="s">  policy = &#34;write&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">}
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">consul acl policy create -name agent -rules @agent-policy.hcl
</span></span><span class="line"><span class="cl">consul acl token create -description <span class="s2">&#34;agent token&#34;</span> -policy-name agent
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">cat &gt; dataplaneapi-policy.hcl <span class="s">&lt;&lt;&#39;EOF&#39;
</span></span></span><span class="line"><span class="cl"><span class="s">node_prefix &#34;&#34; {
</span></span></span><span class="line"><span class="cl"><span class="s">  policy = &#34;read&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">}
</span></span></span><span class="line"><span class="cl"><span class="s">service_prefix &#34;&#34; {
</span></span></span><span class="line"><span class="cl"><span class="s">  policy = &#34;read&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">}
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">consul acl policy create -name dataplaneapi -rules @dataplaneapi-policy.hcl
</span></span><span class="line"><span class="cl">consul acl token create -description <span class="s2">&#34;dataplaneapi&#34;</span> -policy-name dataplaneapi
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># On every agent (servers and service nodes), with the agent token&#39;s SecretID.</span>
</span></span><span class="line"><span class="cl"><span class="c1"># &#34;default&#34; is also used for services defined in /etc/consul.d and for local API calls without a token.</span>
</span></span><span class="line"><span class="cl">consul acl set-agent-token agent   <span class="s2">&#34;&lt;agent token SecretID&gt;&#34;</span>
</span></span><span class="line"><span class="cl">consul acl set-agent-token default <span class="s2">&#34;&lt;agent token SecretID&gt;&#34;</span>
</span></span></code></pre></div><p>The Data Plane API token goes into the Consul registration on the load balancer (below). Export <code>CONSUL_HTTP_TOKEN</code> before running <code>consul members</code> or <code>consul catalog</code> commands.</p>
<h2 id="registering-services-with-consul">Registering Services with Consul <a class="anchor" data-pagefind-ignore href="#registering-services-with-consul" aria-label="Link to this section">#</a></h2>
<p>Only registered, healthy services reach HAProxy. A registration gives a name, a port and a health check.</p>
<p>Two ways to register:</p>
<ul>
<li><strong>File</strong> in <code>/etc/consul.d</code>: for long-lived services managed by configuration management.</li>
<li><strong>HTTP API</strong>: for deployment tooling and short-lived instances.</li>
</ul>
<h3 id="service-registration-file">Service Registration File <a class="anchor" data-pagefind-ignore href="#service-registration-file" aria-label="Link to this section">#</a></h3>
<ul>
<li><code>tags</code> and <code>meta</code>: labels for your own tooling. The Data Plane API ignores them; it selects by service name.</li>
<li><code>success_before_passing</code> / <code>failures_before_critical</code>: stop a flapping instance from going in and out of the pool.</li>
</ul>
<p>Create <code>/etc/consul.d/web-service.hcl</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">service</span> {
</span></span><span class="line"><span class="cl"><span class="n">  name</span> <span class="o">=</span> <span class="s2">&#34;web&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  port</span> <span class="o">=</span> <span class="m">8080</span>
</span></span><span class="line"><span class="cl"><span class="n">  tags</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;http&#34;, &#34;production&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="k">meta</span> {
</span></span><span class="line"><span class="cl"><span class="n">    version</span> <span class="o">=</span> <span class="s2">&#34;1.0.0&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">    environment</span> <span class="o">=</span> <span class="s2">&#34;production&#34;</span>
</span></span><span class="line"><span class="cl">  }<span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1">  
</span></span></span><span class="line"><span class="cl"><span class="c1">  # More than one check: use a &#34;checks&#34; list, repeated check blocks fail to parse
</span></span></span><span class="line"><span class="cl"><span class="n">  checks</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    {
</span></span><span class="line"><span class="cl"><span class="n">      name</span> <span class="o">=</span> <span class="s2">&#34;HTTP Health Check&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      http</span> <span class="o">=</span> <span class="s2">&#34;http://localhost:8080/health&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      interval</span> <span class="o">=</span> <span class="s2">&#34;10s&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      timeout</span> <span class="o">=</span> <span class="s2">&#34;5s&#34;</span>
</span></span><span class="line"><span class="cl">      
</span></span><span class="line"><span class="cl">      <span class="n">success_before_passing</span> <span class="o">=</span> <span class="m">2</span>
</span></span><span class="line"><span class="cl"><span class="n">      failures_before_critical</span> <span class="o">=</span> <span class="m">3</span>
</span></span><span class="line"><span class="cl">    }<span class="p">,</span>
</span></span><span class="line"><span class="cl">    {
</span></span><span class="line"><span class="cl"><span class="n">      name</span> <span class="o">=</span> <span class="s2">&#34;TCP Port Check&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      tcp</span> <span class="o">=</span> <span class="s2">&#34;localhost:8080&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      interval</span> <span class="o">=</span> <span class="s2">&#34;5s&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      timeout</span> <span class="o">=</span> <span class="s2">&#34;2s&#34;</span>
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><p>Apply with <code>consul reload</code>. An instance is healthy only when all its checks pass.</p>
<h3 id="api-service-registration">API Service Registration <a class="anchor" data-pagefind-ignore href="#api-service-registration" aria-label="Link to this section">#</a></h3>
<p><code>weights</code> sets Consul&rsquo;s traffic share per health state, used by Consul DNS. The Data Plane API does not copy them: every server it creates gets weight 128.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">service</span> {
</span></span><span class="line"><span class="cl"><span class="n">  name</span> <span class="o">=</span> <span class="s2">&#34;api&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  port</span> <span class="o">=</span> <span class="m">3000</span>
</span></span><span class="line"><span class="cl"><span class="n">  tags</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;http&#34;, &#34;api&#34;, &#34;v2&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="k">weights</span> {
</span></span><span class="line"><span class="cl"><span class="n">    passing</span> <span class="o">=</span> <span class="m">100</span>
</span></span><span class="line"><span class="cl"><span class="n">    warning</span> <span class="o">=</span> <span class="m">50</span>
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="k">check</span> {
</span></span><span class="line"><span class="cl"><span class="n">    name</span> <span class="o">=</span> <span class="s2">&#34;API Health&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">    http</span> <span class="o">=</span> <span class="s2">&#34;http://localhost:3000/api/health&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">    interval</span> <span class="o">=</span> <span class="s2">&#34;10s&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">    timeout</span> <span class="o">=</span> <span class="s2">&#34;3s&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">header</span> {
</span></span><span class="line"><span class="cl"><span class="n">      Content-Type</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;application/json&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  }
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><h3 id="dynamic-registration-via-http-api">Dynamic Registration via HTTP API <a class="anchor" data-pagefind-ignore href="#dynamic-registration-via-http-api" aria-label="Link to this section">#</a></h3>
<p>Register when the instance is up; deregister before you stop it, so HAProxy drops it before the process goes away.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Register a service dynamically</span>
</span></span><span class="line"><span class="cl">curl -X PUT http://localhost:8500/v1/agent/service/register <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;ID&#34;: &#34;web-1&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;Name&#34;: &#34;web&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;Port&#34;: 8080,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;Tags&#34;: [&#34;http&#34;, &#34;production&#34;],
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;Check&#34;: {
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;HTTP&#34;: &#34;http://localhost:8080/health&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;Interval&#34;: &#34;10s&#34;
</span></span></span><span class="line"><span class="cl"><span class="s1">    }
</span></span></span><span class="line"><span class="cl"><span class="s1">  }&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Deregister a service</span>
</span></span><span class="line"><span class="cl">curl -X PUT http://localhost:8500/v1/agent/service/deregister/web-1
</span></span></code></pre></div><h2 id="the-load-balancer-haproxy-and-the-data-plane-api">The Load Balancer: HAProxy and the Data Plane API <a class="anchor" data-pagefind-ignore href="#the-load-balancer-haproxy-and-the-data-plane-api" aria-label="Link to this section">#</a></h2>
<p>The Data Plane API is HAProxy&rsquo;s management API. Here it does one job: it polls Consul and keeps one backend per service in <code>haproxy.cfg</code> up to date.</p>
<p>The setup on the load balancer:</p>
<ol>
<li>Install HAProxy and write a base <code>haproxy.cfg</code> with the frontend.</li>
<li>Install the Data Plane API.</li>
<li>Register Consul with the API.</li>
</ol>
<h3 id="installing-haproxy">Installing HAProxy <a class="anchor" data-pagefind-ignore href="#installing-haproxy" aria-label="Link to this section">#</a></h3>
<p>On the load balancer, install HAProxy from the official HAProxy packages rather than the distribution&rsquo;s older <code>haproxy</code> package:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Official HAProxy packages for Ubuntu 24.04: one PPA per branch</span>
</span></span><span class="line"><span class="cl">sudo add-apt-repository -y ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">sudo apt update
</span></span><span class="line"><span class="cl">sudo apt install -y <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span> curl jq socat
</span></span><span class="line"><span class="cl">haproxy -v
</span></span></code></pre></div><p>3.3 is the newest branch the PPA builds for Ubuntu 24.04, but it is a short-lived stable branch, maintained until about Q1 2027. The newest LTS branch, 3.4, is built for Ubuntu 26.04 (<code>ppa:vbernat/haproxy-3.4</code>) and for Debian 13 on <a href="https://haproxy.debian.net/" rel="noopener">haproxy.debian.net</a>, which lists the repository lines per release and branch. For a long-lived Ubuntu 24.04 install, use the 3.2 LTS branch instead (<code>ppa:vbernat/haproxy-3.2</code>, maintained until 2030).</p>
<h3 id="base-haproxy-configuration">Base HAProxy Configuration <a class="anchor" data-pagefind-ignore href="#base-haproxy-configuration" aria-label="Link to this section">#</a></h3>
<p>You write the frontend; the Data Plane API writes the backends. Do this before the API starts: from then on the API owns <code>haproxy.cfg</code> and marks it <code># Dataplaneapi managed File</code>.</p>
<ul>
<li><code>defaults consul_services</code>: settings for the generated backends (balance, timeouts, forwardfor). The registration below refers to it by name.</li>
<li><code>frontend http_front</code>: looks up the Host header in <code>hosts.map</code> and sends the request to <code>consul-backend-127.0.0.1-8500-&lt;service&gt;</code>. Unknown hosts go to <code>web</code>.</li>
<li><code>frontend stats</code>: stats page and Prometheus metrics.</li>
</ul>
<p>Create <code>/etc/haproxy/haproxy.cfg</code>:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy

# Settings for the backends the Data Plane API creates from Consul
defaults consul_services
    mode http
    log global
    option httplog
    option dontlognull
    option forwardfor
    balance roundrobin
    timeout connect 5s
    timeout client 50s
    timeout server 50s

defaults
    mode http
    log global
    option httplog
    option dontlognull
    timeout connect 5s
    timeout client 50s
    timeout server 50s

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    http-request use-service prometheus-exporter if { path /metrics }

frontend http_front
    bind *:80
    # Host header -&gt; Consul service name (hosts.map), then its generated backend
    http-request set-var(txn.service) req.hdr(host),field(1,:),lower,map(/etc/haproxy/maps/hosts.map,web)
    use_backend consul-backend-127.0.0.1-8500-%[var(txn.service)]
</code></pre><p>And the map, <code>/etc/haproxy/maps/hosts.map</code> (host, then Consul service name):</p>
<pre tabindex="0"><code>www.example.com web
api.example.com api
</code></pre><p>Check it with <code>haproxy -c -f /etc/haproxy/haproxy.cfg</code> and restart HAProxy. Until the API has created the backends, every request gets a 503.</p>
<h3 id="backend-naming">Backend Naming <a class="anchor" data-pagefind-ignore href="#backend-naming" aria-label="Link to this section">#</a></h3>
<p>The API names each backend <code>consul-backend-&lt;consul address&gt;-&lt;consul port&gt;-&lt;service&gt;</code>. The address and port are the ones you register, so with the local agent:</p>
<table>
	<thead>
			<tr>
					<th>Consul service</th>
					<th>Backend</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>web</code></td>
					<td><code>consul-backend-127.0.0.1-8500-web</code></td>
			</tr>
			<tr>
					<td><code>api</code></td>
					<td><code>consul-backend-127.0.0.1-8500-api</code></td>
			</tr>
	</tbody>
</table>
<p>Servers inside get generated names like <code>SRV_6USXG</code>. Don&rsquo;t rely on them; they move between instances.</p>
<h3 id="installing-the-data-plane-api">Installing the Data Plane API <a class="anchor" data-pagefind-ignore href="#installing-the-data-plane-api" aria-label="Link to this section">#</a></h3>
<p>One static binary from GitHub. It runs as root, because it rewrites <code>haproxy.cfg</code> and reloads HAProxy.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">DPAPI_VERSION</span><span class="o">=</span><span class="s2">&#34;3.4.7&#34;</span>
</span></span><span class="line"><span class="cl">wget https://github.com/haproxytech/dataplaneapi/releases/download/v<span class="si">${</span><span class="nv">DPAPI_VERSION</span><span class="si">}</span>/dataplaneapi_<span class="si">${</span><span class="nv">DPAPI_VERSION</span><span class="si">}</span>_linux_x86_64.tar.gz
</span></span><span class="line"><span class="cl">tar xzf dataplaneapi_<span class="si">${</span><span class="nv">DPAPI_VERSION</span><span class="si">}</span>_linux_x86_64.tar.gz dataplaneapi
</span></span><span class="line"><span class="cl">sudo install -m <span class="m">0755</span> dataplaneapi /usr/local/bin/
</span></span><span class="line"><span class="cl">dataplaneapi --version
</span></span></code></pre></div><h3 id="data-plane-api-configuration">Data Plane API Configuration <a class="anchor" data-pagefind-ignore href="#data-plane-api-configuration" aria-label="Link to this section">#</a></h3>
<ul>
<li><code>host: 127.0.0.1</code>: the API is only for local use here. Nothing remote needs it.</li>
<li><code>user</code>: the API user, with a SHA-512 hash (<code>mkpasswd -m sha-512</code>, package <code>whois</code>).</li>
<li><code>reload</code>: how the API reloads HAProxy; the packaged unit does a seamless reload.</li>
</ul>
<p>Create <code>/etc/haproxy/dataplaneapi.yaml</code> (mode 0600; the API rewrites it in this form on every start):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">config_version</span><span class="p">:</span><span class="w"> </span><span class="m">2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy-1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">dataplaneapi</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="m">127.0.0.1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">5555</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">advertised</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">api_address</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">api_port</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scheme</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">http</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">userlist</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">userlist</span><span class="p">:</span><span class="w"> </span><span class="l">controller</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">userlist_file</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">transaction</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">transaction_dir</span><span class="p">:</span><span class="w"> </span><span class="l">/tmp/haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">user</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">admin</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">insecure</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l">&lt;mkpasswd -m sha-512 output&gt;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">haproxy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">config_file</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/haproxy.cfg</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">haproxy_bin</span><span class="p">:</span><span class="w"> </span><span class="l">/usr/sbin/haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">reload</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">reload_delay</span><span class="p">:</span><span class="w"> </span><span class="m">5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">reload_cmd</span><span class="p">:</span><span class="w"> </span><span class="l">systemctl reload haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">restart_cmd</span><span class="p">:</span><span class="w"> </span><span class="l">systemctl restart haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">reload_strategy</span><span class="p">:</span><span class="w"> </span><span class="l">custom</span><span class="w">
</span></span></span></code></pre></div><p>Create <code>/etc/systemd/system/dataplaneapi.service</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">HAProxy Data Plane API</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network.target haproxy.service consul.service</span>
</span></span><span class="line"><span class="cl"><span class="na">Requires</span><span class="o">=</span><span class="s">haproxy.service</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">simple</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/dataplaneapi -f /etc/haproxy/dataplaneapi.yaml</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
</span></span><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
</span></span></code></pre></div><p>Run <code>systemctl enable --now dataplaneapi</code>.</p>
<h3 id="registering-consul-with-the-data-plane-api">Registering Consul with the Data Plane API <a class="anchor" data-pagefind-ignore href="#registering-consul-with-the-data-plane-api" aria-label="Link to this section">#</a></h3>
<p>One API call connects the two. The API stores it in <code>/etc/haproxy/dataplane/service_discovery/consul.json</code> and keeps polling after restarts.</p>
<table>
	<thead>
			<tr>
					<th>Field</th>
					<th>Value here</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>address</code>, <code>port</code></td>
					<td><code>127.0.0.1</code>, <code>8500</code></td>
					<td>The local Consul agent. Also part of the backend names.</td>
			</tr>
			<tr>
					<td><code>token</code></td>
					<td>Data Plane API token</td>
					<td>The read-only ACL token from &ldquo;Bootstrapping ACLs&rdquo;.</td>
			</tr>
			<tr>
					<td><code>retry_timeout</code></td>
					<td><code>5</code></td>
					<td>Seconds between polls. Changes reach HAProxy within this time.</td>
			</tr>
			<tr>
					<td><code>service_allowlist</code></td>
					<td><code>[&quot;web&quot;, &quot;api&quot;]</code></td>
					<td>Only these services get a backend. Without it, every service in Consul does.</td>
			</tr>
			<tr>
					<td><code>health_check_policy</code></td>
					<td><code>all</code></td>
					<td>Which instances become servers (see below). Default <code>none</code>.</td>
			</tr>
			<tr>
					<td><code>defaults</code></td>
					<td><code>consul_services</code></td>
					<td>The named <code>defaults</code> section the backends use.</td>
			</tr>
			<tr>
					<td><code>server_slots_base</code></td>
					<td><code>10</code></td>
					<td>Server lines per backend to start with. Minimum 10.</td>
			</tr>
			<tr>
					<td><code>server_slots_growth_type</code>, <code>_increment</code></td>
					<td><code>linear</code>, <code>10</code></td>
					<td>How the slots grow when they run out. Default <code>exponential</code>.</td>
			</tr>
	</tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -u admin -X POST -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  http://127.0.0.1:5555/v3/service_discovery/consul -d <span class="s1">&#39;{
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;address&#34;: &#34;127.0.0.1&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;port&#34;: 8500,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;enabled&#34;: true,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;token&#34;: &#34;&lt;dataplaneapi token SecretID&gt;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;retry_timeout&#34;: 5,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;service_allowlist&#34;: [&#34;web&#34;, &#34;api&#34;],
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;health_check_policy&#34;: &#34;all&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;defaults&#34;: &#34;consul_services&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;server_slots_base&#34;: 10,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;server_slots_growth_type&#34;: &#34;linear&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;server_slots_growth_increment&#34;: 10
</span></span></span><span class="line"><span class="cl"><span class="s1">  }&#39;</span>
</span></span></code></pre></div><p>The answer contains an <code>id</code>. Change settings later with <code>PUT /v3/service_discovery/consul/&lt;id&gt;</code> and the full object. A few seconds later <code>haproxy.cfg</code> has the backends:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend consul-backend-127.0.0.1-8500-web from consul_services
  server SRV_YKy8O 10.0.0.21:8080 check weight 128
  server SRV_WZpFa 127.0.0.1:80 disabled weight 128
  ...
</code></pre><h2 id="choosing-services-and-instances">Choosing Services and Instances <a class="anchor" data-pagefind-ignore href="#choosing-services-and-instances" aria-label="Link to this section">#</a></h2>
<p>Two filters decide what ends up in HAProxy: which services get a backend, and which instances of a service become servers.</p>
<h3 id="which-services">Which Services <a class="anchor" data-pagefind-ignore href="#which-services" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Field</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>service_allowlist</code></td>
					<td>A fixed list of services to publish. The safe default.</td>
			</tr>
			<tr>
					<td><code>service_denylist</code></td>
					<td>Publish everything except a few (for example <code>consul</code> itself).</td>
			</tr>
			<tr>
					<td><code>service_name_regexp</code></td>
					<td>A naming convention, like <code>^web-</code>.</td>
			</tr>
	</tbody>
</table>
<p>There is no filter on tags, metadata or datacenter. Give instances that need their own backend their own service name, such as <code>web-blue</code> and <code>web-green</code>.</p>
<h3 id="which-instances-health_check_policy">Which Instances: health_check_policy <a class="anchor" data-pagefind-ignore href="#which-instances-health_check_policy" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Policy</th>
					<th>An instance becomes a server when</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>none</code> (default)</td>
					<td>Always</td>
					<td>Let HAProxy&rsquo;s own <code>check</code> decide. A failing instance stays listed but marked DOWN.</td>
			</tr>
			<tr>
					<td><code>any</code></td>
					<td>At least one check passes</td>
					<td>Services with an optional extra check.</td>
			</tr>
			<tr>
					<td><code>all</code></td>
					<td>Every check passes (node checks included)</td>
					<td>The normal choice: Consul decides.</td>
			</tr>
			<tr>
					<td><code>min</code></td>
					<td>At least <code>health_check_policy_min</code> checks pass</td>
					<td>Many checks, some allowed to fail.</td>
			</tr>
	</tbody>
</table>
<p>Every server also gets HAProxy&rsquo;s own <code>check</code> (a TCP connect with these defaults). It catches instances that are healthy locally but unreachable from the load balancer.</p>
<h2 id="scaling-what-happens-on-a-change">Scaling: What Happens on a Change <a class="anchor" data-pagefind-ignore href="#scaling-what-happens-on-a-change" aria-label="Link to this section">#</a></h2>
<p>The API pre-creates server slots: real servers first, the rest as disabled placeholders (<code>127.0.0.1:80 disabled</code>). When the slots run out, it adds more.</p>
<p>What the lab showed with Data Plane API 3.4.7:</p>
<ul>
<li><strong>Scale up</strong>: the new instance is a server within one poll (<code>retry_timeout</code>).</li>
<li><strong>Scale down or failing check</strong> (policy <code>all</code>): the instance is removed at the next poll; a healed instance comes back.</li>
<li><strong>Every change is one reload</strong>, also when a free slot was used. The reload is seamless: the old worker finishes its connections.</li>
<li><strong>Changes are batched per poll</strong>: ten instances registered together caused two reloads, not ten.</li>
<li><strong>Empty service</strong>: the backend stays, with only disabled slots, so requests get a 503.</li>
</ul>
<p>Runtime changes to servers are lost at the next reload. A weight set through the runtime API went back to 128 when the next instance was added. Don&rsquo;t build on them.</p>
<h2 id="blue-green-deployments">Blue-Green Deployments <a class="anchor" data-pagefind-ignore href="#blue-green-deployments" aria-label="Link to this section">#</a></h2>
<p>Run two copies as two Consul services, <code>web-blue</code> and <code>web-green</code>, both in <code>service_allowlist</code>. The <code>hosts.map</code> entry decides which one is live. Switch it through the runtime maps API: no reload, and <code>force_sync</code> writes the map file so the switch survives a restart.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Switch www.example.com to green</span>
</span></span><span class="line"><span class="cl">curl -u admin -X PUT -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;http://127.0.0.1:5555/v3/services/haproxy/runtime/maps/hosts.map/entries/www.example.com?force_sync=true&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;value&#34;:&#34;web-green&#34;}&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Roll back</span>
</span></span><span class="line"><span class="cl">curl -u admin -X PUT -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;http://127.0.0.1:5555/v3/services/haproxy/runtime/maps/hosts.map/entries/www.example.com?force_sync=true&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;value&#34;:&#34;web-blue&#34;}&#39;</span>
</span></span></code></pre></div><p>The same call with <code>POST .../entries</code> adds a new host route.</p>
<p>Percentage canaries don&rsquo;t fit this model: the API ignores Consul weights, and runtime weights are reset at the next reload. Use a separate host (<code>canary.example.com web-canary</code>) or a different tool for that.</p>
<h2 id="health-checks-and-service-discovery">Health Checks and Service Discovery <a class="anchor" data-pagefind-ignore href="#health-checks-and-service-discovery" aria-label="Link to this section">#</a></h2>
<p>Health checks decide who gets traffic, together with the <code>health_check_policy</code> from the registration. Too shallow, and broken instances keep receiving traffic. Too strict, and healthy ones get pulled.</p>
<h3 id="consul-health-check-types">Consul Health Check Types <a class="anchor" data-pagefind-ignore href="#consul-health-check-types" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Type</th>
					<th>Use it for</th>
					<th>Watch out</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HTTP</td>
					<td>Web apps and APIs with a <code>/health</code> endpoint</td>
					<td>Make the endpoint check real dependencies</td>
			</tr>
			<tr>
					<td>TCP</td>
					<td>Services where you can only test the port</td>
					<td>A hung app that accepts connections still passes</td>
			</tr>
			<tr>
					<td>Script</td>
					<td>Custom logic</td>
					<td>Off by default: needs <code>enable_local_script_checks</code></td>
			</tr>
			<tr>
					<td>gRPC</td>
					<td>gRPC services with the standard health protocol</td>
					<td></td>
			</tr>
			<tr>
					<td>TTL</td>
					<td>Workers and batch jobs without a port</td>
					<td>The service must report in itself</td>
			</tr>
	</tbody>
</table>
<p>The example shows all five for the syntax. A real service needs one or two.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">service</span> {
</span></span><span class="line"><span class="cl"><span class="n">  name</span> <span class="o">=</span> <span class="s2">&#34;web&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">  port</span> <span class="o">=</span> <span class="m">8080</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="n">checks</span> <span class="o">=</span> <span class="p">[</span><span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1">    # HTTP check
</span></span></span><span class="line"><span class="cl">    {
</span></span><span class="line"><span class="cl"><span class="n">      name</span> <span class="o">=</span> <span class="s2">&#34;HTTP Health&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      http</span> <span class="o">=</span> <span class="s2">&#34;http://localhost:8080/health&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      method</span> <span class="o">=</span> <span class="s2">&#34;GET&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      interval</span> <span class="o">=</span> <span class="s2">&#34;10s&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      timeout</span> <span class="o">=</span> <span class="s2">&#34;5s&#34;</span>
</span></span><span class="line"><span class="cl">    }<span class="p">,</span><span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1">    
</span></span></span><span class="line"><span class="cl"><span class="c1">    # TCP check
</span></span></span><span class="line"><span class="cl">    {
</span></span><span class="line"><span class="cl"><span class="n">      name</span> <span class="o">=</span> <span class="s2">&#34;TCP Port&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      tcp</span> <span class="o">=</span> <span class="s2">&#34;localhost:8080&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      interval</span> <span class="o">=</span> <span class="s2">&#34;5s&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      timeout</span> <span class="o">=</span> <span class="s2">&#34;2s&#34;</span>
</span></span><span class="line"><span class="cl">    }<span class="p">,</span><span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1">    
</span></span></span><span class="line"><span class="cl"><span class="c1">    # Script check (the agent needs enable_local_script_checks = true,
</span></span></span><span class="line"><span class="cl"><span class="c1">    # otherwise registering the whole service fails)
</span></span></span><span class="line"><span class="cl">    {
</span></span><span class="line"><span class="cl"><span class="n">      name</span> <span class="o">=</span> <span class="s2">&#34;Script Check&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      args</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;/usr/local/bin/health-check.sh&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">      interval</span> <span class="o">=</span> <span class="s2">&#34;30s&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      timeout</span> <span class="o">=</span> <span class="s2">&#34;10s&#34;</span>
</span></span><span class="line"><span class="cl">    }<span class="p">,</span><span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1">    
</span></span></span><span class="line"><span class="cl"><span class="c1">    # gRPC check
</span></span></span><span class="line"><span class="cl">    {
</span></span><span class="line"><span class="cl"><span class="n">      name</span> <span class="o">=</span> <span class="s2">&#34;gRPC Health&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      grpc</span> <span class="o">=</span> <span class="s2">&#34;localhost:9090&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      grpc_use_tls</span> <span class="o">=</span> <span class="kt">true</span>
</span></span><span class="line"><span class="cl"><span class="n">      interval</span> <span class="o">=</span> <span class="s2">&#34;10s&#34;</span>
</span></span><span class="line"><span class="cl">    }<span class="p">,</span><span class="c1">
</span></span></span><span class="line"><span class="cl"><span class="c1">    
</span></span></span><span class="line"><span class="cl"><span class="c1">    # TTL check (service must update)
</span></span></span><span class="line"><span class="cl">    {
</span></span><span class="line"><span class="cl"><span class="n">      name</span> <span class="o">=</span> <span class="s2">&#34;TTL Check&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      ttl</span> <span class="o">=</span> <span class="s2">&#34;30s&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">      status</span> <span class="o">=</span> <span class="s2">&#34;passing&#34;</span>
</span></span><span class="line"><span class="cl">    }
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><h3 id="updating-ttl-check">Updating TTL Check <a class="anchor" data-pagefind-ignore href="#updating-ttl-check" aria-label="Link to this section">#</a></h3>
<p>The service calls <code>pass</code> before the TTL runs out to stay healthy. Calling <code>fail</code> takes it out on purpose, for example while draining.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check IDs are service:&lt;service-id&gt;:&lt;n&gt;; the TTL check above is the 5th check of &#34;web&#34;</span>
</span></span><span class="line"><span class="cl"><span class="c1"># (curl -s http://localhost:8500/v1/agent/checks | jq keys lists them)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Pass TTL check</span>
</span></span><span class="line"><span class="cl">curl -X PUT http://localhost:8500/v1/agent/check/pass/service:web:5
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Fail TTL check</span>
</span></span><span class="line"><span class="cl">curl -X PUT http://localhost:8500/v1/agent/check/fail/service:web:5
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Warn TTL check</span>
</span></span><span class="line"><span class="cl">curl -X PUT http://localhost:8500/v1/agent/check/warn/service:web:5
</span></span></code></pre></div><h2 id="monitoring-and-observability">Monitoring and Observability <a class="anchor" data-pagefind-ignore href="#monitoring-and-observability" aria-label="Link to this section">#</a></h2>
<p>Compare what Consul thinks is healthy with what HAProxy actually uses. If they differ, look at the Data Plane API.</p>
<h3 id="consul-service-health-queries">Consul Service Health Queries <a class="anchor" data-pagefind-ignore href="#consul-service-health-queries" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># List all healthy web services</span>
</span></span><span class="line"><span class="cl">curl http://localhost:8500/v1/health/service/web?passing<span class="o">=</span><span class="nb">true</span> <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Get service catalog</span>
</span></span><span class="line"><span class="cl">curl http://localhost:8500/v1/catalog/services <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Watch for changes</span>
</span></span><span class="line"><span class="cl">watch -n <span class="m">2</span> <span class="s1">&#39;curl -s http://localhost:8500/v1/health/service/web | jq length&#39;</span>
</span></span></code></pre></div><h3 id="haproxy-stats-integration">HAProxy Stats Integration <a class="anchor" data-pagefind-ignore href="#haproxy-stats-integration" aria-label="Link to this section">#</a></h3>
<p>The base configuration already serves the stats page on <code>:8404/stats</code> and Prometheus metrics on <code>:8404/metrics</code>. See the <a href="https://blog.bsdserver.nl/haproxy-monitoring-with-prometheus-complete-observability-guide/">Prometheus monitoring post</a>.</p>
<h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<p>Follow the chain: registered → checks passing (per <code>health_check_policy</code>) → seen by the Data Plane API → server in the backend → <code>hosts.map</code> points at the service. The first broken link is the problem. On a fresh install it is usually ACLs: look for <code>blocked by ACLs</code> in the Consul log.</p>
<h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<ol>
<li><strong>No <code>consul-backend-…</code> backend at all</strong>
<ul>
<li>Is the service in <code>service_allowlist</code>?</li>
<li>Does the registration have the Data Plane API token? Without it, a default-deny Consul shows no services.</li>
<li>Check the API log: <code>journalctl -u dataplaneapi</code></li>
</ul>
</li>
<li><strong>Backend exists, requests get a 503</strong>
<ul>
<li>No instance passes the policy: every server line is a disabled slot.</li>
<li>Check the checks: <code>curl -s http://localhost:8500/v1/health/checks/web | jq .</code></li>
</ul>
</li>
<li><strong>Requests reach the wrong service</strong>
<ul>
<li>Check the map: <code>echo &quot;show map /etc/haproxy/maps/hosts.map&quot; | sudo socat - /run/haproxy/admin.sock</code></li>
</ul>
</li>
<li><strong>Hand edits to <code>haproxy.cfg</code> disappear</strong>
<ul>
<li>The Data Plane API owns the file. Change it through the API.</li>
</ul>
</li>
</ol>
<h3 id="debug-commands">Debug Commands <a class="anchor" data-pagefind-ignore href="#debug-commands" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># The registration, as the API sees it (the token is included: don&#39;t paste this output)</span>
</span></span><span class="line"><span class="cl">curl -s -u admin http://127.0.0.1:5555/v3/service_discovery/consul <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Generated backends and their servers</span>
</span></span><span class="line"><span class="cl">grep -A3 <span class="s1">&#39;^backend consul-backend&#39;</span> /etc/haproxy/haproxy.cfg
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show servers state consul-backend-127.0.0.1-8500-web&#34;</span> <span class="p">|</span> sudo socat - /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Reloads caused by the API</span>
</span></span><span class="line"><span class="cl">journalctl -u haproxy --since <span class="s2">&#34;10 min ago&#34;</span> <span class="p">|</span> grep <span class="s2">&#34;Reloading HAProxy&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Consul side</span>
</span></span><span class="line"><span class="cl">consul members
</span></span><span class="line"><span class="cl">consul catalog nodes -service<span class="o">=</span>web
</span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<p>Anyone who can register a service named <code>web</code> receives your web traffic. Protect Consul like the rest of your traffic path.</p>
<ul>
<li><strong>Read-only token for the API.</strong> The <code>dataplaneapi</code> policy only reads nodes and services; a KV write with it gets a 403. It is stored in <code>/etc/haproxy/dataplane/service_discovery/consul.json</code> (root, mode 0600).</li>
<li><strong>API on localhost.</strong> <code>host: 127.0.0.1</code>, a hashed password, <code>dataplaneapi.yaml</code> mode 0600. Add TLS before exposing it.</li>
<li><strong>Local Consul agent.</strong> The API talks plain HTTP to <code>127.0.0.1:8500</code>. The registration schema has a <code>mode</code> field (<code>http</code> or <code>https</code>) but no CA or client-certificate settings; not tested here.</li>
<li><strong>Narrow the agent token in production.</strong> The agent policy here can write every service; scope it per node and service.</li>
<li><strong>Encrypt Consul traffic.</strong> Gossip encryption (<code>encrypt</code>) and TLS on RPC between agents and servers.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-consul" rel="noopener">pderuiter/blog-haproxy-consul</a>. It creates a three-server Consul cluster with ACLs, a service node with the web and api services, and an HAProxy load balancer where the Data Plane API discovers the services in Consul; fill in the inventory, four token UUIDs and the API password, then run <code>ansible-playbook site.yml</code>.</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>With Consul service discovery integrated, consider:</p>
<ul>
<li><strong>Prometheus Monitoring</strong> - Add metrics collection for HAProxy and Consul</li>
<li><strong>Consul Connect</strong> - Implement service mesh with automatic mTLS</li>
<li><strong>Data Plane API</strong> - Manage frontends and ACLs through the same API: see the <a href="https://blog.bsdserver.nl/haproxy-data-plane-api-dynamic-configuration-management/">Data Plane API post</a></li>
</ul>
<p>Consul and HAProxy together create a self-healing setup where services join and leave the load balancer pool based on their health.</p>
]]></content:encoded></item><item><title>HAProxy Data Plane API: Dynamic Configuration Management</title><link>https://blog.bsdserver.nl/haproxy-data-plane-api-dynamic-configuration-management/</link><guid isPermaLink="false">693b760079eb54000160a61d</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:55:12 +0000</pubDate><description>Managing HAProxy configuration traditionally requires editing files and reloading the service.</description><category>HAProxy</category><category>API</category><category>Automation</category><category>DevOps</category><category>Infrastructure</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-data-plane-api.png" medium="image"/><content:encoded><![CDATA[<p>Managing HAProxy configuration traditionally requires editing files and reloading the service. That works for a person at a terminal, but not for a deployment pipeline that needs to add a server.</p>
<p>The Data Plane API is HAProxy&rsquo;s REST API for its configuration. This post installs it behind a TLS frontend, then uses it to add and remove servers, switch blue-green environments, shift canary traffic, and drive it from Python and GitLab CI.</p>
<h2 id="why-data-plane-api">Why Data Plane API? <a class="anchor" data-pagefind-ignore href="#why-data-plane-api" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Configuration over HTTP</strong> - Backends and servers through REST calls instead of editing <code>haproxy.cfg</code></li>
<li><strong>Transactions</strong> - Group several changes, commit them as one, or throw them away</li>
<li><strong>Version check</strong> - Every change names the configuration version it builds on, so two clients cannot overwrite each other</li>
<li><strong>Runtime changes</strong> - Drain, maintenance and weights on the running process, without a reload</li>
<li><strong>Integration friendly</strong> - Anything that speaks HTTP: shell scripts, Python, CI pipelines</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when tools change HAProxy: deployment pipelines, blue-green switches, service discovery.</li>
<li><strong>Skip it</strong> when the configuration changes rarely and Ansible or another tool already templates <code>haproxy.cfg</code>. That is simpler and easier to review.</li>
</ul>
<p>Pick one owner for <code>haproxy.cfg</code>. Once the API manages it, make changes through the API, not by hand.</p>
<h3 id="configuration-vs-runtime-changes">Configuration vs runtime changes <a class="anchor" data-pagefind-ignore href="#configuration-vs-runtime-changes" aria-label="Link to this section">#</a></h3>
<p>The API has two kinds of endpoints. Knowing which one you call explains most surprises.</p>
<table>
	<thead>
			<tr>
					<th>Endpoints</th>
					<th>Path</th>
					<th>What happens</th>
					<th>Survives a reload</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Configuration</td>
					<td><code>/services/haproxy/configuration/...</code></td>
					<td>Writes <code>haproxy.cfg</code>, then reloads HAProxy</td>
					<td>Yes</td>
			</tr>
			<tr>
					<td>Runtime</td>
					<td><code>/services/haproxy/runtime/...</code></td>
					<td>Changes the running process, no reload</td>
					<td>No</td>
			</tr>
	</tbody>
</table>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy</td>
					<td>Load balancer</td>
					<td>Serves the application (<code>http_front</code>) and a TLS frontend for the API on port 5556.</td>
			</tr>
			<tr>
					<td>Data Plane API</td>
					<td>Load balancer, port 5555</td>
					<td>Edits <code>haproxy.cfg</code>, reloads HAProxy, and makes runtime changes on the running process.</td>
			</tr>
			<tr>
					<td>Clients</td>
					<td>Admin machine, CI runner</td>
					<td>curl, the scripts below, Python, GitLab CI.</td>
			</tr>
	</tbody>
</table>
<p>The flow: client → HAProxy <code>:5556</code> (TLS, basic auth) → Data Plane API <code>:5555</code> → <code>haproxy.cfg</code> and a reload, or a runtime change.</p>
<h2 id="installing-the-data-plane-api">Installing the Data Plane API <a class="anchor" data-pagefind-ignore href="#installing-the-data-plane-api" aria-label="Link to this section">#</a></h2>
<h3 id="install-haproxy">Install HAProxy <a class="anchor" data-pagefind-ignore href="#install-haproxy" aria-label="Link to this section">#</a></h3>
<p>The Data Plane API 3.x needs a current HAProxy. Install it from the official HAProxy packages rather than the distribution&rsquo;s older <code>haproxy</code> package:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Official HAProxy packages for Ubuntu 24.04: one PPA per branch</span>
</span></span><span class="line"><span class="cl">sudo add-apt-repository -y ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">sudo apt update
</span></span><span class="line"><span class="cl">sudo apt install -y <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">haproxy -v
</span></span></code></pre></div><p>3.3 is the newest branch the PPA builds for Ubuntu 24.04, but it is a short-lived stable branch, maintained until about Q1 2027. The newest LTS branch, 3.4, is built for Ubuntu 26.04 (<code>ppa:vbernat/haproxy-3.4</code>) and for Debian 13 on <a href="https://haproxy.debian.net/" rel="noopener">haproxy.debian.net</a>, which lists the repository lines per release and branch. For a long-lived Ubuntu 24.04 install, use the 3.2 LTS branch instead (<code>ppa:vbernat/haproxy-3.2</code>, maintained until 2030).</p>
<h3 id="download-and-install">Download and Install <a class="anchor" data-pagefind-ignore href="#download-and-install" aria-label="Link to this section">#</a></h3>
<p>The API is one static binary from the HAProxy Technologies GitHub releases.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Download the latest 3.x release (official HAProxy Technologies build)</span>
</span></span><span class="line"><span class="cl"><span class="nv">DATAPLANEAPI_VERSION</span><span class="o">=</span><span class="s2">&#34;3.4.7&#34;</span>
</span></span><span class="line"><span class="cl">wget https://github.com/haproxytech/dataplaneapi/releases/download/v<span class="si">${</span><span class="nv">DATAPLANEAPI_VERSION</span><span class="si">}</span>/dataplaneapi_<span class="si">${</span><span class="nv">DATAPLANEAPI_VERSION</span><span class="si">}</span>_linux_x86_64.tar.gz
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Extract and install</span>
</span></span><span class="line"><span class="cl">tar xzf dataplaneapi_<span class="si">${</span><span class="nv">DATAPLANEAPI_VERSION</span><span class="si">}</span>_linux_x86_64.tar.gz
</span></span><span class="line"><span class="cl">sudo mv dataplaneapi /usr/local/bin/
</span></span><span class="line"><span class="cl">sudo chmod +x /usr/local/bin/dataplaneapi
</span></span></code></pre></div><p>This post uses the Data Plane API 3.x paths: everything lives under <code>/v3</code>, and servers are addressed through their backend (<code>/services/haproxy/configuration/backends/web_servers/servers</code>). The 2.x style <code>/v2/.../servers?backend=web_servers</code> paths no longer exist.</p>
<h3 id="configuration-file">Configuration File <a class="anchor" data-pagefind-ignore href="#configuration-file" aria-label="Link to this section">#</a></h3>
<p>This file tells the API where to listen, who may log in, and how to reload HAProxy.</p>
<ul>
<li><code>host: 0.0.0.0</code>, <code>port: 5555</code>: the API itself, plain HTTP. Clients use the TLS frontend on 5556 (below).</li>
<li><code>user</code>: the API user. <code>insecure: false</code> means <code>password</code> is a hash, not plain text.</li>
<li><code>transaction_dir</code>: where open transactions wait until they are committed.</li>
<li><code>resources</code>: directories for map files, certificates and SPOE files that the API manages.</li>
<li><code>reload</code>: the commands the API runs. <code>reload_delay: 5</code> is the minimum time in seconds between two reloads.</li>
</ul>
<p>Create <code>/etc/haproxy/dataplaneapi.yaml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">config_version</span><span class="p">:</span><span class="w"> </span><span class="m">2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_dataplane</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">dataplaneapi</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="m">0.0.0.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">5555</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scheme</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">http</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">user</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">admin</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l">$6$rounds=500000$...$... </span><span class="w"> </span><span class="c"># Use mkpasswd to generate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">insecure</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">transaction</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">transaction_dir</span><span class="p">:</span><span class="w"> </span><span class="l">/tmp/haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">resources</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">maps_dir</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/maps</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ssl_certs_dir</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/certs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spoe_dir</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/spoe</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spoe_transaction_dir</span><span class="p">:</span><span class="w"> </span><span class="l">/tmp/spoe-haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">haproxy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">config_file</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/haproxy.cfg</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">haproxy_bin</span><span class="p">:</span><span class="w"> </span><span class="l">/usr/sbin/haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">reload</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">reload_delay</span><span class="p">:</span><span class="w"> </span><span class="m">5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">reload_cmd</span><span class="p">:</span><span class="w"> </span><span class="l">systemctl reload haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">restart_cmd</span><span class="p">:</span><span class="w"> </span><span class="l">systemctl restart haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">status_cmd</span><span class="p">:</span><span class="w"> </span><span class="l">systemctl status haproxy</span><span class="w">
</span></span></span></code></pre></div><h3 id="generate-password-hash">Generate Password Hash <a class="anchor" data-pagefind-ignore href="#generate-password-hash" aria-label="Link to this section">#</a></h3>
<p>The SHA-512 hash goes into <code>dataplaneapi.yaml</code> and into the HAProxy <code>userlist</code> below.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Install mkpasswd if needed</span>
</span></span><span class="line"><span class="cl">sudo apt install whois
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Generate SHA-512 password hash</span>
</span></span><span class="line"><span class="cl">mkpasswd -m sha-512 <span class="s1">&#39;your-secure-password&#39;</span>
</span></span></code></pre></div><h2 id="haproxy-configuration-for-api">HAProxy Configuration for API <a class="anchor" data-pagefind-ignore href="#haproxy-configuration-for-api" aria-label="Link to this section">#</a></h2>
<p>HAProxy puts TLS and a second password check in front of the API, and serves the application the examples change.</p>
<ul>
<li><code>stats socket ... level admin</code>: the runtime socket. Runtime changes go through it.</li>
<li><code>frontend dataplane_api</code>: TLS on port 5556 with <code>api.pem</code>, basic auth against <code>api_users</code>, then on to the API on <code>127.0.0.1:5555</code>.</li>
<li><code>userlist api_users</code>: the same user and hash as in <code>dataplaneapi.yaml</code>. The <code>Authorization</code> header is passed on, so the API checks it again.</li>
<li><code>http_front</code> and <code>web_servers</code>: the application. <code>option httpchk GET /health</code> health-checks each server.</li>
</ul>
<p>Add to <code>/etc/haproxy/haproxy.cfg</code>:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    # No &#34;master-worker&#34; here: it is deprecated since HAProxy 3.3. The packaged
    # systemd unit already starts HAProxy in master-worker mode (-Ws).

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

# Data Plane API Frontend
frontend dataplane_api
    bind *:5556 ssl crt /etc/haproxy/certs/api.pem
    mode http
    http-request auth unless { http_auth(api_users) }
    default_backend dataplane_api_backend

backend dataplane_api_backend
    mode http
    server dataplane 127.0.0.1:5555

userlist api_users
    user admin password $6$rounds=500000$...$...

# Application frontends and backends
frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check
</code></pre><h2 id="starting-the-data-plane-api">Starting the Data Plane API <a class="anchor" data-pagefind-ignore href="#starting-the-data-plane-api" aria-label="Link to this section">#</a></h2>
<h3 id="systemd-service">Systemd Service <a class="anchor" data-pagefind-ignore href="#systemd-service" aria-label="Link to this section">#</a></h3>
<p>The unit starts the API after HAProxy and stops it when HAProxy stops (<code>Requires</code>). It runs as root; the comment in the unit explains why.</p>
<p>Create <code>/etc/systemd/system/dataplaneapi.service</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">HAProxy Data Plane API</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network.target haproxy.service</span>
</span></span><span class="line"><span class="cl"><span class="na">Requires</span><span class="o">=</span><span class="s">haproxy.service</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">simple</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/dataplaneapi -f /etc/haproxy/dataplaneapi.yaml</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
</span></span><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Runs as root: it rewrites /etc/haproxy/haproxy.cfg and its own config file,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># and calls systemctl to reload HAProxy. As User=haproxy it exits with</span>
</span></span><span class="line"><span class="cl"><span class="c1"># &#34;Error saving configuration: ... permission denied&#34;.</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo systemctl daemon-reload
</span></span><span class="line"><span class="cl">sudo systemctl <span class="nb">enable</span> dataplaneapi
</span></span><span class="line"><span class="cl">sudo systemctl start dataplaneapi
</span></span></code></pre></div><h2 id="api-operations">API Operations <a class="anchor" data-pagefind-ignore href="#api-operations" aria-label="Link to this section">#</a></h2>
<h3 id="authentication">Authentication <a class="anchor" data-pagefind-ignore href="#authentication" aria-label="Link to this section">#</a></h3>
<p>All examples use these two variables. <code>/info</code> is a quick test of the connection and the credentials.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Set credentials</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">HAPROXY_API</span><span class="o">=</span><span class="s2">&#34;https://haproxy.example.com:5556/v3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">AUTH</span><span class="o">=</span><span class="s2">&#34;admin:your-secure-password&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test connection</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/info <span class="p">|</span> jq .
</span></span></code></pre></div><h3 id="get-current-configuration">Get Current Configuration <a class="anchor" data-pagefind-ignore href="#get-current-configuration" aria-label="Link to this section">#</a></h3>
<p>Read before you write: the raw configuration, one backend, or its servers.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Get HAProxy configuration</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/raw
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Get specific backend</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/web_servers&#34;</span> <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># List all servers in a backend</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/web_servers/servers&#34;</span> <span class="p">|</span> jq .
</span></span></code></pre></div><h3 id="working-with-transactions">Working with Transactions <a class="anchor" data-pagefind-ignore href="#working-with-transactions" aria-label="Link to this section">#</a></h3>
<p>Every configuration change follows the same pattern: version → transaction → changes → commit.</p>
<ul>
<li><strong>Version</strong>: the transaction is tied to the configuration version you read. If another client changed the configuration in between, the API rejects it; start again with a fresh version.</li>
<li><strong>Changes</strong>: calls with <code>?transaction_id=</code> are collected, not applied.</li>
<li><strong>Commit</strong> (<code>PUT</code>): writes all changes to <code>haproxy.cfg</code> at once and reloads HAProxy. <strong>Rollback</strong> (<code>DELETE</code>) throws them away.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Get current configuration version</span>
</span></span><span class="line"><span class="cl"><span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Current version: </span><span class="nv">$VERSION</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Start a transaction</span>
</span></span><span class="line"><span class="cl"><span class="nv">TRANSACTION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$VERSION</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Transaction ID: </span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Make changes within transaction (see examples below)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Commit transaction</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Or rollback if needed</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X DELETE <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span></code></pre></div><h3 id="adding-a-server">Adding a Server <a class="anchor" data-pagefind-ignore href="#adding-a-server" aria-label="Link to this section">#</a></h3>
<p><code>check: enabled</code> turns on health checks for the server. <code>weight</code> is its share of the traffic, relative to the other servers in the backend.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Get version and start transaction</span>
</span></span><span class="line"><span class="cl"><span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nv">TRANSACTION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$VERSION</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Add new server</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/web_servers/servers?transaction_id=</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;name&#34;: &#34;web3&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;address&#34;: &#34;192.168.1.12&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;port&#34;: 8080,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;check&#34;: &#34;enabled&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;weight&#34;: 100
</span></span></span><span class="line"><span class="cl"><span class="s1">  }&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Commit changes</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span></code></pre></div><h3 id="removing-a-server">Removing a Server <a class="anchor" data-pagefind-ignore href="#removing-a-server" aria-label="Link to this section">#</a></h3>
<p>Delete the server by name. Drain it first (see Runtime API Operations) if it still has active connections.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nv">TRANSACTION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$VERSION</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Remove server</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X DELETE <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/web_servers/servers/web3?transaction_id=</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span></code></pre></div><h3 id="modifying-server-weight">Modifying Server Weight <a class="anchor" data-pagefind-ignore href="#modifying-server-weight" aria-label="Link to this section">#</a></h3>
<p>A <code>PUT</code> replaces the whole server object, so send every field, not just the weight. The new weight is written to <code>haproxy.cfg</code>; for a temporary change use the runtime weight below.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nv">TRANSACTION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$VERSION</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Update server weight</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/web_servers/servers/web1?transaction_id=</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;name&#34;: &#34;web1&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;address&#34;: &#34;192.168.1.10&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;port&#34;: 8080,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;check&#34;: &#34;enabled&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;weight&#34;: 50
</span></span></span><span class="line"><span class="cl"><span class="s1">  }&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span></code></pre></div><h3 id="creating-a-new-backend">Creating a New Backend <a class="anchor" data-pagefind-ignore href="#creating-a-new-backend" aria-label="Link to this section">#</a></h3>
<p>The backend and its servers go into one transaction, so HAProxy never loads an empty <code>api_servers</code>.</p>
<ul>
<li><code>balance: leastconn</code>: each request goes to the server with the fewest active connections. Good for requests of uneven length.</li>
<li><code>adv_check: httpchk</code> with <code>httpchk_params</code>: health-check every server with <code>GET /health</code>.</li>
</ul>
<p>No frontend sends traffic to <code>api_servers</code> yet; add a <code>use_backend</code> rule for that.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="nv">TRANSACTION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$VERSION</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create backend</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends?transaction_id=</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;name&#34;: &#34;api_servers&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;mode&#34;: &#34;http&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;balance&#34;: {&#34;algorithm&#34;: &#34;leastconn&#34;},
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;adv_check&#34;: &#34;httpchk&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">    &#34;httpchk_params&#34;: {
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;method&#34;: &#34;GET&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;uri&#34;: &#34;/health&#34;
</span></span></span><span class="line"><span class="cl"><span class="s1">    }
</span></span></span><span class="line"><span class="cl"><span class="s1">  }&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Add servers to new backend</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/api_servers/servers?transaction_id=</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;name&#34;: &#34;api1&#34;, &#34;address&#34;: &#34;192.168.1.20&#34;, &#34;port&#34;: 3000, &#34;check&#34;: &#34;enabled&#34;}&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/api_servers/servers?transaction_id=</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;name&#34;: &#34;api2&#34;, &#34;address&#34;: &#34;192.168.1.21&#34;, &#34;port&#34;: 3000, &#34;check&#34;: &#34;enabled&#34;}&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TRANSACTION</span><span class="s2">&#34;</span>
</span></span></code></pre></div><h2 id="runtime-api-operations">Runtime API Operations <a class="anchor" data-pagefind-ignore href="#runtime-api-operations" aria-label="Link to this section">#</a></h2>
<p>Runtime calls change the running HAProxy directly: no transaction, no reload, nothing written to <code>haproxy.cfg</code>. Use them for short-lived changes like taking a server out during a deploy.</p>
<h3 id="server-state-management">Server State Management <a class="anchor" data-pagefind-ignore href="#server-state-management" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th><code>admin_state</code></th>
					<th>Effect</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>drain</code></td>
					<td>No new traffic; existing connections finish</td>
					<td>Before a deploy or a restart</td>
			</tr>
			<tr>
					<td><code>maint</code></td>
					<td>No traffic at all</td>
					<td>A server down for maintenance</td>
			</tr>
			<tr>
					<td><code>ready</code></td>
					<td>Normal operation</td>
					<td>Putting the server back</td>
			</tr>
	</tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Get server runtime state</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/runtime/backends/web_servers/servers&#34;</span> <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Disable server (drain mode)</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/runtime/backends/web_servers/servers/web1&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;admin_state&#34;: &#34;drain&#34;}&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Set server to maintenance</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/runtime/backends/web_servers/servers/web1&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;admin_state&#34;: &#34;maint&#34;}&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Re-enable server</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/runtime/backends/web_servers/servers/web1&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;admin_state&#34;: &#34;ready&#34;}&#39;</span>
</span></span></code></pre></div><h3 id="runtime-weight-adjustment">Runtime Weight Adjustment <a class="anchor" data-pagefind-ignore href="#runtime-weight-adjustment" aria-label="Link to this section">#</a></h3>
<p>The runtime weight is not written to <code>haproxy.cfg</code>; it lasts until the next reload.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Adjust weight without reload</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/runtime/backends/web_servers/servers/web1&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  -d <span class="s1">&#39;{&#34;operational_state&#34;: &#34;up&#34;, &#34;weight&#34;: 150}&#39;</span>
</span></span></code></pre></div><h2 id="automation-scripts">Automation Scripts <a class="anchor" data-pagefind-ignore href="#automation-scripts" aria-label="Link to this section">#</a></h2>
<h3 id="blue-green-deployment-script">Blue-Green Deployment Script <a class="anchor" data-pagefind-ignore href="#blue-green-deployment-script" aria-label="Link to this section">#</a></h3>
<p>Swaps a whole set of servers in one transaction: <code>green</code> adds the green servers and removes the blue ones, <code>blue</code> does the opposite. One commit, so both changes reach HAProxy in the same reload.</p>
<ul>
<li>Servers are listed as <code>name:address:port</code>.</li>
<li><code>--fail-with-body</code> with <code>set -e</code>: a rejected change stops the script before the commit.</li>
<li>Removing ignores errors (<code>|| true</code>), so it also works when the old servers are already gone.</li>
</ul>
<p>Run it as <code>./blue-green-deploy.sh green</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># blue-green-deploy.sh - Switch traffic between blue and green environments</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -e
</span></span><span class="line"><span class="cl"><span class="c1"># --fail-with-body makes API errors (4xx/5xx) fail the script instead of</span>
</span></span><span class="line"><span class="cl"><span class="c1"># printing &#34;Deployment complete!&#34; after a rejected change</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_API</span><span class="o">=</span><span class="s2">&#34;https://haproxy.example.com:5556/v3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">AUTH</span><span class="o">=</span><span class="s2">&#34;admin:your-secure-password&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">BACKEND</span><span class="o">=</span><span class="s2">&#34;app_servers&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">BLUE_SERVERS</span><span class="o">=(</span><span class="s2">&#34;blue1:192.168.1.10:8080&#34;</span> <span class="s2">&#34;blue2:192.168.1.11:8080&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl"><span class="nv">GREEN_SERVERS</span><span class="o">=(</span><span class="s2">&#34;green1:192.168.1.20:8080&#34;</span> <span class="s2">&#34;green2:192.168.1.21:8080&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">get_version<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">start_transaction<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$1</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">add_server<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">name</span><span class="o">=</span><span class="nv">$1</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">addr</span><span class="o">=</span><span class="nv">$2</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">port</span><span class="o">=</span><span class="nv">$3</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">txn</span><span class="o">=</span><span class="nv">$4</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  curl -s --fail-with-body -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/</span><span class="nv">$BACKEND</span><span class="s2">/servers?transaction_id=</span><span class="nv">$txn</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    -d <span class="s2">&#34;{\&#34;name\&#34;: \&#34;</span><span class="nv">$name</span><span class="s2">\&#34;, \&#34;address\&#34;: \&#34;</span><span class="nv">$addr</span><span class="s2">\&#34;, \&#34;port\&#34;: </span><span class="nv">$port</span><span class="s2">, \&#34;check\&#34;: \&#34;enabled\&#34;}&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">remove_server<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">name</span><span class="o">=</span><span class="nv">$1</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">txn</span><span class="o">=</span><span class="nv">$2</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  curl -s --fail-with-body -u <span class="nv">$AUTH</span> -X DELETE <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/</span><span class="nv">$BACKEND</span><span class="s2">/servers/</span><span class="si">${</span><span class="nv">name</span><span class="si">}</span><span class="s2">?transaction_id=</span><span class="nv">$txn</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">commit_transaction<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  curl -s --fail-with-body -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$1</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">deploy_environment<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">target</span><span class="o">=</span><span class="nv">$1</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> servers_to_add
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> servers_to_remove
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="o">[</span> <span class="s2">&#34;</span><span class="nv">$target</span><span class="s2">&#34;</span> <span class="o">==</span> <span class="s2">&#34;green&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    <span class="nv">servers_to_add</span><span class="o">=(</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">GREEN_SERVERS</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="nv">servers_to_remove</span><span class="o">=(</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">BLUE_SERVERS</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">else</span>
</span></span><span class="line"><span class="cl">    <span class="nv">servers_to_add</span><span class="o">=(</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">BLUE_SERVERS</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">    <span class="nv">servers_to_remove</span><span class="o">=(</span><span class="s2">&#34;</span><span class="si">${</span><span class="nv">GREEN_SERVERS</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span><span class="o">)</span>
</span></span><span class="line"><span class="cl">  <span class="k">fi</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;Switching to </span><span class="nv">$target</span><span class="s2"> environment...&#34;</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>get_version<span class="k">)</span>
</span></span><span class="line"><span class="cl">  <span class="nv">TXN</span><span class="o">=</span><span class="k">$(</span>start_transaction <span class="nv">$VERSION</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="c1"># Add new servers</span>
</span></span><span class="line"><span class="cl">  <span class="k">for</span> server in <span class="s2">&#34;</span><span class="si">${</span><span class="nv">servers_to_add</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    <span class="nv">IFS</span><span class="o">=</span><span class="s1">&#39;:&#39;</span> <span class="nb">read</span> -r name addr port <span class="o">&lt;&lt;&lt;</span> <span class="s2">&#34;</span><span class="nv">$server</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;Adding </span><span class="nv">$name</span><span class="s2"> (</span><span class="nv">$addr</span><span class="s2">:</span><span class="nv">$port</span><span class="s2">)&#34;</span>
</span></span><span class="line"><span class="cl">    add_server <span class="nv">$name</span> <span class="nv">$addr</span> <span class="nv">$port</span> <span class="nv">$TXN</span>
</span></span><span class="line"><span class="cl">  <span class="k">done</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="c1"># Remove old servers</span>
</span></span><span class="line"><span class="cl">  <span class="k">for</span> server in <span class="s2">&#34;</span><span class="si">${</span><span class="nv">servers_to_remove</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    <span class="nv">IFS</span><span class="o">=</span><span class="s1">&#39;:&#39;</span> <span class="nb">read</span> -r name addr port <span class="o">&lt;&lt;&lt;</span> <span class="s2">&#34;</span><span class="nv">$server</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;Removing </span><span class="nv">$name</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">    remove_server <span class="nv">$name</span> <span class="nv">$TXN</span> 2&gt;/dev/null <span class="o">||</span> <span class="nb">true</span>
</span></span><span class="line"><span class="cl">  <span class="k">done</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  commit_transaction <span class="nv">$TXN</span>
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;Deployment complete!&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span> in
</span></span><span class="line"><span class="cl">  blue<span class="o">)</span>  deploy_environment blue <span class="p">;;</span>
</span></span><span class="line"><span class="cl">  green<span class="o">)</span> deploy_environment green <span class="p">;;</span>
</span></span><span class="line"><span class="cl">  *<span class="o">)</span>     <span class="nb">echo</span> <span class="s2">&#34;Usage: </span><span class="nv">$0</span><span class="s2"> {blue|green}&#34;</span> <span class="p">;;</span>
</span></span><span class="line"><span class="cl"><span class="k">esac</span>
</span></span></code></pre></div><h3 id="canary-deployment-script">Canary Deployment Script <a class="anchor" data-pagefind-ignore href="#canary-deployment-script" aria-label="Link to this section">#</a></h3>
<p>Adds one canary server with weight 0, then raises its weight step by step.</p>
<table>
	<thead>
			<tr>
					<th>Command</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>add</code></td>
					<td>Adds <code>web-canary</code> to the configuration with weight 0: it receives no traffic yet.</td>
			</tr>
			<tr>
					<td><code>shift &lt;n&gt;</code></td>
					<td>Sets the canary&rsquo;s runtime weight to <code>n</code>. No reload.</td>
			</tr>
			<tr>
					<td><code>remove</code></td>
					<td>Deletes the canary from the configuration.</td>
			</tr>
	</tbody>
</table>
<p>Two things to keep in mind:</p>
<ul>
<li>The weight is relative to the other servers in the backend, and the script does not change those. The printed percentage only matches when the production weights add up to <code>100 - n</code>.</li>
<li>Runtime weights last until the next reload. Any configuration change sets the canary back to weight 0.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># canary-deploy.sh - Gradual traffic shifting to new version</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">set</span> -e
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">HAPROXY_API</span><span class="o">=</span><span class="s2">&#34;https://haproxy.example.com:5556/v3&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">AUTH</span><span class="o">=</span><span class="s2">&#34;admin:your-secure-password&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">BACKEND</span><span class="o">=</span><span class="s2">&#34;web_servers&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">CANARY_SERVER</span><span class="o">=</span><span class="s2">&#34;web-canary&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">CANARY_ADDRESS</span><span class="o">=</span><span class="s2">&#34;192.168.1.50&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nv">CANARY_PORT</span><span class="o">=</span><span class="s2">&#34;8080&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">shift_traffic<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">canary_weight</span><span class="o">=</span><span class="nv">$1</span>
</span></span><span class="line"><span class="cl">  <span class="nb">local</span> <span class="nv">production_weight</span><span class="o">=</span><span class="k">$((</span><span class="m">100</span> <span class="o">-</span> canary_weight<span class="k">))</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;Shifting traffic: Canary=</span><span class="nv">$canary_weight</span><span class="s2">%, Production=</span><span class="nv">$production_weight</span><span class="s2">%&#34;</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  <span class="c1"># Use runtime API for immediate weight changes</span>
</span></span><span class="line"><span class="cl">  curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/runtime/backends/</span><span class="nv">$BACKEND</span><span class="s2">/servers/</span><span class="si">${</span><span class="nv">CANARY_SERVER</span><span class="si">}</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    -d <span class="s2">&#34;{\&#34;weight\&#34;: </span><span class="nv">$canary_weight</span><span class="s2">}&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">add_canary<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version<span class="k">)</span>
</span></span><span class="line"><span class="cl">  <span class="nv">TXN</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$VERSION</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/</span><span class="nv">$BACKEND</span><span class="s2">/servers?transaction_id=</span><span class="nv">$TXN</span><span class="s2">&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    -H <span class="s2">&#34;Content-Type: application/json&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">    -d <span class="s1">&#39;{
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;name&#34;: &#34;&#39;</span><span class="nv">$CANARY_SERVER</span><span class="s1">&#39;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;address&#34;: &#34;&#39;</span><span class="nv">$CANARY_ADDRESS</span><span class="s1">&#39;&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;port&#34;: &#39;</span><span class="nv">$CANARY_PORT</span><span class="s1">&#39;,
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;check&#34;: &#34;enabled&#34;,
</span></span></span><span class="line"><span class="cl"><span class="s1">      &#34;weight&#34;: 0
</span></span></span><span class="line"><span class="cl"><span class="s1">    }&#39;</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TXN</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;Canary server added with 0% traffic&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">remove_canary<span class="o">()</span> <span class="o">{</span>
</span></span><span class="line"><span class="cl">  <span class="nv">VERSION</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/configuration/version<span class="k">)</span>
</span></span><span class="line"><span class="cl">  <span class="nv">TXN</span><span class="o">=</span><span class="k">$(</span>curl -s -u <span class="nv">$AUTH</span> -X POST <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions?version=</span><span class="nv">$VERSION</span><span class="s2">&#34;</span> <span class="p">|</span> jq -r <span class="s1">&#39;.id&#39;</span><span class="k">)</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  curl -s -u <span class="nv">$AUTH</span> -X DELETE <span class="se">\
</span></span></span><span class="line"><span class="cl">    <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/configuration/backends/</span><span class="nv">$BACKEND</span><span class="s2">/servers/</span><span class="si">${</span><span class="nv">CANARY_SERVER</span><span class="si">}</span><span class="s2">?transaction_id=</span><span class="nv">$TXN</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">  
</span></span><span class="line"><span class="cl">  curl -s -u <span class="nv">$AUTH</span> -X PUT <span class="s2">&#34;</span><span class="nv">$HAPROXY_API</span><span class="s2">/services/haproxy/transactions/</span><span class="nv">$TXN</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="nb">echo</span> <span class="s2">&#34;Canary server removed&#34;</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">case</span> <span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span> in
</span></span><span class="line"><span class="cl">  add<span class="o">)</span>    add_canary <span class="p">;;</span>
</span></span><span class="line"><span class="cl">  remove<span class="o">)</span> remove_canary <span class="p">;;</span>
</span></span><span class="line"><span class="cl">  <span class="nb">shift</span><span class="o">)</span>  shift_traffic <span class="nv">$2</span> <span class="p">;;</span>
</span></span><span class="line"><span class="cl">  *<span class="o">)</span>      <span class="nb">echo</span> <span class="s2">&#34;Usage: </span><span class="nv">$0</span><span class="s2"> {add|remove|shift &lt;percentage&gt;}&#34;</span> <span class="p">;;</span>
</span></span><span class="line"><span class="cl"><span class="k">esac</span>
</span></span></code></pre></div><h2 id="python-client-library">Python Client Library <a class="anchor" data-pagefind-ignore href="#python-client-library" aria-label="Link to this section">#</a></h2>
<p>The same calls wrapped in a small class, for tools written in Python.</p>
<ul>
<li><code>add_server</code> and <code>remove_server</code> each run their own transaction, and roll it back when the API rejects the change.</li>
<li><code>set_server_state</code> and <code>set_server_weight</code> are runtime calls: no transaction.</li>
<li><code>session.verify = True</code> checks the API&rsquo;s TLS certificate. Point it at your CA file rather than turning it off.</li>
<li>The <code>time.sleep(6)</code> in the example waits for the reload: a new server only exists in the running HAProxy after <code>reload_delay</code>.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="ch">#!/usr/bin/env python3</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">requests</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">requests.auth</span> <span class="kn">import</span> <span class="n">HTTPBasicAuth</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">HAProxyAPI</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">base_url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="n">base_url</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s1">&#39;/&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">auth</span> <span class="o">=</span> <span class="n">HTTPBasicAuth</span><span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">auth</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">auth</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">verify</span> <span class="o">=</span> <span class="kc">True</span>  <span class="c1"># Set to False for self-signed certs</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">get_version</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Get current configuration version&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/configuration/version&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">start_transaction</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Start a new transaction&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">version</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">version</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_version</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/transactions?version=</span><span class="si">{</span><span class="n">version</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s1">&#39;id&#39;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">commit_transaction</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">txn_id</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Commit a transaction&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/transactions/</span><span class="si">{</span><span class="n">txn_id</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">202</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">rollback_transaction</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">txn_id</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Rollback a transaction&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/transactions/</span><span class="si">{</span><span class="n">txn_id</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">204</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">get_backends</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;List all backends&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/configuration/backends&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">get_servers</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">backend</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;List servers in a backend&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/configuration/backends/</span><span class="si">{</span><span class="n">backend</span><span class="si">}</span><span class="s2">/servers&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">add_server</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">backend</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">address</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span> <span class="n">weight</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">check</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Add a server to a backend&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">txn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">start_transaction</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        
</span></span><span class="line"><span class="cl">        <span class="n">data</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s1">&#39;name&#39;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="s1">&#39;address&#39;</span><span class="p">:</span> <span class="n">address</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="s1">&#39;port&#39;</span><span class="p">:</span> <span class="n">port</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="s1">&#39;weight&#39;</span><span class="p">:</span> <span class="n">weight</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="s1">&#39;check&#39;</span><span class="p">:</span> <span class="s1">&#39;enabled&#39;</span> <span class="k">if</span> <span class="n">check</span> <span class="k">else</span> <span class="s1">&#39;disabled&#39;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">        
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/configuration/backends/</span><span class="si">{</span><span class="n">backend</span><span class="si">}</span><span class="s2">/servers&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">params</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;transaction_id&#39;</span><span class="p">:</span> <span class="n">txn</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="n">json</span><span class="o">=</span><span class="n">data</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">200</span><span class="p">,</span> <span class="mi">201</span><span class="p">,</span> <span class="mi">202</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">commit_transaction</span><span class="p">(</span><span class="n">txn</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="kc">True</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">rollback_transaction</span><span class="p">(</span><span class="n">txn</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">remove_server</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">backend</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Remove a server from a backend&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">txn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">start_transaction</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/configuration/backends/</span><span class="si">{</span><span class="n">backend</span><span class="si">}</span><span class="s2">/servers/</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">params</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;transaction_id&#39;</span><span class="p">:</span> <span class="n">txn</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">200</span><span class="p">,</span> <span class="mi">202</span><span class="p">,</span> <span class="mi">204</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">commit_transaction</span><span class="p">(</span><span class="n">txn</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="kc">True</span>
</span></span><span class="line"><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">rollback_transaction</span><span class="p">(</span><span class="n">txn</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">set_server_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">backend</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Set server runtime state (ready/drain/maint)&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">put</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/runtime/backends/</span><span class="si">{</span><span class="n">backend</span><span class="si">}</span><span class="s2">/servers/</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;admin_state&#39;</span><span class="p">:</span> <span class="n">state</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">set_server_weight</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">backend</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;&#34;&#34;Set server weight at runtime&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">put</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="si">}</span><span class="s2">/services/haproxy/runtime/backends/</span><span class="si">{</span><span class="n">backend</span><span class="si">}</span><span class="s2">/servers/</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;weight&#39;</span><span class="p">:</span> <span class="n">weight</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Usage example</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">api</span> <span class="o">=</span> <span class="n">HAProxyAPI</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s1">&#39;https://haproxy.example.com:5556/v3&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s1">&#39;admin&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s1">&#39;your-secure-password&#39;</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># List backends</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Backends:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">backend</span> <span class="ow">in</span> <span class="n">api</span><span class="o">.</span><span class="n">get_backends</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;  - </span><span class="si">{</span><span class="n">backend</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># List servers</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;</span><span class="se">\n</span><span class="s2">Servers in web_servers:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">server</span> <span class="ow">in</span> <span class="n">api</span><span class="o">.</span><span class="n">get_servers</span><span class="p">(</span><span class="s1">&#39;web_servers&#39;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;  - </span><span class="si">{</span><span class="n">server</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">server</span><span class="p">[</span><span class="s1">&#39;address&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">server</span><span class="p">[</span><span class="s1">&#39;port&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Add a new server</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;</span><span class="se">\n</span><span class="s2">Adding new server...&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">api</span><span class="o">.</span><span class="n">add_server</span><span class="p">(</span><span class="s1">&#39;web_servers&#39;</span><span class="p">,</span> <span class="s1">&#39;web4&#39;</span><span class="p">,</span> <span class="s1">&#39;192.168.1.14&#39;</span><span class="p">,</span> <span class="mi">8080</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># The new server exists in the running HAProxy only after the reload</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># (reload_delay); before that the runtime call fails with &#34;not found&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">6</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    <span class="c1"># Put server in maintenance</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Setting web4 to maintenance...&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="n">api</span><span class="o">.</span><span class="n">set_server_state</span><span class="p">(</span><span class="s1">&#39;web_servers&#39;</span><span class="p">,</span> <span class="s1">&#39;web4&#39;</span><span class="p">,</span> <span class="s1">&#39;maint&#39;</span><span class="p">))</span>
</span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<p>The API can rewrite your whole load balancer. Protect it like root access.</p>
<ul>
<li><strong>Close port 5555.</strong> In the configuration above the API listens on <code>0.0.0.0:5555</code> without TLS. Firewall it, so clients can only use the frontend on 5556.</li>
<li><strong>Keep passwords out of scripts.</strong> The examples set <code>AUTH</code> inline for readability; use a masked CI variable like <code>HAPROXY_AUTH</code> instead.</li>
<li><strong>Restrict who can connect</strong>, and add TLS or mTLS on the API itself (below).</li>
</ul>
<h3 id="tls-configuration">TLS Configuration <a class="anchor" data-pagefind-ignore href="#tls-configuration" aria-label="Link to this section">#</a></h3>
<p>The API can terminate TLS itself. <code>tls_ca</code> makes it require client certificates signed by that CA (mTLS). Keep these files outside <code>/etc/haproxy/certs/</code>, as the comment explains.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># dataplaneapi.yaml with TLS</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">dataplaneapi</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="m">0.0.0.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">5555</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scheme</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">https</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">tls</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># Not in /etc/haproxy/certs/: HAProxy loads every file in that directory</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># for &#34;bind *:443 ssl crt /etc/haproxy/certs/&#34; and fails on a bare key or CA</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">tls_certificate</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/api-tls/api.pem</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">tls_key</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/api-tls/api.key</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">tls_ca</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/haproxy/api-tls/ca.pem </span><span class="w"> </span><span class="c"># For mTLS</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">tls_port</span><span class="p">:</span><span class="w"> </span><span class="m">5555</span><span class="w">
</span></span></span></code></pre></div><h3 id="network-restrictions">Network Restrictions <a class="anchor" data-pagefind-ignore href="#network-restrictions" aria-label="Link to this section">#</a></h3>
<p>Extra rules on the API frontend:</p>
<ul>
<li><code>acl allowed_networks</code> with <code>http-request deny</code>: only the management networks reach the API.</li>
<li><code>stick-table</code> and <code>track-sc0</code>: count requests per source IP over 10 seconds; above 100 the client gets a 429.</li>
<li>The rules run in order, so blocked clients are refused before the password check.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dataplane_api
    bind *:5556 ssl crt /etc/haproxy/certs/api.pem
    
    # Restrict to management network
    acl allowed_networks src 10.0.0.0/8 192.168.0.0/16
    http-request deny unless allowed_networks
    
    # Rate limiting
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
    
    http-request auth unless { http_auth(api_users) }
    default_backend dataplane_api_backend
</code></pre><h2 id="integration-with-cicd">Integration with CI/CD <a class="anchor" data-pagefind-ignore href="#integration-with-cicd" aria-label="Link to this section">#</a></h2>
<h3 id="gitlab-ci-example">GitLab CI Example <a class="anchor" data-pagefind-ignore href="#gitlab-ci-example" aria-label="Link to this section">#</a></h3>
<p>The canary flow as a pipeline. <code>HAPROXY_AUTH</code> (<code>user:password</code>) and <code>CANARY_IP</code> come from CI/CD variables.</p>
<table>
	<thead>
			<tr>
					<th>Job</th>
					<th>Runs</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>deploy-canary</code></td>
					<td>Automatically on <code>main</code></td>
					<td>Adds <code>canary</code> with weight 10, in a transaction</td>
			</tr>
			<tr>
					<td><code>shift-traffic-50</code></td>
					<td>Manual</td>
					<td>Runtime weight 50</td>
			</tr>
			<tr>
					<td><code>promote-canary</code></td>
					<td>Manual</td>
					<td>Runtime weight 100, drains <code>web1</code> to <code>web3</code></td>
			</tr>
	</tbody>
</table>
<p>The runtime weights go back to 10 at the next reload. Make the promotion permanent with a configuration change.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .gitlab-ci.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">stages</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">deploy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">traffic</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">variables</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">HAPROXY_API</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://haproxy.example.com:5556/v3&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">deploy-canary</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l">deploy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">script</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      # Add canary server
</span></span></span><span class="line"><span class="cl"><span class="sd">      VERSION=$(curl -s -u $HAPROXY_AUTH $HAPROXY_API/services/haproxy/configuration/version)
</span></span></span><span class="line"><span class="cl"><span class="sd">      TXN=$(curl -s -u $HAPROXY_AUTH -X POST \
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;$HAPROXY_API/services/haproxy/transactions?version=$VERSION&#34; | jq -r &#39;.id&#39;)
</span></span></span><span class="line"><span class="cl"><span class="sd">      
</span></span></span><span class="line"><span class="cl"><span class="sd">      curl -s -u $HAPROXY_AUTH -X POST \
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;$HAPROXY_API/services/haproxy/configuration/backends/app_servers/servers?transaction_id=$TXN&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">        -H &#34;Content-Type: application/json&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">        -d &#39;{&#34;name&#34;: &#34;canary&#34;, &#34;address&#34;: &#34;&#39;$CANARY_IP&#39;&#34;, &#34;port&#34;: 8080, &#34;weight&#34;: 10, &#34;check&#34;: &#34;enabled&#34;}&#39;
</span></span></span><span class="line"><span class="cl"><span class="sd">      
</span></span></span><span class="line"><span class="cl"><span class="sd">      curl -s -u $HAPROXY_AUTH -X PUT &#34;$HAPROXY_API/services/haproxy/transactions/$TXN&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">only</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">shift-traffic-50</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l">traffic</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">script</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      curl -s -u $HAPROXY_AUTH -X PUT \
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;$HAPROXY_API/services/haproxy/runtime/backends/app_servers/servers/canary&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">        -H &#34;Content-Type: application/json&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">        -d &#39;{&#34;weight&#34;: 50}&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">manual</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">only</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">promote-canary</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l">traffic</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">script</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      # Shift all traffic to canary
</span></span></span><span class="line"><span class="cl"><span class="sd">      curl -s -u $HAPROXY_AUTH -X PUT \
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;$HAPROXY_API/services/haproxy/runtime/backends/app_servers/servers/canary&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">        -H &#34;Content-Type: application/json&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">        -d &#39;{&#34;weight&#34;: 100}&#39;
</span></span></span><span class="line"><span class="cl"><span class="sd">      
</span></span></span><span class="line"><span class="cl"><span class="sd">      # Set old servers to drain (without the JSON header the API answers 415)
</span></span></span><span class="line"><span class="cl"><span class="sd">      for server in web1 web2 web3; do
</span></span></span><span class="line"><span class="cl"><span class="sd">        curl -s -u $HAPROXY_AUTH -X PUT \
</span></span></span><span class="line"><span class="cl"><span class="sd">          &#34;$HAPROXY_API/services/haproxy/runtime/backends/app_servers/servers/${server}&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">          -H &#34;Content-Type: application/json&#34; \
</span></span></span><span class="line"><span class="cl"><span class="sd">          -d &#39;{&#34;admin_state&#34;: &#34;drain&#34;}&#39;
</span></span></span><span class="line"><span class="cl"><span class="sd">      done</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">manual</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">only</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">main</span><span class="w">
</span></span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="check-api-status">Check API Status <a class="anchor" data-pagefind-ignore href="#check-api-status" aria-label="Link to this section">#</a></h3>
<p>Is the API up, are transactions left open, and what does HAProxy report?</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Test API connectivity</span>
</span></span><span class="line"><span class="cl">curl -v -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/info
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check pending transactions</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/transactions <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># View HAProxy stats</span>
</span></span><span class="line"><span class="cl">curl -s -u <span class="nv">$AUTH</span> <span class="nv">$HAPROXY_API</span>/services/haproxy/stats/native <span class="p">|</span> jq .
</span></span></code></pre></div><h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Transaction or commit rejected over the version</td>
					<td>Another change came first. Fetch a fresh version and start again.</td>
			</tr>
			<tr>
					<td><code>401</code></td>
					<td>Wrong user or password. Through port 5556, check both <code>dataplaneapi.yaml</code> and <code>userlist api_users</code>.</td>
			</tr>
			<tr>
					<td><code>415</code></td>
					<td>The request has no <code>Content-Type: application/json</code> header.</td>
			</tr>
			<tr>
					<td><code>Error saving configuration: ... permission denied</code></td>
					<td>The API does not run as root. See the systemd unit.</td>
			</tr>
			<tr>
					<td>Change rejected as invalid</td>
					<td>Check the current state with the GET endpoints first.</td>
			</tr>
			<tr>
					<td>Reload fails</td>
					<td>Check the syntax with <code>haproxy -c -f /etc/haproxy/haproxy.cfg</code>.</td>
			</tr>
			<tr>
					<td>Runtime call on a new server says &ldquo;not found&rdquo;</td>
					<td>The reload has not happened yet. Wait <code>reload_delay</code>.</td>
			</tr>
			<tr>
					<td>A runtime weight or state is back to its old value</td>
					<td>A reload happened. Runtime changes do not survive it.</td>
			</tr>
	</tbody>
</table>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-dataplane-api" rel="noopener">pderuiter/blog-haproxy-dataplane-api</a>. It creates an Ubuntu 24.04 VM, installs HAProxy from the official packages and the Data Plane API 3.x behind the TLS frontend from this post; put your <code>mkpasswd</code> hash in the vault and run <code>ansible-playbook site.yml</code>.</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Service Discovery Integration</strong> - Let the API build backends from Consul: see the <a href="https://blog.bsdserver.nl/haproxy-and-consul-dynamic-service-discovery-for-load-balancing/">Consul post</a></li>
<li><strong>Monitoring</strong> - Track configuration changes and performance metrics</li>
<li><strong>GitOps</strong> - Version control your HAProxy configuration changes</li>
</ul>
<p>With the Data Plane API, HAProxy changes become HTTP calls that scripts and pipelines can make safely.</p>
]]></content:encoded></item><item><title>Automating HAProxy 3.3 and Data Plane API Deployment with Ansible</title><link>https://blog.bsdserver.nl/automating-haproxy-3-1-and-data-plane-api-deployment-with-ansible/</link><guid isPermaLink="false">693b511079eb54000160a47e</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Wed, 18 Nov 2026 09:00:00 +0100</pubDate><description>Introduction HAProxy is one of the most powerful open-source load balancers, widely used for its reliability, security, and performance.</description><category>HAProxy</category><category>Automation</category><category>Ansible</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/ansible-haproxy-data-plane.png" medium="image"/><content:encoded><![CDATA[<p>HAProxy is one of the most powerful open-source load balancers, widely used for its reliability, security, and performance. The Data Plane API adds a REST API to manage its configuration. Installing both by hand means a repository, a pinned package, a binary from GitHub, two config files and a systemd unit, on every load balancer.</p>
<p>This post walks through one Ansible playbook that does all of that, from the official HAProxy repositories, and gives the same result on every run.</p>
<h2 id="why-ansible-for-this">Why Ansible for this? <a class="anchor" data-pagefind-ignore href="#why-ansible-for-this" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Repeatable</strong> - Every load balancer gets the same packages, files and service</li>
<li><strong>Version control</strong> - Change <code>haproxy_version</code> or <code>dataplaneapi_version</code> and rerun to upgrade</li>
<li><strong>Idempotent</strong> - A second run reports no changes, so you can run it at any time</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> for more than one load balancer, or for one you want to rebuild quickly.</li>
<li><strong>Skip it</strong> for a single throwaway test box; the manual steps in the <a href="https://blog.bsdserver.nl/haproxy-data-plane-api-dynamic-configuration-management/">Data Plane API post</a> are quicker.</li>
</ul>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Comes from</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy</td>
					<td><code>vbernat/haproxy-X.Y</code> PPA (Ubuntu) or haproxy.debian.net (Debian)</td>
					<td>The load balancer.</td>
			</tr>
			<tr>
					<td><code>haproxy.cfg</code></td>
					<td>Template <code>haproxy.cfg.j2</code></td>
					<td>Base config with the stats socket and the API userlist.</td>
			</tr>
			<tr>
					<td>Data Plane API</td>
					<td>GitHub release binary</td>
					<td>REST API that edits <code>haproxy.cfg</code> and reloads HAProxy.</td>
			</tr>
			<tr>
					<td><code>dataplaneapi.yaml</code></td>
					<td>Template <code>dataplaneapi.yaml.j2</code></td>
					<td>API listen address, userlist and reload commands.</td>
			</tr>
			<tr>
					<td><code>dataplaneapi.service</code></td>
					<td>Inline in the playbook</td>
					<td>Starts the API at boot.</td>
			</tr>
	</tbody>
</table>
<p>The flow: add the repository → install HAProxy → write <code>haproxy.cfg</code> → install the API binary → write its config → start the service.</p>
<h2 id="prerequisites">Prerequisites <a class="anchor" data-pagefind-ignore href="#prerequisites" aria-label="Link to this section">#</a></h2>
<p>Before running this playbook, ensure that:</p>
<ul>
<li>You have an Ansible control node with SSH access to the target server.</li>
<li>The target server is running a Debian-based OS (Ubuntu/Debian).</li>
<li>You have sudo privileges on the target machine.</li>
<li>The target can reach the official HAProxy package repositories: the <code>vbernat/haproxy-X.Y</code> PPA on Ubuntu and haproxy.debian.net on Debian. The distribution&rsquo;s own <code>haproxy</code> package is usually a few branches behind.</li>
</ul>
<h3 id="choosing-an-haproxy-branch">Choosing an HAProxy branch <a class="anchor" data-pagefind-ignore href="#choosing-an-haproxy-branch" aria-label="Link to this section">#</a></h3>
<p>HAProxy has two kinds of stable branches:</p>
<table>
	<thead>
			<tr>
					<th>Branch</th>
					<th>Maintained</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>LTS (3.2, 3.4)</td>
					<td>Five years</td>
					<td>Long-lived installs</td>
			</tr>
			<tr>
					<td>Other stable (3.3)</td>
					<td>About a year</td>
					<td>The newest features, if you upgrade often</td>
			</tr>
	</tbody>
</table>
<p>This playbook uses 3.3, the newest branch the PPA builds for Ubuntu 24.04 (3.4 LTS is only built for Ubuntu 26.04 and Debian 13). For a long-lived Ubuntu 24.04 install, set <code>haproxy_version: &quot;3.2&quot;</code> to stay on an LTS branch.</p>
<h2 id="ansible-playbook-for-haproxy-and-data-plane-api-installation">Ansible Playbook for HAProxy and Data Plane API Installation <a class="anchor" data-pagefind-ignore href="#ansible-playbook-for-haproxy-and-data-plane-api-installation" aria-label="Link to this section">#</a></h2>
<p>Below is the complete Ansible playbook to install and configure HAProxy along with the Data Plane API.</p>
<p>The variables at the top:</p>
<table>
	<thead>
			<tr>
					<th>Variable</th>
					<th>Default</th>
					<th>What it sets</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>haproxy_version</code></td>
					<td><code>&quot;3.3&quot;</code></td>
					<td>The repository branch and the pinned package version</td>
			</tr>
			<tr>
					<td><code>dataplaneapi_version</code></td>
					<td><code>&quot;3.4.7&quot;</code></td>
					<td>The API release from GitHub. Changing it upgrades or downgrades.</td>
			</tr>
			<tr>
					<td><code>dataplaneapi_user</code>, <code>dataplaneapi_password</code></td>
					<td><code>admin</code>, <code>adminpwd</code></td>
					<td>The API login. Keep the password in Ansible Vault.</td>
			</tr>
			<tr>
					<td><code>api_bind_address</code>, <code>api_port</code></td>
					<td><code>0.0.0.0</code>, <code>5555</code></td>
					<td>Where the API listens</td>
			</tr>
			<tr>
					<td><code>haproxy_socket</code></td>
					<td><code>/run/haproxy/admin.sock</code></td>
					<td>HAProxy&rsquo;s runtime socket</td>
			</tr>
	</tbody>
</table>
<p>What the tasks do, in order:</p>
<table>
	<thead>
			<tr>
					<th>Tasks</th>
					<th>What they do</th>
					<th>Why this way</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Install <code>python3-debian</code></td>
					<td>Dependency</td>
					<td><code>deb822_repository</code> needs it on the target</td>
			</tr>
			<tr>
					<td>Keyring, <code>stat</code>, <code>get_url</code></td>
					<td>Download the repository signing key</td>
					<td>Only once: the keyserver does not always return identical keys, so a plain download would report a change</td>
			</tr>
			<tr>
					<td><code>deb822_repository</code> (twice)</td>
					<td>Add the PPA on Ubuntu or haproxy.debian.net on Debian</td>
					<td>The suite comes from the OS release, so one playbook fits both</td>
			</tr>
			<tr>
					<td><code>apt haproxy=X.Y.*</code></td>
					<td>Install HAProxy</td>
					<td>Pins the branch. The package cache is only refreshed when the repository was just added.</td>
			</tr>
			<tr>
					<td>Socket directory, <code>haproxy.cfg</code> template</td>
					<td>Base configuration</td>
					<td>A config change restarts HAProxy through the handler</td>
			</tr>
			<tr>
					<td><code>--version</code> check, <code>unarchive</code></td>
					<td>Install the API binary</td>
					<td>Only when the installed version differs from <code>dataplaneapi_version</code></td>
			</tr>
			<tr>
					<td><code>dataplaneapi.yaml</code> template</td>
					<td>API configuration</td>
					<td>root and <code>0600</code>, the same as the API writes it</td>
			</tr>
			<tr>
					<td>systemd unit, <code>systemd</code></td>
					<td>Start the API now and at boot</td>
					<td><code>daemon_reload</code> picks up a new or changed unit</td>
			</tr>
	</tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nn">---</span><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Install and Configure HAProxy with Data Plane API</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">hosts</span><span class="p">:</span><span class="w"> </span><span class="l">your_target_host</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">become</span><span class="p">:</span><span class="w"> </span><span class="kc">yes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">vars</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">haproxy_version</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;3.3&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">dataplaneapi_version</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;3.4.7&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">haproxy_user</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;haproxy&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">haproxy_group</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;haproxy&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">dataplaneapi_user</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;admin&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">dataplaneapi_password</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;adminpwd&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">api_bind_address</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;0.0.0.0&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">api_port</span><span class="p">:</span><span class="w"> </span><span class="m">5555</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">haproxy_socket</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;/run/haproxy/admin.sock&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">haproxy_config_file</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;/etc/haproxy/haproxy.cfg&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">dataplaneapi_config_file</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;/etc/haproxy/dataplaneapi.yaml&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">tasks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># deb822_repository needs python3-debian on the target</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Install python3-debian</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">apt</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">python3-debian</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">present</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">update_cache</span><span class="p">:</span><span class="w"> </span><span class="kc">yes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">cache_valid_time</span><span class="p">:</span><span class="w"> </span><span class="m">3600</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># Download the signing keys only once: the keyserver ignores</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># If-Modified-Since and does not always return byte-identical keys, so a</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># plain get_url would report a change on some runs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Ensure /etc/apt/keyrings exists</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">file</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/apt/keyrings</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">directory</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0755&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Check for the HAProxy repository signing key</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">stat</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;/etc/apt/keyrings/{{ &#39;haproxy-ppa.asc&#39; if ansible_facts[&#39;distribution&#39;] == &#39;Ubuntu&#39; else &#39;haproxy.debian.net.gpg&#39; }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_repo_key</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Download the HAProxy PPA signing key (Ubuntu)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">get_url</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://keyserver.ubuntu.com/pks/lookup?op=get&amp;search=0x3D653970FBAB0A890E4E4E9A0F14D8B0CF4EFE96&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/apt/keyrings/haproxy-ppa.asc</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0644&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">ansible_facts[&#39;distribution&#39;] == &#34;Ubuntu&#34; and not haproxy_repo_key.stat.exists</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Download the haproxy.debian.net signing key (Debian)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">get_url</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://haproxy.debian.net/haproxy-archive-keyring.gpg&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/apt/keyrings/haproxy.debian.net.gpg</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0644&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">ansible_facts[&#39;distribution&#39;] == &#34;Debian&#34; and not haproxy_repo_key.stat.exists</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Add HAProxy {{ haproxy_version }} Repository (Ubuntu PPA)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">deb822_repository</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy-{{ haproxy_version }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">types</span><span class="p">:</span><span class="w"> </span><span class="l">deb</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">uris</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://ppa.launchpadcontent.net/vbernat/haproxy-{{ haproxy_version }}/ubuntu&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">suites</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ ansible_facts[&#39;distribution_release&#39;] }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">components</span><span class="p">:</span><span class="w"> </span><span class="l">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">signed_by</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/apt/keyrings/haproxy-ppa.asc</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_repo_ubuntu</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">ansible_facts[&#39;distribution&#39;] == &#34;Ubuntu&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Add HAProxy {{ haproxy_version }} Repository (haproxy.debian.net)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">deb822_repository</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy-{{ haproxy_version }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">types</span><span class="p">:</span><span class="w"> </span><span class="l">deb</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">uris</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;http://haproxy.debian.net&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">suites</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ ansible_facts[&#39;distribution_release&#39;] }}-backports-{{ haproxy_version }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">components</span><span class="p">:</span><span class="w"> </span><span class="l">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">signed_by</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/apt/keyrings/haproxy.debian.net.gpg</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_repo_debian</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">ansible_facts[&#39;distribution&#39;] == &#34;Debian&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Install HAProxy {{ haproxy_version }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">apt</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy={{ haproxy_version }}.*</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">present</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">update_cache</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ haproxy_repo_ubuntu is changed or haproxy_repo_debian is changed }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Ensure HAProxy socket directory exists</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">file</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ haproxy_socket | dirname }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">directory</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ haproxy_user }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ haproxy_group }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0755&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Create HAProxy Configuration</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy.cfg.j2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ haproxy_config_file }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ haproxy_user }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ haproxy_group }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0644&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">notify</span><span class="p">:</span><span class="w"> </span><span class="l">restart haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># Compare the installed version with the wanted one, so that changing</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c"># dataplaneapi_version upgrades (or downgrades) an existing install</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Check installed Data Plane API version</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l">/usr/local/bin/dataplaneapi --version</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l">dataplaneapi_installed</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">changed_when</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">failed_when</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Download HAProxy Data Plane API {{ dataplaneapi_version }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">unarchive</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://github.com/haproxytech/dataplaneapi/releases/download/v{{ dataplaneapi_version }}/dataplaneapi_{{ dataplaneapi_version }}_linux_x86_64.tar.gz&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="l">/usr/local/bin</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">remote_src</span><span class="p">:</span><span class="w"> </span><span class="kc">yes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">include</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="l">dataplaneapi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0755&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">dataplaneapi_installed.stdout is not search(&#39;v&#39; ~ (dataplaneapi_version | regex_escape) ~ &#39; &#39;)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">notify</span><span class="p">:</span><span class="w"> </span><span class="l">restart dataplaneapi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Create Data Plane API Configuration</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="l">dataplaneapi.yaml.j2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ dataplaneapi_config_file }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c"># root and 0600, like the service: Data Plane API rewrites this</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="c"># file with these permissions every time it starts</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0600&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Create systemd service for Data Plane API</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">copy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/systemd/system/dataplaneapi.service</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">content</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          [Unit]
</span></span></span><span class="line"><span class="cl"><span class="sd">          Description=HAProxy Data Plane API
</span></span></span><span class="line"><span class="cl"><span class="sd">          After=network.target haproxy.service
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          [Service]
</span></span></span><span class="line"><span class="cl"><span class="sd">          # -f is the Data Plane API config; --config-file is haproxy.cfg.
</span></span></span><span class="line"><span class="cl"><span class="sd">          # Runs as root: it rewrites haproxy.cfg and calls systemctl reload.
</span></span></span><span class="line"><span class="cl"><span class="sd">          ExecStart=/usr/local/bin/dataplaneapi -f {{ dataplaneapi_config_file }}
</span></span></span><span class="line"><span class="cl"><span class="sd">          Restart=always
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          [Install]
</span></span></span><span class="line"><span class="cl"><span class="sd">          WantedBy=multi-user.target</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">notify</span><span class="p">:</span><span class="w"> </span><span class="l">reload systemd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Enable and start Data Plane API service</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">systemd</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">dataplaneapi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">yes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">started</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">daemon_reload</span><span class="p">:</span><span class="w"> </span><span class="kc">yes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">handlers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">restart haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">service</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">restarted</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">reload systemd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l">systemctl daemon-reload</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">restart dataplaneapi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">service</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">dataplaneapi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">restarted</span><span class="w">
</span></span></span></code></pre></div><h2 id="haproxy-configuration-template-haproxycfgj2">HAProxy Configuration Template (haproxy.cfg.j2) <a class="anchor" data-pagefind-ignore href="#haproxy-configuration-template-haproxycfgj2" aria-label="Link to this section">#</a></h2>
<p>A minimal starting point. The parts the API needs:</p>
<ul>
<li><code>stats socket ... level admin</code>: the runtime socket for changes without a reload.</li>
<li><code>userlist dataplaneapi</code>: the API login. <code>insecure-password</code> stores it in plain text in <code>haproxy.cfg</code>.</li>
<li><code>frontend http-in</code> / <code>backend servers</code>: a placeholder application to replace with your own.</li>
</ul>
<p>The template task writes this file whenever its content differs, also after changes made through the API. Decide who owns <code>haproxy.cfg</code>: the playbook or the API.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jinja" data-lang="jinja"><span class="line"><span class="cl"><span class="x">global
</span></span></span><span class="line"><span class="cl"><span class="x">    log /dev/log local0
</span></span></span><span class="line"><span class="cl"><span class="x">    log /dev/log local1 notice
</span></span></span><span class="line"><span class="cl"><span class="x">    stats socket </span><span class="cp">{{</span> <span class="nv">haproxy_socket</span> <span class="cp">}}</span><span class="x"> mode 660 level admin
</span></span></span><span class="line"><span class="cl"><span class="x">    user </span><span class="cp">{{</span> <span class="nv">haproxy_user</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">    group </span><span class="cp">{{</span> <span class="nv">haproxy_group</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">defaults
</span></span></span><span class="line"><span class="cl"><span class="x">    log     global
</span></span></span><span class="line"><span class="cl"><span class="x">    mode    http
</span></span></span><span class="line"><span class="cl"><span class="x">    option  httplog
</span></span></span><span class="line"><span class="cl"><span class="x">    option  dontlognull
</span></span></span><span class="line"><span class="cl"><span class="x">    timeout connect 5000ms
</span></span></span><span class="line"><span class="cl"><span class="x">    timeout client  50000ms
</span></span></span><span class="line"><span class="cl"><span class="x">    timeout server  50000ms
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">userlist dataplaneapi
</span></span></span><span class="line"><span class="cl"><span class="x">    user </span><span class="cp">{{</span> <span class="nv">dataplaneapi_user</span> <span class="cp">}}</span><span class="x"> insecure-password </span><span class="cp">{{</span> <span class="nv">dataplaneapi_password</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">frontend http-in
</span></span></span><span class="line"><span class="cl"><span class="x">    bind *:80
</span></span></span><span class="line"><span class="cl"><span class="x">    default_backend servers
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">backend servers
</span></span></span><span class="line"><span class="cl"><span class="x">    server server1 127.0.0.1:8000 maxconn 32
</span></span></span></code></pre></div><h2 id="data-plane-api-configuration-template-dataplaneapiyamlj2">Data Plane API Configuration Template (dataplaneapi.yaml.j2) <a class="anchor" data-pagefind-ignore href="#data-plane-api-configuration-template-dataplaneapiyamlj2" aria-label="Link to this section">#</a></h2>
<p>The Data Plane API authenticates against the <code>dataplaneapi</code> userlist in <code>haproxy.cfg</code>. It also rewrites this file in its own normalized form every time it starts, so the template is written in exactly that form; otherwise every playbook run would report it as changed.</p>
<ul>
<li><code>userlist: dataplaneapi</code>: use the userlist from <code>haproxy.cfg</code> instead of users in this file.</li>
<li><code>reload_cmd</code> / <code>restart_cmd</code> with <code>reload_strategy: custom</code>: the API reloads HAProxy through systemd.</li>
<li><code>reload_delay: 5</code>: at most one reload every 5 seconds.</li>
<li><code>log_targets</code> to stdout: the API log ends up in the journal (<code>journalctl -u dataplaneapi</code>).</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jinja" data-lang="jinja"><span class="line"><span class="cl"><span class="x">config_version: 2
</span></span></span><span class="line"><span class="cl"><span class="x">name: </span><span class="cp">{{</span> <span class="nv">inventory_hostname</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">dataplaneapi:
</span></span></span><span class="line"><span class="cl"><span class="x">  host: </span><span class="cp">{{</span> <span class="nv">api_bind_address</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">  port: </span><span class="cp">{{</span> <span class="nv">api_port</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">  advertised:
</span></span></span><span class="line"><span class="cl"><span class="x">    api_address: &#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="x">    api_port: 0
</span></span></span><span class="line"><span class="cl"><span class="x">  scheme:
</span></span></span><span class="line"><span class="cl"><span class="x">  - http
</span></span></span><span class="line"><span class="cl"><span class="x">  userlist:
</span></span></span><span class="line"><span class="cl"><span class="x">    userlist: dataplaneapi
</span></span></span><span class="line"><span class="cl"><span class="x">    userlist_file: &#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="x">  transaction:
</span></span></span><span class="line"><span class="cl"><span class="x">    transaction_dir: /tmp/haproxy
</span></span></span><span class="line"><span class="cl"><span class="x">haproxy:
</span></span></span><span class="line"><span class="cl"><span class="x">  config_file: </span><span class="cp">{{</span> <span class="nv">haproxy_config_file</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">  haproxy_bin: /usr/sbin/haproxy
</span></span></span><span class="line"><span class="cl"><span class="x">  reload:
</span></span></span><span class="line"><span class="cl"><span class="x">    reload_delay: 5
</span></span></span><span class="line"><span class="cl"><span class="x">    reload_cmd: systemctl reload haproxy
</span></span></span><span class="line"><span class="cl"><span class="x">    restart_cmd: systemctl restart haproxy
</span></span></span><span class="line"><span class="cl"><span class="x">    reload_strategy: custom
</span></span></span><span class="line"><span class="cl"><span class="x">log_targets:
</span></span></span><span class="line"><span class="cl"><span class="x">- log_to: stdout
</span></span></span><span class="line"><span class="cl"><span class="x">  log_level: info
</span></span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Store the password in Ansible Vault</strong>, not in the playbook. <code>adminpwd</code> is an example.</li>
<li><strong>Use a hashed password.</strong> <code>insecure-password</code> keeps it readable in <code>haproxy.cfg</code>; HAProxy userlists also accept a hash with <code>password</code>.</li>
<li><strong>Limit <code>api_bind_address</code>.</strong> <code>0.0.0.0</code> with plain HTTP exposes the API and its password to the network. Bind it to <code>127.0.0.1</code> or a management address, or put TLS in front as in the <a href="https://blog.bsdserver.nl/haproxy-data-plane-api-dynamic-configuration-management/">Data Plane API post</a>.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The playbook from this post, split into roles, plus Terraform code for the machine, is on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-ansible" rel="noopener">pderuiter/blog-haproxy-ansible</a>. It creates an Ubuntu 24.04 VM and installs HAProxy and the Data Plane API exactly as above; set the API password in the vault and run <code>ansible-playbook site.yml</code>.</p>
<h2 id="conclusion">Conclusion <a class="anchor" data-pagefind-ignore href="#conclusion" aria-label="Link to this section">#</a></h2>
<p>One playbook installs HAProxy and the Data Plane API the same way on every machine, and a version bump is a variable change.</p>
<p>Next: use the API to manage backends (<a href="https://blog.bsdserver.nl/haproxy-data-plane-api-dynamic-configuration-management/">Data Plane API post</a>) or to fill them from Consul (<a href="https://blog.bsdserver.nl/haproxy-and-consul-dynamic-service-discovery-for-load-balancing/">Consul post</a>).</p>
]]></content:encoded></item><item><title>HAProxy Monitoring with Prometheus: Complete Observability Guide</title><link>https://blog.bsdserver.nl/haproxy-monitoring-with-prometheus-complete-observability-guide/</link><guid isPermaLink="false">693b76b979eb54000160a637</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:58:17 +0000</pubDate><description>Monitoring HAProxy is essential for maintaining reliable load balancing infrastructure.</description><category>HAProxy</category><category>Prometheus</category><category>Monitoring</category><category>Grafana</category><category>Observability</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-monitoring-with-prometheus.png" medium="image"/><content:encoded><![CDATA[<p>Monitoring HAProxy is essential for maintaining reliable load balancing infrastructure. The stats page shows the current state, but not what happened at 3 a.m. and nothing pages you when a backend goes down.</p>
<p>This post exposes HAProxy&rsquo;s built-in Prometheus metrics, scrapes them with Prometheus, and adds alerting rules, recording rules and Grafana dashboard queries on top.</p>
<h2 id="why-prometheus-for-haproxy">Why Prometheus for HAProxy? <a class="anchor" data-pagefind-ignore href="#why-prometheus-for-haproxy" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Pull-based metrics</strong> - Prometheus scrapes HAProxy metrics endpoints</li>
<li><strong>Time-series database</strong> - Store historical data for trend analysis</li>
<li><strong>Powerful queries</strong> - PromQL for complex metric analysis</li>
<li><strong>Alerting</strong> - Define rules for proactive incident detection</li>
<li><strong>Grafana integration</strong> - Dashboards on the same data</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when you run HAProxy in production and want history, alerts and dashboards, or already run Prometheus for other services.</li>
<li><strong>Skip it</strong> for a single test instance: the built-in stats page (<code>:8404/stats</code> below) is enough.</li>
</ul>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy built-in exporter</td>
					<td>Every HAProxy node</td>
					<td>Serves all counters on <code>:8405/metrics</code>.</td>
			</tr>
			<tr>
					<td>Prometheus</td>
					<td>Monitoring server</td>
					<td>Scrapes every HAProxy every 10s, stores the series, evaluates the rules.</td>
			</tr>
			<tr>
					<td>Alertmanager</td>
					<td>Monitoring server</td>
					<td>Routes firing alerts to mail, chat or paging.</td>
			</tr>
			<tr>
					<td>Grafana</td>
					<td>Monitoring server</td>
					<td>Dashboards that query Prometheus.</td>
			</tr>
	</tbody>
</table>
<p>The flow: HAProxy counts requests → Prometheus scrapes <code>/metrics</code> → rules fire alerts to Alertmanager → Grafana graphs the same series.</p>
<h2 id="haproxy-built-in-prometheus-support">HAProxy Built-in Prometheus Support <a class="anchor" data-pagefind-ignore href="#haproxy-built-in-prometheus-support" aria-label="Link to this section">#</a></h2>
<p>HAProxy 2.0+ includes native Prometheus metrics export via the built-in Prometheus exporter. In 2.0 to 2.3 it had to be compiled in (<code>USE_PROMEX=1</code>); since 2.4 it is part of every standard build, so there is nothing extra to install.</p>
<p>The examples in this post were checked against HAProxy 3.3. Install it from the official HAProxy packages rather than the distribution&rsquo;s older <code>haproxy</code> package:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Official HAProxy packages for Ubuntu 24.04: one PPA per branch</span>
</span></span><span class="line"><span class="cl">sudo add-apt-repository -y ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">sudo apt update
</span></span><span class="line"><span class="cl">sudo apt install -y <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">haproxy -v
</span></span></code></pre></div><p>Which branch to pick:</p>
<table>
	<thead>
			<tr>
					<th>Branch</th>
					<th>Package source</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>3.3</td>
					<td><code>ppa:vbernat/haproxy-3.3</code> (Ubuntu 24.04)</td>
					<td>Newest features; short-lived, maintained until about Q1 2027</td>
			</tr>
			<tr>
					<td>3.2 LTS</td>
					<td><code>ppa:vbernat/haproxy-3.2</code> (Ubuntu 24.04)</td>
					<td>Long-lived installs, maintained until 2030</td>
			</tr>
			<tr>
					<td>3.4 LTS</td>
					<td><code>ppa:vbernat/haproxy-3.4</code> (Ubuntu 26.04), Debian 13 via <a href="https://haproxy.debian.net/" rel="noopener">haproxy.debian.net</a></td>
					<td>Newest LTS on newer releases; the site lists repository lines per release and branch</td>
			</tr>
	</tbody>
</table>
<h3 id="enable-prometheus-metrics-in-haproxy">Enable Prometheus Metrics in HAProxy <a class="anchor" data-pagefind-ignore href="#enable-prometheus-metrics-in-haproxy" aria-label="Link to this section">#</a></h3>
<p>The exporter is an HTTP service inside HAProxy. You give it its own frontend and port, so metrics stay off your application ports.</p>
<ul>
<li><code>frontend prometheus</code> on <code>:8405</code>: <code>http-request use-service prometheus-exporter</code> answers <code>/metrics</code> itself; no backend involved. <code>no log</code> keeps scrapes out of the access log.</li>
<li><code>frontend stats</code> on <code>:8404</code>: the human-readable stats page. <code>stats admin if LOCALHOST</code> allows enabling and disabling servers only from the host itself.</li>
<li><code>option httpchk GET /health</code> with <code>check</code>: health checks, which feed the server status and check metrics below.</li>
</ul>
<p>Add to <code>/etc/haproxy/haproxy.cfg</code>:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

# Prometheus metrics frontend
frontend prometheus
    bind *:8405
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
    no log
    stats enable
    stats uri /stats
    stats refresh 10s

# Stats page for human viewing
frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

# Application frontends and backends
frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check
    server web3 192.168.1.12:8080 check
</code></pre><h3 id="verify-metrics-endpoint">Verify Metrics Endpoint <a class="anchor" data-pagefind-ignore href="#verify-metrics-endpoint" aria-label="Link to this section">#</a></h3>
<p>Reload HAProxy and fetch the endpoint. Every line is one series; <code># HELP</code> and <code># TYPE</code> describe it.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Test Prometheus metrics endpoint</span>
</span></span><span class="line"><span class="cl">curl http://localhost:8405/metrics
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Sample output:</span>
</span></span><span class="line"><span class="cl"><span class="c1"># HELP haproxy_process_nbthread Number of started threads (global.nbthread)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># TYPE haproxy_process_nbthread gauge</span>
</span></span><span class="line"><span class="cl">haproxy_process_nbthread <span class="m">4</span>
</span></span><span class="line"><span class="cl"><span class="c1"># HELP haproxy_process_current_connections Current number of connections on this worker process</span>
</span></span><span class="line"><span class="cl"><span class="c1"># TYPE haproxy_process_current_connections gauge</span>
</span></span><span class="line"><span class="cl">haproxy_process_current_connections <span class="m">42</span>
</span></span></code></pre></div><h2 id="prometheus-configuration">Prometheus Configuration <a class="anchor" data-pagefind-ignore href="#prometheus-configuration" aria-label="Link to this section">#</a></h2>
<h3 id="basic-prometheus-setup">Basic Prometheus Setup <a class="anchor" data-pagefind-ignore href="#basic-prometheus-setup" aria-label="Link to this section">#</a></h3>
<p>This tells Prometheus where HAProxy is, how often to scrape it, which rule files to load and where to send alerts.</p>
<ul>
<li><code>scrape_interval: 10s</code> for the <code>haproxy</code> job: finer than the 15s global default, so short spikes show up.</li>
<li><code>scrape_timeout: 5s</code>: must stay below the interval.</li>
<li><code>rule_files</code>: the alerting and recording rules created later in this post.</li>
<li><code>alertmanagers</code>: where firing alerts go. Leave it out if you only want dashboards.</li>
</ul>
<p>Create <code>/etc/prometheus/prometheus.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">global</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">evaluation_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">alerting</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">alertmanagers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="l">alertmanager:9093</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">rule_files</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;haproxy_alerts.yml&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;haproxy_rules.yml&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;prometheus&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;localhost:9090&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="s1">&#39;haproxy1.example.com:8405&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="s1">&#39;haproxy2.example.com:8405&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">metrics_path</span><span class="p">:</span><span class="w"> </span><span class="l">/metrics</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="l">10s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">scrape_timeout</span><span class="p">:</span><span class="w"> </span><span class="l">5s</span><span class="w">
</span></span></span></code></pre></div><h3 id="service-discovery-for-multiple-haproxy-instances">Service Discovery for Multiple HAProxy Instances <a class="anchor" data-pagefind-ignore href="#service-discovery-for-multiple-haproxy-instances" aria-label="Link to this section">#</a></h3>
<p>A static target list means editing <code>prometheus.yml</code> for every new load balancer. Two alternatives:</p>
<table>
	<thead>
			<tr>
					<th>Option</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>file_sd_configs</code></td>
					<td>A JSON file written by Ansible or a script. Prometheus rereads it, no restart.</td>
			</tr>
			<tr>
					<td><code>consul_sd_configs</code></td>
					<td>HAProxy nodes registered in Consul. New nodes are scraped automatically.</td>
			</tr>
	</tbody>
</table>
<p>With <code>file_sd_configs</code>, the <code>labels</code> (<code>env</code>, <code>dc</code>) are attached to every series from those targets, so you can filter and group by them in queries.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">file_sd_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">files</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="s1">&#39;/etc/prometheus/haproxy_targets.json&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">refresh_interval</span><span class="p">:</span><span class="w"> </span><span class="l">30s</span><span class="w">
</span></span></span></code></pre></div><p>Create <code>/etc/prometheus/haproxy_targets.json</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">[</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">      <span class="s2">&#34;haproxy1.example.com:8405&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="s2">&#34;haproxy2.example.com:8405&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;labels&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="s2">&#34;production&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;dc&#34;</span><span class="p">:</span> <span class="s2">&#34;dc1&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">      <span class="s2">&#34;haproxy-staging.example.com:8405&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;labels&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="s2">&#34;staging&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;dc&#34;</span><span class="p">:</span> <span class="s2">&#34;dc1&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span></code></pre></div><h3 id="consul-service-discovery">Consul Service Discovery <a class="anchor" data-pagefind-ignore href="#consul-service-discovery" aria-label="Link to this section">#</a></h3>
<p>Prometheus asks Consul for instances of the <code>haproxy</code> service. The <code>relabel_configs</code>:</p>
<ul>
<li><code>keep</code> on <code>__meta_consul_tags</code>: only scrape instances tagged <code>prometheus</code>.</li>
<li><code>job</code> and <code>instance</code>: set from the Consul service and node name, so series read <code>instance=&quot;lb1&quot;</code> instead of an IP and port.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">consul_sd_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">server</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;consul.example.com:8500&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">relabel_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__meta_consul_tags]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">regex</span><span class="p">:</span><span class="w"> </span><span class="l">.*,prometheus,.*</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">action</span><span class="p">:</span><span class="w"> </span><span class="l">keep</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__meta_consul_service]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">target_label</span><span class="p">:</span><span class="w"> </span><span class="l">job</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__meta_consul_node]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">target_label</span><span class="p">:</span><span class="w"> </span><span class="l">instance</span><span class="w">
</span></span></span></code></pre></div><h2 id="key-haproxy-metrics">Key HAProxy Metrics <a class="anchor" data-pagefind-ignore href="#key-haproxy-metrics" aria-label="Link to this section">#</a></h2>
<p>The exporter exposes hundreds of series. They come in four levels; the <code>proxy</code> label holds the frontend or backend name, <code>server</code> the server name.</p>
<table>
	<thead>
			<tr>
					<th>Level</th>
					<th>Prefix</th>
					<th>Answers</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Process</td>
					<td><code>haproxy_process_</code></td>
					<td>Is HAProxy itself busy or short on resources?</td>
			</tr>
			<tr>
					<td>Frontend</td>
					<td><code>haproxy_frontend_</code></td>
					<td>What are clients sending and getting back?</td>
			</tr>
			<tr>
					<td>Backend</td>
					<td><code>haproxy_backend_</code></td>
					<td>Is the application pool healthy and fast?</td>
			</tr>
			<tr>
					<td>Server</td>
					<td><code>haproxy_server_</code></td>
					<td>Which single server is the problem?</td>
			</tr>
	</tbody>
</table>
<p>Counters (<code>_total</code>) only grow; wrap them in <code>rate()</code> to get per-second values. Gauges (current sessions, queue) are used as they are.</p>
<h3 id="process-metrics">Process Metrics <a class="anchor" data-pagefind-ignore href="#process-metrics" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Current connections</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_current_connections</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_process_connections_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Memory usage</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_pool_allocated_bytes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_pool_used_bytes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># CPU headroom (HAProxy exports no CPU-seconds counters; low idle = busy)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_idle_time_percent</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Uptime</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_start_time_seconds</span><span class="w">
</span></span></span></code></pre></div><h3 id="frontend-metrics">Frontend Metrics <a class="anchor" data-pagefind-ignore href="#frontend-metrics" aria-label="Link to this section">#</a></h3>
<p>The client view. Response codes are grouped in a <code>code</code> label (<code>2xx</code>, <code>4xx</code>, <code>5xx</code>). A rise in 4xx points at clients (bad URLs, failed auth); a rise in 5xx at the backends.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Request rate per frontend</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Current sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_frontend_current_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Bytes in/out</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_in_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_out_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># HTTP response codes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">2xx</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">4xx</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">5xx</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Request errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_request_errors_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Denied requests</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_requests_denied_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h3 id="backend-metrics">Backend Metrics <a class="anchor" data-pagefind-ignore href="#backend-metrics" aria-label="Link to this section">#</a></h3>
<p>The pool view. The most useful early warnings:</p>
<ul>
<li><code>current_queue</code> above 0: all servers are at <code>maxconn</code>, requests wait.</li>
<li><code>active_servers</code> dropping: health checks are taking servers out.</li>
<li><code>response_time_average_seconds</code>: an average over recent requests, not a percentile.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Backend response time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_response_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_total_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Queue length</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_current_queue</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_max_queue</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Active servers</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_active_servers</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_backend_connection_errors_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Response errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_backend_response_errors_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Retry rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_backend_retry_warnings_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_current_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_max_sessions</span><span class="w">
</span></span></span></code></pre></div><h3 id="server-metrics">Server Metrics <a class="anchor" data-pagefind-ignore href="#server-metrics" aria-label="Link to this section">#</a></h3>
<p>Use these to find the one server behind a backend problem.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Server status: one series per state, value 1 on the current state</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_status</span><span class="p">{</span><span class="nl">state</span><span class="o">=</span><span class="p">&#34;</span><span class="s">UP</span><span class="p">&#34;}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Server weight</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_weight</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Current sessions per server</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_current_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_connect_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Response time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_response_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Health check status</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_check_status</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Failed health checks</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_server_check_failures_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h2 id="alerting-rules">Alerting Rules <a class="anchor" data-pagefind-ignore href="#alerting-rules" aria-label="Link to this section">#</a></h2>
<p>Alerts turn the metrics into notifications. <code>for:</code> is how long the condition must hold before the alert fires; it filters out short blips.</p>
<table>
	<thead>
			<tr>
					<th>Alert</th>
					<th>Fires when</th>
					<th>Severity</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>HAProxyHighErrorRate</code></td>
					<td>More than 5% of responses on a frontend are 5xx</td>
					<td>critical</td>
			</tr>
			<tr>
					<td><code>HAProxyBackendDown</code></td>
					<td>A backend has no active servers</td>
					<td>critical</td>
			</tr>
			<tr>
					<td><code>HAProxyDown</code></td>
					<td>Prometheus cannot scrape an HAProxy</td>
					<td>critical</td>
			</tr>
			<tr>
					<td><code>HAProxyServerDown</code></td>
					<td>A single server is DOWN</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyHighConnections</code></td>
					<td>A frontend uses over 80% of its session limit</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyQueueNotEmpty</code></td>
					<td>More than 10 requests queued on a backend</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyHighResponseTime</code></td>
					<td>Average backend response time over 2s</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyHealthCheckFailures</code></td>
					<td>More than 5 failed checks on a server in 10 minutes</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyConnectionErrors</code></td>
					<td>More than 10 connection errors per second to a backend</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxySSLCertExpiringSoon</code></td>
					<td>A certificate expires within 7 days (needs blackbox_exporter)</td>
					<td>warning</td>
			</tr>
	</tbody>
</table>
<p>Tune the thresholds to your traffic. Create <code>/etc/prometheus/haproxy_alerts.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># High error rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHighErrorRate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total{code=&#34;5xx&#34;}[5m])) by (proxy)
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total[5m])) by (proxy)
</span></span></span><span class="line"><span class="cl"><span class="sd">          &gt; 0.05</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High 5xx error rate on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Error rate is {{ $value | humanizePercentage }} on frontend {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Backend down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyBackendDown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_backend_active_servers == 0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;HAProxy backend {{ $labels.proxy }} has no active servers&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;All servers in backend {{ $labels.proxy }} are down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Server down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyServerDown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_server_status{state=&#34;DOWN&#34;} == 1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">2m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Server {{ $labels.server }} is down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Server {{ $labels.server }} in backend {{ $labels.proxy }} is down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># High connection count</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHighConnections</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_current_sessions
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_limit_sessions
</span></span></span><span class="line"><span class="cl"><span class="sd">          &gt; 0.8</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High connection usage on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Frontend {{ $labels.proxy }} is at {{ $value | humanizePercentage }} of max connections&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Queue building up</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyQueueNotEmpty</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_backend_current_queue &gt; 10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">2m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Queue building up on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Backend {{ $labels.proxy }} has {{ $value }} requests queued&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># High response time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHighResponseTime</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_backend_response_time_average_seconds &gt; 2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High response time on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Backend {{ $labels.proxy }} average response time is {{ $value }}s&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Health check failures</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHealthCheckFailures</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">increase(haproxy_server_check_failures_total[10m]) &gt; 5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Health check failures on {{ $labels.server }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Server {{ $labels.server }} has {{ $value }} health check failures&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Connection errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyConnectionErrors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">rate(haproxy_backend_connection_errors_total[5m]) &gt; 10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Connection errors to {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Backend {{ $labels.proxy }} has {{ $value }}/s connection errors&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># HAProxy instance down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyDown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">up{job=&#34;haproxy&#34;} == 0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;HAProxy instance {{ $labels.instance }} is down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;HAProxy at {{ $labels.instance }} is not responding&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># SSL certificate expiry (HAProxy does not export certificate dates;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># this uses blackbox_exporter probing the HTTPS frontend)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxySSLCertExpiringSoon</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">probe_ssl_earliest_cert_expiry - time() &lt; 7 * 24 * 3600</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1h</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;SSL certificate expiring soon&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Certificate expires in {{ $value | humanizeDuration }}&#34;</span><span class="w">
</span></span></span></code></pre></div><h2 id="grafana-dashboards">Grafana Dashboards <a class="anchor" data-pagefind-ignore href="#grafana-dashboards" aria-label="Link to this section">#</a></h2>
<p>Grafana reads the same series from Prometheus. Add Prometheus as a data source, then build panels from the queries below.</p>
<h3 id="haproxy-overview-dashboard-json">HAProxy Overview Dashboard JSON <a class="anchor" data-pagefind-ignore href="#haproxy-overview-dashboard-json" aria-label="Link to this section">#</a></h3>
<p>A minimal skeleton with five panels: request rate, response codes, backend response time, active connections and server health.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;HAProxy Overview&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;panels&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Request Rate&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;graph&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(haproxy_frontend_http_requests_total[5m])) by (proxy)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{ proxy }}&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;HTTP Response Codes&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;graph&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(haproxy_frontend_http_responses_total[5m])) by (code)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{ code }}&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Backend Response Time&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;graph&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;haproxy_backend_response_time_average_seconds&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{ proxy }}&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Active Connections&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;gauge&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(haproxy_frontend_current_sessions)&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Server Health&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;table&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;haproxy_server_status == 1&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;format&#34;</span><span class="p">:</span> <span class="s2">&#34;table&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;instant&#34;</span><span class="p">:</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><h3 id="key-dashboard-panels">Key Dashboard Panels <a class="anchor" data-pagefind-ignore href="#key-dashboard-panels" aria-label="Link to this section">#</a></h3>
<p>More queries for panels, grouped by the question they answer.</p>
<p><strong>Traffic Overview:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Total requests per second</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Bandwidth in/out</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_in_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="w">  </span><span class="c1"># bits/sec</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_out_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="w">
</span></span></span></code></pre></div><p><strong>Error Analysis:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Error rate percentage</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">100</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=~</span><span class="p">&#34;</span><span class="s">5..</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># 4xx rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">100</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=~</span><span class="p">&#34;</span><span class="s">4..</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span></code></pre></div><p><strong>Backend Performance:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Worst observed response time (HAProxy exports averages and maxima, no histogram)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_max_response_time_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Queue time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_queue_time_average_seconds</span><span class="w">
</span></span></span></code></pre></div><p><strong>Server Status:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Server availability</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="nv">haproxy_server_status</span><span class="p">{</span><span class="nl">state</span><span class="o">=</span><span class="p">&#34;</span><span class="s">UP</span><span class="p">&#34;}</span><span class="o">)</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">proxy</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">count</span><span class="o">(</span><span class="nv">haproxy_server_status</span><span class="p">{</span><span class="nl">state</span><span class="o">=</span><span class="p">&#34;</span><span class="s">UP</span><span class="p">&#34;}</span><span class="o">)</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">proxy</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Server weight distribution</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_weight</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="k">on</span><span class="o">(</span><span class="nv">proxy</span><span class="p">,</span><span class="w"> </span><span class="nv">instance</span><span class="o">)</span><span class="w"> </span><span class="k">group_left</span><span class="w"> </span><span class="k">sum</span><span class="o">(</span><span class="nv">haproxy_server_weight</span><span class="o">)</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">proxy</span><span class="p">,</span><span class="w"> </span><span class="nv">instance</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h2 id="recording-rules">Recording Rules <a class="anchor" data-pagefind-ignore href="#recording-rules" aria-label="Link to this section">#</a></h2>
<p>A recording rule computes a query on every evaluation and stores the result as a new series. Use them for queries that dashboards and alerts run often: they load faster and are reused by name (<code>haproxy:frontend:error_rate:5m</code>).</p>
<p>Create <code>/etc/prometheus/haproxy_rules.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_recording</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Request rate by frontend</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:request_rate:5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">sum(rate(haproxy_frontend_http_requests_total[5m])) by (proxy, instance)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Error rate by frontend</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:error_rate:5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total{code=&#34;5xx&#34;}[5m])) by (proxy, instance)
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total[5m])) by (proxy, instance)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Backend availability</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:backend:availability</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(haproxy_server_status{state=&#34;UP&#34;}) by (proxy, instance)
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          count(haproxy_server_status{state=&#34;UP&#34;}) by (proxy, instance)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Session utilization</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:session_utilization</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_current_sessions
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_limit_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Bytes total rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:bytes_total_rate:5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_bytes_in_total[5m])) by (proxy, instance)
</span></span></span><span class="line"><span class="cl"><span class="sd">          +
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_bytes_out_total[5m])) by (proxy, instance)</span><span class="w">
</span></span></span></code></pre></div><h2 id="haproxy-exporter-alternative">HAProxy Exporter (Alternative) <a class="anchor" data-pagefind-ignore href="#haproxy-exporter-alternative" aria-label="Link to this section">#</a></h2>
<p>For HAProxy versions without native Prometheus support, use the separate HAProxy Exporter. It reads the stats socket and translates it to Prometheus format.</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Built-in exporter</th>
					<th>haproxy_exporter</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy version</td>
					<td>2.0+ (standard since 2.4)</td>
					<td>Any</td>
			</tr>
			<tr>
					<td>Extra process</td>
					<td>No</td>
					<td>Yes, a systemd service</td>
			</tr>
			<tr>
					<td>Port</td>
					<td>Your choice (8405 here)</td>
					<td>9101</td>
			</tr>
			<tr>
					<td>Use it for</td>
					<td>Everything current</td>
					<td>Old HAProxy you cannot upgrade</td>
			</tr>
	</tbody>
</table>
<h3 id="installation">Installation <a class="anchor" data-pagefind-ignore href="#installation" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Download HAProxy Exporter</span>
</span></span><span class="line"><span class="cl"><span class="nv">EXPORTER_VERSION</span><span class="o">=</span><span class="s2">&#34;0.15.0&#34;</span>
</span></span><span class="line"><span class="cl">wget https://github.com/prometheus/haproxy_exporter/releases/download/v<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>/haproxy_exporter-<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>.linux-amd64.tar.gz
</span></span><span class="line"><span class="cl">tar xzf haproxy_exporter-<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>.linux-amd64.tar.gz
</span></span><span class="line"><span class="cl">sudo mv haproxy_exporter-<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>.linux-amd64/haproxy_exporter /usr/local/bin/
</span></span></code></pre></div><h3 id="haproxy-stats-socket-configuration">HAProxy Stats Socket Configuration <a class="anchor" data-pagefind-ignore href="#haproxy-stats-socket-configuration" aria-label="Link to this section">#</a></h3>
<p>The exporter needs a stats socket. The service below uses the Unix socket; the TCP socket on <code>127.0.0.1:9999</code> is an alternative for an exporter that cannot reach the socket file.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats socket ipv4@127.0.0.1:9999 level admin
    stats timeout 30s
</code></pre><h3 id="exporter-systemd-service">Exporter Systemd Service <a class="anchor" data-pagefind-ignore href="#exporter-systemd-service" aria-label="Link to this section">#</a></h3>
<p>It runs as the <code>haproxy</code> user so it can open the socket (mode 660). Create <code>/etc/systemd/system/haproxy_exporter.service</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">HAProxy Exporter</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network.target haproxy.service</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">simple</span>
</span></span><span class="line"><span class="cl"><span class="na">User</span><span class="o">=</span><span class="s">haproxy</span>
</span></span><span class="line"><span class="cl"><span class="na">Group</span><span class="o">=</span><span class="s">haproxy</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/haproxy_exporter \
</span></span></span><span class="line"><span class="cl"><span class="s">  --haproxy.scrape-uri=&#34;unix:/run/haproxy/admin.sock&#34; \
</span></span></span><span class="line"><span class="cl"><span class="s">  --web.listen-address=&#34;:9101&#34;</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
</span></span><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
</span></span></code></pre></div><h3 id="prometheus-configuration-for-exporter">Prometheus Configuration for Exporter <a class="anchor" data-pagefind-ignore href="#prometheus-configuration-for-exporter" aria-label="Link to this section">#</a></h3>
<p>Point the <code>haproxy</code> job at port 9101 instead of 8405.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;haproxy1.example.com:9101&#39;</span><span class="p">]</span><span class="w">
</span></span></span></code></pre></div><h2 id="advanced-monitoring-patterns">Advanced Monitoring Patterns <a class="anchor" data-pagefind-ignore href="#advanced-monitoring-patterns" aria-label="Link to this section">#</a></h2>
<h3 id="multi-instance-aggregation">Multi-Instance Aggregation <a class="anchor" data-pagefind-ignore href="#multi-instance-aggregation" aria-label="Link to this section">#</a></h3>
<p>With several load balancers, sum across instances for the total, or group by the <code>dc</code> label from file discovery.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Total request rate across all instances</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Per-datacenter metrics</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">dc</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Highest error rate instance</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">topk</span><span class="o">(</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="nv">haproxy</span><span class="err">:</span><span class="nv">frontend</span><span class="err">:</span><span class="nv">error_rate</span><span class="err">:</span><span class="s">5m</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h3 id="slo-monitoring">SLO Monitoring <a class="anchor" data-pagefind-ignore href="#slo-monitoring" aria-label="Link to this section">#</a></h3>
<p>An SLO of 99.9% allows 0.1% of requests to fail over 30 days. The first query checks whether you meet it; the second shows how much of that 0.1% budget is left (1 = untouched, below 0 = SLO missed).</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Availability SLO (99.9%)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">5xx</span><span class="p">&#34;}[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">)</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="mf">0.999</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Error budget remaining</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">sum</span><span class="o">(</span><span class="kr">increase</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">5xx</span><span class="p">&#34;}[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="o">(</span><span class="k">sum</span><span class="o">(</span><span class="kr">increase</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mf">0.001</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h3 id="capacity-planning">Capacity Planning <a class="anchor" data-pagefind-ignore href="#capacity-planning" aria-label="Link to this section">#</a></h3>
<ul>
<li><code>predict_linear</code>: where sessions will be in an hour if the last hour&rsquo;s trend continues.</li>
<li><code>deriv</code>: how fast sessions grow per second.</li>
<li><code>max_over_time</code>: the peak, to compare against <code>maxconn</code>.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Predict session exhaustion</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">predict_linear</span><span class="o">(</span><span class="nv">haproxy_frontend_current_sessions</span><span class="p">[</span><span class="s">1h</span><span class="p">],</span><span class="w"> </span><span class="mi">3600</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection growth rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">deriv</span><span class="o">(</span><span class="nv">haproxy_frontend_current_sessions</span><span class="p">[</span><span class="s">30m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Peak hour comparison</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">max_over_time</span><span class="o">(</span><span class="nv">haproxy_frontend_current_sessions</span><span class="p">[</span><span class="s">1h</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="verify-metrics-collection">Verify Metrics Collection <a class="anchor" data-pagefind-ignore href="#verify-metrics-collection" aria-label="Link to this section">#</a></h3>
<p>Ask the Prometheus API whether the HAProxy targets are up and returning data.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check if Prometheus can reach HAProxy</span>
</span></span><span class="line"><span class="cl">curl -s http://prometheus:9090/api/v1/targets <span class="p">|</span> jq <span class="s1">&#39;.data.activeTargets[] | select(.job==&#34;haproxy&#34;)&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Query current metrics</span>
</span></span><span class="line"><span class="cl">curl -s <span class="s2">&#34;http://prometheus:9090/api/v1/query?query=haproxy_process_current_connections&#34;</span> <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check for scrape errors</span>
</span></span><span class="line"><span class="cl">curl -s http://prometheus:9090/api/v1/targets <span class="p">|</span> jq <span class="s1">&#39;.data.activeTargets[] | select(.health!=&#34;up&#34;)&#39;</span>
</span></span></code></pre></div><h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Metrics not appearing</td>
					<td>HAProxy older than 2.4 without <code>USE_PROMEX</code>; port 8405 blocked by a firewall; <code>use-service prometheus-exporter</code> missing</td>
			</tr>
			<tr>
					<td>Prometheus slow, many series</td>
					<td>High cardinality: limit label values in relabeling, use recording rules for aggregations</td>
			</tr>
			<tr>
					<td>Stale metrics</td>
					<td>Check the scrape interval; check that HAProxy still answers on <code>/metrics</code></td>
			</tr>
	</tbody>
</table>
<h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Restrict ports 8404 and 8405.</strong> The configuration binds them on all interfaces; allow only the Prometheus server in the firewall.</li>
<li><strong>Stats admin stays local.</strong> <code>stats admin if LOCALHOST</code> stops remote users from disabling servers.</li>
<li><strong>Admin sockets.</strong> <code>level admin</code> on the stats sockets allows changing HAProxy at runtime; keep the TCP socket on <code>127.0.0.1</code>.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-monitoring" rel="noopener">pderuiter/blog-haproxy-monitoring</a>. It creates an HAProxy host with the built-in exporter and a Prometheus server that scrapes it and loads the alerting and recording rules from this post; run <code>terraform apply</code> for the VMs and <code>ansible-playbook site.yml</code> for the rest.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<p>With Prometheus monitoring configured, you have:</p>
<ul>
<li>Real-time visibility into HAProxy performance</li>
<li>Historical data for trend analysis and capacity planning</li>
<li>Alerts before issues reach users</li>
<li>Grafana dashboards for operations teams</li>
</ul>
<p>This completes the HAProxy monitoring series. Together with the posts on load balancing, mTLS, the Data Plane API and Consul integration, it covers building and operating HAProxy in production.</p>
]]></content:encoded></item></channel></rss>