<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>High availability | A DevOps Blog</title><link>https://blog.bsdserver.nl/tag/high-availability/</link><description>Hands-on DevOps and homelab content: Kubernetes clusters, self-hosted services, CI/CD pipelines and infrastructure as code, by Patrick de Ruiter of We Build Your Cloud.</description><language>en</language><image><url>https://blog.bsdserver.nl/images/app-icon.svg</url><title>A DevOps Blog</title><link>https://blog.bsdserver.nl/</link></image><lastBuildDate>Wed, 18 Nov 2026 09:00:00 +0100</lastBuildDate><atom:link href="https://blog.bsdserver.nl/tag/high-availability/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title>HAProxy DNS Load Balancing: High Availability for DNS Infrastructure</title><link>https://blog.bsdserver.nl/haproxy-dns-load-balancing-high-availability-for-dns-infrastructure/</link><guid isPermaLink="false">693b727a79eb54000160a5e9</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:40:11 +0000</pubDate><description>DNS is critical infrastructure - when DNS fails, nothing works.</description><category>HAProxy</category><category>DNS</category><category>Load balancing</category><category>High availability</category><category>Networking</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-dns-loadbalancing.png" medium="image"/><content:encoded><![CDATA[<p>DNS is critical infrastructure - when DNS fails, nothing works. Clients usually have two resolver addresses at most, and a dead server behind one of them means slow lookups everywhere.</p>
<p>This guide puts a load balancer in front of resolvers and authoritative servers. HAProxy handles DNS over TCP and DNS over TLS; dnsdist handles DNS over UDP next to it, because HAProxy cannot proxy UDP. keepalived makes the load balancer itself redundant.</p>
<h2 id="why-load-balance-dns">Why Load Balance DNS? <a class="anchor" data-pagefind-ignore href="#why-load-balance-dns" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>High availability</strong>: If one DNS server fails, queries go to the healthy ones</li>
<li><strong>Performance</strong>: Spread queries over several servers</li>
<li><strong>Maintenance flexibility</strong>: Take a server offline without clients noticing</li>
<li><strong>Both protocols</strong>: UDP and TCP on one address</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when many clients depend on a few DNS servers, or when you want one stable address in front of a changing set of servers.</li>
<li><strong>Skip it</strong> when clients already list two or three resolvers and can live with a short timeout on failure. The load balancer pair is two more machines to run.</li>
</ul>
<h2 id="dns-protocol-considerations">DNS Protocol Considerations <a class="anchor" data-pagefind-ignore href="#dns-protocol-considerations" aria-label="Link to this section">#</a></h2>
<p>DNS primarily uses UDP on port 53, but can fall back to TCP for:</p>
<ul>
<li>Responses larger than 512 bytes (or EDNS0 buffer size)</li>
<li>Zone transfers (AXFR/IXFR)</li>
<li>DNS over TCP enforcement</li>
</ul>
<p>HAProxy only handles the TCP side. The community edition has no UDP proxy mode: <code>bind ... udp</code> and <code>mode udp</code> are rejected as unknown keywords, also in HAProxy 3.4. So the work is split:</p>
<table>
	<thead>
			<tr>
					<th>Traffic</th>
					<th>Handled by</th>
					<th>Why</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>UDP port 53</td>
					<td><a href="https://www.dnsdist.org/" rel="noopener">dnsdist</a></td>
					<td>A DNS-aware load balancer from PowerDNS; HAProxy has no UDP proxy</td>
			</tr>
			<tr>
					<td>TCP port 53</td>
					<td>HAProxy</td>
					<td>Large responses, zone transfers</td>
			</tr>
			<tr>
					<td>TCP port 853 (DoT)</td>
					<td>HAProxy</td>
					<td>TLS termination or passthrough</td>
			</tr>
	</tbody>
</table>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>dnsdist</td>
					<td>Both load balancers</td>
					<td>Balances UDP queries, health-checks the servers, rate limits per client.</td>
			</tr>
			<tr>
					<td>nftables</td>
					<td>Both load balancers</td>
					<td>Redirects UDP port 53 to dnsdist on port 5353, because TCP port 53 belongs to HAProxy.</td>
			</tr>
			<tr>
					<td>HAProxy</td>
					<td>Both load balancers</td>
					<td>Balances DNS over TCP and DoT.</td>
			</tr>
			<tr>
					<td>keepalived</td>
					<td>Both load balancers</td>
					<td>Moves the virtual IPs to the backup when HAProxy or dnsdist stops.</td>
			</tr>
			<tr>
					<td>DNS servers</td>
					<td>Separate machines</td>
					<td>Resolvers (<code>192.168.1.10-12</code>) and authoritative servers (<code>192.168.1.20-22</code>).</td>
			</tr>
	</tbody>
</table>
<p>The flow: client → virtual IP → UDP: nftables → dnsdist, TCP: HAProxy → healthy DNS server.</p>
<h2 id="basic-udp-dns-load-balancing">Basic UDP DNS Load Balancing <a class="anchor" data-pagefind-ignore href="#basic-udp-dns-load-balancing" aria-label="Link to this section">#</a></h2>
<p>dnsdist runs on the same machine as HAProxy, so clients use one address for both protocols.</p>
<p>Install dnsdist from the official PowerDNS repository (dnsdist 2.1 on Ubuntu 24.04):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">install -d /etc/apt/keyrings
</span></span><span class="line"><span class="cl">curl https://repo.powerdns.com/FD380FBB-pub.asc -o /etc/apt/keyrings/dnsdist-21-pub.asc
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;deb [signed-by=/etc/apt/keyrings/dnsdist-21-pub.asc] http://repo.powerdns.com/ubuntu noble-dnsdist-21 main&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  &gt; /etc/apt/sources.list.d/pdns.list
</span></span><span class="line"><span class="cl">cat &gt; /etc/apt/preferences.d/dnsdist <span class="s">&lt;&lt; EOF
</span></span></span><span class="line"><span class="cl"><span class="s">Package: dnsdist*
</span></span></span><span class="line"><span class="cl"><span class="s">Pin: origin repo.powerdns.com
</span></span></span><span class="line"><span class="cl"><span class="s">Pin-Priority: 600
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install dnsdist nftables
</span></span></code></pre></div><p>On Debian 13 use <code>http://repo.powerdns.com/debian trixie-dnsdist-21 main</code> instead.</p>
<p>dnsdist always listens on UDP and TCP together, and TCP port 53 belongs to HAProxy. So dnsdist listens on port 5353, and an nftables rule hands it the UDP traffic for port 53.</p>
<ul>
<li><code>addLocal</code>: the listen address.</li>
<li><code>newServer</code>: one line per DNS server.</li>
<li><code>setServerPolicy(roundrobin)</code>: queries in turn over all healthy servers.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- UDP port 53 is redirected here by nftables; HAProxy owns TCP port 53</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- dnsdist only answers private (RFC1918) clients by default; widen with setACL() if needed</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.10:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;dns1&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.11:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;dns2&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.12:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;dns3&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">setServerPolicy</span><span class="p">(</span><span class="n">roundrobin</span><span class="p">)</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/nftables.conf (add below the existing tables)</span>
</span></span><span class="line"><span class="cl">table ip dns_udp <span class="o">{</span>
</span></span><span class="line"><span class="cl">    chain prerouting <span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="nb">type</span> nat hook prerouting priority dstnat<span class="p">;</span> policy accept<span class="p">;</span>
</span></span><span class="line"><span class="cl">        udp dport <span class="m">53</span> redirect to :5353
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><p>Load both and start dnsdist:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">dnsdist --check-config
</span></span><span class="line"><span class="cl">systemctl <span class="nb">enable</span> --now nftables
</span></span><span class="line"><span class="cl">systemctl restart dnsdist
</span></span></code></pre></div><p><strong>Warning before you enable <code>nftables.service</code>:</strong> the stock <code>/etc/nftables.conf</code> on Debian and Ubuntu starts with <code>flush ruleset</code>. Loading it removes every other nftables rule on the host, including the iptables-nft rules that Docker, ufw and firewalld create.</p>
<ul>
<li>Run this on dedicated load balancer machines, or leave <code>flush ruleset</code> out and load the <code>dns_udp</code> table from its own file.</li>
<li>The companion repo below adds the table to <code>/etc/nftables.conf</code> in the same way, so it is meant for dedicated load balancer VMs and its README warns about this.</li>
</ul>
<p>What dnsdist does without extra configuration:</p>
<ul>
<li><strong>Health checks</strong>: every server once a second, with a query for <code>a.root-servers.net</code>. Any answer except SERVFAIL counts as up.</li>
<li><strong>Failover</strong>: failing servers leave the rotation until they answer again.</li>
<li><strong>Client addresses</strong>: kept, so per-client rate limits (below) keep working.</li>
</ul>
<h2 id="tcp-dns-load-balancing">TCP DNS Load Balancing <a class="anchor" data-pagefind-ignore href="#tcp-dns-load-balancing" aria-label="Link to this section">#</a></h2>
<p>HAProxy takes TCP port 53 in <code>mode tcp</code>: it passes the connection through without parsing DNS.</p>
<ul>
<li><code>option tcp-check</code> with <code>check</code>: a server is up when it accepts a TCP connection.</li>
<li><code>timeout client</code> / <code>timeout server 30s</code>: room for zone transfers and slow large answers.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_tcp
    bind *:53
    mode tcp
    option tcplog
    timeout client 30s
    default_backend dns_servers_tcp

backend dns_servers_tcp
    mode tcp
    balance roundrobin
    timeout server 30s
    timeout connect 5s
    option tcp-check
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
    server dns3 192.168.1.12:53 check
</code></pre><h2 id="combined-udp-and-tcp-configuration">Combined UDP and TCP Configuration <a class="anchor" data-pagefind-ignore href="#combined-udp-and-tcp-configuration" aria-label="Link to this section">#</a></h2>
<p>A production DNS load balancer should handle both protocols. dnsdist from the previous section takes UDP; HAProxy takes TCP, with health checks and statistics.</p>
<ul>
<li><code>defaults dns</code>: short timeouts (DNS answers fast) and <code>default-server inter 3s fall 3 rise 2</code>: check every 3 seconds, down after 3 failures, up after 2 successes.</li>
<li><code>listen stats</code>: stats page on <code>:8404/stats</code>, Prometheus metrics on <code>:8404/metrics</code>.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    maxconn 100000
    nbthread 4

defaults dns
    log global
    timeout connect 1s
    timeout client  5s
    timeout server  5s
    default-server inter 3s fall 3 rise 2

# UDP DNS is handled by dnsdist (see above)

# TCP DNS
frontend dns_tcp_frontend
    bind *:53
    mode tcp
    option tcplog
    default_backend dns_tcp_backend

backend dns_tcp_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
    server dns3 192.168.1.12:53 check

# Stats interface
listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 5s
    http-request use-service prometheus-exporter if { path /metrics }
</code></pre><h2 id="dns-health-checks">DNS Health Checks <a class="anchor" data-pagefind-ignore href="#dns-health-checks" aria-label="Link to this section">#</a></h2>
<p>Pick the check by how much you need to know about the server:</p>
<table>
	<thead>
			<tr>
					<th>Check</th>
					<th>Proves</th>
					<th>Cost</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>TCP connect</td>
					<td>The port accepts connections</td>
					<td>Lowest</td>
			</tr>
			<tr>
					<td>DNS query in <code>tcp-check</code></td>
					<td>The server answers a real query</td>
					<td>Hand-built packet</td>
			</tr>
			<tr>
					<td>External script</td>
					<td>Anything a script can test</td>
					<td>A process per check; needs <code>insecure-fork-wanted</code></td>
			</tr>
	</tbody>
</table>
<h3 id="basic-tcp-health-check">Basic TCP Health Check <a class="anchor" data-pagefind-ignore href="#basic-tcp-health-check" aria-label="Link to this section">#</a></h3>
<p>The simplest check verifies the TCP port is responding. It does not notice a server that accepts connections but cannot answer.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_tcp_backend
    mode tcp
    option tcp-check
    server dns1 192.168.1.10:53 check inter 2s fall 3 rise 2
</code></pre><h3 id="dns-specific-health-check">DNS-Specific Health Check <a class="anchor" data-pagefind-ignore href="#dns-specific-health-check" aria-label="Link to this section">#</a></h3>
<p>For more thorough checking, verify the server can resolve queries. The check sends a query for <code>health.local</code> and expects a response with no error.</p>
<ul>
<li>Every server needs a <code>health.local</code> A record: an NXDOMAIN answer does not match and marks the server down.</li>
<li>The 2-byte length prefix comes first because DNS over TCP requires it.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_tcp_backend
    mode tcp
    option tcp-check
    
    # DNS query for &#34;health.local&#34; A record
    # This is a raw DNS query packet (DNS over TCP needs the 2-byte length prefix)
    tcp-check connect port 53
    tcp-check send-binary 001e  # Length: 30-byte query follows
    tcp-check send-binary 0000  # Transaction ID
    tcp-check send-binary 0100  # Flags: Standard query
    tcp-check send-binary 0001  # Questions: 1
    tcp-check send-binary 0000  # Answer RRs: 0
    tcp-check send-binary 0000  # Authority RRs: 0
    tcp-check send-binary 0000  # Additional RRs: 0
    tcp-check send-binary 066865616c7468  # &#34;health&#34;
    tcp-check send-binary 056c6f63616c    # &#34;local&#34;
    tcp-check send-binary 00              # End of name
    tcp-check send-binary 0001  # Type: A
    tcp-check send-binary 0001  # Class: IN
    tcp-check expect rbinary ^[0-9A-Fa-f]{8}8[15]80  # Response flags: answer, no error (AA bit optional)
    
    server dns1 192.168.1.10:53 check
</code></pre><h3 id="external-health-check-script">External Health Check Script <a class="anchor" data-pagefind-ignore href="#external-health-check-script" aria-label="Link to this section">#</a></h3>
<p>For complex health checks, use an external script. This needs <code>external-check</code> and <code>insecure-fork-wanted</code> in the <code>global</code> section, otherwise HAProxy refuses the configuration.</p>
<p><code>insecure-fork-wanted</code> lets HAProxy start processes, which it blocks by default for safety. Only use it when the other checks are not enough.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    external-check
    insecure-fork-wanted

backend dns_tcp_backend
    mode tcp
    option external-check
    external-check path &#34;/usr/bin:/bin&#34;
    external-check command /etc/haproxy/check-dns.sh
    server dns1 192.168.1.10:53 check inter 5s
</code></pre><p>Create the health check script. HAProxy passes the server address and port; exit code 0 means up. The script resolves <code>google.com</code>, so it suits resolvers; for authoritative servers, query a zone they serve.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="cl"><span class="c1"># /etc/haproxy/check-dns.sh</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Arguments from HAProxy:</span>
</span></span><span class="line"><span class="cl"><span class="c1"># $1 = proxy address, $2 = proxy port (NOT_USED for a backend)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># $3 = server address</span>
</span></span><span class="line"><span class="cl"><span class="c1"># $4 = server port</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">SERVER</span><span class="o">=</span><span class="nv">$3</span>
</span></span><span class="line"><span class="cl"><span class="nv">PORT</span><span class="o">=</span><span class="nv">$4</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Try to resolve a known domain</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> dig @<span class="si">${</span><span class="nv">SERVER</span><span class="si">}</span> -p <span class="si">${</span><span class="nv">PORT</span><span class="si">}</span> +time<span class="o">=</span><span class="m">2</span> +tries<span class="o">=</span><span class="m">1</span> google.com A &gt; /dev/null 2&gt;<span class="p">&amp;</span>1<span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    <span class="nb">exit</span> <span class="m">0</span>  <span class="c1"># Success</span>
</span></span><span class="line"><span class="cl"><span class="k">else</span>
</span></span><span class="line"><span class="cl">    <span class="nb">exit</span> <span class="m">1</span>  <span class="c1"># Failure</span>
</span></span><span class="line"><span class="cl"><span class="k">fi</span>
</span></span></code></pre></div><p>Make it executable:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">chmod +x /etc/haproxy/check-dns.sh
</span></span></code></pre></div><h2 id="load-balancing-algorithms-for-dns">Load Balancing Algorithms for DNS <a class="anchor" data-pagefind-ignore href="#load-balancing-algorithms-for-dns" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Algorithm</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>roundrobin</code></td>
					<td>The default: equal servers, even spread</td>
			</tr>
			<tr>
					<td><code>leastconn</code></td>
					<td>TCP connections of uneven length, like zone transfers</td>
			</tr>
			<tr>
					<td><code>source</code></td>
					<td>Keeping each client on one resolver, so its cache is reused</td>
			</tr>
			<tr>
					<td><code>first</code></td>
					<td>Active-passive: everything to one server until it fails</td>
			</tr>
	</tbody>
</table>
<h3 id="round-robin">Round Robin <a class="anchor" data-pagefind-ignore href="#round-robin" aria-label="Link to this section">#</a></h3>
<p>Default algorithm, distributes queries evenly:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_servers
    balance roundrobin
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
</code></pre><h3 id="least-connections-tcp-only">Least Connections (TCP only) <a class="anchor" data-pagefind-ignore href="#least-connections-tcp-only" aria-label="Link to this section">#</a></h3>
<p>Best for DNS servers with varying query complexity:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_tcp_backend
    mode tcp
    balance leastconn
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
</code></pre><h3 id="source-ip-hash">Source IP Hash <a class="anchor" data-pagefind-ignore href="#source-ip-hash" aria-label="Link to this section">#</a></h3>
<p>Ensures clients consistently reach the same server (useful for caching). <code>hash-type consistent</code> keeps most clients on their server when one is added or removed.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_servers
    balance source
    hash-type consistent
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check
</code></pre><h3 id="first-available">First Available <a class="anchor" data-pagefind-ignore href="#first-available" aria-label="Link to this section">#</a></h3>
<p>Send all traffic to the first available server (active-passive). The <code>backup</code> server only gets traffic when the first is down.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend dns_servers
    balance first
    server dns-primary   192.168.1.10:53 check
    server dns-secondary 192.168.1.11:53 check backup
</code></pre><h2 id="recursive-vs-authoritative-dns">Recursive vs Authoritative DNS <a class="anchor" data-pagefind-ignore href="#recursive-vs-authoritative-dns" aria-label="Link to this section">#</a></h2>
<p>The two server types need different settings:</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Recursive resolvers</th>
					<th>Authoritative servers</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Clients</td>
					<td>Internal only</td>
					<td>Anyone on the internet</td>
			</tr>
			<tr>
					<td>Address</td>
					<td><code>10.0.0.1</code> / <code>10.0.0.53</code></td>
					<td><code>203.0.113.53</code></td>
			</tr>
			<tr>
					<td>Fallback</td>
					<td>Public DNS when all internal resolvers are down</td>
					<td>None: only your servers have your zones</td>
			</tr>
			<tr>
					<td>dnsdist ACL</td>
					<td>Default (private clients only)</td>
					<td>Open to all</td>
			</tr>
	</tbody>
</table>
<h3 id="load-balancing-recursive-resolvers">Load Balancing Recursive Resolvers <a class="anchor" data-pagefind-ignore href="#load-balancing-recursive-resolvers" aria-label="Link to this section">#</a></h3>
<p>For caching resolvers serving internal clients, HAProxy takes TCP. The public resolvers are <code>backup</code> servers: they only get traffic when both internal resolvers are down.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_resolver
    bind 10.0.0.1:53
    mode tcp
    default_backend resolvers

backend resolvers
    mode tcp
    balance roundrobin
    # Internal resolvers
    server resolver1 192.168.1.10:53 check weight 100
    server resolver2 192.168.1.11:53 check weight 100
    # Public fallback
    server cloudflare 1.1.1.1:53 check weight 50 backup
    server google     8.8.8.8:53 check weight 50 backup
</code></pre><p>And dnsdist takes UDP (with the nftables redirect from above), using a pool for the internal resolvers and one for the public fallback.</p>
<ul>
<li><code>pool</code>: groups servers. Rules pick a pool per query.</li>
<li><code>addAction</code> rules run in order: the first that matches decides. <code>PoolAvailableRule('internal')</code> matches while at least one internal resolver is up.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal resolvers</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.10:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver1&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.11:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver2&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="c1">-- Public fallback</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;1.1.1.1:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;cloudflare&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;8.8.8.8:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;google&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">setServerPolicy</span><span class="p">(</span><span class="n">roundrobin</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal resolvers while at least one is up, public DNS otherwise</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">PoolAvailableRule</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">AllRule</span><span class="p">(),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;fallback&#39;</span><span class="p">))</span>
</span></span></code></pre></div><h3 id="load-balancing-authoritative-servers">Load Balancing Authoritative Servers <a class="anchor" data-pagefind-ignore href="#load-balancing-authoritative-servers" aria-label="Link to this section">#</a></h3>
<p>For authoritative servers serving external queries. They listen on their own public address, separate from the internal resolvers.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_authoritative
    bind 203.0.113.53:53
    mode tcp
    default_backend auth_servers

backend auth_servers
    mode tcp
    balance roundrobin
    option tcp-check
    server auth1 192.168.1.20:53 check
    server auth2 192.168.1.21:53 check
    server auth3 192.168.1.22:53 check
</code></pre><p>The UDP side in dnsdist; an authoritative server answers anyone, so open the ACL:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">setACL</span><span class="p">({</span> <span class="s1">&#39;0.0.0.0/0&#39;</span><span class="p">,</span> <span class="s1">&#39;::/0&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.20:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth1&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.21:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth2&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.22:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth3&#39;</span> <span class="p">})</span>
</span></span></code></pre></div><h2 id="dns-over-tls-dot-load-balancing">DNS over TLS (DoT) Load Balancing <a class="anchor" data-pagefind-ignore href="#dns-over-tls-dot-load-balancing" aria-label="Link to this section">#</a></h2>
<p>For encrypted DNS traffic on port 853, pick one of two modes:</p>
<table>
	<thead>
			<tr>
					<th>Mode</th>
					<th>Where TLS ends</th>
					<th>Use it when</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Termination</td>
					<td>HAProxy, with <code>dns.example.com.pem</code></td>
					<td>The DNS servers only speak plain DNS. One certificate on the load balancer.</td>
			</tr>
			<tr>
					<td>Passthrough</td>
					<td>The DNS servers</td>
					<td>The servers already run DoT, or the load balancer should not see the queries.</td>
			</tr>
	</tbody>
</table>
<p>In passthrough mode, <code>check-ssl verify none</code> health-checks over TLS without verifying the server certificate.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># TLS Termination at HAProxy
frontend dot_frontend
    bind *:853 ssl crt /etc/haproxy/certs/dns.example.com.pem
    mode tcp
    option tcplog
    default_backend dns_tcp_backend

backend dns_tcp_backend
    mode tcp
    balance roundrobin
    server dns1 192.168.1.10:53 check
    server dns2 192.168.1.11:53 check

# TLS Passthrough (use either termination or passthrough on port 853, not both)
frontend dot_passthrough
    bind *:853
    mode tcp
    option tcplog
    default_backend dot_servers

backend dot_servers
    mode tcp
    balance roundrobin
    # check-ssl: health check over TLS, client traffic is passed through untouched
    server dot1 192.168.1.10:853 check check-ssl verify none
    server dot2 192.168.1.11:853 check check-ssl verify none
</code></pre><h2 id="rate-limiting-dns-queries">Rate Limiting DNS Queries <a class="anchor" data-pagefind-ignore href="#rate-limiting-dns-queries" aria-label="Link to this section">#</a></h2>
<p>Protect against DNS amplification attacks. Open DNS servers can be abused to send large answers to a spoofed victim address. For UDP, rate limit in dnsdist:</p>
<ul>
<li><code>MaxQPSIPRule(100)</code> with <code>DropAction()</code>: drops queries above 100 per second per client address.</li>
<li><code>TCAction()</code> for <code>ANY</code>: the client must retry over TCP, which needs a real handshake and so cannot be spoofed.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Drop clients that send more than 100 queries per second</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">MaxQPSIPRule</span><span class="p">(</span><span class="mi">100</span><span class="p">),</span> <span class="n">DropAction</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Answer ANY queries with the TC bit, so the client has to retry over TCP (HAProxy)</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">QTypeRule</span><span class="p">(</span><span class="n">DNSQType.ANY</span><span class="p">),</span> <span class="n">TCAction</span><span class="p">())</span>
</span></span></code></pre></div><p>For TCP with more options. The stick table tracks each client address:</p>
<ul>
<li><code>conn_rate(10s)</code> above 50: new connections are rejected.</li>
<li><code>bytes_out_rate(10s)</code> above 100000: the connection is rejected, to cap how much one client can pull.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_tcp
    bind *:53
    mode tcp
    option tcplog
    
    # Rate limiting table
    stick-table type ip size 100k expire 60s store conn_rate(10s),bytes_out_rate(10s)
    
    # Limit connections per IP
    tcp-request connection track-sc0 src
    tcp-request connection reject if { sc0_conn_rate gt 50 }
    
    # Limit bandwidth (prevent amplification)
    tcp-request content reject if { sc0_bytes_out_rate gt 100000 }
    
    default_backend dns_tcp_backend
</code></pre><h2 id="high-availability-haproxy-setup">High Availability HAProxy Setup <a class="anchor" data-pagefind-ignore href="#high-availability-haproxy-setup" aria-label="Link to this section">#</a></h2>
<p>For the load balancer itself, use keepalived for failover. Both nodes run HAProxy and dnsdist; the virtual IPs live on one node at a time.</p>
<p>The backup node must be able to bind the virtual IP before it owns it, so allow that first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s1">&#39;net.ipv4.ip_nonlocal_bind = 1&#39;</span> &gt; /etc/sysctl.d/90-nonlocal-bind.conf
</span></span><span class="line"><span class="cl">sysctl --system
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">apt install keepalived
</span></span></code></pre></div><p>Configure keepalived on the primary. Both virtual IPs go in <code>virtual_ipaddress</code>: the internal resolver address and the authoritative address. An address that is not listed is never brought up on the backup, so it has no failover.</p>
<ul>
<li><code>vrrp_script check_haproxy</code>: <code>killall -0</code> succeeds only while both <code>haproxy</code> and <code>dnsdist</code> run. While it succeeds, the node&rsquo;s priority goes up by <code>weight 2</code>.</li>
<li><code>priority 101</code> (primary) vs <code>100</code> (backup): with both checks passing, 103 beats 102. When the check fails on the primary, 101 loses to 102 and the IPs move.</li>
<li><code>virtual_router_id 53</code>: must match on both nodes and be unique in the network.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/keepalived/keepalived.conf (primary)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">global_defs <span class="o">{</span>
</span></span><span class="line"><span class="cl">    router_id HAPROXY_DNS_1
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_script check_haproxy <span class="o">{</span>
</span></span><span class="line"><span class="cl">    script <span class="s2">&#34;/usr/bin/killall -0 haproxy dnsdist&#34;</span>
</span></span><span class="line"><span class="cl">    interval <span class="m">2</span>
</span></span><span class="line"><span class="cl">    weight <span class="m">2</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_instance VI_DNS <span class="o">{</span>
</span></span><span class="line"><span class="cl">    state MASTER
</span></span><span class="line"><span class="cl">    interface eth0
</span></span><span class="line"><span class="cl">    virtual_router_id <span class="m">53</span>
</span></span><span class="line"><span class="cl">    priority <span class="m">101</span>
</span></span><span class="line"><span class="cl">    advert_int <span class="m">1</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    authentication <span class="o">{</span>
</span></span><span class="line"><span class="cl">        auth_type PASS
</span></span><span class="line"><span class="cl">        auth_pass secret123
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    virtual_ipaddress <span class="o">{</span>
</span></span><span class="line"><span class="cl">        10.0.0.53/24
</span></span><span class="line"><span class="cl">        <span class="c1"># the authoritative address; add &#34;dev eth1&#34; if it lives on another NIC</span>
</span></span><span class="line"><span class="cl">        203.0.113.53/24
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    track_script <span class="o">{</span>
</span></span><span class="line"><span class="cl">        check_haproxy
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><p>Configure the backup. Only <code>router_id</code>, <code>state</code> and <code>priority</code> differ:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/keepalived/keepalived.conf (backup)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">global_defs <span class="o">{</span>
</span></span><span class="line"><span class="cl">    router_id HAPROXY_DNS_2
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_script check_haproxy <span class="o">{</span>
</span></span><span class="line"><span class="cl">    script <span class="s2">&#34;/usr/bin/killall -0 haproxy dnsdist&#34;</span>
</span></span><span class="line"><span class="cl">    interval <span class="m">2</span>
</span></span><span class="line"><span class="cl">    weight <span class="m">2</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">vrrp_instance VI_DNS <span class="o">{</span>
</span></span><span class="line"><span class="cl">    state BACKUP
</span></span><span class="line"><span class="cl">    interface eth0
</span></span><span class="line"><span class="cl">    virtual_router_id <span class="m">53</span>
</span></span><span class="line"><span class="cl">    priority <span class="m">100</span>
</span></span><span class="line"><span class="cl">    advert_int <span class="m">1</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    authentication <span class="o">{</span>
</span></span><span class="line"><span class="cl">        auth_type PASS
</span></span><span class="line"><span class="cl">        auth_pass secret123
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    virtual_ipaddress <span class="o">{</span>
</span></span><span class="line"><span class="cl">        10.0.0.53/24
</span></span><span class="line"><span class="cl">        <span class="c1"># the authoritative address; add &#34;dev eth1&#34; if it lives on another NIC</span>
</span></span><span class="line"><span class="cl">        203.0.113.53/24
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl">    
</span></span><span class="line"><span class="cl">    track_script <span class="o">{</span>
</span></span><span class="line"><span class="cl">        check_haproxy
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><h2 id="complete-production-configuration">Complete Production Configuration <a class="anchor" data-pagefind-ignore href="#complete-production-configuration" aria-label="Link to this section">#</a></h2>
<p>Here&rsquo;s a complete configuration for a production DNS load balancer. It combines the sections above:</p>
<table>
	<thead>
			<tr>
					<th>Frontend</th>
					<th>Address</th>
					<th>Backend</th>
					<th>Rate limit</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>internal_dns_tcp</code></td>
					<td><code>10.0.0.53:53</code></td>
					<td>Internal resolvers, public DNS as <code>backup</code></td>
					<td>30 connections / 10s per client</td>
			</tr>
			<tr>
					<td><code>authoritative_dns_tcp</code></td>
					<td><code>203.0.113.53:53</code></td>
					<td>Authoritative servers, <code>timeout server 30s</code> for zone transfers</td>
					<td>20 connections / 10s per client</td>
			</tr>
			<tr>
					<td><code>dot_frontend</code></td>
					<td><code>*:853</code>, TLS</td>
					<td>Internal resolvers</td>
					<td>None</td>
			</tr>
	</tbody>
</table>
<p>Other settings:</p>
<ul>
<li><code>nbthread 4</code> with <code>cpu-map</code>: four threads, each pinned to one CPU.</li>
<li><code>retries 2</code>: retry a failed connection to a server twice.</li>
<li><code>stats auth admin:changeme</code>: replace this password.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0 info
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    # Performance
    maxconn 100000
    nbthread 4
    cpu-map auto:1/1-4 0-3
    
    # Security
    tune.ssl.default-dh-param 2048

defaults dns_defaults
    log global
    timeout connect 1s
    timeout client  5s
    timeout server  5s
    default-server inter 3s fall 3 rise 2
    retries 2

# Stats and monitoring
listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 5s
    stats auth admin:changeme
    http-request use-service prometheus-exporter if { path /metrics }

# ====================
# Internal Resolvers
# ====================

# UDP DNS for internal and external clients is handled by dnsdist (below)

# TCP DNS for internal clients
frontend internal_dns_tcp
    bind 10.0.0.53:53
    mode tcp
    option tcplog
    default_backend internal_resolvers_tcp
    
    # Connection limiting
    stick-table type ip size 50k expire 60s store conn_rate(10s)
    tcp-request connection track-sc0 src
    tcp-request connection reject if { sc0_conn_rate gt 30 }

backend internal_resolvers_tcp
    mode tcp
    balance leastconn
    option tcp-check
    server resolver1 192.168.1.10:53 check
    server resolver2 192.168.1.11:53 check
    server resolver3 192.168.1.12:53 check
    server cloudflare 1.1.1.1:53 check backup
    server quad9      9.9.9.9:53 check backup

# ====================
# Authoritative Servers
# ====================

# TCP DNS for zone transfers and large responses
frontend authoritative_dns_tcp
    bind 203.0.113.53:53
    mode tcp
    option tcplog
    default_backend auth_servers_tcp
    
    # Connection limiting
    stick-table type ip size 100k expire 60s store conn_rate(10s),bytes_out_rate(10s)
    tcp-request connection track-sc0 src
    tcp-request connection reject if { sc0_conn_rate gt 20 }

backend auth_servers_tcp
    mode tcp
    balance roundrobin
    option tcp-check
    timeout server 30s  # Longer timeout for zone transfers
    server auth1 192.168.1.20:53 check
    server auth2 192.168.1.21:53 check
    server auth3 192.168.1.22:53 check

# ====================
# DNS over TLS (DoT)
# ====================

frontend dot_frontend
    bind *:853 ssl crt /etc/haproxy/certs/dns.example.com.pem alpn dns
    mode tcp
    option tcplog
    timeout client 30s
    default_backend internal_resolvers_tcp
    
    # TLS client validation (optional)
    # bind *:853 ssl crt /etc/haproxy/certs/dns.example.com.pem ca-file /etc/haproxy/certs/ca.pem verify optional
</code></pre><p>And the UDP side, with one dnsdist listener per virtual IP. nftables sends each virtual IP to its own port, and the rules pick the pool by port:</p>
<ul>
<li>Port 5354 (authoritative): 50 queries per second per client, then the <code>auth</code> pool.</li>
<li>Port 5353 (internal): 100 queries per second per client, then the <code>internal</code> pool, or <code>fallback</code> when all internal resolvers are down.</li>
<li><code>controlSocket</code> and <code>setKey</code>: the console for <code>dnsdist -c</code>, local only.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-lua" data-lang="lua"><span class="line"><span class="cl"><span class="c1">-- /etc/dnsdist/dnsdist.conf</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- nftables redirects UDP 10.0.0.53:53 to 5353 and UDP 203.0.113.53:53 to 5354</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5353&#39;</span><span class="p">)</span>   <span class="c1">-- internal resolvers</span>
</span></span><span class="line"><span class="cl"><span class="n">addLocal</span><span class="p">(</span><span class="s1">&#39;0.0.0.0:5354&#39;</span><span class="p">)</span>   <span class="c1">-- authoritative servers</span>
</span></span><span class="line"><span class="cl"><span class="n">setACL</span><span class="p">({</span> <span class="s1">&#39;0.0.0.0/0&#39;</span><span class="p">,</span> <span class="s1">&#39;::/0&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal resolvers, public DNS as fallback</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.10:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver1&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.11:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver2&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.12:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;resolver3&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;internal&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;1.1.1.1:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;cloudflare&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;9.9.9.9:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;quad9&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;fallback&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Authoritative servers</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.20:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth1&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;auth&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.21:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth2&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;auth&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl"><span class="n">newServer</span><span class="p">({</span> <span class="n">address</span> <span class="o">=</span> <span class="s1">&#39;192.168.1.22:53&#39;</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;auth3&#39;</span><span class="p">,</span> <span class="n">pool</span> <span class="o">=</span> <span class="s1">&#39;auth&#39;</span> <span class="p">})</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">setServerPolicy</span><span class="p">(</span><span class="n">roundrobin</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Authoritative listener: strict rate limiting, authoritative servers only</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">AndRule</span><span class="p">({</span> <span class="n">DSTPortRule</span><span class="p">(</span><span class="mi">5354</span><span class="p">),</span> <span class="n">MaxQPSIPRule</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="p">}),</span> <span class="n">DropAction</span><span class="p">())</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">DSTPortRule</span><span class="p">(</span><span class="mi">5354</span><span class="p">),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;auth&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Internal listener: rate limiting, resolvers with public fallback</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">MaxQPSIPRule</span><span class="p">(</span><span class="mi">100</span><span class="p">),</span> <span class="n">DropAction</span><span class="p">())</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">PoolAvailableRule</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;internal&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"><span class="n">addAction</span><span class="p">(</span><span class="n">AllRule</span><span class="p">(),</span> <span class="n">PoolAction</span><span class="p">(</span><span class="s1">&#39;fallback&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">-- Console for dnsdist -c (generate a key with: head -c 32 /dev/urandom | base64)</span>
</span></span><span class="line"><span class="cl"><span class="n">controlSocket</span><span class="p">(</span><span class="s1">&#39;127.0.0.1:5199&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">setKey</span><span class="p">(</span><span class="s1">&#39;replace-with-your-base64-key&#39;</span><span class="p">)</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/nftables.conf (add below the existing tables)</span>
</span></span><span class="line"><span class="cl">table ip dns_udp <span class="o">{</span>
</span></span><span class="line"><span class="cl">    chain prerouting <span class="o">{</span>
</span></span><span class="line"><span class="cl">        <span class="nb">type</span> nat hook prerouting priority dstnat<span class="p">;</span> policy accept<span class="p">;</span>
</span></span><span class="line"><span class="cl">        ip daddr 10.0.0.53 udp dport <span class="m">53</span> redirect to :5353
</span></span><span class="line"><span class="cl">        ip daddr 203.0.113.53 udp dport <span class="m">53</span> redirect to :5354
</span></span><span class="line"><span class="cl">    <span class="o">}</span>
</span></span><span class="line"><span class="cl"><span class="o">}</span>
</span></span></code></pre></div><h2 id="monitoring-and-logging">Monitoring and Logging <a class="anchor" data-pagefind-ignore href="#monitoring-and-logging" aria-label="Link to this section">#</a></h2>
<h3 id="dns-query-logging">DNS Query Logging <a class="anchor" data-pagefind-ignore href="#dns-query-logging" aria-label="Link to this section">#</a></h3>
<p><code>option tcplog</code> logs one line per connection, not per query. The custom <code>log-format</code> adds the client, the chosen server, the timers (<code>%Tw/%Tc/%Tt</code>), bytes sent and the connection counts.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend dns_tcp
    bind *:53
    mode tcp
    option tcplog
    log-format &#34;%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc&#34;
    default_backend dns_servers
</code></pre><h3 id="prometheus-metrics">Prometheus Metrics <a class="anchor" data-pagefind-ignore href="#prometheus-metrics" aria-label="Link to this section">#</a></h3>
<p>Enable the built-in Prometheus exporter:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">listen stats
    bind *:8404
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
</code></pre><p>Key metrics to monitor:</p>
<ul>
<li><code>haproxy_server_status</code>: Backend server health</li>
<li><code>haproxy_frontend_current_sessions</code>: Active connections</li>
<li><code>haproxy_frontend_bytes_in_total</code>: Incoming traffic</li>
<li><code>haproxy_frontend_bytes_out_total</code>: Outgoing traffic (watch for amplification)</li>
</ul>
<p>These cover HAProxy, so TCP and DoT only. dnsdist handles most queries (UDP); check it with <code>showServers()</code> (below).</p>
<h2 id="testing-your-configuration">Testing Your Configuration <a class="anchor" data-pagefind-ignore href="#testing-your-configuration" aria-label="Link to this section">#</a></h2>
<h3 id="validate-configuration">Validate Configuration <a class="anchor" data-pagefind-ignore href="#validate-configuration" aria-label="Link to this section">#</a></h3>
<p>Check the syntax before every reload.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">haproxy -c -f /etc/haproxy/haproxy.cfg
</span></span></code></pre></div><h3 id="test-dns-resolution">Test DNS Resolution <a class="anchor" data-pagefind-ignore href="#test-dns-resolution" aria-label="Link to this section">#</a></h3>
<p>Query the virtual IP over both protocols, then each server directly. If a direct query works but the virtual IP does not, the problem is in the load balancer.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># UDP query (dnsdist)</span>
</span></span><span class="line"><span class="cl">dig @10.0.0.53 google.com A
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># TCP query (HAProxy)</span>
</span></span><span class="line"><span class="cl">dig @10.0.0.53 +tcp google.com A
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test all servers</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> server in 192.168.1.10 192.168.1.11 192.168.1.12<span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">    <span class="nb">echo</span> <span class="s2">&#34;Testing </span><span class="nv">$server</span><span class="s2">:&#34;</span>
</span></span><span class="line"><span class="cl">    dig @<span class="nv">$server</span> +short google.com A
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span></code></pre></div><h3 id="load-testing">Load Testing <a class="anchor" data-pagefind-ignore href="#load-testing" aria-label="Link to this section">#</a></h3>
<p><code>dnsperf</code> sends queries from a file: <code>-c 100</code> clients, at most <code>-Q 10000</code> queries per second. Run it once per protocol. Keep the rate limits in mind: from one test machine you hit them quickly.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Install dnsperf</span>
</span></span><span class="line"><span class="cl">apt install dnsperf
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Create query file</span>
</span></span><span class="line"><span class="cl">cat &gt; queries.txt <span class="s">&lt;&lt; EOF
</span></span></span><span class="line"><span class="cl"><span class="s">google.com A
</span></span></span><span class="line"><span class="cl"><span class="s">example.com A
</span></span></span><span class="line"><span class="cl"><span class="s">github.com A
</span></span></span><span class="line"><span class="cl"><span class="s">EOF</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Run load test (UDP, through dnsdist)</span>
</span></span><span class="line"><span class="cl">dnsperf -s 10.0.0.53 -d queries.txt -c <span class="m">100</span> -Q <span class="m">10000</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># The same over TCP, through HAProxy</span>
</span></span><span class="line"><span class="cl">dnsperf -m tcp -s 10.0.0.53 -d queries.txt -c <span class="m">100</span> -Q <span class="m">10000</span>
</span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Check</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>UDP not working</td>
					<td>UDP goes through dnsdist, not HAProxy. Check that dnsdist listens (<code>ss -ulpn | grep 5353</code>), that the nftables redirect is loaded (<code>nft list table ip dns_udp</code>), that <code>setACL()</code> allows your clients, and that <code>showServers()</code> lists the backends as up.</td>
			</tr>
			<tr>
					<td>Health checks failing</td>
					<td>Verify DNS servers are responding, check firewall rules. With the DNS query check: does <code>health.local</code> exist on every server?</td>
			</tr>
			<tr>
					<td>High latency</td>
					<td>Check <code>timeout</code> settings and server response times.</td>
			</tr>
			<tr>
					<td>Connection refused</td>
					<td>Verify HAProxy is binding correctly (<code>ss -tulpn | grep 53</code>).</td>
			</tr>
			<tr>
					<td>Other firewall rules gone</td>
					<td><code>flush ruleset</code> in <code>/etc/nftables.conf</code> (see the warning above).</td>
			</tr>
	</tbody>
</table>
<h3 id="debug-commands">Debug Commands <a class="anchor" data-pagefind-ignore href="#debug-commands" aria-label="Link to this section">#</a></h3>
<p>The admin socket shows what HAProxy sees; <code>dnsdist -c</code> does the same for the UDP side.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check HAProxy status</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show info&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check backend servers</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show servers state&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check statistics</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show stat&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Watch connections in real-time</span>
</span></span><span class="line"><span class="cl">watch -n <span class="m">1</span> <span class="s1">&#39;echo &#34;show stat&#34; | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,5,18&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># dnsdist (UDP) backends and their state</span>
</span></span><span class="line"><span class="cl">dnsdist -c -e <span class="s1">&#39;showServers()&#39;</span>
</span></span></code></pre></div><h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Change the example secrets</strong>: <code>auth_pass secret123</code> in keepalived, <code>stats auth admin:changeme</code> in HAProxy, and the dnsdist console key. VRRP <code>PASS</code> authentication is sent in plain text; it only guards against mistakes.</li>
<li><strong>Keep resolvers internal.</strong> The production dnsdist config sets <code>setACL({ '0.0.0.0/0', '::/0' })</code> for the authoritative side, and that ACL also covers the internal listener. Make sure only internal clients can reach <code>10.0.0.53</code> and port 5353, or you run an open resolver.</li>
<li><strong>Keep the rate limits.</strong> They are what stops the load balancer from being used for amplification attacks.</li>
<li><strong>Avoid external checks where you can.</strong> <code>insecure-fork-wanted</code> lifts a safety guard in HAProxy.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-dns-lb" rel="noopener">pderuiter/blog-haproxy-dns-lb</a>. It creates two load balancer VMs on Proxmox and configures HAProxy for TCP and DoT, dnsdist with the nftables redirect for UDP, and keepalived for the virtual IPs, as in the complete production configuration from this post; run <code>terraform apply</code>, then <code>ansible-playbook site.yml</code>.</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>Next in the series: <a href="https://blog.bsdserver.nl/haproxy-with-lets-encrypt-automated-https-certificate-management/">HAProxy with Let&rsquo;s Encrypt</a> for automatic HTTPS certificate management.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<ul>
<li>HAProxy cannot proxy UDP (also not in 3.4): use dnsdist for DNS over UDP</li>
<li>Cover both UDP (dnsdist) and TCP (HAProxy) on the same address for complete DNS coverage</li>
<li>Implement proper health checks for DNS servers</li>
<li>Use rate limiting to protect against abuse</li>
<li>Deploy keepalived for HAProxy and dnsdist high availability</li>
</ul>
]]></content:encoded></item><item><title>HAProxy TCP Load Balancing: A Complete Guide to Layer 4 Load Balancing</title><link>https://blog.bsdserver.nl/haproxy-tcp-load-balancing-a-complete-guide-to-layer-4-load-balancing/</link><guid isPermaLink="false">693b721e79eb54000160a5dc</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:38:38 +0000</pubDate><description>HAProxy is one of the most powerful and reliable load balancers available.</description><category>HAProxy</category><category>Load balancing</category><category>TCP</category><category>High availability</category><category>Networking</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-tcp-loadbalancing.png" medium="image"/><content:encoded><![CDATA[<p>HAProxy is one of the most powerful and reliable load balancers available. It is best known for HTTP, but databases, Redis, message queues and TLS services speak their own protocols over TCP, and they need load balancing and failover too.</p>
<p>This guide covers HAProxy in TCP mode: algorithms, protocol-aware health checks, connection limits, failover patterns, TLS passthrough, client IP preservation and logging, ending with a complete PostgreSQL and Redis configuration.</p>
<h2 id="why-tcp-load-balancing">Why TCP Load Balancing? <a class="anchor" data-pagefind-ignore href="#why-tcp-load-balancing" aria-label="Link to this section">#</a></h2>
<p>Layer 4 load balancing works at the transport layer: it routes on IP addresses and ports and forwards the bytes without reading them.</p>
<ul>
<li><strong>Protocol agnostic</strong>: Works with any TCP-based protocol (databases, message queues, custom protocols)</li>
<li><strong>Lower latency</strong>: No content inspection means faster routing decisions</li>
<li><strong>TLS passthrough</strong>: End-to-end encryption without termination at the load balancer</li>
<li><strong>Simplicity</strong>: Fewer moving parts compared to Layer 7 configuration</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> for non-HTTP protocols, and for HTTPS when the backends must terminate TLS themselves.</li>
<li><strong>Skip it</strong> for plain HTTP(S) services where you want path routing, header changes or HTTP-level logs: use <code>mode http</code>.</li>
</ul>
<table>
	<thead>
			<tr>
					<th></th>
					<th><code>mode tcp</code> (Layer 4)</th>
					<th><code>mode http</code> (Layer 7)</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Routes on</td>
					<td>IP, port, TLS SNI</td>
					<td>Host, path, headers, cookies</td>
			</tr>
			<tr>
					<td>Sees content</td>
					<td>No</td>
					<td>Yes</td>
			</tr>
			<tr>
					<td>Protocols</td>
					<td>Any TCP</td>
					<td>HTTP/1.x, HTTP/2, HTTP/3</td>
			</tr>
			<tr>
					<td>TLS</td>
					<td>Passthrough or termination</td>
					<td>Termination</td>
			</tr>
	</tbody>
</table>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Frontend</td>
					<td>HAProxy</td>
					<td>Listens on a port (<code>bind *:5432</code>) and picks a backend.</td>
			</tr>
			<tr>
					<td>Backend</td>
					<td>HAProxy</td>
					<td>Holds the server list, the algorithm and the health check.</td>
			</tr>
			<tr>
					<td>Servers</td>
					<td>Database, Redis or app hosts</td>
					<td>Receive the forwarded TCP connections.</td>
			</tr>
			<tr>
					<td>Stats socket</td>
					<td>HAProxy</td>
					<td>Runtime control: drain, disable, show state.</td>
			</tr>
	</tbody>
</table>
<p>The flow: client connects to HAProxy → frontend picks a backend → algorithm picks a healthy server → HAProxy opens a connection to it and copies bytes both ways until one side closes.</p>
<h2 id="installation">Installation <a class="anchor" data-pagefind-ignore href="#installation" aria-label="Link to this section">#</a></h2>
<p>On Debian/Ubuntu, use the official HAProxy packages from <a href="https://haproxy.debian.net" rel="noopener">haproxy.debian.net</a> instead of the distribution package, which lags several branches behind:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Ubuntu 24.04: official PPA. 3.3 is the newest branch built for 24.04; it is a</span>
</span></span><span class="line"><span class="cl"><span class="c1"># short-lived, non-LTS branch. HAProxy 3.4 LTS is available for Ubuntu 26.04</span>
</span></span><span class="line"><span class="cl"><span class="c1"># (ppa:vbernat/haproxy-3.4) and for Debian 13 (below).</span>
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install --no-install-recommends software-properties-common
</span></span><span class="line"><span class="cl">add-apt-repository ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">apt install <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Debian 13: official haproxy.debian.net repository (3.4 LTS)</span>
</span></span><span class="line"><span class="cl">curl https://haproxy.debian.net/haproxy-archive-keyring.gpg <span class="se">\
</span></span></span><span class="line"><span class="cl">  --create-dirs --output /etc/apt/keyrings/haproxy-archive-keyring.gpg
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> deb <span class="s2">&#34;[signed-by=/etc/apt/keyrings/haproxy-archive-keyring.gpg]&#34;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl">  https://haproxy.debian.net trixie-backports-3.4 main <span class="se">\
</span></span></span><span class="line"><span class="cl">  &gt; /etc/apt/sources.list.d/haproxy.list
</span></span><span class="line"><span class="cl">apt update
</span></span><span class="line"><span class="cl">apt install <span class="nv">haproxy</span><span class="o">=</span>3.4.<span class="se">\*</span>
</span></span></code></pre></div><p>On RHEL/CentOS there is no official HAProxy package repository, so you get the distribution version:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">yum install haproxy
</span></span><span class="line"><span class="cl"><span class="c1"># RHEL/Rocky/Alma 9 AppStream ships the 2.8 LTS branch; check what you got</span>
</span></span><span class="line"><span class="cl">haproxy -v
</span></span></code></pre></div><p>The RHEL unit file does not create <code>/run/haproxy</code>, so the <code>stats socket /run/haproxy/admin.sock</code> line used throughout this post stops HAProxy from starting. Let systemd create the directory:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">mkdir -p /etc/systemd/system/haproxy.service.d
</span></span><span class="line"><span class="cl"><span class="nb">printf</span> <span class="s1">&#39;[Service]\nRuntimeDirectory=haproxy\n&#39;</span> &gt; /etc/systemd/system/haproxy.service.d/runtime-directory.conf
</span></span><span class="line"><span class="cl">systemctl daemon-reload
</span></span></code></pre></div><h2 id="basic-tcp-load-balancer-configuration">Basic TCP Load Balancer Configuration <a class="anchor" data-pagefind-ignore href="#basic-tcp-load-balancer-configuration" aria-label="Link to this section">#</a></h2>
<p>A minimal load balancer for three MySQL servers. What matters for TCP:</p>
<ul>
<li><code>mode tcp</code> and <code>option tcplog</code> in <code>defaults</code>: every proxy works at Layer 4 and logs connections, not requests.</li>
<li><code>timeout client</code> / <code>timeout server</code>: idle time before HAProxy closes the connection. Too low cuts off idle database sessions.</li>
<li><code>retries 3</code>: connection attempts to a server before giving up.</li>
<li><code>option mysql-check</code>: a health check that logs in to MySQL instead of only opening the port.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    # Performance tuning
    maxconn 50000
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    timeout connect 10s
    timeout client  30s
    timeout server  30s
    retries 3

frontend mysql_frontend
    bind *:3306
    default_backend mysql_servers

backend mysql_servers
    balance roundrobin
    option mysql-check user haproxy
    server mysql1 192.168.1.10:3306 check
    server mysql2 192.168.1.11:3306 check
    server mysql3 192.168.1.12:3306 check
</code></pre><h2 id="load-balancing-algorithms">Load Balancing Algorithms <a class="anchor" data-pagefind-ignore href="#load-balancing-algorithms" aria-label="Link to this section">#</a></h2>
<p>The algorithm decides which server gets the next connection.</p>
<table>
	<thead>
			<tr>
					<th>Algorithm</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>roundrobin</code></td>
					<td>Short connections of similar cost; supports weights</td>
			</tr>
			<tr>
					<td><code>leastconn</code></td>
					<td>Long-lived connections such as database sessions</td>
			</tr>
			<tr>
					<td><code>source</code></td>
					<td>Clients that must stay on the same server</td>
			</tr>
			<tr>
					<td><code>first</code></td>
					<td>Filling servers one by one, for example so spare servers can sleep</td>
			</tr>
	</tbody>
</table>
<h3 id="round-robin">Round Robin <a class="anchor" data-pagefind-ignore href="#round-robin" aria-label="Link to this section">#</a></h3>
<p>Distributes connections in turn, in proportion to <code>weight</code>:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance roundrobin
    server s1 192.168.1.10:5432 check weight 1
    server s2 192.168.1.11:5432 check weight 2  # Gets 2x traffic
    server s3 192.168.1.12:5432 check weight 1
</code></pre><h3 id="least-connections">Least Connections <a class="anchor" data-pagefind-ignore href="#least-connections" aria-label="Link to this section">#</a></h3>
<p>Sends traffic to the server with fewest active connections:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance leastconn
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check
    server s3 192.168.1.12:5432 check
</code></pre><h3 id="source-ip-hash">Source IP Hash <a class="anchor" data-pagefind-ignore href="#source-ip-hash" aria-label="Link to this section">#</a></h3>
<p>Ensures the same client always connects to the same server (session persistence). <code>hash-type consistent</code> moves only a few clients when a server is added or removed.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance source
    hash-type consistent
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check
    server s3 192.168.1.12:5432 check
</code></pre><h3 id="first-available">First Available <a class="anchor" data-pagefind-ignore href="#first-available" aria-label="Link to this section">#</a></h3>
<p>Uses the first server with available connection slots. Requires <code>maxconn</code> on the servers: the next server only gets traffic when the first is full.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    balance first
    server s1 192.168.1.10:5432 check maxconn 100
    server s2 192.168.1.11:5432 check maxconn 100
</code></pre><h2 id="health-checks">Health Checks <a class="anchor" data-pagefind-ignore href="#health-checks" aria-label="Link to this section">#</a></h2>
<p>HAProxy only sends traffic to servers that pass their check. An open port is not a working service, so use a protocol check where one exists.</p>
<table>
	<thead>
			<tr>
					<th>Check</th>
					<th>Proves</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>option tcp-check</code> (no steps)</td>
					<td>The port accepts connections</td>
			</tr>
			<tr>
					<td><code>mysql-check</code>, <code>pgsql-check</code></td>
					<td>The database answers its handshake</td>
			</tr>
			<tr>
					<td><code>tcp-check send</code> / <code>expect</code></td>
					<td>The service gives the right answer to a request you define</td>
			</tr>
	</tbody>
</table>
<h3 id="basic-tcp-check">Basic TCP Check <a class="anchor" data-pagefind-ignore href="#basic-tcp-check" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    option tcp-check
    server s1 192.168.1.10:5432 check
</code></pre><h3 id="protocol-specific-health-checks">Protocol-Specific Health Checks <a class="anchor" data-pagefind-ignore href="#protocol-specific-health-checks" aria-label="Link to this section">#</a></h3>
<h4 id="mysqlmariadb">MySQL/MariaDB <a class="anchor" data-pagefind-ignore href="#mysqlmariadb" aria-label="Link to this section">#</a></h4>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend mysql_servers
    option mysql-check user haproxy
    server mysql1 192.168.1.10:3306 check
</code></pre><p>The check logs in as <code>haproxy</code> without a password. Create the HAProxy user in MySQL:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">USER</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="o">@</span><span class="s1">&#39;%&#39;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="n">FLUSH</span><span class="w"> </span><span class="k">PRIVILEGES</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><h4 id="postgresql">PostgreSQL <a class="anchor" data-pagefind-ignore href="#postgresql" aria-label="Link to this section">#</a></h4>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend postgres_servers
    option pgsql-check user haproxy
    server pg1 192.168.1.10:5432 check
</code></pre><h4 id="redis">Redis <a class="anchor" data-pagefind-ignore href="#redis" aria-label="Link to this section">#</a></h4>
<p>Sends <code>PING</code> and expects <code>+PONG</code>.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend redis_servers
    option tcp-check
    tcp-check connect
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    server redis1 192.168.1.10:6379 check
</code></pre><h4 id="custom-tcp-check">Custom TCP Check <a class="anchor" data-pagefind-ignore href="#custom-tcp-check" aria-label="Link to this section">#</a></h4>
<p>Sends a raw HTTP request and expects a <code>200</code> status line. <code>tcp-check connect port</code> lets the check use another port than the traffic, for a health endpoint on a separate port.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend custom_servers
    option tcp-check
    tcp-check connect port 8080
    tcp-check send &#34;GET /health HTTP/1.1\r\nHost: localhost\r\n\r\n&#34;
    tcp-check expect rstring ^HTTP/1\.[01]\ 200
    server app1 192.168.1.10:8080 check
</code></pre><h3 id="health-check-parameters">Health Check Parameters <a class="anchor" data-pagefind-ignore href="#health-check-parameters" aria-label="Link to this section">#</a></h3>
<p><code>default-server</code> sets the values for all servers in the backend; a value on a <code>server</code> line overrides it.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    default-server inter 3s fall 3 rise 2
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check inter 5s fall 2 rise 3
</code></pre><ul>
<li><code>inter</code>: Interval between checks (default: 2s)</li>
<li><code>fall</code>: Number of failed checks before marking server down</li>
<li><code>rise</code>: Number of successful checks before marking server up</li>
</ul>
<h2 id="connection-management">Connection Management <a class="anchor" data-pagefind-ignore href="#connection-management" aria-label="Link to this section">#</a></h2>
<p>Limits protect the backends: HAProxy queues or rejects connections instead of overloading a server.</p>
<h3 id="maximum-connections">Maximum Connections <a class="anchor" data-pagefind-ignore href="#maximum-connections" aria-label="Link to this section">#</a></h3>
<p><code>maxconn</code> works at three levels:</p>
<ul>
<li><code>global</code>: the whole process. Above it, HAProxy stops accepting.</li>
<li><code>frontend</code>: per listener.</li>
<li><code>server</code>: per backend server. Extra connections wait in the backend queue.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    maxconn 50000

frontend db_frontend
    bind *:5432
    maxconn 10000
    default_backend db_servers

backend db_servers
    server db1 192.168.1.10:5432 check maxconn 500
    server db2 192.168.1.11:5432 check maxconn 500
</code></pre><h3 id="connection-queuing">Connection Queuing <a class="anchor" data-pagefind-ignore href="#connection-queuing" aria-label="Link to this section">#</a></h3>
<p><code>maxqueue 50</code> caps the queue for a server; beyond that, connections go to another server or fail instead of waiting.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    fullconn 1000
    server s1 192.168.1.10:5432 check maxconn 100 maxqueue 50
</code></pre><h3 id="connection-timeouts">Connection Timeouts <a class="anchor" data-pagefind-ignore href="#connection-timeouts" aria-label="Link to this section">#</a></h3>
<p>Each timeout covers one phase of a connection; the comments say which.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">defaults
    timeout connect 5s      # Time to establish connection to server
    timeout client  60s     # Inactivity timeout on client side
    timeout server  60s     # Inactivity timeout on server side
    timeout tunnel  1h      # Timeout for tunneled connections (WebSocket, etc.)
    timeout client-fin 30s  # Timeout for client FIN
    timeout server-fin 30s  # Timeout for server FIN
</code></pre><p>Database clients and pools keep connections open and idle. Raise the client and server timeouts, or HAProxy closes them:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">defaults
    timeout client  3600s
    timeout server  3600s
</code></pre><h2 id="high-availability-patterns">High Availability Patterns <a class="anchor" data-pagefind-ignore href="#high-availability-patterns" aria-label="Link to this section">#</a></h2>
<h3 id="active-passive-with-backup-servers">Active-Passive with Backup Servers <a class="anchor" data-pagefind-ignore href="#active-passive-with-backup-servers" aria-label="Link to this section">#</a></h3>
<p>Servers marked <code>backup</code> only get traffic when all normal servers are down.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend db_servers
    balance roundrobin
    server primary1 192.168.1.10:5432 check
    server primary2 192.168.1.11:5432 check
    server backup1  192.168.1.20:5432 check backup
    server backup2  192.168.1.21:5432 check backup
</code></pre><h3 id="readwrite-splitting">Read/Write Splitting <a class="anchor" data-pagefind-ignore href="#readwrite-splitting" aria-label="Link to this section">#</a></h3>
<p>For database clusters with a primary and replicas. One frontend listens on two ports; <code>dst_port</code> sends writes to the primary and reads to the replicas. The application chooses the port.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend mysql_frontend
    bind *:3306
    bind *:3307
    
    # Port 3306 -&gt; Write (Primary)
    # Port 3307 -&gt; Read (Replicas)
    use_backend mysql_write if { dst_port 3306 }
    use_backend mysql_read if { dst_port 3307 }

backend mysql_write
    option mysql-check user haproxy
    server primary 192.168.1.10:3306 check

backend mysql_read
    balance leastconn
    option mysql-check user haproxy
    server replica1 192.168.1.11:3306 check
    server replica2 192.168.1.12:3306 check
    server primary  192.168.1.10:3306 check backup  # Fallback to primary
</code></pre><h3 id="graceful-server-maintenance">Graceful Server Maintenance <a class="anchor" data-pagefind-ignore href="#graceful-server-maintenance" aria-label="Link to this section">#</a></h3>
<p>Take a server out without cutting active connections. <code>weight 0</code> in the configuration stops new connections; the stats socket does the same at runtime without a reload.</p>
<table>
	<thead>
			<tr>
					<th>Command</th>
					<th>Effect</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>disable server</code></td>
					<td>Maintenance mode: no new connections, existing ones continue</td>
			</tr>
			<tr>
					<td><code>set server ... state drain</code></td>
					<td>No new load-balanced connections, health checks keep running</td>
			</tr>
			<tr>
					<td><code>enable server</code></td>
					<td>Back in rotation</td>
			</tr>
	</tbody>
</table>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">backend servers
    server s1 192.168.1.10:5432 check
    server s2 192.168.1.11:5432 check weight 0  # No new connections (drain)
</code></pre><p>Use the stats socket to manage servers dynamically:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Disable a server (existing connections preserved)</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;disable server servers/s1&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Set server to drain mode</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;set server servers/s1 state drain&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Re-enable a server</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;enable server servers/s1&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check server status</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show servers state&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span></code></pre></div><h2 id="tls-passthrough">TLS Passthrough <a class="anchor" data-pagefind-ignore href="#tls-passthrough" aria-label="Link to this section">#</a></h2>
<p>For end-to-end encryption without terminating TLS at the load balancer. HAProxy forwards the encrypted stream; certificates live on the backends.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend tls_frontend
    bind *:443
    mode tcp
    option tcplog
    default_backend tls_servers

backend tls_servers
    mode tcp
    balance roundrobin
    server s1 192.168.1.10:443 check
    server s2 192.168.1.11:443 check
</code></pre><h3 id="sni-based-routing">SNI-Based Routing <a class="anchor" data-pagefind-ignore href="#sni-based-routing" aria-label="Link to this section">#</a></h3>
<p>Route to different backends based on the TLS SNI header, the hostname the client sends unencrypted in its TLS hello.</p>
<ul>
<li><code>tcp-request inspect-delay 5s</code>: wait up to 5 seconds for the hello before deciding.</li>
<li><code>req_ssl_hello_type 1</code>: accept once a ClientHello arrived.</li>
<li><code>req_ssl_sni</code>: the requested hostname.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend tls_frontend
    bind *:443
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    
    use_backend app1_servers if { req_ssl_sni -i app1.example.com }
    use_backend app2_servers if { req_ssl_sni -i app2.example.com }
    default_backend default_servers

backend app1_servers
    mode tcp
    server app1 192.168.1.10:443 check

backend app2_servers
    mode tcp
    server app2 192.168.1.20:443 check

backend default_servers
    mode tcp
    server default1 192.168.1.30:443 check
</code></pre><h2 id="client-ip-preservation">Client IP Preservation <a class="anchor" data-pagefind-ignore href="#client-ip-preservation" aria-label="Link to this section">#</a></h2>
<h3 id="proxy-protocol">PROXY Protocol <a class="anchor" data-pagefind-ignore href="#proxy-protocol" aria-label="Link to this section">#</a></h3>
<p>In TCP mode the backend sees HAProxy&rsquo;s IP, not the client&rsquo;s, and there are no headers to add <code>X-Forwarded-For</code> to. The PROXY protocol passes client information through TCP load balancers by sending a short header before the data:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend tcp_frontend
    bind *:5432
    default_backend tcp_servers

backend tcp_servers
    server s1 192.168.1.10:5432 check send-proxy-v2
</code></pre><p>The backend server must support PROXY protocol, or it reads the header as garbage and drops the connection. For HAProxy as the backend:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend backend_frontend
    bind *:5432 accept-proxy
    # ...
</code></pre><h2 id="logging">Logging <a class="anchor" data-pagefind-ignore href="#logging" aria-label="Link to this section">#</a></h2>
<h3 id="tcp-specific-logging">TCP-Specific Logging <a class="anchor" data-pagefind-ignore href="#tcp-specific-logging" aria-label="Link to this section">#</a></h3>
<p><code>option tcplog</code> logs one line per connection. The custom <code>log-format</code> adds timings and connection counts.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    log 127.0.0.1:514 local0 info

defaults
    mode tcp
    option tcplog
    log global

# Custom TCP log format
frontend tcp_frontend
    bind *:5432
    log-format &#34;%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq&#34;
</code></pre><p>Log format variables for TCP:</p>
<ul>
<li><code>%ci</code>: Client IP</li>
<li><code>%cp</code>: Client port</li>
<li><code>%ft</code>: Frontend name</li>
<li><code>%b</code>: Backend name</li>
<li><code>%s</code>: Server name</li>
<li><code>%Tw</code>: Time waiting in queue (ms)</li>
<li><code>%Tc</code>: Time to connect to server (ms)</li>
<li><code>%Tt</code>: Total time (ms)</li>
<li><code>%B</code>: Bytes read from server</li>
<li><code>%ts</code>: Termination state</li>
</ul>
<p>HAProxy sends logs to syslog on UDP 514. Configure rsyslog to receive them and write <code>local0</code> to a file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># /etc/rsyslog.d/haproxy.conf</span>
</span></span><span class="line"><span class="cl"><span class="nv">$ModLoad</span> imudp
</span></span><span class="line"><span class="cl"><span class="nv">$UDPServerRun</span> <span class="m">514</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">local0.* /var/log/haproxy.log
</span></span></code></pre></div><h2 id="statistics-and-monitoring">Statistics and Monitoring <a class="anchor" data-pagefind-ignore href="#statistics-and-monitoring" aria-label="Link to this section">#</a></h2>
<p>The stats page is HTTP even when all your proxies are TCP, so it sets <code>mode http</code> itself.</p>
<h3 id="stats-page">Stats Page <a class="anchor" data-pagefind-ignore href="#stats-page" aria-label="Link to this section">#</a></h3>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST
</code></pre><h3 id="prometheus-metrics">Prometheus Metrics <a class="anchor" data-pagefind-ignore href="#prometheus-metrics" aria-label="Link to this section">#</a></h3>
<p>See <a href="https://blog.bsdserver.nl/haproxy-monitoring-with-prometheus-complete-observability-guide/">HAProxy Monitoring with Prometheus</a> for scraping and alerting.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">frontend stats
    bind *:8404
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
</code></pre><h2 id="complete-production-example">Complete Production Example <a class="anchor" data-pagefind-ignore href="#complete-production-example" aria-label="Link to this section">#</a></h2>
<p>A complete configuration for a PostgreSQL cluster and Redis:</p>
<table>
	<thead>
			<tr>
					<th>Frontend</th>
					<th>Port</th>
					<th>Backend</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>pg_write</code></td>
					<td>5432</td>
					<td>Primary, standby as backup; over 5000 connections are rejected</td>
			</tr>
			<tr>
					<td><code>pg_read</code></td>
					<td>5433</td>
					<td>Standbys with <code>leastconn</code>, primary as backup</td>
			</tr>
			<tr>
					<td><code>redis</code></td>
					<td>6379</td>
					<td>Only the server reporting <code>role:master</code> passes the check</td>
			</tr>
			<tr>
					<td><code>stats</code></td>
					<td>8404</td>
					<td>Stats page and <code>/metrics</code></td>
			</tr>
	</tbody>
</table>
<p>Settings worth knowing:</p>
<ul>
<li><code>option redispatch</code>: retry on another server if the chosen one fails.</li>
<li><code>option log-health-checks</code>: log health check state changes.</li>
<li><code>pgsql-check</code> checks that PostgreSQL answers, not whether it is the primary.</li>
</ul>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy"># /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
    maxconn 100000
    nbthread 4
    cpu-map auto:1/1-4 0-3
    
    # Security
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    option  log-health-checks
    
    timeout connect 5s
    timeout client  3600s
    timeout server  3600s
    timeout check   3s
    
    retries 3
    option  redispatch
    
    default-server inter 3s fall 3 rise 2

# Stats frontend
listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST
    http-request use-service prometheus-exporter if { path /metrics }

# PostgreSQL Primary (Read/Write)
frontend pg_write
    bind *:5432
    default_backend pg_primary
    
    # ACL for connection limiting
    acl too_many_conns fe_conn gt 5000
    tcp-request connection reject if too_many_conns

backend pg_primary
    option pgsql-check user haproxy
    server pg-primary 192.168.1.10:5432 check
    server pg-standby1 192.168.1.11:5432 check backup

# PostgreSQL Replicas (Read-Only)
frontend pg_read
    bind *:5433
    default_backend pg_replicas

backend pg_replicas
    balance leastconn
    option pgsql-check user haproxy
    server pg-standby1 192.168.1.11:5432 check
    server pg-standby2 192.168.1.12:5432 check
    server pg-primary  192.168.1.10:5432 check backup  # Fallback

# Redis Cluster
frontend redis
    bind *:6379
    default_backend redis_servers

backend redis_servers
    balance first
    option tcp-check
    tcp-check connect
    tcp-check send &#34;PING\r\n&#34;
    tcp-check expect string +PONG
    tcp-check send &#34;INFO replication\r\n&#34;
    tcp-check expect string role:master
    tcp-check send &#34;QUIT\r\n&#34;
    tcp-check expect string +OK
    server redis1 192.168.1.20:6379 check
    server redis2 192.168.1.21:6379 check
    server redis3 192.168.1.22:6379 check
</code></pre><h2 id="validation-and-testing">Validation and Testing <a class="anchor" data-pagefind-ignore href="#validation-and-testing" aria-label="Link to this section">#</a></h2>
<h3 id="configuration-validation">Configuration Validation <a class="anchor" data-pagefind-ignore href="#configuration-validation" aria-label="Link to this section">#</a></h3>
<p>Check the syntax before every reload.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">haproxy -c -f /etc/haproxy/haproxy.cfg
</span></span></code></pre></div><h3 id="load-testing">Load Testing <a class="anchor" data-pagefind-ignore href="#load-testing" aria-label="Link to this section">#</a></h3>
<p>Open many connections at once, then run a real database benchmark through HAProxy.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Test TCP connection throughput</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> i in <span class="o">{</span>1..100<span class="o">}</span><span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="cl">  nc -z -w1 haproxy.example.com <span class="m">5432</span> <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl"><span class="k">done</span>
</span></span><span class="line"><span class="cl"><span class="nb">wait</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># PostgreSQL specific (initialise the pgbench tables once first)</span>
</span></span><span class="line"><span class="cl">pgbench -i -h haproxy.example.com -p <span class="m">5432</span> mydb
</span></span><span class="line"><span class="cl">pgbench -h haproxy.example.com -p <span class="m">5432</span> -c <span class="m">50</span> -j <span class="m">4</span> -T <span class="m">60</span> mydb
</span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Connections dropped after idle time</td>
					<td>Increase <code>timeout server</code> and <code>timeout client</code></td>
			</tr>
			<tr>
					<td>Server marked DOWN</td>
					<td>Check health check configuration and network connectivity</td>
			</tr>
			<tr>
					<td>Connection refused</td>
					<td>Verify backend servers are listening and firewall rules</td>
			</tr>
			<tr>
					<td>Slow connections</td>
					<td>Check <code>balance</code> algorithm and server capacity</td>
			</tr>
			<tr>
					<td>HAProxy does not start on RHEL</td>
					<td>Missing <code>/run/haproxy</code>; see the <code>RuntimeDirectory</code> fix above</td>
			</tr>
	</tbody>
</table>
<h3 id="debug-mode">Debug Mode <a class="anchor" data-pagefind-ignore href="#debug-mode" aria-label="Link to this section">#</a></h3>
<p><code>haproxy -d</code> runs in the foreground and prints every event. The socket commands show process info, per-server stats and the last protocol errors.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Run HAProxy in debug mode</span>
</span></span><span class="line"><span class="cl">haproxy -d -f /etc/haproxy/haproxy.cfg
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check runtime status</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show info&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show stat&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;show errors&#34;</span> <span class="p">|</span> socat stdio /run/haproxy/admin.sock
</span></span></code></pre></div><h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-tcp-lb" rel="noopener">pderuiter/blog-haproxy-tcp-lb</a>. It creates an HAProxy VM on Proxmox and configures the complete production example from this post, PostgreSQL read/write and read-only frontends and the Redis master check included; set your database and Redis servers in <code>group_vars</code>, run <code>terraform apply</code>, then <code>ansible-playbook site.yml</code>.</p>
<h2 id="next-steps">Next Steps <a class="anchor" data-pagefind-ignore href="#next-steps" aria-label="Link to this section">#</a></h2>
<p>Next in the series: DNS load balancing with HAProxy, to spread DNS queries across multiple resolvers.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<p>Key takeaways:</p>
<ul>
<li>Use <code>mode tcp</code> for Layer 4 load balancing</li>
<li>Choose the right balancing algorithm for your use case</li>
<li>Implement proper health checks for each protocol</li>
<li>Configure appropriate timeouts for your workload</li>
<li>Use the stats socket for runtime management</li>
<li>Enable logging for troubleshooting and monitoring</li>
</ul>
<p>TCP load balancing is the building block for highly available infrastructure, whether you&rsquo;re running databases, message queues, or custom services.</p>
]]></content:encoded></item><item><title>Automating iSCSI Multipathing with Ansible</title><link>https://blog.bsdserver.nl/automating-iscsi-multipathing-with-ansible/</link><guid isPermaLink="false">693b680679eb54000160a52b</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 00:55:34 +0000</pubDate><description>Multipathing (MPIO - Multipath I/O) is essential for high availability and performance in iSCSI storage environments.</description><category>Linux</category><category>Storage</category><category>iSCSI</category><category>Ansible</category><category>High availability</category><category>Automation</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/ansible-iscsi-multipathing.png" medium="image"/><content:encoded><![CDATA[<p>A single iSCSI connection is a single point of failure. One broken switch port or NIC, and the server loses its disk.</p>
<p>Multipathing (MPIO, Multipath I/O) connects the initiator to the target over two or more network paths and shows them as one disk. This post builds an Ansible role that installs the iSCSI and multipath tools, logs in over every path and combines them into one multipath device.</p>
<h2 id="why-use-multipathing">Why use multipathing? <a class="anchor" data-pagefind-ignore href="#why-use-multipathing" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Redundancy</strong>: if one network path fails, another takes over</li>
<li><strong>Load balancing</strong>: I/O is spread across all paths</li>
<li><strong>Higher bandwidth</strong>: more network interfaces, more throughput</li>
<li><strong>Automatic failover</strong>: no manual action when a path fails</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when the target has portals on two or more storage networks and the server needs its disk to survive a network failure.</li>
<li><strong>Skip it</strong> when there is only one network path. Multipath adds nothing then; the plain initiator setup from <a href="https://blog.bsdserver.nl/automating-iscsi-initiator-configuration-with-ansible/">Automating iSCSI Initiator Configuration with Ansible</a> is enough.</li>
</ul>
<p>Want to understand each step by hand first? See <a href="https://blog.bsdserver.nl/setting-up-iscsi-multipathing-manually-on-linux-2/">Setting Up iSCSI Multipathing Manually on Linux</a>.</p>
<h2 id="how-it-works">How it works <a class="anchor" data-pagefind-ignore href="#how-it-works" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Ansible role <code>iscsi_multipath</code></td>
					<td>Control machine</td>
					<td>Installs packages, renders configs, logs in to every portal.</td>
			</tr>
			<tr>
					<td><code>iscsid</code></td>
					<td>Initiator</td>
					<td>One iSCSI session per portal (path).</td>
			</tr>
			<tr>
					<td><code>multipathd</code></td>
					<td>Initiator</td>
					<td>Sees the same LUN on every path (same WWID) and combines the paths into one <code>mpatha</code> device.</td>
			</tr>
			<tr>
					<td>iSCSI target</td>
					<td>Storage server</td>
					<td>One portal per storage network, same LUN behind both.</td>
			</tr>
	</tbody>
</table>
<p>The flow: log in to portal 1 → <code>sdb</code> appears → log in to portal 2 → <code>sdc</code> appears → <code>multipathd</code> sees the same WWID on both → creates <code>/dev/mapper/mpatha</code> → you use only <code>mpatha</code>.</p>
<h2 id="the-role-iscsi_multipath">The role: iscsi_multipath <a class="anchor" data-pagefind-ignore href="#the-role-iscsi_multipath" aria-label="Link to this section">#</a></h2>
<p>The role:</p>
<ul>
<li>Installs the required packages</li>
<li>Configures <code>iscsid.conf</code> and <code>multipath.conf</code></li>
<li>Discovers the iSCSI targets on every portal</li>
<li>Logs in to every path</li>
<li>Keeps <code>multipathd</code> enabled and running</li>
</ul>
<h3 id="role-directory-structure">Role directory structure <a class="anchor" data-pagefind-ignore href="#role-directory-structure" aria-label="Link to this section">#</a></h3>
<p>A standard role layout; the files below fill <code>defaults</code>, <code>tasks</code>, <code>handlers</code> and <code>templates</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">iscsi_multipath/
</span></span><span class="line"><span class="cl">├── tasks/
</span></span><span class="line"><span class="cl">│   ├── main.yml
</span></span><span class="line"><span class="cl">├── handlers/
</span></span><span class="line"><span class="cl">│   ├── main.yml
</span></span><span class="line"><span class="cl">├── templates/
</span></span><span class="line"><span class="cl">│   ├── iscsid.conf.j2
</span></span><span class="line"><span class="cl">│   ├── multipath.conf.j2
</span></span><span class="line"><span class="cl">├── vars/
</span></span><span class="line"><span class="cl">│   ├── main.yml
</span></span><span class="line"><span class="cl">├── defaults/
</span></span><span class="line"><span class="cl">│   ├── main.yml
</span></span><span class="line"><span class="cl">├── meta/
</span></span><span class="line"><span class="cl">│   ├── main.yml
</span></span><span class="line"><span class="cl">├── README.md
</span></span></code></pre></div><h2 id="role-configuration">Role configuration <a class="anchor" data-pagefind-ignore href="#role-configuration" aria-label="Link to this section">#</a></h2>
<h3 id="define-variables">Define variables <a class="anchor" data-pagefind-ignore href="#define-variables" aria-label="Link to this section">#</a></h3>
<p><code>iscsi_target_ips</code> lists one portal per path; the role discovers and logs in to each. Create <code>defaults/main.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">iscsi_target_ips</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;192.168.2.253&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;192.168.3.253&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">iscsi_target_name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;iqn.2025-03.com.example:storage&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">iscsi_chap_username</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ vault_iscsi_chap_username }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">iscsi_chap_password</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ vault_iscsi_chap_password }}&#34;</span><span class="w">
</span></span></span></code></pre></div><p>The CHAP credentials themselves don&rsquo;t belong in the role. Keep them in an Ansible Vault encrypted file next to your playbook and inventory, <code>group_vars/all/vault.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">ansible-vault create group_vars/all/vault.yml
</span></span></code></pre></div><p>Ansible asks for a new vault password and opens your editor. Add the CHAP credentials, prefixed with <code>vault_</code> so it is obvious where they come from:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">vault_iscsi_chap_username</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;your_username&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">vault_iscsi_chap_password</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;your_secure_password&#34;</span><span class="w">
</span></span></span></code></pre></div><ul>
<li>On disk the file starts with <code>$ANSIBLE_VAULT;1.1;AES256</code> and is safe to commit.</li>
<li>Already have a plaintext file? Encrypt it in place with <code>ansible-vault encrypt group_vars/all/vault.yml</code>.</li>
<li>Use <code>ansible-vault view</code> or <code>ansible-vault edit</code> to read or change it later.</li>
</ul>
<h3 id="tasks">Tasks <a class="anchor" data-pagefind-ignore href="#tasks" aria-label="Link to this section">#</a></h3>
<p>The tasks install the tools, write both configs, load the multipath config and then log in over every path. Create <code>tasks/main.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nn">---</span><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Install iSCSI and multipath packages</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.apt</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">open-iscsi</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">multipath-tools</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">present</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">update_cache</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Deploy iSCSI configuration</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="l">iscsid.conf.j2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/iscsi/iscsid.conf</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0644&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">no_log</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">  </span><span class="c"># the rendered file contains the CHAP password</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Deploy Multipath configuration</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="l">multipath.conf.j2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/multipath.conf</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0644&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">notify</span><span class="p">:</span><span class="w"> </span><span class="l">Reload multipathd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Enable and start multipathd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.systemd</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">multipathd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">started</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Load a changed multipath configuration before logging in</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.meta</span><span class="p">:</span><span class="w"> </span><span class="l">flush_handlers</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Discover iSCSI targets</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.command</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;iscsiadm -m discovery -t st -p {{ item }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">loop</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ iscsi_target_ips }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l">iscsi_discovery</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">changed_when</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Check active iSCSI sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.command</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;iscsiadm -m session&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l">iscsi_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">changed_when</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">failed_when</span><span class="p">:</span><span class="w"> </span><span class="l">iscsi_sessions.rc not in [0, 21] </span><span class="w"> </span><span class="c"># 21 = no active sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Log in to iSCSI targets</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.command</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;iscsiadm -m node -T {{ iscsi_target_name }} -p {{ item }} --login&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">loop</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ iscsi_target_ips }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l">(item ~ &#39;:3260&#39;) not in iscsi_sessions.stdout</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">changed_when</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">  </span><span class="c"># only runs for portals without a session</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Ensure iSCSI login persists across reboots</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.command</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;iscsiadm -m node -T {{ iscsi_target_name }} -p {{ item }} --op=update --name=node.startup --value=automatic&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">loop</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ iscsi_target_ips }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">changed_when</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span></code></pre></div><table>
	<thead>
			<tr>
					<th>Task</th>
					<th>What it does</th>
					<th>Why</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Install packages</td>
					<td><code>open-iscsi</code> and <code>multipath-tools</code></td>
					<td>Initiator and multipath daemon.</td>
			</tr>
			<tr>
					<td>Deploy iSCSI configuration</td>
					<td>Renders <code>iscsid.conf</code></td>
					<td><code>no_log: true</code>: the file contains the CHAP password.</td>
			</tr>
			<tr>
					<td>Deploy Multipath configuration</td>
					<td>Renders <code>multipath.conf</code></td>
					<td>Triggers a reload of <code>multipathd</code> when it changes.</td>
			</tr>
			<tr>
					<td>Enable and start multipathd</td>
					<td>Starts the daemon, enables it at boot</td>
					<td>It must run before the paths appear.</td>
			</tr>
			<tr>
					<td><code>flush_handlers</code></td>
					<td>Runs the reload now</td>
					<td>The new <code>multipath.conf</code> is loaded before the logins.</td>
			</tr>
			<tr>
					<td>Discover iSCSI targets</td>
					<td>Discovery on every portal</td>
					<td>Creates a node record per path.</td>
			</tr>
			<tr>
					<td>Check active iSCSI sessions</td>
					<td>Lists sessions</td>
					<td>Exit code 21 means &ldquo;no sessions&rdquo;, not an error.</td>
			</tr>
			<tr>
					<td>Log in to iSCSI targets</td>
					<td>Logs in per portal</td>
					<td>Skipped for portals that already have a session.</td>
			</tr>
			<tr>
					<td>Ensure login persists</td>
					<td>Sets <code>node.startup</code> to <code>automatic</code> per portal</td>
					<td>All paths come back after a reboot.</td>
			</tr>
	</tbody>
</table>
<h3 id="handlers">Handlers <a class="anchor" data-pagefind-ignore href="#handlers" aria-label="Link to this section">#</a></h3>
<p>One handler: reload <code>multipathd</code> when its config changes. Create <code>handlers/main.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nn">---</span><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Reload multipathd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ansible.builtin.systemd</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">multipathd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">reloaded</span><span class="w">
</span></span></span></code></pre></div><p>Why the handler and the session check matter:</p>
<ul>
<li><strong>Reload, not start.</strong> <code>multipathd</code> is normally already running, so <code>state: started</code> alone never loads a new <code>multipath.conf</code>. The handler reloads it when the template changes, and <code>flush_handlers</code> runs it before the logins.</li>
<li><strong>Skip existing sessions.</strong> <code>iscsiadm --login</code> fails with &ldquo;session already present&rdquo; on a second run. The session check skips portals that are already logged in.</li>
</ul>
<h2 id="templates-for-configuration-files">Templates for configuration files <a class="anchor" data-pagefind-ignore href="#templates-for-configuration-files" aria-label="Link to this section">#</a></h2>
<h3 id="templatesiscsidconfj2">templates/iscsid.conf.j2 <a class="anchor" data-pagefind-ignore href="#templatesiscsidconfj2" aria-label="Link to this section">#</a></h3>
<p>The initiator settings: start <code>iscsid</code> via its socket, log in automatically at boot, authenticate with CHAP, scan for LUNs after login.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jinja" data-lang="jinja"><span class="line"><span class="cl"><span class="x">iscsid.startup = /bin/systemctl start iscsid.socket
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">node.startup = automatic
</span></span></span><span class="line"><span class="cl"><span class="x">node.leading_login = No
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">node.session.auth.authmethod = CHAP
</span></span></span><span class="line"><span class="cl"><span class="x">node.session.auth.username = </span><span class="cp">{{</span> <span class="nv">iscsi_chap_username</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">node.session.auth.password = </span><span class="cp">{{</span> <span class="nv">iscsi_chap_password</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">node.session.scan = auto
</span></span></span></code></pre></div><ul>
<li><code>node.startup = automatic</code>: log in to all discovered paths at boot.</li>
<li><code>node.session.auth.*</code>: CHAP with the credentials from the vault.</li>
<li><code>node.session.scan = auto</code>: scan for LUNs automatically after login.</li>
</ul>
<h3 id="templatesmultipathconfj2">templates/multipath.conf.j2 <a class="anchor" data-pagefind-ignore href="#templatesmultipathconfj2" aria-label="Link to this section">#</a></h3>
<p>This tells <code>multipathd</code> which devices to manage and how to use the paths:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jinja" data-lang="jinja"><span class="line"><span class="cl"><span class="x">defaults {
</span></span></span><span class="line"><span class="cl"><span class="x">    user_friendly_names yes
</span></span></span><span class="line"><span class="cl"><span class="x">}
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">blacklist {
</span></span></span><span class="line"><span class="cl"><span class="x">    wwid &#34;.*&#34;
</span></span></span><span class="line"><span class="cl"><span class="x">}
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">blacklist_exceptions {
</span></span></span><span class="line"><span class="cl"><span class="x">    wwid &#34;</span><span class="cp">{{</span> <span class="nv">iscsi_wwid</span> <span class="cp">}}</span><span class="x">&#34;
</span></span></span><span class="line"><span class="cl"><span class="x">}
</span></span></span><span class="line"><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="cl"><span class="x">multipaths {
</span></span></span><span class="line"><span class="cl"><span class="x">    multipath {
</span></span></span><span class="line"><span class="cl"><span class="x">        wwid &#34;</span><span class="cp">{{</span> <span class="nv">iscsi_wwid</span> <span class="cp">}}</span><span class="x">&#34;
</span></span></span><span class="line"><span class="cl"><span class="x">        alias mpatha
</span></span></span><span class="line"><span class="cl"><span class="x">        path_grouping_policy &#34;multibus&#34;
</span></span></span><span class="line"><span class="cl"><span class="x">        path_selector &#34;round-robin 0&#34;
</span></span></span><span class="line"><span class="cl"><span class="x">        failback immediate
</span></span></span><span class="line"><span class="cl"><span class="x">        rr_weight priorities
</span></span></span><span class="line"><span class="cl"><span class="x">        no_path_retry 5
</span></span></span><span class="line"><span class="cl"><span class="x">    }
</span></span></span><span class="line"><span class="cl"><span class="x">}
</span></span></span></code></pre></div><table>
	<thead>
			<tr>
					<th>Setting</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>user_friendly_names yes</code></td>
					<td>Names maps <code>mpathN</code> instead of by WWID.</td>
			</tr>
			<tr>
					<td><code>blacklist { wwid &quot;.*&quot; }</code> + <code>blacklist_exceptions</code></td>
					<td>Ignore every device except your iSCSI LUN. Keeps local disks out.</td>
			</tr>
			<tr>
					<td><code>alias mpatha</code></td>
					<td>Fixed name for this LUN: <code>/dev/mapper/mpatha</code>.</td>
			</tr>
			<tr>
					<td><code>path_grouping_policy &quot;multibus&quot;</code></td>
					<td>All paths in one group, so all of them carry I/O.</td>
			</tr>
			<tr>
					<td><code>path_selector &quot;round-robin 0&quot;</code></td>
					<td>Rotate I/O over the paths in the group.</td>
			</tr>
			<tr>
					<td><code>failback immediate</code></td>
					<td>Switch back to the preferred path group as soon as it is available again.</td>
			</tr>
			<tr>
					<td><code>rr_weight priorities</code></td>
					<td>Weight the round-robin by path priority.</td>
			</tr>
			<tr>
					<td><code>no_path_retry 5</code></td>
					<td>When all paths are down, queue I/O for 5 checker intervals, then fail it.</td>
			</tr>
	</tbody>
</table>
<p><strong>Note:</strong> Don&rsquo;t blacklist <code>devnode &quot;^sd[a-z]&quot;</code>: the iSCSI paths are <code>sdX</code> devices as well, and that rule makes multipathd drop the map. The <code>wwid &quot;.*&quot;</code> blacklist with an exception for your LUN keeps local disks out without hiding the iSCSI paths.</p>
<h3 id="finding-the-wwid">Finding the WWID <a class="anchor" data-pagefind-ignore href="#finding-the-wwid" aria-label="Link to this section">#</a></h3>
<p>The template needs the WWID (World Wide Identifier) of the LUN in <code>iscsi_wwid</code>. Retrieve it manually, on a node that is already logged in to the target:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo multipath -ll
</span></span></code></pre></div><p>The WWID is the value between brackets after the map name. Then add it as <code>iscsi_wwid</code> to <code>vars/main.yml</code>.</p>
<h2 id="running-the-ansible-playbook">Running the Ansible playbook <a class="anchor" data-pagefind-ignore href="#running-the-ansible-playbook" aria-label="Link to this section">#</a></h2>
<p>A playbook that applies the role to all hosts. Create <code>setup_iscsi_multipath.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Setup iSCSI Multipathing</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">hosts</span><span class="p">:</span><span class="w"> </span><span class="l">all</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">become</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">roles</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">iscsi_multipath</span><span class="w">
</span></span></span></code></pre></div><p>Run the playbook and enter the vault password when asked:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">ansible-playbook -i inventory setup_iscsi_multipath.yml --ask-vault-pass
</span></span></code></pre></div><p>Or, for unattended runs, with a password file kept outside the repository:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">ansible-playbook -i inventory setup_iscsi_multipath.yml --vault-password-file ~/.vault_pass.txt
</span></span></code></pre></div><p>Thanks to <code>no_log: true</code> on the <code>iscsid.conf</code> task the CHAP password never shows up in the output, not even with <code>-v</code> or <code>--diff</code>.</p>
<h2 id="verifying-the-setup">Verifying the setup <a class="anchor" data-pagefind-ignore href="#verifying-the-setup" aria-label="Link to this section">#</a></h2>
<p>Check that there is one session per path and that both paths are part of one multipath device.</p>
<p><strong>Active iSCSI sessions</strong> (one line per portal):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo iscsiadm -m session
</span></span></code></pre></div><p>Expected output:</p>
<pre tabindex="0"><code>tcp: [1] 192.168.2.253:3260,1 iqn.2025-03.com.example:storage (non-flash)
tcp: [2] 192.168.3.253:3260,1 iqn.2025-03.com.example:storage (non-flash)
</code></pre><p><strong>Multipath devices:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo multipath -ll
</span></span></code></pre></div><p>Expected output:</p>
<pre tabindex="0"><code>mpatha (3600140547f55689de4b49c098d9b1a3c) dm-0 LIO-ORG,iscsi_target
size=20G features=&#39;1 queue_if_no_path&#39; hwhandler=&#39;1 alua&#39; wp=rw
`-+- policy=&#39;round-robin 0&#39; prio=50 status=active
  |- 2:0:0:0 sdb 8:16 active ready running
  `- 3:0:0:0 sdc 8:32 active ready running
</code></pre><p>What to look for:</p>
<ul>
<li><code>mpatha (3600…)</code>: the alias and the WWID.</li>
<li><code>policy='round-robin 0'</code>: both paths are in one group.</li>
<li><code>sdb</code> and <code>sdc</code> both <code>active ready running</code>: both paths work. A failed path shows as <code>failed faulty</code>.</li>
</ul>
<h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>multipath -ll</code> shows nothing</td>
					<td>WWID in <code>iscsi_wwid</code> does not match the LUN, so the blacklist hides it. Compare with <code>/lib/udev/scsi_id -g -u -d /dev/sdb</code>.</td>
			</tr>
			<tr>
					<td>Map disappears after a config change</td>
					<td>A <code>devnode &quot;^sd[a-z]&quot;</code> blacklist hides the iSCSI paths. Use the WWID blacklist.</td>
			</tr>
			<tr>
					<td>Only one path in the map</td>
					<td>Login to one portal failed. Check <code>iscsiadm -m session</code> and the network to that portal.</td>
			</tr>
			<tr>
					<td>New <code>multipath.conf</code> has no effect</td>
					<td><code>multipathd</code> was not reloaded. Run <code>systemctl reload multipathd</code>.</td>
			</tr>
			<tr>
					<td>&ldquo;session already present&rdquo;</td>
					<td>Already logged in. The role skips logged-in portals.</td>
			</tr>
	</tbody>
</table>
<h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li>Keep the CHAP credentials in Ansible Vault; <code>no_log: true</code> keeps them out of the output.</li>
<li><code>iscsid.conf</code> on each node holds the CHAP password in plaintext. Consider mode <code>0600</code>.</li>
<li>Put the storage networks on separate VLANs or switches, not on the client network.</li>
<li>Limit LUN access on the target to the expected initiator IQNs.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub: <a href="https://github.com/pderuiter/blog-iscsi-multipath-ansible" rel="noopener">pderuiter/blog-iscsi-multipath-ansible</a>. It creates an iSCSI target with a portal on each of two storage networks and an initiator on Proxmox, and applies the <code>iscsi_multipath</code> role from this post with the CHAP secret in Ansible Vault; run <code>terraform apply</code> and then <code>ansible-playbook site.yml --ask-vault-pass</code>.</p>
<h2 id="conclusion">Conclusion <a class="anchor" data-pagefind-ignore href="#conclusion" aria-label="Link to this section">#</a></h2>
<p>With this role every node:</p>
<ul>
<li>Logs in to the target over every path</li>
<li>Keeps its sessions across reboots</li>
<li>Uses one multipath device with failover and load balancing</li>
<li>Gets the same setup as every other node</li>
</ul>
<p>Use <code>/dev/mapper/mpatha</code>, never the <code>sdX</code> paths, for filesystems or LVM.</p>
]]></content:encoded></item><item><title>Setting Up iSCSI Multipathing Manually on Linux</title><link>https://blog.bsdserver.nl/setting-up-iscsi-multipathing-manually-on-linux-2/</link><guid isPermaLink="false">693b67e379eb54000160a523</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 00:54:59 +0000</pubDate><description>Multipathing (MPIO - Multipath I/O) is crucial for high availability and performance in iSCSI storage environments.</description><category>Linux</category><category>Storage</category><category>iSCSI</category><category>High availability</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/iscsi-multipathing.png" medium="image"/><content:encoded><![CDATA[<p>A server with one iSCSI connection loses its disk when that one network path fails. A switch reboot or a bad cable is enough.</p>
<p>Multipathing (MPIO, Multipath I/O) connects the server (initiator) to the storage (target) over two or more networks and shows the paths as one disk. This post configures iSCSI multipathing by hand on Linux, without Ansible, and tests the failover.</p>
<h2 id="why-use-multipathing">Why use multipathing? <a class="anchor" data-pagefind-ignore href="#why-use-multipathing" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Redundancy</strong>: if one network path fails, another takes over</li>
<li><strong>Load balancing</strong>: I/O is spread across the paths</li>
<li><strong>Higher bandwidth</strong>: more network interfaces, more throughput</li>
<li><strong>Failover support</strong>: storage stays available while a path is down</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when the storage has two separate networks and the server must survive losing one.</li>
<li><strong>Skip it</strong> with a single network path; a plain iSCSI login is enough.</li>
<li><strong>Do it by hand</strong> to learn the steps or for one server. For more servers, see <a href="https://blog.bsdserver.nl/automating-iscsi-multipathing-with-ansible/">Automating iSCSI Multipathing with Ansible</a>.</li>
</ul>
<h2 id="how-it-works">How it works <a class="anchor" data-pagefind-ignore href="#how-it-works" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>iSCSI target</td>
					<td>Storage server</td>
					<td>Exports the LUN on two IP addresses (portals).</td>
			</tr>
			<tr>
					<td><code>iscsid</code></td>
					<td>Initiator</td>
					<td>One session per portal; each session shows the LUN as its own <code>sdX</code> device.</td>
			</tr>
			<tr>
					<td><code>multipathd</code></td>
					<td>Initiator</td>
					<td>Sees that <code>sdb</code> and <code>sdc</code> have the same WWID and combines them into <code>/dev/mapper/mpatha</code>.</td>
			</tr>
	</tbody>
</table>
<p>The flow: discover the target → log in over path 1 (<code>sdb</code>) and path 2 (<code>sdc</code>) → <code>multipathd</code> matches the WWID → <code>mpatha</code> → filesystem or LVM on <code>mpatha</code>.</p>
<h2 id="requirements">Requirements <a class="anchor" data-pagefind-ignore href="#requirements" aria-label="Link to this section">#</a></h2>
<h3 id="server-side-iscsi-target">Server side (iSCSI target) <a class="anchor" data-pagefind-ignore href="#server-side-iscsi-target" aria-label="Link to this section">#</a></h3>
<ul>
<li>The target must support multiple network paths.</li>
<li>Each LUN must be accessible via <strong>at least two separate IP addresses</strong>.</li>
<li>Example target setup:</li>
</ul>
<pre tabindex="0"><code>iSCSI Target:
IP1: 192.168.2.253 (eth1)
IP2: 192.168.3.253 (eth2)
</code></pre><h3 id="client-side-iscsi-initiator">Client side (iSCSI initiator) <a class="anchor" data-pagefind-ignore href="#client-side-iscsi-initiator" aria-label="Link to this section">#</a></h3>
<ul>
<li>Multiple network interfaces, one per storage network.</li>
<li>The <code>open-iscsi</code> and <code>multipath-tools</code> packages installed.</li>
<li>Two or more iSCSI sessions to the same LUN using different IPs.</li>
</ul>
<h2 id="install-required-packages">Install required packages <a class="anchor" data-pagefind-ignore href="#install-required-packages" aria-label="Link to this section">#</a></h2>
<p><code>open-iscsi</code> provides the initiator, <code>multipath-tools</code> the multipath daemon. On the iSCSI initiator (client):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo apt update
</span></span><span class="line"><span class="cl">sudo apt install -y open-iscsi multipath-tools
</span></span></code></pre></div><p>Start both daemons now and at boot:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo systemctl <span class="nb">enable</span> --now iscsid multipathd
</span></span></code></pre></div><h2 id="discover-and-log-in-to-iscsi-targets">Discover and log in to iSCSI targets <a class="anchor" data-pagefind-ignore href="#discover-and-log-in-to-iscsi-targets" aria-label="Link to this section">#</a></h2>
<h3 id="discover-available-targets">Discover available targets <a class="anchor" data-pagefind-ignore href="#discover-available-targets" aria-label="Link to this section">#</a></h3>
<p>Discovery (<code>-t st</code>, SendTargets) asks the target which portals and IQNs it offers and stores them as node records. Run it for each target IP. Each discovery returns every portal the target advertises, so both commands print the same two lines:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo iscsiadm -m discovery -t st -p 192.168.2.253
</span></span><span class="line"><span class="cl">sudo iscsiadm -m discovery -t st -p 192.168.3.253
</span></span></code></pre></div><p>Expected output:</p>
<pre tabindex="0"><code>192.168.2.253:3260,1 iqn.2025-03.com.example:storage
192.168.3.253:3260,1 iqn.2025-03.com.example:storage
</code></pre><h3 id="log-in-to-both-paths">Log in to both paths <a class="anchor" data-pagefind-ignore href="#log-in-to-both-paths" aria-label="Link to this section">#</a></h3>
<p>One login per portal gives one session, and one <code>sdX</code> device, per path:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo iscsiadm -m node -T iqn.2025-03.com.example:storage -p 192.168.2.253 --login
</span></span><span class="line"><span class="cl">sudo iscsiadm -m node -T iqn.2025-03.com.example:storage -p 192.168.3.253 --login
</span></span></code></pre></div><p>Verify active iSCSI sessions:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo iscsiadm -m session
</span></span></code></pre></div><p>Expected output:</p>
<pre tabindex="0"><code>tcp: [1] 192.168.2.253:3260,1 iqn.2025-03.com.example:storage (non-flash)
tcp: [2] 192.168.3.253:3260,1 iqn.2025-03.com.example:storage (non-flash)
</code></pre><p>Set the login to persist across reboots, for both paths. <code>node.startup=automatic</code> makes the initiator log in again at boot:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo iscsiadm -m node -T iqn.2025-03.com.example:storage -p 192.168.2.253 --op<span class="o">=</span>update --name<span class="o">=</span>node.startup --value<span class="o">=</span>automatic
</span></span><span class="line"><span class="cl">sudo iscsiadm -m node -T iqn.2025-03.com.example:storage -p 192.168.3.253 --op<span class="o">=</span>update --name<span class="o">=</span>node.startup --value<span class="o">=</span>automatic
</span></span></code></pre></div><h2 id="configure-multipathing">Configure multipathing <a class="anchor" data-pagefind-ignore href="#configure-multipathing" aria-label="Link to this section">#</a></h2>
<h3 id="enable-multipath">Enable multipath <a class="anchor" data-pagefind-ignore href="#enable-multipath" aria-label="Link to this section">#</a></h3>
<p><code>/etc/multipath.conf</code> tells <code>multipathd</code> which devices to combine and how to use the paths. Edit the file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo nano /etc/multipath.conf
</span></span></code></pre></div><p>Add the following configuration:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="na">defaults {</span>
</span></span><span class="line"><span class="cl">    <span class="na">user_friendly_names yes</span>
</span></span><span class="line"><span class="cl"><span class="na">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="na">blacklist {</span>
</span></span><span class="line"><span class="cl">    <span class="na">wwid &#34;.*&#34;</span>
</span></span><span class="line"><span class="cl"><span class="na">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="na">blacklist_exceptions {</span>
</span></span><span class="line"><span class="cl">    <span class="na">wwid &#34;3600140547f55689de4b49c098d9b1a3c&#34;</span>
</span></span><span class="line"><span class="cl"><span class="na">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="na">multipaths {</span>
</span></span><span class="line"><span class="cl">    <span class="na">multipath {</span>
</span></span><span class="line"><span class="cl">        <span class="na">wwid &#34;3600140547f55689de4b49c098d9b1a3c&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="na">alias mpatha</span>
</span></span><span class="line"><span class="cl">        <span class="na">path_grouping_policy &#34;multibus&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="na">path_selector &#34;round-robin 0&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="na">failback immediate</span>
</span></span><span class="line"><span class="cl">        <span class="na">rr_weight priorities</span>
</span></span><span class="line"><span class="cl">        <span class="na">no_path_retry 5</span>
</span></span><span class="line"><span class="cl">    <span class="na">}</span>
</span></span><span class="line"><span class="cl"><span class="na">}</span>
</span></span></code></pre></div><table>
	<thead>
			<tr>
					<th>Setting</th>
					<th>What it does</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>user_friendly_names yes</code></td>
					<td>Names maps <code>mpathN</code> instead of by WWID.</td>
			</tr>
			<tr>
					<td><code>blacklist { wwid &quot;.*&quot; }</code></td>
					<td>Ignore every device by default.</td>
			</tr>
			<tr>
					<td><code>blacklist_exceptions</code></td>
					<td>&hellip;except the iSCSI LUN with this WWID.</td>
			</tr>
			<tr>
					<td><code>alias mpatha</code></td>
					<td>Fixed name: <code>/dev/mapper/mpatha</code>.</td>
			</tr>
			<tr>
					<td><code>path_grouping_policy &quot;multibus&quot;</code></td>
					<td>All paths in one group; all of them carry I/O.</td>
			</tr>
			<tr>
					<td><code>path_selector &quot;round-robin 0&quot;</code></td>
					<td>Rotate I/O over the paths in the group.</td>
			</tr>
			<tr>
					<td><code>failback immediate</code></td>
					<td>Switch back to the preferred path group as soon as it is available again.</td>
			</tr>
			<tr>
					<td><code>rr_weight priorities</code></td>
					<td>Weight the round-robin by path priority.</td>
			</tr>
			<tr>
					<td><code>no_path_retry 5</code></td>
					<td>When all paths are down, queue I/O for 5 checker intervals, then fail it.</td>
			</tr>
	</tbody>
</table>
<p>Do not blacklist <code>devnode &quot;^sd[a-z]&quot;</code> here:</p>
<ul>
<li>iSCSI LUNs show up as <code>sdX</code> devices too, so that rule blacklists both paths and multipathd removes the <code>mpatha</code> map.</li>
<li>Blacklisting every WWID with an exception for the LUNs you want multipathed leaves your local disks alone without hiding the iSCSI paths.</li>
</ul>
<h3 id="find-the-wwid">Find the WWID <a class="anchor" data-pagefind-ignore href="#find-the-wwid" aria-label="Link to this section">#</a></h3>
<p>The config needs the <strong>WWID</strong> of your LUN. Get it before you edit the file; the default Ubuntu configuration already builds the map once both sessions are logged in:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo multipath -ll
</span></span></code></pre></div><p>Or query one of the path devices directly:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo /lib/udev/scsi_id -g -u -d /dev/sdb
</span></span></code></pre></div><h3 id="restart-the-multipath-service">Restart the multipath service <a class="anchor" data-pagefind-ignore href="#restart-the-multipath-service" aria-label="Link to this section">#</a></h3>
<p>Restart <code>multipathd</code> so it reads the new configuration:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo systemctl restart multipathd
</span></span></code></pre></div><p>Enable it at boot:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo systemctl <span class="nb">enable</span> multipathd
</span></span></code></pre></div><h2 id="verifying-the-multipathing-setup">Verifying the multipathing setup <a class="anchor" data-pagefind-ignore href="#verifying-the-multipathing-setup" aria-label="Link to this section">#</a></h2>
<h3 id="check-multipath-devices">Check multipath devices <a class="anchor" data-pagefind-ignore href="#check-multipath-devices" aria-label="Link to this section">#</a></h3>
<p>The map should show the alias, the WWID and both paths:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo multipath -ll
</span></span></code></pre></div><p>Expected output:</p>
<pre tabindex="0"><code>mpatha (3600140547f55689de4b49c098d9b1a3c) dm-0 LIO-ORG,iscsi_target
size=20G features=&#39;1 queue_if_no_path&#39; hwhandler=&#39;1 alua&#39; wp=rw
`-+- policy=&#39;round-robin 0&#39; prio=50 status=active
  |- 2:0:0:0 sdb 8:16 active ready running
  `- 3:0:0:0 sdc 8:32 active ready running
</code></pre><p>With <code>path_grouping_policy &quot;multibus&quot;</code> both paths sit in a single, active path group, so I/O is spread across them. Both <code>sdb</code> and <code>sdc</code> should be <code>active ready running</code>.</p>
<h3 id="monitor-multipathing">Monitor multipathing <a class="anchor" data-pagefind-ignore href="#monitor-multipathing" aria-label="Link to this section">#</a></h3>
<p>The daemon status and its log show path failures and recoveries:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo systemctl status multipathd
</span></span><span class="line"><span class="cl">sudo journalctl -u multipathd --no-pager --lines<span class="o">=</span><span class="m">50</span>
</span></span></code></pre></div><h2 id="simulating-a-path-failure">Simulating a path failure <a class="anchor" data-pagefind-ignore href="#simulating-a-path-failure" aria-label="Link to this section">#</a></h2>
<p>Test the failover before you rely on it. Disable one of the network interfaces:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo ip link <span class="nb">set</span> eth1 down
</span></span></code></pre></div><p>Check multipath status:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo multipath -ll
</span></span></code></pre></div><p>The failed path should be marked failed, but the system should remain operational using the second path:</p>
<pre tabindex="0"><code>`-+- policy=&#39;round-robin 0&#39; prio=50 status=active
  |- 2:0:0:0 sdb 8:16 active ready  running
  `- 3:0:0:0 sdc 8:32 failed faulty running
</code></pre><ul>
<li>Expect writes to pause for roughly 15 seconds while the iSCSI layer notices the dead connection.</li>
<li>After that, I/O continues on the remaining path without errors.</li>
</ul>
<p>Restore the path:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo ip link <span class="nb">set</span> eth1 up
</span></span></code></pre></div><p>Run <code>multipath -ll</code> again: the path returns to <code>active ready running</code>.</p>
<h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>mpatha</code> disappears after editing the config</td>
					<td>A <code>devnode &quot;^sd[a-z]&quot;</code> blacklist hides the iSCSI paths. Use the WWID blacklist.</td>
			</tr>
			<tr>
					<td><code>multipath -ll</code> shows nothing</td>
					<td>The WWID in the config does not match the LUN. Check it with <code>scsi_id</code>.</td>
			</tr>
			<tr>
					<td>Only one path in the map</td>
					<td>One login failed. Check <code>iscsiadm -m session</code> and the network to that portal.</td>
			</tr>
			<tr>
					<td>No sessions after a reboot</td>
					<td><code>node.startup</code> is not <code>automatic</code> for that portal.</td>
			</tr>
	</tbody>
</table>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub: <a href="https://github.com/pderuiter/blog-iscsi-multipath-manual" rel="noopener">pderuiter/blog-iscsi-multipath-manual</a>. It creates an iSCSI target with two portals and an initiator on Proxmox and runs the steps from this post as Ansible roles, including the WWID-based <code>multipath.conf</code>; run <code>terraform apply</code> and then <code>ansible-playbook site.yml</code>.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<p>In this guide we:</p>
<ul>
<li>Logged in to the target over two paths</li>
<li>Enabled automatic iSCSI login at boot</li>
<li>Combined the paths into one multipath device with failover and load balancing</li>
<li>Verified redundancy by simulating a path failure</li>
</ul>
<p>Put your filesystem or LVM on <code>/dev/mapper/mpatha</code>, never on <code>sdb</code> or <code>sdc</code> directly.</p>
]]></content:encoded></item></channel></rss>