<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>Prometheus | A DevOps Blog</title><link>https://blog.bsdserver.nl/tag/prometheus/</link><description>Hands-on DevOps and homelab content: Kubernetes clusters, self-hosted services, CI/CD pipelines and infrastructure as code, by Patrick de Ruiter of We Build Your Cloud.</description><language>en</language><image><url>https://blog.bsdserver.nl/images/app-icon.svg</url><title>A DevOps Blog</title><link>https://blog.bsdserver.nl/</link></image><lastBuildDate>Wed, 18 Nov 2026 09:00:00 +0100</lastBuildDate><atom:link href="https://blog.bsdserver.nl/tag/prometheus/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title>HAProxy Monitoring with Prometheus: Complete Observability Guide</title><link>https://blog.bsdserver.nl/haproxy-monitoring-with-prometheus-complete-observability-guide/</link><guid isPermaLink="false">693b76b979eb54000160a637</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:58:17 +0000</pubDate><description>Monitoring HAProxy is essential for maintaining reliable load balancing infrastructure.</description><category>HAProxy</category><category>Prometheus</category><category>Monitoring</category><category>Grafana</category><category>Observability</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/haproxy-monitoring-with-prometheus.png" medium="image"/><content:encoded><![CDATA[<p>Monitoring HAProxy is essential for maintaining reliable load balancing infrastructure. The stats page shows the current state, but not what happened at 3 a.m. and nothing pages you when a backend goes down.</p>
<p>This post exposes HAProxy&rsquo;s built-in Prometheus metrics, scrapes them with Prometheus, and adds alerting rules, recording rules and Grafana dashboard queries on top.</p>
<h2 id="why-prometheus-for-haproxy">Why Prometheus for HAProxy? <a class="anchor" data-pagefind-ignore href="#why-prometheus-for-haproxy" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Pull-based metrics</strong> - Prometheus scrapes HAProxy metrics endpoints</li>
<li><strong>Time-series database</strong> - Store historical data for trend analysis</li>
<li><strong>Powerful queries</strong> - PromQL for complex metric analysis</li>
<li><strong>Alerting</strong> - Define rules for proactive incident detection</li>
<li><strong>Grafana integration</strong> - Dashboards on the same data</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> when you run HAProxy in production and want history, alerts and dashboards, or already run Prometheus for other services.</li>
<li><strong>Skip it</strong> for a single test instance: the built-in stats page (<code>:8404/stats</code> below) is enough.</li>
</ul>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy built-in exporter</td>
					<td>Every HAProxy node</td>
					<td>Serves all counters on <code>:8405/metrics</code>.</td>
			</tr>
			<tr>
					<td>Prometheus</td>
					<td>Monitoring server</td>
					<td>Scrapes every HAProxy every 10s, stores the series, evaluates the rules.</td>
			</tr>
			<tr>
					<td>Alertmanager</td>
					<td>Monitoring server</td>
					<td>Routes firing alerts to mail, chat or paging.</td>
			</tr>
			<tr>
					<td>Grafana</td>
					<td>Monitoring server</td>
					<td>Dashboards that query Prometheus.</td>
			</tr>
	</tbody>
</table>
<p>The flow: HAProxy counts requests → Prometheus scrapes <code>/metrics</code> → rules fire alerts to Alertmanager → Grafana graphs the same series.</p>
<h2 id="haproxy-built-in-prometheus-support">HAProxy Built-in Prometheus Support <a class="anchor" data-pagefind-ignore href="#haproxy-built-in-prometheus-support" aria-label="Link to this section">#</a></h2>
<p>HAProxy 2.0+ includes native Prometheus metrics export via the built-in Prometheus exporter. In 2.0 to 2.3 it had to be compiled in (<code>USE_PROMEX=1</code>); since 2.4 it is part of every standard build, so there is nothing extra to install.</p>
<p>The examples in this post were checked against HAProxy 3.3. Install it from the official HAProxy packages rather than the distribution&rsquo;s older <code>haproxy</code> package:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Official HAProxy packages for Ubuntu 24.04: one PPA per branch</span>
</span></span><span class="line"><span class="cl">sudo add-apt-repository -y ppa:vbernat/haproxy-3.3
</span></span><span class="line"><span class="cl">sudo apt update
</span></span><span class="line"><span class="cl">sudo apt install -y <span class="nv">haproxy</span><span class="o">=</span>3.3.<span class="se">\*</span>
</span></span><span class="line"><span class="cl">haproxy -v
</span></span></code></pre></div><p>Which branch to pick:</p>
<table>
	<thead>
			<tr>
					<th>Branch</th>
					<th>Package source</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>3.3</td>
					<td><code>ppa:vbernat/haproxy-3.3</code> (Ubuntu 24.04)</td>
					<td>Newest features; short-lived, maintained until about Q1 2027</td>
			</tr>
			<tr>
					<td>3.2 LTS</td>
					<td><code>ppa:vbernat/haproxy-3.2</code> (Ubuntu 24.04)</td>
					<td>Long-lived installs, maintained until 2030</td>
			</tr>
			<tr>
					<td>3.4 LTS</td>
					<td><code>ppa:vbernat/haproxy-3.4</code> (Ubuntu 26.04), Debian 13 via <a href="https://haproxy.debian.net/" rel="noopener">haproxy.debian.net</a></td>
					<td>Newest LTS on newer releases; the site lists repository lines per release and branch</td>
			</tr>
	</tbody>
</table>
<h3 id="enable-prometheus-metrics-in-haproxy">Enable Prometheus Metrics in HAProxy <a class="anchor" data-pagefind-ignore href="#enable-prometheus-metrics-in-haproxy" aria-label="Link to this section">#</a></h3>
<p>The exporter is an HTTP service inside HAProxy. You give it its own frontend and port, so metrics stay off your application ports.</p>
<ul>
<li><code>frontend prometheus</code> on <code>:8405</code>: <code>http-request use-service prometheus-exporter</code> answers <code>/metrics</code> itself; no backend involved. <code>no log</code> keeps scrapes out of the access log.</li>
<li><code>frontend stats</code> on <code>:8404</code>: the human-readable stats page. <code>stats admin if LOCALHOST</code> allows enabling and disabling servers only from the host itself.</li>
<li><code>option httpchk GET /health</code> with <code>check</code>: health checks, which feed the server status and check metrics below.</li>
</ul>
<p>Add to <code>/etc/haproxy/haproxy.cfg</code>:</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

# Prometheus metrics frontend
frontend prometheus
    bind *:8405
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
    no log
    stats enable
    stats uri /stats
    stats refresh 10s

# Stats page for human viewing
frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

# Application frontends and backends
frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    server web1 192.168.1.10:8080 check
    server web2 192.168.1.11:8080 check
    server web3 192.168.1.12:8080 check
</code></pre><h3 id="verify-metrics-endpoint">Verify Metrics Endpoint <a class="anchor" data-pagefind-ignore href="#verify-metrics-endpoint" aria-label="Link to this section">#</a></h3>
<p>Reload HAProxy and fetch the endpoint. Every line is one series; <code># HELP</code> and <code># TYPE</code> describe it.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Test Prometheus metrics endpoint</span>
</span></span><span class="line"><span class="cl">curl http://localhost:8405/metrics
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Sample output:</span>
</span></span><span class="line"><span class="cl"><span class="c1"># HELP haproxy_process_nbthread Number of started threads (global.nbthread)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># TYPE haproxy_process_nbthread gauge</span>
</span></span><span class="line"><span class="cl">haproxy_process_nbthread <span class="m">4</span>
</span></span><span class="line"><span class="cl"><span class="c1"># HELP haproxy_process_current_connections Current number of connections on this worker process</span>
</span></span><span class="line"><span class="cl"><span class="c1"># TYPE haproxy_process_current_connections gauge</span>
</span></span><span class="line"><span class="cl">haproxy_process_current_connections <span class="m">42</span>
</span></span></code></pre></div><h2 id="prometheus-configuration">Prometheus Configuration <a class="anchor" data-pagefind-ignore href="#prometheus-configuration" aria-label="Link to this section">#</a></h2>
<h3 id="basic-prometheus-setup">Basic Prometheus Setup <a class="anchor" data-pagefind-ignore href="#basic-prometheus-setup" aria-label="Link to this section">#</a></h3>
<p>This tells Prometheus where HAProxy is, how often to scrape it, which rule files to load and where to send alerts.</p>
<ul>
<li><code>scrape_interval: 10s</code> for the <code>haproxy</code> job: finer than the 15s global default, so short spikes show up.</li>
<li><code>scrape_timeout: 5s</code>: must stay below the interval.</li>
<li><code>rule_files</code>: the alerting and recording rules created later in this post.</li>
<li><code>alertmanagers</code>: where firing alerts go. Leave it out if you only want dashboards.</li>
</ul>
<p>Create <code>/etc/prometheus/prometheus.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">global</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">evaluation_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">alerting</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">alertmanagers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="l">alertmanager:9093</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">rule_files</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;haproxy_alerts.yml&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;haproxy_rules.yml&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;prometheus&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;localhost:9090&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="s1">&#39;haproxy1.example.com:8405&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="s1">&#39;haproxy2.example.com:8405&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">metrics_path</span><span class="p">:</span><span class="w"> </span><span class="l">/metrics</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="l">10s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">scrape_timeout</span><span class="p">:</span><span class="w"> </span><span class="l">5s</span><span class="w">
</span></span></span></code></pre></div><h3 id="service-discovery-for-multiple-haproxy-instances">Service Discovery for Multiple HAProxy Instances <a class="anchor" data-pagefind-ignore href="#service-discovery-for-multiple-haproxy-instances" aria-label="Link to this section">#</a></h3>
<p>A static target list means editing <code>prometheus.yml</code> for every new load balancer. Two alternatives:</p>
<table>
	<thead>
			<tr>
					<th>Option</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>file_sd_configs</code></td>
					<td>A JSON file written by Ansible or a script. Prometheus rereads it, no restart.</td>
			</tr>
			<tr>
					<td><code>consul_sd_configs</code></td>
					<td>HAProxy nodes registered in Consul. New nodes are scraped automatically.</td>
			</tr>
	</tbody>
</table>
<p>With <code>file_sd_configs</code>, the <code>labels</code> (<code>env</code>, <code>dc</code>) are attached to every series from those targets, so you can filter and group by them in queries.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">file_sd_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">files</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="s1">&#39;/etc/prometheus/haproxy_targets.json&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">refresh_interval</span><span class="p">:</span><span class="w"> </span><span class="l">30s</span><span class="w">
</span></span></span></code></pre></div><p>Create <code>/etc/prometheus/haproxy_targets.json</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">[</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">      <span class="s2">&#34;haproxy1.example.com:8405&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="s2">&#34;haproxy2.example.com:8405&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;labels&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="s2">&#34;production&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;dc&#34;</span><span class="p">:</span> <span class="s2">&#34;dc1&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">      <span class="s2">&#34;haproxy-staging.example.com:8405&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;labels&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="s2">&#34;staging&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;dc&#34;</span><span class="p">:</span> <span class="s2">&#34;dc1&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span>
</span></span></code></pre></div><h3 id="consul-service-discovery">Consul Service Discovery <a class="anchor" data-pagefind-ignore href="#consul-service-discovery" aria-label="Link to this section">#</a></h3>
<p>Prometheus asks Consul for instances of the <code>haproxy</code> service. The <code>relabel_configs</code>:</p>
<ul>
<li><code>keep</code> on <code>__meta_consul_tags</code>: only scrape instances tagged <code>prometheus</code>.</li>
<li><code>job</code> and <code>instance</code>: set from the Consul service and node name, so series read <code>instance=&quot;lb1&quot;</code> instead of an IP and port.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">consul_sd_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">server</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;consul.example.com:8500&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">relabel_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__meta_consul_tags]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">regex</span><span class="p">:</span><span class="w"> </span><span class="l">.*,prometheus,.*</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">action</span><span class="p">:</span><span class="w"> </span><span class="l">keep</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__meta_consul_service]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">target_label</span><span class="p">:</span><span class="w"> </span><span class="l">job</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__meta_consul_node]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">target_label</span><span class="p">:</span><span class="w"> </span><span class="l">instance</span><span class="w">
</span></span></span></code></pre></div><h2 id="key-haproxy-metrics">Key HAProxy Metrics <a class="anchor" data-pagefind-ignore href="#key-haproxy-metrics" aria-label="Link to this section">#</a></h2>
<p>The exporter exposes hundreds of series. They come in four levels; the <code>proxy</code> label holds the frontend or backend name, <code>server</code> the server name.</p>
<table>
	<thead>
			<tr>
					<th>Level</th>
					<th>Prefix</th>
					<th>Answers</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Process</td>
					<td><code>haproxy_process_</code></td>
					<td>Is HAProxy itself busy or short on resources?</td>
			</tr>
			<tr>
					<td>Frontend</td>
					<td><code>haproxy_frontend_</code></td>
					<td>What are clients sending and getting back?</td>
			</tr>
			<tr>
					<td>Backend</td>
					<td><code>haproxy_backend_</code></td>
					<td>Is the application pool healthy and fast?</td>
			</tr>
			<tr>
					<td>Server</td>
					<td><code>haproxy_server_</code></td>
					<td>Which single server is the problem?</td>
			</tr>
	</tbody>
</table>
<p>Counters (<code>_total</code>) only grow; wrap them in <code>rate()</code> to get per-second values. Gauges (current sessions, queue) are used as they are.</p>
<h3 id="process-metrics">Process Metrics <a class="anchor" data-pagefind-ignore href="#process-metrics" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Current connections</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_current_connections</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_process_connections_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Memory usage</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_pool_allocated_bytes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_pool_used_bytes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># CPU headroom (HAProxy exports no CPU-seconds counters; low idle = busy)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_idle_time_percent</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Uptime</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_process_start_time_seconds</span><span class="w">
</span></span></span></code></pre></div><h3 id="frontend-metrics">Frontend Metrics <a class="anchor" data-pagefind-ignore href="#frontend-metrics" aria-label="Link to this section">#</a></h3>
<p>The client view. Response codes are grouped in a <code>code</code> label (<code>2xx</code>, <code>4xx</code>, <code>5xx</code>). A rise in 4xx points at clients (bad URLs, failed auth); a rise in 5xx at the backends.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Request rate per frontend</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Current sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_frontend_current_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Bytes in/out</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_in_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_out_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># HTTP response codes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">2xx</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">4xx</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">5xx</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Request errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_request_errors_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Denied requests</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_requests_denied_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h3 id="backend-metrics">Backend Metrics <a class="anchor" data-pagefind-ignore href="#backend-metrics" aria-label="Link to this section">#</a></h3>
<p>The pool view. The most useful early warnings:</p>
<ul>
<li><code>current_queue</code> above 0: all servers are at <code>maxconn</code>, requests wait.</li>
<li><code>active_servers</code> dropping: health checks are taking servers out.</li>
<li><code>response_time_average_seconds</code>: an average over recent requests, not a percentile.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Backend response time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_response_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_total_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Queue length</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_current_queue</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_max_queue</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Active servers</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_active_servers</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_backend_connection_errors_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Response errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_backend_response_errors_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Retry rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_backend_retry_warnings_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_current_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_max_sessions</span><span class="w">
</span></span></span></code></pre></div><h3 id="server-metrics">Server Metrics <a class="anchor" data-pagefind-ignore href="#server-metrics" aria-label="Link to this section">#</a></h3>
<p>Use these to find the one server behind a backend problem.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Server status: one series per state, value 1 on the current state</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_status</span><span class="p">{</span><span class="nl">state</span><span class="o">=</span><span class="p">&#34;</span><span class="s">UP</span><span class="p">&#34;}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Server weight</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_weight</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Current sessions per server</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_current_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_connect_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Response time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_response_time_average_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Health check status</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_check_status</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Failed health checks</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_server_check_failures_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h2 id="alerting-rules">Alerting Rules <a class="anchor" data-pagefind-ignore href="#alerting-rules" aria-label="Link to this section">#</a></h2>
<p>Alerts turn the metrics into notifications. <code>for:</code> is how long the condition must hold before the alert fires; it filters out short blips.</p>
<table>
	<thead>
			<tr>
					<th>Alert</th>
					<th>Fires when</th>
					<th>Severity</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>HAProxyHighErrorRate</code></td>
					<td>More than 5% of responses on a frontend are 5xx</td>
					<td>critical</td>
			</tr>
			<tr>
					<td><code>HAProxyBackendDown</code></td>
					<td>A backend has no active servers</td>
					<td>critical</td>
			</tr>
			<tr>
					<td><code>HAProxyDown</code></td>
					<td>Prometheus cannot scrape an HAProxy</td>
					<td>critical</td>
			</tr>
			<tr>
					<td><code>HAProxyServerDown</code></td>
					<td>A single server is DOWN</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyHighConnections</code></td>
					<td>A frontend uses over 80% of its session limit</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyQueueNotEmpty</code></td>
					<td>More than 10 requests queued on a backend</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyHighResponseTime</code></td>
					<td>Average backend response time over 2s</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyHealthCheckFailures</code></td>
					<td>More than 5 failed checks on a server in 10 minutes</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxyConnectionErrors</code></td>
					<td>More than 10 connection errors per second to a backend</td>
					<td>warning</td>
			</tr>
			<tr>
					<td><code>HAProxySSLCertExpiringSoon</code></td>
					<td>A certificate expires within 7 days (needs blackbox_exporter)</td>
					<td>warning</td>
			</tr>
	</tbody>
</table>
<p>Tune the thresholds to your traffic. Create <code>/etc/prometheus/haproxy_alerts.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># High error rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHighErrorRate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total{code=&#34;5xx&#34;}[5m])) by (proxy)
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total[5m])) by (proxy)
</span></span></span><span class="line"><span class="cl"><span class="sd">          &gt; 0.05</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High 5xx error rate on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Error rate is {{ $value | humanizePercentage }} on frontend {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Backend down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyBackendDown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_backend_active_servers == 0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;HAProxy backend {{ $labels.proxy }} has no active servers&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;All servers in backend {{ $labels.proxy }} are down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Server down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyServerDown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_server_status{state=&#34;DOWN&#34;} == 1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">2m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Server {{ $labels.server }} is down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Server {{ $labels.server }} in backend {{ $labels.proxy }} is down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># High connection count</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHighConnections</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_current_sessions
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_limit_sessions
</span></span></span><span class="line"><span class="cl"><span class="sd">          &gt; 0.8</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High connection usage on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Frontend {{ $labels.proxy }} is at {{ $value | humanizePercentage }} of max connections&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Queue building up</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyQueueNotEmpty</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_backend_current_queue &gt; 10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">2m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Queue building up on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Backend {{ $labels.proxy }} has {{ $value }} requests queued&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># High response time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHighResponseTime</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_backend_response_time_average_seconds &gt; 2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High response time on {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Backend {{ $labels.proxy }} average response time is {{ $value }}s&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Health check failures</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyHealthCheckFailures</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">increase(haproxy_server_check_failures_total[10m]) &gt; 5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Health check failures on {{ $labels.server }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Server {{ $labels.server }} has {{ $value }} health check failures&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Connection errors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyConnectionErrors</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">rate(haproxy_backend_connection_errors_total[5m]) &gt; 10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Connection errors to {{ $labels.proxy }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Backend {{ $labels.proxy }} has {{ $value }}/s connection errors&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># HAProxy instance down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxyDown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">up{job=&#34;haproxy&#34;} == 0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;HAProxy instance {{ $labels.instance }} is down&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;HAProxy at {{ $labels.instance }} is not responding&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># SSL certificate expiry (HAProxy does not export certificate dates;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># this uses blackbox_exporter probing the HTTPS frontend)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HAProxySSLCertExpiringSoon</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">probe_ssl_earliest_cert_expiry - time() &lt; 7 * 24 * 3600</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1h</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;SSL certificate expiring soon&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Certificate expires in {{ $value | humanizeDuration }}&#34;</span><span class="w">
</span></span></span></code></pre></div><h2 id="grafana-dashboards">Grafana Dashboards <a class="anchor" data-pagefind-ignore href="#grafana-dashboards" aria-label="Link to this section">#</a></h2>
<p>Grafana reads the same series from Prometheus. Add Prometheus as a data source, then build panels from the queries below.</p>
<h3 id="haproxy-overview-dashboard-json">HAProxy Overview Dashboard JSON <a class="anchor" data-pagefind-ignore href="#haproxy-overview-dashboard-json" aria-label="Link to this section">#</a></h3>
<p>A minimal skeleton with five panels: request rate, response codes, backend response time, active connections and server health.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;HAProxy Overview&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;panels&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Request Rate&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;graph&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(haproxy_frontend_http_requests_total[5m])) by (proxy)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{ proxy }}&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;HTTP Response Codes&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;graph&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(haproxy_frontend_http_responses_total[5m])) by (code)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{ code }}&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Backend Response Time&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;graph&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;haproxy_backend_response_time_average_seconds&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{ proxy }}&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Active Connections&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;gauge&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(haproxy_frontend_current_sessions)&#34;</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Server Health&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;table&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;haproxy_server_status == 1&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;format&#34;</span><span class="p">:</span> <span class="s2">&#34;table&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">          <span class="nt">&#34;instant&#34;</span><span class="p">:</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">      <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><h3 id="key-dashboard-panels">Key Dashboard Panels <a class="anchor" data-pagefind-ignore href="#key-dashboard-panels" aria-label="Link to this section">#</a></h3>
<p>More queries for panels, grouped by the question they answer.</p>
<p><strong>Traffic Overview:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Total requests per second</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Bandwidth in/out</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_in_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="w">  </span><span class="c1"># bits/sec</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_bytes_out_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="w">
</span></span></span></code></pre></div><p><strong>Error Analysis:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Error rate percentage</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">100</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=~</span><span class="p">&#34;</span><span class="s">5..</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># 4xx rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">100</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=~</span><span class="p">&#34;</span><span class="s">4..</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span></code></pre></div><p><strong>Backend Performance:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Worst observed response time (HAProxy exports averages and maxima, no histogram)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_max_response_time_seconds</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Queue time</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_backend_queue_time_average_seconds</span><span class="w">
</span></span></span></code></pre></div><p><strong>Server Status:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Server availability</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="nv">haproxy_server_status</span><span class="p">{</span><span class="nl">state</span><span class="o">=</span><span class="p">&#34;</span><span class="s">UP</span><span class="p">&#34;}</span><span class="o">)</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">proxy</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">count</span><span class="o">(</span><span class="nv">haproxy_server_status</span><span class="p">{</span><span class="nl">state</span><span class="o">=</span><span class="p">&#34;</span><span class="s">UP</span><span class="p">&#34;}</span><span class="o">)</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">proxy</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Server weight distribution</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">haproxy_server_weight</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="k">on</span><span class="o">(</span><span class="nv">proxy</span><span class="p">,</span><span class="w"> </span><span class="nv">instance</span><span class="o">)</span><span class="w"> </span><span class="k">group_left</span><span class="w"> </span><span class="k">sum</span><span class="o">(</span><span class="nv">haproxy_server_weight</span><span class="o">)</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">proxy</span><span class="p">,</span><span class="w"> </span><span class="nv">instance</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h2 id="recording-rules">Recording Rules <a class="anchor" data-pagefind-ignore href="#recording-rules" aria-label="Link to this section">#</a></h2>
<p>A recording rule computes a query on every evaluation and stores the result as a new series. Use them for queries that dashboards and alerts run often: they load faster and are reused by name (<code>haproxy:frontend:error_rate:5m</code>).</p>
<p>Create <code>/etc/prometheus/haproxy_rules.yml</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy_recording</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Request rate by frontend</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:request_rate:5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">sum(rate(haproxy_frontend_http_requests_total[5m])) by (proxy, instance)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Error rate by frontend</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:error_rate:5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total{code=&#34;5xx&#34;}[5m])) by (proxy, instance)
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_http_responses_total[5m])) by (proxy, instance)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Backend availability</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:backend:availability</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(haproxy_server_status{state=&#34;UP&#34;}) by (proxy, instance)
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          count(haproxy_server_status{state=&#34;UP&#34;}) by (proxy, instance)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Session utilization</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:session_utilization</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_current_sessions
</span></span></span><span class="line"><span class="cl"><span class="sd">          /
</span></span></span><span class="line"><span class="cl"><span class="sd">          haproxy_frontend_limit_sessions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Bytes total rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">haproxy:frontend:bytes_total_rate:5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_bytes_in_total[5m])) by (proxy, instance)
</span></span></span><span class="line"><span class="cl"><span class="sd">          +
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(haproxy_frontend_bytes_out_total[5m])) by (proxy, instance)</span><span class="w">
</span></span></span></code></pre></div><h2 id="haproxy-exporter-alternative">HAProxy Exporter (Alternative) <a class="anchor" data-pagefind-ignore href="#haproxy-exporter-alternative" aria-label="Link to this section">#</a></h2>
<p>For HAProxy versions without native Prometheus support, use the separate HAProxy Exporter. It reads the stats socket and translates it to Prometheus format.</p>
<table>
	<thead>
			<tr>
					<th></th>
					<th>Built-in exporter</th>
					<th>haproxy_exporter</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>HAProxy version</td>
					<td>2.0+ (standard since 2.4)</td>
					<td>Any</td>
			</tr>
			<tr>
					<td>Extra process</td>
					<td>No</td>
					<td>Yes, a systemd service</td>
			</tr>
			<tr>
					<td>Port</td>
					<td>Your choice (8405 here)</td>
					<td>9101</td>
			</tr>
			<tr>
					<td>Use it for</td>
					<td>Everything current</td>
					<td>Old HAProxy you cannot upgrade</td>
			</tr>
	</tbody>
</table>
<h3 id="installation">Installation <a class="anchor" data-pagefind-ignore href="#installation" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Download HAProxy Exporter</span>
</span></span><span class="line"><span class="cl"><span class="nv">EXPORTER_VERSION</span><span class="o">=</span><span class="s2">&#34;0.15.0&#34;</span>
</span></span><span class="line"><span class="cl">wget https://github.com/prometheus/haproxy_exporter/releases/download/v<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>/haproxy_exporter-<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>.linux-amd64.tar.gz
</span></span><span class="line"><span class="cl">tar xzf haproxy_exporter-<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>.linux-amd64.tar.gz
</span></span><span class="line"><span class="cl">sudo mv haproxy_exporter-<span class="si">${</span><span class="nv">EXPORTER_VERSION</span><span class="si">}</span>.linux-amd64/haproxy_exporter /usr/local/bin/
</span></span></code></pre></div><h3 id="haproxy-stats-socket-configuration">HAProxy Stats Socket Configuration <a class="anchor" data-pagefind-ignore href="#haproxy-stats-socket-configuration" aria-label="Link to this section">#</a></h3>
<p>The exporter needs a stats socket. The service below uses the Unix socket; the TCP socket on <code>127.0.0.1:9999</code> is an alternative for an exporter that cannot reach the socket file.</p>
<pre tabindex="0"><code class="language-haproxy" data-lang="haproxy">global
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats socket ipv4@127.0.0.1:9999 level admin
    stats timeout 30s
</code></pre><h3 id="exporter-systemd-service">Exporter Systemd Service <a class="anchor" data-pagefind-ignore href="#exporter-systemd-service" aria-label="Link to this section">#</a></h3>
<p>It runs as the <code>haproxy</code> user so it can open the socket (mode 660). Create <code>/etc/systemd/system/haproxy_exporter.service</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">HAProxy Exporter</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network.target haproxy.service</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">simple</span>
</span></span><span class="line"><span class="cl"><span class="na">User</span><span class="o">=</span><span class="s">haproxy</span>
</span></span><span class="line"><span class="cl"><span class="na">Group</span><span class="o">=</span><span class="s">haproxy</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/haproxy_exporter \
</span></span></span><span class="line"><span class="cl"><span class="s">  --haproxy.scrape-uri=&#34;unix:/run/haproxy/admin.sock&#34; \
</span></span></span><span class="line"><span class="cl"><span class="s">  --web.listen-address=&#34;:9101&#34;</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
</span></span><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
</span></span></code></pre></div><h3 id="prometheus-configuration-for-exporter">Prometheus Configuration for Exporter <a class="anchor" data-pagefind-ignore href="#prometheus-configuration-for-exporter" aria-label="Link to this section">#</a></h3>
<p>Point the <code>haproxy</code> job at port 9101 instead of 8405.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;haproxy&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;haproxy1.example.com:9101&#39;</span><span class="p">]</span><span class="w">
</span></span></span></code></pre></div><h2 id="advanced-monitoring-patterns">Advanced Monitoring Patterns <a class="anchor" data-pagefind-ignore href="#advanced-monitoring-patterns" aria-label="Link to this section">#</a></h2>
<h3 id="multi-instance-aggregation">Multi-Instance Aggregation <a class="anchor" data-pagefind-ignore href="#multi-instance-aggregation" aria-label="Link to this section">#</a></h3>
<p>With several load balancers, sum across instances for the total, or group by the <code>dc</code> label from file discovery.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Total request rate across all instances</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Per-datacenter metrics</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">dc</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Highest error rate instance</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">topk</span><span class="o">(</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="nv">haproxy</span><span class="err">:</span><span class="nv">frontend</span><span class="err">:</span><span class="nv">error_rate</span><span class="err">:</span><span class="s">5m</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h3 id="slo-monitoring">SLO Monitoring <a class="anchor" data-pagefind-ignore href="#slo-monitoring" aria-label="Link to this section">#</a></h3>
<p>An SLO of 99.9% allows 0.1% of requests to fail over 30 days. The first query checks whether you meet it; the second shows how much of that 0.1% budget is left (1 = untouched, below 0 = SLO missed).</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Availability SLO (99.9%)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">5xx</span><span class="p">&#34;}[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">)</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="mf">0.999</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Error budget remaining</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">sum</span><span class="o">(</span><span class="kr">increase</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">{</span><span class="nl">code</span><span class="o">=</span><span class="p">&#34;</span><span class="s">5xx</span><span class="p">&#34;}[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="o">(</span><span class="k">sum</span><span class="o">(</span><span class="kr">increase</span><span class="o">(</span><span class="nv">haproxy_frontend_http_responses_total</span><span class="p">[</span><span class="s">30d</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mf">0.001</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h3 id="capacity-planning">Capacity Planning <a class="anchor" data-pagefind-ignore href="#capacity-planning" aria-label="Link to this section">#</a></h3>
<ul>
<li><code>predict_linear</code>: where sessions will be in an hour if the last hour&rsquo;s trend continues.</li>
<li><code>deriv</code>: how fast sessions grow per second.</li>
<li><code>max_over_time</code>: the peak, to compare against <code>maxconn</code>.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Predict session exhaustion</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">predict_linear</span><span class="o">(</span><span class="nv">haproxy_frontend_current_sessions</span><span class="p">[</span><span class="s">1h</span><span class="p">],</span><span class="w"> </span><span class="mi">3600</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Connection growth rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">deriv</span><span class="o">(</span><span class="nv">haproxy_frontend_current_sessions</span><span class="p">[</span><span class="s">30m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Peak hour comparison</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">max_over_time</span><span class="o">(</span><span class="nv">haproxy_frontend_current_sessions</span><span class="p">[</span><span class="s">1h</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h2 id="troubleshooting">Troubleshooting <a class="anchor" data-pagefind-ignore href="#troubleshooting" aria-label="Link to this section">#</a></h2>
<h3 id="verify-metrics-collection">Verify Metrics Collection <a class="anchor" data-pagefind-ignore href="#verify-metrics-collection" aria-label="Link to this section">#</a></h3>
<p>Ask the Prometheus API whether the HAProxy targets are up and returning data.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Check if Prometheus can reach HAProxy</span>
</span></span><span class="line"><span class="cl">curl -s http://prometheus:9090/api/v1/targets <span class="p">|</span> jq <span class="s1">&#39;.data.activeTargets[] | select(.job==&#34;haproxy&#34;)&#39;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Query current metrics</span>
</span></span><span class="line"><span class="cl">curl -s <span class="s2">&#34;http://prometheus:9090/api/v1/query?query=haproxy_process_current_connections&#34;</span> <span class="p">|</span> jq .
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Check for scrape errors</span>
</span></span><span class="line"><span class="cl">curl -s http://prometheus:9090/api/v1/targets <span class="p">|</span> jq <span class="s1">&#39;.data.activeTargets[] | select(.health!=&#34;up&#34;)&#39;</span>
</span></span></code></pre></div><h3 id="common-issues">Common Issues <a class="anchor" data-pagefind-ignore href="#common-issues" aria-label="Link to this section">#</a></h3>
<table>
	<thead>
			<tr>
					<th>Symptom</th>
					<th>Cause / fix</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Metrics not appearing</td>
					<td>HAProxy older than 2.4 without <code>USE_PROMEX</code>; port 8405 blocked by a firewall; <code>use-service prometheus-exporter</code> missing</td>
			</tr>
			<tr>
					<td>Prometheus slow, many series</td>
					<td>High cardinality: limit label values in relabeling, use recording rules for aggregations</td>
			</tr>
			<tr>
					<td>Stale metrics</td>
					<td>Check the scrape interval; check that HAProxy still answers on <code>/metrics</code></td>
			</tr>
	</tbody>
</table>
<h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Restrict ports 8404 and 8405.</strong> The configuration binds them on all interfaces; allow only the Prometheus server in the firewall.</li>
<li><strong>Stats admin stays local.</strong> <code>stats admin if LOCALHOST</code> stops remote users from disabling servers.</li>
<li><strong>Admin sockets.</strong> <code>level admin</code> on the stats sockets allows changing HAProxy at runtime; keep the TCP socket on <code>127.0.0.1</code>.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub:
<a href="https://github.com/pderuiter/blog-haproxy-monitoring" rel="noopener">pderuiter/blog-haproxy-monitoring</a>. It creates an HAProxy host with the built-in exporter and a Prometheus server that scrapes it and loads the alerting and recording rules from this post; run <code>terraform apply</code> for the VMs and <code>ansible-playbook site.yml</code> for the rest.</p>
<h2 id="summary">Summary <a class="anchor" data-pagefind-ignore href="#summary" aria-label="Link to this section">#</a></h2>
<p>With Prometheus monitoring configured, you have:</p>
<ul>
<li>Real-time visibility into HAProxy performance</li>
<li>Historical data for trend analysis and capacity planning</li>
<li>Alerts before issues reach users</li>
<li>Grafana dashboards for operations teams</li>
</ul>
<p>This completes the HAProxy monitoring series. Together with the posts on load balancing, mTLS, the Data Plane API and Consul integration, it covers building and operating HAProxy in production.</p>
]]></content:encoded></item><item><title>Monitoring BIND DNS with Prometheus and Grafana</title><link>https://blog.bsdserver.nl/monitoring-bind-dns-with-prometheus-and-grafana/</link><guid isPermaLink="false">693b6f7d79eb54000160a5c8</guid><dc:creator>Patrick de Ruiter</dc:creator><pubDate>Fri, 12 Dec 2025 01:27:25 +0000</pubDate><description>A well-monitored DNS infrastructure provides visibility into query patterns, performance bottlenecks, and potential security issues.</description><category>BIND</category><category>DNS</category><category>Prometheus</category><category>Grafana</category><category>Monitoring</category><category>Observability</category><category>Homelab</category><category>Linux</category><media:content url="https://blog.bsdserver.nl/content/images/2026/01/monitoring-bind-dns-with-prometheus.png" medium="image"/><content:encoded><![CDATA[<p>A well-monitored DNS infrastructure provides visibility into query patterns, performance bottlenecks, and potential security issues. Without it, you hear about DNS problems from your users.</p>
<p>This final post in the BIND series sets up monitoring with Prometheus and Grafana: an exporter on every BIND server, dashboards, alerts, and query logs in Loki. It ends with a Docker Compose stack that runs it all on the DNS server.</p>
<h2 id="why-prometheus-and-grafana-for-bind">Why Prometheus and Grafana for BIND? <a class="anchor" data-pagefind-ignore href="#why-prometheus-and-grafana-for-bind" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Metrics without parsing logs</strong>: BIND&rsquo;s statistics channel already counts queries, responses and resolver activity.</li>
<li><strong>Alerting</strong>: Prometheus rules catch SERVFAIL spikes, failed zone transfers and floods.</li>
<li><strong>Dashboards</strong>: a ready-made Grafana dashboard, plus panels you can build yourself.</li>
<li><strong>One stack</strong>: if you already run Prometheus for other services, BIND is one more scrape target.</li>
</ul>
<h3 id="when-to-use-it">When to use it <a class="anchor" data-pagefind-ignore href="#when-to-use-it" aria-label="Link to this section">#</a></h3>
<ul>
<li><strong>Use it</strong> for any BIND server others depend on, especially resolvers and slaves.</li>
<li><strong>Skip the full stack</strong> for a single lab resolver. <code>rndc stats</code> or a look at the statistics channel may be enough.</li>
</ul>
<h3 id="bind-statistics-sources">BIND statistics sources <a class="anchor" data-pagefind-ignore href="#bind-statistics-sources" aria-label="Link to this section">#</a></h3>
<p>BIND provides extensive statistics through several channels:</p>
<table>
	<thead>
			<tr>
					<th>Source</th>
					<th>What you get</th>
					<th>Use it for</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Statistics channel</td>
					<td>XML/JSON HTTP endpoint with counters</td>
					<td>Prometheus, via bind_exporter</td>
			</tr>
			<tr>
					<td>Query logging</td>
					<td>Detailed query/response logs</td>
					<td>Per-domain and per-client analysis in Loki</td>
			</tr>
			<tr>
					<td>RNDC statistics</td>
					<td>Command-line statistics dump</td>
					<td>Ad-hoc checks</td>
			</tr>
	</tbody>
</table>
<h2 id="architecture-overview">Architecture Overview <a class="anchor" data-pagefind-ignore href="#architecture-overview" aria-label="Link to this section">#</a></h2>
<table>
	<thead>
			<tr>
					<th>Component</th>
					<th>Runs on</th>
					<th>Job</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>BIND statistics channel</td>
					<td>Every DNS server, <code>127.0.0.1:8053</code></td>
					<td>Serves BIND&rsquo;s counters over HTTP.</td>
			</tr>
			<tr>
					<td>bind_exporter</td>
					<td>Every DNS server, port 9119</td>
					<td>Reads the statistics channel and turns it into Prometheus metrics.</td>
			</tr>
			<tr>
					<td>Prometheus</td>
					<td>Monitoring host (or the DNS server, with Compose)</td>
					<td>Scrapes the exporters, stores metrics, evaluates alert rules.</td>
			</tr>
			<tr>
					<td>Alertmanager</td>
					<td>Next to Prometheus</td>
					<td>Groups alerts and sends notifications.</td>
			</tr>
			<tr>
					<td>Grafana</td>
					<td>Next to Prometheus</td>
					<td>Dashboards.</td>
			</tr>
			<tr>
					<td>Promtail + Loki (optional)</td>
					<td>DNS server, log host</td>
					<td>Ship and store the query log.</td>
			</tr>
	</tbody>
</table>
<p>The flow: BIND → statistics channel → bind_exporter → Prometheus → Grafana dashboards and Alertmanager.</p>
<h2 id="enabling-bind-statistics-channel">Enabling BIND Statistics Channel <a class="anchor" data-pagefind-ignore href="#enabling-bind-statistics-channel" aria-label="Link to this section">#</a></h2>
<h3 id="configure-statistics-in-bind">Configure Statistics in BIND <a class="anchor" data-pagefind-ignore href="#configure-statistics-in-bind" aria-label="Link to this section">#</a></h3>
<p>The statistics channel is a small HTTP server inside BIND. Keep it on localhost: the exporter runs on the same machine, and the statistics reveal a lot about your server.</p>
<ul>
<li><code>statistics-channels</code>: where the endpoint listens, and who may connect (<code>allow { localhost; }</code>).</li>
<li><code>zone-statistics full</code>: also count queries per zone.</li>
</ul>
<pre tabindex="0"><code class="language-named" data-lang="named">// /etc/bind/named.conf.options

// Enable statistics channel (a top-level statement, not an option)
statistics-channels {
    inet 127.0.0.1 port 8053 allow { localhost; };
    inet ::1 port 8053 allow { localhost; };
};

options {
    directory &#34;/var/cache/bind&#34;;
    
    // Enable zone statistics
    zone-statistics full;
    
    // Other options...
    recursion yes;
    dnssec-validation auto;
};
</code></pre><h3 id="verify-statistics-channel">Verify Statistics Channel <a class="anchor" data-pagefind-ignore href="#verify-statistics-channel" aria-label="Link to this section">#</a></h3>
<p>Restart BIND and check that the endpoint answers before you install the exporter:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Restart BIND</span>
</span></span><span class="line"><span class="cl">systemctl restart named
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test statistics endpoint (XML)</span>
</span></span><span class="line"><span class="cl">curl http://127.0.0.1:8053/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Get XML statistics</span>
</span></span><span class="line"><span class="cl">curl http://127.0.0.1:8053/xml/v3/server
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Get JSON statistics (BIND 9.10+)</span>
</span></span><span class="line"><span class="cl">curl http://127.0.0.1:8053/json/v1/server
</span></span></code></pre></div><h2 id="installing-bind_exporter">Installing bind_exporter <a class="anchor" data-pagefind-ignore href="#installing-bind_exporter" aria-label="Link to this section">#</a></h2>
<p>The bind_exporter converts BIND statistics to Prometheus metrics format. Install it on every BIND server.</p>
<h3 id="download-and-install">Download and Install <a class="anchor" data-pagefind-ignore href="#download-and-install" aria-label="Link to this section">#</a></h3>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Download latest release</span>
</span></span><span class="line"><span class="cl">wget https://github.com/prometheus-community/bind_exporter/releases/download/v0.7.0/bind_exporter-0.7.0.linux-amd64.tar.gz
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Extract</span>
</span></span><span class="line"><span class="cl">tar xzf bind_exporter-0.7.0.linux-amd64.tar.gz
</span></span><span class="line"><span class="cl">mv bind_exporter-0.7.0.linux-amd64/bind_exporter /usr/local/bin/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Set permissions</span>
</span></span><span class="line"><span class="cl">chmod +x /usr/local/bin/bind_exporter
</span></span></code></pre></div><h3 id="create-systemd-service">Create Systemd Service <a class="anchor" data-pagefind-ignore href="#create-systemd-service" aria-label="Link to this section">#</a></h3>
<p>The exporter needs no privileges, so it runs as <code>nobody</code>.</p>
<ul>
<li><code>--bind.stats-url</code>: the local statistics channel.</li>
<li><code>--bind.stats-groups</code>: which parts of the statistics to export (table below).</li>
<li><code>--web.listen-address=:9119</code>: where Prometheus scrapes it. It listens on all interfaces; firewall it to your Prometheus host.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># /etc/systemd/system/bind_exporter.service</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Unit]</span>
</span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">BIND Exporter for Prometheus</span>
</span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">network.target named.service</span>
</span></span><span class="line"><span class="cl"><span class="na">Wants</span><span class="o">=</span><span class="s">named.service</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Service]</span>
</span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">simple</span>
</span></span><span class="line"><span class="cl"><span class="na">User</span><span class="o">=</span><span class="s">nobody</span>
</span></span><span class="line"><span class="cl"><span class="na">Group</span><span class="o">=</span><span class="s">nogroup</span>
</span></span><span class="line"><span class="cl"><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/bind_exporter \
</span></span></span><span class="line"><span class="cl"><span class="s">    --bind.stats-url=http://127.0.0.1:8053/ \
</span></span></span><span class="line"><span class="cl"><span class="s">    --bind.stats-groups=server,view,tasks \
</span></span></span><span class="line"><span class="cl"><span class="s">    --web.listen-address=:9119</span>
</span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">always</span>
</span></span><span class="line"><span class="cl"><span class="na">RestartSec</span><span class="o">=</span><span class="s">5</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Install]</span>
</span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
</span></span></code></pre></div><p>Enable it and check the metrics:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">systemctl daemon-reload
</span></span><span class="line"><span class="cl">systemctl <span class="nb">enable</span> bind_exporter
</span></span><span class="line"><span class="cl">systemctl start bind_exporter
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Test metrics endpoint</span>
</span></span><span class="line"><span class="cl">curl http://localhost:9119/metrics
</span></span></code></pre></div><h3 id="available-statistics-groups">Available Statistics Groups <a class="anchor" data-pagefind-ignore href="#available-statistics-groups" aria-label="Link to this section">#</a></h3>
<p>The exporter supports these statistics groups:</p>
<table>
	<thead>
			<tr>
					<th>Group</th>
					<th>Description</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>server</td>
					<td>Overall server statistics</td>
			</tr>
			<tr>
					<td>view</td>
					<td>Per-view statistics</td>
			</tr>
			<tr>
					<td>tasks</td>
					<td>Task manager statistics</td>
			</tr>
	</tbody>
</table>
<h2 id="prometheus-configuration">Prometheus Configuration <a class="anchor" data-pagefind-ignore href="#prometheus-configuration" aria-label="Link to this section">#</a></h2>
<h3 id="add-scrape-target">Add Scrape Target <a class="anchor" data-pagefind-ignore href="#add-scrape-target" aria-label="Link to this section">#</a></h3>
<p>Add one target per DNS server. The <code>relabel_configs</code> strip the port from the <code>instance</code> label, so graphs show <code>dns1.example.com</code> instead of <code>dns1.example.com:9119</code>. Leave it out if you use dashboard 12309 (see below).</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># /etc/prometheus/prometheus.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">global</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">evaluation_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;prometheus&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;localhost:9090&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;bind&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="s1">&#39;dns1.example.com:9119&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="s1">&#39;dns2.example.com:9119&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">relabel_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__address__]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">target_label</span><span class="p">:</span><span class="w"> </span><span class="l">instance</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">regex</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;([^:]+).*&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">replacement</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;${1}&#39;</span><span class="w">
</span></span></span></code></pre></div><p>bind_exporter has no multi-target mode: it always scrapes the one <code>--bind.stats-url</code> it was started with and ignores a <code>?target=</code> parameter. Run one exporter per BIND server (as above), since the statistics channel should only listen on localhost anyway.</p>
<h3 id="reload-prometheus">Reload Prometheus <a class="anchor" data-pagefind-ignore href="#reload-prometheus" aria-label="Link to this section">#</a></h3>
<p>Either command makes Prometheus read the new configuration without a restart:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nb">kill</span> -HUP <span class="k">$(</span>pidof prometheus<span class="k">)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Or</span>
</span></span><span class="line"><span class="cl">systemctl reload prometheus
</span></span></code></pre></div><h2 id="key-metrics-to-monitor">Key Metrics to Monitor <a class="anchor" data-pagefind-ignore href="#key-metrics-to-monitor" aria-label="Link to this section">#</a></h2>
<p>Most BIND metrics are counters, so wrap them in <code>rate()</code> or <code>increase()</code> to get something you can read.</p>
<h3 id="query-metrics">Query Metrics <a class="anchor" data-pagefind-ignore href="#query-metrics" aria-label="Link to this section">#</a></h3>
<p>How much traffic, of which type, and how it was answered. Note the two response metrics: <code>bind_responses_total</code> groups by result (NOERROR is &ldquo;Success&rdquo;), <code>bind_response_rcodes_total</code> by real RCODE.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Total queries received</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_incoming_queries_total</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Queries by type (A, AAAA, MX, etc.)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_incoming_queries_total</span><span class="p">{</span><span class="nl">type</span><span class="o">=</span><span class="p">&#34;</span><span class="s">A</span><span class="p">&#34;}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Query rate (queries per second)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">bind_incoming_queries_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Queries by result (NOERROR answers are counted as &#34;Success&#34;)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_responses_total</span><span class="p">{</span><span class="nl">result</span><span class="o">=</span><span class="p">&#34;</span><span class="s">Success</span><span class="p">&#34;}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_responses_total</span><span class="p">{</span><span class="nl">result</span><span class="o">=</span><span class="p">&#34;</span><span class="s">NXDOMAIN</span><span class="p">&#34;}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_responses_total</span><span class="p">{</span><span class="nl">result</span><span class="o">=</span><span class="p">&#34;</span><span class="s">SERVFAIL</span><span class="p">&#34;}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Responses by real RCODE</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_response_rcodes_total</span><span class="p">{</span><span class="nl">rcode</span><span class="o">=</span><span class="p">&#34;</span><span class="s">NOERROR</span><span class="p">&#34;}</span><span class="w">
</span></span></span></code></pre></div><h3 id="resolver-metrics">Resolver Metrics <a class="anchor" data-pagefind-ignore href="#resolver-metrics" aria-label="Link to this section">#</a></h3>
<p>Only relevant on resolvers. A falling cache hit rate or rising upstream latency means slower answers for your clients.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Approximate cache hit rate: share of client queries that needed no recursion</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># (bind_exporter exposes no direct cache hit/miss counters)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">(</span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">bind_query_recursions_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">instance</span><span class="o">)</span><span class="w"> </span><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">     </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">bind_incoming_queries_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">instance</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Recursive queries sent</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">rate</span><span class="o">(</span><span class="nv">bind_resolver_queries_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Response time distribution</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_resolver_query_duration_seconds_bucket</span><span class="w">
</span></span></span></code></pre></div><h3 id="server-health-metrics">Server Health Metrics <a class="anchor" data-pagefind-ignore href="#server-health-metrics" aria-label="Link to this section">#</a></h3>
<ul>
<li><code>bind_recursive_clients</code>: recursive queries in progress. Near <code>recursive-clients</code> (default 1000) BIND starts dropping queries.</li>
<li>Zone transfer counters: on slaves, failures mean stale zones.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Current number of recursive clients (exporter 0.8.0 omits it while it is 0)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_recursive_clients</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Zone transfers</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_zone_transfer_success_total</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nv">bind_zone_transfer_failure_total</span><span class="w">
</span></span></span></code></pre></div><h2 id="grafana-dashboard">Grafana Dashboard <a class="anchor" data-pagefind-ignore href="#grafana-dashboard" aria-label="Link to this section">#</a></h2>
<h3 id="import-pre-built-dashboard">Import Pre-built Dashboard <a class="anchor" data-pagefind-ignore href="#import-pre-built-dashboard" aria-label="Link to this section">#</a></h3>
<p>The fastest start. The BIND Exporter community provides dashboards:</p>
<ol>
<li>In Grafana, go to Dashboards -&gt; Import</li>
<li>Enter dashboard ID: <strong>12309</strong> (BIND Exporter)</li>
<li>Select your Prometheus data source</li>
<li>Click Import</li>
</ol>
<p>Dashboard 12309 selects series with <code>instance=&quot;$node:$port&quot;</code>, so it expects the default <code>host:port</code> instance label. If you use the <code>relabel_configs</code> above that strip the port, its panels stay empty; leave the relabeling out for this dashboard. Its panels use the legacy graph type, which current Grafana converts automatically on load.</p>
<h3 id="custom-dashboard-panels">Custom Dashboard Panels <a class="anchor" data-pagefind-ignore href="#custom-dashboard-panels" aria-label="Link to this section">#</a></h3>
<p>Build your own panels for what matters to you. Each JSON snippet below is one panel; paste it via a panel&rsquo;s JSON view, or copy the <code>expr</code> into a new panel.</p>
<table>
	<thead>
			<tr>
					<th>Panel</th>
					<th>Type</th>
					<th>Shows</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Query Rate</td>
					<td>Time series</td>
					<td>Queries per second per server</td>
			</tr>
			<tr>
					<td>Query Type Distribution</td>
					<td>Pie chart</td>
					<td>Share of A, AAAA, MX, … over the last hour</td>
			</tr>
			<tr>
					<td>Response Codes</td>
					<td>Stat</td>
					<td>Rate of NOERROR, NXDOMAIN and SERVFAIL</td>
			</tr>
			<tr>
					<td>Cache Hit Rate</td>
					<td>Gauge</td>
					<td>Approximate cache hit rate, red below 70%, green above 90%</td>
			</tr>
	</tbody>
</table>
<h4 id="query-rate-panel">Query Rate Panel <a class="anchor" data-pagefind-ignore href="#query-rate-panel" aria-label="Link to this section">#</a></h4>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;DNS Query Rate&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;timeseries&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(bind_incoming_queries_total[5m])) by (instance)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{instance}}&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">],</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;fieldConfig&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;defaults&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;unit&#34;</span><span class="p">:</span> <span class="s2">&#34;reqps&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;custom&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;axisLabel&#34;</span><span class="p">:</span> <span class="s2">&#34;Queries/sec&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><h4 id="query-type-distribution">Query Type Distribution <a class="anchor" data-pagefind-ignore href="#query-type-distribution" aria-label="Link to this section">#</a></h4>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Query Types&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;piechart&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(increase(bind_incoming_queries_total[1h])) by (type)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;{{type}}&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><h4 id="response-codes">Response Codes <a class="anchor" data-pagefind-ignore href="#response-codes" aria-label="Link to this section">#</a></h4>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Response Codes&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;stat&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(bind_responses_total{result=\&#34;Success\&#34;}[5m]))&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;NOERROR&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(bind_responses_total{result=\&#34;NXDOMAIN\&#34;}[5m]))&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;NXDOMAIN&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;sum(rate(bind_responses_total{result=\&#34;SERVFAIL\&#34;}[5m]))&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;legendFormat&#34;</span><span class="p">:</span> <span class="s2">&#34;SERVFAIL&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><h4 id="cache-hit-rate">Cache Hit Rate <a class="anchor" data-pagefind-ignore href="#cache-hit-rate" aria-label="Link to this section">#</a></h4>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;Cache Hit Rate&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;gauge&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;targets&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;expr&#34;</span><span class="p">:</span> <span class="s2">&#34;(1 - sum(rate(bind_query_recursions_total[5m])) / sum(rate(bind_incoming_queries_total[5m]))) * 100&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">],</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;fieldConfig&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;defaults&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;unit&#34;</span><span class="p">:</span> <span class="s2">&#34;percent&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;min&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;max&#34;</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;thresholds&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;steps&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">          <span class="p">{</span><span class="nt">&#34;value&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="nt">&#34;color&#34;</span><span class="p">:</span> <span class="s2">&#34;red&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">          <span class="p">{</span><span class="nt">&#34;value&#34;</span><span class="p">:</span> <span class="mi">70</span><span class="p">,</span> <span class="nt">&#34;color&#34;</span><span class="p">:</span> <span class="s2">&#34;yellow&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">          <span class="p">{</span><span class="nt">&#34;value&#34;</span><span class="p">:</span> <span class="mi">90</span><span class="p">,</span> <span class="nt">&#34;color&#34;</span><span class="p">:</span> <span class="s2">&#34;green&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">]</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><h2 id="alerting-rules">Alerting Rules <a class="anchor" data-pagefind-ignore href="#alerting-rules" aria-label="Link to this section">#</a></h2>
<h3 id="prometheus-alert-rules">Prometheus Alert Rules <a class="anchor" data-pagefind-ignore href="#prometheus-alert-rules" aria-label="Link to this section">#</a></h3>
<p>Each rule has a <code>for</code> duration, so a short spike does not page anyone. Tune the thresholds to your normal traffic.</p>
<table>
	<thead>
			<tr>
					<th>Alert</th>
					<th>Fires when</th>
					<th>Why it matters</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>BindHighQueryRate</code></td>
					<td>Over 1000 queries/s for 5 minutes</td>
					<td>Unexpected load or a misbehaving client</td>
			</tr>
			<tr>
					<td><code>BindHighServfailRate</code></td>
					<td>Over 5% SERVFAIL for 5 minutes</td>
					<td>Upstream, DNSSEC or zone problems; clients see failures</td>
			</tr>
			<tr>
					<td><code>BindLowCacheHitRate</code></td>
					<td>Cache hit rate under 70% for 15 minutes</td>
					<td>More recursion, slower answers</td>
			</tr>
			<tr>
					<td><code>BindRecursiveClientsHigh</code></td>
					<td>Over 900 recursive clients for 2 minutes</td>
					<td>Close to the default limit of 1000</td>
			</tr>
			<tr>
					<td><code>BindZoneTransferFailure</code></td>
					<td>Any failed transfer in the last hour</td>
					<td>A slave may serve stale data</td>
			</tr>
			<tr>
					<td><code>BindExporterDown</code></td>
					<td>Exporter unreachable for 2 minutes</td>
					<td>No metrics, so no other alert can fire</td>
			</tr>
	</tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># /etc/prometheus/rules/bind.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">bind</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># High query rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindHighQueryRate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">sum(rate(bind_incoming_queries_total[5m])) by (instance) &gt; 1000</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High DNS query rate on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Query rate is {{ $value }} queries/sec&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># SERVFAIL rate too high</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindHighServfailRate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(bind_responses_total{result=&#34;SERVFAIL&#34;}[5m])) by (instance) /
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(bind_responses_total[5m])) by (instance) &gt; 0.05</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High SERVFAIL rate on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;SERVFAIL rate is {{ $value | humanizePercentage }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Low cache hit rate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindLowCacheHitRate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          1 - (sum(rate(bind_query_recursions_total[5m])) by (instance) /
</span></span></span><span class="line"><span class="cl"><span class="sd">               sum(rate(bind_incoming_queries_total[5m])) by (instance)) &lt; 0.7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">15m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Low cache hit rate on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Cache hit rate is {{ $value | humanizePercentage }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Recursive clients exhausted</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindRecursiveClientsHigh</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">bind_recursive_clients &gt; 900</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">2m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High recursive clients on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Recursive clients: {{ $value }} (default max: 1000)&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Zone transfer failures</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindZoneTransferFailure</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">increase(bind_zone_transfer_failure_total[1h]) &gt; 0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">0m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Zone transfer failure on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Zone transfers have failed in the last hour&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Exporter down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindExporterDown</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">up{job=&#34;bind&#34;} == 0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">2m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;BIND exporter down on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span></code></pre></div><h2 id="query-logging-for-analysis">Query Logging for Analysis <a class="anchor" data-pagefind-ignore href="#query-logging-for-analysis" aria-label="Link to this section">#</a></h2>
<p>Metrics give totals; the query log says which client asked for which name. Enable it when you need that detail. It costs disk space and some performance on busy servers.</p>
<h3 id="enable-query-logging">Enable Query Logging <a class="anchor" data-pagefind-ignore href="#enable-query-logging" aria-label="Link to this section">#</a></h3>
<p>Separate files for queries and security events, rotated by size (<code>versions</code> and <code>size</code>):</p>
<pre tabindex="0"><code class="language-named" data-lang="named">// /etc/bind/named.conf

logging {
    channel query_log {
        file &#34;/var/log/named/queries.log&#34; versions 10 size 100m;
        severity info;
        print-time yes;
        print-category yes;
        print-severity yes;
    };
    
    channel security_log {
        file &#34;/var/log/named/security.log&#34; versions 5 size 50m;
        severity info;
        print-time yes;
        print-category yes;
    };
    
    category queries { query_log; };
    category security { security_log; };
    category query-errors { query_log; };
};
</code></pre><p>Defining the <code>queries</code> category does not switch query logging on: it is enabled at startup, but <code>rndc reconfig</code> or <code>rndc reload</code> leaves it off. Turn it on with <code>rndc querylog on</code>, or add <code>querylog yes;</code> to <code>options</code> so it survives reloads.</p>
<h3 id="parse-logs-with-promtailloki">Parse Logs with Promtail/Loki <a class="anchor" data-pagefind-ignore href="#parse-logs-with-promtailloki" aria-label="Link to this section">#</a></h3>
<p>Promtail ships the logs to Loki. The <code>regex</code> stage pulls client IP, domain and type out of each query line, and the <code>timestamp</code> stage uses BIND&rsquo;s time instead of the read time.</p>
<p><code>client_ip</code> and <code>domain</code> as labels create a Loki stream per client and per name. That is fine for a small server; on a busy one, keep them out of the labels and extract them at query time instead.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># /etc/promtail/promtail.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">server</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">http_listen_port</span><span class="p">:</span><span class="w"> </span><span class="m">9080</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">positions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">filename</span><span class="p">:</span><span class="w"> </span><span class="l">/var/lib/promtail/positions.yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">clients</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">http://loki:3100/loki/api/v1/push</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="l">bind_queries</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="l">localhost</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">job</span><span class="p">:</span><span class="w"> </span><span class="l">bind_queries</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">__path__</span><span class="p">:</span><span class="w"> </span><span class="l">/var/log/named/queries.log</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">pipeline_stages</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">regex</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">expression</span><span class="p">:</span><span class="w"> </span><span class="nt">&#39;^(?P&lt;timestamp&gt;\d+-\w+-\d+ \d+:\d+:\d+.\d+) queries</span><span class="p">:</span><span class="w"> </span><span class="nt">info</span><span class="p">:</span><span class="w"> </span><span class="l">client @\S+ (?P&lt;client_ip&gt;[\d.]+)#\d+ \((?P&lt;query&gt;[^)]+)\): query: (?P&lt;domain&gt;\S+) IN (?P&lt;type&gt;\w+)&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">client_ip</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">query</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">domain</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">type</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">timestamp</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l">timestamp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">format</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;02-Jan-2006 15:04:05.000&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="l">bind_security</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="l">localhost</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">job</span><span class="p">:</span><span class="w"> </span><span class="l">bind_security</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">__path__</span><span class="p">:</span><span class="w"> </span><span class="l">/var/log/named/security.log</span><span class="w">
</span></span></span></code></pre></div><h2 id="complete-monitoring-stack-with-docker-compose">Complete Monitoring Stack with Docker Compose <a class="anchor" data-pagefind-ignore href="#complete-monitoring-stack-with-docker-compose" aria-label="Link to this section">#</a></h2>
<p>Deploy a complete monitoring stack on the DNS server itself. Good for a single server or a lab; for several DNS servers, run Prometheus and Grafana once, centrally.</p>
<p>The statistics channel only listens on 127.0.0.1 with <code>allow { localhost; }</code>, so a bind_exporter container on a Docker bridge network cannot reach it. Run the exporter with <code>network_mode: host</code> instead: it then talks to 127.0.0.1:8053 like the systemd service above, and listens on the host&rsquo;s port 9119. Prometheus reaches it through <code>host.docker.internal</code>. Use either this container or the systemd service, not both, because both listen on port 9119.</p>
<pre tabindex="0"><code>bind-monitoring/
├── docker-compose.yml
├── .env
├── prometheus/
│   ├── prometheus.yml
│   └── rules/
│       ├── bind.yml
│       └── bind_security.yml
├── alertmanager/
│   └── alertmanager.yml
└── grafana/
    └── provisioning/
        └── datasources/
            └── prometheus.yml
</code></pre><p>The services:</p>
<ul>
<li><code>prometheus</code>: <code>--web.enable-lifecycle</code> allows reloads over HTTP; <code>extra_hosts</code> maps <code>host.docker.internal</code> to the host.</li>
<li><code>grafana</code>: admin password from <code>.env</code>, sign-up disabled.</li>
<li><code>alertmanager</code>: receives alerts from Prometheus.</li>
<li><code>bind_exporter</code>: on the host network, as explained above.</li>
</ul>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># docker-compose.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">prometheus</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">prom/prometheus:v3.16.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">prometheus</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">unless-stopped</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">./prometheus:/etc/prometheus</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">prometheus_data:/prometheus</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">command</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;--config.file=/etc/prometheus/prometheus.yml&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;--storage.tsdb.path=/prometheus&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;--web.enable-lifecycle&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;9090:9090&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">extra_hosts</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;host.docker.internal:host-gateway&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">networks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">monitoring</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">grafana</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">grafana/grafana:13.3.1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">grafana</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">unless-stopped</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">grafana_data:/var/lib/grafana</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">./grafana/provisioning:/etc/grafana/provisioning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">GF_USERS_ALLOW_SIGN_UP=false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;3000:3000&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">networks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">monitoring</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">alertmanager</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">prom/alertmanager:v0.33.1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">alertmanager</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">unless-stopped</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">./alertmanager:/etc/alertmanager</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">command</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;--config.file=/etc/alertmanager/alertmanager.yml&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;9093:9093&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">networks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">monitoring</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="c"># Host network: the statistics channel only listens on 127.0.0.1:8053 (allow localhost).</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="c"># The exporter listens on the host&#39;s port 9119; Prometheus reaches it via host.docker.internal.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">bind_exporter</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">prometheuscommunity/bind-exporter:v0.7.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">bind_exporter</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">unless-stopped</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">network_mode</span><span class="p">:</span><span class="w"> </span><span class="l">host</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">command</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;--bind.stats-url=http://127.0.0.1:8053/&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s1">&#39;--bind.stats-groups=server,view,tasks&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">prometheus_data</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">grafana_data</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">networks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">monitoring</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">driver</span><span class="p">:</span><span class="w"> </span><span class="l">bridge</span><span class="w">
</span></span></span></code></pre></div><p>Keep the Grafana admin password out of the compose file. Put it in <code>.env</code> next to <code>docker-compose.yml</code> (mode 0600), which Docker Compose reads automatically:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">GRAFANA_ADMIN_PASSWORD</span><span class="o">=</span>change-me
</span></span></code></pre></div><p>Prometheus in the stack scrapes the exporter on the host and sends alerts to Alertmanager. Put the two rule files from the alerting sections in <code>prometheus/rules/</code>. The relabeling sets a fixed <code>instance</code> name, because <code>host.docker.internal</code> says nothing:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># prometheus/prometheus.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">global</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">evaluation_interval</span><span class="p">:</span><span class="w"> </span><span class="l">15s</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">alerting</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">alertmanagers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;alertmanager:9093&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">rule_files</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">/etc/prometheus/rules/*.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;prometheus&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;localhost:9090&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;bind&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">static_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;host.docker.internal:9119&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">relabel_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">target_label</span><span class="p">:</span><span class="w"> </span><span class="l">instance</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">replacement</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;dns1&#39;</span><span class="w">
</span></span></span></code></pre></div><p>Alertmanager refuses to start without a configuration file. A minimal one sends every alert to a single receiver without notifications; add <code>email_configs</code>, <code>slack_configs</code> or <code>webhook_configs</code> to the receiver to get notified:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># alertmanager/alertmanager.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">route</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">receiver</span><span class="p">:</span><span class="w"> </span><span class="l">default</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;alertname&#39;</span><span class="p">,</span><span class="w"> </span><span class="s1">&#39;instance&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">receivers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">default</span><span class="w">
</span></span></span></code></pre></div><p>Provision the Prometheus datasource so Grafana is ready to use on first start:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># grafana/provisioning/datasources/prometheus.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">datasources</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Prometheus</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">uid</span><span class="p">:</span><span class="w"> </span><span class="l">prometheus</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">prometheus</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">access</span><span class="p">:</span><span class="w"> </span><span class="l">proxy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">http://prometheus:9090</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">isDefault</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span></code></pre></div><p>Start the stack and check that both targets are up:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">docker compose up -d
</span></span><span class="line"><span class="cl">curl -s http://localhost:9090/api/v1/targets <span class="p">|</span> grep -o <span class="s1">&#39;&#34;health&#34;:&#34;[a-z]*&#34;&#39;</span>
</span></span></code></pre></div><h2 id="security-monitoring">Security Monitoring <a class="anchor" data-pagefind-ignore href="#security-monitoring" aria-label="Link to this section">#</a></h2>
<h3 id="detecting-dns-attacks">Detecting DNS Attacks <a class="anchor" data-pagefind-ignore href="#detecting-dns-attacks" aria-label="Link to this section">#</a></h3>
<p>Traffic patterns reveal abuse before it becomes an outage. These rules go in <code>prometheus/rules/bind_security.yml</code>:</p>
<table>
	<thead>
			<tr>
					<th>Alert</th>
					<th>Fires when</th>
					<th>Possible cause</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>BindPossibleAmplificationAttack</code></td>
					<td>Over 100 ANY queries/s</td>
					<td>Your server used to amplify a DDoS</td>
			</tr>
			<tr>
					<td><code>BindHighNxdomainRate</code></td>
					<td>Over 30% NXDOMAIN for 10 minutes</td>
					<td>Enumeration, or malware generating random names</td>
			</tr>
			<tr>
					<td><code>BindQueryFlood</code></td>
					<td>Over 5000 queries/s</td>
					<td>Query flood</td>
			</tr>
	</tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># Security-focused alerts</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">bind_security</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Possible DNS amplification attack</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindPossibleAmplificationAttack</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(bind_incoming_queries_total{type=&#34;ANY&#34;}[5m])) by (instance) &gt; 100</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">2m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Possible DNS amplification attack on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High rate of ANY queries detected&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Unusual NXDOMAIN rate (possible enumeration)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindHighNxdomainRate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(bind_responses_total{result=&#34;NXDOMAIN&#34;}[5m])) by (instance) /
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(bind_responses_total[5m])) by (instance) &gt; 0.3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">10m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">warning</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;High NXDOMAIN rate on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;NXDOMAIN rate is {{ $value | humanizePercentage }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># Query flood detection</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">BindQueryFlood</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">sum(rate(bind_incoming_queries_total[1m])) by (instance) &gt; 5000</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">1m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">critical</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;DNS query flood on {{ $labels.instance }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Query rate: {{ $value }} q/s&#34;</span><span class="w">
</span></span></span></code></pre></div><h2 id="useful-promql-queries">Useful PromQL Queries <a class="anchor" data-pagefind-ignore href="#useful-promql-queries" aria-label="Link to this section">#</a></h2>
<h3 id="performance-analysis">Performance Analysis <a class="anchor" data-pagefind-ignore href="#performance-analysis" aria-label="Link to this section">#</a></h3>
<p>How long upstream queries take, and whether DNSSEC validation fails:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># 95th percentile upstream query latency</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">histogram_quantile</span><span class="o">(</span><span class="mf">0.95</span><span class="p">,</span><span class="w"> </span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">bind_resolver_query_duration_seconds_bucket</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">le</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># DNSSEC validation failures</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">increase</span><span class="o">(</span><span class="nv">bind_resolver_dnssec_validation_errors_total</span><span class="p">[</span><span class="s">1h</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><p>Top queried domains come from the parsed query log, not from Prometheus. In Loki (LogQL):</p>
<pre tabindex="0"><code class="language-logql" data-lang="logql">topk(10, sum by (domain) (count_over_time({job=&#34;bind_queries&#34;}[1h])))
</code></pre><h3 id="capacity-planning">Capacity Planning <a class="anchor" data-pagefind-ignore href="#capacity-planning" aria-label="Link to this section">#</a></h3>
<p>Peak and daily volume tell you when to add a server; the cache size shows memory use growing:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="c1"># Peak query rate (last 7 days)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kr">max_over_time</span><span class="o">(</span><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">bind_incoming_queries_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="p">[</span><span class="s">7d</span><span class="err">:</span><span class="s">5m</span><span class="p">]</span><span class="o">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Average queries per day</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">increase</span><span class="o">(</span><span class="nv">bind_incoming_queries_total</span><span class="p">[</span><span class="s">24h</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1"># Cache size (RRsets in cache)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">instance</span><span class="o">)</span><span class="w"> </span><span class="o">(</span><span class="nv">bind_resolver_cache_rrsets</span><span class="o">)</span><span class="w">
</span></span></span></code></pre></div><h2 id="best-practices">Best Practices <a class="anchor" data-pagefind-ignore href="#best-practices" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Monitor all DNS servers</strong>: deploy bind_exporter on every BIND instance.</li>
<li><strong>Set appropriate scrape intervals</strong>: 15-30 seconds is usually sufficient.</li>
<li><strong>Retain metrics appropriately</strong>: keep high-resolution data for 15 days, aggregated data longer.</li>
<li><strong>Alert on trends, not spikes</strong>: use <code>for</code> duration to avoid alert fatigue.</li>
<li><strong>Correlate with other metrics</strong>: link DNS performance to application metrics.</li>
<li><strong>Document your dashboards</strong>: add descriptions to panels explaining what they show.</li>
<li><strong>Test your alerts</strong>: periodically verify alerts fire correctly.</li>
</ul>
<h2 id="security-considerations">Security Considerations <a class="anchor" data-pagefind-ignore href="#security-considerations" aria-label="Link to this section">#</a></h2>
<ul>
<li><strong>Statistics channel on localhost</strong>: it exposes server internals; keep <code>allow { localhost; }</code>.</li>
<li><strong>Firewall the exporter</strong>: port 9119 listens on all interfaces; only Prometheus needs it.</li>
<li><strong>Query logs are personal data</strong>: they show who looked up what. Limit access and retention.</li>
<li><strong>Grafana password in <code>.env</code></strong>: mode 0600, not in the compose file or Git.</li>
</ul>
<h2 id="automate-it">Automate it <a class="anchor" data-pagefind-ignore href="#automate-it" aria-label="Link to this section">#</a></h2>
<p>The Ansible roles and Terraform code that build this setup are on GitHub: <a href="https://github.com/pderuiter/blog-bind-monitoring" rel="noopener">pderuiter/blog-bind-monitoring</a>. It creates an Ubuntu 24.04 VM on Proxmox and runs one playbook that enables the BIND statistics channel and query logging, and deploys this Docker Compose stack (or the systemd exporter); <code>tests/verify.yml</code> then checks bind_up, the Prometheus targets and rules, Alertmanager and the Grafana datasource.</p>
<h2 id="conclusion">Conclusion <a class="anchor" data-pagefind-ignore href="#conclusion" aria-label="Link to this section">#</a></h2>
<p>bind_exporter turns BIND&rsquo;s statistics channel into Prometheus metrics; Grafana shows them and Alertmanager tells you when something breaks.</p>
<p>This completes our BIND DNS series covering:</p>
<ol>
<li>Resolver configuration</li>
<li>Authoritative server setup</li>
<li>RNDC and remote management</li>
<li>Zone security and ACLs</li>
<li>Dynamic DNS updates</li>
<li>Catalog zones</li>
<li>DNSSEC validation</li>
<li>DNSSEC zone signing</li>
<li>Response Policy Zones</li>
<li>DNS over TLS</li>
<li>DNS over HTTPS</li>
<li>External-DNS integration</li>
<li>Prometheus monitoring</li>
</ol>
<p>With these building blocks, you can build a robust, secure, and observable DNS infrastructure for any environment - from homelab to enterprise.</p>
]]></content:encoded></item></channel></rss>