Traefik was my ingress controller for years, on Docker hosts first and then on every Kubernetes cluster I run. It’s a good proxy. Labels and CRDs feel natural, Let’s Encrypt is built in, and the dashboard is nice to have.

A month ago I replaced it with HAProxy on all seven clusters anyway. This post explains why, describes the setup before and after, and is honest about what I gave up.

One naming note first, because it confuses everyone. There are two HAProxy-based ingress controllers:

Both are fine. Everything below is about the first one, at controller 3.4.1 (Helm chart 1.57.1, HAProxy 3.4).

The setup before #

Seven single-node Talos clusters (production, non-production, management, security, monitoring, identity and a pilot cluster), all managed with Flux from one Git repository. Each cluster ran the same Traefik stack:

                 internet / LAN
                        |
             MetalLB LoadBalancer IP
           (externalTrafficPolicy: Local)
                        |
            +-----------------------+
            |        Traefik        |   Helm chart, one per cluster
            |  entryPoints 80/443   |   HTTP -> HTTPS redirect
            |  Middlewares:         |
            |   ipAllowList         |
            |   basicAuth           |
            |   headers             |
            |  IngressRouteTCP      |   TLS passthrough for a few apps
            +-----------------------+
              |        |        |
           metrics   access   OTLP traces
              |       logs       |
              +-------> Alloy -> Prometheus / Loki / Tempo

   TLS: cert-manager, Let's Encrypt DNS-01
   DNS: external-dns, reading Ingress and IngressRoute objects
   WAF: none in the cluster

It worked. Certificates renewed, routing was predictable, and the dashboards looked good. The gap was the WAF: there wasn’t one inside the clusters. A web application firewall is the piece I wanted most.

The setup now #

Same clusters, same Flux repository, same MetalLB address per cluster. Traefik is gone:

                 internet / LAN
                        |
             MetalLB LoadBalancer IP   (the same address Traefik had)
                        |
   +----------------------------------------------------------+
   |  haproxy-ingress pod                                     |
   |                                                          |
   |  +--------------------+  SPOP   +---------------------+  |
   |  | HAProxy 3.4        |-------->| coraza-spoa         |  |
   |  | Ingress controller |  :9000  | OWASP CRS 4.30      |  |
   |  |                    |         +---------------------+  |
   |  | IngressClass       |  SPOP   +---------------------+  |
   |  |   haproxy-waf      |-------->| otel-spoa           |--+--> Alloy -> Tempo
   |  +--------------------+  :12345 +---------------------+  |
   +----------------------------------------------------------+
        |               |                 |
     stats/metrics   one log line      Gatus checks the
     (Alloy scrapes) per request       default backend
                     (WAF verdict + trace id)

   TLS: cert-manager, Let's Encrypt DNS-01 (unchanged)
   DNS: external-dns, reading Ingress objects
   CI:  render the chart, then `haproxy -c` on the generated config

The controller runs with two sidecars in the same pod:

  • coraza-spoa is the Coraza WAF with the OWASP Core Rule Set. HAProxy hands each request to it over SPOE (the Stream Processing Offload Engine) and acts on the verdict.
  • otel-spoa turns HAProxy’s events into OpenTelemetry spans and sends them to the cluster’s Alloy, which forwards them to Tempo.

Both sidecars listen on 127.0.0.1 only. Nothing outside the pod can talk to them.

Each cluster switched over in one merge request: HAProxy took over Traefik’s LoadBalancer address and Traefik was removed. Each switch took about a minute. Before it, I recorded status code, redirect target and key headers for every host; after it, all 37 hosts on the first five clusters matched that baseline.

That comparison also caught the one real bug, which no lint or render check would have found. It’s under “what I gave up” below.

The eight reasons #

1. WAF integration that doesn’t live inside the proxy #

This was the trigger. Traefik can run Coraza through its WebAssembly plugin system, and I tried it. It ran, but never reliably enough to switch it from logging to blocking. The Traefik plugin has had no release since version 0.3.0 in October 2024. I don’t want a WAF I can’t trust to block.

HAProxy takes a different approach. The WAF is a separate process, and HAProxy talks to it over SPOE, a protocol built for exactly this. The proxy and the WAF fail separately, scale separately, and can be upgraded separately. The Coraza SPOA release I run is from October 2026, and the Core Rule Set is current.

The wiring is a few lines of HAProxy configuration in the controller’s frontend snippet:

option http-buffer-request
filter spoe engine coraza config /etc/haproxy/spoe/coraza.cfg
http-request send-spoe-group coraza coraza-req
http-request deny deny_status 403 if { var(txn.coraza.action) -m found }

The SPOE file defines what HAProxy sends to Coraza and how long it waits:

[coraza]
spoe-agent coraza-agent
    groups      coraza-req
    option      var-prefix   coraza
    option      set-on-error error
    timeout     processing   500ms
    use-backend coraza-spoa

spoe-message coraza-req
    args app=var(txn.coraza.app) src-ip=src src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body

spoe-group coraza-req
    messages coraza-req

My earlier post, Securing HAProxy with Coraza-SPOA, covers the agent itself.

2. WAF policy as plain HAProxy ACLs #

Because the WAF verdict comes back as HAProxy variables, deciding what to do with it is ordinary HAProxy configuration. That matters more than I expected:

  • Per-application profiles. txn.coraza.app picks a Coraza profile per host. A telemetry ingest endpoint gets a relaxed profile (no body inspection); everything else gets the default.
  • Gradual enforcement. Coraza started in DetectionOnly on every cluster. I then moved hosts to blocking one at a time, by adding them to an ACL, after their logs had been clean for a while.
  • An explicit failure policy. If Coraza errors or times out, set-on-error sets a variable. The config decides, per host, whether that means “let it through and log it” or “block”.
acl waf_enforce_host req.hdr(host),field(1,:) -m str -i blog.example.com docs.example.com
http-request set-var(txn.waf_decision) str(blocked) if waf_enforce_host { var(txn.coraza.action) -m found }
http-request set-var(txn.waf_decision) str(error)   if !{ var(txn.waf_decision) -m found } { var(txn.coraza.error) -m int gt 0 }
http-request set-var(txn.waf_decision) str(detected) if !{ var(txn.waf_decision) -m found } { var(txn.coraza.rules_hit) -m int gt 0 }
http-request deny deny_status 403 if { var(txn.waf_decision) -m str blocked }

In Traefik a plugin is a black box with a config block. Here every decision is a line I can read, review and test.

3. Performance #

HAProxy is an event-driven, multi-threaded C proxy with a long record of doing exactly this job at very high request rates. Traefik is fast too, and on single-node clusters neither one is the bottleneck.

The difference that matters to me is where the WAF runs. A WebAssembly plugin runs inside the proxy’s own process, on every request. With SPOE, HAProxy hands the request to Coraza and moves on to other connections while it waits, with a hard timeout I set myself (500 ms). A slow inspection can’t stall the proxy.

I didn’t run a formal benchmark, so I won’t quote numbers. What I can say: since the switch, the ingress has never been the reason a page was slow.

4. Load-balancing options and methods #

Traefik has caught up here: version 3.7 offers weighted round robin, power-of-two-choices, consistent hashing by client IP and a least-time strategy. That covers most workloads.

HAProxy goes further, and the controller exposes it as plain Ingress annotations:

NeedAnnotation
Algorithm: roundrobin, leastconn, first, source, uri, hdr(<name>), url_param, randomhaproxy.org/load-balance
Limit concurrent requests per pod, queue the resthaproxy.org/pod-maxconn, haproxy.org/timeout-queue
Active health checks with their own intervalhaproxy.org/check, haproxy.org/check-http, haproxy.org/check-interval
Rate limiting per sourcehaproxy.org/rate-limit-requests, haproxy.org/rate-limit-period
Sticky sessionshaproxy.org/cookie-persistence

Queueing is the one I value most. With pod-maxconn, a pod that can handle 20 concurrent requests gets 20, and request 21 waits in HAProxy’s queue instead of piling onto a busy pod. For a slow application that is the difference between “a bit slower under load” and “falls over under load”.

metadata:
  annotations:
    haproxy.org/load-balance: "leastconn"
    haproxy.org/pod-maxconn: "20"
    haproxy.org/timeout-queue: "30s"

5. Stability #

Three things make HAProxy boring, in the good sense:

  • LTS branches. Even-numbered HAProxy releases (3.2, 3.4) are long-term support, maintained for five years. The controller follows them.
  • Endpoint changes without reloads. When pods scale or move, the controller updates HAProxy’s server slots through the runtime API. Only real config changes trigger a reload, and HAProxy reloads without dropping connections.
  • One locked combination. I pin the chart, the controller image, both sidecar images and the rule set together in a release lock. A combination is tested on the pilot cluster first, then promoted cluster by cluster to production.

None of that is impossible with Traefik. It’s just the default way of working with HAProxy.

6. Observability #

Traefik’s observability is good: Prometheus metrics, access logs and OpenTelemetry tracing out of the box. I’m not claiming HAProxy is better at any single one of those.

What I gained is correlation. HAProxy’s log-format can include any variable, so one log line per request carries:

  • the timings: time to receive the request, time in the queue, time to connect, time until the response;
  • the backend and server that handled it;
  • the WAF verdict, matched rule IDs and anomaly score;
  • the W3C traceparent of the request’s trace.
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %tsc \"%HM %[var(txn.base)] %HV\" id=%ID waf=%[var(txn.waf_decision)] waf_ids=%[var(txn.coraza.rule_ids)] waf_score=%[var(txn.coraza.anomaly_score)] traceparent=%[var(txn.otel.traceparent)]"

When someone reports a 403, one Loki query shows which rule blocked them, and the trace ID takes me from the log line to the trace in Tempo. Alloy also scrapes HAProxy’s stats endpoint and the Coraza metrics port, and Gatus checks every cluster’s default backend. My HAProxy monitoring post shows the metrics side.

7. Project maturity #

HAProxy has been around since 2001. Its configuration language, logging format and runtime API have stayed recognisable for more than two decades, and HAProxy Technologies maintains both the proxy and the ingress controller. The controller sees regular releases: two branches got updates in the week this post was written.

For the WAF, maturity also means the pieces are maintained. Coraza SPOA is an official Coraza project with releases this autumn, and the Core Rule Set ships signed releases. That beats a plugin that hasn’t been released in two years.

8. One proxy everywhere #

Outside Kubernetes I already ran HAProxy: on the edge, as a load balancer for DNS and for TCP services. Those setups have their own posts in the HAProxy series.

With HAProxy as the ingress too, there’s one proxy to understand, one log format to parse, one set of metrics and one place to look when something misbehaves. A WAF rule tuned at the edge works the same inside the cluster. That reason alone wouldn’t have made me switch, but it’s why the switch felt like simplifying rather than adding.

How I test the config #

The controller generates haproxy.cfg from Ingress objects plus my snippets, so a typo in a snippet only shows up when the pod starts. Two CI jobs catch that before merge:

  1. Render. Build the Flux overlay with kustomize, merge the values the HelmRelease would get, check the chart checksum against the release lock, and run helm template. The job also asserts that images are pinned by digest and that the pod meets the restricted Pod Security Standard.
  2. Validate. Run haproxy -c against the snippets inside the exact controller image that will run them.
# A sketch: the render script assembles the snippets into a full config.
haproxy-ingress-config:
  image:
    name: haproxytech/kubernetes-ingress:3.4.1
    entrypoint: [""]
  script:
    - ./scripts/render-haproxy-config.sh > /tmp/haproxy.cfg
    - haproxy -c -f /tmp/haproxy.cfg   # silent on success, non-zero on error

With Traefik, a broken Middleware reference meant a 404 in production. Now a broken snippet means a red pipeline.

What I gave up #

Fair is fair. Traefik does some things better:

  • CRDs over annotations. Traefik’s Middleware objects are reusable and composable. HAProxy annotations live on a backend, so two Ingresses that share a service port also share its auth and allow-list settings. The workaround is haproxy.org/standalone-backend: "true", which gives an Ingress its own backend. That works, but you need to know it exists.
  • A dashboard. Traefik’s dashboard shows routers and services at a glance. HAProxy has a stats page; Grafana does the rest.
  • Redirect surprise. The controller put its internal HTTPS port (8443) into the HTTP-to-HTTPS redirect. Clients got sent to https://host:8443/, a port the LoadBalancer doesn’t expose. The fix is one line in the controller config, ssl-redirect-port: "443", but every cluster shipped with the bug for three days. Only the per-host before-and-after comparison caught it.
  • Learning curve. HAProxy’s config language is powerful and terse. ACLs, fetches and converters take time to learn, and a snippet you don’t understand is a liability.

Should you switch? #

  • Switch if you want a WAF you can enforce, need queueing or fine-grained load balancing, or already run HAProxy elsewhere.
  • Stay on Traefik if your clusters are small, you don’t need a WAF in the cluster, and Middlewares and the dashboard make your life easier. It’s a good proxy; it just stopped being the right one for me.