haproxy

HAProxy and Consul: Dynamic Service Discovery for Load Balancing

Patrick de Ruiter

Patrick de Ruiter

— 7 min read
HAProxy and Consul: Dynamic Service Discovery for Load Balancing

In dynamic environments where services scale up and down automatically, static HAProxy configurations become a management burden. Consul integration enables HAProxy to automatically discover and route traffic to healthy service instances without manual configuration updates.

Why Consul + HAProxy?

The combination provides:

  • Automatic service registration - Services announce themselves to Consul
  • Health-aware routing - Only healthy instances receive traffic
  • Dynamic scaling - New instances are automatically added to load balancing
  • Multi-datacenter - Route traffic across datacenters with Consul federation
  • No downtime updates - Configuration changes without service interruption

Architecture Overview

                    ┌─────────────────────────────────┐
                    │           Consul Cluster        │
                    │  ┌───────┐ ┌───────┐ ┌───────┐ │
                    │  │Server1│ │Server2│ │Server3│ │
                    │  └───────┘ └───────┘ └───────┘ │
                    └────────────────┬────────────────┘
                                     │
              ┌──────────────────────┼──────────────────────┐
              │                      │                      │
              ▼                      ▼                      ▼
      ┌───────────────┐      ┌───────────────┐      ┌───────────────┐
      │   Service A   │      │   Service B   │      │   Service C   │
      │ + Consul Agent│      │ + Consul Agent│      │ + Consul Agent│
      └───────────────┘      └───────────────┘      └───────────────┘
              │                      │                      │
              └──────────────────────┼──────────────────────┘
                                     │
                                     ▼
                          ┌─────────────────────┐
                          │      HAProxy        │
                          │  + consul-template  │
                          └─────────────────────┘
                                     │
                                     ▼
                               Clients/Users

Setting Up Consul

Consul Server Installation

# Download Consul
CONSUL_VERSION="1.17.0"
wget https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip
unzip consul_${CONSUL_VERSION}_linux_amd64.zip
sudo mv consul /usr/local/bin/

# Create directories
sudo mkdir -p /etc/consul.d /var/lib/consul
sudo useradd --system --home /var/lib/consul --shell /bin/false consul
sudo chown -R consul:consul /etc/consul.d /var/lib/consul

Consul Server Configuration

Create /etc/consul.d/consul.hcl:

datacenter = "dc1"
node_name = "consul-server-1"
data_dir = "/var/lib/consul"
log_level = "INFO"

server = true
bootstrap_expect = 3

bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
client_addr = "0.0.0.0"

ui_config {
  enabled = true
}

connect {
  enabled = true
}

addresses {
  http = "0.0.0.0"
  grpc = "0.0.0.0"
}

ports {
  http = 8500
  grpc = 8502
  grpc_tls = 8503
}

retry_join = [
  "consul-server-1.example.com",
  "consul-server-2.example.com",
  "consul-server-3.example.com"
]

acl {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

Consul Agent Configuration (for service nodes)

Create /etc/consul.d/consul.hcl on service nodes:

datacenter = "dc1"
node_name = "service-node-1"
data_dir = "/var/lib/consul"
log_level = "INFO"

server = false

bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
client_addr = "127.0.0.1"

retry_join = [
  "consul-server-1.example.com",
  "consul-server-2.example.com",
  "consul-server-3.example.com"
]

Systemd Service

Create /etc/systemd/system/consul.service:

[Unit]
Description=Consul Service Discovery Agent
Documentation=https://www.consul.io/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Registering Services with Consul

Service Registration File

Create /etc/consul.d/web-service.hcl:

service {
  name = "web"
  port = 8080
  tags = ["http", "production"]
  
  meta {
    version = "1.0.0"
    environment = "production"
  }
  
  check {
    name = "HTTP Health Check"
    http = "http://localhost:8080/health"
    interval = "10s"
    timeout = "5s"
    
    success_before_passing = 2
    failures_before_critical = 3
  }
  
  check {
    name = "TCP Port Check"
    tcp = "localhost:8080"
    interval = "5s"
    timeout = "2s"
  }
}

API Service Registration

service {
  name = "api"
  port = 3000
  tags = ["http", "api", "v2"]
  
  weights {
    passing = 100
    warning = 50
  }
  
  check {
    name = "API Health"
    http = "http://localhost:3000/api/health"
    interval = "10s"
    timeout = "3s"
    header {
      Content-Type = ["application/json"]
    }
  }
}

Dynamic Registration via HTTP API

# Register a service dynamically
curl -X PUT http://localhost:8500/v1/agent/service/register \
  -H "Content-Type: application/json" \
  -d '{
    "ID": "web-1",
    "Name": "web",
    "Port": 8080,
    "Tags": ["http", "production"],
    "Check": {
      "HTTP": "http://localhost:8080/health",
      "Interval": "10s"
    }
  }'

# Deregister a service
curl -X PUT http://localhost:8500/v1/agent/service/deregister/web-1

Consul Template for HAProxy

Installing Consul Template

CT_VERSION="0.35.0"
wget https://releases.hashicorp.com/consul-template/${CT_VERSION}/consul-template_${CT_VERSION}_linux_amd64.zip
unzip consul-template_${CT_VERSION}_linux_amd64.zip
sudo mv consul-template /usr/local/bin/

Consul Template Configuration

Create /etc/consul-template/config.hcl:

consul {
  address = "127.0.0.1:8500"
  
  retry {
    enabled = true
    attempts = 5
    backoff = "250ms"
  }
}

template {
  source = "/etc/consul-template/haproxy.ctmpl"
  destination = "/etc/haproxy/haproxy.cfg"
  perms = 0644
  
  command = "systemctl reload haproxy"
  command_timeout = "60s"
  
  backup = true
  
  wait {
    min = "5s"
    max = "10s"
  }
}

log_level = "info"

HAProxy Template

Create /etc/consul-template/haproxy.ctmpl:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 4096

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    option http-server-close
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

# Stats endpoint
frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

# Main HTTP Frontend
frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/
    
    # ACL definitions based on hostname
    acl host_web hdr(host) -i www.example.com example.com
    acl host_api hdr(host) -i api.example.com
    acl host_admin hdr(host) -i admin.example.com
    
    # Backend selection
    use_backend web_servers if host_web
    use_backend api_servers if host_api
    use_backend admin_servers if host_admin
    
    default_backend web_servers

# Web Servers Backend - dynamically populated from Consul
backend web_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    {{- range service "web" }}
    server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check weight {{ if .Weights.Passing }}{{ .Weights.Passing }}{{ else }}100{{ end }}{{ if not .Checks.AggregatedStatus | eq "passing" }} disabled{{ end }}
    {{- end }}

# API Servers Backend
backend api_servers
    balance leastconn
    option httpchk GET /api/health
    http-check expect status 200
    {{- range service "api" }}
    server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check weight {{ if .Weights.Passing }}{{ .Weights.Passing }}{{ else }}100{{ end }}
    {{- end }}

# Admin Servers Backend
backend admin_servers
    balance roundrobin
    option httpchk GET /health
    {{- range service "admin" }}
    server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check
    {{- end }}

Advanced Template with Tags and Filtering

# Production web servers only (filtered by tag)
backend web_production
    balance roundrobin
    option httpchk GET /health
    {{- range service "web|production" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

# Staging web servers
backend web_staging
    balance roundrobin
    option httpchk GET /health
    {{- range service "web|staging" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

# Services in specific datacenter
backend api_dc1
    balance leastconn
    {{- range service "api" "dc1" }}
    server {{ .Node }}-{{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

# Multi-datacenter with failover
backend api_multi_dc
    balance roundrobin
    {{- range service "api" "dc1" }}
    server {{ .Node }}-{{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}
    {{- range service "api" "dc2" }}
    server {{ .Node }}-{{ .ID }} {{ .Address }}:{{ .Port }} check backup
    {{- end }}

Template with Service Metadata

backend web_servers
    balance roundrobin
    {{- range service "web" }}
    {{- if .ServiceMeta.version | eq "2.0" }}
    # Version 2.0 servers get more weight
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight 150
    {{- else }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight 100
    {{- end }}
    {{- end }}

Consul Template Systemd Service

Create /etc/systemd/system/consul-template.service:

[Unit]
Description=Consul Template
After=network-online.target consul.service
Wants=network-online.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/consul-template -config=/etc/consul-template/config.hcl
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Health Checks and Service Discovery

Consul Health Check Types

service {
  name = "web"
  port = 8080
  
  # HTTP check
  check {
    name = "HTTP Health"
    http = "http://localhost:8080/health"
    method = "GET"
    interval = "10s"
    timeout = "5s"
  }
  
  # TCP check
  check {
    name = "TCP Port"
    tcp = "localhost:8080"
    interval = "5s"
    timeout = "2s"
  }
  
  # Script check
  check {
    name = "Script Check"
    args = ["/usr/local/bin/health-check.sh"]
    interval = "30s"
    timeout = "10s"
  }
  
  # gRPC check
  check {
    name = "gRPC Health"
    grpc = "localhost:9090"
    grpc_use_tls = true
    interval = "10s"
  }
  
  # TTL check (service must update)
  check {
    name = "TTL Check"
    ttl = "30s"
    status = "passing"
  }
}

Updating TTL Check

# Pass TTL check
curl -X PUT http://localhost:8500/v1/agent/check/pass/service:web-1

# Fail TTL check
curl -X PUT http://localhost:8500/v1/agent/check/fail/service:web-1

# Warn TTL check
curl -X PUT http://localhost:8500/v1/agent/check/warn/service:web-1

Advanced HAProxy + Consul Patterns

Blue-Green Deployments

# Template for blue-green deployment
frontend http_front
    bind *:80
    
    # Read active environment from Consul KV
    {{- $active := key "app/active-environment" }}
    
    default_backend web_{{ $active }}

backend web_blue
    balance roundrobin
    {{- range service "web|blue" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

backend web_green
    balance roundrobin
    {{- range service "web|green" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

Switch environments:

# Switch to green
consul kv put app/active-environment green

# Switch to blue
consul kv put app/active-environment blue

Canary Deployments with Weights

backend web_servers
    balance roundrobin
    
    {{- $canary_weight := keyOrDefault "app/canary-weight" "10" }}
    
    # Stable servers
    {{- range service "web|stable" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight {{ subtract 100 (parseInt $canary_weight) }}
    {{- end }}
    
    # Canary servers
    {{- range service "web|canary" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight {{ $canary_weight }}
    {{- end }}

Rate Limiting per Service

frontend http_front
    bind *:80
    
    {{- $rate_limit := keyOrDefault "app/rate-limit" "100" }}
    
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt {{ $rate_limit }} }
    
    default_backend web_servers

Maintenance Mode

frontend http_front
    bind *:80
    
    {{- $maintenance := keyOrDefault "app/maintenance" "false" }}
    
    {{- if eq $maintenance "true" }}
    # Maintenance mode - serve static page
    http-request deny deny_status 503
    {{- else }}
    default_backend web_servers
    {{- end }}

Monitoring and Observability

Consul Service Health Queries

# List all healthy web services
curl http://localhost:8500/v1/health/service/web?passing=true | jq .

# Get service catalog
curl http://localhost:8500/v1/catalog/services | jq .

# Watch for changes
watch -n 2 'curl -s http://localhost:8500/v1/health/service/web | jq length'

HAProxy Stats Integration

frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    
    # Prometheus metrics endpoint
    http-request use-service prometheus-exporter if { path /metrics }

Troubleshooting

Common Issues

  1. Services not appearing in HAProxy
    • Check Consul agent logs: journalctl -u consul
    • Verify service health: consul catalog services
    • Check consul-template logs: journalctl -u consul-template
  2. HAProxy not reloading
    • Verify template syntax: consul-template -template "haproxy.ctmpl:test.cfg" -once -dry
    • Check HAProxy config: haproxy -c -f /etc/haproxy/haproxy.cfg
  3. Services marked unhealthy
    • Check health check status: consul catalog service web
    • Review check definitions in service registration

Debug Commands

# Test template rendering
consul-template -template "/etc/consul-template/haproxy.ctmpl:/tmp/haproxy-test.cfg" -once -dry

# Verify Consul connectivity
consul members

# List all services
consul catalog services

# Get service details
consul catalog service web

# Check service health
consul health checks web

Security Considerations

Consul ACL Tokens

# consul-template config with ACL
consul {
  address = "127.0.0.1:8500"
  token = "your-acl-token"
}

TLS Configuration

consul {
  address = "consul.example.com:8501"
  
  ssl {
    enabled = true
    verify = true
    ca_cert = "/etc/consul-template/ca.pem"
    cert = "/etc/consul-template/client.pem"
    key = "/etc/consul-template/client-key.pem"
  }
}

Next Steps

With Consul service discovery integrated, consider:

  • Prometheus Monitoring - Add metrics collection for HAProxy and Consul
  • Consul Connect - Implement service mesh with automatic mTLS
  • Multi-datacenter - Extend to multiple Consul datacenters for geo-routing

Consul and HAProxy together create a powerful, self-healing infrastructure where services automatically join and leave the load balancer pool based on their health status.

Read more

HAProxy Monitoring with Prometheus: Complete Observability Guide

HAProxy Monitoring with Prometheus: Complete Observability Guide

Monitoring HAProxy is essential for maintaining reliable load balancing infrastructure. Prometheus provides powerful metrics collection, alerting capabilities, and seamless Grafana integration for visualizing HAProxy performance and health. Why Prometheus for HAProxy? Prometheus offers: * Pull-based metrics - Prometheus scrapes HAProxy metrics endpoints * Time-series database - Store historical data for trend analysis

By Patrick de Ruiter