haproxy

HAProxy and Consul: Dynamic Service Discovery for Load Balancing

Patrick de Ruiter

Patrick de Ruiter

— 7 min read
Share
HAProxy and Consul: Dynamic Service Discovery for Load Balancing

In dynamic environments where services scale up and down automatically, static HAProxy configurations become a management burden. Consul integration enables HAProxy to automatically discover and route traffic to healthy service instances without manual configuration updates.

Why Consul + HAProxy?

The combination provides:

  • Automatic service registration - Services announce themselves to Consul
  • Health-aware routing - Only healthy instances receive traffic
  • Dynamic scaling - New instances are automatically added to load balancing
  • Multi-datacenter - Route traffic across datacenters with Consul federation
  • No downtime updates - Configuration changes without service interruption

Architecture Overview

                    ┌─────────────────────────────────┐
                    │           Consul Cluster        │
                    │  ┌───────┐ ┌───────┐ ┌───────┐ │
                    │  │Server1│ │Server2│ │Server3│ │
                    │  └───────┘ └───────┘ └───────┘ │
                    └────────────────┬────────────────┘
                                     │
              ┌──────────────────────┼──────────────────────┐
              │                      │                      │
              ▼                      ▼                      ▼
      ┌───────────────┐      ┌───────────────┐      ┌───────────────┐
      │   Service A   │      │   Service B   │      │   Service C   │
      │ + Consul Agent│      │ + Consul Agent│      │ + Consul Agent│
      └───────────────┘      └───────────────┘      └───────────────┘
              │                      │                      │
              └──────────────────────┼──────────────────────┘
                                     │
                                     ▼
                          ┌─────────────────────┐
                          │      HAProxy        │
                          │  + consul-template  │
                          └─────────────────────┘
                                     │
                                     ▼
                               Clients/Users

Setting Up Consul

Consul Server Installation

# Download Consul
CONSUL_VERSION="1.17.0"
wget https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip
unzip consul_${CONSUL_VERSION}_linux_amd64.zip
sudo mv consul /usr/local/bin/

# Create directories
sudo mkdir -p /etc/consul.d /var/lib/consul
sudo useradd --system --home /var/lib/consul --shell /bin/false consul
sudo chown -R consul:consul /etc/consul.d /var/lib/consul

Consul Server Configuration

Create /etc/consul.d/consul.hcl:

datacenter = "dc1"
node_name = "consul-server-1"
data_dir = "/var/lib/consul"
log_level = "INFO"

server = true
bootstrap_expect = 3

bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
client_addr = "0.0.0.0"

ui_config {
  enabled = true
}

connect {
  enabled = true
}

addresses {
  http = "0.0.0.0"
  grpc = "0.0.0.0"
}

ports {
  http = 8500
  grpc = 8502
  grpc_tls = 8503
}

retry_join = [
  "consul-server-1.example.com",
  "consul-server-2.example.com",
  "consul-server-3.example.com"
]

acl {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

Consul Agent Configuration (for service nodes)

Create /etc/consul.d/consul.hcl on service nodes:

datacenter = "dc1"
node_name = "service-node-1"
data_dir = "/var/lib/consul"
log_level = "INFO"

server = false

bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
client_addr = "127.0.0.1"

retry_join = [
  "consul-server-1.example.com",
  "consul-server-2.example.com",
  "consul-server-3.example.com"
]

Systemd Service

Create /etc/systemd/system/consul.service:

[Unit]
Description=Consul Service Discovery Agent
Documentation=https://www.consul.io/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Registering Services with Consul

Service Registration File

Create /etc/consul.d/web-service.hcl:

service {
  name = "web"
  port = 8080
  tags = ["http", "production"]
  
  meta {
    version = "1.0.0"
    environment = "production"
  }
  
  check {
    name = "HTTP Health Check"
    http = "http://localhost:8080/health"
    interval = "10s"
    timeout = "5s"
    
    success_before_passing = 2
    failures_before_critical = 3
  }
  
  check {
    name = "TCP Port Check"
    tcp = "localhost:8080"
    interval = "5s"
    timeout = "2s"
  }
}

API Service Registration

service {
  name = "api"
  port = 3000
  tags = ["http", "api", "v2"]
  
  weights {
    passing = 100
    warning = 50
  }
  
  check {
    name = "API Health"
    http = "http://localhost:3000/api/health"
    interval = "10s"
    timeout = "3s"
    header {
      Content-Type = ["application/json"]
    }
  }
}

Dynamic Registration via HTTP API

# Register a service dynamically
curl -X PUT http://localhost:8500/v1/agent/service/register \
  -H "Content-Type: application/json" \
  -d '{
    "ID": "web-1",
    "Name": "web",
    "Port": 8080,
    "Tags": ["http", "production"],
    "Check": {
      "HTTP": "http://localhost:8080/health",
      "Interval": "10s"
    }
  }'

# Deregister a service
curl -X PUT http://localhost:8500/v1/agent/service/deregister/web-1

Consul Template for HAProxy

Installing Consul Template

CT_VERSION="0.35.0"
wget https://releases.hashicorp.com/consul-template/${CT_VERSION}/consul-template_${CT_VERSION}_linux_amd64.zip
unzip consul-template_${CT_VERSION}_linux_amd64.zip
sudo mv consul-template /usr/local/bin/

Consul Template Configuration

Create /etc/consul-template/config.hcl:

consul {
  address = "127.0.0.1:8500"
  
  retry {
    enabled = true
    attempts = 5
    backoff = "250ms"
  }
}

template {
  source = "/etc/consul-template/haproxy.ctmpl"
  destination = "/etc/haproxy/haproxy.cfg"
  perms = 0644
  
  command = "systemctl reload haproxy"
  command_timeout = "60s"
  
  backup = true
  
  wait {
    min = "5s"
    max = "10s"
  }
}

log_level = "info"

HAProxy Template

Create /etc/consul-template/haproxy.ctmpl:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 4096

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    option http-server-close
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

# Stats endpoint
frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

# Main HTTP Frontend
frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/
    
    # ACL definitions based on hostname
    acl host_web hdr(host) -i www.example.com example.com
    acl host_api hdr(host) -i api.example.com
    acl host_admin hdr(host) -i admin.example.com
    
    # Backend selection
    use_backend web_servers if host_web
    use_backend api_servers if host_api
    use_backend admin_servers if host_admin
    
    default_backend web_servers

# Web Servers Backend - dynamically populated from Consul
backend web_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    {{- range service "web" }}
    server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check weight {{ if .Weights.Passing }}{{ .Weights.Passing }}{{ else }}100{{ end }}{{ if not .Checks.AggregatedStatus | eq "passing" }} disabled{{ end }}
    {{- end }}

# API Servers Backend
backend api_servers
    balance leastconn
    option httpchk GET /api/health
    http-check expect status 200
    {{- range service "api" }}
    server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check weight {{ if .Weights.Passing }}{{ .Weights.Passing }}{{ else }}100{{ end }}
    {{- end }}

# Admin Servers Backend
backend admin_servers
    balance roundrobin
    option httpchk GET /health
    {{- range service "admin" }}
    server {{ .Node }}-{{ .Port }} {{ .Address }}:{{ .Port }} check
    {{- end }}

Advanced Template with Tags and Filtering

# Production web servers only (filtered by tag)
backend web_production
    balance roundrobin
    option httpchk GET /health
    {{- range service "web|production" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

# Staging web servers
backend web_staging
    balance roundrobin
    option httpchk GET /health
    {{- range service "web|staging" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

# Services in specific datacenter
backend api_dc1
    balance leastconn
    {{- range service "api" "dc1" }}
    server {{ .Node }}-{{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

# Multi-datacenter with failover
backend api_multi_dc
    balance roundrobin
    {{- range service "api" "dc1" }}
    server {{ .Node }}-{{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}
    {{- range service "api" "dc2" }}
    server {{ .Node }}-{{ .ID }} {{ .Address }}:{{ .Port }} check backup
    {{- end }}

Template with Service Metadata

backend web_servers
    balance roundrobin
    {{- range service "web" }}
    {{- if .ServiceMeta.version | eq "2.0" }}
    # Version 2.0 servers get more weight
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight 150
    {{- else }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight 100
    {{- end }}
    {{- end }}

Consul Template Systemd Service

Create /etc/systemd/system/consul-template.service:

[Unit]
Description=Consul Template
After=network-online.target consul.service
Wants=network-online.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/consul-template -config=/etc/consul-template/config.hcl
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Health Checks and Service Discovery

Consul Health Check Types

service {
  name = "web"
  port = 8080
  
  # HTTP check
  check {
    name = "HTTP Health"
    http = "http://localhost:8080/health"
    method = "GET"
    interval = "10s"
    timeout = "5s"
  }
  
  # TCP check
  check {
    name = "TCP Port"
    tcp = "localhost:8080"
    interval = "5s"
    timeout = "2s"
  }
  
  # Script check
  check {
    name = "Script Check"
    args = ["/usr/local/bin/health-check.sh"]
    interval = "30s"
    timeout = "10s"
  }
  
  # gRPC check
  check {
    name = "gRPC Health"
    grpc = "localhost:9090"
    grpc_use_tls = true
    interval = "10s"
  }
  
  # TTL check (service must update)
  check {
    name = "TTL Check"
    ttl = "30s"
    status = "passing"
  }
}

Updating TTL Check

# Pass TTL check
curl -X PUT http://localhost:8500/v1/agent/check/pass/service:web-1

# Fail TTL check
curl -X PUT http://localhost:8500/v1/agent/check/fail/service:web-1

# Warn TTL check
curl -X PUT http://localhost:8500/v1/agent/check/warn/service:web-1

Advanced HAProxy + Consul Patterns

Blue-Green Deployments

# Template for blue-green deployment
frontend http_front
    bind *:80
    
    # Read active environment from Consul KV
    {{- $active := key "app/active-environment" }}
    
    default_backend web_{{ $active }}

backend web_blue
    balance roundrobin
    {{- range service "web|blue" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

backend web_green
    balance roundrobin
    {{- range service "web|green" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check
    {{- end }}

Switch environments:

# Switch to green
consul kv put app/active-environment green

# Switch to blue
consul kv put app/active-environment blue

Canary Deployments with Weights

backend web_servers
    balance roundrobin
    
    {{- $canary_weight := keyOrDefault "app/canary-weight" "10" }}
    
    # Stable servers
    {{- range service "web|stable" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight {{ subtract 100 (parseInt $canary_weight) }}
    {{- end }}
    
    # Canary servers
    {{- range service "web|canary" }}
    server {{ .ID }} {{ .Address }}:{{ .Port }} check weight {{ $canary_weight }}
    {{- end }}

Rate Limiting per Service

frontend http_front
    bind *:80
    
    {{- $rate_limit := keyOrDefault "app/rate-limit" "100" }}
    
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt {{ $rate_limit }} }
    
    default_backend web_servers

Maintenance Mode

frontend http_front
    bind *:80
    
    {{- $maintenance := keyOrDefault "app/maintenance" "false" }}
    
    {{- if eq $maintenance "true" }}
    # Maintenance mode - serve static page
    http-request deny deny_status 503
    {{- else }}
    default_backend web_servers
    {{- end }}

Monitoring and Observability

Consul Service Health Queries

# List all healthy web services
curl http://localhost:8500/v1/health/service/web?passing=true | jq .

# Get service catalog
curl http://localhost:8500/v1/catalog/services | jq .

# Watch for changes
watch -n 2 'curl -s http://localhost:8500/v1/health/service/web | jq length'

HAProxy Stats Integration

frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    
    # Prometheus metrics endpoint
    http-request use-service prometheus-exporter if { path /metrics }

Troubleshooting

Common Issues

  1. Services not appearing in HAProxy
    • Check Consul agent logs: journalctl -u consul
    • Verify service health: consul catalog services
    • Check consul-template logs: journalctl -u consul-template
  2. HAProxy not reloading
    • Verify template syntax: consul-template -template "haproxy.ctmpl:test.cfg" -once -dry
    • Check HAProxy config: haproxy -c -f /etc/haproxy/haproxy.cfg
  3. Services marked unhealthy
    • Check health check status: consul catalog service web
    • Review check definitions in service registration

Debug Commands

# Test template rendering
consul-template -template "/etc/consul-template/haproxy.ctmpl:/tmp/haproxy-test.cfg" -once -dry

# Verify Consul connectivity
consul members

# List all services
consul catalog services

# Get service details
consul catalog service web

# Check service health
consul health checks web

Security Considerations

Consul ACL Tokens

# consul-template config with ACL
consul {
  address = "127.0.0.1:8500"
  token = "your-acl-token"
}

TLS Configuration

consul {
  address = "consul.example.com:8501"
  
  ssl {
    enabled = true
    verify = true
    ca_cert = "/etc/consul-template/ca.pem"
    cert = "/etc/consul-template/client.pem"
    key = "/etc/consul-template/client-key.pem"
  }
}

Next Steps

With Consul service discovery integrated, consider:

  • Prometheus Monitoring - Add metrics collection for HAProxy and Consul
  • Consul Connect - Implement service mesh with automatic mTLS
  • Multi-datacenter - Extend to multiple Consul datacenters for geo-routing

Consul and HAProxy together create a powerful, self-healing infrastructure where services automatically join and leave the load balancer pool based on their health status.

Read more