HAProxy HTTP Headers Debugging: A Complete Troubleshooting Guide
When applications misbehave behind a load balancer, HTTP headers are often the culprit. This guide shows you how to debug, inspect, and manipulate HTTP headers in HAProxy to troubleshoot web application issues.
Why Header Debugging Matters
HTTP headers control critical aspects of web traffic:
- Client IP forwarding (X-Forwarded-For)
- Protocol information (X-Forwarded-Proto)
- Host routing and virtual hosts
- Session persistence (cookies)
- Caching behavior
- Security policies (CORS, CSP, HSTS)
- Authentication tokens
When these headers are wrong or missing, applications break in mysterious ways.
Logging Headers
Basic HTTP Logging
defaults
mode http
option httplog
log global
Detailed Custom Logging
frontend http_frontend
bind *:80
# Capture specific headers for logging
capture request header Host len 50
capture request header User-Agent len 100
capture request header X-Forwarded-For len 50
capture request header Authorization len 100
capture request header Content-Type len 50
capture response header Content-Type len 50
capture response header Location len 200
capture response header Set-Cookie len 100
default_backend web_servers
Custom Log Format with Headers
frontend http_frontend
bind *:80
# Define captures
capture request header Host len 50
capture request header X-Request-ID len 50
capture response header X-Response-Time len 20
# Custom log format
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
default_backend web_servers
Log format variables:
%hr: Captured request headers%hs: Captured response headers%{+Q}r: Quoted HTTP request line%ST: HTTP status code%TR: Request receive time%Ta: Total active time
Logging All Headers
frontend debug_frontend
bind *:8080
mode http
# Log all request headers
http-request capture req.hdrs len 1024
# Log format with all headers
log-format "%ci [%t] %r headers:{%[capture.req.hdr(0)]}"
default_backend web_servers
Inspecting Headers in Real-Time
Using the Stats Socket
# Show current sessions with header info
echo "show sess" | socat stdio /run/haproxy/admin.sock
# Show detailed session info
echo "show sess <session_id>" | socat stdio /run/haproxy/admin.sock
Debug Mode
Run HAProxy in debug mode to see all traffic:
# Stop HAProxy service
systemctl stop haproxy
# Run in foreground with debug
haproxy -d -f /etc/haproxy/haproxy.cfg
This outputs all headers to stdout.
Mirror Traffic for Debugging
frontend http_frontend
bind *:80
# Mirror traffic to debug backend
http-request set-header X-Mirror true
use_backend debug_backend if { path_beg /api }
default_backend web_servers
backend debug_backend
server debug 127.0.0.1:9999
Run a simple debug server:
# Using netcat
while true; do nc -l -p 9999 -c 'cat'; done
# Using Python
python3 -c "
import http.server
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
print(f'Path: {self.path}')
print('Headers:')
for h, v in self.headers.items():
print(f' {h}: {v}')
self.send_response(200)
self.end_headers()
self.wfile.write(b'OK')
do_POST = do_GET
http.server.HTTPServer(('', 9999), Handler).serve_forever()
"
Adding and Modifying Request Headers
Setting Headers
frontend http_frontend
bind *:80
# Add static header
http-request set-header X-Forwarded-Proto http
# Add header from variable
http-request set-header X-Client-IP %[src]
# Add header with HAProxy version
http-request set-header X-Proxy-Version %[var(proc.version)]
# Add timestamp
http-request set-header X-Request-Start %[date()]
default_backend web_servers
Common Forwarding Headers
frontend http_frontend
bind *:80
# X-Forwarded-For (client IP)
http-request set-header X-Forwarded-For %[src]
# Or append to existing
http-request set-header X-Forwarded-For %[req.hdr(X-Forwarded-For)],%[src] if { req.hdr(X-Forwarded-For) -m found }
http-request set-header X-Forwarded-For %[src] unless { req.hdr(X-Forwarded-For) -m found }
# X-Real-IP
http-request set-header X-Real-IP %[src]
# X-Forwarded-Proto
http-request set-header X-Forwarded-Proto http
# X-Forwarded-Host
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
# X-Forwarded-Port
http-request set-header X-Forwarded-Port %[dst_port]
default_backend web_servers
frontend https_frontend
bind *:443 ssl crt /etc/haproxy/certs/
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Proto https
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Forwarded-Port 443
default_backend web_servers
Adding Request ID for Tracing
frontend http_frontend
bind *:80
# Generate unique request ID if not present
unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
unique-id-header X-Request-ID
# Or use UUID format
http-request set-header X-Request-ID %[uuid()] unless { req.hdr(X-Request-ID) -m found }
default_backend web_servers
Conditional Header Setting
frontend http_frontend
bind *:80
# Set header based on path
http-request set-header X-API-Version v2 if { path_beg /api/v2 }
http-request set-header X-API-Version v1 if { path_beg /api/v1 }
# Set header based on client IP
acl internal_net src 192.168.0.0/16 10.0.0.0/8
http-request set-header X-Internal-Request true if internal_net
# Set header based on user agent
acl is_mobile req.hdr(User-Agent) -m sub -i mobile android iphone
http-request set-header X-Device-Type mobile if is_mobile
http-request set-header X-Device-Type desktop unless is_mobile
default_backend web_servers
Modifying Response Headers
Adding Security Headers
frontend http_frontend
bind *:443 ssl crt /etc/haproxy/certs/
# HSTS
http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
# Content Security Policy
http-response set-header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'"
# X-Content-Type-Options
http-response set-header X-Content-Type-Options nosniff
# X-Frame-Options
http-response set-header X-Frame-Options SAMEORIGIN
# X-XSS-Protection
http-response set-header X-XSS-Protection "1; mode=block"
# Referrer Policy
http-response set-header Referrer-Policy strict-origin-when-cross-origin
# Permissions Policy
http-response set-header Permissions-Policy "geolocation=(), microphone=(), camera=()"
default_backend web_servers
CORS Headers
frontend http_frontend
bind *:80
# Handle CORS preflight
acl is_options method OPTIONS
acl is_cors_request req.hdr(Origin) -m found
# Return 204 for preflight
http-request return status 204 if is_options is_cors_request
# Add CORS headers to responses
http-response set-header Access-Control-Allow-Origin "*" if is_cors_request
http-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" if is_cors_request
http-response set-header Access-Control-Allow-Headers "Content-Type, Authorization, X-Requested-With" if is_cors_request
http-response set-header Access-Control-Max-Age "86400" if is_cors_request
default_backend web_servers
Conditional CORS by Origin
frontend http_frontend
bind *:80
# Define allowed origins
acl allowed_origin req.hdr(Origin) -i https://example.com https://app.example.com
# Echo back the allowed origin
http-response set-header Access-Control-Allow-Origin %[req.hdr(Origin)] if allowed_origin
http-response set-header Access-Control-Allow-Credentials true if allowed_origin
default_backend web_servers
Deleting Headers
Remove Sensitive Headers
frontend http_frontend
bind *:80
# Remove potentially dangerous headers from requests
http-request del-header X-Forwarded-For
http-request del-header X-Real-IP
http-request del-header X-Forwarded-Proto
http-request del-header X-Forwarded-Host
# Now add our own trusted values
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Proto http
default_backend web_servers
Remove Headers from Responses
backend web_servers
# Remove server version headers
http-response del-header Server
http-response del-header X-Powered-By
http-response del-header X-AspNet-Version
http-response del-header X-AspNetMvc-Version
# Remove debug headers
http-response del-header X-Debug-Token
http-response del-header X-Debug-Token-Link
server web1 192.168.1.10:8080 check
Header-Based Routing
Route by Host Header
frontend http_frontend
bind *:80
# ACLs based on Host header
acl host_api hdr(host) -i api.example.com
acl host_admin hdr(host) -i admin.example.com
acl host_www hdr(host) -i www.example.com example.com
use_backend api_servers if host_api
use_backend admin_servers if host_admin
use_backend www_servers if host_www
default_backend www_servers
Route by Custom Header
frontend http_frontend
bind *:80
# Route based on X-Version header
acl version_v2 req.hdr(X-Version) -i v2
acl version_v1 req.hdr(X-Version) -i v1
use_backend api_v2 if version_v2
use_backend api_v1 if version_v1
default_backend api_v2
Route by Accept Header
frontend http_frontend
bind *:80
acl accepts_json req.hdr(Accept) -m sub application/json
acl accepts_xml req.hdr(Accept) -m sub application/xml
use_backend json_api if accepts_json
use_backend xml_api if accepts_xml
default_backend json_api
Debugging Specific Issues
Session Persistence Problems
backend web_servers
balance roundrobin
cookie SERVERID insert indirect nocache
# Log cookie info
capture cookie SERVERID len 32
server web1 192.168.1.10:8080 cookie s1 check
server web2 192.168.1.11:8080 cookie s2 check
frontend http_frontend
bind *:80
# Capture cookies for debugging
capture request header Cookie len 200
capture response header Set-Cookie len 200
# Log format showing cookies
log-format "%ci [%t] %r Cookie:{%[capture.req.hdr(0)]} Set-Cookie:{%[capture.res.hdr(0)]}"
default_backend web_servers
Authentication Header Issues
frontend http_frontend
bind *:80
# Capture auth header (be careful with logging sensitive data!)
capture request header Authorization len 100
# Check if auth header exists
acl has_auth req.hdr(Authorization) -m found
acl valid_bearer req.hdr(Authorization) -m beg Bearer
# Add debug header
http-request set-header X-Auth-Debug has_auth if has_auth
http-request set-header X-Auth-Debug missing unless has_auth
http-request set-header X-Auth-Type bearer if valid_bearer
http-request set-header X-Auth-Type other if has_auth !valid_bearer
default_backend web_servers
Content-Type Mismatches
frontend http_frontend
bind *:80
# Capture content types
capture request header Content-Type len 50
capture response header Content-Type len 50
# Add debug info
http-request set-header X-Request-Content-Type %[req.hdr(Content-Type)]
http-response set-header X-Response-Content-Type %[res.hdr(Content-Type)]
default_backend web_servers
Caching Issues
frontend http_frontend
bind *:80
# Capture cache headers
capture response header Cache-Control len 100
capture response header ETag len 50
capture request header If-None-Match len 50
capture request header If-Modified-Since len 50
default_backend web_servers
backend web_servers
# Force no caching for debugging
http-response set-header Cache-Control "no-store, no-cache, must-revalidate"
http-response del-header ETag
http-response del-header Last-Modified
server web1 192.168.1.10:8080 check
Creating a Debug Endpoint
Echo All Headers
frontend http_frontend
bind *:80
# Debug endpoint that returns request info
acl is_debug path /debug/headers
http-request return status 200 content-type "text/plain" lf-string "Client IP: %[src]\nMethod: %[method]\nPath: %[path]\nQuery: %[query]\nHost: %[req.hdr(Host)]\nUser-Agent: %[req.hdr(User-Agent)]\nAccept: %[req.hdr(Accept)]\nContent-Type: %[req.hdr(Content-Type)]\nAuthorization: %[req.hdr(Authorization)]\nX-Forwarded-For: %[req.hdr(X-Forwarded-For)]\nCookies: %[req.hdr(Cookie)]" if is_debug
default_backend web_servers
JSON Debug Response
frontend http_frontend
bind *:80
acl is_debug path /debug/info
http-request return status 200 content-type "application/json" lf-string '{"client_ip":"%[src]","method":"%[method]","path":"%[path]","host":"%[req.hdr(Host)]","user_agent":"%[req.hdr(User-Agent)]","timestamp":"%[date]"}' if is_debug
default_backend web_servers
Lua Scripting for Advanced Debugging
Install Lua Support
apt install haproxy-lua
Header Dump Lua Script
-- /etc/haproxy/lua/debug-headers.lua
core.register_service("debug_headers", "http", function(applet)
local response = "Request Headers:\n"
response = response .. "================\n"
-- Get all request headers
local headers = applet.headers
for name, values in pairs(headers) do
for _, value in pairs(values) do
response = response .. name .. ": " .. value .. "\n"
end
end
response = response .. "\nConnection Info:\n"
response = response .. "================\n"
response = response .. "Client IP: " .. applet.sf:src() .. "\n"
response = response .. "Client Port: " .. applet.sf:src_port() .. "\n"
response = response .. "Server IP: " .. applet.sf:dst() .. "\n"
response = response .. "Server Port: " .. applet.sf:dst_port() .. "\n"
response = response .. "Method: " .. applet.method .. "\n"
response = response .. "Path: " .. applet.path .. "\n"
response = response .. "Query: " .. (applet.qs or "") .. "\n"
applet:set_status(200)
applet:add_header("Content-Type", "text/plain")
applet:add_header("Content-Length", string.len(response))
applet:start_response()
applet:send(response)
end)
Use in HAProxy Config
global
lua-load /etc/haproxy/lua/debug-headers.lua
frontend http_frontend
bind *:80
acl is_debug path /debug/headers
http-request use-service lua.debug_headers if is_debug
default_backend web_servers
Complete Debug Configuration
# /etc/haproxy/haproxy.cfg
global
log /dev/log local0 debug
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
user haproxy
group haproxy
daemon
# Enable Lua for advanced debugging
# lua-load /etc/haproxy/lua/debug-headers.lua
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
timeout connect 5s
timeout client 50s
timeout server 50s
# Stats with header info
listen stats
bind *:8404
stats enable
stats uri /stats
stats refresh 5s
http-request use-service prometheus-exporter if { path /metrics }
frontend http_frontend
bind *:80
# Generate unique request ID
unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
unique-id-header X-Request-ID
# Capture headers for logging
capture request header Host len 50
capture request header User-Agent len 100
capture request header X-Forwarded-For len 50
capture request header Authorization len 20
capture request header Content-Type len 50
capture request header Cookie len 100
capture request header X-Request-ID len 50
capture response header Content-Type len 50
capture response header Location len 200
capture response header Set-Cookie len 100
capture response header Cache-Control len 50
# Detailed log format
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq {%hrl} {%hsl} %{+Q}r"
# Remove potentially spoofed headers
http-request del-header X-Forwarded-For
http-request del-header X-Real-IP
http-request del-header X-Forwarded-Proto
# Add trusted forwarding headers
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Real-IP %[src]
http-request set-header X-Forwarded-Proto http
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Forwarded-Port %[dst_port]
# Debug endpoints
acl is_debug_headers path /debug/headers
acl is_debug_json path /debug/json
http-request return status 200 content-type "text/plain" lf-string "Client: %[src]:%[src_port]\nHost: %[req.hdr(Host)]\nMethod: %[method]\nPath: %[path]\nQuery: %[query]\nProtocol: %[req.ver]\nUser-Agent: %[req.hdr(User-Agent)]\nAccept: %[req.hdr(Accept)]\nContent-Type: %[req.hdr(Content-Type)]\nX-Request-ID: %[req.hdr(X-Request-ID)]\nCookies: %[req.hdr(Cookie)]" if is_debug_headers
http-request return status 200 content-type "application/json" lf-string '{"client_ip":"%[src]","host":"%[req.hdr(Host)]","method":"%[method]","path":"%[path]","request_id":"%[req.hdr(X-Request-ID)]"}' if is_debug_json
default_backend web_servers
backend web_servers
balance roundrobin
option httpchk GET /health
# Remove server identification from responses
http-response del-header Server
http-response del-header X-Powered-By
# Add debug headers to response
http-response set-header X-Backend-Server %s
http-response set-header X-Request-ID %[req.hdr(X-Request-ID)]
http-response set-header X-Response-Time %Ta
server web1 192.168.1.10:8080 check
server web2 192.168.1.11:8080 check
Quick Debugging Commands
# Watch HAProxy logs in real-time
tail -f /var/log/haproxy.log | grep -E "(Host|User-Agent|X-Request-ID)"
# Test with curl and show headers
curl -v http://example.com/
curl -H "X-Custom: test" http://example.com/debug/headers
# Test from specific IP (if accessible)
curl -H "X-Forwarded-For: 1.2.3.4" http://example.com/debug/headers
# Check HAProxy session info
echo "show sess" | socat stdio /run/haproxy/admin.sock
# Show backend status
echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d',' -f1,2,18
Next Steps
This post covered HTTP header debugging in HAProxy. In the next post, we'll explore mTLS (mutual TLS) for securing API access, building on the TLS knowledge from previous posts.
Summary
Key debugging techniques for HAProxy headers:
- Use
capture request/response headerfor logging - Create debug endpoints with
http-request return - Remove untrusted headers before adding your own
- Use
http-request/response set-headerfor manipulation - Add X-Request-ID for tracing across services
- Use Lua for advanced header inspection
- Always secure sensitive header logging in production
Effective header debugging is essential for troubleshooting modern web applications behind load balancers.
