From nginx to traefik (and solving X-Forwarded-For)
I recently switched my home setup from nginx to traefik. I had to get used to the new configuration styles and weird doc style of traefik, but suffice to say, I am happy at the end results.
Traefik ships with SNI, allowing me to snoop the connection and prevent unauthorized access to my file server. (One must have the correct hostname to see my shared photos.) My first action was to replace my edge nginx reverse proxy with traefik. If this was a success, then I would move onto replacing nginx in my k8s cluster. I was surprized that after zero OS tuning changes, traefik was able to serve pages faster for my k8s cluster. I initially thought that traefik would perform marginally slower than nginx due to additional HostSNI checks, but that was totally disproved by the first page load.
After switching to traefik inside my k8s cluster, performance improved even more. However, a new problem emerged: I was getting the IPs of my internal traffic logged instead of the real origin IP. While I would not normally care, malicious bots were attempting to brute force my WordPress installations, and I need to block their IPs.
I was able to get the X-Forwarded-For header populated with the true ip by enabling the Proxy Protocol on my edge traefik, setting externalTrafficPolicy: Local on my traefik k8s service, and finally telling traefik to accept the proxyProtocol information from the edge server.
# Edge Router providers.yaml
...
tcp:
routers:
to-web:
service: web
rule: HostSNIRegexp(`www.andrewwippler.com`, `andrewwippler.com`, ...)
entryPoints:
- web
to-websecure:
service: websecure
rule: HostSNIRegexp(`www.andrewwippler.com`, `andrewwippler.com`, ...)
entryPoints:
- websecure
tls:
passthrough: true
services:
web:
loadBalancer:
proxyProtocol:
version: 2
servers:
- ...
websecure:
loadBalancer:
proxyProtocol:
version: 2
servers:
- ...
...# traefik k8s deployment and service
# missing service account definition and CRDs
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
spec:
replicas: 3 #<-- one for each of my nodes, for failover
strategy: #<-- forces k8s to update pods on configuration change
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
affinity: #<-- one replica per node
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- traefik
topologyKey: "kubernetes.io/hostname"
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.10.3
args:
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.websecure.http.tls
- --entrypoints.websecure.http3
- --experimental.http3=true
- --providers.kubernetesingress
- --providers.kubernetescrd
- --log.level=DEBUG
- --entrypoints.web.http.redirections.entrypoint.scheme=https
- --entrypoints.web.http.redirections.entrypoint.to=websecure
# - --accesslog
- --entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 #<-- local CIDRs
- --entryPoints.websecure.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 #<-- local CIDRs
ports:
- name: web
containerPort: 80
- name: websecure
containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
type: NodePort
selector:
app: traefik
ports:
- protocol: TCP
port: 80
name: web
targetPort: 80
- protocol: TCP
port: 443
name: websecure
targetPort: 443
externalTrafficPolicy: Local # <--- changed
