From nginx to traefik (and solving X-Forwarded-For)

I recently switched my home setup from nginx to traefik. I had to get used to the new configuration styles and weird doc style of traefik, but suffice to say, I am happy at the end results.

Traefik ships with SNI, allowing me to snoop the connection and prevent unauthorized access to my file server. (One must have the correct hostname to see my shared photos.) My first action was to replace my edge nginx reverse proxy with traefik. If this was a success, then I would move onto replacing nginx in my k8s cluster. I was surprized that after zero OS tuning changes, traefik was able to serve pages faster for my k8s cluster. I initially thought that traefik would perform marginally slower than nginx due to additional HostSNI checks, but that was totally disproved by the first page load.

After switching to traefik inside my k8s cluster, performance improved even more. However, a new problem emerged: I was getting the IPs of my internal traffic logged instead of the real origin IP. While I would not normally care, malicious bots were attempting to brute force my WordPress installations, and I need to block their IPs.

I was able to get the X-Forwarded-For header populated with the true ip by enabling the Proxy Protocol on my edge traefik, setting externalTrafficPolicy: Local on my traefik k8s service, and finally telling traefik to accept the proxyProtocol information from the edge server.

# Edge Router providers.yaml
...
tcp:
  routers:
    to-web:
      service: web
      rule: HostSNIRegexp(`www.andrewwippler.com`, `andrewwippler.com`, ...)
      entryPoints:
        - web
    to-websecure:
      service: websecure
      rule: HostSNIRegexp(`www.andrewwippler.com`, `andrewwippler.com`, ...)
      entryPoints:
        - websecure
      tls: 
        passthrough: true
services:
    web:
      loadBalancer:
        proxyProtocol:
          version: 2
        servers:
        - ...
    websecure:
      loadBalancer:
        proxyProtocol:
          version: 2
        servers:
        - ...
...

# traefik k8s deployment and service
# missing service account definition and CRDs
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik
spec:
  replicas: 3 #<-- one for each of my nodes, for failover
  strategy: #<-- forces k8s to update pods on configuration change
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      affinity: #<-- one replica per node
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - traefik
            topologyKey: "kubernetes.io/hostname"
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.10.3
          args:
            - --entrypoints.web.address=:80
            - --entrypoints.websecure.address=:443
            - --entrypoints.websecure.http.tls
            - --entrypoints.websecure.http3
            - --experimental.http3=true
            - --providers.kubernetesingress
            - --providers.kubernetescrd
            - --log.level=DEBUG
            - --entrypoints.web.http.redirections.entrypoint.scheme=https
            - --entrypoints.web.http.redirections.entrypoint.to=websecure
            # - --accesslog
            - --entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 #<-- local CIDRs
            - --entryPoints.websecure.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 #<-- local CIDRs
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: NodePort
  selector:
    app: traefik
  ports:
    - protocol: TCP
      port: 80
      name: web
      targetPort: 80
    - protocol: TCP
      port: 443
      name: websecure
      targetPort: 443
  externalTrafficPolicy: Local # <--- changed