Andrew Wippler's Sketchpad - k8s

From nginx to traefik (and solving X-Forwarded-For)

2023-07-18T11:23:43+00:00

I recently switched my home setup from nginx to traefik. I had to get used to the new configuration styles and weird doc style of traefik, but suffice to say, I am happy at the end results.

Traefik ships with SNI, allowing me to snoop the connection and prevent unauthorized access to my file server. (One must have the correct hostname to see my shared photos.) My first action was to replace my edge nginx reverse proxy with traefik. If this was a success, then I would move onto replacing nginx in my k8s cluster. I was surprized that after zero OS tuning changes, traefik was able to serve pages faster for my k8s cluster. I initially thought that traefik would perform marginally slower than nginx due to additional HostSNI checks, but that was totally disproved by the first page load.

After switching to traefik inside my k8s cluster, performance improved even more. However, a new problem emerged: I was getting the IPs of my internal traffic logged instead of the real origin IP. While I would not normally care, malicious bots were attempting to brute force my WordPress installations, and I need to block their IPs.

I was able to get the X-Forwarded-For header populated with the true ip by enabling the Proxy Protocol on my edge traefik, setting externalTrafficPolicy: Local on my traefik k8s service, and finally telling traefik to accept the proxyProtocol information from the edge server.

# Edge Router providers.yaml
</span>...
</span>tcp:
</span>  routers:
</span>    to-web:
</span>      service: web
</span>      rule: HostSNIRegexp(`www.andrewwippler.com`, `andrewwippler.com`, ...)
</span>      entryPoints:
</span>        - web
</span>    to-websecure:
</span>      service: websecure
</span>      rule: HostSNIRegexp(`www.andrewwippler.com`, `andrewwippler.com`, ...)
</span>      entryPoints:
</span>        - websecure
</span>      tls:
</span>        passthrough: true
</span>services:
</span>    web:
</span>      loadBalancer:
</span>        proxyProtocol:
</span>          version: 2
</span>        servers:
</span>        - ...
</span>    websecure:
</span>      loadBalancer:
</span>        proxyProtocol:
</span>          version: 2
</span>        servers:
</span>        - ...
</span>...
</span></code></pre>
# traefik k8s deployment and service
</span># missing service account definition and CRDs
</span>apiVersion: apps/v1
</span>kind: Deployment
</span>metadata:
</span>  name: traefik
</span>spec:
</span>  replicas: 3 #<-- one for each of my nodes, for failover
</span>  strategy: #<-- forces k8s to update pods on configuration change
</span>    rollingUpdate:
</span>      maxSurge: 1
</span>      maxUnavailable: 1
</span>    type: RollingUpdate
</span>  selector:
</span>    matchLabels:
</span>      app: traefik
</span>  template:
</span>    metadata:
</span>      labels:
</span>        app: traefik
</span>    spec:
</span>      affinity: #<-- one replica per node
</span>        podAntiAffinity:
</span>          requiredDuringSchedulingIgnoredDuringExecution:
</span>          - labelSelector:
</span>              matchExpressions:
</span>              - key: app
</span>                operator: In
</span>                values:
</span>                - traefik
</span>            topologyKey: "kubernetes.io/hostname"
</span>      serviceAccountName: traefik-ingress-controller
</span>      containers:
</span>        - name: traefik
</span>          image: traefik:v2.10.3
</span>          args:
</span>            - --entrypoints.web.address=:80
</span>            - --entrypoints.websecure.address=:443
</span>            - --entrypoints.websecure.http.tls
</span>            - --entrypoints.websecure.http3
</span>            - --experimental.http3=true
</span>            - --providers.kubernetesingress
</span>            - --providers.kubernetescrd
</span>            - --log.level=DEBUG
</span>            - --entrypoints.web.http.redirections.entrypoint.scheme=https
</span>            - --entrypoints.web.http.redirections.entrypoint.to=websecure
</span>            # - --accesslog
</span>            - --entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 #<-- local CIDRs
</span>            - --entryPoints.websecure.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 #<-- local CIDRs
</span>          ports:
</span>            - name: web
</span>              containerPort: 80
</span>            - name: websecure
</span>              containerPort: 443
</span>---
</span>apiVersion: v1
</span>kind: Service
</span>metadata:
</span>  name: traefik
</span>spec:
</span>  type: NodePort
</span>  selector:
</span>    app: traefik
</span>  ports:
</span>    - protocol: TCP
</span>      port: 80
</span>      name: web
</span>      targetPort: 80
</span>    - protocol: TCP
</span>      port: 443
</span>      name: websecure
</span>      targetPort: 443
</span>  externalTrafficPolicy: Local # <--- changed
</span></code></pre>

Kubernetes: Heapster to Metrics Server 2018-07-09T13:58:16+00:00 I recently updated my kubernetes cluster from 1.10.2</code> to 1.11.0</code>. I noticed heapster was being deprecated and completely removed by version 1.13.0</code>. I thought this would be the perfect time to try out metrics-server. I had to download the git repo to apply the kubernetes yaml to my cluster. Since this is sometimes not as ideal as I would like, (I prefer kubectl apply -f http://</code> when it comes from a trusted source) I am writing the below for easy access in the future: kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/auth-delegator.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/auth-reader.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/metrics-apiservice.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/metrics-server-deployment.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/metrics-server-service.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/resource-reader.yaml </code></pre> Note: this is for clusters running v1.8.0 or greater. Jenkins-x on home kubernetes cluster 2018-06-06T12:15:42+00:00 Jenkins-x</a> appears to be the next big thing in CI/CD workflows - especially if you develop applications on kubernetes. There were a few tweaks I needed to do to set it up: I had to manually create Persistent Volumes (no big deal, below are what I have for my NFS share) apiVersion: v1 kind: PersistentVolume metadata: name: jenkins namespace: jx labels: app: jenkins spec: capacity: storage: 30Gi accessModes: - ReadWriteOnce nfs: server: 192.168.0.101 path: "/volume1/k8s/jenkins/jenkins" --- apiVersion: v1 kind: PersistentVolume metadata: name: jenkins-x-chartmuseum namespace: jx labels: app: jenkins-x-chartmuseum spec: capacity: storage: 100Gi accessModes: - ReadWriteOnce nfs: server: 192.168.0.101 path: "/volume1/k8s/jenkins/jenkins-x-chartmuseum" --- apiVersion: v1 kind: PersistentVolume metadata: name: jenkins-x-docker-registry namespace: jx labels: app: jenkins-x-docker-registry spec: capacity: storage: 30Gi accessModes: - ReadWriteOnce nfs: server: 192.168.0.101 path: "/volume1/k8s/jenkins/jenkins-x-docker-registry" --- apiVersion: v1 kind: PersistentVolume metadata: name: jenkins-x-mongodb namespace: jx labels: app: jenkins-x-mongodb spec: capacity: storage: 30Gi accessModes: - ReadWriteOnce nfs: server: 192.168.0.101 path: "/volume1/k8s/jenkins/jenkins-x-mongodb" --- apiVersion: v1 kind: PersistentVolume metadata: name: jenkins-x-nexus namespace: jx labels: app: jenkins-x-nexus spec: capacity: storage: 30Gi accessModes: - ReadWriteOnce nfs: server: 192.168.0.101 path: "/volume1/k8s/jenkins/jenkins-x-nexus" </code></pre> </li> I had to modify the install linejx install --ingress-namespace ingress-nginx --domain wplr.rocks --tls-acme true --skip-ingress</code> </li> I had to modify the jenkins-x-mongodb</code> deployment to use image mongo:3.6.5-jessie</code>. Still wonder why people use bitnami images. </li> I had to securityContext: runAsUser: 1024 </code></pre> on the jenkins-x-nexus</code> deployment. The container was trying to change permissions on my nfs mount. Not sure why my Synology NFS does not like permission changes. </li> </ol> Even after those changes, jenkins-x-monocular-ui</code> still fails to start -_- ... I have run out of time for now. More debugging to come later (MUCH MUCH later) Allowing outside access to Home Kubernetes Cluster 2018-04-03T15:36:25+00:00 After I created a home kubernetes cluster, I immediately wanted to allow external access to pods/services/ingresses hosted inside the cluster. One must be aware that in bare metal environments, there is no receiver of an api call to create a load balancer. Since there is not a scriptable environment available to kubernetes, kubernetes cannot request external IP addresses or provision resources that one has come to expect in cloud environments such as AWS. This is a huge bummer - especially since dynamically built environments are fun to have. To route traffic to web services inside of kubernetes, you have to options available: ingress</code> and service</code>. Services can be exposed via NodePort</code>, LoadBalancer</code>, or ClusterIP</code>. In bare metal, LoadBalancer</code> would never work (unless you coded your own API call to configure a load balancer outside of kubernetes). ClusterIP</code> might work if you want to manage a routing table somewhere inside your network, and NodePort</code> will work if you want to manage a port forwarding table on your router. None of these options are fun for home labs on bare metal. An Ingress</code> is like a layer 7 firewall in that it reads the hostname and path of the incoming HTTP request and can route to applicable services. This works great for a dynamic environment where I am going to host multiple http endpoints. The overall view of this traffic is going to be: Internet</code> > Router</code> > k8s Ingress</code> > k8s Service</code> > Pod(s)</code>. To create an ingress in kubernetes, you have to make it a Service</code>. In cloud environments, the Ingress</code> is created as type LoadBalancer</code> in home labs, we create this as type NodePort</code> and port forward on the router to any node in the kubernetes cluster. $ kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.102.173.184 80/TCP 3d ingress-nginx NodePort 10.110.162.247 80:30746/TCP,443:32641/TCP 3d </code></pre> In my home lab, I am port forwarding on my router 0.0.0.0:80 -> <any_node>:30746</code> and 0.0.0.0:443 -> <any_node>:32641</code>. Since I have a non-traditional home router (a Linux server with two nics</a>), I had to either enter these into iptables</code> or I could improve upon that by setting up a load balancer such as nginx</code>. nginx</code> will allow me to port-forward load balance across all my nodes and have an easy config file to edit. Because I also want to use cert-manager</code> with Let's Encrypt free SSLs, I chose to use the TCP stream server of nginx</code>. Another hiccup (so to speak) in home based labs, is that ISPs give DHCP addresses. So when my internet IP changes, I need to update the DNS of all my http endpoints. Rather than doing that, I have all my home urls (*.wplr.rocks</code>) CNAME to a single hostname which I update with a script with the correct IP. Home Kubernetes cluster 2018-04-02T15:26:52+00:00 So I admit it - I am completely obsessed with Kubernetes. All of the web app/api deployment challenges in the past 20 years have been somewhat solved with the Kubernetes orchestration and scheduling platform. Kubernetes brings fault-tolerance, and Highly available systems if set up correctly (i.e. use a Kubernetes cloud installer). I enjoy having the power to write yaml and apply it to infrastructure and it eventually becomes what I told it to be. No longer do I need to write the scripts to do it - it does it for me :) In my first kubernetes home cluster, I re-used my home router and my old media center as a single node + master, but I was hit by a 2 year old Kubernetes bug. It appears my old PC was out in the open. Since my 2 year old son likes to press power buttons, he came over and constantly pressed the power button on my Kubernetes master. This caused me to find a small mini computer that I could place in my cabinet out of view. I finally settled on this as my computer of choice</a>. At $150/each for 4 cores, 4GB RAM, and 32GB nve, I thought it was a good deal and ample to run a few containers per node with a nfs-backed storage array. These little machines booted via UEFI pxe (after pressing DEL for bios and selecting the Realtek boot in the save/exit page). I used this kickstart file which installed CentOS 7 minimal by passing in the ks</code> argument to a CentOS initramfs: After the servers installed, I ran this script: Note: for one of the three machines I received, I had to put modprobe br_netfilter</code> in /etc/rc.local</code> before /etc/sysctl.d/k8s.conf</code> would apply. kubernetes health check 2017-12-08T15:00:22+00:00 The day before thanksgiving, I was pondering an issue I was having. I was pinning a package to a specific version in my Docker container and the repository I grabbed it from stopped offering this specific version. This resulted in a container that Jenkins responded as being built correctly, but missing an integral package that allowed my application to function properly. This led me to believe I had to implement Puppet's Lumogon</a> in my Jenkins build process. Curious if anyone had something like this already developed, I headed over to github.com which eventually led me to compose this tweet: Step 1: Have k8s technical problem Step 2: Review @kelseyhightower</a>'s recent git pushes Step 3: Find: https://github.com/kelseyhightower/ato/blob/master/deployments/helloworld.yaml#L26-L29</a>Step 4: Realize this is exactly something I needed to implement Step 5: Implement feature Step 6: Enjoy thanksgiving without worry Step 7: Blog post — Andrew J. Wippler (@AndrewWippler) November 22, 2017</a> </blockquote> This readinessProbe</code> communicates with kubeproxy</code> to either allow a pod to be in service or out of service. At first I thought the readinessProbe</code> was a once-and-done check, but I found out later this is not the case. When a pod gets launched, kubernetes waits until the container is in the ready state. We can define what consists of a ready container by the use of probes. Coupled with a kubernetes strategy, we can also define and ensure our application survives broken container updates. Since the application I am supporting is already HTTP based, making an HTTP check to an endpoint that reports on connectivity to core services was the most trivial to implement. I created a script to verify connectivity to MariaDB, MongoDB, Memcached, Message Queue, and verified certain paths on the NFS share were present. All of these items are important to my application and most of them require certain configuration values in my containers to operate. Having kubernetes run this script every time there is a new pod verifies I will never an experience an outage due to a missing package again. As I mentioned before, I thought the readinessProbe</code> was a once-and-done, however, I found that after implementing it, my metrics indicated the script was running every 10 seconds per every replica... this quickly added up! After some chatting in the #kubernetes-users slack, I was able to get more understanding of the readinessProbe</code> and how it was designed to communicate with kubeproxy so that you could "shut off" a container by taking it out of rotation. This was not the behavior I wanted so it was suggested that I create a state file. This state file is created after the check and, if it is present, it skips all checks. Due to the ephemeral nature of container storage, it can be assumed this file will never exist on a pod where this check has not been performed. Adding a user to k8s RBAC 2017-12-01T15:00:36+00:00 In order to add a user to a kubernetes cluster, we will need several things: kubectl, CA.crt and CA.key (found in your head node's /etc/kubernetes/pki</code> folder), and openssl. First, create a private key for the new user. In this example, we will name the file employee.key: openssl genrsa -out employee.key 2048 </code></pre> Next, we will need to create a certificate sign request - employee.csr - using the private key we just created (employee.key in this example). Make sure to specify your username and group in the -subj</code> section (CN is for the username and O for the group). openssl req -new -key employee.key -out employee.csr -subj "/CN=username/O=developer" </code></pre> Generate the final certificate employee.crt by approving the certificate sign request, employee.csr, you made earlier. In this example, the certificate will be valid for 90 days. openssl x509 -req -in employee.csr -CA CA.crt -CAkey CA.key -CAcreateserial -out employee.crt -days 90 </code></pre> Give employee.crt, employee.key, and CA.crt to the new employee and have the employee follow the below steps. # Set up the cluster $ kubectl config set-cluster k8s.domain.tld --server https://api.k8s.domain.tld --certificate-authority /path/to/CA.crt --embed-certs=true # Set up the credentials (a.k.a login information) $ kubectl config set-credentials <name> --client-certificate=/path/to/cert.crt --client-key=/path/to/cert.key --embed-certs=true # bind login to server $ kubectl config set-context k8s.domain.tld --cluster= k8s.domain.tld --user=<name> # Optional: append `--namespace=<namespace>` to the command to set a default namespace. </code></pre> Note: You may move the certificates to a safe location since the commands included --embed-certs=true</code>. This saved the certs in base64 format in the kubernetes config. Sometimes I post to my blog so I remember how to do a particular thing. This is one of those times.