{"id":544,"date":"2017-12-08T07:00:22","date_gmt":"2017-12-08T15:00:22","guid":{"rendered":"http:\/\/andrewwippler.com\/?p=544"},"modified":"2017-11-30T14:36:15","modified_gmt":"2017-11-30T22:36:15","slug":"kubernetes-health-check","status":"publish","type":"post","link":"https:\/\/andrewwippler.com\/2017\/12\/08\/kubernetes-health-check\/","title":{"rendered":"kubernetes health check"},"content":{"rendered":"<p>The day before thanksgiving, I was pondering an issue I was having. I was pinning a package to a specific version in my Docker container and the repository I grabbed it from stopped offering this specific version. This resulted in a container that Jenkins responded as being built correctly, but missing an integral package that allowed my application to function properly. This led me to believe I had to implement <a href=\"https:\/\/puppet.com\/blog\/using-lumogon-inspect-your-containers\">Puppet&#8217;s Lumogon<\/a> in my Jenkins build process. Curious if anyone had something like this already developed, I headed over to github.com which eventually led me to compose this tweet:<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">Step 1: Have k8s technical problem<br \/>\nStep 2: Review <a href=\"https:\/\/twitter.com\/kelseyhightower?ref_src=twsrc%5Etfw\">@kelseyhightower<\/a>&#8216;s recent git pushes<br \/>\nStep 3: Find: <a href=\"https:\/\/github.com\/kelseyhightower\/ato\/blob\/master\/deployments\/helloworld.yaml#L26-L29\">https:\/\/github.com\/kelseyhightower\/ato\/blob\/master\/deployments\/helloworld.yaml#L26-L29<\/a><br \/>\nStep 4: Realize this is exactly something I needed to implement<br \/>\nStep 5: Implement feature<br \/>\nStep 6: Enjoy thanksgiving without worry<br \/>\nStep 7: Blog post<\/p>\n<p>\u2014 Andrew J. Wippler (@AndrewWippler) <a href=\"https:\/\/twitter.com\/AndrewWippler\/status\/933413365837418498?ref_src=twsrc%5Etfw\">November 22, 2017<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>This <code>readinessProbe<\/code> communicates with <code>kubeproxy<\/code> to either allow a pod to be in service or out of service. At first I thought the <code>readinessProbe<\/code> was a once-and-done check, but I found out later this is not the case. When a pod gets launched, kubernetes waits until the container is in the ready state. We can define what consists of a ready container by the use of probes. Coupled with a kubernetes strategy, we can also define and ensure our application survives broken container updates.<\/p>\n<p>Since the application I am supporting is already HTTP based, making an HTTP check to an endpoint that reports on connectivity to core services was the most trivial to implement. I created a script to verify connectivity to MariaDB, MongoDB, Memcached, Message Queue, and verified certain paths on the NFS share were present. All of these items are important to my application and most of them require certain configuration values in my containers to operate. Having kubernetes run this script every time there is a new pod verifies I will never an experience an outage due to a missing package again. As I mentioned before, I thought the <code>readinessProbe<\/code> was a once-and-done, however, I found that after implementing it, my metrics indicated the script was running every 10 seconds per every replica&#8230; this quickly added up!<\/p>\n<p>After some chatting in the #kubernetes-users slack, I was able to get more understanding of the <code>readinessProbe<\/code> and how it was designed to communicate with kubeproxy so that you could &#8220;shut off&#8221; a container by taking it out of rotation. This was not the behavior I wanted so it was suggested that I create a state file. This state file is created after the check and, if it is present, it skips all checks. Due to the ephemeral nature of container storage, it can be assumed this file will never exist on a pod where this check has not been performed.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The day before thanksgiving, I was pondering an issue I was having. I was pinning a package to a specific version in my Docker container and the repository I grabbed it from stopped offering this specific version. This resulted in a container that Jenkins responded as being built correctly, but missing an integral package that [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[11],"tags":[75,51,83],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/posts\/544"}],"collection":[{"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/comments?post=544"}],"version-history":[{"count":1,"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/posts\/544\/revisions"}],"predecessor-version":[{"id":545,"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/posts\/544\/revisions\/545"}],"wp:attachment":[{"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/media?parent=544"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/categories?post=544"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/andrewwippler.com\/wp-json\/wp\/v2\/tags?post=544"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}