This was the most obvious one. Our infrastructure today has far less compute, memory

8 min readNov 19, 2020

ou can make an attempt to push towards GitOps if you will. If you can’t do that, reducing manual steps to a bare minimum is a great start. We use a combination of eksctl, terraform and our cluster configuration manifests (including manifests for platform services) to set up what we call the “Grofers Kubernetes Platform”. To make the setup and deployment process simpler and repeatable, we have built an automated pipeline to set up new clusters and deploy changes to existing ones.

Setting up a basic cluster is perhaps not as difficult. We were able to get up our first cluster running within a week. Most issues happen when you start deploying your workloads. From tuning cluster autoscaler to provisioning resources at the right time to configuring the network correctly for the right performance, you have to do research and configure it all yourself. Defaults don’t work most of the time (or at least they didn’t work for us back then) for production.
However, Naval also quotes others during the interview. A lot. “Specialization is for insects,” he shares less than two minutes in — a line from a friend. He also talks about Bruce Lee, Elon Musk, Madonna, Paul Simon, U2, the ancient Romans, and beginner’s mind, a zen concept, before we even hit ten minutes.
Even when using a managed Kubernetes service, invest early in infrastructure-as-code setup to make disaster recovery and upgrade process relatively less painful in the future and be able to recover fast in face of disasters.
In non-production environments, we safely over commit resources as much as possible by keeping resource requests extremely low and limits extremely high. The limiting factor in this case is memory i.e. no matter how low the memory request is and how high the memory limit is, pod eviction is a function of sum of memory utilized by all containers scheduled on a node.
We mostly operate out of the Singapore region on AWS. At the time we started our journey with Kubernetes, EKS was not available as a service in the Singapore region. So we had to set up our own Kubernetes cluster on EC2 using kops.
Our learning is that operating Kubernetes is complex. There are a lot of moving parts. And learning how to operate Kubernetes is most likely not core to your business. Offload as much as possible to cloud service providers (EKS, GKE, AKS). There is no value in doing this yourself.
An important aspect of Kubernetes is to think about how developers are going to interact with the cluster and deploy their workloads. We wanted to keep things simple and easy to scale. We are converging towards Kustomize, Skaffold along with a few home-grown CRDs as the way for developers to deploy and manage applications. Having said that, any team is free to use whatever tools they would like to use to interact with the cluster as long as they are open-source and built on open standards.
Deploying Open Policy Agent to build the right controls helped automate the entire change management process and build the right safety nets for our developers. With Open Policy Agent, we can restrict scenarios like one just mentioned before — it is possible to restrict service objects from getting created unless the right annotation is present so that developers don’t accidentally create public ELBs.
Naval doesn’t have a quote collection he looks at every morning. He doesn’t even take notes when he reads: “I think taking notes is the same as taking photos when you’re on a trip. All it’s doing is taking you out of the moment,” he says in another interview.
Naval’s quotes flow into the conversation as an organic extension of his thoughts. They just happen to be what he remembers in the moment. That’s why they feel flawless — and he sounds smart as a result.
Many of us want to be like Naval. We’d love to shoot quotes left and right, dropping our own wisdom and that of others as casually as others drop $2 on a coffee. The problem is we often try too hard, and it makes us look sleazy rather than intelligent.
Sometimes this independence could pose severe risks. For example, using the LoadBalancer type service in EKS provisions a public-network facing ELB by default. Adding a certain annotation would ensure that an internal ELB is provisioned.We made some of these mistakes early on.
Kubernetes is meant to unlock the cloud platform for developers, make them more independent and push the DevOps culture. Opening up the platform to developers, reducing intervention by cloud engineering teams (or sysadmins) and making development teams independent should be one of the important goals.
Our learning was to keep resource requests high enough but not too high so that during low traffic hours we are wasting resources and keep resource limits relatively close to resource requests to allow for some breathing room for spiky traffic without pod evictions due to memory pressure on nodes. How close must the limits be to requests depends on your traffic patterns.
After we started migrating, we observed a lot of performance and functional issues in our cluster due to incorrect configuration. One of the effects of that was adding a lot of buffers in resource requests and limits to eliminate resource constraints as a possibility for performance degradation.This does not apply in case of non-production environments (such as development, staging and CI). These environments don’t get any spike in traffic. Theoretically you can run infinite containers if you set CPU requests to zero and set a high enough CPU limit for your containers. If your containers start utilizing a lot of CPU, they will get throttled. You can do the same with memory requests and limits as well. However, the behaviour of reaching memory limits is different than that of CPU. If you utilize more than the set memory limit, your containers get OOM killed and they restart. If your memory limit is abnormally high (let’s say higher than the node’s capacity), you can keep using memory but eventually the scheduler will start evicting pods when the node runs out of available memory.One of the first observations was pod evictions due to memory constraints on nodes. The reason for this was disproportionately high resource limits as compared to their resource requests. With surge in traffic, increase in memory consumption could lead to memory saturation on nodes, further leading to pod eviction.

This was the most obvious one. Our infrastructure today has far less compute, memory

Written by helal patha

No responses yet