What are /r/kubernetes' favorite Products & Services?

From 3.5 billion Reddit comments

Can't believe no-one has mentioned prometheus? It is the standard when it comes to monitoring anything in Kubernetes.

This helm-chart installs everything that you need, including alerts and dashboards: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#kube-prometheus-stack . It can obviously also be extended.

Loki is great for logs; it doesn't do metrics. Although you can extract metrics from logs.

im willing to bet you have not set proper resource limits on your pods:

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/

and your applications are eating up all the available cpus.

Operators are meant to manage the lifecycle of a specific application (e.g drain a database, move replicas around, take backups, coordinate version upgrades etc). I think a lot of people are mixing up what operators are vs a custom controller.

This is not that use case. Just because you can do something doesn't make it a valid pattern.

https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

> The Operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.

Here are the slides:

https://www.slideshare.net/GregoryTaylor11/kubernetes-at-reddit-an-origin-story-kubecon-na-2018

They'll have recordings posted on the CNCF Youtube channel eventually.

You have to add a admission controller to get the same thing in vanilla k8s, https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector

Close enough to the openshift version that it doesn't feel too alien.

Hey! This is one of my specialties. We use an in cluster Prometheus to do service discovery in cluster based off a service account token that we give cluster read permission to.

Then, there is a central Prometheus server that uses the /federate endpoint of each cluster Prometheus to gather data centrally and add annotations like cluster name to each metric.

You can read more about federation in Prometheus here: https://prometheus.io/docs/prometheus/latest/federation/

No it's not. "A CPU" is one vCPU/core/hyperthread/whatever-your-infra-calls-it. How fast that is will vary based on your hardware; there is no such 1GHz approximation. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu

IMO, it would be better if there were such an approximation, because as it stands it's impossible to write good CPU limits/requests without knowing where it will be deployed.

There are some good tips in there but not all great for CKA. No one is going to remember a bajillion aliases and set up their bashrc for 20 minutes for a shortcut you use 3 times. I don't think you really need more than:

k=kubectl kn='kubectl config set-context --current --namespace' export dry='--dry-run=client -o yaml'

Along with the completion stuff and the complete -F __start_kubectl k

I also edit /etc/vim/vimrc & at bottom put: set ts=2 sts=2 sw=2 ai

Honestly the CKA is its own beast and shouldn't be conflated with useful stuff to actually admin a cluster.

Anyway IMO the actual cheat sheet you can access during the exam is more useful for CKA; https://kubernetes.io/docs/reference/kubectl/cheatsheet/

Getting the kubeadm CM from kube-system is a good tip, nice to get both pod and svc networks from one place quickly.

There is also some deprecated stuff in there.

k get cs Warning: v1 ComponentStatus is deprecated in v1.19+

I'm fond of Renovate, which can be used to update dependencies, configs, and the like. I think it would slot in near Helm and Argocd in your ecosystem there.

Basically, your build system would update something, and Renovate would go change the configs in Github, and then Argocd would reconcile the new configs with the cluster.

But Traefik supports standard ingress resources as well. They are simply just festure rich enough. The community is working on a standard for things such as IngressRoutes in the Gateway API. https://kubernetes.io/blog/2021/04/22/evolving-kubernetes-networking-with-the-gateway-api/

Prometheus and node-exporter work well for my environment. Grafana also has support for Prometheus data sources out of the box. There are plenty of other exporters but you can also instrument existing code fairly easily or write a custom sidecar.

1. Don't run containers root
2. Have good RBAC roles set up. Don't wildly grant people cluster-reader or cluster-admin because it's easy -- take the time to build up the policy that you actually need.
   
3. Have good pod security policies set up (https://kubernetes.io/docs/concepts/policy/pod-security-policy/). For some examples, take a look at the OpenShift default cluster policy, which is significantly more restrictive than the default Kubernetes policy.
4. Make use of quota to prevent accidental overconsumption of resources.
5. If you're fairly concerned, know where your images are coming from. There's a few different ways that you could restrict where images come from (blocking registries in /etc/containers/registries.conf in crio, validating registries in a custom admission plugin, etc).

The recommended way is to use Pod Topology Spread Constraints

Specifically using the key kubernetes.io/hostname

Is there a reason you dont use ConfigMaps?

https://kubernetes.io/docs/concepts/configuration/configmap/#configmaps-and-pods

Nice guide. I bookmarked it and going to send it some who are just starting with kubernetes.

​

I have one suggestion:

When talking about the K8s master components you listed etcd instead of controller-manager. In my opinion its far more important to know about controller-manager than etcd in that context.

https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager

It sounds sounds very much like the "debug container" feature, don't you think? 😉 https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container-example

Disclaimer: I took the CKA before this question/feature existed so I'm just guessing.

Define "bad".

The CPU architecture is slightly different, but compatible, so beside the performance difference, it's no biggie. Yet if pods run on the Pi3, they'll be slower than if they run on the Pi4. That's bad for CPU intensive programs.

It's a good educational exercise to work around this (mark the 3b as "low priority" or put special non-CPU-intensive workload on it), but you don't gain much here in practical terms. Some would thus see this educational value as "wasted time". In reality you try to keep all nodes identical. Simplifies management a lot.

There's more educational value if you add in an even older RPi. Or x86_64 nodes. Then the binaries are no longer compatible. See multi arch images. Again: you'd usually not do this in reality.

2 Pi4 and 1 Pi3b will work. So I guess that counts as "not bad". It's not good either (3 Pi4 would be absolutely ok), but for the sake of having 3 nodes to set up and test a HA setup, it'll do.

For a quick start, I am a fan of minikube. By default it will run in a single VM, but like a "real" kubernetes cluster you can join additional worker nodes to it.

https://kubernetes.io/docs/setup/minikube/

I don't think it's fair to say that Google isn't contributing. I've personally been involved since the beginning (https://www.docker.com/blog/community-collaboration-on-notary-v2/) and have spent a substantial amount of time giving feedback from the perspective of a registry and client maintainer. We're also working on some changes to the image-spec and distribution-spec that (if adopted) could enable the notary v2 prototypes to be portable across real registries.

I'm just not heavily involved in refining the requirements documents, because that's not really interesting to me as an engineer, and time is a limited resource :)

> Just wanted to learn some docker and kubernetes

Windows Docker Desktop has a built in Kubernetes setting that you can toggle to run K8:

https://www.docker.com/blog/docker-windows-desktop-now-kubernetes/

That is all fine, especially when you say our docs do not explain it well enough, we should fix that. Later if you have time please specify where you landed and which docs were to generic/complex or whatever your thoughts are.

​

The whole story of Kyma I tried to explain here as clear as possible https://kubernetes.io/blog/2019/05/23/kyma-extend-and-build-on-kubernetes-with-ease/ and also recorded a 7min video about it's key features.

​

In short:

In general, main thing about Kyma is that it is a way to go with monoliths that you want to turn into cloud native apps. It was donated to open source by SAP so a company that you might suspect knows somethign about monoliths.

Also, to speed up Kubernetes adoption for large teams that are used to working with monoliths, we designed Kyma to come with all batteries included. So you do not have to, on your own, decide on what tools to use, which service mesh, what monitoring, what logging, because we bring them all integrated. So Kyma is ready not only to write solutions on it but also to operate them easily.

Use Gitlab container registries. While you're there, use their free CI minutes, private repositories, Gitlab Pages integration, Kanban-style issue tracking boards, and all the other features.

Gitlab is dope.

[EDIT] - Oh I also forgot -- there is the whole AutoDeploy pipeline which is heavily aligned with Kubernetes to try out. Things like per-branch environments (which is generally only implemented at companies with good software engineering shops).

We run elastic clusters as a service out of Kubernetes with ECK. It's great. With the correct operator running a database on k8s makes a ton of sense.

https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html

I would encourage raising awareness of how to adjust the stabilization window on scale down or scale up to change the speed of scaling. The default window is 300 seconds, which can be a long time. Here's the link to that documentation.

apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: target-hpa spec: behavior: scaleDown: stabilizationWindowSeconds: 60 maxReplicas: 10 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: target-deployment metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 75

You can whitelist IP addresses by specifying .spec.loadBalancerSourceRanges in your service yaml.

See https://kubernetes.io/docs/concepts/services-networking/service for an example as well as several other options for configuring ELBs.

This is very specifically out of scope for Kubernetes, see "What is Kubernetes?".

Kubernetes makes it easy for users to implement a modular PaaS that suites their needs, or even FaaS. The main issue with PaaS is that it's hard to get it right for different use-cases, it's either too simple and opinionated or too complicated and has very opinionated defaults. It's much easier to think of Kubernetes as the Lego for building PaaS (or FaaS, as I said, and OpenFaaS is a good example of that).

If most of the projects are low traffic you might be faster by using some managed services instead of self-hosting. It comes at a cost but you don't waste time on infrastructure.

Managed services are becoming more expensive when you want to scale out which than might be a reason to move to self-hosting to save some real bucks.

Not sure what db type you are using, but since you mention that you are a frontend dev: check what https://firebase.google.com/ does offer for you.

Isn't the nginx ingress controller also able to route based on the hostname via name based routing? Guessing I'm missing something here! https://kubernetes.io/docs/concepts/services-networking/ingress/#name-based-virtual-hosting

I run a selfhosted NUC-based cluster, where all relevant nodes are schedulable. I use rook as storage.

Before I set proper limits on the OSD's, the memory would frequently run out on nodes, that would sometimes lead to the following fun sideeffects:

• All pods on that node killed
• Ceph cluster marking the OSD as out, and start to rebuild
  • this caused even more load on the remaining OSD's
  • which then trigger higher RAM usage on the remaining nodes, which caused a nice loop all over.

Now, this was a consequence of me not knowing what I was doing, using too small nodes, and copying in a fairly substansial amount of data.

When the node that had stuff killed was a master-node also, the cluster also started doing funny stuff, as one can assume it does when a 3-master-cluster suddenly only consists of two, or even one Master-node being ready.

All in all, for me this was a great learning experience.

It made me hugely impressed of the resilience of both rook, ceph and k8s in general - stuff might break, but the parts come together again fairly easy.

But to the topic;

Yes, you can use masters as worker-nodes.

That might also mean that your jobs will affect the cluster's stability in general, only you can say for certain if that cost is worth it.

I have also read that there are some security implications, but I'm not there yet - so I can't provide any information about that.

edit: Also, for reference:

kubectl taint nodes --all node-role.kubernetes.io/master-

This is what allows master-nodes to run schedules.

https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ This is the documentation behind the consept of tains.

Kubernetes by default sets ndots to five, so it doesn't assume it's an fqdn unless there are at least five periods in the host name. You can adjust this with a custom DNS config:

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config

Normally the resolution should fail fast enough even looking up all the possible search domains that the fqdn does get looked up, but you can manually lower ndots to help with this.

You may also be running into a conntrack race condition. This can be fixed by using the single request reopen option on libc based pods. I don't know if there's a fix for Alpine base pods yet. Here's an article that explains the problem in more detail:

https://medium.com/techmindtickle/intermittent-delays-in-kubernetes-e9de8239e2fa

The first thing I always do is run

Kubectl logs [pod name]

That command will display the application logs.

If the issue is that the pods are not spinning up you can also use

Kubectl describe pod [pod name]

This command gives a lot of information, the section that is helpful is the events section.

The link below is for the interacting with pods section of the kubernetes documents, and there is helpful info there. Kubernetes documents

I wrote a blog, for just this reason: https://coreos.com/blog/what-is-kubernetes.html

There's an attached whitepaper that goes into a bit more detail on the "Why" of containers and Kubernetes. But, be forewarned that it'll ask ya for an email address

Elasticsearch-Fluentd-Kibana (EFK), which is kinda agnostic (both for on-prem or cloud), Papertrail, Scalyr, and the cloud-provider specific ones. Pretty much depends on where you run Kubernetes …

You should define rds instance as Service of type: ExternalName and use it as regular service.

https://kubernetes.io/docs/concepts/services-networking/service/#externalname

If the file doesn't contain secret data (i.e. passwords, private certificates) then you should use a ConfigMap. If it does contain secret data then a secret is the right choice. See the documentation neilwatson posted for secrets, or the ConfigMap documentation

You'll need one of these for Kubernetes and one of these for OpenShift.

Kubernetes is a complex system to solve complex distributed computing problems, which makes it difficult to summarize in a 5 minute tutorial. The docs are your friend.

> Do you foresee kubeadm becoming the standard way to create / upgrade clusters (outside of hosted installations)?

Kubeadm is widely recognized as a standard way to deploy Kubernetes clusters with a wide variety of options. While the Kubernetes community supports multiple cluster deployment solutions simultaneously (mainly because there is no single best solution that will satisfy all the needs of everyone), Kubeadm (https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) - is the right solution that we may suggest for deploying the production-ready Kubernetes clusters.

But again, there are a lot of multiple solutions to deploy and manage Kubernetes clusters, all of them have their own pros&cons. Kubeadm is just one of them.

You may look into AppImage if you have strict dependencies on some specific version (like for GTK3): https://appimage.org/.

This allows to have a binary that works universally on every Linux distributions. And it's way easier to integrate into a Linux distribution than a .deb.

you can use ephemeral containers (which moves into beta in 1.23)

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

come on - this is barely "advanced patterns". You don't even go into things like aggregated roles.

I wouldn't bother with this article - just go to the official docs if you want actual "advanced" patterns: https://kubernetes.io/docs/reference/access-authn-authz/rbac/

The biggest things which helped me speed up were:

• As I went through the trial tests I bookmarked the page which gave me information if I needed it (such as how to backup etcd). This way I had helpful bookmarks instead of just links to subjects.
• Name the bookmarks something useful, assuming you would be looking quickly for an answer.
• Setting up the alias k=kubectl and autocomplete was a real timesaver. (https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#enable-shell-autocompletion)

It sounds like you're on the right track here. Integrate git into your CD tool of choice, managed with a particular SA, monitor any create events against the k8's API that aren't executed from that SA, and I feel like you're 90% of the way there.

Edit: here's the documentation on audit logging. https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

You need to set up ingress with host based routing inside your AKS cluster (https://kubernetes.io/docs/concepts/services-networking/ingress/). You would set up a service for your pods, and ingress should be configured so that each host (the subdomain) routes traffic to the appropriate pod. Ideally you would want to deploy your pod as a deployment. Your DNS records would point to your AKS cluster. I think that should be able to do it.

Essentially, DNS record -> AKS IP -> ingress looks at the host and routes the traffic -> service -> pod

This. You can communicate with services in other namespaces, on the same cluster, using the full service discovery DNS name, unless it has been restricted with a network policy. We use these for intra-cluster rest API calls, and they are of the form:

<service_name>.<service_namespace>.svc.cluster.local:<service_port>

It'll run 4 processes. CPU limits in K8s are not CPU cores, either.

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu

I don't think the CPU limits work the way you're thinking they do. It's more of a weighting thing, rather than a limit.

"A Container with spec.containers.resources.requests.cpu of 0.5 is guaranteed half as much CPU as one that asks for 1 CPU. "

"A Container might or might not be allowed to exceed its CPU limit for extended periods of time. However, it will not be killed for excessive CPU usage."

"The spec.containers[].resources.limits.cpu is converted to its millicore value and multiplied by 100. The resulting value is the total amount of CPU time that a container can use every 100ms. A container cannot use more than its share of CPU time during this interval."

This last part is important. If you spec a CPU limit and it gets limited, it's a "stop the world" limiting. The process is just frozen until the next 100ms interval. This can lead to situations where you hit the limit in, say, 5ms, then for the next 95ms the process is just frozen and unresponsive to anything.

Awesome, love the diagrams.

Any way to note that services and endpoints can refer to other entities, not just pods? Eg I've used that to refer to an elastic cluster running outside kubernetes.

Look at this: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically

You need to mount the config map as a volume. Environment variables from a config map don't get updates.

Also you should probably use a StatefulSet instead of Deployment. Just look it up.

And instead of scaling down and then up, you can run:

kubectl rollout restart statefulset mysql

k8s (and CoreDNS) inherits DNS configs from your nodes. Make sure your /etc/resolv.conf is set up right on your rPis that make up your cluster, specifically the ndots option, nameservers, and search list.

Edit: this might also be helpful

Put a taint on the nodes with effect NoSchedule, then use nodeSelector on your pod template to favor those nodes by label and include a toleration to overrule the taint for workloads that are specially permitted to run there. This is a textbook example of the purpose for those features.

https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

Unless there is some specific need for Ubuntu, you should move to a container-specific distro like CoreOS's Container Linux, which is self-updating using A/B partitions for fall-back. Container-specific distros are much lighter weight, and don't have package managers meaning you never have to worry about keeping any packages up to date on the nodes.

Also, as the updates happen automatically, there is no delay between a release being published and the upgrades being deployed to your fleet. There is no manual intervention or operator overhead required, often allowing CVE's to be mitigated before your operations team is even aware of them.

You can use the Container Linux Update Operator to control the rollout of updates, ensuring minimal impact on your cluster workloads while updates are being deployed.

You are taking the only instance offline to "force" an upgrade instead of performing a "rolling upgrade". Use a Deployment instead. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

You need to specify the name of the service account like this: system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT) as described here. The full command then becomes this: $ kubectl auth can-i get pods --namespace rbac-test --as system:serviceaccount:rbac-test:rbac-test-sa yes

Sorry to hear this; out intention with AGPL is that it’s totally fine to use the projects, even expose them on the internet - as long as you release any changes you make. If you don’t make any changes, you have nothing to worry about.

We published a FAQ on the blog which goes into more detail https://grafana.com/blog/2021/04/20/qa-with-our-ceo-on-relicensing/

I actually really like that Kubernetes is on GitHub. I'm already there, it fits nicely into my daily workflow, and I don't have to wrestle with whatever horrible workflow and review tooling that might be used. In the case of drupal.org, I really had to dig to find the git repo.

https://kubernetes.io/docs/concepts/storage/storage-classes/

You can create your own provisioner, then a storageClassName that uses the provisioner

As far as persistence, that is controlled by the policy you set on the reclaimPolicy

So its basically an order of:

• provisioner (netapp, glusterfs, whatever)
• storageClass (how you references that provisioner & other settings you need to make it work the way you want)
• persistent volume (creating the actual volume, which the provisioner takes care of under the hood, also where you define sizes & usage, readwrite, reclaimPolicy)
• persistent volume claim (where you actually use the PV in your pods)

It won't be scheduled to a node without the spare ram if you have appropriate memory requests set on the pod: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

A limit of 3 per node isn't dynamic, what if one node has twice the ram?

You can specify a limit to the resources that a container may access.

You might enjoy this part of the docs. There are examples...

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Hey yes I would take a look at the official docs https://kubernetes.io/docs/reference/ports-and-protocols/ It can sometimes be hard to find things in the docs but I would recommend getting familiar with them if you are intending to use kube especially in places where you are worried about specifics like ports in this case. Good luck!

If creating a service imperitavely, I would go with `kubectl expose`: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#expose

I guess in your case you are just creating the service but not the endpoints for it.

Just wanted to add here that your namespace may have some LimitRange[1] assigned to it, which means that if you don't set resources in your pod you may get some defaults applied to it anyway. This is good practice specially in multi-tenant clusters.

I tend to set strict LimitRange to my clusters so it "encourage" developers to specify theirs apps resources properly or get throttled/OOMKilled often. :)

[1] https://kubernetes.io/docs/concepts/policy/limit-range/

Becareful with >10 deprecated apis removed https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/#removal-of-several-deprecated-beta-apis (>10 seems a bit high to still use the word several....)

What I usually do is label namespaces with name: <namespace-name> and then I can use the following construct everywhere that needs a <code>LabelSelector</code> (e.g. NetworkPolicy):

namespaceSelector: matchLabels: name: <source-ns>

When I took the exam, I just went ahead and labelled the source namespace with kubectl label ns <source-ns> name=<source-ns> :D

OP ignore all the others that say you’re being stupid. Follow this:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Kubeadm is the official installer. Once you remove the noschedule taint you can deploy work to it. The biggest things to know here:

1) Make sure to take backups and ship them somewhere if the etcd cluster created by Kubeadm, it holds all your state.

2) Since your work is running on the same as the master, your performance under heavy load will suffer. Just be aware.

3) Since your work is running on the same as master, your security surface area is much larger and well.

4) When you do scale out, skip to 3 nodes. Etcd needs it for quorum and you’ll have a bad time with just 2. That is assuming you scale out your masters as well.

I’ve been deploying k8s to bare metal for a few years, happy to answer any questions.

Persistent volumes on AWS have node affinities set that will ensure the pod is scheduled in the correct zone, assuming everything is configured correctly.

Oh fu.. I mixed it up. This is on their Docs

&#x200B;

"Minor releases occur approximately every 3 months, so each minor release branch is maintained for approximately 9 months"

&#x200B;

Check this : https://kubernetes.io/docs/setup/version-skew-policy/

&#x200B;

You will only see most of the components (Kubectl and kubeapiserver) supports only upto 1.12

&#x200B;

I guess your on the edge of the cliff mate. They only support three minor version from the current version. 1.14, 1.13 and 1.12,

&#x200B;

Better to upgrade right now.

-- Going through Documentation and practicing all the tasks twice

-- Kubernetes in Action by Marko Luksa

-- kubectl --dry-run --save-config is your friend

-- Go thru kubectl cheat sheet and practice creating all k8s objects (Deployments,services,bare pods,etc )on the commandline using kubectl than prepping up yaml files from scratch.

&#x200B;

During exam ,

-- Time management is important . You dont know a question , skip it and come back at last.

-- You can use documentation so use search functionality in kubernetes.io wisely and widely.

All the best for the exam.

So, there's tunable behavior as to how CronJobs actually deal with delay and failure, and the defaults aren't always the best option, depending on your workload. There are a couple knobs that you can tune, IIRC, but the first one to look at would probably be <code>concurrencyPolicy</code>, which controls what happens if a CronJob would have two running at the same time.

The default solution, developed by Kubernetes project itself is Kubernetes Dashboard (https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/).

There are also a few third-party UI solutions, where my personal preference is the Scope by Weave (https://github.com/weaveworks/scope).

For stateful applications like databases, you'd do a rolling update like described here: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#rolling-update

For a stateless service, you'd have two different deployments e.g. called blue and green, each with their own special label that lets you know which pods are running from which deployment. For example blue could have a label release=blue and green have a label release=green. In front of these deployments there would be a single service for your application that would use a LabelSelector release=blue. When your new deployment is ready to go, updating the service to use the selector release=green will switch all incoming traffic to the green deployment.

For something like a Canary deployment, you could follow my quick tutorial here. There might be small downtimes in the tutorial because I'm not properly catching the SIGTERM being sent to the container and closing gracefully, but it can be done with zero downtime.

FYI, that "DefaultDeny" Namespace annotation was removed in Kubernetes 1.7 and you should instead create a NetworkPolicy to get that behavior.

If you like u/mogthesprog's idea of sticking it in git but you're going down the immutable infrastructure road and don't need to update the json often you could try out mounting the git repo directly:

https://kubernetes.io/docs/user-guide/volumes/#gitrepo

Never tried this myself. I'd be interested to know if you change the revision reference would it update the mount. Also if you point it at a branch instead of a revision, world it pick up changes next time you update the deployment. Sorry if I've provided more questions than answers 😛

If you try it please let me know how you get on!

In general storage is hard. Kubernetes is no exception. If you want fault tolerant storage block storage will probably be an easier option than Gluster or Ceph. In our cluster we use Amazon's EFS, but that can get a bit pricey.

No, you don't need an externally exposed endpoint.

You can get a LetsEncrypt certificate by solving a DNS challenge for a domain where you can change the DNS records. You can even get certs for wildcard domains this way. https://letsencrypt.org/docs/challenge-types/#dns-01-challenge

Then you can put the certificate and key wherever you want. Eg, on your local Kubernetes cluster with a 172.16.0.0/12 IP. And change your /etc/hosts file to let the domain you got the certificate for point to the Kubernetes IP.

To get a better understanding of K8s and its concepts I would read through the concepts docs.
If you want do get a little bit hands on take a look at the tutorials section.

It would also be helpful if you could elaborate on "I currently do" and give a brief summary of what you still want to/should know. Knowing what tasks you might work on could also help to point you to the right resources.

I've yet to do this but the docs seem pretty straight forward.

You shouldn't be adverse to creating new clusters. That's what IaC is all about. I've destroyed and re-built my clusters a number of times.

Edit: Oh I just noticed you're running on Bare metal. I'd suggest using VMs to easier destroy and rebuild clusters. But if I was forced to run kubernetes on bare metal I'd probably ensure I had a way to re-provision the nodes with a PXE server and kickstart.

Excellent, in that case I think you can use this paradigm below. I apologize because I work way more in the OpenShift kubernetes world with the majority of my kubeadm experience coming from when I got the CKA.

https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/

Great peace of documentation! I would mention that PSPs are leaving us soon and point people to something like OPA Gatekeeper until Kubernetes has a usable PSPRP version!

https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/#what-s-next

P.S. If you need contributors, lmk!

I think it might have been the new Gateway API: https://kubernetes.io/blog/2021/04/22/evolving-kubernetes-networking-with-the-gateway-api/

I'm sure I feel like I read about Ingress being canned though... can't shake the feeling.

This page should be your bible: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ for actual doing the upgrade of Kubernetes. When you go for CKA (Always do when and where possible) you will probably do this on the exam as well.

There's a few problems with recommending Pi's here.

Price: It's way cheaper to just spin up and kill a k8s cluster in the cloud for training when you need it, than it is to buy Pi's. The repetition is valuable in training, because people learn best by repetitious training. That $45 price tag can easily be less than $10 as long as the person only has a running cluster while using it actively.

There are pitfalls with arm based computing:

You can't really do things like run longhorn.io on a Pi cluster. The IO is not viable.

There are still many popular container images that don't have arm versions available, such as the ones provided in the bitnami helm repo.

Then there's the consideration of if you /can/ use kubeadm or if you are going to learn on k3s to squeeze out a little more of that valuable compute power you purchased on a budget.

I think k3s is really cool, but I found more value in understanding kubeadm and revisiting k3s much later. It doesn't expose mirror containers, which makes Kubernetes look a bit magical without knowing the parts of the all-in-one binary.

Also, wanted to clarify that I'm not hating on ARM. I love flirting with the idea of ARM based computing, but it's important to know that it has shortcomings.

I would probably point OP this way:

What I have found a lot of success with for starting off with training is like so.

1. Learning to use Kubernetes with Docker for Desktop or minikube is very easy. I think most people could benefit by starting with the Workloads section inside of either of these options before crawling out to administration.

2. Learning to deploy Kubernetes with kubeadm in either cloud servers or VMs. Deploy a single node. Then learn to do multi-master clusters. Then learn how to add/remove workers. How to do upgrades. How to backup the cluster state.

Adding my 2 cents here. The service mesh and ingress components of Istio are completely decoupled, so they aren't really tied together. They are just available as part of the same suite of tools. Since both components do similar things (proxy traffic with Envoy and deal with TLS certs), I assume the creators figured that they would be complimentary features. For example, you can apply envoyfilters to both sidecars and ingress gateway using the same syntax/config.

In addition, the vanilla kubernetes ingress resources (implemented by Ambassador, Contror, and Nginx ingress controllers) just didn't support some advanced features of Istio ingress, which was the inspiration for Istio to create its own approach. Now, however, the Kubernetes Gateway API will hopefully solve those limitations with the Kubernetes ingress resources.

Although I am not so familiar with cdk8s, the things you mentioned are native k8s stuff that you need to understand before you would be able to apply them in IaC of any kind.

For volumes - look up persistent volumes in the official k8s docs. For more help you would need to share how you run k8s (managed or Bare metal? Cloud provider?)

For example on azure you would use storage account/disk, on aws EFS or EBS, etc.

For private registries, you can use image pull secrets. Again, the official docs are great. But again with more context on how you run k8s there might be a better solution, like in azure you can link ACR with AKS and in AWS you can use an IAM role, both options are superior compared to image pull secrets.

After understanding how to do it without IaC, making it declarative on any IaC will probably be much much easier.

I believe this is the default scoring weight applied by the scheduler: https://github.com/kubernetes/kubernetes/blob/851b7276a5deef9b5ee78bed59e5226d591efaf9/pkg/scheduler/apis/config/v1beta2/default_plugins.go#L82 You can however customise these weights, either for the default scheduler or for a specific scheduler instance - https://kubernetes.io/docs/reference/scheduling/config/#extension-points

Deploy a service for your database pod: https://kubernetes.io/docs/concepts/services-networking/service/

Then you only have to configure the name in your application.

https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/

Browse up a few dirs and you'll have the base spot to find the API parts you're asking for

I think this is what you’re looking for:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

Make sure that your Kubernetes API endpoint is accessible from your Jenkins worker, ie you can ping the IP from the machine, otherwise this is a network config issue.

You can go through setting up the service account, and a context for it in a namespace, from your local machine or wherever you have kubectl working. Then the same ~/.kube/config section for that context should be able to copy straight to a jenkins worker. Assuming the correct kubectl is installed and on the PATH, it should be able to run those commands

You would include the configmap as a volume in your pod. Then, you can mount configmap in your container. See example and details in the docs: https://kubernetes.io/docs/concepts/configuration/configmap/

Kubeinvaders looks like a nice one. and who doesn't like to game?
https://kubernetes.io/blog/2020/01/22/kubeinvaders-gamified-chaos-engineering-tool-for-kubernetes/

Indeed. OP can also use rclone (docker hub image) to sync data to and from various cloud storages (amazon S3, google cloud storage, google drive, google photos, backblaze, etc), selfhosteds nextcloud, minio, webdav, SFTP, HTTP, Ceph... as an initContainer as you mentionned to populate the PV, and as a cronJob to regularly synchronize data back to storage from PV.

We use it at work in a cronJob to sync data pushed by a contractor on a S3 storage to our "data to process" PV, and it's been reliable since.

I am not familiar with Hetzner's specifics but we do a lot of work to make the bare metal install on CoreOS Tectonic as simple as possible. Here is an overview video and the install instructions.

I've also been using the Prometheus operator in my local dev kubernetes cluster recently: https://github.com/coreos/prometheus-operator

General info on kubernetes operators: https://coreos.com/blog/introducing-operators.html

I love how operators capture and automate admin tasks in software and make deploying/managing complicated applications a lot easier. Are there other interesting kubernetes operators out there you all have been using?

Yes, add prometheus to your application, use histogram metrics (https://prometheus.io/docs/concepts/metric_types/#histogram)

Alternatively you can deploy a service mesh and get out the info from there, but personally I would rather instrument the application directly. (https://istio.io/latest/docs/tasks/observability/metrics/)

The reason as to why it takes so long to register nodes as down is due to node heartbeats. By default the cluster waits for 5 minutes before removing them.

https://kubernetes.io/docs/concepts/architecture/nodes/#heartbeats

You do not need to run kubelet, it will be configured by kubeadm completely.

Official guide for your notes. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Terraform

• sounds a good idea for this. What you need to consider is that where and how to store the state file.

Vault

• for secret management, hence Kubernetes Secrets are encoded and **not** encrypted.
• integrates well into Kubernetes (ServiceAccount authentication)

Consul

• consul-template/envconsul for consume the secrets if the apps not capable for this
• I'd use Kubernetes native Services, but things can grow quickly to complex level when service mesh becomes a must have

DNS

• Ingress (Traefik is the new buzz but there are many. Check this out)

Monitoring

• Prometheus all the way
• Grafana for visualize metrics

&#x200B;

Hope it helps

&#x200B;

*edit* grammar and formatting

Privileged mode is intended for this usecase. Keep running the script inside a container, but give that container sufficient rights to perform the work it needs to do. https://kubernetes.io/docs/concepts/workloads/pods/pod/#privileged-mode-for-pod-containers

This is the same concept CNIs in Daemonsets use. For example, read https://docs.projectcalico.org/v3.8/manifests/calico.yaml

There is an except option in the networkpolicy spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#ipblock-v1-networking-k8s-io

This means you can do:

egress: - to: - ipBlock: cidr: 0.0.0.0/0 except: - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16

which allows traffic to the world, except for any private IP addresses.

Using kubeadm to create new masters (HA clusters is the terminology that kubeadm uses) is still fairly experimental, but we've been able to write automation around it to bootstrap multi-master control planes. Most of it revolves around the --experimental-control-plane flag of kubeadm join.

The main things you need to do is copy over most of the certificates, keys, and configs from /etc/kubernetes on your first master since it needs to share the same PKI. Then, you just need to have the cluster endpoint and a valid kubeadm token and CA cert hash (similar to how you join worker nodes).

This has been really successful for us for adding new masters, though there's still a lot of work in the replacing/removing/upgrading masters department.

Here's the official docs for it.

Hope this helps!

Just adding an extra 2c to what most comments already described...

Aside from being a complete layer 4/7 proxy with advanced features idealized for a cloud environment running a complex mesh of microservices, it's important to realize that migrating an existing architecture to Istio means practically leaving your entire legacy of frontends and reverse-proxies behind. If you are using Nginx or Apache, those become obsolete in the face of Istio. And you need to evaluate whether the move makes sense for your team and infrastructure.

Istio was designed to be independent of Kubernetes. But even inside k8s, if you are currently using solutions like ingress-nginx, migrating to Istio means you are no longer depending on Kubernetes native objects, like Ingress, to expose services. Istio completely abandons some native k8s objects in favor of its own CRDs. You will end up with a much more flexible solution for a new platform of services, but you will nonetheless need to rewrite your entire frontend layer... not to mention the cost of running a full regression test to make sure all services are still working as expected from the user's perspective.

My team is still digesting the idea of Istio. Quite frankly, we haven't felt the need to rush. Our infrastructure serves close to 100 monolithic/legacy applications and Nginx, while old and not as flexible, compensates entirely due its outstanding maturity and stability.

Disclaimer: I'm not familiar with Traefik, I'm familiar with nginx-ingress.

Going from ports to path, instead of calling your application at www.example.com:4444, you'll want to call your application at www.example.com/4444/ or www.example.com/applicationa/ . nginx-ingress will send data to it automatically by looking at the kubernetes ingresses, as long as you put the host as www.example.com and the path as /applicationa/ . You can have both /4444/ and /applicationa/ going to the same application. I assume Traefik is similar in using the ingresses.

There are options to drop the path if you don't want your application to handle them. So traffic to www.example.com/applicationa/stuff/morestuff/ will hit your application as www.example.com/stuff/morestuff/

Hope this helps.