Traditional virtualization requires both a host operating system (OS) and a guest OS inside each virtual machine. The upside is that you can run multiple OSes (or versions of them) on a single base host. The downside is that they are big, slow, and not very portable.
Linux containers (or containers for short) have simplified things by allowing you to deploy multiple applications in separate buckets (or containers) that all share the same base OS. Containers generally speaking are smaller, boot up faster, and are generally more portable across multiple environments when compared to traditional VMs.
Kubernetes, which is a technology created at Google, is one of the most popular container orchestration tools. It enables teams to operationalize containers at scale. It provides multi-tenancy functionality – helping users to create and manage clusters, and then break those clusters up into virtual clusters to support multiple teams, projects, or departments. Each of those users and teams can have distinct security permissions.
Another solution Kubernetes offers is auto-scaling, of which there are two types. Workload (or pod) autoscaling is the ability to scale up the number of pods running a given workload as needed. This works with multi-tenancy to take full advantage of the resources available in a given Kubernetes cluster. Cluster autoscaling is the ability to scale up or down the number of nodes in the entire cluster without disrupting the cluster or any of its applications or users. This is helpful, but of limited utility, when you’re running the cluster on physical nodes in your own datacenter. When you’re running Kubernetes on virtual servers in the cloud, though, this allows you to continually right-size your cluster so you’re only paying for the resources you need at any given time.
Because Kubernetes-enabled workloads can be run on any Kubernetes cluster in any location without any changes to the application’s code, users can move applications between deployment environments with relative ease. Kubernetes has effectively become the default abstraction layer across the data center and the public clouds – essentially enabling hybrid cloud infrastructure.
Regardless of the specific implementation, many End-to-End ML platforms now have extensive built-in support for Kubernetes as a part of their ML Infrastructure Orchestration capabilities.