Kubernetes is an open-source system for automated deployment, scaling, and management of containerized applications. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation.
By the publically available information such as this CNCF survey, Amazon AWS seems to be running more Kubernetes clusters nowadays than any other cloud provider in the world and this percent keeps increasing.
Recently, Amazon presented it’s new Amazon EKS service, which became generally available few days ago. Today we finished reviewing and testing it so we are glad to provide some more information and recommendations. So, to begin with,
What is Amazon EKS?
Amazon EKS is Amazon’s ‘Kubernetes as a service’. It simplifies the process of spinning up and maintaining highly-available Kubernetes clusters in AWS cloud.
Typical EKS cluster consists of 2 main components, each is running on it’s own VPC — Control plane and Worker Nodes
- Control plane consists of 3 Kubernetes Master nodes running in 3 different AZs for high availability. Incoming traffic to Kubernetes API comes through the AWS network load balancer. Control Plane runs on the Amazon-controlled VPC that cannot be managed by the customer and is fully managed by Amazon
- Worker Nodes run on usual Amazon EC2 instances at the customer-controlled VPC. Any AWS instance type can be used as a worker node. Worker nodes can be accessed via SSH or provisioned with any existing automation.
EKS is flexible in terms of layout. You can deploy one EKS (read ‘Kubernetes’) cluster per environment or per system or per application. Also a single Amazon EKS cluster an be used to run multiple applications by taking advantage of Kubernetes namespaces and IAM security policies.
Below are the official pictures from AWS that better describe how does the typical EKS cluster look like in AWS and how does it work:
EKS usage algorithm. Source: Amazon AWS
EKS architecture schemas. Source: Amazon AWS
What is good about Amazon EKS?
Reduced maintenance overhead
All Kubernetes cluster spinup and management operations are well automated by AWS which allows AWS customers that uses EKS so focus more on their business and applications instead of spending time of engineers on Kubernetes clusters provisioning and maintenance.
Integration with AWS services
EKS does not force customers to migrate all workloads to AWS. With EKS one can still spread the workload over different clouds. Migrating from self-hosted Kubernetes installations to Amazon EKS typically does not require any significant changes to the existing system. In the same time, EKS integrates well with many common Amazon services, namely:
- VPC and availability zones. EKS supports multi-AZ deployment of worker nodes for high availability
- EKS load balances traffic for services using AWS Network Load Balancers, AWS Application Load Balancers and Elastic Load Balancers
- Route53. EKS supports assigning Route 53 DNS records to any service running in the cluster
- Autoscaling: worker nodes are deployed into the autoscaling group. I.e., cluster can automatically scale up and down by the load or by any CloudWatch metric
- AWS CloudWatch — all worker nodes are monitored with CloudWatch, like any other EC2 instances
- AWS IAM. Via open-source Heptio authenticator you can use IAM roles and policies to authenticate users to EKS and configure the fine-grained access to any Kubernetes namespaces created in it.
- AWS EBS. EBS volumes are used for Kubernetes persistent storage
- AWS databases. RDS, DynamoDB, Elasticache, Redshift, and all other AWS databases are available from EKS which for many cases allow to keep in EKS only stateless components while running databases separately on AWS which provides the more flexible failover paths for EKS-deployed components and in the same time reduces the maintenance overhead.
- Elastic Network Interfaces (ENI) can be used for EKS Kubernetes cluster networking via the CNI plugin
- AWS CloudTrail records any actions that are done with your EKS clusters via AWS Console, SDK or rest API
As AWS EKS fully runs on AWS VPC and EC2, it inherits the security measures from these services. For example, you can use your own VPC security groups and network ACLs for restricting the network access to your EKS clusters.
AWS team continuously commits to Kubernetes and several related products community, such as Heptio Authenticator. And vice versa, EKS supports many of open-source solutions built by Kubernetes community.
Answers for some most asked questions
What instance types are supported by EKS?
EKS supports many of EC2 instance types, such as t2, m3, m4, m5, c4, c5, i3, r3, r4, x1, p2 and p3 instances. The smallest recommended instance type is t2.small where ~1.1Gb of 2Gb memory will be available for the Kubernetes pods after ‘warming up’ Docker, Kubelet, and OS under some load.
What versions of OS, Docker and Kubernetes are used by EKS?
As of this writing, AWS EKS supports only Kubernetes version 1.10.3.
By default, Amazon EKS provides AWS CloudFormation templates to spin up your worker nodes with the Amazon EKS-optimized AMI. This AMI is built on top of Amazon Linux 2. The AMI is configured to work with Amazon EKS out of the box, and it includes Docker 17.06.2-ce (with overlay2 as a Docker storage driver), Kubelet 1.10.3, and the AWS authenticator. The AMI also launches with specialized Amazon EC2 user data that allows it to discover and connect to your cluster's control plane automatically. The example AMI for us-east-1 region is ami-dea4d5a1
But it is possible to build your own AMI that would include different OS, different kernel, different Docker version and settings and would still integrate with Amazon EKS.
What is used as a network driver for EKS and how well the network performs?
The AWS VPC container network interface (CNI) plugin is responsible for providing pod networking in Kubernetes using Elastic Network Interfaces on AWS. Amazon EKS works with Calico by Tigera to integrate with the CNI plugin to provide fine grained networking policies.
Our tests shown that the CNI plugin itself is not becoming the network bottleneck when running worker instances with 10Gbps network which we can’t say about other popular network drivers for Kubernetes, such as Flannel and Opencontrail.
Is adding / removing nodes automatically by the load possible?
Yes, you can launch an Auto Scaling group of worker nodes and register it with your EKS cluster. The recommended AWS CloudFormation templates for EKS already come with autoscaling group that launches the on-demand worker instances.
Can I SSH to worker nodes and manage them?
Yes, as worker nodes are just usual EC2 instances running in your VPC.
Can EKS use reserved or spot instances?
Yes. You can use reserved or spot instances for EKS worker nodes in the same way as for your other EC2 instances. As of this writing, there are no reservation plans for EKS control plain aka Master nodes, but probably such ones will appear in future.
What existing Kubernetes addons can be used with Amazon EKS?
As of this writing, we tested the DNS and Kubernetes Dashboard addons with EKS, and they both worked well. For example, the DNS addon comes enabled out-the-box and runs DNS pod in 3 replicas which is, by our experience, absolutely enough for production usage. We will provide more updates when we test logs and metrics aggregation addons under production load.
How to migrate the existing Kubernetes cluster to EKS?
If your system is already running in Kubernetes the migration will not require changing the code of your applications.
But for legacy applications that are not designed for running in Docker and Kubernetes or doesn’t support running more than one replica of the same app, some code and design changes may be required.
Some AWS partner companies already have a big experience of moving legacy applications to the cloud (including containerization, refactoring workloads for better scaling and moving them into the cloud). But only some of these companies, including us, already learned Amazon EKS and added it to the list of officially supported cloud services for migration 🙂 So we recommend you to contact us for complicated cloud migration cases, especially ones that involve Amazon EKS.
I already have the kubectl installed. What else do I need to connect to EKS?
If you already have kubectl version 1.10+ then for using Amazon EKS you will only need to install the heptio-authenticator-aws. See this guide.
kubectl exec/proxy/logs/port-forward commands work with EKS?
Yes, thanks to the fact that Amazon EKS provisions elastic network interfaces in your VPC subnets to provide connectivity from the control plane instances to the worker nodes
More words on EKS security
- User management and RBAC. When an Amazon EKS cluster is created, the IAM user that creates the cluster is added to the Kubernetes RBAC as the administrator. Then it is a purpose of this EKS/RBAC administrator user to add more users and setup access restrictions for them.
- IAM root user. It is possible to create the EKS cluster under root AWS user but it is impossible to authenticate to it later so EKS clusters only an be used when created under a non-root IAM user.
- Private Kubernetes cluster with EKS? When deploying Kubernetes on EC2 hosts without EKS, it is possible to setup the internal load balancer and VPN around K8s master and worker nodes so kubectl access to the cluster is only allowed via VPN. We didn’t found this to be possible with Amazon EKS. But we believe this is a suspect to be changed in future and to be documented properly in AWS docs.
Nuances that we faced with Amazon EKS
- IAM permissions: we noticed that IAM user with AdministratorAccess role assigned can’t use AWS EKS until additionally assigned the eks:* permissions to him. This behavior differs from all other AWS services, for which AWS admin users with AdministratorAccess permission (“Action”: “*”, “Resource”: “*”) usually have the full access automatically
- The heptio-authenticator binary download is very slow. We experienced 10Kb/sec download speeds from AWS-recommended endpoints serving the heptio-authenticator. Probably, this is caused by the increased activity over EKS and will be fixed soon but currently, downloading the heptio-authenticator is taking a half of time to spin up the AWS EKS cluster from scratch 😉
- For authentication via heptio-authenticatior, EKS uses the alpha feature of runnning command on authentication (note the
apiVersion: client.authentication.k8s.io/v1alpha1line on example kubectl config at this AWS documentation page. But we haven’t seen any issues with this alpha feature so far after using it on production for some time, so we still can recommend it.
Amazon changes Kubernetes code base? What does it mean for Kubernetes?
Yes, Amazon team commits to Kubernetes and several related products, such as Heptio Authenticator, but Amazon is not even in the top 100 committees list for any of those. So don't be afraid, AWS will not redesign Kubernetes 🙂 In the same time, many AWS fixes to Kubernetes platform not only optimizes Kubernetes for better running on AWS, but also improves the Kubernetes as a platform itself which is very good for community.
Should I redesign the system to start using EKS?
If your system already consists of dockerized microservices that can run in Kubernetes, then the short answer is “no”, you shouldn’t. Amazon EKS is certified to be a fully Kubernetes-compatible. Applications running on any standard Kubernetes environment are fully compatible and can be easily migrated to Amazon EKS. That being said, if your system is ready for Kubernetes (or runs in Kubernetes already), then it is migratable to EKS. Otherwise, you need to dockerize the system components first and make sure that they may scale in Kubernetes. The good example of legacy system migration to Kubernetes can be found here. The more information about the migration process itself can be found here.
How much Amazon EKS costs?
As of this writing, the EKS cost is looking very impressive, especially for medium and large-scale workloads. Here are the main 4 items to review when forecasting your AWS EKS costs:
- You pay $0.20 per hour for each Amazon EKS cluster (control plane) that you create that makes about ~$150/month. In terms of infra costs, this is almost the same as expensive as running 3 Kubernetes master hosts on EC2 with load balancer but with this price AWS also takes all heavy lifting of Kubernetes master nodes management, such as security upgrades and failovers.
- In EKS, you also additionally pay the usual hourly price for any EC2 instances, EBS volumes and ELBs that you create to run your Kubernetes worker nodes and apps.
- The same as for all other AWS services, for EKS you only pay for what you use, as you use it; there are no minimum fees and no upfront commitments. For example, if you deploy the EKS cluster and run your workload on it for 24 hours you will be only changed for these 24 hours. There is also an option to stop your EKS worker nodes when you don’t need them to save costs without missing data on that nodes.
- You may use reserved instances to save costs in long term or use spot instances to save costs in short term if your workload can tolerate one or more instances eventually going down because the spot price became bigger than the spot bid.
Design rules for saving costs with EKS are the same as for designing system on ‘raw’ EC2, but with one key difference — Kubernetes platform can automatically failover any services deployed to it and it does it well!
Just as an example, if you use Kubernetes for scaling stateless services and you have properly configured the K8s ‘anti-affinity rules’ and enough replicas for each, you can replace each Kubernetes worker instance with a different one (different instance type, different capacity, switch from on-demand to spot instance) without downtime.
That being said, you can build an automation that monitors the AWS spot prices and failovers your cluster to on-demand instances faster than your current instances will be terminated because of spot price spike and failovers back when the spot spike is over. Or automation can select the most cost-effective spot instance type and migrate you EKS to it automatically.
That can effectively help you to save up to 90% of your AWS costs without impacting the SLAs or HA requirements too much which, by our experience, is usually affordable for most data processing systems.
When we recommend to use AWS EKS?
AWS EKS brings the benefits of container-based computing to organizations that want to focus on building applications instead of setting up a Kubernetes cluster from scratch.
Here are use cases that we recommend for AWS EKS:
- Running scalable stateless apps. Good examples are REST APIs and various data processing microservices
- Running resource-aggressive workflows on auto-scaled spot instances using failover capabilities provided by Kubernetes
- Running DB and queue server clusters that support running in Kubernetes PetSets and tolerates the failure of one node without downtime. Examples: MongoDB, Elasticsearch, Kafka
When we don’t recommend to use AWS EKS?
Use cases that we don’t recommend for AWS EKS:
- Deploying stateful Big data clusters. We don’t recommend to put your big DBs such as Hadoop/Hbase, HP Vertica or Cassandra into EKS
- Deploying products that are not designed for running and scaling in Docker. Any product that requires operator to change config files at disk to add or remove nodes (such as Apache HDFS) will become a pain when deployed into Kubernetes
- Deploying applications that can’t run in more than one replica. Kubernetes can only show it’s full power (such as automatic failovers through the pods eviction) on apps that run in more than 1 replica. If you app does not support running more than one replica of itself in the same network or with same DB or only supports the so-called ‘hot-standby’ mode then Kubernetes (and hence EKS) is probably not the best choice to deploy it. If you have legacy applications that can’t run it Docker or scale properly, then there are many companies on the market (including our one) that can help you to fix that seamlessly for your business.
We are also excited to announce that Bekitzur now supports the migration of self-hosted Kubernetes clusters to AWS EKS at any scale. So if you have such a migration case and you need help, please contact us!