Kubernetes is an open-source automation system originally designed by Google and now maintained under the Cloud Native Computing Foundation (CNCF) for the management, scaling, and deployment of containerized applications. A recent survey by the CNCF concluded that AWS has witnessed an increase in more Kubernetes clusters than any of the other leading cloud providers with ~ 69% coverage as the leading container deployment environment in use today as of late 2017.
Recently Amazon has presented its Amazon Elastic Container Service (EKS) for Kubernetes which has become generally available as of June 5th. We’ve recently completed our testing and review of Amazon EKS and would like to provide you with some information and recommendations regarding this service.
What is Amazon EKS?
Amazon EKS is Amazon’s ‘Kubernetes as a service’. It simplifies the process of spinning up and maintaining highly-available Kubernetes clusters in the AWS cloud.
The typical EKS cluster consists of two main components, each running on its own VPC – The Control Plane and Worker Nodes.
- Control Plane – Consists of three Kubernetes Master nodes running in three different AZs for high availability. Incoming traffic to Kubernetes API comes through the AWS network load balancer (NLB). Control Plane runs on the Amazon-controlled VPC that cannot be managed by the customer and is fully managed by Amazon.
- Worker Nodes – Run on usual Amazon EC2 instances at the customer-controlled VPC. Any AWS instance type can be used as a worker node. Worker nodes can be accessed via SSH or provisioned with any existing automation.
EKS is flexible in terms of layout. You can deploy one EKS (read ‘Kubernetes’) cluster per environment, system or application. Also, a single Amazon EKS cluster can be used to run multiple applications by taking advantage of Kubernetes namespaces and IAM security policies.
Below are the official visuals from AWS that better describe how the typical EKS cluster appears in AWS and how it works.
EKS usage algorithm. Source: Amazon AWS
EKS architecture schemas. Source: Amazon AWS
Frequently Asked Questions – FAQ
What instance types are supported by EKS?
EKS supports many EC2 instance types, such as t2, m3, m4, m5, c4, c5, i3, r3, r4, x1, p2 and p3 instances. The smallest recommended instance type is t2.small where ~1.1Gb of 2Gb memory will be available for the Kubernetes pods after ‘warming up’ Docker, Kubelet, and OS under some load.
What versions of OS, Docker and Kubernetes are used by EKS?
As of this blog post, AWS EKS supports only Kubernetes version 1.10.3.
By default, Amazon EKS provides AWS CloudFormation templates to spin up your worker nodes with the Amazon EKS-optimized AMI. This AMI is built on top of Amazon Linux 2. The AMI is configured to work with Amazon EKS out of the box and it includes Docker 17.06.2-ce (with overlay2 as a Docker storage driver), Kubelet 1.10.3, and the AWS authenticator. The AMI also launches with specialized Amazon EC2 user data that allows it to discover and connect to your cluster’s control plane automatically. The example AMI for us-east-1 region is ami-dea4d5a1.
It is possible, however, to build your own AMI that would include a different OS, kernel, or Docker version as well as different settings that would still integrate with Amazon EKS.
What is used as a network driver for EKS and how well does the network perform?
The AWS VPC container network interface (CNI) plugin is responsible for providing pod networking in Kubernetes using Elastic Network Interfaces on AWS. Amazon EKS works with Calico by Tigera to integrate with the CNI plugin to provide fine grained networking policies.
Our tests have shown that the CNI plugin itself is not producing a network bottleneck when running worker instances with a 10GB/s network – which we can’t say about some other popular network drivers for Kubernetes, such as Flannel and Opencontrail.
Is automatically adding / removing nodes by the load possible?
Yes, you can launch an Auto Scaling group of worker nodes and register it with your EKS cluster. The recommended AWS CloudFormation templates for EKS already come with an autoscaling group that launches the on-demand worker instances.
Can I SSH to worker nodes and manage them?
Yes, as worker nodes are just usual EC2 instances running in your VPC.
Can EKS use reserved or spot instances?
Yes. You can use reserved or spot instances for EKS worker nodes in the same way as your other EC2 instances. As of this writing, there are no reservation plans for the EKS control plane aka Master nodes, but such plans will likely appear in the future.
What existing Kubernetes addons can be used with Amazon EKS?
As of this writing, we tested the DNS and Kubernetes Dashboard addons with EKS, and they both worked well. For example, the DNS addon comes enabled out-the-box and runs DNS pod in 3 replicas which is, by our experience, quite enough for production usage. We will provide more updates when we test logs and metrics aggregation addons under production load.
How do you migrate the existing Kubernetes cluster to EKS?
If your system is already running in Kubernetes the migration will not require changing the code of your applications. However, for legacy applications that are not designed to run in Docker and Kubernetes or that don’t support running more than one replica of the same app, some code and design changes may be required.
Some AWS partner companies already have quite a bit of experience in moving legacy applications to the cloud (including containerization, refactoring workloads for better scaling, and moving them into the cloud). But only some of these companies, us included, have mastered Amazon EKS and added it to the list of officially supported cloud services for migration 🙂 As such, we encourage you to contact us for complicated cloud migration cases, especially ones that involve Amazon EKS.
I already have the kubectl installed. What else do I need to connect to EKS?
If you already have kubectl version 1.10+, then in order to use Amazon EKS you will only need to install the heptio-authenticator-aws. See this guide.
kubectl exec/proxy/logs/port-forward commands work with EKS?
Yes, thanks to the fact that Amazon EKS provisions elastic network interfaces in your VPC subnets to provide connectivity from the control plane instances to the worker nodes.
What is good about Amazon EKS?
Reduced maintenance overhead
All Kubernetes cluster spinup and management operations are well-automated by AWS which allows AWS customers that use EKS to focus more on their business and applications instead of spending their engineers’ time on Kubernetes clusters provisioning and maintenance.
Integration with AWS services
EKS does not force customers to migrate all workloads to AWS. With EKS one can still spread the workload over different clouds. Migrating from self-hosted Kubernetes installations to Amazon EKS typically does not require any significant changes to the existing system. At the same time, EKS integrates well with many common Amazon services, namely:
- VPC and availability zones. EKS supports multi-AZ deployment of worker nodes for high availability
- EKS load balances traffic for services using AWS Network Load Balancers, AWS Application Load Balancers and Elastic Load Balancers
- Route53. EKS supports assigning Route 53 DNS records to any service running in the cluster
- Autoscaling. Worker nodes are deployed in the autoscaling group. i.e., the cluster can automatically scale up and down by the load or by any CloudWatch metric
- AWS CloudWatch. All worker nodes are monitored with CloudWatch, like any other EC2 instances
- AWS IAM. Via open-source Heptio authenticator you can use IAM roles and policies to authenticate users to EKS and configure the fine-grained access to any Kubernetes namespaces created within it
- AWS EBS. EBS volumes are used for Kubernetes persistent storage
- AWS databases. RDS, DynamoDB, Elasticache, Redshift, and all other AWS databases are available from EKS. In many cases this allows you to keep only stateless components in EKS while running databases separately on AWS, which provides the more flexible failover paths for EKS-deployed components and at the same time reduces the maintenance overhead
- Elastic Network Interfaces (ENI) can be used for EKS Kubernetes cluster networking via the CNI plugin
- AWS CloudTrail records any actions that are done with your EKS clusters via AWS Console, SDK or rest API
As AWS EKS fully runs on AWS VPC and EC2, it inherits the security measures from these services. For example, you can use your own VPC security groups and network ACLs for restricting the network access to your EKS clusters. Amazon web application firewall and DDoS protection services such as AWS WAF and AWS Shield can be applied to your EKS installation in the same way as other systems hosted on Amazon.
The AWS team continuously supports Kubernetes as well as several related products in the community, such as Heptio Authenticator. Additionally, EKS supports many of the open-source solutions built by the Kubernetes community.
More on EKS security
- User management and RBAC. When an Amazon EKS cluster is created, the IAM user that creates the cluster is added to the Kubernetes RBAC as the administrator. It is then the responsibility of this EKS/RBAC administrator user to add more users and setup access restrictions for them.
- IAM root user. It is possible to create the EKS cluster under root AWS user but it is impossible to authenticate to it later, therefore EKS clusters can only be used when created under a non-root IAM user.
- Private Kubernetes cluster with EKS? When deploying Kubernetes on EC2 hosts without EKS, it is possible to set up the internal load balancer and VPN around K8s master and worker nodes so kubectl access to the cluster is only allowed via VPN. AWS EKS supports this and recommends deploying worker nodes to private subnets for better security.
Difficulties that we faced with Amazon EKS
- IAM permissions. We noticed that IAM user with the AdministratorAccess role assigned cannot use AWS EKS until additionally assigned the eks:* permissions. This behavior differs from all other AWS services, for which AWS admin users with AdministratorAccess permission (“Action”: “*”, “Resource”: “*”) are usually granted full access automatically.
- The heptio-authenticator binary download is very slow. We experienced 10Kb/s download speeds from AWS-recommended endpoints serving the heptio-authenticator. In all likelihood, this is caused by the increased activity across EKS and will be fixed soon – But currently, downloading the heptio-authenticator is taking half the time to spin up the AWS EKS cluster from scratch 😉
- For authentication via heptio-authenticator, EKS uses the alpha feature for running the command on authentication (note the apiVersion: client.authentication.k8s.io/v1alpha1 line as an example of a kubectl config at this AWS documentation page). We haven’t encountered any issues with this alpha feature so far after using it in production for some time, so we can still recommend it.
Amazon changes the Kubernetes code base – What does that mean for Kubernetes?
Yes, the Amazon team commits to Kubernetes and several related products, such as Heptio Authenticator, but Amazon is not even in the top 100 committees list for any of those. So don’t be afraid, AWS will not redesign Kubernetes 🙂 At the same time, many AWS fixes to the Kubernetes platform have not only optimized Kubernetes to better run on AWS but have also improved Kubernetes as a platform itself which is, overall, good for the community.
Should I redesign my system to start using EKS?
If your system already consists of dockerized microservices that can run in Kubernetes, then the short answer is “no”, you shouldn’t. Amazon EKS is certified to be fully Kubernetes-compatible. Applications running on any standard Kubernetes environment are fully compatible and can be easily migrated to Amazon EKS. That being said, if your system is ready for Kubernetes (or runs in Kubernetes already), then it can easily be migrated to EKS. Otherwise, you need to dockerize the system components first and make sure that they can scale in Kubernetes. A good example of legacy system migration to Kubernetes can be found here. More information about the migration process itself can be found here.
How much does Amazon EKS cost?
As of this writing, EKS is quite affordable, especially for medium and large-scale workloads. Here are the 4 main items to review when forecasting your AWS EKS costs:
- You pay $0.20 per hour for each Amazon EKS cluster (control plane) that you create which makes about ~$150/month. In terms of infra costs, this is almost the same as running 3 Kubernetes master hosts on EC2 with a load balancer. But, with this price AWS also shoulders the heavy lifting of Kubernetes master nodes management, such as security upgrades and failovers.
- In EKS, you also additionally pay the usual hourly price for any EC2 instances, EBS volumes and ELBs that you create to run your Kubernetes worker nodes and apps.
- As is the same for all other AWS services, with EKS you only pay for what you use, as you use it; there are no minimum fees and no upfront commitments. For example, if you deploy the EKS cluster and run your workload on it for 24 hours, you will only be charged for these 24 hours. There is also an option to stop your EKS worker nodes when you don’t need them in order to save costs without missing data on those nodes.
- You may use reserved instances to save costs in the long term or use spot instances to save costs in the short term if your workload can tolerate one or more instances eventually going down; the spot price becomes more expensive than the spot bid.
The design rules for saving costs with EKS are the same as for designing a system on ‘raw’ EC2, but with one key difference — the Kubernetes platform can automatically failover any services deployed to it, and it does it well!
Just as an example, if you use Kubernetes for scaling stateless services and you have properly configured the K8s ‘anti-affinity rules’ and enough replicas for each, you can replace each Kubernetes worker instance with a different one (different instance type, different capacity, switch from on-demand to spot instance) without downtime.
That being said, you can build an automation that:
- Automatically selects the best (e.g. the most cost-effective) AWS instance type and switches Kubernetes workers to it one by one
- When there is a spot price spike, failovers to on-demand instances faster than spots will be terminated by the AWS
That can effectively help you to save up to 80% of your AWS costs without impacting the worst-case SLAs or HA requirements too much, which, by our experience, is usually affordable for most data processing systems.
When would we recommend to use AWS EKS?
AWS EKS brings the benefits of container-based computing to organizations that want to focus on building applications instead of setting up a Kubernetes cluster from scratch.
Use cases that we recommend for AWS EKS:
- Running scalable stateless apps. Examples: REST APIs and various data processing microservices
- Running resource-aggressive workflows on auto-scaled spot instances using failover capabilities provided by Kubernetes
- Running DB and queue server clusters that support running in Kubernetes PetSets and tolerate the failure of one node without downtime. Examples: MongoDB, Elasticsearch, Kafka
When would we not recommend to use AWS EKS?
Use cases that we don’t recommend for AWS EKS:
- Deploying stateful Big data clusters. We don’t recommend to put your big DBs such as Hadoop/Hbase, HP Vertica or Cassandra into EKS.
- Deploying products that are not designed for running and scaling in Docker. Any product that requires the operator to change config files on the disk to add or remove nodes (such as Apache HDFS) will become a pain when deployed in Kubernetes.
- Deploying applications that can’t run in more than one replica. Kubernetes can only show it’s full power (such as automatic failovers through the pods eviction) on apps that run in more than one replica. If your app does not support running more than one replica of itself in the same network or with the same DB or only supports the so-called ‘hot-standby’ mode, then Kubernetes (and hence EKS) is probably not the best choice to deploy it. If you have legacy applications that can’t run in Docker or scale properly, then there are many companies on the market (including our own) that can help you seamlessly fix this issue for your business.
We are also excited to announce that Bekitzur now supports the migration of self-hosted Kubernetes clusters to AWS EKS at any scale. So, if you have such a migration case and you need help, please contact us!