Amazon EKS

pod-id-configure-pods.title

Amazon EKS aims to give you a wide selection of options to cover all use cases. If you develop a commercially supported OIDC compatible identity provider that is not listed here, then contact our partner team at aws-container-partners@amazon. com for more information.

8.4.6. Disassociate an `OIDC` identity provider from your cluster

If you disassociate an OIDC identity provider from your cluster, users included in the provider can no longer access the cluster. However, you can still access the cluster with IAM principals.

Open the Amazon EKS console.
In the OIDC Identity Providers section, select Disassociate, enter the identity provider name, and then select Disassociate.

8.5. View `Kubernetes` resources in the `consolelong`

Learn how to view Kubernetes resources in the consolelong.

You can view the Kubernetes resources deployed to your cluster with the consolelong. You can’t view Kubernetes resources with the AWS CLI or eksctl. To view Kubernetes resources using a command-line tool, use kubectl.

To view the Resources tab and Nodes section on the Compute tab in the consolelong, the IAM principal that you’re using must have specific IAM and Kubernetes permissions. For more information, see view-kubernetes-resources-permissions.title.

Open the Amazon EKS console.
In the Clusters list, select the cluster that contains the Kubernetes resources that you want to view.
Select the Resources tab.
Select a Resource type group that you want to view resources for, such as Workloads. You see a list of resource types in that group.
Select a resource type, such as Deployments, in the Workloads group. You see a description of the resource type, a link to the Kubernetes documentation for more information about the resource type, and a list of resources of that type that are deployed on your cluster. If the list is empty, then there are no resources of that type deployed to your cluster.
Select a resource to view more information about it. Try the following examples:
- Select the Workloads group, select the Deployments resource type, and then select the coredns resource. When you select a resource, you are in Structured view, by default. For some resource types, you see a Pods section in Structured view. This section lists the Pods managed by the workload. You can select any Pod listed to view information about the Pod. Not all resource types display information in Structured View. If you select Raw view in the top right corner of the page for the resource, you see the complete JSON response from the Kubernetes API for the resource.
- Select the Cluster group and then select the Nodes resource type. You see a list of all nodes in your cluster. The nodes can be any Amazon EKS node type. This is the same list that you see in the Nodes section when you select the Compute tab for your cluster. Select a node resource from the list. In Structured view, you also see a Pods section. This section shows you all Pods running on the node.

8.5.1. Required permissions

To view the Resources tab and Nodes section on the Compute tab in the consolelong, the IAM principal that you’re using must have specific minimum IAM and Kubernetes permissions. Complete the following steps to assign the required permissions to your IAM principals.

Make sure that the eks:AccessKubernetesApi, and other necessary IAM permissions to view Kubernetes resources, are assigned to the IAM principal that you’re using. For more information about how to edit permissions for an IAM principal, see Controlling access for principals in the IAM User Guide. For more information about how to edit permissions for a role, see Modifying a role permissions policy (console) in the IAM User Guide.

The following example policy includes the necessary permissions for a principal to view Kubernetes resources for all clusters in your account. Replace 111122223333 with your AWS account ID.
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:ListFargateProfiles",
                "eks:DescribeNodegroup",
                "eks:ListNodegroups",
                "eks:ListUpdates",
                "eks:AccessKubernetesApi",
                "eks:ListAddons",
                "eks:DescribeCluster",
                "eks:DescribeAddonVersions",
                "eks:ListClusters",
                "eks:ListIdentityProviderConfigs",
                "iam:ListRoles"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "ssm:GetParameter",
            "Resource": "region.arnssm:*:111122223333:parameter/*"
        }
    ]
}
```
To view nodes in connected clusters, the Amazon EKS connector IAM role should be able to impersonate the principal in the cluster. This allows the Connect a Kubernetes cluster to an Amazon EKS Management Console with Amazon EKS Connector to map the principal to a Kubernetes user.
Create a Kubernetes rolebinding or clusterrolebinding that is bound to a Kubernetes role or clusterrole that has the necessary permissions to view the Kubernetes resources. To learn more about Kubernetes roles and role bindings, see Using RBAC Authorization in the Kubernetes documentation. You can apply one of the following manifests to your cluster that create a role and rolebinding or a clusterrole and clusterrolebinding with the necessary Kubernetes permissions:
View Kubernetes resources in all namespaces
The group name in the file is eks-console-dashboard-full-access-group. Apply the manifest to your cluster with the following command:

kubectl apply -f https://s3.us-west-2.amazonaws.com/amazon-eks/docs/eks-console-full-access.yaml
View Kubernetes resources in a specific namespace
The namespace in this file is default. The group name in the file is eks-console-dashboard-restricted-access-group. Apply the manifest to your cluster with the following command:

kubectl apply -f https://s3.us-west-2.amazonaws.com/amazon-eks/docs/eks-console-restricted-access.yaml

If you need to change the Kubernetes group name, namespace, permissions, or any other configuration in the file, then download the file and edit it before applying it to your cluster:

Download the file with one of the following commands:

curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/docs/eks-console-full-access.yaml

curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/docs/eks-console-restricted-access.yaml

Edit the file as necessary.

Apply the manifest to your cluster with one of the following commands:

kubectl apply -f eks-console-full-access.yaml

kubectl apply -f eks-console-restricted-access.yaml

Map the IAM principal to the Kubernetes user or group in the aws-auth ConfigMap. You can use a tool such as eksctl to update the ConfigMap or you can update it manually by editing it.

Edit with eksctl

You need version 0.199.0 or later of the eksctl command line tool installed on your device or AWS CloudShell. To install or update eksctl, see Installation in the eksctl documentation.

View the current mappings in the ConfigMap. Replace my-cluster with the name of your cluster. Replace region-code with the AWS Region that your cluster is in.

eksctl get iamidentitymapping --cluster my-cluster --region=region-code

An example output is as follows.

ARN                                                                                             USERNAME                                GROUPS                          ACCOUNT
region.arniam::111122223333:role/eksctl-my-cluster-my-nodegroup-NodeInstanceRole-1XLS7754U3ZPA    system:node:{{EC2PrivateDNSName}}       system:bootstrappers,system:nodes

Add a mapping for a role. This example assume that you attached the IAM permissions in the first step to a role named my-console-viewer-role. Replace 111122223333 with your account ID.

eksctl create iamidentitymapping \
    --cluster my-cluster \
    --region=region-code \
    --arn region.arniam::111122223333:role/my-console-viewer-role \
    --group eks-console-dashboard-full-access-group \
    --no-duplicate-arns

An example output is as follows.

[...]
2022-05-09 14:51:20 [ℹ]  adding identity "region.arniam::111122223333:role/my-console-viewer-role" to auth ConfigMap

Add a mapping for a user. IAM best practices recommend that you grant permissions to roles instead of users. This example assume that you attached the IAM permissions in the first step to a user named my-user. Replace 111122223333 with your account ID.

eksctl create iamidentitymapping \
    --cluster my-cluster \
    --region=region-code \
    --arn region.arniam::111122223333:user/my-user \
    --group eks-console-dashboard-restricted-access-group \
    --no-duplicate-arns

An example output is as follows.

[...]
2022-05-09 14:53:48 [ℹ]  adding identity "region.arniam::111122223333:user/my-user" to auth ConfigMap

View the mappings in the ConfigMap again.

eksctl get iamidentitymapping --cluster my-cluster --region=region-code

An example output is as follows.

ARN                                                                                             USERNAME                                GROUPS                                  ACCOUNT
region.arniam::111122223333:role/eksctl-my-cluster-my-nodegroup-NodeInstanceRole-1XLS7754U3ZPA    system:node:{{EC2PrivateDNSName}}       system:bootstrappers,system:nodes
region.arniam::111122223333:role/my-console-viewer-role                                                                                   eks-console-dashboard-full-access-group
region.arniam::111122223333:user/my-user                                                                                                  eks-console-dashboard-restricted-access-group

Edit ConfigMap manually

For more information about adding users or roles to the aws-auth ConfigMap, see aws-auth-users.title.

Open the aws-auth ConfigMap for editing.

kubectl edit -n kube-system configmap/aws-auth

Add the mappings to the aws-auth ConfigMap, but don’t replace any of the existing mappings. The following example adds mappings between IAM principals with permissions added in the first step and the Kubernetes groups created in the previous step:

The my-console-viewer-role role and the eks-console-dashboard-full-access-group.

The my-user user and the eks-console-dashboard-restricted-access-group.

These examples assume that you attached the IAM permissions in the first step to a role named my-console-viewer-role and a user named my-user. Replace 111122223333 with your AWS account ID.

apiVersion: v1
data:
mapRoles: |
  - groups:
    - eks-console-dashboard-full-access-group
    rolearn: region.arniam::111122223333:role/my-console-viewer-role
    username: my-console-viewer-role
mapUsers: |
  - groups:
    - eks-console-dashboard-restricted-access-group
    userarn: region.arniam::111122223333:user/my-user
    username: my-user

The role ARN can’t include a path such as role/my-team/developers/my-console-viewer-role. The format of the ARN must be region.arniam::111122223333:role/my-console-viewer-role. In this example, my-team/developers/ needs to be removed.

Save the file and exit your text editor.

8.6. Connect `kubectl` to an EKS cluster by creating a `kubeconfig` file

Learn how to create or update a kubeconfig file for authenticating with your Amazon EKS cluster using kubectl. Follow prerequisites for required tools and permissions.

In this topic, you create a kubeconfig file for your cluster (or update an existing one).

The kubectl command-line tool uses configuration information in kubeconfig files to communicate with the API server of a cluster. For more information, see Organizing Cluster Access Using kubeconfig Files in the Kubernetes documentation.

Amazon EKS uses the aws eks get-token command with kubectl for cluster authentication. By default, the AWS CLI uses the same credentials that are returned with the following command:

aws sts get-caller-identity

An existing Amazon EKS cluster. To deploy one, see getting-started.title.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
An IAM user or role with permission to use the eks:DescribeCluster API action for the cluster that you specify. For more information, see security-iam-id-based-policy-examples.title. If you use an identity from your own OpenID Connect provider to access your cluster, then see Using kubectl in the Kubernetes documentation to create or update your kube config file.

8.6.1. Create `kubeconfig` file automatically

Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
Permission to use the eks:DescribeCluster API action for the cluster that you specify. For more information, see security-iam-id-based-policy-examples.title.
1. Create or update a kubeconfig file for your cluster. Replace region-code with the AWS Region that your cluster is in and replace my-cluster with the name of your cluster.
  aws eks update-kubeconfig --region region-code --name my-cluster
  By default, the resulting configuration file is created at the default kubeconfig path (.kube) in your home directory or merged with an existing config file at that location. You can specify another path with the --kubeconfig option.
  
  You can specify an IAM role ARN with the --role-arn option to use for authentication when you issue kubectl commands. Otherwise, the IAM principal in your default AWS CLI or SDK credential chain is used. You can view your default AWS CLI or SDK identity by running the aws sts get-caller-identity command.
  
  For all available options, run the aws eks update-kubeconfig help command or see update-kubeconfig in the AWS CLI Command Reference.
2. Test your configuration.
  kubectl get svc
  An example output is as follows.
  NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1m
  If you receive any authorization or resource type errors, see unauthorized.title in the troubleshooting topic.

8.7. Grant Kubernetes workloads access to `AWS` using `Kubernetes` Service Accounts

A Kubernetes service account provides an identity for processes that run in a Pod. For more information see Managing Service Accounts in the Kubernetes documentation. If your Pod needs access to AWS services, you can map the service account to an AWS Identity and Access Management identity to grant that access. For more information, see iam-roles-for-service-accounts.title.

8.7.1. Service account tokens

The BoundServiceAccountTokenVolume feature is enabled by default in Kubernetes versions. This feature improves the security of service account tokens by allowing workloads running on Kubernetes to request JSON web tokens that are audience, time, and key bound. Service account tokens have an expiration of one hour. In earlier Kubernetes versions, the tokens didn’t have an expiration. This means that clients that rely on these tokens must refresh the tokens within an hour. The following Kubernetes client SDKs refresh tokens automatically within the required time frame:

Go version 0.15.7 and later
Python version 12.0.0 and later
Java version 9.0.0 and later
JavaScript version 0.10.3 and later
Ruby master branch
Haskell version 0.3.0.0
C# version 7.0.5 and later

If your workload is using an earlier client version, then you must update it. To enable a smooth migration of clients to the newer time-bound service account tokens, Kubernetes adds an extended expiry period to the service account token over the default one hour. For Amazon EKS clusters, the extended expiry period is 90 days. Your Amazon EKS cluster’s Kubernetes API server rejects requests with tokens that are greater than 90 days old. We recommend that you check your applications and their dependencies to make sure that the Kubernetes client SDKs are the same or later than the versions listed previously.

When the API server receives requests with tokens that are greater than one hour old, it annotates the API audit log event with annotations.authentication.k8s.io/stale-token. The value of the annotation looks like the following example:

subject: system:serviceaccount:common:fluent-bit, seconds after warning threshold: 4185802.

If your cluster has control plane logging enabled, then the annotations are in the audit logs. You can use the following CloudWatch Logs Insights query to identify all the Pods in your Amazon EKS cluster that are using stale tokens:

fields @timestamp
|filter @logStream like /kube-apiserver-audit/
|filter @message like /seconds after warning threshold/
|parse @message "subject: *, seconds after warning threshold:*\"" as subject, elapsedtime

The subject refers to the service account that the Pod used. The elapsedtime indicates the elapsed time (in seconds) after reading the latest token. The requests to the API server are denied when the elapsedtime exceeds 90 days (7,776,000 seconds). You should proactively update your applications' Kubernetes client SDK to use one of the version listed previously that automatically refresh the token. If the service account token used is close to 90 days and you don’t have sufficient time to update your client SDK versions before token expiration, then you can terminate existing Pods and create new ones. This results in refetching of the service account token, giving you an additional 90 days to update your client version SDKs.

If the Pod is part of a deployment, the suggested way to terminate Pods while keeping high availability is to perform a roll out with the following command. Replace my-deployment with the name of your deployment.

kubectl rollout restart deployment/my-deployment

8.7.2. Cluster add-ons

The following cluster add-ons have been updated to use the Kubernetes client SDKs that automatically refetch service account tokens. We recommend making sure that the listed versions, or later versions, are installed on your cluster.

Amazon VPC CNI plugin for Kubernetes and metrics helper plugins version 1.8.0 and later. To check your current version or update it, see managing-vpc-cni.title and cni-metrics-helper.
CoreDNS version 1.8.4 and later. To check your current version or update it, see managing-coredns.title.
AWS Load Balancer Controller version 2.0.0 and later. To check your current version or update it, see aws-load-balancer-controller.title.
A current kube-proxy version. To check your current version or update it, see managing-kube-proxy.title.
AWS for Fluent Bit version 2.25.0 or later. To update your current version, see Releases on GitHub.
Fluentd image version 1.14.6-1.2 or later and Fluentd filter plugin for Kubernetes metadata version 2.11.1 or later.

8.7.3. Granting `AWS` Identity and Access Management permissions to workloads on Amazon Elastic Kubernetes Service clusters

Amazon EKS provides two ways to grant AWS Identity and Access Management permissions to workloads that run in Amazon EKS clusters: IAM roles for service accounts, and EKS Pod Identities.

IAM roles for service accounts: IAM roles for service accounts (IRSA) configures Kubernetes applications running on AWS with fine-grained IAM permissions to access various other AWS resources such as Amazon S3 buckets, Amazon DynamoDB tables, and more. You can run multiple applications together in the same Amazon EKS cluster, and ensure each application has only the minimum set of permissions that it needs. IRSA was build to support various Kubernetes deployment options supported by AWS such as Amazon EKS, Amazon EKS Anywhere, Red Hat OpenShift Service on AWS, and self managed Kubernetes clusters on Amazon EC2 instances. Thus, IRSA was build using foundational AWS service like IAM, and did not take any direct dependency on the Amazon EKS service and the EKS API. For more information, see iam-roles-for-service-accounts.title.
EKS Pod Identities: EKS Pod Identity offers cluster administrators a simplified workflow for authenticating applications to access various other AWS resources such as Amazon S3 buckets, Amazon DynamoDB tables, and more. EKS Pod Identity is for EKS only, and as a result, it simplifies how cluster administrators can configure Kubernetes applications to obtain IAM permissions. These permissions can now be easily configured with fewer steps directly through consolelong, EKS API, and AWS CLI, and there isn’t any action to take inside the cluster in any Kubernetes objects. Cluster administrators don’t need to switch between the EKS and IAM services, or use privileged IAM operations to configure permissions required by your applications. IAM roles can now be used across multiple clusters without the need to update the role trust policy when creating new clusters. IAM credentials supplied by EKS Pod Identity include role session tags, with attributes such as cluster name, namespace, service account name. Role session tags enable administrators to author a single role that can work across service accounts by allowing access to AWS resources based on matching tags. For more information, see pod-identities.title.

Comparing EKS Pod Identity and IRSA

At a high level, both EKS Pod Identity and IRSA enables you to grant IAM permissions to applications running on Kubernetes clusters. But they are fundamentally different in how you configure them, the limits supported, and features enabled. Below, we compare some of the key facets of both solutions.

Attribute EKS Pod Identity IRSA

Role extensibility

You have to setup each role once to establish trust with the newly-introduced Amazon EKS service principal pods.eks.amazonaws.com. After this one-time step, you don’t need to update the role’s trust policy each time that it is used in a new cluster.

You have to update the IAM role’s trust policy with the new EKS cluster OIDC provider endpoint each time you want to use the role in a new cluster.

Cluster scalability

EKS Pod Identity doesn’t require users to setup IAM OIDC provider, so this limit doesn’t apply.

Each EKS cluster has an OpenID Connect (OIDC) issuer URL associated with it. To use IRSA, a unique OpenID Connect provider needs to be created for each EKS cluster in IAM. IAM has a default global limit of 100 OIDC providers for each AWS account. If you plan to have more than 100 EKS clusters for each AWS account with IRSA, then you will reach the IAM OIDC provider limit.

Role scalability

EKS Pod Identity doesn’t require users to define trust relationship between IAM role and service account in the trust policy, so this limit doesn’t apply.

In IRSA, you define the trust relationship between an IAM role and service account in the role’s trust policy. By default, the length of trust policy size is 2048. This means that you can typically define 4 trust relationships in a single trust policy. While you can get the trust policy length limit increased, you are typically limited to a max of 8 trust relationships within a single trust policy.

Role reusability

AWS STS temporary credentials supplied by EKS Pod Identity include role session tags, such as cluster name, namespace, service account name. Role session tags enable administrators to author a single IAM role that can be used with multiple service accounts, with different effective permission, by allowing access to AWS resources based on tags attached to them. This is also called attribute-based access control (ABAC). For more information, see pod-id-abac.title.

AWS STS session tags are not supported. You can reuse a role between clusters but every pod receives all of the permissions of the role.

Environments supported

EKS Pod Identity is only available on Amazon EKS.

IRSA can be used such as Amazon EKS, Amazon EKS Anywhere, Red Hat OpenShift Service on AWS, and self managed Kubernetes clusters on Amazon EC2 instances.

EKS versions supported

EKS Kubernetes versions 1.24 or later. For the specific platform versions, see pod-id-cluster-versions.title.

All of the supported EKS cluster versions.

8.7.4. Learn how `EKS Pod Identity` grants pods access to `AWS` services

Learn how to provide AWS service access to your Kubernetes workloads with Amazon EKS Pod Identities, offering least privilege access, credential isolation, and auditability for enhanced security. Discover the benefits and considerations of this identity management solution for your Amazon EKS clusters.

Applications in a Pod’s containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials.

EKS Pod Identities provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance’s role, you associate an IAM role with a Kubernetes service account and configure your Pods to use the service account.

Each EKS Pod Identity association maps a role to a service account in a namespace in the specified cluster. If you have the same application in multiple clusters, you can make identical associations in each cluster without modifying the trust policy of the role.

If a pod uses a service account that has an association, Amazon EKS sets environment variables in the containers of the pod. The environment variables configure the AWS SDKs, including the AWS CLI, to use the EKS Pod Identity credentials.

Benefits of EKS Pod Identities

EKS Pod Identities provide the following benefits:

Least privilege – You can scope IAM permissions to a service account, and only Pods that use that service account have access to those permissions. This feature also eliminates the need for third-party solutions such as kiam or kube2iam.
Credential isolation – A Pod’s containers can only retrieve credentials for the IAM role that’s associated with the service account that the container uses. A container never has access to credentials that are used by other containers in other Pods. When using Pod Identities, the Pod’s containers also have the permissions assigned to the create-node-role.title, unless you block Pod access to the Amazon EC2 Instance Metadata Service (IMDS). For more information, see Restrict access to the instance profile assigned to the worker node.
Auditability – Access and event logging is available through AWS CloudTrail to help facilitate retrospective auditing.

EKS Pod Identity is a simpler method than iam-roles-for-service-accounts.title, as this method doesn’t use OIDC identity providers. EKS Pod Identity has the following enhancements:

Independent operations – In many organizations, creating OIDC identity providers is a responsibility of different teams than administering the Kubernetes clusters. EKS Pod Identity has clean separation of duties, where all configuration of EKS Pod Identity associations is done in Amazon EKS and all configuration of the IAM permissions is done in IAM.
Reusability – EKS Pod Identity uses a single IAM principal instead of the separate principals for each cluster that IAM roles for service accounts use. Your IAM administrator adds the following principal to the trust policy of any role to make it usable by EKS Pod Identities.
```
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            }
```
Scalability — Each set of temporary credentials are assumed by the EKS Auth service in EKS Pod Identity, instead of each AWS SDK that you run in each pod. Then, the Amazon EKS Pod Identity Agent that runs on each node issues the credentials to the SDKs. Thus the load is reduced to once for each node and isn’t duplicated in each pod. For more details of the process, see pod-id-how-it-works.title.

For more information to compare the two alternatives, see service-accounts.title.

Overview of setting up EKS Pod Identities

Turn on EKS Pod Identities by completing the following procedures:

pod-id-agent-setup.title — You only complete this procedure once for each cluster. You do not need to complete this step if EKS Auto Mode is enabled on your cluster.
pod-id-association.title — Complete this procedure for each unique set of permissions that you want an application to have.
pod-id-configure-pods.title — Complete this procedure for each Pod that needs access to AWS services.
pod-id-minimum-sdk.title — Confirm that the workload uses an AWS SDK of a supported version and that the workload uses the default credential chain.

EKS Pod Identity considerations

You can associate one IAM role to each Kubernetes service account in each cluster. You can change which role is mapped to the service account by editing the EKS Pod Identity association.
You can only associate roles that are in the same AWS account as the cluster. You can delegate access from another account to the role in this account that you configure for EKS Pod Identities to use. For a tutorial about delegating access and AssumeRole, see Delegate access across AWS accounts using IAM roles in the IAM User Guide.
The EKS Pod Identity Agent is required. It runs as a Kubernetes DaemonSet on your nodes and only provides credentials to pods on the node that it runs on. For more information about EKS Pod Identity Agent compatibility, see the following section pod-id-restrictions.title.
If you are using Security Group for Pods along with Pod Identity Agent, you may need to set the POD_SECURITY_GROUP_ENFORCING_MODE Flag for the AWS VPC CNI. For more information on security group for pods considerations, see security-groups-for-pods.title.
The EKS Pod Identity Agent uses the hostNetwork of the node and it uses port 80 and port 2703 on a link-local address on the node. This address is 169.254.170.23 for IPv4 and [fd00:ec2::23] for IPv6 clusters.

If you disable IPv6 addresses, or otherwise prevent localhost IPv6 IP addresses, the agent can’t start. To start the agent on nodes that can’t use IPv6, follow the steps in pod-id-agent-config-ipv6.title to disable the IPv6 configuration.

EKS Pod Identity cluster versions

To use EKS Pod Identities, the cluster must have a platform version that is the same or later than the version listed in the following table, or a Kubernetes version that is later than the versions listed in the table.

Kubernetes version Platform version

1.31

eks.4

1.30

eks.2

1.29

eks.1

1.28

eks.4

1.27

eks.8

1.26

eks.9

1.25

eks.10

1.24

eks.13

EKS Pod Identity restrictions

EKS Pod Identities are available on the following:

Amazon EKS cluster versions listed in the previous topic pod-id-cluster-versions.title.
Worker nodes in the cluster that are Linux Amazon EC2 instances.

EKS Pod Identities aren’t available on the following:

AWS Outposts.
Amazon EKS Anywhere.
Kubernetes clusters that you create and run on Amazon EC2. The EKS Pod Identity components are only available on Amazon EKS.

You can’t use EKS Pod Identities with:

Pods that run anywhere except Linux Amazon EC2 instances. Linux and Windows pods that run on AWS Fargate (Fargate) aren’t supported. Pods that run on Windows Amazon EC2 instances aren’t supported.

Understand how `EKS Pod Identity` works

Learn how Amazon EKS Pod Identity works to provide temporary credentials to your Kubernetes workloads, using an agent running on each node and the AWS SDKs.

Amazon EKS Pod Identity associations provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances.

Amazon EKS Pod Identity provides credentials to your workloads with an additional EKS Auth API and an agent pod that runs on each node.

In your add-ons, such as Amazon EKS add-ons and self-managed controller, operators, and other add-ons, the author needs to update their software to use the latest AWS SDKs. For the list of compatibility between EKS Pod Identity and the add-ons produced by Amazon EKS, see the previous section pod-id-restrictions.title.

Using EKS Pod Identities in your code

In your code, you can use the AWS SDKs to access AWS services. You write code to create a client for an AWS service with an SDK, and by default the SDK searches in a chain of locations for AWS Identity and Access Management credentials to use. After valid credentials are found, the search is stopped. For more information about the default locations used, see the Credential provider chain in the AWS SDKs and Tools Reference Guide.

EKS Pod Identities have been added to the Container credential provider which is searched in a step in the default credential chain. If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an EKS Pod Identity association for the same workload. This way you can safely migrate from other types of credentials by creating the association first, before removing the old credentials.

The container credentials provider provides temporary credentials from an agent that runs on each node. In Amazon EKS, the agent is the Amazon EKS Pod Identity Agent and on Amazon Elastic Container Service the agent is the amazon-ecs-agent. The SDKs use environment variables to locate the agent to connect to.

In contrast, IAM roles for service accounts provides a web identity token that the AWS SDK must exchange with AWS Security Token Service by using AssumeRoleWithWebIdentity.

How EKS Pod Identity Agent works with a `Pod`

When Amazon EKS starts a new pod that uses a service account with an EKS Pod Identity association, the cluster adds the following content to the Pod manifest:

    env:
    - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
      value: "/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token"
    - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
      value: "http://169.254.170.23/v1/credentials"
    volumeMounts:
    - mountPath: "/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/"
      name: eks-pod-identity-token
  volumes:
  - name: eks-pod-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: pods.eks.amazonaws.com
          expirationSeconds: 86400 # 24 hours
          path: eks-pod-identity-token

Kubernetes selects which node to run the pod on. Then, the Amazon EKS Pod Identity Agent on the node uses the AssumeRoleForPodIdentity action to retrieve temporary credentials from the EKS Auth API.
The EKS Pod Identity Agent makes these credentials available for the AWS SDKs that you run inside your containers.
You use the SDK in your application without specifying a credential provider to use the default credential chain. Or, you specify the container credential provider. For more information about the default locations used, see the Credential provider chain in the AWS SDKs and Tools Reference Guide.

The SDK uses the environment variables to connect to the EKS Pod Identity Agent and retrieve the credentials.

If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an EKS Pod Identity association for the same workload.

Set up the Amazon EKS Pod Identity Agent

Learn how to set up the EKS Pod Identity Agent for your cluster.

Amazon EKS Pod Identity associations provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances.

Amazon EKS Pod Identity provides credentials to your workloads with an additional EKS Auth API and an agent pod that runs on each node.

You do not need to install the EKS Pod Identity Agent on EKS Auto Mode Clusters. This capability is built into EKS Auto Mode.

Considerations

By default, the EKS Pod Identity Agent listens on an IPv4 and IPv6 address for pods to request credentials. The agent uses the loopback (localhost) IP address 169.254.170.23 for IPv4 and the localhost IP address [fd00:ec2::23] for IPv6.
If you disable IPv6 addresses, or otherwise prevent localhost IPv6 IP addresses, the agent can’t start. To start the agent on nodes that can’t use IPv6, follow the steps in pod-id-agent-config-ipv6.title to disable the IPv6 configuration.

Creating the Amazon EKS Pod Identity Agent

Agent prerequisites

An existing Amazon EKS cluster. To deploy one, see getting-started.title. The cluster version and platform version must be the same or later than the versions listed in EKS Pod Identity cluster versions.
The node role has permissions for the agent to do the AssumeRoleForPodIdentity action in the EKS Auth API. You can use the AWS managed policy: AmazonEKSWorkerNodePolicy or add a custom policy similar to the following:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks-auth:AssumeRoleForPodIdentity"
            ],
            "Resource": "*"
        }
    ]
}
```
This action can be limited by tags to restrict which roles can be assumed by pods that use the agent.
The nodes can reach and download images from Amazon ECR. The container image for the add-on is in the registries listed in View Amazon container image registries for Amazon EKS add-ons.

Note that you can change the image location and provide imagePullSecrets for EKS add-ons in the Optional configuration settings in the consolelong, and in the --configuration-values in the AWS CLI.
The nodes can reach the Amazon EKS Auth API. For private clusters, the eks-auth endpoint in AWS PrivateLink is required.

Setup agent with AWS console

Open the Amazon EKS console.
In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the EKS Pod Identity Agent add-on for.
Choose the Add-ons tab.
Choose Get more add-ons.
Select the box in the top right of the add-on box for EKS Pod Identity Agent and then choose Next.
On the Configure selected add-ons settings page, select any version in the Version dropdown list.
(Optional) Expand Optional configuration settings to enter additional configuration. For example, you can provide an alternative container image location and ImagePullSecrets. The JSON Schema with accepted keys is shown in Add-on configuration schema.

Enter the configuration keys and values in Configuration values.
Choose Next.

Confirm that the EKS Pod Identity Agent pods are running on your cluster.

kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'

An example output is as follows.

eks-pod-identity-agent-gmqp7                                          1/1     Running   1 (24h ago)   24h
eks-pod-identity-agent-prnsh                                          1/1     Running   1 (24h ago)   24h

You can now use EKS Pod Identity associations in your cluster. For more information, see pod-id-association.title.

Setup agent with AWS CLI

Run the following AWS CLI command. Replace my-cluster with the name of your cluster.
```
aws eks create-addon --cluster-name my-cluster --addon-name eks-pod-identity-agent --addon-version v1.0.0-eksbuild.1
```
The EKS Pod Identity Agent doesn’t use the service-account-role-arn for IAM roles for service accounts. You must provide the EKS Pod Identity Agent with permissions in the node role.

Confirm that the EKS Pod Identity Agent pods are running on your cluster.

kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'

An example output is as follows.

eks-pod-identity-agent-gmqp7                                          1/1     Running   1 (24h ago)   24h
eks-pod-identity-agent-prnsh                                          1/1     Running   1 (24h ago)   24h

You can now use EKS Pod Identity associations in your cluster. For more information, see pod-id-association.title.

Assign an `IAM` role to a `Kubernetes` service account

Learn how to configure a Kubernetes service account to assume an AWS IAM role with Amazon EKS Pod Identity for securely accessing AWS services from your pods.

This topic covers how to configure a Kubernetes service account to assume an AWS Identity and Access Management (IAM) role with EKS Pod Identity. Any Pods that are configured to use the service account can then access any AWS service that the role has permissions to access.

To create an EKS Pod Identity association, there is only a single step; you create the association in EKS through the consolelong, AWS CLI, AWS SDKs, AWS CloudFormation and other tools. There isn’t any data or metadata about the associations inside the cluster in any Kubernetes objects and you don’t add any annotations to the service accounts.

An existing cluster. If you don’t have one, you can create one by following one of the guides in getting-started.title.
The IAM principal that is creating the association must have iam:PassRole.
The latest version of the AWS CLI installed and configured on your device or AWS CloudShell. You can check your current version with aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version installed in the AWS CloudShell may also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.
An existing kubectl config file that contains your cluster configuration. To create a kubectl config file, see create-kubeconfig.title.

Create a Pod Identity association (`AWS` Console)

Open the Amazon EKS console.
In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the EKS Pod Identity Agent add-on for.
Choose the Access tab.
In the Pod Identity associations, choose Create.
For the IAM role, select the IAM role with the permissions that you want the workload to have.

The list only contains roles that have the following trust policy which allows EKS Pod Identity to use them.
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}
```
sts:AssumeRole — EKS Pod Identity uses AssumeRole to assume the IAM role before passing the temporary credentials to your pods.

sts:TagSession — EKS Pod Identity uses TagSession to include session tags in the requests to AWS STS.

You can use these tags in the condition keys in the trust policy to restrict which service accounts, namespaces, and clusters can use this role.

For a list of Amazon EKS condition keys, see Conditions defined by Amazon Elastic Kubernetes Service in the Service Authorization Reference. To learn which actions and resources you can use a condition key with, see Actions defined by Amazon Elastic Kubernetes Service.
For the Kubernetes namespace, select the Kubernetes namespace that contains the service account and workload. Optionally, you can specify a namespace by name that doesn’t exist in the cluster.
For the Kubernetes service account, select the Kubernetes service account to use. The manifest for your Kubernetes workload must specify this service account. Optionally, you can specify a service account by name that doesn’t exist in the cluster.
(Optional) For the Tags, choose Add tag to add metadata in a key and value pair. These tags are applied to the association and can be used in IAM policies.

You can repeat this step to add multiple tags.
Choose Create.

Create a Pod Identity association (`AWS` CLI)

If you want to associate an existing IAM policy to your IAM role, skip to the next step.

Create an IAM policy. You can create your own policy, or copy an AWS managed policy that already grants some of the permissions that you need and customize it to your specific requirements. For more information, see Creating IAM policies in the IAM User Guide.
1. Create a file that includes the permissions for the AWS services that you want your Pods to access. For a list of all actions for all AWS services, see the Service Authorization Reference.
  
  You can run the following command to create an example policy file that allows read-only access to an Amazon S3 bucket. You can optionally store configuration information or a bootstrap script in this bucket, and the containers in your Pod can read the file from the bucket and load it into your application. If you want to create this example policy, copy the following contents to your device. Replace my-pod-secrets-bucket with your bucket name and run the command.
  cat >my-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "region.arns3:::my-pod-secrets-bucket" } ] } EOF
2. Create the IAM policy.
  aws iam create-policy --policy-name my-policy --policy-document file://my-policy.json

Create an IAM role and associate it with a Kubernetes service account.

If you have an existing Kubernetes service account that you want to assume an IAM role, then you can skip this step.

Create a Kubernetes service account. Copy the following contents to your device. Replace my-service-account with your desired name and default with a different namespace, if necessary. If you change default, the namespace must already exist.
```
cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default
EOF
kubectl apply -f my-service-account.yaml
```
Run the following command.
```
kubectl apply -f my-service-account.yaml
```

Run the following command to create a trust policy file for the IAM role.

cat >trust-relationship.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}
EOF

Create the role. Replace my-role with a name for your IAM role, and my-role-description with a description for your role.

aws iam create-role --role-name my-role --assume-role-policy-document file://trust-relationship.json --description "my-role-description"

Attach an IAM policy to your role. Replace my-role with the name of your IAM role and my-policy with the name of an existing policy that you created.
```
aws iam attach-role-policy --role-name my-role --policy-arn=region.arniam::111122223333:policy/my-policy
```
Unlike IAM roles for service accounts, EKS Pod Identity doesn’t use an annotation on the service account.

Run the following command to create the association. Replace my-cluster with the name of the cluster, replace my-service-account with your desired name and default with a different namespace, if necessary.

aws eks create-pod-identity-association --cluster-name my-cluster --role-arn region.arniam::111122223333:role/my-role --namespace default --service-account my-service-account

An example output is as follows.

{
    "association": {
        "clusterName": "my-cluster",
        "namespace": "default",
        "serviceAccount": "my-service-account",
        "roleArn": "region.arniam::111122223333:role/my-role",
        "associationArn": "region.arn:111122223333:podidentityassociation/my-cluster/a-abcdefghijklmnop1",
        "associationId": "a-abcdefghijklmnop1",
        "tags": {},
        "createdAt": 1700862734.922,
        "modifiedAt": 1700862734.922
    }
}

You can specify a namespace and service account by name that doesn’t exist in the cluster. You must create the namespace, service account, and the workload that uses the service account for the EKS Pod Identity association to function.

Confirm configuration

Confirm that the IAM role’s trust policy is configured correctly.

aws iam get-role --role-name my-role --query Role.AssumeRolePolicyDocument

An example output is as follows.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow EKS Auth service to assume this role for Pod Identities",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

Confirm that the policy that you attached to your role in a previous step is attached to the role.

aws iam list-attached-role-policies --role-name my-role --query AttachedPolicies[].PolicyArn --output text

An example output is as follows.

region.arniam::111122223333:policy/my-policy

Set a variable to store the Amazon Resource Name (ARN) of the policy that you want to use. Replace my-policy with the name of the policy that you want to confirm permissions for.
```
export policy_arn=region.arniam::111122223333:policy/my-policy
```

View the default version of the policy.

aws iam get-policy --policy-arn $policy_arn

An example output is as follows.

{
    "Policy": {
        "PolicyName": "my-policy",
        "PolicyId": "EXAMPLEBIOWGLDEXAMPLE",
        "Arn": "region.arniam::111122223333:policy/my-policy",
        "Path": "/",
        "DefaultVersionId": "v1",
        [...]
    }
}

View the policy contents to make sure that the policy includes all the permissions that your Pod needs. If necessary, replace 1 in the following command with the version that’s returned in the previous output.
```
aws iam get-policy-version --policy-arn $policy_arn --version-id v1
```
An example output is as follows.
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "region.arns3:::my-pod-secrets-bucket"
        }
    ]
}
```
If you created the example policy in a previous step, then your output is the same. If you created a different policy, then the example content is different.

Next Steps

Configure `pods` to access `AWS` services with service accounts

Learn how to configure Pods to use a Kubernetes service account with an associated IAM role for accessing AWS services on Amazon EKS.

If a Pod needs to access AWS services, then you must configure it to use a Kubernetes service account. The service account must be associated to an AWS Identity and Access Management (IAM) role that has permissions to access the AWS services.

An existing cluster. If you don’t have one, you can create one using one of the guides in getting-started.title.
An existing Kubernetes service account and an EKS Pod Identity association that associates the service account with an IAM role. The role must have an associated IAM policy that contains the permissions that you want your Pods to have to use AWS services. For more information about how to create the service account and role, and configure them, see pod-id-association.title.
The latest version of the AWS CLI installed and configured on your device or AWS CloudShell. You can check your current version with aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version installed in the AWS CloudShell may also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.

An existing kubectl config file that contains your cluster configuration. To create a kubectl config file, see create-kubeconfig.title.

Use the following command to create a deployment manifest that you can deploy a Pod to confirm configuration with. Replace the example values with your own values.

cat >my-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      serviceAccountName: my-service-account
      containers:
      - name: my-app
        image: public.ecr.aws/nginx/nginx:X.XX
EOF

Deploy the manifest to your cluster.
```
kubectl apply -f my-deployment.yaml
```

Confirm that the required environment variables exist for your Pod.

View the Pods that were deployed with the deployment in the previous step.
```
kubectl get pods | grep my-app
```
An example output is as follows.
```
my-app-6f4dfff6cb-76cv9   1/1     Running   0          3m28s
```

Confirm that the Pod has a service account token file mount.

kubectl describe pod my-app-6f4dfff6cb-76cv9 | grep AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE:

An example output is as follows.

AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE:  /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

Confirm that your Pods can interact with the AWS services using the permissions that you assigned in the IAM policy attached to your role.

When a Pod uses AWS credentials from an IAM role that’s associated with a service account, the AWS CLI or other SDKs in the containers for that Pod use the credentials that are provided by that role. If you don’t restrict access to the credentials that are provided to the Amazon EKS node IAM role, the Pod still has access to these credentials. For more information, see Restrict access to the instance profile assigned to the worker node.

If your Pods can’t interact with the services as you expected, complete the following steps to confirm that everything is properly configured.

Confirm that your Pods use an AWS SDK version that supports assuming an IAM role through an EKS Pod Identity association. For more information, see pod-id-minimum-sdk.title.

Confirm that the deployment is using the service account.

kubectl describe deployment my-app | grep "Service Account"

An example output is as follows.

Service Account:  my-service-account

Grant `pods` access to `AWS` resources based on tags

Learn how to use Amazon EKS Pod Identity to attach tags for cluster, namespace, and service account to temporary credentials, enabling attribute-based access control (ABAC) for EKS pods to AWS resources based on matching tags.

EKS Pod Identity attaches tags to the temporary credentials to each pod with attributes such as cluster name, namespace, service account name. These role session tags enable administrators to author a single role that can work across service accounts by allowing access to AWS resources based on matching tags. By adding support for role session tags, customers can enforce tighter security boundaries between clusters, and workloads within clusters, while reusing the same IAM roles and IAM policies.

For example, the following policy allows the s3:GetObject action if the object is tagged with the name of the EKS cluster.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectTagging"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "s3:ExistingObjectTag/eks-cluster-name": "${aws:PrincipalTag/eks-cluster-name}"
                }
            }
        }
    ]
}

List of session tags added by EKS Pod Identity

The following list contains all of the keys for tags that are added to the AssumeRole request made by Amazon EKS. To use these tags in policies, use ${aws:PrincipalTag/ followed by the key, for example ${aws:PrincipalTag/kubernetes-namespace}.

eks-cluster-arn
eks-cluster-name
kubernetes-namespace
kubernetes-service-account
kubernetes-pod-name
kubernetes-pod-uid

Cross-account tags

All of the session tags that are added by EKS Pod Identity are transitive; the tag keys and values are passed to any AssumeRole actions that your workloads use to switch roles into another account. You can use these tags in policies in other accounts to limit access in cross-account scenarios. For more infromation, see Chaining roles with session tags in the IAM User Guide.

Custom tags

EKS Pod Identity can’t add additional custom tags to the AssumeRole action that it performs. However, tags that you apply to the IAM role are always available though the same format: ${aws:PrincipalTag/ followed by the key, for example ${aws:PrincipalTag/MyCustomTag}.

Tags added to the session through the sts:AssumeRole request take precedence in the case of conflict. For example, say that:

Amazon EKS adds a key eks-cluster-name and value my-cluster to the session when EKS assumes the customer role and
You add an eks-cluster-name tag to the IAM role with the value my-own-cluster.

In this case, the former takes precedence and the value for the eks-cluster-name tag will be my-cluster.

Use pod identity with the `AWS` SDK

Using EKS Pod Identity credentials

To use the credentials from a EKS Pod Identity association, your code can use any AWS SDK to create a client for an AWS service with an SDK, and by default the SDK searches in a chain of locations for AWS Identity and Access Management credentials to use. The EKS Pod Identity credentials will be used if you don’t specify a credential provider when you create the client or otherwise initialized the SDK.

This works because EKS Pod Identities have been added to the Container credential provider which is searched in a step in the default credential chain. If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an EKS Pod Identity association for the same workload.

For more information about how EKS Pod Identities work, see pod-id-how-it-works.title.

When using Learn how EKS Pod Identity grants pods access to AWS services, the containers in your Pods must use an AWS SDK version that supports assuming an IAM role from the EKS Pod Identity Agent. Make sure that you’re using the following versions, or later, for your AWS SDK:

Java (Version 2) – 2.21.30
Java – 1.12.746
Go v1 – v1.47.11
Go v2 – release-2023-11-14
Python (Boto3) – 1.34.41
Python (botocore) – 1.34.41
AWS CLI – 1.30.0

AWS CLI – 2.15.0
JavaScript v2 – 2.1550.0
JavaScript v3 – v3.458.0
Kotlin – v1.0.1
Ruby – 3.188.0
Rust – release-2024-03-13
C++ – 1.11.263
.NET – 3.7.734.0
PowerShell – https://www.powershellgallery.com/packages/AWS.Tools.Common/4.1.502[4.1.502]
PHP – 3.287.1

To ensure that you’re using a supported SDK, follow the installation instructions for your preferred SDK at Tools to Build on AWS when you build your containers.

For a list of add-ons that support EKS Pod Identity, see pod-id-add-on-versions.title.

Disable `IPv6` in the EKS Pod Identity Agent

`consolelong`

To disable IPv6 in the EKS Pod Identity Agent, add the following configuration to the Optional configuration settings of the EKS Add-on.
1. Open the Amazon EKS console.
2. In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the add-on for.
3. Choose the Add-ons tab.
4. Select the box in the top right of the EKS Pod Identity Agent add-on box and then choose Edit.
5. On the Configure EKS Pod Identity Agent page:
  1. Select the Version that you’d like to use. We recommend that you keep the same version as the previous step, and update the version and configuration in separate actions.
  2. Expand the Optional configuration settings.
  3. Enter the JSON key "agent": and value of a nested JSON object with a key "additionalArgs": in Configuration values. The resulting text must be a valid JSON object. If this key and value are the only data in the text box, surround the key and value with curly braces { }. The following example shows network policy is enabled:
    
    { "agent": { "additionalArgs": { "-b": "169.254.170.23" } } }
    
    This configuration sets the IPv4 address to be the only address used by the agent.
6. To apply the new configuration by replacing the EKS Pod Identity Agent pods, choose Save changes.
  
  Amazon EKS applies changes to the EKS Add-ons by using a rollout of the Kubernetes DaemonSet for EKS Pod Identity Agent. You can track the status of the rollout in the Update history of the add-on in the consolelong and with kubectl rollout status daemonset/eks-pod-identity-agent --namespace kube-system.
  
  kubectl rollout has the following commands:
  $ kubectl rollout history -- View rollout history pause -- Mark the provided resource as paused restart -- Restart a resource resume -- Resume a paused resource status -- Show the status of the rollout undo -- Undo a previous rollout
  If the rollout takes too long, Amazon EKS will undo the rollout, and a message with the type of Addon Update and a status of Failed will be added to the Update history of the add-on. To investigate any issues, start from the history of the rollout, and run kubectl logs on a EKS Pod Identity Agent pod to see the logs of EKS Pod Identity Agent.
If the new entry in the Update history has a status of Successful, then the rollout has completed and the add-on is using the new configuration in all of the EKS Pod Identity Agent pods.

`AWS` CLI

To disable IPv6 in the EKS Pod Identity Agent, add the following configuration to the configuration values of the EKS Add-on.

Run the following AWS CLI command. Replace my-cluster with the name of your cluster and the IAM role ARN with the role that you are using.
```
aws eks update-addon --cluster-name my-cluster --addon-name eks-pod-identity-agent \
    --resolve-conflicts PRESERVE --configuration-values '{"agent":{"additionalArgs": { "-b": "169.254.170.23"}}}'
```
This configuration sets the IPv4 address to be the only address used by the agent.

Amazon EKS applies changes to the EKS Add-ons by using a rollout of the Kubernetes DaemonSet for EKS Pod Identity Agent. You can track the status of the rollout in the Update history of the add-on in the consolelong and with kubectl rollout status daemonset/eks-pod-identity-agent --namespace kube-system.

kubectl rollout has the following commands:
```
kubectl rollout

history  -- View rollout history
pause    -- Mark the provided resource as paused
restart  -- Restart a resource
resume   -- Resume a paused resource
status   -- Show the status of the rollout
undo     -- Undo a previous rollout
```
If the rollout takes too long, Amazon EKS will undo the rollout, and a message with the type of Addon Update and a status of Failed will be added to the Update history of the add-on. To investigate any issues, start from the history of the rollout, and run kubectl logs on a EKS Pod Identity Agent pod to see the logs of EKS Pod Identity Agent.

Create `IAM` role with trust policy required by `EKS Pod Identity`

Learn how to configure the IAM trust policy for Amazon EKS Pod Identity to allow Kubernetes pods to assume IAM roles and access AWS resources securely using Amazon EKS condition keys.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

sts:AssumeRole: EKS Pod Identity uses AssumeRole to assume the IAM role before passing the temporary credentials to your pods.
sts:TagSession: EKS Pod Identity uses TagSession to include session tags in the requests to AWS STS.

You can use these tags in the condition keys in the trust policy to restrict which service accounts, namespaces, and clusters can use this role.

For a list of Amazon EKS condition keys, see Conditions defined by Amazon Elastic Kubernetes Service in the Service Authorization Reference. To learn which actions and resources you can use a condition key with, see Actions defined by Amazon Elastic Kubernetes Service.

8.7.5. IAM roles for service accounts

Learn how applications in your Pods can access AWS services.

Applications in a Pod’s containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials. IAM roles for service accounts provide the ability to manage credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance’s role, you associate an IAM role with a Kubernetes service account and configure your Pods to use the service account. You can’t use IAM roles for service accounts with local clusters for Amazon EKS on AWS Outposts.

IAM roles for service accounts provide the following benefits:

Least privilege – You can scope IAM permissions to a service account, and only Pods that use that service account have access to those permissions. This feature also eliminates the need for third-party solutions such as kiam or kube2iam.
Credential isolation – A Pod’s containers can only retrieve credentials for the IAM role that’s associated with the service account that the container uses. A container never has access to credentials that are used by other containers in other Pods. When using IAM roles for service accounts, the Pod’s containers also have the permissions assigned to the Amazon EKS node IAM role, unless you block Pod access to the Amazon EC2 Instance Metadata Service (IMDS). For more information, see Restrict access to the instance profile assigned to the worker node.
Auditability – Access and event logging is available through AWS CloudTrail to help ensure retrospective auditing.

Enable IAM roles for service accounts by completing the following procedures:

Create an IAM OIDC provider for your cluster – You only complete this procedure once for each cluster.

If you enabled the EKS VPC endpoint, the EKS OIDC service endpoint couldn’t be accessed from inside that VPC. Consequently, your operations such as creating an OIDC provider with eksctl in the VPC will not work and will result in a timeout when attempting to request https://oidc.eks.region.amazonaws.com. An example error message follows:

server cant find oidc.eks.region.amazonaws.com: NXDOMAIN

To complete this step, you can run the command outside the VPC, for example in AWS CloudShell or on a computer connected to the internet. Alternatively, you can create a split-horizon conditional resolver in the VPC, such as Route 53 Resolver to use a different resolver for the OIDC Issuer URL and not use the VPC DNS for it. For an example of conditional forwarding in CoreDNS, see the Amazon EKS feature request on GitHub.

Assign IAM roles to Kubernetes service accounts – Complete this procedure for each unique set of permissions that you want an application to have.
Configure Pods to use a Kubernetes service account – Complete this procedure for each Pod that needs access to AWS services.
Use IRSA with the AWS SDK – Confirm that the workload uses an AWS SDK of a supported version and that the workload uses the default credential chain.

IAM, `Kubernetes`, and `OpenID Connect` (`OIDC`) background information

In 2014, AWS Identity and Access Management added support for federated identities using OpenID Connect (OIDC). This feature allows you to authenticate AWS API calls with supported identity providers and receive a valid OIDC JSON web token (JWT). You can pass this token to the AWS STS AssumeRoleWithWebIdentity API operation and receive IAM temporary role credentials. You can use these credentials to interact with any AWS service, including Amazon S3 and DynamoDB.

Each JWT token is signed by a signing key pair. The keys are served on the OIDC provider managed by Amazon EKS and the private key rotates every 7 days. Amazon EKS keeps the public keys until they expire. If you connect external OIDC clients, be aware that you need to refresh the signing keys before the public key expires. Learn how to Fetch signing keys to validate OIDC tokens.

Kubernetes has long used service accounts as its own internal identity system. Pods can authenticate with the Kubernetes API server using an auto-mounted token (which was a non-OIDC JWT) that only the Kubernetes API server could validate. These legacy service account tokens don’t expire, and rotating the signing key is a difficult process. In Kubernetes version 1.12, support was added for a new ProjectedServiceAccountToken feature. This feature is an OIDC JSON web token that also contains the service account identity and supports a configurable audience.

Amazon EKS hosts a public OIDC discovery endpoint for each cluster that contains the signing keys for the ProjectedServiceAccountToken JSON web tokens so external systems, such as IAM, can validate and accept the OIDC tokens that are issued by Kubernetes.

Create an IAM `OIDC` provider for your cluster

Learn how to create an AWS Identity and Access Management OpenID Connect provider for your cluster.

Your cluster has an OpenID Connect (OIDC) issuer URL associated with it. To use AWS Identity and Access Management (IAM) roles for service accounts, an IAM OIDC provider must exist for your cluster’s OIDC issuer URL.

An existing Amazon EKS cluster. To deploy one, see getting-started.title.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.
An existing kubectl config file that contains your cluster configuration. To create a kubectl config file, see create-kubeconfig.title.

You can create an IAM OIDC provider for your cluster using eksctl or the consolelong.

Create OIDC provider (eksctl)

Version 0.199.0 or later of the eksctl command line tool installed on your device or AWS CloudShell. To install or update eksctl, see Installation in the eksctl documentation.
Determine the OIDC issuer ID for your cluster.

Retrieve your cluster’s OIDC issuer ID and store it in a variable. Replace my-cluster with your own value.
```
cluster_name=my-cluster
```

oidc_id=$(aws eks describe-cluster --name $cluster_name --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

echo $oidc_id

Determine whether an IAM OIDC provider with your cluster’s issuer ID is already in your account.
```
aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4
```
If output is returned, then you already have an IAM OIDC provider for your cluster and you can skip the next step. If no output is returned, then you must create an IAM OIDC provider for your cluster.

Create an IAM OIDC identity provider for your cluster with the following command.

eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve

** server cant find oidc.eks.region.amazonaws.com: NXDOMAIN

Create OIDC provider (`AWS` Console)

Open the Amazon EKS console.
In the left pane, select Clusters, and then select the name of your cluster on the Clusters page.
In the Details section on the Overview tab, note the value of the OpenID Connect provider URL.
Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Identity Providers under Access management. If a Provider is listed that matches the URL for your cluster, then you already have a provider for your cluster. If a provider isn’t listed that matches the URL for your cluster, then you must create one.
To create a provider, choose Add provider.
For Provider type, select OpenID Connect.
For Provider URL, enter the OIDC provider URL for your cluster.
For Audience, enter sts.amazonaws.com.
(Optional) Add any tags, for example a tag to identify which cluster is for this provider.
Choose Add provider.

Next step: associate-service-account-role.title

Assign `IAM` roles to `Kubernetes` service accounts

Discover how to configure a Kubernetes service account to assume an IAM role, enabling Pods to securely access AWS services with granular permissions.

This topic covers how to configure a Kubernetes service account to assume an AWS Identity and Access Management (IAM) role. Any Pods that are configured to use the service account can then access any AWS service that the role has permissions to access.

Prerequisites

An existing cluster. If you don’t have one, you can create one by following one of the guides in getting-started.title.
An existing IAM OpenID Connect (OIDC) provider for your cluster. To learn if you already have one or how to create one, see enable-iam-roles-for-service-accounts.title.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.
An existing kubectl config file that contains your cluster configuration. To create a kubectl config file, see create-kubeconfig.title.

Step 1: Create IAM Policy

If you want to associate an existing IAM policy to your IAM role, skip to the next step.

Create an IAM policy. You can create your own policy, or copy an AWS managed policy that already grants some of the permissions that you need and customize it to your specific requirements. For more information, see Creating IAM policies in the IAM User Guide.
Create a file that includes the permissions for the AWS services that you want your Pods to access. For a list of all actions for all AWS services, see the Service Authorization Reference.

You can run the following command to create an example policy file that allows read-only access to an Amazon S3 bucket. You can optionally store configuration information or a bootstrap script in this bucket, and the containers in your Pod can read the file from the bucket and load it into your application. If you want to create this example policy, copy the following contents to your device. Replace my-pod-secrets-bucket with your bucket name and run the command.
```
cat >my-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "region.arns3:::my-pod-secrets-bucket"
        }
    ]
}
EOF
```

Create the IAM policy.

aws iam create-policy --policy-name my-policy --policy-document file://my-policy.json

Step 2: Create and associate IAM Role

Create an IAM role and associate it with a Kubernetes service account. You can use either eksctl or the AWS CLI.

Create and associate role (eksctl)

Version 0.199.0 or later of the eksctl command line tool installed on your device or AWS CloudShell. To install or update eksctl, see Installation in the eksctl documentation.

Replace my-service-account with the name of the Kubernetes service account that you want eksctl to create and associate with an IAM role. Replace default with the namespace that you want eksctl to create the service account in. Replace my-cluster with the name of your cluster. Replace my-role with the name of the role that you want to associate the service account to. If it doesn’t already exist, eksctl creates it for you. Replace 111122223333 with your account ID and my-policy with the name of an existing policy.

eksctl create iamserviceaccount --name my-service-account --namespace default --cluster my-cluster --role-name my-role \
    --attach-policy-arn region.arniam::111122223333:policy/my-policy --approve

If the role or service account already exist, the previous command might fail. eksctl has different options that you can provide in those situations. For more information run eksctl create iamserviceaccount --help.

Create and associate role (AWS CLI)

If you have an existing Kubernetes service account that you want to assume an IAM role, then you can skip this step.

Create a Kubernetes service account. Copy the following contents to your device. Replace my-service-account with your desired name and default with a different namespace, if necessary. If you change default, the namespace must already exist.
```
cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default
EOF
kubectl apply -f my-service-account.yaml
```
Set your AWS account ID to an environment variable with the following command.
```
account_id=$(aws sts get-caller-identity --query "Account" --output text)
```

Set your cluster’s OIDC identity provider to an environment variable with the following command. Replace my-cluster with the name of your cluster.

oidc_provider=$(aws eks describe-cluster --name my-cluster --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

Set variables for the namespace and name of the service account. Replace my-service-account with the Kubernetes service account that you want to assume the role. Replace default with the namespace of the service account.
```
export namespace=default
export service_account=my-service-account
```
Run the following command to create a trust policy file for the IAM role. If you want to allow all service accounts within a namespace to use the role, then copy the following contents to your device. Replace StringEquals with StringLike and replace $service_account with *. You can add multiple entries in the StringEquals or StringLike conditions to allow multiple service accounts or namespaces to assume the role. To allow roles from a different AWS account than the account that your cluster is in to assume the role, see cross-account-access.title for more information.
```
cat >trust-relationship.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "region.arniam::$account_id:oidc-provider/$oidc_provider"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$oidc_provider:aud": "sts.amazonaws.com",
          "$oidc_provider:sub": "system:serviceaccount:$namespace:$service_account"
        }
      }
    }
  ]
}
EOF
```

Create the role. Replace my-role with a name for your IAM role, and my-role-description with a description for your role.

aws iam create-role --role-name my-role --assume-role-policy-document file://trust-relationship.json --description "my-role-description"

Attach an IAM policy to your role. Replace my-role with the name of your IAM role and my-policy with the name of an existing policy that you created.
```
aws iam attach-role-policy --role-name my-role --policy-arn=region.arniam::$account_id:policy/my-policy
```
Annotate your service account with the Amazon Resource Name (ARN) of the IAM role that you want the service account to assume. Replace my-role with the name of your existing IAM role. Suppose that you allowed a role from a different AWS account than the account that your cluster is in to assume the role in a previous step. Then, make sure to specify the AWS account and role from the other account. For more information, see cross-account-access.title.
```
kubectl annotate serviceaccount -n $namespace $service_account eks.amazonaws.com/role-arn=region.arniam::$account_id:role/my-role
```
(Optional) Configure the AWS Security Token Service endpoint for a service account. AWS recommends using a regional AWS STS endpoint instead of the global endpoint. This reduces latency, provides built-in redundancy, and increases session token validity.

Step 3: Confirm configuration

Confirm that the IAM role’s trust policy is configured correctly.

aws iam get-role --role-name my-role --query Role.AssumeRolePolicyDocument

An example output is as follows.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "region.arniam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:default:my-service-account",
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Confirm that the policy that you attached to your role in a previous step is attached to the role.

aws iam list-attached-role-policies --role-name my-role --query AttachedPolicies[].PolicyArn --output text

An example output is as follows.

region.arniam::111122223333:policy/my-policy

Set a variable to store the Amazon Resource Name (ARN) of the policy that you want to use. Replace my-policy with the name of the policy that you want to confirm permissions for.
```
export policy_arn=region.arniam::111122223333:policy/my-policy
```

View the default version of the policy.

aws iam get-policy --policy-arn $policy_arn

An example output is as follows.

{
    "Policy": {
        "PolicyName": "my-policy",
        "PolicyId": "EXAMPLEBIOWGLDEXAMPLE",
        "Arn": "region.arniam::111122223333:policy/my-policy",
        "Path": "/",
        "DefaultVersionId": "v1",
        [...]
    }
}

View the policy contents to make sure that the policy includes all the permissions that your Pod needs. If necessary, replace 1 in the following command with the version that’s returned in the previous output.
```
aws iam get-policy-version --policy-arn $policy_arn --version-id v1
```
An example output is as follows.
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "region.arns3:::my-pod-secrets-bucket"
        }
    ]
}
```
If you created the example policy in a previous step, then your output is the same. If you created a different policy, then the example content is different.

Confirm that the Kubernetes service account is annotated with the role.

kubectl describe serviceaccount my-service-account -n default

An example output is as follows.

Name:                my-service-account
Namespace:           default
Annotations:         eks.amazonaws.com/role-arn: region.arniam::111122223333:role/my-role
Image pull secrets:  <none>
Mountable secrets:   my-service-account-token-qqjfl
Tokens:              my-service-account-token-qqjfl
[...]

Next steps

pod-configuration.title

Configure `Pods` to use a `Kubernetes` service account

Learn how to configure your Pods to use a Kubernetes service account that you allowed to assume an AWS Identity and Access Management role.

An existing cluster. If you don’t have one, you can create one using one of the guides in getting-started.title.
An existing IAM OpenID Connect (OIDC) provider for your cluster. To learn if you already have one or how to create one, see enable-iam-roles-for-service-accounts.title.
An existing Kubernetes service account that’s associated with an IAM role. The service account must be annotated with the Amazon Resource Name (ARN) of the IAM role. The role must have an associated IAM policy that contains the permissions that you want your Pods to have to use AWS services. For more information about how to create the service account and role, and configure them, see associate-service-account-role.title.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.

An existing kubectl config file that contains your cluster configuration. To create a kubectl config file, see create-kubeconfig.title.

Use the following command to create a deployment manifest that you can deploy a Pod to confirm configuration with. Replace the example values with your own values.

cat >my-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      serviceAccountName: my-service-account
      containers:
      - name: my-app
        image: public.ecr.aws/nginx/nginx:X.XX
EOF

Deploy the manifest to your cluster.
```
kubectl apply -f my-deployment.yaml
```
Confirm that the required environment variables exist for your Pod.
1. View the Pods that were deployed with the deployment in the previous step.
  kubectl get pods | grep my-app
  An example output is as follows.
  my-app-6f4dfff6cb-76cv9 1/1 Running 0 3m28s
2. View the ARN of the IAM role that the Pod is using.
  kubectl describe pod my-app-6f4dfff6cb-76cv9 | grep AWS_ROLE_ARN:
  An example output is as follows.
  AWS_ROLE_ARN: region.arniam::111122223333:role/my-role
  The role ARN must match the role ARN that you annotated the existing service account with. For more about annotating the service account, see associate-service-account-role.title.
3. Confirm that the Pod has a web identity token file mount.
  kubectl describe pod my-app-6f4dfff6cb-76cv9 | grep AWS_WEB_IDENTITY_TOKEN_FILE:
  An example output is as follows.
  AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
  The kubelet requests and stores the token on behalf of the Pod. By default, the kubelet refreshes the token if the token is older than 80 percent of its total time to live or older than 24 hours. You can modify the expiration duration for any account other than the default service account by using the settings in your Pod spec. For more information, see Service Account Token Volume Projection in the Kubernetes documentation.
  
  The Amazon EKS Pod Identity Webhook on the cluster watches for Pods that use a service account with the following annotation:
  eks.amazonaws.com/role-arn: region.arniam::111122223333:role/my-role
  The webhook applies the previous environment variables to those Pods. Your cluster doesn’t need to use the webhook to configure the environment variables and token file mounts. You can manually configure Pods to have these environment variables. The supported versions of the AWS SDK look for these environment variables first in the credential chain provider. The role credentials are used for Pods that meet this criteria.

Confirm that your Pods can interact with the AWS services using the permissions that you assigned in the IAM policy attached to your role.

If your Pods can’t interact with the services as you expected, complete the following steps to confirm that everything is properly configured.

Confirm that your Pods use an AWS SDK version that supports assuming an IAM role through an OpenID Connect web identity token file. For more information, see iam-roles-for-service-accounts-minimum-sdk.title.

Confirm that the deployment is using the service account.

kubectl describe deployment my-app | grep "Service Account"

An example output is as follows.

Service Account:  my-service-account

If your Pods still can’t access services, review the steps that are described in Assign IAM roles to Kubernetes service accounts to confirm that your role and service account are configured properly.

Configure the `AWS` Security Token Service endpoint for a service account

If you’re using a Kubernetes service account with IAM roles for service accounts, then you can configure the type of AWS Security Token Service endpoint that’s used by the service account if your cluster and platform version are the same or later than those listed in the following table. If your Kubernetes or platform version are earlier than those listed in the table, then your service accounts can only use the global endpoint.

Kubernetes version Platform version Default endpoint type

1.31

eks.4

Regional

1.30

eks.2

Regional

1.29

eks.1

Regional

1.28

eks.1

Regional

1.27

eks.1

Regional

1.26

eks.1

Regional

1.25

eks.1

Regional

1.24

eks.2

Regional

1.23

eks.1

Regional

AWS recommends using the regional AWS STS endpoints instead of the global endpoint. This reduces latency, provides built-in redundancy, and increases session token validity. The AWS Security Token Service must be active in the AWS Region where the Pod is running. Moreover, your application must have built-in redundancy for a different AWS Region in the event of a failure of the service in the AWS Region. For more information, see Managing AWS STS in an AWS Region in the IAM User Guide.

An existing cluster. If you don’t have one, you can create one using one of the guides in getting-started.title.
An existing IAM OIDC provider for your cluster. For more information, see enable-iam-roles-for-service-accounts.title.
An existing Kubernetes service account configured for use with the Amazon EKS IAM for service accounts feature.

The following examples all use the aws-node Kubernetes service account used by the Amazon VPC CNI plugin. You can replace the example values with your own service accounts, Pods, namespaces, and other resources.

Select a Pod that uses a service account that you want to change the endpoint for. Determine which AWS Region that the Pod runs in. Replace aws-node-6mfgv with your Pod name and kube-system with your Pod’s namespace.
```
kubectl describe pod aws-node-6mfgv -n kube-system |grep Node:
```
An example output is as follows.
```
ip-192-168-79-166.us-west-2/192.168.79.166
```
In the previous output, the Pod is running on a node in the us-west-2 AWS Region.
Determine the endpoint type that the Pod’s service account is using.
```
kubectl describe pod aws-node-6mfgv -n kube-system |grep AWS_STS_REGIONAL_ENDPOINTS
```
An example output is as follows.
```
AWS_STS_REGIONAL_ENDPOINTS: regional
```
If the current endpoint is global, then global is returned in the output. If no output is returned, then the default endpoint type is in use and has not been overridden.
If your cluster or platform version are the same or later than those listed in the table, then you can change the endpoint type used by your service account from the default type to a different type with one of the following commands. Replace aws-node with the name of your service account and kube-system with the namespace for your service account.
- If your default or current endpoint type is global and you want to change it to regional:
  kubectl annotate serviceaccount -n kube-system aws-node eks.amazonaws.com/sts-regional-endpoints=true
  If you’re using IAM roles for service accounts to generate pre-signed S3 URLs in your application running in Pods' containers, the format of the URL for regional endpoints is similar to the following example:
  https://bucket.s3.us-west-2.amazonaws.com/path?...&X-Amz-Credential=your-access-key-id/date/us-west-2/s3/aws4_request&...
- If your default or current endpoint type is regional and you want to change it to global:
  kubectl annotate serviceaccount -n kube-system aws-node eks.amazonaws.com/sts-regional-endpoints=false
  If your application is explicitly making requests to AWS STS global endpoints and you don’t override the default behavior of using regional endpoints in Amazon EKS clusters, then requests will fail with an error. For more information, see security-iam-troubleshoot-wrong-sts-endpoint.title.
  
  If you’re using IAM roles for service accounts to generate pre-signed S3 URLs in your application running in Pods' containers, the format of the URL for global endpoints is similar to the following example:
  https://bucket.s3.amazonaws.com/path?...&X-Amz-Credential=your-access-key-id/date/us-west-2/s3/aws4_request&...
If you have automation that expects the pre-signed URL in a certain format or if your application or downstream dependencies that use pre-signed URLs have expectations for the AWS Region targeted, then make the necessary changes to use the appropriate AWS STS endpoint.
Delete and re-create any existing Pods that are associated with the service account to apply the credential environment variables. The mutating web hook doesn’t apply them to Pods that are already running. You can replace Pods, kube-system, and -l k8s-app=aws-node with the information for the Pods that you set your annotation for.
```
kubectl delete Pods -n kube-system -l k8s-app=aws-node
```

Confirm that the all Pods restarted.

kubectl get Pods -n kube-system -l k8s-app=aws-node

View the environment variables for one of the Pods. Verify that the AWS_STS_REGIONAL_ENDPOINTS value is what you set it to in a previous step.
```
kubectl describe pod aws-node-kzbtr -n kube-system |grep AWS_STS_REGIONAL_ENDPOINTS
```
An example output is as follows.
```
AWS_STS_REGIONAL_ENDPOINTS=regional
```

Authenticate to another account with IRSA

Learn how to configure cross-account IAM permissions for Amazon EKS clusters by creating an identity provider from another account’s cluster or using chained AssumeRole operations, enabling secure access to AWS resources across multiple accounts.

You can configure cross-account IAM permissions either by creating an identity provider from another account’s cluster or by using chained AssumeRole operations. In the following examples, Account A owns an Amazon EKS cluster that supports IAM roles for service accounts. Pods that are running on that cluster must assume IAM permissions from Account B.

Example 1. Create an identity provider from another account’s cluster

In this example, Account A provides Account B with the OpenID Connect (OIDC) issuer URL from their cluster. Account B follows the instructions in Create an IAM OIDC provider for your cluster and associate-service-account-role.title using the OIDC issuer URL from Account A’s cluster. Then, a cluster administrator annotates the service account in Account A’s cluster to use the role from Account B (444455556666).

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: region.arniam::444455556666:role/account-b-role

Example 2. Use chained AssumeRole operations

In this example, Account B creates an IAM policy with the permissions to give to Pods in Account A’s cluster. Account B (444455556666) attaches that policy to an IAM role with a trust relationship that allows AssumeRole permissions to Account A (111122223333).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "region.arniam::111122223333:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

Account A creates a role with a trust policy that gets credentials from the identity provider created with the cluster’s OIDC issuer address.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "region.arniam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity"
    }
  ]
}

Account A attaches a policy to that role with the following permissions to assume the role that Account B created.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "region.arniam::444455556666:role/account-b-role"
        }
    ]
}

The application code for Pods to assume Account B’s role uses two profiles: account_b_role and account_a_role. The account_b_role profile uses the account_a_role profile as its source. For the AWS CLI, the ~/.aws/config file is similar to the following.

[profile account_b_role]
source_profile = account_a_role
role_arn=region.arniam::444455556666:role/account-b-role

[profile account_a_role]
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token
role_arn=region.arniam::111122223333:role/account-a-role

To specify chained profiles for other AWS SDKs, consult the documentation for the SDK that you’re using. For more information, see Tools to Build on AWS.

Use IRSA with the `AWS` SDK

Using the credentials

To use the credentials from IAM roles for service accounts, your code can use any AWS SDK to create a client for an AWS service with an SDK, and by default the SDK searches in a chain of locations for AWS Identity and Access Management credentials to use. The IAM roles for service accounts credentials will be used if you don’t specify a credential provider when you create the client or otherwise initialized the SDK.

This works because IAM roles for service accounts have been added as a step in the default credential chain. If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an IAM roles for service accounts for the same workload.

The SDK automatically exchanges the service account OIDC token for temporary credentials from AWS Security Token Service by using the AssumeRoleWithWebIdentity action. Amazon EKS and this SDK action continue to rotate the temporary credentials by renewing them before they expire.

When using IAM roles for service accounts, the containers in your Pods must use an AWS SDK version that supports assuming an IAM role through an OpenID Connect web identity token file. Make sure that you’re using the following versions, or later, for your AWS SDK:

Java (Version 2) – 2.10.11
Java – 1.11.704
Go – 1.23.13
Python (Boto3) – 1.9.220
Python (botocore) – 1.12.200
AWS CLI – 1.16.232
Node – 2.525.0 and 3.27.0
Ruby – 3.58.0
C++ – 1.7.174
.NET – 3.3.659.1 – You must also include AWSSDK.SecurityToken.
PHP – 3.110.7

Many popular Kubernetes add-ons, such as the Cluster Autoscaler, the Route internet traffic with AWS Load Balancer Controller, and the Amazon VPC CNI plugin for Kubernetes support IAM roles for service accounts.

To ensure that you’re using a supported SDK, follow the installation instructions for your preferred SDK at Tools to Build on AWS when you build your containers.

Fetch signing keys to validate `OIDC` tokens

Discover how to fetch the OIDC public signing keys (JSON Web Key Set) required to validate the ProjectedServiceAccountToken for Amazon EKS clusters, enabling external systems to authenticate with IAM roles for Kubernetes service accounts.

Kubernetes issues a ProjectedServiceAccountToken to each Kubernetes Service Account. This token is an OIDC token, which is further a type of JSON web token (JWT). Amazon EKS hosts a public OIDC endpoint for each cluster that contains the signing keys for the token so external systems can validate it.

To validate a ProjectedServiceAccountToken, you need to fetch the OIDC public signing keys, also called the JSON Web Key Set (JWKS). Use these keys in your application to validate the token. For example, you can use the PyJWT Python library to validate tokens using these keys. For more information on the ProjectedServiceAccountToken, see irsa-oidc-background.title.

Prerequisites

An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.
AWS CLI — A command line tool for working with AWS services, including Amazon EKS. For more information, see Installing in the AWS Command Line Interface User Guide. After installing the AWS CLI, we recommend that you also configure it. For more information, see Quick configuration with aws configure in the AWS Command Line Interface User Guide.

Procedure

Retrieve the OIDC URL for your Amazon EKS cluster using the AWS CLI.

$ aws eks describe-cluster --name my-cluster --query 'cluster.identity.oidc.issuer'
"https://oidc.eks.us-west-2.amazonaws.com/id/8EBDXXXX00BAE"

Retrieve the public signing key using curl, or a similar tool. The result is a JSON Web Key Set (JWKS).

Amazon EKS throttles calls to the OIDC endpoint. You should cache the public signing key. Respect the cache-control header included in the response.

Amazon EKS rotates the OIDC signing key every seven days.
```
$ curl https://oidc.eks.us-west-2.amazonaws.com/id/8EBDXXXX00BAE/keys
{"keys":[{"kty":"RSA","kid":"2284XXXX4a40","use":"sig","alg":"RS256","n":"wklbXXXXMVfQ","e":"AQAB"}]}
```

9. Manage compute resources by using nodes

Your Amazon EKS cluster can schedule Pods on any combination of self-managed nodes, Amazon EKS managed node groups, Fargate, and Amazon EKS Hybrid Nodes in the AWS Cloud and hybrid nodes on-premises.

A Kubernetes node is a machine that runs containerized applications. Each node has the following components:

Container runtime – Software that’s responsible for running the containers.
kubelet – Makes sure that containers are healthy and running within their associated Pod.
kube-proxy – Maintains network rules that allow communication to your Pods.

For more information, see Nodes in the Kubernetes documentation.

Your Amazon EKS cluster can schedule Pods on any combination of EKS Auto Mode managed nodes, self-managed nodes, Amazon EKS managed node groups, AWS Fargate, and Amazon EKS Hybrid Nodes. To learn more about nodes deployed in your cluster, see view-kubernetes-resources.title.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West). Amazon EKS Hybrid Nodes isn’t available in AWS GovCloud Regions and China Regions.

Excluding hybrid nodes, nodes must be in the same VPC as the subnets you selected when you created the cluster. However, the nodes don’t have to be in the same subnets.

9.1. Compare compute options

The following table provides several criteria to evaluate when deciding which options best meet your requirements. Self-managed nodes are another option which support all of the criteria listed, but they require a lot more manual maintenance. For more information, see worker.title.

Bottlerocket has some specific differences from the general information in this table. For more information, see the Bottlerocket documentation on GitHub.

Criteria EKS managed node groups EKS Auto Mode Amazon EKS Hybrid Nodes

Can be deployed to AWS Outposts

Can be deployed to an AWS Local Zone

Yes

Can run containers that require Windows

Yes

Can run containers that require Linux

Yes

Can run workloads that require the Inferentia chip

Yes – Amazon Linux nodes only

Yes

Can run workloads that require a GPU

Yes – Amazon Linux nodes only

Yes

Can run workloads that require Arm processors

Can run AWS Bottlerocket

Yes

Pods share CPU, memory, storage, and network resources with other Pods.

Yes

Must deploy and manage Amazon EC2 instances

Yes

No - Learn about EC2 managed instances

Yes – the on-premises physical or virtual machines are managed by you with your choice of tooling.

Must secure, maintain, and patch the operating system of Amazon EC2 instances

Yes

Yes – the operating system running on your physical or virtual machines are managed by you with your choice of tooling.

Can provide bootstrap arguments at deployment of a node, such as extra kubelet arguments.

Yes – Using eksctl or a launch template with a custom AMI.

No - Use a NodeClass to configure nodes

Yes - you can customize bootstrap arguments with nodeadm. See hybrid-nodes-nodeadm.title.

Can assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node.

Yes – Using a launch template with a custom AMI. For more information, see launch-templates.title.

Yes - see hybrid-nodes-cni.title.

Can SSH into node

Yes

No - Learn how to troubleshoot nodes

Yes

Can deploy your own custom AMI to nodes

Yes – Using a launch template

Yes

Can deploy your own custom CNI to nodes

Yes – Using a launch template with a custom AMI

Yes

Must update node AMI on your own

Yes – If you deployed an Amazon EKS optimized AMI, you’re notified in the Amazon EKS console when updates are available. You can perform the update with one-click in the console. If you deployed a custom AMI, you’re not notified in the Amazon EKS console when updates are available. You must perform the update on your own.

Yes - the operating system running on your physical or virtual machines is managed by you with your choice of tooling. See hybrid-nodes-os.title.

Must update node Kubernetes version on your own

Yes - you manage hybrid nodes upgrades with your own choice of tooling or with nodeadm. See hybrid-nodes-upgrade.title.

Can use Amazon EBS storage with Pods

Yes, as an integrated capability. Learn how to create a storage class.

Can use Amazon EFS storage with Pods

Can use Amazon FSx for Lustre storage with Pods

Can use Network Load Balancer for services

All Amazon EKS supported regions

Yes - must use target type ip.

Pods can run in a public subnet

Yes

No - pods run in on-premises environment.

Can assign different VPC security groups to individual Pods

Yes – Linux nodes only

Can run Kubernetes DaemonSets

Yes

Support HostPort and HostNetwork in the Pod manifest

Yes

AWS Region availability

All Amazon EKS supported regions except the AWS GovCloud (US) Regions and the China Regions.

Can run containers on Amazon EC2 dedicated hosts

Yes

Pricing

Cost of Amazon EC2 instance that runs multiple Pods. For more information, see Amazon EC2 pricing.

When EKS Auto Mode is enabled in your cluster, you pay a separate fee, in addition to the standard EC2 instance charges, for the instances launched using Auto Mode’s compute capability. The amount varies with the instance type launched and the AWS region where your cluster is located. For more information, see Amazon EKS pricing.

Cost of hybrid nodes vCPU per hour. For more information, see Amazon EKS pricing.

9.2. Simplify node lifecycle with managed node groups

9.2.1. Create a managed node group for your cluster

This topic describes how you can launch Amazon EKS managed node groups of nodes that register with your Amazon EKS cluster.

This topic describes how you can launch Amazon EKS managed node groups of nodes that register with your Amazon EKS cluster. After the nodes join the cluster, you can deploy Kubernetes applications to them.

If this is your first time launching an Amazon EKS managed node group, we recommend that you instead follow one of our guides in getting-started.title. These guides provide walkthroughs for creating an Amazon EKS cluster with nodes.

Amazon EKS nodes are standard Amazon EC2 instances. You’re billed based on the normal Amazon EC2 prices. For more information, see Amazon EC2 Pricing.
You can’t create managed nodes in an AWS Region where you have AWS Outposts or AWS Wavelength enabled. You can create self-managed nodes instead. For more information, see launch-workers.title, launch-windows-workers.title, and launch-node-bottlerocket.title. You can also create a self-managed Amazon Linux node group on an Outpost. For more information, see eks-outposts-self-managed-nodes.title.
If you don’t specify an AMI ID for the bootstrap.sh file included with Amazon EKS optimized Linux or Bottlerocket, managed node groups enforce a maximum number on the value of maxPods. For instances with less than 30 vCPUs, the maximum number is 110. For instances with greater than 30 vCPUs, the maximum number jumps to 250. These numbers are based on Kubernetes scalability thresholds and recommended settings by internal Amazon EKS scalability team testing. For more information, see the Amazon VPC CNI plugin increases pods per node limits blog post.

An existing Amazon EKS cluster. To deploy one, see create-cluster.title.
An existing IAM role for the nodes to use. To create one, see create-node-role.title. If this role doesn’t have either of the policies for the VPC CNI, the separate role that follows is required for the VPC CNI pods.
(Optional, but recommended) The Amazon VPC CNI plugin for Kubernetes add-on configured with its own IAM role that has the necessary IAM policy attached to it. For more information, see cni-iam-role.title.
Familiarity with the considerations listed in Choose an optimal Amazon EC2 node instance type. Depending on the instance type you choose, there may be additional prerequisites for your cluster and VPC.
To add a Windows managed node group, you must first enable Windows support for your cluster. For more information, see windows-support.title.

You can create a managed node group with either of the following:

eksctl
consolelong

`eksctl`

Create a managed node group with eksctl

This procedure requires eksctl version 0.199.0 or later. You can check your version with the following command:

eksctl version

For instructions on how to install or upgrade eksctl, see Installation in the eksctl documentation.

(Optional) If the AmazonEKS_CNI_Policy managed IAM policy is attached to your Amazon EKS node IAM role, we recommend assigning it to an IAM role that you associate to the Kubernetes aws-node service account instead. For more information, see cni-iam-role.title.

Create a managed node group with or without using a custom launch template. Manually specifying a launch template allows for greater customization of a node group. For example, it can allow deploying a custom AMI or providing arguments to the boostrap.sh script in an Amazon EKS optimized AMI. For a complete list of every available option and default, enter the following command.

eksctl create nodegroup --help

In the following command, replace my-cluster with the name of your cluster and replace my-mng with the name of your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.

If you don’t use a custom launch template when first creating a managed node group, don’t use one at a later time for the node group. If you didn’t specify a custom launch template, the system auto-generates a launch template that we don’t recommend that you modify manually. Manually modifying this auto-generated launch template might cause errors.

Without a launch template

eksctl creates a default Amazon EC2 launch template in your account and deploys the node group using a launch template that it creates based on options that you specify. Before specifying a value for --node-type, see choosing-instance-type.title.

Replace ami-family with an allowed keyword. For more information, see Setting the node AMI Family in the eksctl documentation. Replace my-key with the name of your Amazon EC2 key pair or public key. This key is used to SSH into your nodes after they launch.

For Windows, this command doesn’t enable SSH. Instead, it associates your Amazon EC2 key pair with the instance and allows you to RDP into the instance.

If you don’t already have an Amazon EC2 key pair, you can create one in the consolelong. For Linux information, see Amazon EC2 key pairs and Linux instances in the Amazon EC2 User Guide. For Windows information, see Amazon EC2 key pairs and Windows instances in the Amazon EC2 User Guide.

We recommend blocking Pod access to IMDS if the following conditions are true:

You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.

For more information, see Restrict access to the instance profile assigned to the worker node.

If you want to block Pod access to IMDS, then add the --disable-pod-imds option to the following command.

eksctl create nodegroup \
  --cluster my-cluster \
  --region region-code \
  --name my-mng \
  --node-ami-family ami-family \
  --node-type m5.large \
  --nodes 3 \
  --nodes-min 2 \
  --nodes-max 4 \
  --ssh-access \
  --ssh-public-key my-key

Your instances can optionally assign a significantly higher number of IP addresses to Pods, assign IP addresses to Pods from a different CIDR block than the instance’s, and be deployed to a cluster without internet access. For more information, see cni-increase-ip-addresses.title, cni-custom-network.title, and private-clusters.title for additional options to add to the previous command.

Managed node groups calculates and applies a single value for the maximum number of Pods that can run on each node of your node group, based on instance type. If you create a node group with different instance types, the smallest value calculated across all instance types is applied as the maximum number of Pods that can run on every instance type in the node group. Managed node groups calculates the value using the script referenced in Amazon EKS recommended maximum Pods for each Amazon EC2 instance type.

With a launch template

The launch template must already exist and must meet the requirements specified in Launch template configuration basics. We recommend blocking Pod access to IMDS if the following conditions are true:

You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.

For more information, see Restrict access to the instance profile assigned to the worker node.

If you want to block Pod access to IMDS, then specify the necessary settings in the launch template.

Copy the following contents to your device. Replace the example values and then run the modified command to create the eks-nodegroup.yaml file. Several settings that you specify when deploying without a launch template are moved into the launch template. If you don’t specify a version, the template’s default version is used.
```
cat >eks-nodegroup.yaml <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: region-code
managedNodeGroups:
- name: my-mng
  launchTemplate:
    id: lt-id
    version: "1"
EOF
```
For a complete list of eksctl config file settings, see Config file schema in the eksctl documentation. Your instances can optionally assign a significantly higher number of IP addresses to Pods, assign IP addresses to Pods from a different CIDR block than the instance’s, use the containerd runtime, and be deployed to a cluster without outbound internet access. For more information, see cni-increase-ip-addresses.title, cni-custom-network.title, containerd-bootstrap.title, and private-clusters.title for additional options to add to the config file.

If you didn’t specify an AMI ID in your launch template, managed node groups calculates and applies a single value for the maximum number of Pods that can run on each node of your node group, based on instance type. If you create a node group with different instance types, the smallest value calculated across all instance types is applied as the maximum number of Pods that can run on every instance type in the node group. Managed node groups calculates the value using the script referenced in Amazon EKS recommended maximum Pods for each Amazon EC2 instance type.

If you specified an AMI ID in your launch template, specify the maximum number of Pods that can run on each node of your node group if you’re using custom networking or want to increase the number of IP addresses assigned to your instance. For more information, see determine-max-pods.title.

Deploy the nodegroup with the following command.

eksctl create nodegroup --config-file eks-nodegroup.yaml

`consolelong`

Create a managed node group using the consolelong

Wait for your cluster status to show as ACTIVE. You can’t create a managed node group for a cluster that isn’t already ACTIVE.
Open the Amazon EKS console.
Choose the name of the cluster that you want to create a managed node group in.
Select the Compute tab.
Choose Add node group.

On the Configure node group page, fill out the parameters accordingly, and then choose Next.

Name – Enter a unique name for your managed node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
Node IAM role – Choose the node instance role to use with your node group. For more information, see create-node-role.title.

You can’t use the same role that is used to create any clusters.
We recommend using a role that’s not currently in use by any self-managed node group. Otherwise, you plan to use with a new self-managed node group. For more information, see delete-managed-node-group.title.

Use launch template – (Optional) Choose if you want to use an existing launch template. Select a Launch Template Name. Then, select a Launch template version. If you don’t select a version, then Amazon EKS uses the template’s default version. Launch templates allow for more customization of your node group, such as allowing you to deploy a custom AMI, assign a significantly higher number of IP addresses to Pods, assign IP addresses to Pods from a different CIDR block than the instance’s, enable the containerd runtime for your instances, and deploying nodes to a cluster without outbound internet access. For more information, see cni-increase-ip-addresses.title, cni-custom-network.title, containerd-bootstrap.title, and private-clusters.title.

The launch template must meet the requirements in Customize managed nodes with launch templates. If you don’t use your own launch template, the Amazon EKS API creates a default Amazon EC2 launch template in your account and deploys the node group using the default launch template.

If you implement IAM roles for service accounts, assign necessary permissions directly to every Pod that requires access to AWS services, and no Pods in your cluster require access to IMDS for other reasons, such as retrieving the current AWS Region, then you can also disable access to IMDS for Pods that don’t use host networking in a launch template. For more information, see Restrict access to the instance profile assigned to the worker node.
Kubernetes labels – (Optional) You can choose to apply Kubernetes labels to the nodes in your managed node group.
Kubernetes taints – (Optional) You can choose to apply Kubernetes taints to the nodes in your managed node group. The available options in the Effect menu are NoSchedule, NoExecute, and PreferNoSchedule. For more information, see node-taints-managed-node-groups.title.
Tags – (Optional) You can choose to tag your Amazon EKS managed node group. These tags don’t propagate to other resources in the node group, such as Auto Scaling groups or instances. For more information, see eks-using-tags.title.

On the Set compute and scaling configuration page, fill out the parameters accordingly, and then choose Next.

AMI type – Select an AMI type.If you are deploying Arm instances, be sure to review the considerations in Amazon EKS optimized Arm Amazon Linux AMIs before deploying.

If you specified a launch template on the previous page, and specified an AMI in the launch template, then you can’t select a value. The value from the template is displayed. The AMI specified in the template must meet the requirements in Specifying an AMI.
Capacity type – Select a capacity type. For more information about choosing a capacity type, see managed-node-group-capacity-types.title. You can’t mix different capacity types within the same node group. If you want to use both capacity types, create separate node groups, each with their own capacity and instance types. See capacity-blocks-mng.title for information on provisioning and scaling GPU-accelerated worker nodes.
Instance types – By default, one or more instance type is specified. To remove a default instance type, select the X on the right side of the instance type. Choose the instance types to use in your managed node group. For more information, see choosing-instance-type.title.

The console displays a set of commonly used instance types. If you need to create a managed node group with an instance type that’s not displayed, then use eksctl, the AWS CLI, AWS CloudFormation, or an SDK to create the node group. If you specified a launch template on the previous page, then you can’t select a value because the instance type must be specified in the launch template. The value from the launch template is displayed. If you selected Spot for Capacity type, then we recommend specifying multiple instance types to enhance availability.
Disk size – Enter the disk size (in GiB) to use for your node’s root volume.

If you specified a launch template on the previous page, then you can’t select a value because it must be specified in the launch template.

Desired size – Specify the current number of nodes that the managed node group should maintain at launch.

Amazon EKS doesn’t automatically scale your node group in or out. However, you can configure the Kubernetes Cluster Autoscaler to do this for you. For more information, see Cluster Autoscaler on AWS.

Minimum size – Specify the minimum number of nodes that the managed node group can scale in to.
Maximum size – Specify the maximum number of nodes that the managed node group can scale out to.
Node group update configuration – (Optional) You can select the number or percentage of nodes to be updated in parallel. These nodes will be unavailable during the update. For Maximum unavailable, select one of the following options and specify a Value:
- Number – Select and specify the number of nodes in your node group that can be updated in parallel.
- Percentage – Select and specify the percentage of nodes in your node group that can be updated in parallel. This is useful if you have a large number of nodes in your node group.
Node auto repair configuration – (Optional) If you activate the Enable node auto repair checkbox, Amazon EKS will automatically replace nodes when detected issues occur. For more information, see node-health.title.

On the Specify networking page, fill out the parameters accordingly, and then choose Next.

Subnets – Choose the subnets to launch your managed nodes into.

If you are running a stateful application across multiple Availability Zones that is backed by Amazon EBS volumes and using the Kubernetes Cluster Autoscaler, you should configure multiple node groups, each scoped to a single Availability Zone. In addition, you should enable the --balance-similar-node-groups feature.

If you choose a public subnet, and your cluster has only the public API server endpoint enabled, then the subnet must have MapPublicIPOnLaunch set to true for the instances to successfully join a cluster. If the subnet was created using eksctl or the Amazon EKS vended AWS CloudFormation templates on or after March 26, 2020, then this setting is already set to true. If the subnets were created with eksctl or the AWS CloudFormation templates before March 26, 2020, then you need to change the setting manually. For more information, see Modifying the public IPv4 addressing attribute for your subnet.
If you use a launch template and specify multiple network interfaces, Amazon EC2 won’t auto-assign a public IPv4 address, even if MapPublicIpOnLaunch is set to true. For nodes to join the cluster in this scenario, you must either enable the cluster’s private API server endpoint, or launch nodes in a private subnet with outbound internet access provided through an alternative method, such as a NAT Gateway. For more information, see Amazon EC2 instance IP addressing in the Amazon EC2 User Guide.

Configure SSH access to nodes (Optional). Enabling SSH allows you to connect to your instances and gather diagnostic information if there are issues. We highly recommend enabling remote access when you create a node group. You can’t enable remote access after the node group is created.

If you chose to use a launch template, then this option isn’t shown. To enable remote access to your nodes, specify a key pair in the launch template and ensure that the proper port is open to the nodes in the security groups that you specify in the launch template. For more information, see launch-template-security-groups.title.

For Windows, this command doesn’t enable SSH. Instead, it associates your Amazon EC2 key pair with the instance and allows you to RDP into the instance.
For SSH key pair (Optional), choose an Amazon EC2 SSH key to use. For Linux information, see Amazon EC2 key pairs and Linux instances in the Amazon EC2 User Guide. For Windows information, see Amazon EC2 key pairs and Windows instances in the Amazon EC2 User Guide. If you chose to use a launch template, then you can’t select one. When an Amazon EC2 SSH key is provided for node groups using Bottlerocket AMIs, the administrative container is also enabled. For more information, see Admin container on GitHub.
For Allow SSH remote access from, if you want to limit access to specific instances, then select the security groups that are associated to those instances. If you don’t select specific security groups, then SSH access is allowed from anywhere on the internet (0.0.0.0/0).

On the Review and create page, review your managed node group configuration and choose Create.

If nodes fail to join the cluster, then see worker-node-fail.title in the Troubleshooting chapter.
Watch the status of your nodes and wait for them to reach the Ready status.
```
kubectl get nodes --watch
```
(GPU nodes only) If you chose a GPU instance type and an Amazon EKS optimized accelerated AMI, then you must apply the NVIDIA device plugin for Kubernetes as a DaemonSet on your cluster. Replace vX.X.X with your desired NVIDIA/k8s-device-plugin version before running the following command.
```
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/vX.X.X/deployments/static/nvidia-device-plugin.yml
```

Install Kubernetes add-ons

Now that you have a working Amazon EKS cluster with nodes, you’re ready to start installing Kubernetes add-ons and deploying applications to your cluster. The following documentation topics help you to extend the functionality of your cluster.

The IAM principal that created the cluster is the only principal that can make calls to the Kubernetes API server with kubectl or the consolelong. If you want other IAM principals to have access to your cluster, then you need to add them. For more information, see grant-k8s-access.title and view-kubernetes-resources-permissions.title.
We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see Restrict access to the instance profile assigned to the worker node.
Configure the Kubernetes Cluster Autoscaler to automatically adjust the number of nodes in your node groups.
Deploy a sample application to your cluster.
Organize and monitor cluster resources with important tools for managing your cluster.

9.2.2. Update a managed node group for your cluster

When you initiate a managed node group update, Amazon EKS automatically updates your nodes for you.

When you initiate a managed node group update, Amazon EKS automatically updates your nodes for you, completing the steps listed in Understand each phase of node updates. If you’re using an Amazon EKS optimized AMI, Amazon EKS automatically applies the latest security patches and operating system updates to your nodes as part of the latest AMI release version.

There are several scenarios where it’s useful to update your Amazon EKS managed node group’s version or configuration:

You have updated the Kubernetes version for your Amazon EKS cluster and want to update your nodes to use the same Kubernetes version.
A new AMI release version is available for your managed node group. For more information about AMI versions, see these sections:
You want to adjust the minimum, maximum, or desired count of the instances in your managed node group.
You want to add or remove Kubernetes labels from the instances in your managed node group.
You want to add or remove AWS tags from your managed node group.
You need to deploy a new version of a launch template with configuration changes, such as an updated custom AMI.
You have deployed version 1.9.0 or later of the Amazon VPC CNI add-on, enabled the add-on for prefix delegation, and want new AWS Nitro System instances in a node group to support a significantly increased number of Pods. For more information, see cni-increase-ip-addresses.title.
You have enabled IP prefix delegation for Windows nodes and want new AWS Nitro System instances in a node group to support a significantly increased number of Pods. For more information, see cni-increase-ip-addresses.title.

If there’s a newer AMI release version for your managed node group’s Kubernetes version, you can update your node group’s version to use the newer AMI version. Similarly, if your cluster is running a Kubernetes version that’s newer than your node group, you can update the node group to use the latest AMI release version to match your cluster’s Kubernetes version.

When a node in a managed node group is terminated due to a scaling operation or update, the Pods in that node are drained first. For more information, see managed-node-update-behavior.title.

Update a node group version

You can update a node group version with either of the following:

eksctl
consolelong

The version that you update to can’t be greater than the control plane’s version.

`eksctl`

Update a managed node group using eksctl

Update a managed node group to the latest AMI release of the same Kubernetes version that’s currently deployed on the nodes with the following command. Replace every example value with your own values.

eksctl upgrade nodegroup \
  --name=node-group-name \
  --cluster=my-cluster \
  --region=region-code

If you’re upgrading a node group that’s deployed with a launch template to a new launch template version, add --launch-template-version version-number to the preceding command. The launch template must meet the requirements described in Customize managed nodes with launch templates. If the launch template includes a custom AMI, the AMI must meet the requirements in Specifying an AMI. When you upgrade your node group to a newer version of your launch template, every node is recycled to match the new configuration of the launch template version that’s specified.

You can’t directly upgrade a node group that’s deployed without a launch template to a new launch template version. Instead, you must deploy a new node group using the launch template to update the node group to a new launch template version.

You can upgrade a node group to the same version as the control plane’s Kubernetes version. For example, if you have a cluster running Kubernetes 1.29, you can upgrade nodes currently running Kubernetes 1.28 to version 1.29 with the following command.

eksctl upgrade nodegroup \
  --name=node-group-name \
  --cluster=my-cluster \
  --region=region-code \
  --kubernetes-version=1.29

`consolelong`

Update a managed node group using the consolelong

Open the Amazon EKS console.
Choose the cluster that contains the node group to update.
If at least one node group has an available update, a box appears at the top of the page notifying you of the available update. If you select the Compute tab, you’ll see Update now in the AMI release version column in the Node groups table for the node group that has an available update. To update the node group, choose Update now.

You won’t see a notification for node groups that were deployed with a custom AMI. If your nodes are deployed with a custom AMI, complete the following steps to deploy a new updated custom AMI.
1. Create a new version of your AMI.
2. Create a new launch template version with the new AMI ID.
3. Upgrade the nodes to the new version of the launch template.
On the Update node group version dialog box, activate or deactivate the following options:
- Update node group version – This option is unavailable if you deployed a custom AMI or your Amazon EKS optimized AMI is currently on the latest version for your cluster.
- Change launch template version – This option is unavailable if the node group is deployed without a custom launch template. You can only update the launch template version for a node group that has been deployed with a custom launch template. Select the Launch template version that you want to update the node group to. If your node group is configured with a custom AMI, then the version that you select must also specify an AMI. When you upgrade to a newer version of your launch template, every node is recycled to match the new configuration of the launch template version specified.
For Update strategy, select one of the following options:
- Rolling update – This option respects the Pod disruption budgets for your cluster. Updates fail if there’s a Pod disruption budget issue that causes Amazon EKS to be unable to gracefully drain the Pods that are running on this node group.
- Force update – This option doesn’t respect Pod disruption budgets. Updates occur regardless of Pod disruption budget issues by forcing node restarts to occur.
Choose Update.

Edit a node group configuration

You can modify some of the configurations of a managed node group.

Open the Amazon EKS console.
Choose the cluster that contains the node group to edit.
Select the Compute tab.
Select the node group to edit, and then choose Edit.
(Optional) On the Edit node group page, do the following:
1. Edit the Node group scaling configuration.
  - Desired size – Specify the current number of nodes that the managed node group should maintain.
  - Minimum size – Specify the minimum number of nodes that the managed node group can scale in to.
  - Maximum size – Specify the maximum number of nodes that the managed node group can scale out to. For the maximum number of nodes supported in a node group, see service-quotas.title.
2. (Optional) Add or remove Kubernetes labels to the nodes in your node group. The labels shown here are only the labels that you have applied with Amazon EKS. Other labels may exist on your nodes that aren’t shown here.
3. (Optional) Add or remove Kubernetes taints to the nodes in your node group. Added taints can have the effect of either NoSchedule, NoExecute, or PreferNoSchedule. For more information, see node-taints-managed-node-groups.title.
4. (Optional) Add or remove Tags from your node group resource. These tags are only applied to the Amazon EKS node group. They don’t propagate to other resources, such as subnets or Amazon EC2 instances in the node group.
5. (Optional) Edit the Node Group update configuration. Select either Number or Percentage.
  - Number – Select and specify the number of nodes in your node group that can be updated in parallel. These nodes will be unavailable during update.
  - Percentage – Select and specify the percentage of nodes in your node group that can be updated in parallel. These nodes will be unavailable during update. This is useful if you have many nodes in your node group.
6. When you’re finished editing, choose Save changes.

When updating the node group configuration, modifying the NodegroupScalingConfig does not respect Pod disruption budgets (PDBs). Unlike the update node group process (which drains nodes and respects PDBs during the upgrade phase), updating the scaling configuration causes nodes to be terminated immediately through an Auto Scaling Group (ASG) scale-down call. This happens without considering PDBs, regardless of the target size you’re scaling down to. That means when you reduce the desiredSize of an Amazon EKS managed node group, Pods are evicted as soon as the nodes are terminated, without honoring any PDBs.

9.2.3. Understand each phase of node updates

The Amazon EKS managed worker node upgrade strategy has four different phases.

The Amazon EKS managed worker node upgrade strategy has four different phases described in the following sections.

Setup phase

The setup phase has these steps:

It creates a new Amazon EC2 launch template version for the Auto Scaling group that’s associated with your node group. The new launch template version uses the target AMI or a custom launch template version for the update.
It updates the Auto Scaling group to use the latest launch template version.
It determines the maximum quantity of nodes to upgrade in parallel using the updateConfig property for the node group. The maximum unavailable has a quota of 100 nodes. The default value is one node. For more information, see the updateConfig property in the Amazon EKS API Reference.

Scale up phase

When upgrading the nodes in a managed node group, the upgraded nodes are launched in the same Availability Zone as those that are being upgraded. To guarantee this placement, we use Amazon EC2’s Availability Zone Rebalancing. For more information, see Availability Zone Rebalancing in the Amazon EC2 Auto Scaling User Guide. To meet this requirement, it’s possible that we’d launch up to two instances per Availability Zone in your managed node group.

The scale up phase has these steps:

It increments the Auto Scaling Group’s maximum size and desired size by the larger of either:
- Up to twice the number of Availability Zones that the Auto Scaling group is deployed in.
- The maximum unavailable of upgrade.
  
  For example, if your node group has five Availability Zones and maxUnavailable as one, the upgrade process can launch a maximum of 10 nodes. However when maxUnavailable is 20 (or anything higher than 10), the process would launch 20 new nodes.
After scaling the Auto Scaling group, it checks if the nodes using the latest configuration are present in the node group. This step succeeds only when it meets these criteria:
- At least one new node is launched in every Availability Zone where the node exists.
- Every new node should be in Ready state.
- New nodes should have Amazon EKS applied labels.
  
  These are the Amazon EKS applied labels on the worker nodes in a regular node group:
  - eks.amazonaws.com/nodegroup-image=$amiName
  - eks.amazonaws.com/nodegroup=$nodeGroupName
These are the Amazon EKS applied labels on the worker nodes in a custom launch template or AMI node group:

+
- eks.amazonaws.com/nodegroup-image=$amiName
- eks.amazonaws.com/nodegroup=$nodeGroupName
- eks.amazonaws.com/sourceLaunchTemplateId=$launchTemplateId
- eks.amazonaws.com/sourceLaunchTemplateVersion=$launchTemplateVersion
It marks nodes as unschedulable to avoid scheduling new Pods. It also labels nodes with node.kubernetes.io/exclude-from-external-load-balancers=true to remove the nodes from load balancers before terminating the nodes.

The following are known reasons which lead to a NodeCreationFailure error in this phase:

Insufficient capacity in the Availability Zone: There is a possibility that the Availability Zone might not have capacity of requested instance types. It’s recommended to configure multiple instance types while creating a managed node group.
EC2 instance limits in your account: You may need to increase the number of Amazon EC2 instances your account can run simultaneously using Service Quotas. For more information, see EC2 Service Quotas in the Amazon Elastic Compute Cloud User Guide for Linux Instances.
Custom user data: Custom user data can sometimes break the bootstrap process. This scenario can lead to the kubelet not starting on the node or nodes not getting expected Amazon EKS labels on them. For more information, see launch-template-custom-ami.title.
Any changes which make a node unhealthy or not ready: Node disk pressure, memory pressure, and similar conditions can lead to a node not going to Ready state.

Upgrade phase

The upgrade phase has these steps:

It randomly selects a node that needs to be upgraded, up to the maximum unavailable configured for the node group.
It drains the Pods from the node. If the Pods don’t leave the node within 15 minutes and there’s no force flag, the upgrade phase fails with a PodEvictionFailure error. For this scenario, you can apply the force flag with the update-nodegroup-version request to delete the Pods.
It cordons the node after every Pod is evicted and waits for 60 seconds. This is done so that the service controller doesn’t send any new requests to this node and removes this node from its list of active nodes.
It sends a termination request to the Auto Scaling Group for the cordoned node.
It repeats the previous upgrade steps until there are no nodes in the node group that are deployed with the earlier version of the launch template.

The following are known reasons which lead to a PodEvictionFailure error in this phase:

Aggressive PDB: Aggressive PDB is defined on the Pod or there are multiple PDBs pointing to the same Pod.
Deployment tolerating all the taints: Once every Pod is evicted, it’s expected for the node to be empty because the node is tainted in the earlier steps. However, if the deployment tolerates every taint, then the node is more likely to be non-empty, leading to Pod eviction failure.

Scale down phase

The scale down phase decrements the Auto Scaling group maximum size and desired size by one to return to values before the update started.

If the Upgrade workflow determines that the Cluster Autoscaler is scaling up the node group during the scale down phase of the workflow, it exits immediately without bringing the node group back to its original size.

9.2.4. Customize managed nodes with launch templates

For the highest level of customization, you can deploy managed nodes using your own launch template and a custom AMI.

For the highest level of customization, you can deploy managed nodes using your own launch template. Using a launch template allows capabilities such as the following:

Provide bootstrap arguments at deployment of a node, such as extra kubelet arguments.
Assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node.
Deploy your own custom AMI to nodes.
Deploy your own custom CNI to nodes.

When you give your own launch template upon first creating a managed node group, you will also have greater flexibility later. As long as you deploy a managed node group with your own launch template, you can iteratively update it with a different version of the same launch template. When you update your node group to a different version of your launch template, all nodes in the group are recycled to match the new configuration of the specified launch template version.

Managed node groups are always deployed with a launch template to be used with the Amazon EC2 Auto Scaling group. When you don’t provide a launch template, the Amazon EKS API creates one automatically with default values in your account. However, we don’t recommend that you modify auto-generated launch templates. Furthermore, existing node groups that don’t use a custom launch template can’t be updated directly. Instead, you must create a new node group with a custom launch template to do so.

Launch template configuration basics

You can create an Amazon EC2 Auto Scaling launch template with the consolelong, AWS CLI, or an AWS SDK. For more information, see Creating a Launch Template for an Auto Scaling group in the Amazon EC2 Auto Scaling User Guide. Some of the settings in a launch template are similar to the settings used for managed node configuration. When deploying or updating a node group with a launch template, some settings must be specified in either the node group configuration or the launch template. Don’t specify a setting in both places. If a setting exists where it shouldn’t, then operations such as creating or updating a node group fail.

The following table lists the settings that are prohibited in a launch template. It also lists similar settings, if any are available, that are required in the managed node group configuration. The listed settings are the settings that appear in the console. They might have similar but different names in the AWS CLI and SDK.

Launch template – Prohibited

Amazon EKS node group configuration

Subnet under Network interfaces (Add network interface)

Subnets under Node group network configuration on the Specify networking page

IAM instance profile under Advanced details

Node IAM role under Node group configuration on the Configure Node group page

Shutdown behavior and Stop - Hibernate behavior under Advanced details. Retain default Don’t include in launch template setting in launch template for both settings.

No equivalent. Amazon EKS must control the instance lifecycle, not the Auto Scaling group.

The following table lists the prohibited settings in a managed node group configuration. It also lists similar settings, if any are available, which are required in a launch template. The listed settings are the settings that appear in the console. They might have similar names in the AWS CLI and SDK.

Amazon EKS node group configuration – Prohibited Launch template

(Only if you specified a custom AMI in a launch template) AMI type under Node group compute configuration on Set compute and scaling configuration page – Console displays Specified in launch template and the AMI ID that was specified.

If Application and OS Images (Amazon Machine Image) wasn’t specified in the launch template, you can select an AMI in the node group configuration.

Application and OS Images (Amazon Machine Image) under Launch template contents – You must specify an ID if you have either of the following requirements:

* Using a custom AMI. If you specify an AMI that doesn’t meet the requirements listed in Specifying an AMI, the node group deployment will fail. * Want to provide user data to provide arguments to the bootstrap.sh file included with an Amazon EKS optimized AMI. You can enable your instances to assign a significantly higher number of IP addresses to Pods, assign IP addresses to Pods from a different CIDR block than the instance’s, or deploy a private cluster without outbound internet access. For more information, see the following topics: + Assign more IP addresses to Amazon EKS nodes with prefixes Deploy pods in alternate subnets with custom networking Deploy private clusters with limited internet access Specifying an AMI

Disk size under Node group compute configuration on Set compute and scaling configuration page – Console displays Specified in launch template.

Size under Storage (Volumes) (Add new volume). You must specify this in the launch template.

SSH key pair under Node group configuration on the Specify Networking page – The console displays the key that was specified in the launch template or displays Not specified in launch template.

Key pair name under Key pair (login).

You can’t specify source security groups that are allowed remote access when using a launch template.

Security groups under Network settings for the instance or Security groups under Network interfaces (Add network interface), but not both. For more information, see launch-template-security-groups.title.

If you deploy a node group using a launch template, specify zero or one Instance type under Launch template contents in a launch template. Alternatively, you can specify 0–20 instance types for Instance types on the Set compute and scaling configuration page in the console. Or, you can do so using other tools that use the Amazon EKS API. If you specify an instance type in a launch template, and use that launch template to deploy your node group, then you can’t specify any instance types in the console or using other tools that use the Amazon EKS API. If you don’t specify an instance type in a launch template, in the console, or using other tools that use the Amazon EKS API, the t3.medium instance type is used. If your node group is using the Spot capacity type, then we recommend specifying multiple instance types using the console. For more information, see managed-node-group-capacity-types.title.
If any containers that you deploy to the node group use the Instance Metadata Service Version 2, make sure to set the Metadata response hop limit to 2 in your launch template. For more information, see Instance metadata and user data in the Amazon EC2 User Guide. If you deploy a managed node group without using a custom launch template, this value is automatically set for the node group in the default launch template.

Tagging Amazon EC2 instances

You can use the TagSpecification parameter of a launch template to specify which tags to apply to Amazon EC2 instances in your node group. The IAM entity calling the CreateNodegroup or UpdateNodegroupVersion APIs must have permissions for ec2:RunInstances and ec2:CreateTags, and the tags must be added to the launch template.

Using custom security groups

You can use a launch template to specify custom Amazon EC2 security groups to apply to instances in your node group. This can be either in the instance level security groups parameter or as part of the network interface configuration parameters. However, you can’t create a launch template that specifies both instance level and network interface security groups. Consider the following conditions that apply to using custom security groups with managed node groups:

When using the consolelong, Amazon EKS only allows launch templates with a single network interface specification.
By default, Amazon EKS applies the cluster security group to the instances in your node group to facilitate communication between nodes and the control plane. If you specify custom security groups in the launch template using either option mentioned earlier, Amazon EKS doesn’t add the cluster security group. So, you must ensure that the inbound and outbound rules of your security groups enable communication with the endpoint of your cluster. If your security group rules are incorrect, the worker nodes can’t join the cluster. For more information about security group rules, see sec-group-reqs.title.
If you need SSH access to the instances in your node group, include a security group that allows that access.

Amazon EC2 user data

The launch template includes a section for custom user data. You can specify configuration settings for your node group in this section without manually creating individual custom AMIs. For more information about the settings available for Bottlerocket, see Using user data on GitHub.

You can supply Amazon EC2 user data in your launch template using cloud-init when launching your instances. For more information, see the cloud-init documentation. Your user data can be used to perform common configuration operations. This includes the following operations:

Amazon EC2 user data in launch templates that are used with managed node groups must be in the MIME multi-part archive format for Amazon Linux AMIs and TOML format for Bottlerocket AMIs. This is because your user data is merged with Amazon EKS user data required for nodes to join the cluster. Don’t specify any commands in your user data that starts or modifies kubelet. This is performed as part of the user data merged by Amazon EKS. Certain kubelet parameters, such as setting labels on nodes, can be configured directly through the managed node groups API.

For more information about advanced kubelet customization, including manually starting it or passing in custom configuration parameters, see launch-template-custom-ami.title. If a custom AMI ID is specified in a launch template, Amazon EKS doesn’t merge user data.

The following details provide more information about the user data section.

Amazon Linux 2 user data

You can combine multiple user data blocks together into a single MIME multi-part file. For example, you can combine a cloud boothook that configures the Docker daemon with a user data shell script that installs a custom package. A MIME multi-part file consists of the following components:

The content type and part boundary declaration – Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
The MIME version declaration – MIME-Version: 1.0
One or more user data blocks, which contain the following components:
- The opening boundary, which signals the beginning of a user data block – --==MYBOUNDARY==
- The content type declaration for the block: Content-Type: text/cloud-config; charset="us-ascii". For more information about content types, see the cloud-init documentation.
- The content of the user data (for example, a list of shell commands or cloud-init directives).
- The closing boundary, which signals the end of the MIME multi-part file: --==MYBOUNDARY==--

The following is an example of a MIME multi-part file that you can use to create your own.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script"

--==MYBOUNDARY==--

Amazon Linux 2023 user data

Amazon Linux 2023 (AL2023) introduces a new node initialization process nodeadm that uses a YAML configuration schema. If you’re using self-managed node groups or an AMI with a launch template, you’ll now need to provide additional cluster metadata explicitly when creating a new node group. An example of the minimum required parameters is as follows, where apiServerEndpoint, certificateAuthority, and service cidr are now required:

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: my-cluster
    apiServerEndpoint: https://example.com
    certificateAuthority: Y2VydGlmaWNhdGVBdXRob3JpdHk=
    cidr: 10.100.0.0/16

You’ll typically set this configuration in your user data, either as-is or embedded within a MIME multi-part document:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig spec: [...]

--BOUNDARY--

In AL2, the metadata from these parameters was discovered from the Amazon EKS DescribeCluster API call. With AL2023, this behavior has changed since the additional API call risks throttling during large node scale ups. This change doesn’t affect you if you’re using managed node groups without a launch template or if you’re using Karpenter. For more information on certificateAuthority and service cidr, see ` DescribeCluster` in the Amazon EKS API Reference.

Bottlerocket user data

Bottlerocket structures user data in the TOML format. You can provide user data to be merged with the user data provided by Amazon EKS. For example, you can provide additional kubelet settings.

[settings.kubernetes.system-reserved]
cpu = "10m"
memory = "100Mi"
ephemeral-storage= "1Gi"

For more information about the supported settings, see Bottlerocket documentation. You can configure node labels and taints in your user data. However, we recommend that you configure these within your node group instead. Amazon EKS applies these configurations when you do so.

When user data is merged, formatting isn’t preserved, but the content remains the same. The configuration that you provide in your user data overrides any settings that are configured by Amazon EKS. So, if you set settings.kubernetes.max-pods or settings.kubernetes.cluster-dns-ip, these values in your user data are applied to the nodes.

Amazon EKS doesn’t support all valid TOML. The following is a list of known unsupported formats:

Quotes within quoted keys: 'quoted "value"' = "value"
Escaped quotes in values: str = "I’m a string. \"You can quote me\""
Mixed floats and integers: numbers = [ 0.1, 0.2, 0.5, 1, 2, 5 ]
Mixed types in arrays: contributors = ["foo@example.com", { name = "Baz", email = "baz@example.com" }]
Bracketed headers with quoted keys: [foo."bar.baz"]

Windows user data

Windows user data uses PowerShell commands. When creating a managed node group, your custom user data combines with Amazon EKS managed user data. Your PowerShell commands come first, followed by the managed user data commands, all within one <powershell></powershell> tag.

When no AMI ID is specified in the launch template, don’t use the Windows Amazon EKS Bootstrap script in user data to configure Amazon EKS.

Example user data is as follows.

<powershell>
Write-Host "Running custom user data script"
</powershell>

Specifying an AMI

If you have either of the following requirements, then specify an AMI ID in the ImageId field of your launch template. Select the requirement you have for additional information.

Provide user data to pass arguments to the bootstrap.sh file included with an Amazon EKS optimized Linux/Bottlerocket AMI

Bootstrapping is a term used to describe adding commands that can be run when an instance starts. For example, bootstrapping allows using extra kubelet arguments. You can pass arguments to the bootstrap.sh script by using eksctl without specifying a launch template. Or you can do so by specifying the information in the user data section of a launch template.

eksctl without specifying a launch template

Create a file named my-nodegroup.yaml with the following contents. Replace every example value with your own values. The --apiserver-endpoint, --b64-cluster-ca, and --dns-cluster-ip arguments are optional. However, defining them allows the bootstrap.sh script to avoid making a describeCluster call. This is useful in private cluster setups or clusters where you’re scaling in and out nodes frequently. For more information on the bootstrap.sh script, see the bootstrap.sh file on GitHub.

The only required argument is the cluster name (my-cluster).
To retrieve an optimized AMI ID for ami-1234567890abcdef0, you can use the tables in the following sections:

To retrieve the certificate-authority for your cluster, run the following command.

aws eks describe-cluster --query "cluster.certificateAuthority.data" --output text --name my-cluster --region region-code

To retrieve the api-server-endpoint for your cluster, run the following command.

aws eks describe-cluster --query "cluster.endpoint" --output text --name my-cluster --region region-code

The value for --dns-cluster-ip is your service CIDR with .10 at the end. To retrieve the service-cidr for your cluster, run the following command. For example, if the returned value for is ipv4 10.100.0.0/16, then your value is 10.100.0.10.
```
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
```

This example provides a kubelet argument to set a custom max-pods value using the bootstrap.sh script included with the Amazon EKS optimized AMI. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. For help with selecting my-max-pods-value, see determine-max-pods.title.

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my-cluster
  region: region-code

managedNodeGroups:
  - name: my-nodegroup
    ami: ami-1234567890abcdef0
    instanceType: m5.large
    privateNetworking: true
    disableIMDSv1: true
    labels: { x86-al2-specified-mng }
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh my-cluster \
        --b64-cluster-ca certificate-authority \
        --apiserver-endpoint api-server-endpoint \
        --dns-cluster-ip service-cidr.10 \
        --kubelet-extra-args '--max-pods=my-max-pods-value' \
        --use-max-pods false

For every available eksctl config file option, see Config file schema in the eksctl documentation. The eksctl utility still creates a launch template for you and populates its user data with the data that you provide in the config file.

Create a node group with the following command.

eksctl create nodegroup --config-file=my-nodegroup.yaml

User data in a launch template

Specify the following information in the user data section of your launch template. Replace every example value with your own values. The --apiserver-endpoint, --b64-cluster-ca, and --dns-cluster-ip arguments are optional. However, defining them allows the bootstrap.sh script to avoid making a describeCluster call. This is useful in private cluster setups or clusters where you’re scaling in and out nodes frequently. For more information on the bootstrap.sh script, see the bootstrap.sh file on GitHub.

The only required argument is the cluster name (my-cluster).

To retrieve the certificate-authority for your cluster, run the following command.

aws eks describe-cluster --query "cluster.certificateAuthority.data" --output text --name my-cluster --region region-code

To retrieve the api-server-endpoint for your cluster, run the following command.

aws eks describe-cluster --query "cluster.endpoint" --output text --name my-cluster --region region-code

The value for --dns-cluster-ip is your service CIDR with .10 at the end. To retrieve the service-cidr for your cluster, run the following command. For example, if the returned value for is ipv4 10.100.0.0/16, then your value is 10.100.0.10.
```
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
```

This example provides a kubelet argument to set a custom max-pods value using the bootstrap.sh script included with the Amazon EKS optimized AMI. For help with selecting my-max-pods-value, see determine-max-pods.title.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
set -ex
/etc/eks/bootstrap.sh my-cluster \
  --b64-cluster-ca certificate-authority \
  --apiserver-endpoint api-server-endpoint \
  --dns-cluster-ip service-cidr.10 \
  --kubelet-extra-args '--max-pods=my-max-pods-value' \
  --use-max-pods false

--==MYBOUNDARY==--

Provide user data to pass arguments to the Start-EKSBootstrap.ps1 file included with an Amazon EKS optimized Windows AMI

Bootstrapping is a term used to describe adding commands that can be run when an instance starts. You can pass arguments to the Start-EKSBootstrap.ps1 script by using eksctl without specifying a launch template. Or you can do so by specifying the information in the user data section of a launch template.

If you want to specify a custom Windows AMI ID, keep in mind the following considerations:

You must use a launch template and give the required bootstrap commands in the user data section. To retrieve your desired Windows ID, you can use the table in Create nodes with optimized Windows AMIs.
There are several limits and conditions. For example, you must add eks:kube-proxy-windows to your AWS IAM Authenticator configuration map. For more information, see mng-ami-id-conditions.title.

Specify the following information in the user data section of your launch template. Replace every example value with your own values. The -APIServerEndpoint, -Base64ClusterCA, and -DNSClusterIP arguments are optional. However, defining them allows the Start-EKSBootstrap.ps1 script to avoid making a describeCluster call.

The only required argument is the cluster name (my-cluster).

To retrieve the certificate-authority for your cluster, run the following command.

aws eks describe-cluster --query "cluster.certificateAuthority.data" --output text --name my-cluster --region region-code

To retrieve the api-server-endpoint for your cluster, run the following command.

aws eks describe-cluster --query "cluster.endpoint" --output text --name my-cluster --region region-code

The value for --dns-cluster-ip is your service CIDR with .10 at the end. To retrieve the service-cidr for your cluster, run the following command. For example, if the returned value for is ipv4 10.100.0.0/16, then your value is 10.100.0.10.
```
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
```
For additional arguments, see bootstrap-script-configuration-parameters.title.

If you’re using custom service CIDR, then you need to specify it using the -ServiceCIDR parameter. Otherwise, the DNS resolution for Pods in the cluster will fail.

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName my-cluster `
	 -Base64ClusterCA certificate-authority `
	 -APIServerEndpoint api-server-endpoint `
	 -DNSClusterIP service-cidr.10
</powershell>

Run a custom AMI due to specific security, compliance, or internal policy requirements

For more information, see Amazon Machine Images (AMI) in the Amazon EC2 User Guide. The Amazon EKS AMI build specification contains resources and configuration scripts for building a custom Amazon EKS AMI based on Amazon Linux. For more information, see Amazon EKS AMI Build Specification on GitHub. To build custom AMIs installed with other operating systems, see Amazon EKS Sample Custom AMIs on GitHub.

When specifying an AMI, Amazon EKS doesn’t merge any user data. Rather, you’re responsible for supplying the required bootstrap commands for nodes to join the cluster. If your nodes fail to join the cluster, the Amazon EKS CreateNodegroup and UpdateNodegroupVersion actions also fail.

Limits and conditions when specifying an AMI ID

The following are the limits and conditions involved with specifying an AMI ID with managed node groups:

You must create a new node group to switch between specifying an AMI ID in a launch template and not specifying an AMI ID.
You aren’t notified in the console when a newer AMI version is available. To update your node group to a newer AMI version, you need to create a new version of your launch template with an updated AMI ID. Then, you need to update the node group with the new launch template version.
The following fields can’t be set in the API if you specify an AMI ID:
- amiType
- releaseVersion
- version
Any taints set in the API are applied asynchronously if you specify an AMI ID. To apply taints prior to a node joining the cluster, you must pass the taints to kubelet in your user data using the --register-with-taints command line flag. For more information, see kubelet in the Kubernetes documentation.
When specifying a custom AMI ID for Windows managed node groups, add eks:kube-proxy-windows to your AWS IAM Authenticator configuration map. This is required for DNS to function properly.
1. Open the AWS IAM Authenticator configuration map for editing.
  kubectl edit -n kube-system cm aws-auth
2. Add this entry to the groups list under each rolearn associated with Windows nodes. Your configuration map should look similar to aws-auth-cm-windows.yaml.
  - eks:kube-proxy-windows
3. Save the file and exit your text editor.

9.2.5. Delete a managed node group from your cluster

This topic describes how you can delete an Amazon EKS managed node group.

This topic describes how you can delete an Amazon EKS managed node group. When you delete a managed node group, Amazon EKS first sets the minimum, maximum, and desired size of your Auto Scaling group to zero. This then causes your node group to scale down.

Before each instance is terminated, Amazon EKS sends a signal to drain the Pods from that node. If the Pods haven’t drained after a few minutes, Amazon EKS lets Auto Scaling continue the termination of the instance. After every instance is terminated, the Auto Scaling group is deleted.

If you delete a managed node group that uses a node IAM role that isn’t used by any other managed node group in the cluster, the role is removed from the aws-auth ConfigMap. If any of the self-managed node groups in the cluster are using the same node IAM role, the self-managed nodes move to the NotReady status. Additionally, the cluster operation is also disrupted. To add a mapping for the role you’re using only for the self-managed node groups, see creating-access-entries.title, if your cluster’s platform version is at least minimum version listed in the prerequisites section of Grant IAM users access to Kubernetes with EKS access entries. If your platform version is earlier than the required minimum version for access entries, you can add the entry back to the aws-auth ConfigMap. For more information, enter eksctl create iamidentitymapping --help in your terminal.

You can delete a managed node group with:

eksctl
consolelong
AWS CLI

`eksctl`

Delete a managed node group with eksctl

Enter the following command. Replace every example value with your own values.

eksctl delete nodegroup \
  --cluster my-cluster \
  --name my-mng \
  --region region-code

For more options, see Deleting and draining nodegroups in the eksctl documentation.

`consolelong`

Delete a managed node group with consolelong

Open the Amazon EKS console.
On the Clusters page, choose the cluster that contains the node group to delete.
On the selected cluster page, choose the Compute tab.
In the Node groups section, choose the node group to delete. Then choose Delete.
In the Delete node group confirmation dialog box, enter the name of the node group. Then choose Delete.

`AWS` CLI

Delete a managed node group with AWS CLI

Enter the following command. Replace every example value with your own values.

aws eks delete-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name my-mng \
  --region region-code

Use the arrow keys on your keyboard to scroll through the response output. Press the q key when you’re finished.

For more options, see the delete-nodegroup command in the AWS CLI Command Reference.

Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters.

Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters.

With Amazon EKS managed node groups, you don’t need to separately provision or register the Amazon EC2 instances that provide compute capacity to run your Kubernetes applications. You can create, automatically update, or terminate nodes for your cluster with a single operation. Node updates and terminations automatically drain nodes to ensure that your applications stay available.

Every managed node is provisioned as part of an Amazon EC2 Auto Scaling group that’s managed for you by Amazon EKS. Every resource including the instances and Auto Scaling groups runs within your AWS account. Each node group runs across multiple Availability Zones that you define.

Managed node groups can also optionally leverage node auto repair, which continuously monitors the health of nodes. It automatically reacts to detected problems and replaces nodes when possible. This helps overall availability of the cluster with minimal manual intervention. For more information, see node-health.title.

You can add a managed node group to new or existing clusters using the Amazon EKS console, eksctl, AWS CLI, AWS API, or infrastructure as code tools including AWS CloudFormation. Nodes launched as part of a managed node group are automatically tagged for auto-discovery by the Kubernetes Cluster Autoscaler. You can use the node group to apply Kubernetes labels to nodes and update them at any time.

There are no additional costs to use Amazon EKS managed node groups, you only pay for the AWS resources you provision. These include Amazon EC2 instances, Amazon EBS volumes, Amazon EKS cluster hours, and any other AWS infrastructure. There are no minimum fees and no upfront commitments.

To get started with a new Amazon EKS cluster and managed node group, see getting-started-console.title.

To add a managed node group to an existing cluster, see create-managed-node-group.title.

9.2.6. Managed node groups concepts

Amazon EKS managed node groups create and manage Amazon EC2 instances for you.
Every managed node is provisioned as part of an Amazon EC2 Auto Scaling group that’s managed for you by Amazon EKS. Moreover, every resource including Amazon EC2 instances and Auto Scaling groups run within your AWS account.
The Auto Scaling group of a managed node group spans every subnet that you specify when you create the group.

Amazon EKS tags managed node group resources so that they are configured to use the Kubernetes Cluster Autoscaler.

You can use a custom launch template for a greater level of flexibility and customization when deploying managed nodes. For example, you can specify extra kubelet arguments and use a custom AMI. For more information, see launch-templates.title. If you don’t use a custom launch template when first creating a managed node group, there is an auto-generated launch template. Don’t manually modify this auto-generated template or errors occur.
Amazon EKS follows the shared responsibility model for CVEs and security patches on managed node groups. When managed nodes run an Amazon EKS optimized AMI, Amazon EKS is responsible for building patched versions of the AMI when bugs or issues are reported. We can publish a fix. However, you’re responsible for deploying these patched AMI versions to your managed node groups. When managed nodes run a custom AMI, you’re responsible for building patched versions of the AMI when bugs or issues are reported and then deploying the AMI. For more information, see update-managed-node-group.title.
Amazon EKS managed node groups can be launched in both public and private subnets. If you launch a managed node group in a public subnet on or after April 22, 2020, the subnet must have MapPublicIpOnLaunch set to true for the instances to successfully join a cluster. If the public subnet was created using eksctl or the Amazon EKS vended AWS CloudFormation templates on or after March 26, 2020, then this setting is already set to true. If the public subnets were created before March 26, 2020, you must change the setting manually. For more information, see Modifying the public IPv4 addressing attribute for your subnet.
When deploying a managed node group in private subnets, you must ensure that it can access Amazon ECR for pulling container images. You can do this by connecting a NAT gateway to the route table of the subnet or by adding the following AWS PrivateLink VPC endpoints:
- Amazon ECR API endpoint interface – com.amazonaws.region-code.ecr.api
- Amazon ECR Docker registry API endpoint interface – com.amazonaws.region-code.ecr.dkr
- Amazon S3 gateway endpoint – com.amazonaws.region-code.s3
For other commonly-used services and endpoints, see private-clusters.title.
Managed node groups can’t be deployed on AWS Outposts or in AWS Wavelength. Managed node groups can be created on AWS Local Zones. For more information, see local-zones.title.
You can create multiple managed node groups within a single cluster. For example, you can create one node group with the standard Amazon EKS optimized Amazon Linux AMI for some workloads and another with the GPU variant for workloads that require GPU support.
If your managed node group encounters an Amazon EC2 instance status check failure, Amazon EKS returns an error code to help you to diagnose the issue. For more information, see troubleshoot-managed-node-groups.title.
Amazon EKS adds Kubernetes labels to managed node group instances. These Amazon EKS provided labels are prefixed with eks.amazonaws.com.
Amazon EKS automatically drains nodes using the Kubernetes API during terminations or updates.
Pod disruption budgets aren’t respected when terminating a node with AZRebalance or reducing the desired node count. These actions try to evict Pods on the node. But if it takes more than 15 minutes, the node is terminated regardless of whether all Pods on the node are terminated. To extend the period until the node is terminated, add a lifecycle hook to the Auto Scaling group. For more information, see Add lifecycle hooks in the Amazon EC2 Auto Scaling User Guide.
In order to run the drain process correctly after receiving a Spot interruption notification or a capacity rebalance notification, CapacityRebalance must be set to true.
Updating managed node groups respects the Pod disruption budgets that you set for your Pods. For more information, see managed-node-update-behavior.title.
There are no additional costs to use Amazon EKS managed node groups. You only pay for the AWS resources that you provision.
If you want to encrypt Amazon EBS volumes for your nodes, you can deploy the nodes using a launch template. To deploy managed nodes with encrypted Amazon EBS volumes without using a launch template, encrypt all new Amazon EBS volumes created in your account. For more information, see Encryption by default in the Amazon EC2 User Guide.

9.2.7. Managed node group capacity types

When creating a managed node group, you can choose either the On-Demand or Spot capacity type. Amazon EKS deploys a managed node group with an Amazon EC2 Auto Scaling group that either contains only On-Demand or only Amazon EC2 Spot Instances. You can schedule Pods for fault tolerant applications to Spot managed node groups, and fault intolerant applications to On-Demand node groups within a single Kubernetes cluster. By default, a managed node group deploys On-Demand Amazon EC2 instances.

On-Demand

With On-Demand Instances, you pay for compute capacity by the second, with no long-term commitments.

By default, if you don’t specify a Capacity Type, the managed node group is provisioned with On-Demand Instances. A managed node group configures an Amazon EC2 Auto Scaling group on your behalf with the following settings applied:

The allocation strategy to provision On-Demand capacity is set to prioritized. Managed node groups use the order of instance types passed in the API to determine which instance type to use first when fulfilling On-Demand capacity. For example, you might specify three instance types in the following order: c5.large, c4.large, and c3.large. When your On-Demand Instances are launched, the managed node group fulfills On-Demand capacity by starting with c5.large, then c4.large, and then c3.large. For more information, see Amazon EC2 Auto Scaling group in the Amazon EC2 Auto Scaling User Guide.
Amazon EKS adds the following Kubernetes label to all nodes in your managed node group that specifies the capacity type: eks.amazonaws.com/capacityType: ON_DEMAND. You can use this label to schedule stateful or fault intolerant applications on On-Demand nodes.

Spot

Amazon EC2 Spot Instances are spare Amazon EC2 capacity that offers steep discounts off of On-Demand prices. Amazon EC2 Spot Instances can be interrupted with a two-minute interruption notice when EC2 needs the capacity back. For more information, see Spot Instances in the Amazon EC2 User Guide. You can configure a managed node group with Amazon EC2 Spot Instances to optimize costs for the compute nodes running in your Amazon EKS cluster.

To use Spot Instances inside a managed node group, create a managed node group by setting the capacity type as spot. A managed node group configures an Amazon EC2 Auto Scaling group on your behalf with the following Spot best practices applied:

To ensure that your Spot nodes are provisioned in the optimal Spot capacity pools, the allocation strategy is set to one of the following:
- price-capacity-optimized (PCO) – When creating new node groups in a cluster with Kubernetes version 1.28 or higher, the allocation strategy is set to price-capacity-optimized. However, the allocation strategy won’t be changed for node groups already created with capacity-optimized before Amazon EKS managed node groups started to support PCO.
- capacity-optimized (CO) – When creating new node groups in a cluster with Kubernetes version 1.27 or lower, the allocation strategy is set to capacity-optimized.
To increase the number of Spot capacity pools available for allocating capacity from, configure a managed node group to use multiple instance types.
Amazon EC2 Spot Capacity Rebalancing is enabled so that Amazon EKS can gracefully drain and rebalance your Spot nodes to minimize application disruption when a Spot node is at elevated risk of interruption. For more information, see Amazon EC2 Auto Scaling Capacity Rebalancing in the Amazon EC2 Auto Scaling User Guide.
- When a Spot node receives a rebalance recommendation, Amazon EKS automatically attempts to launch a new replacement Spot node.
- If a Spot two-minute interruption notice arrives before the replacement Spot node is in a Ready state, Amazon EKS starts draining the Spot node that received the rebalance recommendation. Amazon EKS drains the node on a best-effort basis. As a result, there’s no guarantee that Amazon EKS will wait for the replacement node to join the cluster before draining the existing node.
- When a replacement Spot node is bootstrapped and in the Ready state on Kubernetes, Amazon EKS cordons and drains the Spot node that received the rebalance recommendation. Cordoning the Spot node ensures that the service controller doesn’t send any new requests to this Spot node. It also removes it from its list of healthy, active Spot nodes. Draining the Spot node ensures that running Pods are evicted gracefully.
Amazon EKS adds the following Kubernetes label to all nodes in your managed node group that specifies the capacity type: eks.amazonaws.com/capacityType: SPOT. You can use this label to schedule fault tolerant applications on Spot nodes.

When deciding whether to deploy a node group with On-Demand or Spot capacity, you should consider the following conditions:

Spot Instances are a good fit for stateless, fault-tolerant, flexible applications. These include batch and machine learning training workloads, big data ETLs such as Apache Spark, queue processing applications, and stateless API endpoints. Because Spot is spare Amazon EC2 capacity, which can change over time, we recommend that you use Spot capacity for interruption-tolerant workloads. More specifically, Spot capacity is suitable for workloads that can tolerate periods where the required capacity isn’t available.
We recommend that you use On-Demand for applications that are fault intolerant. This includes cluster management tools such as monitoring and operational tools, deployments that require StatefulSets, and stateful applications, such as databases.
To maximize the availability of your applications while using Spot Instances, we recommend that you configure a Spot managed node group to use multiple instance types. We recommend applying the following rules when using multiple instance types:
- Within a managed node group, if you’re using the Cluster Autoscaler, we recommend using a flexible set of instance types with the same amount of vCPU and memory resources. This is to ensure that the nodes in your cluster scale as expected. For example, if you need four vCPUs and eight GiB memory, use c3.xlarge, c4.xlarge, c5.xlarge, c5d.xlarge, c5a.xlarge, c5n.xlarge, or other similar instance types.
- To enhance application availability, we recommend deploying multiple Spot managed node groups. For this, each group should use a flexible set of instance types that have the same vCPU and memory resources. For example, if you need 4 vCPUs and 8 GiB memory, we recommend that you create one managed node group with c3.xlarge, c4.xlarge, c5.xlarge, c5d.xlarge, c5a.xlarge, c5n.xlarge, or other similar instance types, and a second managed node group with m3.xlarge, m4.xlarge, m5.xlarge, m5d.xlarge, m5a.xlarge, m5n.xlarge or other similar instance types.
- When deploying your node group with the Spot capacity type that’s using a custom launch template, use the API to pass multiple instance types. Don’t pass a single instance type through the launch template. For more information about deploying a node group using a launch template, see launch-templates.title.

9.3. Maintain nodes yourself with self-managed nodes

9.3.1. Create self-managed Amazon Linux nodes

This topic describes how you can launch Auto Scaling groups of Linux nodes that register with your Amazon EKS cluster.

This topic describes how you can launch Auto Scaling groups of Linux nodes that register with your Amazon EKS cluster. After the nodes join the cluster, you can deploy Kubernetes applications to them. You can also launch self-managed Amazon Linux nodes with eksctl or the consolelong. If you need to launch nodes on AWS Outposts, see eks-outposts-self-managed-nodes.title.

An existing Amazon EKS cluster. To deploy one, see create-cluster.title. If you have subnets in the AWS Region where you have AWS Outposts, AWS Wavelength, or AWS Local Zones enabled, those subnets must not have been passed in when you created your cluster.
An existing IAM role for the nodes to use. To create one, see create-node-role.title. If this role doesn’t have either of the policies for the VPC CNI, the separate role that follows is required for the VPC CNI pods.
(Optional, but recommended) The Amazon VPC CNI plugin for Kubernetes add-on configured with its own IAM role that has the necessary IAM policy attached to it. For more information, see cni-iam-role.title.
Familiarity with the considerations listed in Choose an optimal Amazon EC2 node instance type. Depending on the instance type you choose, there may be additional prerequisites for your cluster and VPC.

You can launch self-managed Linux nodes using either of the following:

eksctl
consolelong

`eksctl`

Launch self-managed Linux nodes using eksctl

Install version 0.199.0 or later of the eksctl command line tool installed on your device or AWS CloudShell. To install or update eksctl, see Installation in the eksctl documentation.
(Optional) If the AmazonEKS_CNI_Policy managed IAM policy is attached to your Amazon EKS node IAM role, we recommend assigning it to an IAM role that you associate to the Kubernetes aws-node service account instead. For more information, see cni-iam-role.title.

The following command creates a node group in an existing cluster. Replace al-nodes with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. Replace my-cluster with the name of your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in. Replace the remaining example value with your own values. The nodes are created with the same Kubernetes version as the control plane, by default.

Before choosing a value for --node-type, review Choose an optimal Amazon EC2 node instance type.

Replace my-key with the name of your Amazon EC2 key pair or public key. This key is used to SSH into your nodes after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the consolelong. For more information, see Amazon EC2 key pairs in the Amazon EC2 User Guide.

Create your node group with the following command.

If you want to deploy a node group to AWS Outposts, Wavelength, or Local Zone subnets, there are additional considerations:

The subnets must not have been passed in when you created the cluster.
You must create the node group with a config file that specifies the subnets and volumeType: gp2. For more information, see Create a nodegroup from a config file and Config file schema in the eksctl documentation.

eksctl create nodegroup \
  --cluster my-cluster \
  --name al-nodes \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4 \
  --ssh-access \
  --managed=false \
  --ssh-public-key my-key

To deploy a node group that:

can assign a significantly higher number of IP addresses to Pods than the default configuration, see cni-increase-ip-addresses.title.
can assign IPv4 addresses to Pods from a different CIDR block than that of the instance, see cni-custom-network.title.
can assign IPv6 addresses to Pods and services, see cni-ipv6.title.
use the containerd runtime, you must deploy the node group using a config file. For more information, see containerd-bootstrap.title.
don’t have outbound internet access, see private-clusters.title.

For a complete list of all available options and defaults, enter the following command.
```
eksctl create nodegroup --help
```
If nodes fail to join the cluster, then see worker-node-fail.title in the Troubleshooting chapter.

An example output is as follows. Several lines are output while the nodes are created. One of the last lines of output is the following example line.
```
[✔]  created 1 nodegroup(s) in cluster "my-cluster"
```

(Optional) Deploy a sample application to test your cluster and Linux nodes.
We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see Restrict access to the instance profile assigned to the worker node.

`consolelong`

Step 1: Launch self-managed Linux nodes using consolelong

Download the latest version of the AWS CloudFormation template.

curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2022-12-23/amazon-eks-nodegroup.yaml

Wait for your cluster status to show as ACTIVE. If you launch your nodes before the cluster is active, the nodes fail to register with the cluster and you will have to relaunch them.
Open the AWS CloudFormation console.
Choose Create stack and then select With new resources (standard).
For Specify template, select Upload a template file and then select Choose file.
Select the amazon-eks-nodegroup.yaml file that you downloaded.
Select Next.

On the Specify stack details page, enter the following parameters accordingly, and then choose Next:

Stack name: Choose a stack name for your AWS CloudFormation stack. For example, you can call it my-cluster-nodes. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
ClusterName: Enter the name that you used when you created your Amazon EKS cluster. This name must equal the cluster name or your nodes can’t join the cluster.
ClusterControlPlaneSecurityGroup: Choose the SecurityGroups value from the AWS CloudFormation output that you generated when you created your VPC.

The following steps show one operation to retrieve the applicable group.
1. Open the Amazon EKS console.
2. Choose the name of the cluster.
3. Choose the Networking tab.
4. Use the Additional security groups value as a reference when selecting from the ClusterControlPlaneSecurityGroup dropdown list.
NodeGroupName: Enter a name for your node group. This name can be used later to identify the Auto Scaling node group that’s created for your nodes. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
NodeAutoScalingGroupMinSize: Enter the minimum number of nodes that your node Auto Scaling group can scale in to.
NodeAutoScalingGroupDesiredCapacity: Enter the desired number of nodes to scale to when your stack is created.
NodeAutoScalingGroupMaxSize: Enter the maximum number of nodes that your node Auto Scaling group can scale out to.
NodeInstanceType: Choose an instance type for your nodes. For more information, see choosing-instance-type.title.

NodeImageIdSSMParam: Pre-populated with the Amazon EC2 Systems Manager parameter of a recent Amazon EKS optimized AMI for a variable Kubernetes version. To use a different Kubernetes minor version supported with Amazon EKS, replace 1.XX with a different supported version. We recommend specifying the same Kubernetes version as your cluster.

You can also replace amazon-linux-2 with a different AMI type. For more information, see retrieve-ami-id.title.

The Amazon EKS node AMIs are based on Amazon Linux. You can track security or privacy events for Amazon Linux 2 at the Amazon Linux Security Center or subscribe to the associated RSS feed. Security and privacy events include an overview of the issue, what packages are affected, and how to update your instances to correct the issue.

NodeImageId: (Optional) If you’re using your own custom AMI (instead of an Amazon EKS optimized AMI), enter a node AMI ID for your AWS Region. If you specify a value here, it overrides any values in the NodeImageIdSSMParam field.
NodeVolumeSize: Specify a root volume size for your nodes, in GiB.
NodeVolumeType: Specify a root volume type for your nodes.
KeyName: Enter the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the consolelong. For more information, see Amazon EC2 key pairs in the Amazon EC2 User Guide.

If you don’t provide a key pair here, the AWS CloudFormation stack creation fails.
BootstrapArguments: Specify any optional arguments to pass to the node bootstrap script, such as extra kubelet arguments. For more information, view the bootstrap script usage information on GitHub.

To deploy a node group that:
- can assign a significantly higher number of IP addresses to Pods than the default configuration, see cni-increase-ip-addresses.title.
- can assign IPv4 addresses to Pods from a different CIDR block than that of the instance, see cni-custom-network.title.
- can assign IPv6 addresses to Pods and services, see cni-ipv6.title.
- use the containerd runtime, you must deploy the node group using a config file. For more information, see containerd-bootstrap.title.
- don’t have outbound internet access, see private-clusters.title.
DisableIMDSv1: By default, each node supports the Instance Metadata Service Version 1 (IMDSv1) and IMDSv2. You can disable IMDSv1. To prevent future nodes and Pods in the node group from using MDSv1, set DisableIMDSv1 to true. For more information about IMDS, see Configuring the instance metadata service. For more information about restricting access to it on your nodes, see Restrict access to the instance profile assigned to the worker node.
VpcId: Enter the ID for the VPC that you created.

Subnets: Choose the subnets that you created for your VPC. If you created your VPC using the steps that are described in Create an Amazon VPC for your Amazon EKS cluster, specify only the private subnets within the VPC for your nodes to launch into. You can see which subnets are private by opening each subnet link from the Networking tab of your cluster.

If any of the subnets are public subnets, then they must have the automatic public IP address assignment setting enabled. If the setting isn’t enabled for the public subnet, then any nodes that you deploy to that public subnet won’t be assigned a public IP address and won’t be able to communicate with the cluster or other AWS services. If the subnet was deployed before March 26, 2020 using either of the Amazon EKS AWS CloudFormation VPC templates, or by using eksctl, then automatic public IP address assignment is disabled for public subnets. For information about how to enable public IP address assignment for a subnet, see Modifying the public IPv4 addressing attribute for your subnet. If the node is deployed to a private subnet, then it’s able to communicate with the cluster and other AWS services through a NAT gateway.
If the subnets don’t have internet access, make sure that you’re aware of the considerations and extra steps in Deploy private clusters with limited internet access.
If you select AWS Outposts, Wavelength, or Local Zone subnets, the subnets must not have been passed in when you created the cluster.

Select your desired choices on the Configure stack options page, and then choose Next.
Select the check box to the left of I acknowledge that AWS CloudFormation might create IAM resources., and then choose Create stack.
When your stack has finished creating, select it in the console and choose Outputs.
Record the NodeInstanceRole for the node group that was created. You need this when you configure your Amazon EKS nodes.

Step 2: Enable nodes to join your cluster

If you launched nodes inside a private VPC without outbound internet access, make sure to enable nodes to join your cluster from within the VPC.

Check to see if you already have an aws-auth ConfigMap.
```
kubectl describe configmap -n kube-system aws-auth
```
If you are shown an aws-auth ConfigMap, then update it as needed.
1. Open the ConfigMap for editing.
  kubectl edit -n kube-system configmap/aws-auth
2. Add a new mapRoles entry as needed. Set the rolearn value to the NodeInstanceRole value that you recorded in the previous procedure.
  [...] data: mapRoles: | - rolearn: <ARN of instance role (not instance profile)> username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes [...]
3. Save the file and exit your text editor.
If you received an error stating "Error from server (NotFound): configmaps "aws-auth" not found, then apply the stock ConfigMap.
1. Download the configuration map.
  curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/aws-auth-cm.yaml
2. In the aws-auth-cm.yaml file, set the rolearn value to the NodeInstanceRole value that you recorded in the previous procedure. You can do this with a text editor, or by replacing my-node-instance-role and running the following command:
  sed -i.bak -e 's|<ARN of instance role (not instance profile)>|my-node-instance-role|' aws-auth-cm.yaml
3. Apply the configuration. This command may take a few minutes to finish.
  kubectl apply -f aws-auth-cm.yaml
Watch the status of your nodes and wait for them to reach the Ready status.
```
kubectl get nodes --watch
```
Enter Ctrl+C to return to a shell prompt.

If you receive any authorization or resource type errors, see unauthorized.title in the troubleshooting topic.

If nodes fail to join the cluster, then see worker-node-fail.title in the Troubleshooting chapter.
(GPU nodes only) If you chose a GPU instance type and the Amazon EKS optimized accelerated AMI, you must apply the NVIDIA device plugin for Kubernetes as a DaemonSet on your cluster. Replace vX.X.X with your desired NVIDIA/k8s-device-plugin version before running the following command.
```
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/vX.X.X/deployments/static/nvidia-device-plugin.yml
```

Step 3: Additional actions

(Optional) Deploy a sample application to test your cluster and Linux nodes.
(Optional) If the AmazonEKS_CNI_Policy managed IAM policy (if you have an IPv4 cluster) or the AmazonEKS_CNI_IPv6_Policy (that you created yourself if you have an IPv6 cluster) is attached to your Amazon EKS node IAM role, we recommend assigning it to an IAM role that you associate to the Kubernetes aws-node service account instead. For more information, see cni-iam-role.title.
We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see Restrict access to the instance profile assigned to the worker node.

9.3.2. Create self-managed `Bottlerocket` nodes

This topic describes how to launch Auto Scaling groups of Bottlerocket nodes that register with your Amazon EKS cluster

Managed node groups might offer some advantages for your use case. For more information, see managed-node-groups.title.

This topic describes how to launch Auto Scaling groups of Bottlerocket nodes that register with your Amazon EKS cluster. Bottlerocket is a Linux-based open-source operating system from AWS that you can use for running containers on virtual machines or bare metal hosts. After the nodes join the cluster, you can deploy Kubernetes applications to them. For more information about Bottlerocket, see Using a Bottlerocket AMI with Amazon EKS on GitHub and Custom AMI support in the eksctl documentation.

For information about in-place upgrades, see Bottlerocket Update Operator on GitHub.

Amazon EKS nodes are standard Amazon EC2 instances, and you are billed for them based on normal Amazon EC2 instance prices. For more information, see Amazon EC2 pricing.
You can launch Bottlerocket nodes in Amazon EKS extended clusters on AWS Outposts, but you can’t launch them in local clusters on AWS Outposts. For more information, see eks-outposts.title.
You can deploy to Amazon EC2 instances with x86 or Arm processors. However, you can’t deploy to instances that have Inferentia chips.
Bottlerocket is compatible with AWS CloudFormation. However, there is no official CloudFormation template that can be copied to deploy Bottlerocket nodes for Amazon EKS.
Bottlerocket images don’t come with an SSH server or a shell. You can use out-of-band access methods to allow SSH enabling the admin container and to pass some bootstrapping configuration steps with user data. For more information, see these sections in the bottlerocket README.md on GitHub:

This procedure requires eksctl version 0.199.0 or later. You can check your version with the following command:

eksctl version

For instructions on how to install or upgrade eksctl, see Installation in the eksctl documentation.NOTE: This procedure only works for clusters that were created with eksctl.

+ . Copy the following contents to your device. Replace my-cluster with the name of your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in. Replace ng-bottlerocket with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. To deploy on Arm instances, replace m5.large with an Arm instance type. Replace my-ec2-keypair-name with the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the consolelong. For more information, see Amazon EC2 key pairs in the Amazon EC2 User Guide. Replace all remaining example values with your own values. Once you’ve made the replacements, run the modified command to create the bottlerocket.yaml file.

+ If specifying an Arm Amazon EC2 instance type, then review the considerations in Amazon EKS optimized Arm Amazon Linux AMIs before deploying. For instructions on how to deploy using a custom AMI, see Building Bottlerocket on GitHub and Custom AMI support in the eksctl documentation. To deploy a managed node group, deploy a custom AMI using a launch template. For more information, see launch-templates.title.

+ IMPORTANT: To deploy a node group to AWS Outposts, AWS Wavelength, or AWS Local Zone subnets, don’t pass AWS Outposts, AWS Wavelength, or AWS Local Zone subnets when you create the cluster. You must specify the subnets in the following example. For more information see Create a nodegroup from a config file and Config file schema in the eksctl documentation. Replace region-code with the AWS Region that your cluster is in.

cat >bottlerocket.yaml <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my-cluster
  region: region-code
  version: '1.30'

iam:
  withOIDC: true

nodeGroups:
  - name: ng-bottlerocket
    instanceType: m5.large
    desiredCapacity: 3
    amiFamily: Bottlerocket
    ami: auto-ssm
    iam:
       attachPolicyARNs:
          - region.arniam::aws:policy/AmazonEKSWorkerNodePolicy
          - region.arniam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - region.arniam::aws:policy/AmazonSSMManagedInstanceCore
          - region.arniam::aws:policy/AmazonEKS_CNI_Policy
    ssh:
        allow: true
        publicKeyName: my-ec2-keypair-name
EOF

Deploy your nodes with the following command.
```
eksctl create nodegroup --config-file=bottlerocket.yaml
```
An example output is as follows.

Several lines are output while the nodes are created. One of the last lines of output is the following example line.
```
[✔]  created 1 nodegroup(s) in cluster "my-cluster"
```
(Optional) Create a Kubernetes persistent volume on a Bottlerocket node using the Amazon EBS CSI Plugin. The default Amazon EBS driver relies on file system tools that aren’t included with Bottlerocket. For more information about creating a storage class using the driver, see ebs-csi.title.
(Optional) By default, kube-proxy sets the nf_conntrack_max kernel parameter to a default value that may differ from what Bottlerocket originally sets at boot. To keep Bottlerocket’s default setting, edit the `kube-proxy configuration with the following command.
```
kubectl edit -n kube-system daemonset kube-proxy
```
Add --conntrack-max-per-core and --conntrack-min to the kube-proxy arguments that are in the following example. A setting of 0 implies no change.
```
      containers:
      - command:
        - kube-proxy
        - --v=2
        - --config=/var/lib/kube-proxy-config/config
        - --conntrack-max-per-core=0
        - --conntrack-min=0
```
(Optional) Deploy a sample application to test your Bottlerocket nodes.
We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see Restrict access to the instance profile assigned to the worker node.

9.3.3. Create self-managed `Microsoft Windows` nodes

This topic describes how to launch Auto Scaling groups of Windows nodes that register with your Amazon EKS cluster.

This topic describes how to launch Auto Scaling groups of Windows nodes that register with your Amazon EKS cluster. After the nodes join the cluster, you can deploy Kubernetes applications to them.

Amazon EKS nodes are standard Amazon EC2 instances, and you are billed for them based on normal Amazon EC2 instance prices. For more information, see Amazon EC2 pricing.
You can launch Windows nodes in Amazon EKS extended clusters on AWS Outposts, but you can’t launch them in local clusters on AWS Outposts. For more information, see eks-outposts.title.

Enable Windows support for your cluster. We recommend that you review important considerations before you launch a Windows node group. For more information, see enable-windows-support.title.

You can launch self-managed Windows nodes with either of the following:

eksctl
consolelong

`eksctl`

Launch self-managed Windows nodes using eksctl

This procedure requires that you have installed eksctl, and that your eksctl version is at least 0.199.0. You can check your version with the following command.

eksctl version

For instructions on how to install or upgrade eksctl, see Installation in the eksctl documentation.

This procedure only works for clusters that were created with eksctl.

(Optional) If the AmazonEKS_CNI_Policy managed IAM policy (if you have an IPv4 cluster) or the AmazonEKS_CNI_IPv6_Policy (that you created yourself if you have an IPv6 cluster) is attached to your Amazon EKS node IAM role, we recommend assigning it to an IAM role that you associate to the Kubernetes aws-node service account instead. For more information, see cni-iam-role.title.

This procedure assumes that you have an existing cluster. If you don’t already have an Amazon EKS cluster and an Amazon Linux node group to add a Windows node group to, we recommend that you follow getting-started-eksctl.title. This guide provides a complete walkthrough for how to create an Amazon EKS cluster with Amazon Linux nodes.

Create your node group with the following command. Replace region-code with the AWS Region that your cluster is in. Replace my-cluster with your cluster name. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in. Replace ng-windows with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. For Kubernetes version 1.24 or later, you can replace 2019 with 2022 to use Windows Server 2022. Replace the rest of the example values with your own values.

To deploy a node group to AWS Outposts, AWS Wavelength, or AWS Local Zone subnets, don’t pass the AWS Outposts, Wavelength, or Local Zone subnets when you create the cluster. Create the node group with a config file, specifying the AWS Outposts, Wavelength, or Local Zone subnets. For more information, see Create a nodegroup from a config file and Config file schema in the eksctl documentation.

eksctl create nodegroup \
    --region region-code \
    --cluster my-cluster \
    --name ng-windows \
    --node-type t2.large \
    --nodes 3 \
    --nodes-min 1 \
    --nodes-max 4 \
    --managed=false \
    --node-ami-family WindowsServer2019FullContainer

If nodes fail to join the cluster, see worker-node-fail.title in the Troubleshooting guide.
To see the available options for eksctl commands, enter the following command.
```
eksctl command -help
```

An example output is as follows. Several lines are output while the nodes are created. One of the last lines of output is the following example line.

[✔]  created 1 nodegroup(s) in cluster "my-cluster"

(Optional) Deploy a sample application to test your cluster and Windows nodes.
We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see Restrict access to the instance profile assigned to the worker node.

`consolelong`

Prerequisites

An existing Amazon EKS cluster and a Linux node group. If you don’t have these resources, we recommend that you create them using one of our guides in getting-started.title. These guides describe how to create an Amazon EKS cluster with Linux nodes.
An existing VPC and security group that meet the requirements for an Amazon EKS cluster. For more information, see network-reqs.title and sec-group-reqs.title. The guides in getting-started.title create a VPC that meets the requirements. Alternatively, you can also follow Create an Amazon VPC for your Amazon EKS cluster to create one manually.
An existing Amazon EKS cluster that uses a VPC and security group that meets the requirements of an Amazon EKS cluster. For more information, see create-cluster.title. If you have subnets in the AWS Region where you have AWS Outposts, AWS Wavelength, or AWS Local Zones enabled, those subnets must not have been passed in when you created the cluster.

Step 1: Launch self-managed Windows nodes using the consolelong

Wait for your cluster status to show as ACTIVE. If you launch your nodes before the cluster is active, the nodes fail to register with the cluster and you need to relaunch them.
Open the AWS CloudFormation console
Choose Create stack.
For Specify template, select Amazon S3 URL.

Copy the following URL and paste it into Amazon S3 URL.

https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2023-02-09/amazon-eks-windows-nodegroup.yaml

Select Next twice.

On the Quick create stack page, enter the following parameters accordingly:

Stack name: Choose a stack name for your AWS CloudFormation stack. For example, you can call it my-cluster-nodes````.
ClusterName: Enter the name that you used when you created your Amazon EKS cluster.

This name must exactly match the name that you used in Step 1: Create your Amazon EKS cluster. Otherwise, your nodes can’t join the cluster.
ClusterControlPlaneSecurityGroup: Choose the security group from the AWS CloudFormation output that you generated when you created your VPC. The following steps show one method to retrieve the applicable group.
1. Open the Amazon EKS console.
2. Choose the name of the cluster.
3. Choose the Networking tab.
4. Use the Additional security groups value as a reference when selecting from the ClusterControlPlaneSecurityGroup dropdown list.
NodeGroupName: Enter a name for your node group. This name can be used later to identify the Auto Scaling node group that’s created for your nodes. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
NodeAutoScalingGroupMinSize: Enter the minimum number of nodes that your node Auto Scaling group can scale in to.
NodeAutoScalingGroupDesiredCapacity: Enter the desired number of nodes to scale to when your stack is created.
NodeAutoScalingGroupMaxSize: Enter the maximum number of nodes that your node Auto Scaling group can scale out to.

NodeInstanceType: Choose an instance type for your nodes. For more information, see choosing-instance-type.title.

The supported instance types for the latest version of the Amazon VPC CNI plugin for Kubernetes are listed in vpc_ip_resource_limit.go on GitHub. You might need to update your CNI version to use the latest supported instance types. For more information, see managing-vpc-cni.title.

NodeImageIdSSMParam: Pre-populated with the Amazon EC2 Systems Manager parameter of the current recommended Amazon EKS optimized Windows Core AMI ID. To use the full version of Windows, replace Core with Full.
NodeImageId: (Optional) If you’re using your own custom AMI (instead of an Amazon EKS optimized AMI), enter a node AMI ID for your AWS Region. If you specify a value for this field, it overrides any values in the NodeImageIdSSMParam field.
NodeVolumeSize: Specify a root volume size for your nodes, in GiB.
KeyName: Enter the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the consolelong. For more information, see Amazon EC2 key pairs in the Amazon EC2 User Guide.

If you don’t provide a key pair here, the AWS CloudFormation stack fails to be created.
BootstrapArguments: Specify any optional arguments to pass to the node bootstrap script, such as extra kubelet arguments using -KubeletExtraArgs.
DisableIMDSv1: By default, each node supports the Instance Metadata Service Version 1 (IMDSv1) and IMDSv2. You can disable IMDSv1. To prevent future nodes and Pods in the node group from using MDSv1, set DisableIMDSv1 to true. For more information about IMDS, see Configuring the instance metadata service.
VpcId: Select the ID for the VPC that you created.
NodeSecurityGroups: Select the security group that was created for your Linux node group when you created your VPC. If your Linux nodes have more than one security group attached to them, specify all of them. This for, for example, if the Linux node group was created with eksctl.

Subnets: Choose the subnets that you created. If you created your VPC using the steps in Create an Amazon VPC for your Amazon EKS cluster, then specify only the private subnets within the VPC for your nodes to launch into.

If any of the subnets are public subnets, then they must have the automatic public IP address assignment setting enabled. If the setting isn’t enabled for the public subnet, then any nodes that you deploy to that public subnet won’t be assigned a public IP address and won’t be able to communicate with the cluster or other AWS services. If the subnet was deployed before March 26, 2020 using either of the Amazon EKS AWS CloudFormation VPC templates, or by using eksctl, then automatic public IP address assignment is disabled for public subnets. For information about how to enable public IP address assignment for a subnet, see Modifying the public IPv4 addressing attribute for your subnet. If the node is deployed to a private subnet, then it’s able to communicate with the cluster and other AWS services through a NAT gateway.
If the subnets don’t have internet access, then make sure that you’re aware of the considerations and extra steps in Deploy private clusters with limited internet access.
If you select AWS Outposts, Wavelength, or Local Zone subnets, then the subnets must not have been passed in when you created the cluster.

Acknowledge that the stack might create IAM resources, and then choose Create stack.
When your stack has finished creating, select it in the console and choose Outputs.
Record the NodeInstanceRole for the node group that was created. You need this when you configure your Amazon EKS Windows nodes.

Step 2: Enable nodes to join your cluster

Check to see if you already have an aws-auth ConfigMap.
```
kubectl describe configmap -n kube-system aws-auth
```

If you are shown an aws-auth ConfigMap, then update it as needed.

Open the ConfigMap for editing.

kubectl edit -n kube-system configmap/aws-auth

Add new mapRoles entries as needed. Set the rolearn values to the NodeInstanceRole values that you recorded in the previous procedures.

[...]
data:
  mapRoles: |
- rolearn: <ARN of linux instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: <ARN of windows instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
        - eks:kube-proxy-windows
[...]

Save the file and exit your text editor.

If you received an error stating "Error from server (NotFound): configmaps "aws-auth" not found, then apply the stock ConfigMap.
1. Download the configuration map.
  curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/aws-auth-cm-windows.yaml
2. In the aws-auth-cm-windows.yaml file, set the rolearn values to the applicable NodeInstanceRole values that you recorded in the previous procedures. You can do this with a text editor, or by replacing the example values and running the following command:
  sed -i.bak -e 's|<ARN of linux instance role (not instance profile)>|my-node-linux-instance-role|' \ -e 's|<ARN of windows instance role (not instance profile)>|my-node-windows-instance-role|' aws-auth-cm-windows.yaml
  Don’t modify any other lines in this file.
  
  Don’t use the same IAM role for both Windows and Linux nodes.
3. Apply the configuration. This command might take a few minutes to finish.
  kubectl apply -f aws-auth-cm-windows.yaml
Watch the status of your nodes and wait for them to reach the Ready status.
```
kubectl get nodes --watch
```
Enter Ctrl+C to return to a shell prompt.

If you receive any authorization or resource type errors, see unauthorized.title in the troubleshooting topic.

If nodes fail to join the cluster, then see worker-node-fail.title in the Troubleshooting chapter.

Step 3: Additional actions

(Optional) Deploy a sample application to test your cluster and Windows nodes.
(Optional) If the AmazonEKS_CNI_Policy managed IAM policy (if you have an IPv4 cluster) or the AmazonEKS_CNI_IPv6_Policy (that you created yourself if you have an IPv6 cluster) is attached to your Amazon EKS node IAM role, we recommend assigning it to an IAM role that you associate to the Kubernetes aws-node service account instead. For more information, see cni-iam-role.title.
We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see Restrict access to the instance profile assigned to the worker node.

9.3.4. Create self-managed `Ubuntu Linux` nodes

This topic describes how to launch Auto Scaling groups of Ubuntu nodes that register with your Amazon EKS cluster

Managed node groups might offer some advantages for your use case. For more information, see managed-node-groups.title.

This topic describes how to launch Auto Scaling groups of Ubuntu on Amazon Elastic Kubernetes Service (EKS) or Ubuntu Pro on Amazon Elastic Kubernetes Service (EKS) nodes that register with your Amazon EKS cluster. Ubuntu and Ubuntu Pro for EKS are based on the official Ubuntu Minimal LTS, include the custom AWS kernel that is jointly developed with AWS, and have been built specifically for EKS. Ubuntu Pro adds additional security coverage by supporting EKS extended support periods, kernel livepatch, FIPS compliance and the ability to run unlimited Pro containers.

After the nodes join the cluster, you can deploy containerized applications to them. For more information, visit the documentation for Ubuntu on AWS and Custom AMI support in the eksctl documentation.

Amazon EKS nodes are standard Amazon EC2 instances, and you are billed for them based on normal Amazon EC2 instance prices. For more information, see Amazon EC2 pricing.
You can launch Ubuntu nodes in Amazon EKS extended clusters on AWS Outposts, but you can’t launch them in local clusters on AWS Outposts. For more information, see eks-outposts.title.
You can deploy to Amazon EC2 instances with x86 or Arm processors. However, instances that have Inferentia chips might need to install the Neuron SDK first.

This procedure requires eksctl version 0.199.0 or later. You can check your version with the following command:

eksctl version

For instructions on how to install or upgrade eksctl, see Installation in the eksctl documentation.NOTE: This procedure only works for clusters that were created with eksctl.

+ . Copy the following contents to your device. Replace my-cluster with the name of your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphabetic character and can’t be longer than 100 characters. Replace ng-ubuntu with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. To deploy on Arm instances, replace m5.large with an Arm instance type. Replace my-ec2-keypair-name with the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the consolelong. For more information, see Amazon EC2 key pairs in the Amazon EC2 User Guide. Replace all remaining example values with your own values. Once you’ve made the replacements, run the modified command to create the ubuntu.yaml file.

cat >ubuntu.yaml <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my-cluster
  region: region-code
  version: '1.30'

iam:
  withOIDC: true

nodeGroups:
  - name: ng-ubuntu
    instanceType: m5.large
    desiredCapacity: 3
    amiFamily: Ubuntu2204
    iam:
       attachPolicyARNs:
          - region.arniam::aws:policy/AmazonEKSWorkerNodePolicy
          - region.arniam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - region.arniam::aws:policy/AmazonSSMManagedInstanceCore
          - region.arniam::aws:policy/AmazonEKS_CNI_Policy
    ssh:
        allow: true
        publicKeyName: my-ec2-keypair-name
EOF

+ To create an Ubuntu Pro node group, just change the amiFamily value to UbuntuPro2204. . Deploy your nodes with the following command.

eksctl create nodegroup --config-file=ubuntu.yaml

+ An example output is as follows.

+ Several lines are output while the nodes are created. One of the last lines of output is the following example line.

[✔]  created 1 nodegroup(s) in cluster "my-cluster"

(Optional) Deploy a sample application to test your Ubuntu nodes.
We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see Restrict access to the instance profile assigned to the worker node.

9.3.5. Update self-managed nodes for your cluster

When a new Amazon EKS optimized AMI is released, consider replacing the nodes in your self-managed node group with the new AMI.

When a new Amazon EKS optimized AMI is released, consider replacing the nodes in your self-managed node group with the new AMI. Likewise, if you have updated the Kubernetes version for your Amazon EKS cluster, update the nodes to use nodes with the same Kubernetes version.

This topic covers node updates for self-managed nodes. If you are using Simplify node lifecycle with managed node groups, see update-managed-node-group.title.

There are two basic ways to update self-managed node groups in your clusters to use a new AMI:

Migrate applications to a new node group: Create a new node group and migrate your Pods to that group. Migrating to a new node group is more graceful than simply updating the AMI ID in an existing AWS CloudFormation stack. This is because the migration process taints the old node group as NoSchedule and drains the nodes after a new stack is ready to accept the existing Pod workload.
Update an AWS CloudFormation node stack: Update the AWS CloudFormation stack for an existing node group to use the new AMI. This method isn’t supported for node groups that were created with eksctl.

Migrate applications to a new node group

This topic describes how you can create a new node group, gracefully migrate your existing applications to the new group, and remove the old node group from your cluster.

This topic describes how you can create a new node group, gracefully migrate your existing applications to the new group, and remove the old node group from your cluster. You can migrate to a new node group using eksctl or the consolelong.

eksctl
consolelong and AWS CLI

`eksctl`

Migrate your applications to a new node group with eksctl

For more information on using eksctl for migration, see Unmanaged nodegroups in the eksctl documentation.

This procedure requires eksctl version 0.199.0 or later. You can check your version with the following command:

eksctl version

For instructions on how to install or upgrade eksctl, see Installation in the eksctl documentation.

This procedure only works for clusters and node groups that were created with eksctl.

Retrieve the name of your existing node groups, replacing my-cluster with your cluster name.

eksctl get nodegroups --cluster=my-cluster

An example output is as follows.

CLUSTER      NODEGROUP          CREATED               MIN SIZE      MAX SIZE     DESIRED CAPACITY     INSTANCE TYPE     IMAGE ID
default      standard-nodes   2019-05-01T22:26:58Z  1             4            3                    t3.medium         ami-05a71d034119ffc12

Launch a new node group with eksctl with the following command. In the command, replace every example value with your own values. The version number can’t be later than the Kubernetes version for your control plane. Also, it can’t be more than two minor versions earlier than the Kubernetes version for your control plane. We recommend that you use the same version as your control plane.

We recommend blocking Pod access to IMDS if the following conditions are true:
- You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
- No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
  
  For more information, see Restrict access to the instance profile assigned to the worker node.
  
  To block Pod access to IMDS, add the --disable-pod-imds option to the following command.
  
  For more available flags and their descriptions, see https://eksctl.io/.
```
eksctl create nodegroup \
  --cluster my-cluster \
  --version 1.30 \
  --name standard-nodes-new \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4 \
  --managed=false
```
When the previous command completes, verify that all of your nodes have reached the Ready state with the following command:
```
kubectl get nodes
```
Delete the original node group with the following command. In the command, replace every example value with your cluster and node group names:
```
eksctl delete nodegroup --cluster my-cluster --name standard-nodes-old
```

`consolelong` and `AWS` CLI

Migrate your applications to a new node group with the consolelong and AWS CLI

Launch a new node group by following the steps that are outlined in Create self-managed Amazon Linux nodes.
When your stack has finished creating, select it in the console and choose Outputs.

Record the NodeInstanceRole for the node group that was created. You need this to add the new Amazon EKS nodes to your cluster.

If you attached any additional IAM policies to your old node group IAM role, attach those same policies to your new node group IAM role to maintain that functionality on the new group. This applies to you if you added permissions for the Kubernetes Cluster Autoscaler, for example.

Update the security groups for both node groups so that they can communicate with each other. For more information, see sec-group-reqs.title.
1. Record the security group IDs for both node groups. This is shown as the NodeSecurityGroup value in the AWS CloudFormation stack outputs.
  
  You can use the following AWS CLI commands to get the security group IDs from the stack names. In these commands, oldNodes is the AWS CloudFormation stack name for your older node stack, and newNodes is the name of the stack that you are migrating to. Replace every example value with your own values.
  oldNodes="old_node_CFN_stack_name" newNodes="new_node_CFN_stack_name" oldSecGroup=$(aws cloudformation describe-stack-resources --stack-name $oldNodes \ --query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \ --output text) newSecGroup=$(aws cloudformation describe-stack-resources --stack-name $newNodes \ --query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \ --output text)
2. Add ingress rules to each node security group so that they accept traffic from each other.
  
  The following AWS CLI commands add inbound rules to each security group that allow all traffic on all protocols from the other security group. This configuration allows Pods in each node group to communicate with each other while you’re migrating your workload to the new group.
  aws ec2 authorize-security-group-ingress --group-id $oldSecGroup \ --source-group $newSecGroup --protocol -1 aws ec2 authorize-security-group-ingress --group-id $newSecGroup \ --source-group $oldSecGroup --protocol -1

Edit the aws-auth configmap to map the new node instance role in RBAC.

kubectl edit configmap -n kube-system aws-auth

Add a new mapRoles entry for the new node group. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.

apiVersion: v1
data:
  mapRoles: |
    - rolearn: ARN of instance role (not instance profile)
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes>
    - rolearn: region.arniam::111122223333:role/nodes-1-16-NodeInstanceRole-U11V27W93CX5
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

Replace the ARN of instance role (not instance profile) snippet with the NodeInstanceRole value that you recorded in a previous step. Then, save and close the file to apply the updated configmap.

Watch the status of your nodes and wait for your new nodes to join your cluster and reach the Ready status.
```
kubectl get nodes --watch
```
(Optional) If you’re using the Kubernetes Cluster Autoscaler, scale the deployment down to zero (0) replicas to avoid conflicting scaling actions.
```
kubectl scale deployments/cluster-autoscaler --replicas=0 -n kube-system
```
Use the following command to taint each of the nodes that you want to remove with NoSchedule. This is so that new Pods aren’t scheduled or rescheduled on the nodes that you’re replacing. For more information, see Taints and Tolerations in the Kubernetes documentation.
```
kubectl taint nodes node_name key=value:NoSchedule
```
If you’re upgrading your nodes to a new Kubernetes version, you can identify and taint all of the nodes of a particular Kubernetes version (in this case, 1.28) with the following code snippet. The version number can’t be later than the Kubernetes version of your control plane. It also can’t be more than two minor versions earlier than the Kubernetes version of your control plane. We recommend that you use the same version as your control plane.
```
K8S_VERSION=1.28
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
    echo "Tainting $node"
    kubectl taint nodes $node key=value:NoSchedule
done
```

Determine your cluster’s DNS provider.

kubectl get deployments -l k8s-app=kube-dns -n kube-system

An example output is as follows. This cluster is using CoreDNS for DNS resolution, but your cluster can return kube-dns instead):

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
coredns   1         1         1            1           31m

If your current deployment is running fewer than two replicas, scale out the deployment to two replicas. Replace coredns with kubedns if your previous command output returned that instead.
```
kubectl scale deployments/coredns --replicas=2 -n kube-system
```

Drain each of the nodes that you want to remove from your cluster with the following command:

kubectl drain node_name --ignore-daemonsets --delete-local-data

If you’re upgrading your nodes to a new Kubernetes version, identify and drain all of the nodes of a particular Kubernetes version (in this case, 1.28) with the following code snippet.

K8S_VERSION=1.28
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
    echo "Draining $node"
    kubectl drain $node --ignore-daemonsets --delete-local-data
done

After your old nodes finished draining, revoke the security group inbound rules you authorized earlier. Then, delete the AWS CloudFormation stack to terminate the instances.

If you attached any additional IAM policies to your old node group IAM role, such as adding permissions for the Kubernetes Cluster Autoscaler, detach those additional policies from the role before you can delete your AWS CloudFormation stack.

Revoke the inbound rules that you created for your node security groups earlier. In these commands, oldNodes is the AWS CloudFormation stack name for your older node stack, and newNodes is the name of the stack that you are migrating to.

oldNodes="old_node_CFN_stack_name"
newNodes="new_node_CFN_stack_name"

oldSecGroup=$(aws cloudformation describe-stack-resources --stack-name $oldNodes \
--query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \
--output text)
newSecGroup=$(aws cloudformation describe-stack-resources --stack-name $newNodes \
--query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \
--output text)
aws ec2 revoke-security-group-ingress --group-id $oldSecGroup \
--source-group $newSecGroup --protocol -1
aws ec2 revoke-security-group-ingress --group-id $newSecGroup \
--source-group $oldSecGroup --protocol -1

Open the AWS CloudFormation console.
Select your old node stack.
Choose Delete.
In the Delete stack confirmation dialog box, choose Delete stack.

Edit the aws-auth configmap to remove the old node instance role from RBAC.

kubectl edit configmap -n kube-system aws-auth

Delete the mapRoles entry for the old node group. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.

apiVersion: v1
data:
  mapRoles: |
    - rolearn: region.arniam::111122223333:role/nodes-1-16-NodeInstanceRole-W70725MZQFF8
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: region.arniam::111122223333:role/nodes-1-15-NodeInstanceRole-U11V27W93CX5
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes>

Save and close the file to apply the updated configmap.

(Optional) If you are using the Kubernetes Cluster Autoscaler, scale the deployment back to one replica.

You must also tag your new Auto Scaling group appropriately (for example, k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster) and update the command for your Cluster Autoscaler deployment to point to the newly tagged Auto Scaling group. For more information, see Cluster Autoscaler on AWS.

kubectl scale deployments/cluster-autoscaler --replicas=1 -n kube-system

(Optional) Verify that you’re using the latest version of the Amazon VPC CNI plugin for Kubernetes. You might need to update your CNI version to use the latest supported instance types. For more information, see managing-vpc-cni.title.
If your cluster is using kube-dns for DNS resolution (see [migrate-determine-dns-step]), scale in the kube-dns deployment to one replica.
```
kubectl scale deployments/kube-dns --replicas=1 -n kube-system
```

Update an `AWS` CloudFormation node stack

This topic describes how you can update an existing AWS CloudFormation self-managed node stack with a new AMI.

This topic describes how you can update an existing AWS CloudFormation self-managed node stack with a new AMI. You can use this procedure to update your nodes to a new version of Kubernetes following a cluster update. Otherwise, you can update to the latest Amazon EKS optimized AMI for an existing Kubernetes version.

This topic covers node updates for self-managed nodes. For information about using Simplify node lifecycle with managed node groups, see update-managed-node-group.title.

The latest default Amazon EKS node AWS CloudFormation template is configured to launch an instance with the new AMI into your cluster before removing an old one, one at a time. This configuration ensures that you always have your Auto Scaling group’s desired count of active instances in your cluster during the rolling update.

This method isn’t supported for node groups that were created with eksctl. If you created your cluster or node group with eksctl, see migrate-stack.title.

Determine the DNS provider for your cluster.
```
kubectl get deployments -l k8s-app=kube-dns -n kube-system
```
An example output is as follows. This cluster is using CoreDNS for DNS resolution, but your cluster might return kube-dns instead. Your output might look different depending on the version of kubectl that you’re using.
```
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
coredns   1         1         1            1           31m
```
If your current deployment is running fewer than two replicas, scale out the deployment to two replicas. Replace coredns with kube-dns if your previous command output returned that instead.
```
kubectl scale deployments/coredns --replicas=2 -n kube-system
```
(Optional) If you’re using the Kubernetes Cluster Autoscaler, scale the deployment down to zero (0) replicas to avoid conflicting scaling actions.
```
kubectl scale deployments/cluster-autoscaler --replicas=0 -n kube-system
```
Determine the instance type and desired instance count of your current node group. You enter these values later when you update the AWS CloudFormation template for the group.
1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the left navigation pane, choose Launch Configurations, and note the instance type for your existing node launch configuration.
3. In the left navigation pane, choose Auto Scaling Groups, and note the Desired instance count for your existing node Auto Scaling group.
Open the AWS CloudFormation console.
Select your node group stack, and then choose Update.
Select Replace current template and select Amazon S3 URL.
For Amazon S3 URL, paste the following URL into the text area to ensure that you’re using the latest version of the node AWS CloudFormation template. Then, choose Next:
```
https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2022-12-23/amazon-eks-nodegroup.yaml
```

On the Specify stack details page, fill out the following parameters, and choose Next:

NodeAutoScalingGroupDesiredCapacity – Enter the desired instance count that you recorded in a previous step. Or, enter your new desired number of nodes to scale to when your stack is updated.
NodeAutoScalingGroupMaxSize – Enter the maximum number of nodes to which your node Auto Scaling group can scale out. This value must be at least one node more than your desired capacity. This is so that you can perform a rolling update of your nodes without reducing your node count during the update.

NodeInstanceType – Choose the instance type your recorded in a previous step. Alternatively, choose a different instance type for your nodes. Before choosing a different instance type, review Choose an optimal Amazon EC2 node instance type. Each Amazon EC2 instance type supports a maximum number of elastic network interfaces (network interface) and each network interface supports a maximum number of IP addresses. Because each worker node and Pod ,is assigned its own IP address, it’s important to choose an instance type that will support the maximum number of Pods that you want to run on each Amazon EC2 node. For a list of the number of network interfaces and IP addresses supported by instance types, see IP addresses per network interface per instance type. For example, the m5.large instance type supports a maximum of 30 IP addresses for the worker node and Pods.

The supported instance types for the latest version of the Amazon VPC CNI plugin for Kubernetes are shown in vpc_ip_resource_limit.go on GitHub. You might need to update your Amazon VPC CNI plugin for Kubernetes version to use the latest supported instance types. For more information, see managing-vpc-cni.title.

Some instance types might not be available in all AWS Regions.

NodeImageIdSSMParam – The Amazon EC2 Systems Manager parameter of the AMI ID that you want to update to. The following value uses the latest Amazon EKS optimized AMI for Kubernetes version 1.30.

/aws/service/eks/optimized-ami/1.30/amazon-linux-2/recommended/image_id

You can replace 1.30 with a supported Kubernetes version that’s the same. Or, it should be up to one version earlier than the Kubernetes version running on your control plane. We recommend that you keep your nodes at the same version as your control plane. You can also replace amazon-linux-2 with a different AMI type. For more information, see retrieve-ami-id.title.

Using the Amazon EC2 Systems Manager parameter enables you to update your nodes in the future without having to look up and specify an AMI ID. If your AWS CloudFormation stack is using this value, any stack update always launches the latest recommended Amazon EKS optimized AMI for your specified Kubernetes version. This is even the case even if you don’t change any values in the template.

NodeImageId – To use your own custom AMI, enter the ID for the AMI to use.

This value overrides any value specified for NodeImageIdSSMParam. If you want to use the NodeImageIdSSMParam value, ensure that the value for NodeImageId is blank.
DisableIMDSv1 – By default, each node supports the Instance Metadata Service Version 1 (IMDSv1) and IMDSv2. However, you can disable IMDSv1. Select true if you don’t want any nodes or any Pods scheduled in the node group to use IMDSv1. For more information about IMDS, see Configuring the instance metadata service. If you’ve implemented IAM roles for service accounts, assign necessary permissions directly to all Pods that require access to AWS services. This way, no Pods in your cluster require access to IMDS for other reasons, such as retrieving the current AWS Region. Then, you can also disable access to IMDSv2 for Pods that don’t use host networking. For more information, see Restrict access to the instance profile assigned to the worker node.

(Optional) On the Options page, tag your stack resources. Choose Next.
On the Review page, review your information, acknowledge that the stack might create IAM resources, and then choose Update stack.

The update of each node in the cluster takes several minutes. Wait for the update of all nodes to complete before performing the next steps.
If your cluster’s DNS provider is kube-dns, scale in the kube-dns deployment to one replica.
```
kubectl scale deployments/kube-dns --replicas=1 -n kube-system
```
(Optional) If you are using the Kubernetes Cluster Autoscaler, scale the deployment back to your desired amount of replicas.
```
kubectl scale deployments/cluster-autoscaler --replicas=1 -n kube-system
```
(Optional) Verify that you’re using the latest version of the Amazon VPC CNI plugin for Kubernetes. You might need to update your Amazon VPC CNI plugin for Kubernetes version to use the latest supported instance types. For more information, see managing-vpc-cni.title.

A cluster contains one or more Amazon EC2 nodes that Pods are scheduled on.

A cluster contains one or more Amazon EC2 nodes that Pods are scheduled on. Amazon EKS nodes run in your AWS account and connect to the control plane of your cluster through the cluster API server endpoint. You’re billed for them based on Amazon EC2 prices. For more information, see Amazon EC2 pricing.

A cluster can contain several node groups. Each node group contains one or more nodes that are deployed in an Amazon EC2 Auto Scaling group. The instance type of the nodes within the group can vary, such as when using attribute-based instance type selection with Karpenter. All instances in a node group must use the Amazon EKS node IAM role.

Amazon EKS provides specialized Amazon Machine Images (AMIs) that are called Amazon EKS optimized AMIs. The AMIs are configured to work with Amazon EKS. Their components include containerd, kubelet, and the AWS IAM Authenticator. The AMIs also contain a specialized bootstrap script that allows it to discover and connect to your cluster’s control plane automatically.

If you restrict access to the public endpoint of your cluster using CIDR blocks, we recommend that you also enable private endpoint access. This is so that nodes can communicate with the cluster. Without the private endpoint enabled, the CIDR blocks that you specify for public access must include the egress sources from your VPC. For more information, see cluster-endpoint.title.

To add self-managed nodes to your Amazon EKS cluster, see the topics that follow. If you launch self-managed nodes manually, add the following tag to each node. For more information, see Adding and deleting tags on an individual resource. If you follow the steps in the guides that follow, the required tag is automatically added to nodes for you.

[[Topic List]]

9.4. Simplify compute management with `AWS` Fargate

This topic discusses using Amazon EKS to run Kubernetes Pods on AWS Fargate.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West).

This topic discusses using Amazon EKS to run Kubernetes Pods on AWS Fargate. Fargate is a technology that provides on-demand, right-sized compute capacity for containers. With Fargate, you don’t have to provision, configure, or scale groups of virtual machines on your own to run containers. You also don’t need to choose server types, decide when to scale your node groups, or optimize cluster packing.

You can control which Pods start on Fargate and how they run with Fargate profiles. Fargate profiles are defined as part of your Amazon EKS cluster. Amazon EKS integrates Kubernetes with Fargate by using controllers that are built by AWS using the upstream, extensible model provided by Kubernetes. These controllers run as part of the Amazon EKS managed Kubernetes control plane and are responsible for scheduling native Kubernetes Pods onto Fargate. The Fargate controllers include a new scheduler that runs alongside the default Kubernetes scheduler in addition to several mutating and validating admission controllers. When you start a Pod that meets the criteria for running on Fargate, the Fargate controllers that are running in the cluster recognize, update, and schedule the Pod onto Fargate.

This topic describes the different components of Pods that run on Fargate, and calls out special considerations for using Fargate with Amazon EKS.

9.4.1. `AWS` Fargate considerations

Here are some things to consider about using Fargate on Amazon EKS.

Each Pod that runs on Fargate has its own isolation boundary. They don’t share the underlying kernel, CPU resources, memory resources, or elastic network interface with another Pod.
Network Load Balancers and Application Load Balancers (ALBs) can be used with Fargate with IP targets only. For more information, see network-load-balancer.title and alb-ingress.title.
Fargate exposed services only run on target type IP mode, and not on node IP mode. The recommended way to check the connectivity from a service running on a managed node and a service running on Fargate is to connect via service name.
Pods must match a Fargate profile at the time that they’re scheduled to run on Fargate. Pods that don’t match a Fargate profile might be stuck as Pending. If a matching Fargate profile exists, you can delete pending Pods that you have created to reschedule them onto Fargate.
Daemonsets aren’t supported on Fargate. If your application requires a daemon, reconfigure that daemon to run as a sidecar container in your Pods.
Privileged containers aren’t supported on Fargate.
Pods running on Fargate can’t specify HostPort or HostNetwork in the Pod manifest.
The default nofile and nproc soft limit is 1024 and the hard limit is 65535 for Fargate Pods.
GPUs aren’t currently available on Fargate.
Pods that run on Fargate are only supported on private subnets (with NAT gateway access to AWS services, but not a direct route to an Internet Gateway), so your cluster’s VPC must have private subnets available. For clusters without outbound internet access, see private-clusters.title.
You can use the Adjust pod resources with Vertical Pod Autoscaler to set the initial correct size of CPU and memory for your Fargate Pods, and then use the Scale pod deployments with Horizontal Pod Autoscaler to scale those Pods. If you want the Vertical Pod Autoscaler to automatically re-deploy Pods to Fargate with larger CPU and memory combinations, set the mode for the Vertical Pod Autoscaler to either Auto or Recreate to ensure correct functionality. For more information, see the Vertical Pod Autoscaler documentation on GitHub.
DNS resolution and DNS hostnames must be enabled for your VPC. For more information, see Viewing and updating DNS support for your VPC.
Amazon EKS Fargate adds defense-in-depth for Kubernetes applications by isolating each Pod within a Virtual Machine (VM). This VM boundary prevents access to host-based resources used by other Pods in the event of a container escape, which is a common method of attacking containerized applications and gain access to resources outside of the container.

Using Amazon EKS doesn’t change your responsibilities under the shared responsibility model. You should carefully consider the configuration of cluster security and governance controls. The safest way to isolate an application is always to run it in a separate cluster.
Fargate profiles support specifying subnets from VPC secondary CIDR blocks. You might want to specify a secondary CIDR block. This is because there’s a limited number of IP addresses available in a subnet. As a result, there’s also a limited number of Pods that can be created in the cluster. By using different subnets for Pods, you can increase the number of available IP addresses. For more information, see Adding IPv4 CIDR blocks to a VPC.
The Amazon EC2 instance metadata service (IMDS) isn’t available to Pods that are deployed to Fargate nodes. If you have Pods that are deployed to Fargate that need IAM credentials, assign them to your Pods using IAM roles for service accounts. If your Pods need access to other information available through IMDS, then you must hard code this information into your Pod spec. This includes the AWS Region or Availability Zone that a Pod is deployed to.
You can’t deploy Fargate Pods to AWS Outposts, AWS Wavelength, or AWS Local Zones.
Amazon EKS must periodically patch Fargate Pods to keep them secure. We attempt the updates in a way that reduces impact, but there are times when Pods must be deleted if they aren’t successfully evicted. There are some actions you can take to minimize disruption. For more information, see fargate-pod-patching.title.
The Amazon VPC CNI plugin for Amazon EKS is installed on Fargate nodes. You can’t use Alternate CNI plugins for Amazon EKS clusters with Fargate nodes.
A Pod running on Fargate automatically mounts an Amazon EFS file system, without needing manual driver installation steps. You can’t use dynamic persistent volume provisioning with Fargate nodes, but you can use static provisioning.
Amazon EKS doesn’t support Fargate Spot.
You can’t mount Amazon EBS volumes to Fargate Pods.
You can run the Amazon EBS CSI controller on Fargate nodes, but the Amazon EBS CSI node DaemonSet can only run on Amazon EC2 instances.
After a Kubernetes Job is marked Completed or Failed, the Pods that the Job creates normally continue to exist. This behavior allows you to view your logs and results, but with Fargate you will incur costs if you don’t clean up the Job afterwards.

To automatically delete the related Pods after a Job completes or fails, you can specify a time period using the time-to-live (TTL) controller. The following example shows specifying .spec.ttlSecondsAfterFinished in your Job manifest.
```
apiVersion: batch/v1
kind: Job
metadata:
  name: busybox
spec:
  template:
    spec:
      containers:
      - name: busybox
        image: busybox
        command: ["/bin/sh", "-c", "sleep 10"]
      restartPolicy: Never
  ttlSecondsAfterFinished: 60 # <-- TTL controller
```

9.4.2. Fargate Comparison Table

Criteria AWS Fargate

Can be deployed to AWS Outposts

Can be deployed to an AWS Local Zone

Can run containers that require Windows

Can run containers that require Linux

Yes

Can run workloads that require the Inferentia chip

Can run workloads that require a GPU

Can run workloads that require Arm processors

Can run AWS Bottlerocket

Pods share a kernel runtime environment with other Pods

No – Each Pod has a dedicated kernel

Pods share CPU, memory, storage, and network resources with other Pods.

No – Each Pod has dedicated resources and can be sized independently to maximize resource utilization.

Pods can use more hardware and memory than requested in Pod specs

No – The Pod can be re-deployed using a larger vCPU and memory configuration though.

Must deploy and manage Amazon EC2 instances

Must secure, maintain, and patch the operating system of Amazon EC2 instances

Can provide bootstrap arguments at deployment of a node, such as extra kubelet arguments.

Can assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node.

Can SSH into node

No – There’s no node host operating system to SSH to.

Can deploy your own custom AMI to nodes

Can deploy your own custom CNI to nodes

Must update node AMI on your own

Must update node Kubernetes version on your own

No – You don’t manage nodes.

Can use Amazon EBS storage with Pods

Can use Amazon EFS storage with Pods

Some Amazon EKS supported regions

Can use Amazon FSx for Lustre storage with Pods

Can use Network Load Balancer for services

Yes, when using the Create a network load balancer

Pods can run in a public subnet

Can assign different VPC security groups to individual Pods

Yes

Can run Kubernetes DaemonSets

Support HostPort and HostNetwork in the Pod manifest

AWS Region availability

Can run containers on Amazon EC2 dedicated hosts

Pricing

Cost of an individual Fargate memory and CPU configuration. Each Pod has its own cost. For more information, see AWS Fargate pricing.

9.4.3. Get started with `AWS` Fargate for your cluster

This topic describes how to get started running Pods on AWS Fargate with your Amazon EKS cluster.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West).

This topic describes how to get started running Pods on AWS Fargate with your Amazon EKS cluster.

If you restrict access to the public endpoint of your cluster using CIDR blocks, we recommend that you also enable private endpoint access. This way, Fargate Pods can communicate with the cluster. Without the private endpoint enabled, the CIDR blocks that you specify for public access must include the outbound sources from your VPC. For more information, see cluster-endpoint.title.

Prerequisite

An existing cluster. If you don’t already have an Amazon EKS cluster, see getting-started.title.

Step 1: Ensure that existing nodes can communicate with Fargate `Pods`

If you’re working with a new cluster with no nodes, or a cluster with only managed node groups (see managed-node-groups.title), you can skip to fargate-sg-pod-execution-role.title.

Assume that you’re working with an existing cluster that already has nodes that are associated with it. Make sure that Pods on these nodes can communicate freely with the Pods that are running on Fargate. Pods that are running on Fargate are automatically configured to use the cluster security group for the cluster that they’re associated with. Ensure that any existing nodes in your cluster can send and receive traffic to and from the cluster security group. Managed node groups are automatically configured to use the cluster security group as well, so you don’t need to modify or check them for this compatibility (see managed-node-groups.title).

For existing node groups that were created with eksctl or the Amazon EKS managed AWS CloudFormation templates, you can add the cluster security group to the nodes manually. Or, alternatively, you can modify the Auto Scaling group launch template for the node group to attach the cluster security group to the instances. For more information, see Changing an instance’s security groups in the Amazon VPC User Guide.

You can check for a security group for your cluster in the consolelong under the Networking section for the cluster. Or, you can do this using the following AWS CLI command. When using this command, replace my-cluster with the name of your cluster.

aws eks describe-cluster --name my-cluster --query cluster.resourcesVpcConfig.clusterSecurityGroupId

Step 2: Create a Fargate `Pod` execution role

When your cluster creates Pods on AWS Fargate, the components that run on the Fargate infrastructure must make calls to AWS APIs on your behalf. The Amazon EKS Pod execution role provides the IAM permissions to do this. To create an AWS Fargate Pod execution role, see pod-execution-role.title.

If you created your cluster with eksctl using the --fargate option, your cluster already has a Pod execution role that you can find in the IAM console with the pattern eksctl-my-cluster-FargatePodExecutionRole-ABCDEFGHIJKL. Similarly, if you use eksctl to create your Fargate profiles, eksctl creates your Pod execution role if one isn’t already created.

Step 3: Create a Fargate profile for your cluster

Before you can schedule Pods that are running on Fargate in your cluster, you must define a Fargate profile that specifies which Pods use Fargate when they’re launched. For more information, see fargate-profile.title.

If you created your cluster with eksctl using the --fargate option, then a Fargate profile is already created for your cluster with selectors for all Pods in the kube-system and default namespaces. Use the following procedure to create Fargate profiles for any other namespaces you would like to use with Fargate.

You can create a Fargate profile using either of these tools:

eksctl
consolelong

`eksctl`

This procedure requires eksctl version 0.199.0 or later. You can check your version with the following command:

eksctl version

For instructions on how to install or upgrade eksctl, see Installation in the eksctl documentation.

To create a Fargate profile with eksctl

Create your Fargate profile with the following eksctl command, replacing every example value with your own values. You’re required to specify a namespace. However, the --labels option isn’t required.

eksctl create fargateprofile \
    --cluster my-cluster \
    --name my-fargate-profile \
    --namespace my-kubernetes-namespace \
    --labels key=value

You can use certain wildcards for my-kubernetes-namespace and key=value labels. For more information, see fargate-profile-wildcards.title.

`consolelong`

To create a Fargate profile with consolelong

Open the Amazon EKS console.
Choose the cluster to create a Fargate profile for.
Choose the Compute tab.
Under Fargate profiles, choose Add Fargate profile.
On the Configure Fargate profile page, do the following:
1. For Name, enter a name for your Fargate profile. The name must be unique.
2. For Pod execution role, choose the Pod execution role to use with your Fargate profile. Only the IAM roles with the eks-fargate-pods.amazonaws.com service principal are shown. If you don’t see any roles listed, you must create one. For more information, see pod-execution-role.title.
3. Modify the selected Subnets as needed.
  
  Only private subnets are supported for Pods that are running on Fargate.
4. For Tags, you can optionally tag your Fargate profile. These tags don’t propagate to other resources that are associated with the profile such as Pods.
5. Choose Next.
On the Configure Pod selection page, do the following:
1. For Namespace, enter a namespace to match for Pods.
  - You can use specific namespaces to match, such as kube-system or default.
  - You can use certain wildcards (for example, prod-*) to match multiple namespaces (for example, prod-deployment and prod-test). For more information, see fargate-profile-wildcards.title.
2. (Optional) Add Kubernetes labels to the selector. Specifically add them to the one that the Pods in the specified namespace need to match.
  - You can add the label infrastructure: fargate to the selector so that only Pods in the specified namespace that also have the infrastructure: fargate Kubernetes label match the selector.
  - You can use certain wildcards (for example, key?: value?) to match multiple namespaces (for example, keya: valuea and keyb: valueb). For more information, see fargate-profile-wildcards.title.
3. Choose Next.
On the Review and create page, review the information for your Fargate profile and choose Create.

Step 4: Update `CoreDNS`

By default, CoreDNS is configured to run on Amazon EC2 infrastructure on Amazon EKS clusters. If you want to only run your Pods on Fargate in your cluster, complete the following steps.

If you created your cluster with eksctl using the --fargate option, then you can skip to fargate-gs-next-steps.title.

Create a Fargate profile for CoreDNS with the following command. Replace my-cluster with your cluster name, 111122223333 with your account ID, AmazonEKSFargatePodExecutionRole with the name of your Pod execution role, and 0000000000000001, 0000000000000002, and 0000000000000003 with the IDs of your private subnets. If you don’t have a Pod execution role, you must create one first (see fargate-sg-pod-execution-role.title).

The role ARN can’t include a path other than /. For example, if the name of your role is development/apps/my-role, you need to change it to my-role when specifying the ARN for the role. The format of the role ARN must be region.arniam::111122223333:role/role-name.

aws eks create-fargate-profile \
    --fargate-profile-name coredns \
    --cluster-name my-cluster \
    --pod-execution-role-arn region.arniam::111122223333:role/AmazonEKSFargatePodExecutionRole \
    --selectors namespace=kube-system,labels={k8s-app=kube-dns} \
    --subnets subnet-0000000000000001 subnet-0000000000000002 subnet-0000000000000003

Run the following command to remove the eks.amazonaws.com/compute-type : ec2 annotation from the CoreDNS Pods.

kubectl patch deployment coredns \
    -n kube-system \
    --type json \
    -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'

Next steps

You can start migrating your existing applications to run on Fargate with the following workflow.
1. create-fargate-profile.title that matches your application’s Kubernetes namespace and Kubernetes labels.
2. Delete and re-create any existing Pods so that they’re scheduled on Fargate. For example, the following command triggers a rollout of the coredns deployment. You can modify the namespace and deployment type to update your specific Pods.
  kubectl rollout restart -n kube-system deployment coredns
Deploy the alb-ingress.title to allow Ingress objects for your Pods running on Fargate.
You can use the vertical-pod-autoscaler.title to set the initial correct size of CPU and memory for your Fargate Pods, and then use the horizontal-pod-autoscaler.title to scale those Pods. If you want the Vertical Pod Autoscaler to automatically re-deploy Pods to Fargate with higher CPU and memory combinations, set the Vertical Pod Autoscaler’s mode to either Auto or Recreate. This is to ensure correct functionality. For more information, see the Vertical Pod Autoscaler documentation on GitHub.
You can set up the AWS Distro for OpenTelemetry (ADOT) collector for application monitoring by following these instructions.

9.4.4. Define which `Pods` use `AWS` Fargate when launched

Before you schedule Pods on Fargate in your cluster, you must define at least one Fargate profile that specifies which Pods use Fargate when launched.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West).

Before you schedule Pods on Fargate in your cluster, you must define at least one Fargate profile that specifies which Pods use Fargate when launched.

As an administrator, you can use a Fargate profile to declare which Pods run on Fargate. You can do this through the profile’s selectors. You can add up to five selectors to each profile. Each selector must contain a namespace. The selector can also include labels. The label field consists of multiple optional key-value pairs. Pods that match a selector are scheduled on Fargate. Pods are matched using a namespace and the labels that are specified in the selector. If a namespace selector is defined without labels, Amazon EKS attempts to schedule all the Pods that run in that namespace onto Fargate using the profile. If a to-be-scheduled Pod matches any of the selectors in the Fargate profile, then that Pod is scheduled on Fargate.

If a Pod matches multiple Fargate profiles, you can specify which profile a Pod uses by adding the following Kubernetes label to the Pod specification: eks.amazonaws.com/fargate-profile: my-fargate-profile. The Pod must match a selector in that profile to be scheduled onto Fargate. Kubernetes affinity/anti-affinity rules do not apply and aren’t necessary with Amazon EKS Fargate Pods.

When you create a Fargate profile, you must specify a Pod execution role. This execution role is for the Amazon EKS components that run on the Fargate infrastructure using the profile. It’s added to the cluster’s Kubernetes Role Based Access Control (RBAC) for authorization. That way, the kubelet that runs on the Fargate infrastructure can register with your Amazon EKS cluster and appear in your cluster as a node. The Pod execution role also provides IAM permissions to the Fargate infrastructure to allow read access to Amazon ECR image repositories. For more information, see pod-execution-role.title.

Fargate profiles can’t be changed. However, you can create a new updated profile to replace an existing profile, and then delete the original.

Any Pods that are running using a Fargate profile are stopped and put into a pending state when the profile is deleted.

If any Fargate profiles in a cluster are in the DELETING status, you must wait until after the Fargate profile is deleted before you create other profiles in that cluster.

Fargate does not currently support Kubernetes topologySpreadConstraints.

Amazon EKS and Fargate spread Pods across each of the subnets that’s defined in the Fargate profile. However, you might end up with an uneven spread. If you must have an even spread, use two Fargate profiles. Even spread is important in scenarios where you want to deploy two replicas and don’t want any downtime. We recommend that each profile has only one subnet.

Fargate profile components

The following components are contained in a Fargate profile.

Pod execution role

When your cluster creates Pods on AWS Fargate, the kubelet that’s running on the Fargate infrastructure must make calls to AWS APIs on your behalf. For example, it needs to make calls to pull container images from Amazon ECR. The Amazon EKS Pod execution role provides the IAM permissions to do this.

When you create a Fargate profile, you must specify a Pod execution role to use with your Pods. This role is added to the cluster’s Kubernetes Role-based access control (RBAC) for authorization. This is so that the kubelet that’s running on the Fargate infrastructure can register with your Amazon EKS cluster and appear in your cluster as a node. For more information, see pod-execution-role.title.

Subnets

The IDs of subnets to launch Pods into that use this profile. At this time, Pods that are running on Fargate aren’t assigned public IP addresses. Therefore, only private subnets with no direct route to an Internet Gateway are accepted for this parameter.

Selectors

The selectors to match for Pods to use this Fargate profile. You might specify up to five selectors in a Fargate profile. The selectors have the following components:

Namespace – You must specify a namespace for a selector. The selector only matches Pods that are created in this namespace. However, you can create multiple selectors to target multiple namespaces.
Labels – You can optionally specify Kubernetes labels to match for the selector. The selector only matches Pods that have all of the labels that are specified in the selector.

Fargate profile wildcards

In addition to characters allowed by Kubernetes, you’re allowed to use * and ? in the selector criteria for namespaces, label keys, and label values:

* represents none, one, or multiple characters. For example, prod* can represent prod and prod-metrics.
? represents a single character (for example, value? can represent valuea). However, it can’t represent value and value-a, because ? can only represent exactly one character.

These wildcard characters can be used in any position and in combination (for example, prod*, *dev, and frontend*?). Other wildcards and forms of pattern matching, such as regular expressions, aren’t supported.

If there are multiple matching profiles for the namespace and labels in the Pod spec, Fargate picks up the profile based on alphanumeric sorting by profile name. For example, if both profile A (with the name beta-workload) and profile B (with the name prod-workload) have matching selectors for the Pods to be launched, Fargate picks profile A (beta-workload) for the Pods. The Pods have labels with profile A on the Pods (for example, eks.amazonaws.com/fargate-profile=beta-workload).

If you want to migrate existing Fargate Pods to new profiles that use wildcards, there are two ways to do so:

Create a new profile with matching selectors, then delete the old profiles. Pods labeled with old profiles are rescheduled to new matching profiles.
If you want to migrate workloads but aren’t sure what Fargate labels are on each Fargate Pod, you can use the following method. Create a new profile with a name that sorts alphanumerically first among the profiles on the same cluster. Then, recycle the Fargate Pods that need to be migrated to new profiles.

Create a Fargate profile

This section describes how to create a Fargate profile. You also must have created a Pod execution role to use for your Fargate profile. For more information, see pod-execution-role.title. Pods that are running on Fargate are only supported on private subnets with NAT gateway access to AWS services, but not a direct route to an Internet Gateway. This is so that your cluster’s VPC must have private subnets available.

You can create a profile with the following:

eksctl
consolelong

`eksctl`

To create a Fargate profile with eksctl

eksctl create fargateprofile \
    --cluster my-cluster \
    --name my-fargate-profile \
    --namespace my-kubernetes-namespace \
    --labels key=value

You can use certain wildcards for my-kubernetes-namespace and key=value labels. For more information, see fargate-profile-wildcards.title.

`consolelong`

To create a Fargate profile with consolelong

Open the Amazon EKS console.
Choose the cluster to create a Fargate profile for.
Choose the Compute tab.
Under Fargate profiles, choose Add Fargate profile.
On the Configure Fargate profile page, do the following:
1. For Name, enter a unique name for your Fargate profile, such as my-profile.
2. For Pod execution role, choose the Pod execution role to use with your Fargate profile. Only the IAM roles with the eks-fargate-pods.amazonaws.com service principal are shown. If you don’t see any roles listed, you must create one. For more information, see pod-execution-role.title.
3. Modify the selected Subnets as needed.
  
  Only private subnets are supported for Pods that are running on Fargate.
4. For Tags, you can optionally tag your Fargate profile. These tags don’t propagate to other resources that are associated with the profile, such as Pods.
5. Choose Next.
On the Configure Pod selection page, do the following:
1. For Namespace, enter a namespace to match for Pods.
  - You can use specific namespaces to match, such as kube-system or default.
  - You can use certain wildcards (for example, prod-*) to match multiple namespaces (for example, prod-deployment and prod-test). For more information, see fargate-profile-wildcards.title.
2. (Optional) Add Kubernetes labels to the selector. Specifically, add them to the one that the Pods in the specified namespace need to match.
  - You can add the label infrastructure: fargate to the selector so that only Pods in the specified namespace that also have the infrastructure: fargate Kubernetes label match the selector.
  - You can use certain wildcards (for example, key?: value?) to match multiple namespaces (for example, keya: valuea and keyb: valueb). For more information, see fargate-profile-wildcards.title.
3. Choose Next.
On the Review and create page, review the information for your Fargate profile and choose Create.

9.4.5. Delete a Fargate profile

When you delete a Fargate profile, any Pods that were scheduled onto Fargate with the profile are deleted.

This topic describes how to delete a Fargate profile. When you delete a Fargate profile, any Pods that were scheduled onto Fargate with the profile are deleted. If those Pods match another Fargate profile, then they’re scheduled on Fargate with that profile. If they no longer match any Fargate profiles, then they aren’t scheduled onto Fargate and might remain as pending.

Only one Fargate profile in a cluster can be in the DELETING status at a time. Wait for a Fargate profile to finish deleting before you can delete any other profiles in that cluster.

You can delete a profile with any of the following tools:

eksctl
consolelong
AWS CLI

`eksctl`

Delete a Fargate profile with eksctl

Use the following command to delete a profile from a cluster. Replace every example value with your own values.

eksctl delete fargateprofile  --name my-profile --cluster my-cluster

`consolelong`

Delete a Fargate profile with consolelong

Open the Amazon EKS console.
In the left navigation pane, choose Clusters. In the list of clusters, choose the cluster that you want to delete the Fargate profile from.
Choose the Compute tab.
Choose the Fargate profile to delete, and then choose Delete.
On the Delete Fargate profile page, enter the name of the profile, and then choose Delete.

`AWS` CLI

Delete a Fargate profile with AWS CLI

Use the following command to delete a profile from a cluster. Replace every example value with your own values.

aws eks delete-fargate-profile --fargate-profile-name my-profile --cluster-name my-cluster

9.4.6. Understand Fargate `Pod` configuration details

This section describes some of the unique Pod configuration details for running Kubernetes Pods on AWS Fargate.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West).

This section describes some of the unique Pod configuration details for running Kubernetes Pods on AWS Fargate.

`Pod` CPU and memory

With Kubernetes, you can define requests, a minimum vCPU amount, and memory resources that are allocated to each container in a Pod. Pods are scheduled by Kubernetes to ensure that at least the requested resources for each Pod are available on the compute resource. For more information, see Managing compute resources for containers in the Kubernetes documentation.

Since Amazon EKS Fargate runs only one Pod per node, the scenario of evicting Pods in case of fewer resources doesn’t occur. All Amazon EKS Fargate Pods run with guaranteed priority, so the requested CPU and memory must be equal to the limit for all of the containers. For more information, see Configure Quality of Service for Pods in the Kubernetes documentation.

When Pods are scheduled on Fargate, the vCPU and memory reservations within the Pod specification determine how much CPU and memory to provision for the Pod.

The maximum request out of any Init containers is used to determine the Init request vCPU and memory requirements.
Requests for all long-running containers are added up to determine the long-running request vCPU and memory requirements.
The larger of the previous two values is chosen for the vCPU and memory request to use for your Pod.
Fargate adds 256 MB to each Pod’s memory reservation for the required Kubernetes components (kubelet, kube-proxy, and containerd).

Fargate rounds up to the following compute configuration that most closely matches the sum of vCPU and memory requests in order to ensure Pods always have the resources that they need to run.

If you don’t specify a vCPU and memory combination, then the smallest available combination is used (.25 vCPU and 0.5 GB memory).

The following table shows the vCPU and memory combinations that are available for Pods running on Fargate.

vCPU value

Memory value

.25 vCPU

0.5 GB, 1 GB, 2 GB

.5 vCPU

1 GB, 2 GB, 3 GB, 4 GB

1 vCPU

2 GB, 3 GB, 4 GB, 5 GB, 6 GB, 7 GB, 8 GB

2 vCPU

Between 4 GB and 16 GB in 1-GB increments

4 vCPU

Between 8 GB and 30 GB in 1-GB increments

8 vCPU

Between 16 GB and 60 GB in 4-GB increments

16 vCPU

Between 32 GB and 120 GB in 8-GB increments

The additional memory reserved for the Kubernetes components can cause a Fargate task with more vCPUs than requested to be provisioned. For example, a request for 1 vCPU and 8 GB memory will have 256 MB added to its memory request, and will provision a Fargate task with 2 vCPUs and 9 GB memory, since no task with 1 vCPU and 9 GB memory is available.

There is no correlation between the size of the Pod running on Fargate and the node size reported by Kubernetes with kubectl get nodes. The reported node size is often larger than the Pod’s capacity. You can verify Pod capacity with the following command. Replace default with your Pod’s namespace and pod-name with the name of your Pod.

kubectl describe pod --namespace default pod-name

An example output is as follows.

[...]
annotations:
    CapacityProvisioned: 0.25vCPU 0.5GB
[...]

The CapacityProvisioned annotation represents the enforced Pod capacity and it determines the cost of your Pod running on Fargate. For pricing information for the compute configurations, see AWS Fargate Pricing.

Fargate storage

A Pod running on Fargate automatically mounts an Amazon EFS file system, without needing manual driver installation steps. You can’t use dynamic persistent volume provisioning with Fargate nodes, but you can use static provisioning. For more information, see Amazon EFS CSI Driver on GitHub.

When provisioned, each Pod running on Fargate receives a default 20 GiB of ephemeral storage. This type of storage is deleted after a Pod stops. New Pods launched onto Fargate have encryption of the ephemeral storage volume enabled by default. The ephemeral Pod storage is encrypted with an AES-256 encryption algorithm using AWS Fargate managed keys.

The default usable storage for Amazon EKS Pods that run on Fargate is less than 20 GiB. This is because some space is used by the kubelet and other Kubernetes modules that are loaded inside the Pod.

You can increase the total amount of ephemeral storage up to a maximum of 175 GiB. To configure the size with Kubernetes, specify the requests of ephemeral-storage resource to each container in a Pod. When Kubernetes schedules Pods, it ensures that the sum of the resource requests for each Pod is less than the capacity of the Fargate task. For more information, see Resource Management for Pods and Containers in the Kubernetes documentation.

Amazon EKS Fargate provisions more ephemeral storage than requested for the purposes of system use. For example, a request of 100 GiB will provision a Fargate task with 115 GiB ephemeral storage.

9.4.7. Set actions for `AWS` Fargate OS patching events

Amazon EKS periodically patches the OS for AWS Fargate nodes to keep them secure.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West).

Amazon EKS periodically patches the OS for AWS Fargate nodes to keep them secure. As part of the patching process, we recycle the nodes to install OS patches. Updates are attempted in a way that creates the least impact on your services. However, if Pods aren’t successfully evicted, there are times when they must be deleted. The following are actions that you can take to minimize potential disruptions:

Set appropriate Pod disruption budgets (PDBs) to control the number of Pods that are down simultaneously.
Create Amazon EventBridge rules to handle failed evictions before the Pods are deleted.
Manually restart your affected pods before the eviction date posted in the notification you receive.
Create a notification configuration in AWS User Notifications.

Amazon EKS works closely with the Kubernetes community to make bug fixes and security patches available as quickly as possible. All Fargate Pods start on the most recent Kubernetes patch version, which is available from Amazon EKS for the Kubernetes version of your cluster. If you have a Pod with an older patch version, Amazon EKS might recycle it to update it to the latest version. This ensures that your Pods are equipped with the latest security updates. That way, if there’s a critical Common Vulnerabilities and Exposures (CVE) issue, you’re kept up to date to reduce security risks.

When the AWS Fargate OS is updated, Amazon EKS will send you a notification that includes your affected resources and the date of upcoming pod evictions. If the provided eviction date is inconvenient, you have the option to manually restart your affected pods before the eviction date posted in the notification. Any pods created before the time at which you receive the notification are subject to eviction. Refer to the Kubernetes Documentation for further instructions on how to manually restart your pods.

To limit the number of Pods that are down at one time when Pods are recycled, you can set Pod disruption budgets (PDBs). You can use PDBs to define minimum availability based on the requirements of each of your applications while still allowing updates to occur. Your PDB’s minimum availability must be less than 100%. For more information, see Specifying a Disruption Budget for your Application in the Kubernetes Documentation.

Amazon EKS uses the Eviction API to safely drain the Pod while respecting the PDBs that you set for the application. Pods are evicted by Availability Zone to minimize impact. If the eviction succeeds, the new Pod gets the latest patch and no further action is required.

When the eviction for a Pod fails, Amazon EKS sends an event to your account with details about the Pods that failed eviction. You can act on the message before the scheduled termination time. The specific time varies based on the urgency of the patch. When it’s time, Amazon EKS attempts to evict the Pods again. However, this time a new event isn’t sent if the eviction fails. If the eviction fails again, your existing Pods are deleted periodically so that the new Pods can have the latest patch.

The following is a sample event received when the Pod eviction fails. It contains details about the cluster, Pod name, Pod namespace, Fargate profile, and the scheduled termination time.

{
    "version": "0",
    "id": "12345678-90ab-cdef-0123-4567890abcde",
    "detail-type": "EKS Fargate Pod Scheduled Termination",
    "source": "aws.eks",
    "account": "111122223333",
    "time": "2021-06-27T12:52:44Z",
    "region": "region-code",
    "resources": [
        "default/my-database-deployment"
    ],
    "detail": {
        "clusterName": "my-cluster",
        "fargateProfileName": "my-fargate-profile",
        "podName": "my-pod-name",
        "podNamespace": "default",
        "evictErrorMessage": "Cannot evict pod as it would violate the pod's disruption budget",
        "scheduledTerminationTime": "2021-06-30T12:52:44.832Z[UTC]"
    }
}

In addition, having multiple PDBs associated with a Pod can cause an eviction failure event. This event returns the following error message.

"evictErrorMessage": "This pod has multiple PodDisruptionBudget, which the eviction subresource does not support",

You can create a desired action based on this event. For example, you can adjust your Pod disruption budget (PDB) to control how the Pods are evicted. More specifically, suppose that you start with a PDB that specifies the target percentage of Pods that are available. Before your Pods are force terminated during an upgrade, you can adjust the PDB to a different percentage of Pods. To receive this event, you must create an Amazon EventBridge rule in the AWS account and AWS Region that the cluster belongs to. The rule must use the following Custom pattern. For more information, see Creating Amazon EventBridge rules that react to events in the Amazon EventBridge User Guide.

{
  "source": ["aws.eks"],
  "detail-type": ["EKS Fargate Pod Scheduled Termination"]
}

A suitable target can be set for the event to capture it. For a complete list of available targets, see Amazon EventBridge targets in the Amazon EventBridge User Guide. You can also create a notification configuration in AWS User Notifications. When using the consolelong to create the notification, under Event Rules, choose Elastic Kubernetes Service (EKS) for AWS service name and EKS Fargate Pod Scheduled Termination for Event type. For more information, see Getting started with AWS User Notifications in the AWS User Notifications User Guide.

See FAQs: Fargate Pod eviction notice in AWS re:Post for frequently asked questions regarding EKS Pod Evictions.

9.4.8. Collect `AWS` Fargate app and usage metrics

You can collect system metrics and CloudWatch usage metrics for AWS Fargate.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West).

You can collect system metrics and CloudWatch usage metrics for AWS Fargate.

Application metrics

For applications running on Amazon EKS and AWS Fargate, you can use the AWS Distro for OpenTelemetry (ADOT). ADOT allows you to collect system metrics and send them to CloudWatch Container Insights dashboards. To get started with ADOT for applications running on Fargate, see Using CloudWatch Container Insights with AWS Distro for OpenTelemetry in the ADOT documentation.

Usage metrics

You can use CloudWatch usage metrics to provide visibility into your account’s usage of resources. Use these metrics to visualize your current service usage on CloudWatch graphs and dashboards.

AWS Fargate usage metrics correspond to AWS service quotas. You can configure alarms that alert you when your usage approaches a service quota. For more information about Fargate service quotas, see service-quotas.title.

AWS Fargate publishes the following metrics in the AWS/Usage namespace.

Metric Description

ResourceCount

The total number of the specified resource running on your account. The resource is defined by the dimensions associated with the metric.

The following dimensions are used to refine the usage metrics that are published by AWS Fargate.

Dimension Description

Service

The name of the AWS service containing the resource. For AWS Fargate usage metrics, the value for this dimension is Fargate.

Type

The type of entity that’s being reported. Currently, the only valid value for AWS Fargate usage metrics is Resource.

Resource

The type of resource that’s running.

Currently, AWS Fargate returns information on your Fargate On-Demand usage. The resource value for Fargate On-Demand usage is OnDemand.

[NOTE] ====

Fargate On-Demand usage combines Amazon EKS Pods using Fargate, Amazon ECS tasks using the Fargate launch type and Amazon ECS tasks using the FARGATE capacity provider.

====

Class

The class of resource being tracked. Currently, AWS Fargate doesn’t use the class dimension.

Creating a CloudWatch alarm to monitor Fargate resource usage metrics

AWS Fargate provides CloudWatch usage metrics that correspond to the AWS service quotas for Fargate On-Demand resource usage. In the Service Quotas console, you can visualize your usage on a graph. You can also configure alarms that alert you when your usage approaches a service quota. For more information, see monitoring-fargate-usage.title.

Use the following steps to create a CloudWatch alarm based on the Fargate resource usage metrics.

Open the Service Quotas console at https://console.aws.amazon.com/servicequotas/.
In the left navigation pane, choose AWS services.
From the AWS services list, search for and select AWS Fargate.
In the Service quotas list, choose the Fargate usage quota you want to create an alarm for.
In the Amazon CloudWatch alarms section, choose Create.
For Alarm threshold, choose the percentage of your applied quota value that you want to set as the alarm value.
For Alarm name, enter a name for the alarm and then choose Create.

9.4.9. Start `AWS` Fargate logging for your cluster

Amazon EKS on Fargate offers a built-in log router based on Fluent Bit.

AWS Fargate with Amazon EKS isn’t available in AWS GovCloud (US-East) and AWS GovCloud (US-West).

Amazon EKS on Fargate offers a built-in log router based on Fluent Bit. This means that you don’t explicitly run a Fluent Bit container as a sidecar, but Amazon runs it for you. All that you have to do is configure the log router. The configuration happens through a dedicated ConfigMap that must meet the following criteria:

Named aws-logging
Created in a dedicated namespace called aws-observability
Can’t exceed 5300 characters.

Once you’ve created the ConfigMap, Amazon EKS on Fargate automatically detects it and configures the log router with it. Fargate uses a version of AWS for Fluent Bit, an upstream compliant distribution of Fluent Bit managed by AWS. For more information, see AWS for Fluent Bit on GitHub.

The log router allows you to use the breadth of services at AWS for log analytics and storage. You can stream logs from Fargate directly to Amazon CloudWatch, Amazon OpenSearch Service. You can also stream logs to destinations such as Amazon S3, Amazon Kinesis Data Streams, and partner tools through Amazon Data Firehose.

An existing Fargate profile that specifies an existing Kubernetes namespace that you deploy Fargate Pods to. For more information, see fargate-gs-create-profile.title.
An existing Fargate Pod execution role. For more information, see fargate-sg-pod-execution-role.title.

Log router configuration

In the following steps, replace every example value with your own values.

Create a dedicated Kubernetes namespace named aws-observability.
1. Save the following contents to a file named aws-observability-namespace.yaml on your computer. The value for name must be aws-observability and the aws-observability: enabled label is required.
  kind: Namespace apiVersion: v1 metadata: name: aws-observability labels: aws-observability: enabled
2. Create the namespace.
  kubectl apply -f aws-observability-namespace.yaml

Create a ConfigMap with a Fluent Conf data value to ship container logs to a destination. Fluent Conf is Fluent Bit, which is a fast and lightweight log processor configuration language that’s used to route container logs to a log destination of your choice. For more information, see Configuration File in the Fluent Bit documentation.

The main sections included in a typical Fluent Conf are Service, Input, Filter, and Output. The Fargate log router however, only accepts:

The Filter and Output sections.
A Parser section.

If you provide any other sections, they will be rejected.

The Fargate log router manages the Service and Input sections. It has the following Input section, which can’t be modified and isn’t needed in your ConfigMap. However, you can get insights from it, such as the memory buffer limit and the tag applied for logs.

[INPUT]
    Name tail
    Buffer_Max_Size 66KB
    DB /var/log/flb_kube.db
    Mem_Buf_Limit 45MB
    Path /var/log/containers/*.log
    Read_From_Head On
    Refresh_Interval 10
    Rotate_Wait 30
    Skip_Long_Lines On
    Tag kube.*

When creating the ConfigMap, take into account the following rules that Fargate uses to validate fields:

[FILTER], [OUTPUT], and [PARSER] are supposed to be specified under each corresponding key. For example, [FILTER] must be under filters.conf. You can have one or more [FILTER]s under filters.conf. The [OUTPUT] and [PARSER] sections should also be under their corresponding keys. By specifying multiple [OUTPUT] sections, you can route your logs to different destinations at the same time.
Fargate validates the required keys for each section. Name and match are required for each [FILTER] and [OUTPUT]. Name and format are required for each [PARSER]. The keys are case-insensitive.
Environment variables such as ${ENV_VAR} aren’t allowed in the ConfigMap.
The indentation has to be the same for either directive or key-value pair within each filters.conf, output.conf, and parsers.conf. Key-value pairs have to be indented more than directives.
Fargate validates against the following supported filters: grep, parser, record_modifier, rewrite_tag, throttle, nest, modify, and kubernetes.
Fargate validates against the following supported output: es, firehose, kinesis_firehose, cloudwatch, cloudwatch_logs, and kinesis.

At least one supported Output plugin has to be provided in the ConfigMap to enable logging. Filter and Parser aren’t required to enable logging.

You can also run Fluent Bit on Amazon EC2 using the desired configuration to troubleshoot any issues that arise from validation. Create your ConfigMap using one of the following examples.

Amazon EKS Fargate logging doesn’t support dynamic configuration of a ConfigMap. Any changes to a ConfigMap are applied to new Pods only. Changes aren’t applied to existing Pods.

Create a ConfigMap using the example for your desired log destination.

You can also use Amazon Kinesis Data Streams for your log destination. If you use Kinesis Data Streams, make sure that the pod execution role has been granted the kinesis:PutRecords permission. For more information, see Amazon Kinesis Data Streams Permissions in the Fluent Bit: Official Manual.

CloudWatch

To create a ConfigMap for CloudWatch

You have two output options when using CloudWatch:

The following example shows you how to use the cloudwatch_logs plugin to send logs to CloudWatch.

Save the following contents to a file named aws-logging-cloudwatch-configmap.yaml. Replace region-code with the AWS Region that your cluster is in. The parameters under [OUTPUT] are required.

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  flb_log_cw: "false"  # Set to true to ship Fluent Bit process logs to CloudWatch.
  filters.conf: |
    [FILTER]
        Name parser
        Match *
        Key_name log
        Parser crio
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Buffer_Size 0
        Kube_Meta_Cache_TTL 300s
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   kube.*
        region region-code
        log_group_name my-logs
        log_stream_prefix from-fluent-bit-
        log_retention_days 60
        auto_create_group true
  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Apply the manifest to your cluster.

kubectl apply -f aws-logging-cloudwatch-configmap.yaml

Download the CloudWatch IAM policy to your computer. You can also view the policy on GitHub.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json

Amazon OpenSearch Service

To create a ConfigMap for Amazon OpenSearch Service

If you want to send logs to Amazon OpenSearch Service, you can use es output, which is a plugin written in C. The following example shows you how to use the plugin to send logs to OpenSearch.

Save the following contents to a file named aws-logging-opensearch-configmap.yaml. Replace every example value with your own values.

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
      Name  es
      Match *
      Host  search-example-gjxdcilagiprbglqn42jsty66y.region-code.es.amazonaws.com
      Port  443
      Index example
      Type  example_type
      AWS_Auth On
      AWS_Region region-code
      tls   On

Apply the manifest to your cluster.

kubectl apply -f aws-logging-opensearch-configmap.yaml

Download the OpenSearch IAM policy to your computer. You can also view the policy on GitHub.
```
curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json
```
Make sure that OpenSearch Dashboards' access control is configured properly. The all_access role in OpenSearch Dashboards needs to have the Fargate Pod execution role and the IAM role mapped. The same mapping must be done for the security_manager role. You can add the previous mappings by selecting Menu, then Security, then Roles, and then select the respective roles. For more information, see How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?.

Firehose

To create a ConfigMap for Firehose

You have two output options when sending logs to Firehose:

kinesis_firehose – An output plugin written in C.

firehose – An output plugin written in Golang.

The following example shows you how to use the kinesis_firehose plugin to send logs to Firehose.

Save the following contents to a file named aws-logging-firehose-configmap.yaml. Replace region-code with the AWS Region that your cluster is in.

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
     Name  kinesis_firehose
     Match *
     region region-code
     delivery_stream my-stream-firehose

Apply the manifest to your cluster.

kubectl apply -f aws-logging-firehose-configmap.yaml

Download the Firehose IAM policy to your computer. You can also view the policy on GitHub.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json

Create an IAM policy from the policy file you downloaded in a previous step.

aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json

Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace 111122223333 with your account ID. Replace AmazonEKSFargatePodExecutionRole with your Pod execution role (for more information, see fargate-sg-pod-execution-role.title).
```
aws iam attach-role-policy \
  --policy-arn region.arniam::111122223333:policy/eks-fargate-logging-policy \
  --role-name AmazonEKSFargatePodExecutionRole
```

`Kubernetes` filter support

This feature requires the following minimum Kubernetes version and platform level, or later.

Kubernetes version

Platform level

1.23 and later

eks.1

The Fluent Bit Kubernetes filter allows you to add Kubernetes metadata to your log files. For more information about the filter, see Kubernetes in the Fluent Bit documentation. You can apply a filter using the API server endpoint.

filters.conf: |
    [FILTER]
        Name             kubernetes
        Match            kube.*
        Merge_Log           On
        Buffer_Size         0
        Kube_Meta_Cache_TTL 300s

Kube_URL, Kube_CA_File, Kube_Token_Command, and Kube_Token_File are service owned configuration parameters and must not be specified. Amazon EKS Fargate populates these values.
Kube_Meta_Cache_TTL is the time Fluent Bit waits until it communicates with the API server for the latest metadata. If Kube_Meta_Cache_TTL isn’t specified, Amazon EKS Fargate appends a default value of 30 minutes to lessen the load on the API server.

To ship `Fluent Bit` process logs to your account

You can optionally ship Fluent Bit process logs to Amazon CloudWatch using the following ConfigMap. Shipping Fluent Bit process logs to CloudWatch requires additional log ingestion and storage costs. Replace region-code with the AWS Region that your cluster is in.

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
  labels:
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  flb_log_cw: "true"  # Ships Fluent Bit process logs to CloudWatch.

  output.conf: |
    [OUTPUT]
        Name cloudwatch
        Match kube.*
        region region-code
        log_group_name fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group true

The logs are in the AWS Region that the cluster resides in under CloudWatch. The log group name is my-cluster-fluent-bit-logs and the Fluent Bit logstream name is fluent-bit-podname-pod-namespace.

The process logs are shipped only when the Fluent Bit process successfully starts. If there is a failure while starting Fluent Bit, the process logs are missed. You can only ship process logs to CloudWatch.
To debug shipping process logs to your account, you can apply the previous ConfigMap to get the process logs. Fluent Bit failing to start is usually due to your ConfigMap not being parsed or accepted by Fluent Bit while starting.

To stop shipping `Fluent Bit` process logs

Shipping Fluent Bit process logs to CloudWatch requires additional log ingestion and storage costs. To exclude process logs in an existing ConfigMap setup, do the following steps.

Locate the CloudWatch log group automatically created for your Amazon EKS cluster’s Fluent Bit process logs after enabling Fargate logging. It follows the format {cluster_name}-fluent-bit-logs.
Delete the existing CloudWatch log streams created for each Pod’s process logs in the CloudWatch log group.
Edit the ConfigMap and set flb_log_cw: "false".
Restart any existing Pods in the cluster.

Test application

Deploy a sample Pod.

Save the following contents to a file named sample-app.yaml on your computer.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: same-namespace-as-your-fargate-profile
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - name: http
              containerPort: 80

Apply the manifest to the cluster.
```
kubectl apply -f sample-app.yaml
```

View the NGINX logs using the destination(s) that you configured in the ConfigMap.

Size considerations

We suggest that you plan for up to 50 MB of memory for the log router. If you expect your application to generate logs at very high throughput then you should plan for up to 100 MB.

Troubleshooting

To confirm whether the logging feature is enabled or disabled for some reason, such as an invalid ConfigMap, and why it’s invalid, check your Pod events with kubectl describe pod pod-name. The output might include Pod events that clarify whether logging is enabled or not, such as the following example output.

[...]
Annotations:          CapacityProvisioned: 0.25vCPU 0.5GB
                      Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND
                      kubernetes.io/psp: eks.privileged
[...]
Events:
  Type     Reason           Age        From                                                           Message
  ----     ------           ----       ----                                                           -------
  Warning  LoggingDisabled  <unknown>  fargate-scheduler                                              Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found

The Pod events are ephemeral with a time period depending on the settings. You can also view a Pod’s annotations using kubectl describe pod pod-name. In the Pod annotation, there is information about whether the logging feature is enabled or disabled and the reason.

9.5. Choose an optimal Amazon EC2 node instance type

Each Amazon EC2 instance type offers different compute, memory, storage, and network capabilities.

Amazon EC2 provides a wide selection of instance types for worker nodes. Each instance type offers different compute, memory, storage, and network capabilities. Each instance is also grouped in an instance family based on these capabilities. For a list, see Available instance types in the Amazon EC2 User Guide. Amazon EKS releases several variations of Amazon EC2 AMIs to enable support. To make sure that the instance type you select is compatible with Amazon EKS, consider the following criteria.

All Amazon EKS AMIs don’t currently support the g5g and mac families.
Arm and non-accelerated Amazon EKS AMIs don’t support the g3, g4, inf, and p families.
Accelerated Amazon EKS AMIs don’t support the a, c, hpc, m, and t families.
For Arm-based instances, Amazon Linux 2023 (AL2023) only supports instance types that use Graviton2 or later processors. AL2023 doesn’t support A1 instances.

When choosing between instance types that are supported by Amazon EKS, consider the following capabilities of each type.

Number of instances in a node group: In general, fewer, larger instances are better, especially if you have a lot of Daemonsets. Each instance requires API calls to the API server, so the more instances you have, the more load on the API server.
Operating system: Review the supported instance types for Linux, Windows, and Bottlerocket. Before creating Windows instances, review Deploy Windows nodes on EKS clusters.
Hardware architecture: Do you need x86 or Arm? Before deploying Arm instances, review Amazon EKS optimized Arm Amazon Linux AMIs. Do you need instances built on the Nitro System ( Linux or Windows) or that have Accelerated capabilities? If you need accelerated capabilities, you can only use Linux with Amazon EKS.
Maximum number of Pods: Since each Pod is assigned its own IP address, the number of IP addresses supported by an instance type is a factor in determining the number of Pods that can run on the instance. To manually determine how many Pods an instance type supports, see determine-max-pods.title. +NOTE: If you’re using an Amazon EKS optimized Amazon Linux 2 AMI that’s `v20220406 or newer, you can use a new instance type without upgrading to the latest AMI. For these AMIs, the AMI auto-calculates the necessary max-pods value if it isn’t listed in the eni-max-pods.txt file. Instance types that are currently in preview may not be supported by Amazon EKS by default. Values for max-pods` for such types still need to be added to eni-max-pods.txt in our AMI.

AWS Nitro System instance types optionally support significantly more IP addresses than non-Nitro System instance types. However, not all IP addresses assigned for an instance are available to Pods. To assign a significantly larger number of IP addresses to your instances, you must have version 1.9.0 or later of the Amazon VPC CNI add-on installed in your cluster and configured appropriately. For more information, see cni-increase-ip-addresses.title. To assign the largest number of IP addresses to your instances, you must have version 1.10.1 or later of the Amazon VPC CNI add-on installed in your cluster and deploy the cluster with the IPv6 family.
IP family: You can use any supported instance type when using the IPv4 family for a cluster, which allows your cluster to assign private IPv4 addresses to your Pods and Services. But if you want to use the IPv6 family for your cluster, then you must use AWS Nitro System instance types or bare metal instance types. Only IPv4 is supported for Windows instances. Your cluster must be running version 1.10.1 or later of the Amazon VPC CNI add-on. For more information about using IPv6, see cni-ipv6.title.
Version of the Amazon VPC CNI add-on that you’re running: The latest version of the Amazon VPC CNI plugin for Kubernetes supports these instance types. You may need to update your Amazon VPC CNI add-on version to take advantage of the latest supported instance types. For more information, see managing-vpc-cni.title. The latest version supports the latest features for use with Amazon EKS. Earlier versions don’t support all features. You can view features supported by different versions in the Changelog on GitHub.
AWS Region that you’re creating your nodes in: Not all instance types are available in all AWS Regions.
Whether you’re using security groups for Pods: If you’re using security groups for Pods, only specific instance types are supported. For more information, see security-groups-for-pods.title.

9.5.1. Amazon EKS recommended maximum `Pods` for each Amazon EC2 instance type

Since each Pod is assigned its own IP address, the number of IP addresses supported by an instance type is a factor in determining the number of Pods that can run on the instance. Amazon EKS provides a script that you can download and run to determine the Amazon EKS recommended maximum number of Pods to run on each instance type. The script uses hardware attributes of each instance, and configuration options, to determine the maximum Pods number. You can use the number returned in these steps to enable capabilities such as assigning IP addresses to Pods from a different subnet than the instance’s and significantly increasing the number of IP addresses for your instance. If you’re using a managed node group with multiple instance types, use a value that would work for all instance types.

Download a script that you can use to calculate the maximum number of Pods for each instance type.

curl -O https://raw.githubusercontent.com/awslabs/amazon-eks-ami/master/templates/al2/runtime/max-pods-calculator.sh

Mark the script as executable on your computer.
```
chmod +x max-pods-calculator.sh
```
Run the script, replacing m5.large with the instance type that you plan to deploy and 1.9.0-eksbuild.1 with your Amazon VPC CNI add-on version. To determine your add-on version, see the update procedures in Assign IPs to Pods with the Amazon VPC CNI.
```
./max-pods-calculator.sh --instance-type m5.large --cni-version 1.9.0-eksbuild.1
```
An example output is as follows.
```
29
```
You can add the following options to the script to see the maximum Pods supported when using optional capabilities.
- --cni-custom-networking-enabled – Use this option when you want to assign IP addresses from a different subnet than your instance’s. For more information, see cni-custom-network.title. Adding this option to the previous script with the same example values yields 20.
- --cni-prefix-delegation-enabled – Use this option when you want to assign significantly more IP addresses to each elastic network interface. This capability requires an Amazon Linux instance that run on the Nitro System and version 1.9.0 or later of the Amazon VPC CNI add-on. For more information, see cni-increase-ip-addresses.title. Adding this option to the previous script with the same example values yields 110.

You can also run the script with the --help option to see all available options.

The max Pods calculator script limits the return value to 110 based on Kubernetes scalability thresholds and recommended settings. If your instance type has greater than 30 vCPUs, this limit jumps to 250, a number based on internal Amazon EKS scalability team testing. For more information, see the Amazon VPC CNI plugin increases pods per node limits blog post.

9.5.2. Considerations for EKS Auto Mode

EKS Auto Mode limits the number of pods on nodes to the lower of:

110 pods hard cap
The result of the max pods calculation described above.

9.6. Create nodes with pre-built optimized images

You can deploy nodes with pre-built Amazon EKS optimized Amazon Machine Images (AMIs) or your own custom AMIs.

You can deploy nodes with pre-built Amazon EKS optimized Amazon Machine Images (AMIs) or your own custom AMIs when you use managed node groups or self-managed nodes. If you are running hybrid nodes, see hybrid-nodes-os.title. For information about each type of Amazon EKS optimized AMI, see one of the following topics. For instructions on how to create your own custom AMI, see eks-ami-build-scripts.title.

With Amazon EKS Auto Mode, EKS manages the EC2 instance including selecting and updating the AMI.

[[Topic List]]

9.6.1. Migrate from `dockershim` to `containerd`

Starting with Kubernetes version 1.24, Amazon EKS AMIs that are officially published only include the containerd runtime.

Kubernetes no longer supports dockershim. The Kubernetes team removed the runtime in Kubernetes version 1.24. For more information, see Kubernetes is Moving on From Dockershim: Commitments and Next Steps on the Kubernetes Blog.

Amazon EKS also ended support for dockershim starting with the Kubernetes version 1.24 release. Amazon EKS AMIs that are officially published have containerd as the only runtime starting with version 1.24. This topic covers some details, but more information is available in All you need to know about moving to containerd on Amazon EKS.

There’s a kubectl plugin that you can use to see which of your Kubernetes workloads mount the Docker socket volume. For more information, see Detector for Docker Socket (DDS) on GitHub. Amazon EKS AMIs that run Kubernetes versions that are earlier than 1.24 use Docker as the default runtime. However, these Amazon EKS AMIs have a bootstrap flag option that you can use to test out your workloads on any supported cluster using containerd. For more information, see containerd-bootstrap.title.

We will continue to publish AMIs for existing Kubernetes versions until the end of their support date. For more information, see kubernetes-release-calendar.title. If you require more time to test your workloads on containerd, use a supported version before 1.24. But, when you want to upgrade official Amazon EKS AMIs to version 1.24 or later, make sure to validate that your workloads run on containerd.

The containerd runtime provides more reliable performance and security. containerd is the runtime that’s being standardized on across Amazon EKS. Fargate and Bottlerocket already use containerd only. containerd helps to minimize the number of Amazon EKS AMI releases that are required to address dockershim Common Vulnerabilities and Exposures (CVEs). Because dockershim already uses containerd internally, you might not need to make any changes. However, there are some situations where changes might or must be required:

You must make changes to applications that mount the Docker socket. For example, container images that are built with a container are impacted. Many monitoring tools also mount the Docker socket. You might need to wait for updates or re-deploy workloads for runtime monitoring.
You might need to make changes for applications that are reliant on specific Docker settings. For example, the HTTPS_PROXY protocol is no longer supported. You must update applications that use this protocol. For more information, see dockerd in the Docker Documentation.
If you use the Amazon ECR credential helper to pull images, you must switch to the kubelet image credential provider. For more information, see Configure a kubelet image credential provider in the Kubernetes documentation.
Because Amazon EKS 1.24 no longer supports Docker, some flags that the Amazon EKS bootstrap script previously supported are no longer supported. Before moving to Amazon EKS 1.24 or later, you must remove any reference to flags that are now unsupported:
- --container-runtime dockerd (containerd is the only supported value)
- --enable-docker-bridge
- --docker-config-json
If you already have Fluentd configured for Container Insights, then you must migrate Fluentd to Fluent Bit before changing to containerd. The Fluentd parsers are configured to only parse log messages in JSON format. Unlike dockerd, the containerd container runtime has log messages that aren’t in JSON format. If you don’t migrate to Fluent Bit, some of the configured Fluentd’s parsers will generate a massive amount of errors inside the Fluentd container. For more information on migrating, see Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs.
If you use a custom AMI and you are upgrading to Amazon EKS 1.24, then you must make sure that IP forwarding is enabled for your worker nodes. This setting wasn’t needed with Docker but is required for containerd. It is needed to troubleshoot Pod-to-Pod, Pod-to-external, or Pod-to-apiserver network connectivity.

To verify this setting on a worker node, run either of the following commands:
- sysctl net.ipv4.ip_forward
- cat /proc/sys/net/ipv4/ip_forward
If the output is 0, then run either of the following commands to activate the net.ipv4.ip_forward kernel variable:

+
- sysctl -w net.ipv4.ip_forward=1
- echo 1 > /proc/sys/net/ipv4/ip_forward

For the setting’s activation on Amazon EKS AMIs for Amazon Linux 2 in the containerd runtime, see install-worker.sh on GitHub.

Test Amazon Linux 2 migration from `Docker` to `containerd`

For Kubernetes version 1.23, you can use an optional bootstrap flag to enable the containerd runtime for Amazon EKS optimized AL2 AMIs. This feature gives you a clear path to migrate to containerd when updating to version 1.24 or later. Amazon EKS ended support for Docker starting with the Kubernetes version 1.24 launch. The containerd runtime is widely adopted in the Kubernetes community and is a graduated project with the CNCF. You can test it by adding a node group to a new or existing cluster.

You can enable the boostrap flag by creating one of the following types of node groups.

Self-managed

Create the node group using the instructions in Create self-managed Amazon Linux nodes. Specify an Amazon EKS optimized AMI and the following text for the BootstrapArguments parameter.

--container-runtime containerd

Managed

If you use eksctl, create a file named my-nodegroup.yaml with the following contents. Replace every example value with your own values. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. To retrieve an optimized AMI ID for ami-1234567890abcdef0, see retrieve-ami-id.title.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: region-code
  version: 1.23
managedNodeGroups:
  - name: my-nodegroup
    ami: ami-1234567890abcdef0
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh my-cluster --container-runtime containerd

If you launch many nodes simultaneously, you may also want to specify values for the --apiserver-endpoint, --b64-cluster-ca, and --dns-cluster-ip bootstrap arguments to avoid errors. For more information, see launch-template-custom-ami.title.

Run the following command to create the node group.

eksctl create nodegroup -f my-nodegroup.yaml

If you prefer to use a different tool to create your managed node group, you must deploy the node group using a launch template. In your launch template, specify an Amazon EKS optimized AMI ID, then deploy the node group using a launch template and provide the following user data. This user data passes arguments into the bootstrap.sh file. For more information about the bootstrap file, see bootstrap.sh on GitHub.

/etc/eks/bootstrap.sh my-cluster --container-runtime containerd

9.6.2. Create nodes with optimized Amazon Linux AMIs

Upgrade from Amazon Linux 2 to Amazon Linux 2023

AL2023 is a new Linux-based operating system designed to provide a secure, stable, and high-performance environment for your cloud applications.

The Amazon EKS optimized AMIs are available in two families based on AL2 and AL2023. AL2023 is a new Linux-based operating system designed to provide a secure, stable, and high-performance environment for your cloud applications. It’s the next generation of Amazon Linux from Amazon Web Services and is available across all supported Amazon EKS versions, including versions 1.23 and 1.24 in extended support.

AL2023 offers several improvements over AL2. For a full comparison, see Comparing AL2 and Amazon Linux 2023 in the Amazon Linux 2023 User Guide. Several packages have been added, upgraded, and removed from AL2. It’s highly recommended to test your applications with AL2023 before upgrading. For a list of all package changes in AL2023, see Package changes in Amazon Linux 2023 in the Amazon Linux 2023 Release Notes.

In addition to these changes, you should be aware of the following:

AL2023 introduces a new node initialization process nodeadm that uses a YAML configuration schema. If you’re using self-managed node groups or an AMI with a launch template, you’ll now need to provide additional cluster metadata explicitly when creating a new node group. An example of the minimum required parameters is as follows, where apiServerEndpoint, certificateAuthority, and service cidr are now required:
```
---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: my-cluster
    apiServerEndpoint: https://example.com
    certificateAuthority: Y2VydGlmaWNhdGVBdXRob3JpdHk=
    cidr: 10.100.0.0/16
```
In AL2, the metadata from these parameters was discovered from the Amazon EKS DescribeCluster API call. With AL2023, this behavior has changed since the additional API call risks throttling during large node scale ups. This change doesn’t affect you if you’re using managed node groups without a launch template or if you’re using Karpenter. For more information on certificateAuthority and service cidr, see ` DescribeCluster` in the Amazon EKS API Reference.
Docker isn’t supported in AL2023 for all supported Amazon EKS versions. Support for Docker has ended and been removed with Amazon EKS version 1.24 or greater in AL2. For more information on deprecation, see dockershim-deprecation.title.
Amazon VPC CNI version 1.16.2 or greater is required for AL2023.

AL2023 requires IMDSv2 by default. IMDSv2 has several benefits that help improve security posture. It uses a session-oriented authentication method that requires the creation of a secret token in a simple HTTP PUT request to start the session. A session’s token can be valid for anywhere between 1 second and 6 hours. For more information on how to transition from IMDSv1 to IMDSv2, see Transition to using Instance Metadata Service Version 2 and Get the full benefits of IMDSv2 and disable IMDSv1 across your AWS infrastructure. If you would like to use IMDSv1, you can still do so by manually overriding the settings using instance metadata option launch properties.

For IMDSv2, the default hop count for managed node groups is set to 1. This means that containers won’t have access to the node’s credentials using IMDS. If you require container access to the node’s credentials, you can still do so by manually overriding the HttpPutResponseHopLimit in a custom Amazon EC2 launch template, increasing it to 2.Alternatively, you can use Amazon EKS Pod Identity to provide credentials instead of IMDSv2.

AL2023 features the next generation of unified control group hierarchy (cgroupv2). cgroupv2 is used to implement a container runtime, and by systemd. While AL2023 still includes code that can make the system run using cgroupv1, this isn’t a recommended or supported configuration. This configuration will be completely removed in a future major release of Amazon Linux.
eksctl version 0.176.0 or greater is required for eksctl to support AL2023.

For previously existing managed node groups, you can either perform an in-place upgrade or a blue/green upgrade depending on how you’re using a launch template:

If you’re using a custom AMI with a managed node group, you can perform an in-place upgrade by swapping the AMI ID in the launch template. You should ensure that your applications and any user data transfer over to AL2023 first before performing this upgrade strategy.
If you’re using managed node groups with either the standard launch template or with a custom launch template that doesn’t specify the AMI ID, you’re required to upgrade using a blue/green strategy. A blue/green upgrade is typically more complex and involves creating an entirely new node group where you would specify AL2023 as the AMI type. The new node group will need to then be carefully configured to ensure that all custom data from the AL2 node group is compatible with the new OS. Once the new node group has been tested and validated with your applications, Pods can be migrated from the old node group to the new node group. Once the migration is completed, you can delete the old node group.

If you’re using Karpenter and want to use AL2023, you’ll need to modify the EC2NodeClass amiFamily field with AL2023. By default, Drift is enabled in Karpenter. This means that once the amiFamily field has been changed, Karpenter will automatically update your worker nodes to the latest AMI when available.

Retrieve Amazon Linux AMI version information

This topic gives the location of Amazon EKS optimized Amazon Linux AMIs version information.

Amazon EKS optimized Amazon Linux AMIs are versioned by Kubernetes version and the release date of the AMI in the following format:

k8s_major_version.k8s_minor_version.k8s_patch_version-release_date

Each AMI release includes various versions of kubelet, the Linux kernel, and containerd. The accelerated AMIs also include various versions of the NVIDIA driver. You can find this version information in the Changelog on GitHub.

Retrieve recommended Amazon Linux AMI IDs

You can programmatically retrieve the Amazon Machine Image (AMI) ID for Amazon EKS optimized AMIs by querying the AWS Systems Manager Parameter Store API.

When deploying nodes, you can specify an ID for a pre-built Amazon EKS optimized Amazon Machine Image (AMI). To retrieve an AMI ID that fits your desired configuration, query the AWS Systems Manager Parameter Store API. Using this API eliminates the need to manually look up Amazon EKS optimized AMI IDs. For more information, see GetParameter. The IAM principal that you use must have the ssm:GetParameter IAM permission to retrieve the Amazon EKS optimized AMI metadata.

You can retrieve the image ID of the latest recommended Amazon EKS optimized Amazon Linux AMI with the following command, which uses the sub-parameter image_id. Make the following modifications to the command as needed and then run the modified command:

Replace kubernetes-version with a supported Amazon EKS version.
Replace ami-type with one of the following options. For information about the types of Amazon EC2 instances, see Amazon EC2 instance types.
- Use amazon-linux-2023/x86_64/standard for Amazon Linux 2023 (AL2023) x86 based instances.
- Use amazon-linux-2023/arm64/standard for AL2023 ARM instances.
- Use amazon-linux-2023/x86_64/nvidia for the latest approved AL2023 NVIDIA instances.
- Use amazon-linux-2023/x86_64/neuron for the latest AL2023 AWS Neuron instances.
- Use amazon-linux-2 for Amazon Linux 2 (AL2) x86 based instances.
- Use amazon-linux-2-arm64 for AL2 ARM instances, such as AWS Graviton based instances.
- Use amazon-linux-2-gpu for AL2 hardware accelerated x86 based instances for NVIDIA GPU, Inferentia, and Trainium based workloads.
Replace region-code with an Amazon EKS supported AWS Region for which you want the AMI ID.

aws ssm get-parameter --name /aws/service/eks/optimized-ami/kubernetes-version/ami-type/recommended/image_id \
    --region region-code --query "Parameter.Value" --output text

Here’s an example command after placeholder replacements have been made.

aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.31/amazon-linux-2023/x86_64/standard/recommended/image_id \
    --region us-west-2 --query "Parameter.Value" --output text

An example output is as follows.

ami-1234567890abcdef0

Build a custom Amazon Linux AMI with a script

Amazon Elastic Kubernetes Service (Amazon EKS) has open-source scripts that are used to build the Amazon EKS optimized AMI.

Amazon Elastic Kubernetes Service (Amazon EKS) has open-source scripts that are used to build the Amazon EKS optimized AMI. These build scripts are available on GitHub.

The Amazon EKS optimized Amazon Linux AMIs are built on top of Amazon Linux 2 (AL2) and Amazon Linux 2023 (AL2023), specifically for use as a node in Amazon EKS clusters. You can use this repository to view the specifics of how the Amazon EKS team configures kubelet, the runtime, the AWS IAM Authenticator for Kubernetes, and build your own Amazon Linux based AMI from scratch.

The build scripts repository includes a HashiCorp packer template and build scripts to generate an AMI. These scripts are the source of truth for Amazon EKS optimized AMI builds, so you can follow the GitHub repository to monitor changes to our AMIs. For example, perhaps you want your own AMI to use the same version of Docker that the Amazon EKS team uses for the official AMI.

The GitHub repository also contains the specialized bootstrap script and nodeadm script that runs at boot time to configure your instance’s certificate data, control plane endpoint, cluster name, and more.

Additionally, the GitHub repository contains our Amazon EKS node AWS CloudFormation templates. These templates make it easier to spin up an instance running an Amazon EKS optimized AMI and register it with a cluster.

For more information, see the repositories on GitHub at https://github.com/awslabs/amazon-eks-ami.

Amazon EKS optimized AL2 contains an optional bootstrap flag to enable the containerd runtime.

The Amazon EKS optimized Amazon Linux AMIs are built on top of Amazon Linux 2 (AL2) and Amazon Linux 2023 (AL2023). They are configured to serve as the base images for Amazon EKS nodes.

The Amazon EKS optimized Amazon Linux AMIs are built on top of Amazon Linux 2 (AL2) and Amazon Linux 2023 (AL2023). They are configured to serve as the base images for Amazon EKS nodes. The AMIs are configured to work with Amazon EKS and they include the following components:

kubelet
AWS IAM Authenticator
Docker (Amazon EKS version 1.23 and earlier)
containerd

You can track security or privacy events for Amazon Linux at the Amazon Linux security center by choosing the tab for your desired version. You can also subscribe to the applicable RSS feed. Security and privacy events include an overview of the issue, what packages are affected, and how to update your instances to correct the issue.
Before deploying an accelerated or Arm AMI, review the information in Amazon EKS optimized accelerated Amazon Linux AMIs and arm-ami.title.
For Kubernetes version 1.23, you can use an optional bootstrap flag to test migration from Docker to containerd. For more information, see containerd-bootstrap.title.
Amazon EC2 P2 instances aren’t supported on Amazon EKS because they require NVIDIA driver version 470 or earlier.
Any newly created managed node groups in clusters on version 1.30 or newer will automatically default to using AL2023 as the node operating system. Previously, new node groups would default to AL2. You can continue to use AL2 by choosing it as the AMI type when creating a new node group.
Support for AL2 will end on June 30th, 2025. For more information, see Amazon Linux 2 FAQs.

Amazon EKS optimized accelerated Amazon Linux AMIs

The Amazon EKS optimized accelerated Amazon Linux AMIs are built on top of the standard Amazon EKS optimized Amazon Linux AMIs. They are configured to serve as optional images for Amazon EKS nodes to support GPU, Inferentia, and Trainium based workloads.

In addition to the standard Amazon EKS optimized AMI configuration, the accelerated AMIs include the following:

NVIDIA drivers
nvidia-container-toolkit
AWS Neuron driver

For a list of the latest components included in the accelerated AMIs, see the amazon-eks-ami Releases on GitHub.

Make sure to specify the applicable instance type in your node AWS CloudFormation template. By using the Amazon EKS optimized accelerated AMIs, you agree to NVIDIA’s Cloud End User License Agreement (EULA).
The Amazon EKS optimized accelerated AMIs were previously referred to as the Amazon EKS optimized AMIs with GPU support.
Previous versions of the Amazon EKS optimized accelerated AMIs installed the nvidia-docker repository. The repository is no longer included in Amazon EKS AMI version v20200529 and later.

For details on running workloads on Amazon EKS optimized accelerated Amazon Linux AMIs, see ml-eks-optimized-ami.title.

Amazon EKS optimized `Arm` Amazon Linux AMIs

Arm instances deliver significant cost savings for scale-out and Arm-based applications such as web servers, containerized microservices, caching fleets, and distributed data stores. When adding Arm nodes to your cluster, review the following considerations.

If your cluster was deployed before August 17, 2020, you must do a one-time upgrade of critical cluster add-on manifests. This is so that Kubernetes can pull the correct image for each hardware architecture in use in your cluster. For more information about updating cluster add-ons, see update-existing-cluster.title. If you deployed your cluster on or after August 17, 2020, then your CoreDNS, kube-proxy, and Amazon VPC CNI plugin for Kubernetes add-ons are already multi-architecture capable.
Applications deployed to Arm nodes must be compiled for Arm.
If you have DaemonSets that are deployed in an existing cluster, or you want to deploy them to a new cluster that you also want to deploy Arm nodes in, then verify that your DaemonSet can run on all hardware architectures in your cluster.
You can run Arm node groups and x86 node groups in the same cluster. If you do, consider deploying multi-architecture container images to a container repository such as Amazon Elastic Container Registry and then adding node selectors to your manifests so that Kubernetes knows what hardware architecture a Pod can be deployed to. For more information, see Pushing a multi-architecture image in the Amazon ECR User Guide and the Introducing multi-architecture container images for Amazon ECR blog post.

More information

For more information about using Amazon EKS optimized Amazon Linux AMIs, see the following sections:

To use Amazon Linux with managed node groups, see managed-node-groups.title.
To launch self-managed Amazon Linux nodes, see retrieve-ami-id.title.
For version information, see eks-linux-ami-versions.title.
To retrieve the latest IDs of the Amazon EKS optimized Amazon Linux AMIs, see retrieve-ami-id.title.
For open-source scripts that are used to build the Amazon EKS optimized AMIs, see eks-ami-build-scripts.title.

9.6.3. Create nodes with optimized `Bottlerocket` AMIs

Bottlerocket is an open source Linux distribution that’s sponsored and supported by AWS. Bottlerocket includes only the essential software to run containers, which improves resource usage, reduces security threats, and lowers management overhead.

Bottlerocket is an open source Linux distribution that’s sponsored and supported by AWS. Bottlerocket is purpose-built for hosting container workloads. With Bottlerocket, you can improve the availability of containerized deployments and reduce operational costs by automating updates to your container infrastructure. Bottlerocket includes only the essential software to run containers, which improves resource usage, reduces security threats, and lowers management overhead. The Bottlerocket AMI includes containerd, kubelet, and AWS IAM Authenticator. In addition to managed node groups and self-managed nodes, Bottlerocket is also supported by Karpenter.

Advantages

Using Bottlerocket with your Amazon EKS cluster has the following advantages:

Higher uptime with lower operational cost and lower management complexity – Bottlerocket has a smaller resource footprint, shorter boot times, and is less vulnerable to security threats than other Linux distributions. Bottlerocket’s smaller footprint helps to reduce costs by using less storage, compute, and networking resources.
Improved security from automatic OS updates – Updates to Bottlerocket are applied as a single unit which can be rolled back, if necessary. This removes the risk of corrupted or failed updates that can leave the system in an unusable state. With Bottlerocket, security updates can be automatically applied as soon as they’re available in a minimally disruptive manner and be rolled back if failures occur.
Premium support – AWS provided builds of Bottlerocket on Amazon EC2 is covered under the same AWS Support plans that also cover AWS services such as Amazon EC2, Amazon EKS, and Amazon ECR.

Considerations

Consider the following when using Bottlerocket for your AMI type:

Bottlerocket supports Amazon EC2 instances with x86_64 and arm64 processors. The Bottlerocket AMI isn’t recommended for use with Amazon EC2 instances with an Inferentia chip.
Bottlerocket images don’t include an SSH server or a shell. You can employ out-of-band access methods to allow SSH. These approaches enable the admin container and to pass some bootstrapping configuration steps with user data. For more information, refer to the following sections in Bottlerocket OS on GitHub:
Bottlerocket uses different container types:
- By default, a control container is enabled. This container runs the AWS Systems Manager agent that you can use to run commands or start shell sessions on Amazon EC2 Bottlerocket instances. For more information, see Setting up Session Manager in the AWS Systems Manager User Guide.
- If an SSH key is given when creating the node group, an admin container is enabled. We recommend using the admin container only for development and testing scenarios. We don’t recommend using it for production environments. For more information, see Admin container on GitHub.

More information

For more information about using Amazon EKS optimized Bottlerocket AMIs, see the following sections:

For details about Bottlerocket, see the Bottlerocket Documentation.
For version information resources, see eks-ami-versions-bottlerocket.title.
To use Bottlerocket with managed node groups, see managed-node-groups.title.
To launch self-managed Bottlerocket nodes, see launch-node-bottlerocket.title.
To retrieve the latest IDs of the Amazon EKS optimized Bottlerocket AMIs, see retrieve-ami-id-bottlerocket.title.
For details on compliance support, see bottlerocket-compliance-support.title.

Retrieve `Bottlerocket` AMI version information

This topic gives resources for Amazon EKS optimized Bottlerocket AMIs version information.

Each Bottlerocket AMI release includes various versions of kubelet, the Bottlerocket kernel, and containerd. Accelerated AMI variants also include various versions of the NVIDIA driver. You can find this version information in the OS topic of the Bottlerocket Documentation. From this page, navigate to the applicable Version Information sub-topic.

The Bottlerocket Documentation can sometimes lag behind the versions that are available on GitHub. You can find a list of changes for the latest versions in the releases on GitHub.

Retrieve recommended `Bottlerocket` AMI IDs

You can retrieve the Amazon Machine Image (AMI) ID for Amazon EKS optimized AMIs by querying the AWS Systems Manager Parameter Store API.

You can retrieve the image ID of the latest recommended Amazon EKS optimized Bottlerocket AMI with the following AWS CLI command, which uses the sub-parameter image_id. Make the following modifications to the command as needed and then run the modified command:

Replace kubernetes-version with a supported Amazon EKS version.
Replace -flavor with one of the following options.
- Remove -flavor for variants without a GPU.
- Use -nvidia for GPU-enabled variants.
- Use -fips for FIPS-enabled variants.
Replace architecture with one of the following options.
- Use x86_64 for x86 based instances.
- Use arm64 for ARM instances.
Replace region-code with an Amazon EKS supported AWS Region for which you want the AMI ID.

aws ssm get-parameter --name /aws/service/bottlerocket/aws-k8s-kubernetes-version-flavor/architecture/latest/image_id \
    --region region-code --query "Parameter.Value" --output text

Here’s an example command after placeholder replacements have been made.

aws ssm get-parameter --name /aws/service/bottlerocket/aws-k8s-1.31/x86_64/latest/image_id \
    --region us-west-2 --query "Parameter.Value" --output text

An example output is as follows.

ami-1234567890abcdef0

Meet compliance requirements with `Bottlerocket`

Bottlerocket complies with recommendations defined by various organizations.

Bottlerocket complies with recommendations defined by various organizations:

There is a CIS Benchmark defined for Bottlerocket. In a default configuration, Bottlerocket image has most of the controls required by CIS Level 1 configuration profile. You can implement the controls required for a CIS Level 2 configuration profile. For more information, see Validating Amazon EKS optimized Bottlerocket AMI against the CIS Benchmark on the AWS blog.
The optimized feature set and reduced attack surface means that Bottlerocket instances require less configuration to satisfy PCI DSS requirements. The CIS Benchmark for Bottlerocket is an excellent resource for hardening guidance, and supports your requirements for secure configuration standards under PCI DSS requirement 2.2. You can also leverage Fluent Bit to support your requirements for operating system level audit logging under PCI DSS requirement 10.2. AWS publishes new (patched) Bottlerocket instances periodically to help you meet PCI DSS requirement 6.2 (for v3.2.1) and requirement 6.3.3 (for v4.0).
Bottlerocket is an HIPAA-eligible feature authorized for use with regulated workloads for both Amazon EC2 and Amazon EKS. For more information, see the Architecting for HIPAA Security and Compliance on Amazon EKS whitepaper.
Bottlerocket AMIs are available that are preconfigured to use FIPS 140-3 validated cryptographic modules. This includes the Amazon Linux 2023 Kernel Crypto API Cryptographic Module and the AWS-LC Cryptographic Module. For more information on selecting FIPS-enabled variants, see retrieve-ami-id-bottlerocket.title.

9.6.4. Create nodes with optimized `Ubuntu Linux` AMIs

Canonical has partnered with Amazon EKS to create node AMIs that you can use in your clusters.

Canonical has partnered with Amazon EKS to create node AMIs that you can use in your clusters.

Canonical delivers a built-for-purpose Kubernetes Node OS image. This minimized Ubuntu image is optimized for Amazon EKS and includes the custom AWS kernel that is jointly developed with AWS. For more information, see Ubuntu on Amazon Elastic Kubernetes Service (EKS) and launch-node-ubuntu.title . For information about support, see the Third-party software section of the AWS Premium Support FAQs.

9.6.5. Create nodes with optimized `Windows` AMIs

Windows Amazon EKS optimized AMIs are built on top of Windows Server 2019.

Windows Amazon EKS optimized AMIs are built on top of Windows Server 2019 and Windows Server 2022. They are configured to serve as the base image for Amazon EKS nodes. By default, the AMIs include the following components:

You can track security or privacy events for Windows Server with the Microsoft security update guide.

Amazon EKS offers AMIs that are optimized for Windows containers in the following variants:

Amazon EKS-optimized Windows Server 2019 Core AMI
Amazon EKS-optimized Windows Server 2019 Full AMI
Amazon EKS-optimized Windows Server 2022 Core AMI
Amazon EKS-optimized Windows Server 2022 Full AMI

The Amazon EKS-optimized Windows Server 20H2 Core AMI is deprecated. No new versions of this AMI will be released.
To ensure that you have the latest security updates by default, Amazon EKS maintains optimized Windows AMIs for the last 4 months. Each new AMI will be available for 4 months from the time of initial release. After this period, older AMIs are made private and are no longer accessible. We encourage using the latest AMIs to avoid security vulnerabilities and losing access to older AMIs which have reached the end of their supported lifetime. While we can’t guarantee that we can provide access to AMIs that have been made private, you can request access by filing a ticket with AWS Support.

Release calendar

The following table lists the release and end of support dates for Windows versions on Amazon EKS. If an end date is blank, it’s because the version is still supported.

Windows version Amazon EKS release Amazon EKS end of support

Windows Server 2022 Core

10/17/2022

Windows Server 2022 Full

10/17/2022

Windows Server 20H2 Core

8/12/2021

8/9/2022

Windows Server 2004 Core

8/19/2020

12/14/2021

Windows Server 2019 Core

10/7/2019

Windows Server 2019 Full

10/7/2019

Windows Server 1909 Core

10/7/2019

12/8/2020

Bootstrap script configuration parameters

When you create a Windows node, there’s a script on the node that allows for configuring different parameters. Depending on your setup, this script can be found on the node at a location similar to: C:\Program Files\Amazon\EKS\Start-EKSBootstrap.ps1. You can specify custom parameter values by specifying them as arguments to the bootstrap script. For example, you can update the user data in the launch template. For more information, see launch-template-user-data.title.

The script includes the following command-line parameters:

-EKSClusterName – Specifies the Amazon EKS cluster name for this worker node to join.
-KubeletExtraArgs – Specifies extra arguments for kubelet (optional).
-KubeProxyExtraArgs – Specifies extra arguments for kube-proxy (optional).
-APIServerEndpoint – Specifies the Amazon EKS cluster API server endpoint (optional). Only valid when used with -Base64ClusterCA. Bypasses calling Get-EKSCluster.
-Base64ClusterCA – Specifies the base64 encoded cluster CA content (optional). Only valid when used with -APIServerEndpoint. Bypasses calling Get-EKSCluster.
-DNSClusterIP – Overrides the IP address to use for DNS queries within the cluster (optional). Defaults to 10.100.0.10 or 172.20.0.10 based on the IP address of the primary interface.
-ServiceCIDR – Overrides the Kubernetes service IP address range from which cluster services are addressed. Defaults to 172.20.0.0/16 or 10.100.0.0/16 based on the IP address of the primary interface.
-ExcludedSnatCIDRs – A list of IPv4 CIDRs to exclude from Source Network Address Translation (SNAT). This means that the pod private IP which is VPC addressable wouldn’t be translated to the IP address of the instance ENI’s primary IPv4 address for outbound traffic. By default, the IPv4 CIDR of the VPC for the Amazon EKS Windows node is added. Specifying CIDRs to this parameter also additionally excludes the specified CIDRs. For more information, see external-snat.title.

In addition to the command line parameters, you can also specify some environment variable parameters. When specifying a command line parameter, it takes precedence over the respective environment variable. The environment variable(s) should be defined as machine (or system) scoped as the bootstrap script will only read machine-scoped variables.

The script takes into account the following environment variables:

SERVICE_IPV4_CIDR – Refer to the ServiceCIDR command line parameter for the definition.
EXCLUDED_SNAT_CIDRS – Should be a comma separated string. Refer to the ExcludedSnatCIDRs command line parameter for the definition.

`gMSA` authentication support

Amazon EKS Windows Pods allow different types of group Managed Service Account (gMSA) authentication.

Amazon EKS supports Active Directory domain identities for authentication. For more information on domain-joined gMSA, see Windows Authentication on Amazon EKS Windowspods on the AWS blog.
Amazon EKS offers a plugin that enables non-domain-joined Windows nodes to retrieve gMSA credentials with a portable user identity. For more information on domainless gMSA, see Domainless Windows Authentication for Amazon EKS Windowspods on the AWS blog.

Cached container images

Amazon EKS Windows optimized AMIs have certain container images cached for the containerd runtime. Container images are cached when building custom AMIs using Amazon-managed build components. For more information, see custom-windows-ami-build-component.title.

The following cached container images are for the containerd runtime:

amazonaws.com/eks/pause-windows
mcr.microsoft.com/windows/nanoserver
mcr.microsoft.com/windows/servercore

More information

For more information about using Amazon EKS optimized Windows AMIs, see the following sections:

For details on running workloads on Amazon EKS optimized accelerated Windows AMIs, see ml-eks-windows-optimized-ami.title.
To use Windows with managed node groups, see managed-node-groups.title.
To launch self-managed Windows nodes, see launch-windows-workers.title.
For version information, see eks-ami-versions-windows.title.
To retrieve the latest IDs of the Amazon EKS optimized Windows AMIs, see retrieve-windows-ami-id.title.
To use Amazon EC2 Image Builder to create custom Amazon EKS optimized Windows AMIs, see eks-custom-ami-windows.title.
For best practices, see Amazon EKS optimized Windows AMI management in the EKS Best Practices Guide.

Create self-managed `Windows` Server 2022 nodes with `eksctl`

This topic includes a YAML file as reference for creating self-managed Windows Server 2022 nodes.

You can use the following test-windows-2022.yaml as reference for creating self-managed Windows Server 2022 nodes. Replace every example value with your own values.

You must use eksctl version 0.116.0 or later to run self-managed Windows Server 2022 nodes.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: windows-2022-cluster
  region: region-code
  version: '1.31'

nodeGroups:
  - name: windows-ng
    instanceType: m5.2xlarge
    amiFamily: WindowsServer2022FullContainer
    volumeSize: 100
    minSize: 2
    maxSize: 3
  - name: linux-ng
    amiFamily: AmazonLinux2
    minSize: 2
    maxSize: 3

The node groups can then be created using the following command.

eksctl create cluster -f test-windows-2022.yaml

Retrieve `Windows` AMI version information

This topic lists versions of the Amazon EKS optimized Windows AMIs and their corresponding versions of kubelet, containerd, and csi-proxy.

Extended Support for Amazon EKS optimized Windows AMIs that are published by AWS isn’t available for Kubernetes version 1.23 but is available for Kubernetes version 1.24 and higher.

This topic lists versions of the Amazon EKS optimized Windows AMIs and their corresponding versions of kubelet, containerd, and csi-proxy.

The Amazon EKS optimized AMI metadata, including the AMI ID, for each variant can be retrieved programmatically. For more information, see retrieve-windows-ami-id.title.

AMIs are versioned by Kubernetes version and the release date of the AMI in the following format:

k8s_major_version.k8s_minor_version-release_date

Amazon EKS managed node groups support the November 2022 and later releases of the Windows AMIs.

Amazon EKS optimized `Windows` Server 2022 Core AMI

The following tables list the current and previous versions of the Amazon EKS optimized Windows Server 2022 Core AMI.

Kubernetes version 1.31

AMI version kubelet version containerd version csi-proxy version Release notes

1.31-2025-01-01

1.31.4

1.7.20

1.1.3

Includes patches for CVE-2024-9042.

1.31-2024.12.13

1.31.3

1.7.20

1.1.3

1.31-2024.11.12

1.31.1

1.7.20

1.1.3

1.31-2024.10.08

1.31.1

1.7.20

1.1.3

1.31-2024.10.01

1.31.1

1.7.20

1.1.3

1.31-2024.09.10

1.31.0

1.7.20

1.1.3

Kubernetes version 1.30

AMI version kubelet version containerd version csi-proxy version Release notes

1.30-2025-01-01

1.30.8

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.30-2024.12.11

1.30.7

1.7.14

1.1.3

1.30-2024.11.12

1.30.4

1.7.14

1.1.3

1.30-2024.10.08

1.30.4

1.7.14

1.1.3

1.30-2024.09.10

1.30.2

1.7.14

1.1.3

1.30-2024.08.13

1.30.2

1.7.14

1.1.3

1.30-2024.07.10

1.30.2

1.7.14

1.1.2

Includes patches for CVE-2024-5321.

1.30-2024.06.17

1.30.0

1.7.14

1.1.2

Upgraded containerd to 1.7.14.

1.30-2024.05.15

1.30.0

1.6.28

1.1.2

Kubernetes version 1.29

AMI version kubelet version containerd version csi-proxy version Release notes

1.29-2025.01.01

1.29.12

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.29-2024.12.11

1.29.10

1.7.14

1.1.3

1.29-2024.11.12

1.29.8

1.7.14

1.1.3

1.29-2024.10.08

1.29.8

1.7.14

1.1.3

1.29-2024.09.10

1.29.6

1.7.14

1.1.3

1.29-2024.08.13

1.29.6

1.7.14

1.1.3

1.29-2024.07.10

1.29.6

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.29-2024.06.17

1.29.3

1.7.11

1.1.2

1.29-2024.05.15

1.29.3

1.7.11

1.1.2

Upgraded containerd to 1.7.11. Upgraded kubelet to 1.29.3.

1.29-2024.04.09

1.29.0

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.29-2024.03.12

1.29.0

1.6.25

1.1.2

1.29-2024.02.13

1.29.0

1.6.25

1.1.2

1.29-2024.02.06

1.29.0

1.6.25

1.1.2

Fixed a bug where the pause image was incorrectly deleted by kubelet garbage collection process.

1.29-2024.01.11

1.29.0

1.6.18

1.1.2

Excluded Standalone Windows Update KB5034439 on Windows Server 2022 Core AMIs. The KB applies only to Windows installations with a separate WinRE partition, which aren’t included with any of our Amazon EKS Optimized Windows AMIs.

Kubernetes version 1.28

AMI version kubelet version containerd version csi-proxy version Release notes

1.28-2025-01-01

1.28.15

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.28-2024.12.11

1.28.15

1.7.14

1.1.3

1.28-2024.11.12

1.28.13

1.7.14

1.1.3

1.28-2024.10.08

1.28.13

1.7.14

1.1.3

1.28-2024.09.10

1.28.11

1.7.14

1.1.3

1.28-2024.08.13

1.28.11

1.7.14

1.1.3

1.28-2024.07.10

1.28.11

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.28-2024.06.17

1.28.8

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.28-2024.05.14

1.28.8

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.28.8.

1.28-2024.04.09

1.28.5

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.28-2024.03.12

1.28.5

1.6.18

1.1.2

1.28-2024.02.13

1.28.5

1.6.18

1.1.2

1.28-2024.01.11

1.28.5

1.6.18

1.1.2

1.28-2023.12.12

1.28.3

1.6.18

1.1.2

1.28-2023.11.14

1.28.3

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.28-2023.10.19

1.28.2

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.28-2023-09.27

1.28.2

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.28-2023.09.12

1.28.1

1.6.6

1.1.2

Kubernetes version 1.27

AMI version kubelet version containerd version csi-proxy version Release notes

1.27-2025-01-01

1.27.16

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.27-2024.12.11

1.27.16

1.7.14

1.1.3

1.27-2024.11.12

1.27.16

1.7.14

1.1.3

1.27-2024.10.08

1.27.16

1.7.14

1.1.3

1.27-2024.09.10

1.27.15

1.7.14

1.1.3

1.27-2024.08.13

1.27.15

1.7.14

1.1.3

1.27-2024.07.10

1.27.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.27-2024.06.17

1.27.12

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.27-2024.05.14

1.27.12

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.27.12.

1.27-2024.04.09

1.27.9

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.27-2024.03.12

1.27.9

1.6.18

1.1.2

1.27-2024.02.13

1.27.9

1.6.18

1.1.2

1.27-2024.01.11

1.27.9

1.6.18

1.1.2

1.27-2023.12.12

1.27.7

1.6.18

1.1.2

1.27-2023.11.14

1.27.7

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.27-2023.10.19

1.27.6

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.27-2023-09.27

1.27.6

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.27-2023.09.12

1.27.4

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.27-2023.08.17

1.27.4

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.27-2023.08.08

1.27.3

1.6.6

1.1.1

1.27-2023.07.11

1.27.3

1.6.6

1.1.1

1.27-2023.06.20

1.27.1

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.27-2023.06.14

1.27.1

1.6.6

1.1.1

Added support for host port mapping in CNI. Merged pull request #93.

1.27-2023.06.06

1.27.1

1.6.6

1.1.1

Fixed containers-roadmap issue #2042, which caused nodes to fail pulling private Amazon ECR images.

1.27-2023.05.17

1.27.1

1.6.6

1.1.1

Kubernetes version 1.26

AMI version kubelet version containerd version csi-proxy version Release notes

1.26-2024.12.11

1.26.15

1.7.14

1.1.3

1.26-2024.11.12

1.26.15

1.7.14

1.1.3

1.26-2024.10.08

1.26.15

1.7.14

1.1.3

1.26-2024.09.10

1.26.15

1.7.14

1.1.3

1.26-2024.08.13

1.26.15

1.7.14

1.1.3

1.26-2024.07.10

1.26.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.26-2024.06.17

1.26.15

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.26-2024.05.14

1.26.15

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.26.15.

1.26-2024.04.09

1.26.12

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.26-2024.03.12

1.26.12

1.6.18

1.1.2

1.26-2024.02.13

1.26.12

1.6.18

1.1.2

1.26-2024.01.11

1.26.12

1.6.18

1.1.2

1.26-2023.12.12

1.26.10

1.6.18

1.1.2

1.26-2023.11.14

1.26.10

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.26-2023.10.19

1.26.9

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.26.9. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.26-2023.09.12

1.26.7

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.26-2023.08.17

1.26.7

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.26-2023.08.08

1.26.6

1.6.6

1.1.1

1.26-2023.07.11

1.26.6

1.6.6

1.1.1

1.26-2023.06.20

1.26.4

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.26-2023.06.14

1.26.4

1.6.6

1.1.1

Upgraded Kubernetes to 1.26.4. Added support for host port mapping in CNI. Merged pull request #93.

1.26-2023.05.09

1.26.2

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.26-2023.04.26

1.26.2

1.6.6

1.1.1

1.26-2023.04.11

1.26.2

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.26-2023.03.24

1.26.2

1.6.6

1.1.1

Kubernetes version 1.25

AMI version kubelet version containerd version csi-proxy version Release notes

1.25-2024.12.13

1.25.16

1.7.14

1.1.3

1.25-2024.11.12

1.25.16

1.7.14

1.1.3

1.25-2024.10.08

1.25.16

1.7.14

1.1.3

1.25-2024.09.10

1.25.16

1.7.14

1.1.3

1.25-2024.08.13

1.25.16

1.7.14

1.1.3

1.25-2024.07.10

1.25.16

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.25-2024.06.17

1.25.16

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.25-2024.05.14

1.25.16

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.25-2024.04.09

1.25.16

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.25-2024.03.12

1.25.16

1.6.18

1.1.2

1.25-2024.02.13

1.25.16

1.6.18

1.1.2

1.25-2024.01.11

1.25.16

1.6.18

1.1.2

1.25-2023.12.12

1.25.15

1.6.18

1.1.2

1.25-2023.11.14

1.25.15

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.25-2023.10.19

1.25.14

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.25.14. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.25-2023.09.12

1.25.12

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.25-2023.08.17

1.25.12

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.25-2023.08.08

1.25.9

1.6.6

1.1.1

1.25-2023.07.11

1.25.9

1.6.6

1.1.1

1.25-2023.06.20

1.25.9

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.25-2023.06.14

1.25.9

1.6.6

1.1.1

Upgraded Kubernetes to 1.25.9. Added support for host port mapping in CNI. Merged pull request #93.

1.25-2023.05.09

1.25.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.25-2023.04.11

1.25.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.25-2023.03.27

1.25.6

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.25-2023.03.20

1.25.6

1.6.6

1.1.1

1.25-2023.02.14

1.25.6

1.6.6

1.1.1

Kubernetes version 1.24

AMI version kubelet version containerd version csi-proxy version Release notes

1.24-2024.12.11

1.24.17

1.7.14

1.1.3

1.24-2024.11.12

1.24.17

1.7.14

1.1.3

1.24-2024.10.08

1.24.17

1.7.14

1.1.3

1.24-2024.09.10

1.24.17

1.7.14

1.1.3

1.24-2024.08.13

1.24.17

1.7.14

1.1.3

1.24-2024.07.10

1.24.17

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.24-2024.06.17

1.24.17

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.24-2024.05.14

1.24.17

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.24-2024.04.09

1.24.17

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.24-2024.03.12

1.24.17

1.6.18

1.1.2

1.24-2024.02.13

1.24.17

1.6.18

1.1.2

1.24-2024.01.11

1.24.17

1.6.18

1.1.2

1.24-2023.12.12

1.24.17

1.6.18

1.1.2

1.24-2023.11.14

1.24.17

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.24-2023.10.19

1.24.17

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.24.17. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.24-2023.09.12

1.24.16

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.24-2023.08.17

1.24.16

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.24-2023.08.08

1.24.13

1.6.6

1.1.1

1.24-2023.07.11

1.24.13

1.6.6

1.1.1

1.24-2023.06.20

1.24.13

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.24-2023.06.14

1.24.13

1.6.6

1.1.1

Upgraded Kubernetes to 1.24.13. Added support for host port mapping in CNI. Merged pull request #93.

1.24-2023.05.09

1.24.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.24-2023.04.11

1.24.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.24-2023.03.27

1.24.7

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.24-2023.03.20

1.24.7

1.6.6

1.1.1

Kubernetes version downgraded to 1.24.7 because 1.24.10 has a reported issue in kube-proxy.

1.24-2023.02.14

1.24.10

1.6.6

1.1.1

1.24-2023.01.23

1.24.7

1.6.6

1.1.1

1.24-2023.01.11

1.24.7

1.6.6

1.1.1

1.24-2022.12.13

1.24.7

1.6.6

1.1.1

1.24-2022.10.11

1.24.7

1.6.6

1.1.1

Amazon EKS optimized `Windows` Server 2022 Full AMI

The following tables list the current and previous versions of the Amazon EKS optimized Windows Server 2022 Full AMI.

Kubernetes version 1.31

AMI version kubelet version containerd version csi-proxy version Release notes

1.31-2025-01-01

1.31.4

1.7.20

1.1.3

Includes patches for CVE-2024-9042.

1.31-2024.12.13

1.31.3

1.7.20

1.1.3

1.31-2024.11.12

1.31.1

1.7.20

1.1.3

1.31-2024.10.08

1.31.1

1.7.20

1.1.3

1.31-2024.10.01

1.31.1

1.7.20

1.1.3

1.31-2024.09.10

1.31.0

1.7.20

1.1.3

Kubernetes version 1.30

AMI version kubelet version containerd version csi-proxy version Release notes

1.30-2025-01-01

1.30.8

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.30-2024.12.11

1.30.7

1.7.14

1.1.3

1.30-2024.11.12

1.30.4

1.7.14

1.1.3

1.30-2024.10.08

1.30.4

1.7.14

1.1.3

1.30-2024.09.10

1.30.2

1.7.14

1.1.3

1.30-2024.08.13

1.30.2

1.7.14

1.1.3

1.30-2024.07.10

1.30.2

1.7.14

1.1.2

Includes patches for CVE-2024-5321.

1.30-2024.06.17

1.30.0

1.7.14

1.1.2

Upgraded containerd to 1.7.14.

1.30-2024.05.15

1.30.0

1.6.28

1.1.2

Kubernetes version 1.29

AMI version kubelet version containerd version csi-proxy version Release notes

1.29-2025.01.01

1.29.12

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.29-2024.12.11

1.29.10

1.7.14

1.1.3

1.29-2024.11.12

1.29.8

1.7.14

1.1.3

1.29-2024.10.08

1.29.8

1.7.14

1.1.3

1.29-2024.09.10

1.29.6

1.7.14

1.1.3

1.29-2024.08.13

1.29.6

1.7.14

1.1.3

1.29-2024.07.10

1.29.6

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.29-2024.06.17

1.29.3

1.7.11

1.1.2

1.29-2024.05.15

1.29.3

1.7.11

1.1.2

Upgraded containerd to 1.7.11. Upgraded kubelet to 1.29.3.

1.29-2024.04.09

1.29.0

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.29-2024.03.12

1.29.0

1.6.25

1.1.2

1.29-2024.02.13

1.29.0

1.6.25

1.1.2

1.29-2024.02.06

1.29.0

1.6.25

1.1.2

Fixed a bug where the pause image was incorrectly deleted by kubelet garbage collection process.

1.29-2024.01.09

1.29.0

1.6.18

1.1.2

Kubernetes version 1.28

AMI version kubelet version containerd version csi-proxy version Release notes

1.28-2025-01-01

1.28.15

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.28-2024.12.11

1.28.15

1.7.14

1.1.3

1.28-2024.11.12

1.28.13

1.7.14

1.1.3

1.28-2024.10.08

1.28.13

1.7.14

1.1.3

1.28-2024.09.10

1.28.11

1.7.14

1.1.3

1.28-2024.08.13

1.28.11

1.7.14

1.1.3

1.28-2024.07.10

1.28.11

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.28-2024.06.17

1.28.8

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.28-2024.05.14

1.28.8

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.28.8.

1.28-2024.04.09

1.28.5

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.28-2024.03.12

1.28.5

1.6.18

1.1.2

1.28-2024.02.13

1.28.5

1.6.18

1.1.2

1.28-2024.01.09

1.28.5

1.6.18

1.1.2

1.28-2023.12.12

1.28.3

1.6.18

1.1.2

1.28-2023.11.14

1.28.3

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.28-2023.10.19

1.28.2

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.28-2023-09.27

1.28.2

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.28-2023.09.12

1.28.1

1.6.6

1.1.2

Kubernetes version 1.27

AMI version kubelet version containerd version csi-proxy version Release notes

1.27-2025-01-01

1.27.16

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.27-2024.12.11

1.27.16

1.7.14

1.1.3

1.27-2024.11.12

1.27.16

1.7.14

1.1.3

1.27-2024.10.08

1.27.16

1.7.14

1.1.3

1.27-2024.09.10

1.27.15

1.7.14

1.1.3

1.27-2024.08.13

1.27.15

1.7.14

1.1.3

1.27-2024.07.10

1.27.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.27-2024.06.17

1.27.12

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.27-2024.05.14

1.27.12

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.27.12.

1.27-2024.04.09

1.27.9

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.27-2024.03.12

1.27.9

1.6.18

1.1.2

1.27-2024.02.13

1.27.9

1.6.18

1.1.2

1.27-2024.01.09

1.27.9

1.6.18

1.1.2

1.27-2023.12.12

1.27.7

1.6.18

1.1.2

1.27-2023.11.14

1.27.7

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.27-2023.10.19

1.27.6

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.27-2023-09.27

1.27.6

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.27-2023.09.12

1.27.4

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.27-2023.08.17

1.27.4

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.27-2023.08.08

1.27.3

1.6.6

1.1.1

1.27-2023.07.11

1.27.3

1.6.6

1.1.1

1.27-2023.06.20

1.27.1

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.27-2023.06.14

1.27.1

1.6.6

1.1.1

Added support for host port mapping in CNI. Merged pull request #93.

1.27-2023.06.06

1.27.1

1.6.6

1.1.1

Fixed containers-roadmap issue #2042, which caused nodes to fail pulling private Amazon ECR images.

1.27-2023.05.18

1.27.1

1.6.6

1.1.1

Kubernetes version 1.26

AMI version kubelet version containerd version csi-proxy version Release notes

1.26-2024.12.11

1.26.15

1.7.14

1.1.3

1.26-2024.11.12

1.26.15

1.7.14

1.1.3

1.26-2024.10.08

1.26.15

1.7.14

1.1.3

1.26-2024.09.10

1.26.15

1.7.14

1.1.3

1.26-2024.08.13

1.26.15

1.7.14

1.1.3

1.26-2024.07.10

1.26.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.26-2024.06.17

1.26.15

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.26-2024.05.14

1.26.15

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.26.15.

1.26-2024.04.09

1.26.12

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.26-2024.03.12

1.26.12

1.6.18

1.1.2

1.26-2024.02.13

1.26.12

1.6.18

1.1.2

1.26-2024.01.09

1.26.12

1.6.18

1.1.2

1.26-2023.12.12

1.26.10

1.6.18

1.1.2

1.26-2023.11.14

1.26.10

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.26-2023.10.19

1.26.9

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.26.9. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.26-2023.09.12

1.26.7

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.26-2023.08.17

1.26.7

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.26-2023.08.08

1.26.6

1.6.6

1.1.1

1.26-2023.07.11

1.26.6

1.6.6

1.1.1

1.26-2023.06.20

1.26.4

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.26-2023.06.14

1.26.4

1.6.6

1.1.1

Upgraded Kubernetes to 1.26.4. Added support for host port mapping in CNI. Merged pull request #93.

1.26-2023.05.09

1.26.2

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.26-2023.04.26

1.26.2

1.6.6

1.1.1

1.26-2023.04.11

1.26.2

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.26-2023.03.24

1.26.2

1.6.6

1.1.1

Kubernetes version 1.25

AMI version kubelet version containerd version csi-proxy version Release notes

1.25-2024.12.13

1.25.16

1.7.14

1.1.3

1.25-2024.11.12

1.25.16

1.7.14

1.1.3

1.25-2024.10.08

1.25.16

1.7.14

1.1.3

1.25-2024.09.10

1.25.16

1.7.14

1.1.3

1.25-2024.08.13

1.25.16

1.7.14

1.1.3

1.25-2024.07.10

1.25.16

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.25-2024.06.17

1.25.16

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.25-2024.05.14

1.25.16

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.25-2024.04.09

1.25.16

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.25-2024.03.12

1.25.16

1.6.18

1.1.2

1.25-2024.02.13

1.25.16

1.6.18

1.1.2

1.25-2024.01.09

1.25.16

1.6.18

1.1.2

1.25-2023.12.12

1.25.15

1.6.18

1.1.2

1.25-2023.11.14

1.25.15

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.25-2023.10.19

1.25.14

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.25.14. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.25-2023.09.12

1.25.12

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.25-2023.08.17

1.25.12

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.25-2023.08.08

1.25.9

1.6.6

1.1.1

1.25-2023.07.11

1.25.9

1.6.6

1.1.1

1.25-2023.06.20

1.25.9

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.25-2023.06.14

1.25.9

1.6.6

1.1.1

Upgraded Kubernetes to 1.25.9. Added support for host port mapping in CNI. Merged pull request #93.

1.25-2023.05.09

1.25.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.25-2023.04.11

1.25.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.25-2023.03.27

1.25.6

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.25-2023.03.20

1.25.6

1.6.6

1.1.1

1.25-2023.02.14

1.25.6

1.6.6

1.1.1

Kubernetes version 1.24

AMI version kubelet version containerd version csi-proxy version Release notes

1.24-2024.12.11

1.24.17

1.7.14

1.1.3

1.24-2024.11.12

1.24.17

1.7.14

1.1.3

1.24-2024.10.08

1.24.17

1.7.14

1.1.3

1.24-2024.09.10

1.24.17

1.7.14

1.1.3

1.24-2024.08.13

1.24.17

1.7.14

1.1.3

1.24-2024.07.10

1.24.17

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.24-2024.06.17

1.24.17

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.24-2024.05.14

1.24.17

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.24-2024.04.09

1.24.17

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.24-2024.03.12

1.24.17

1.6.18

1.1.2

1.24-2024.02.13

1.24.17

1.6.18

1.1.2

1.24-2024.01.09

1.24.17

1.6.18

1.1.2

1.24-2023.12.12

1.24.17

1.6.18

1.1.2

1.24-2023.11.14

1.24.17

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.24-2023.10.19

1.24.17

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.24.17. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.24-2023.09.12

1.24.16

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.24-2023.08.17

1.24.16

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.24-2023.08.08

1.24.13

1.6.6

1.1.1

1.24-2023.07.11

1.24.13

1.6.6

1.1.1

1.24-2023.06.20

1.24.13

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.24-2023.06.14

1.24.13

1.6.6

1.1.1

Upgraded Kubernetes to 1.24.13. Added support for host port mapping in CNI. Merged pull request #93.

1.24-2023.05.09

1.24.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.24-2023.04.11

1.24.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.24-2023.03.27

1.24.7

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.24-2023.03.20

1.24.7

1.6.6

1.1.1

Kubernetes version downgraded to 1.24.7 because 1.24.10 has a reported issue in kube-proxy.

1.24-2023.02.14

1.24.10

1.6.6

1.1.1

1.24-2023.01.23

1.24.7

1.6.6

1.1.1

1.24-2023.01.11

1.24.7

1.6.6

1.1.1

1.24-2022.12.14

1.24.7

1.6.6

1.1.1

1.24-2022.10.11

1.24.7

1.6.6

1.1.1

Amazon EKS optimized `Windows` Server 2019 Core AMI

The following tables list the current and previous versions of the Amazon EKS optimized Windows Server 2019 Core AMI.

Kubernetes version 1.31

AMI version kubelet version containerd version csi-proxy version Release notes

1.31-2025-01-01

1.31.4

1.7.20

1.1.3

Includes patches for CVE-2024-9042.

1.31-2024.12.13

1.31.3

1.7.20

1.1.3

1.31-2024.11.12

1.31.1

1.7.20

1.1.3

1.31-2024.10.08

1.31.1

1.7.20

1.1.3

1.31-2024.10.01

1.31.1

1.7.20

1.1.3

1.31-2024.09.10

1.31.0

1.7.20

1.1.3

Kubernetes version 1.30

AMI version kubelet version containerd version csi-proxy version Release notes

1.30-2025-01-01

1.30.8

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.30-2024.12.11

1.30.7

1.7.14

1.1.3

1.30-2024.11.12

1.30.4

1.7.14

1.1.3

1.30-2024.10.08

1.30.4

1.7.14

1.1.3

1.30-2024.09.10

1.30.2

1.7.14

1.1.3

1.30-2024.08.13

1.30.2

1.7.14

1.1.3

1.30-2024.07.10

1.30.2

1.7.14

1.1.2

Includes patches for CVE-2024-5321.

1.30-2024.06.17

1.30.0

1.7.14

1.1.2

Upgraded containerd to 1.7.14.

1.30-2024.05.15

1.30.0

1.6.28

1.1.2

Kubernetes version 1.29

AMI version kubelet version containerd version csi-proxy version Release notes

1.29-2025.01.01

1.29.12

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.29-2024.12.11

1.29.10

1.7.14

1.1.3

1.29-2024.11.12

1.29.8

1.7.14

1.1.3

1.29-2024.10.08

1.29.8

1.7.14

1.1.3

1.29-2024.09.10

1.29.6

1.7.14

1.1.3

1.29-2024.08.13

1.29.6

1.7.14

1.1.3

1.29-2024.07.10

1.29.6

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.29-2024.06.17

1.29.3

1.7.11

1.1.2

1.29-2024.05.15

1.29.3

1.7.11

1.1.2

Upgraded containerd to 1.7.11. Upgraded kubelet to 1.29.3.

1.29-2024.04.09

1.29.0

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.29-2024.03.13

1.29.0

1.6.25

1.1.2

1.29-2024.02.13

1.29.0

1.6.25

1.1.2

1.29-2024.02.06

1.29.0

1.6.25

1.1.2

Fixed a bug where the pause image was incorrectly deleted by kubelet garbage collection process.

1.29-2024.01.09

1.29.0

1.6.18

1.1.2

Kubernetes version 1.28

AMI version kubelet version containerd version csi-proxy version Release notes

1.28-2025-01-01

1.28.15

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.28-2024.12.11

1.28.15

1.7.14

1.1.3

1.28-2024.11.12

1.28.13

1.7.14

1.1.3

1.28-2024.10.08

1.28.13

1.7.14

1.1.3

1.28-2024.09.10

1.28.11

1.7.14

1.1.3

1.28-2024.08.13

1.28.11

1.7.14

1.1.3

1.28-2024.07.10

1.28.11

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.28-2024.06.17

1.28.8

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.28-2024.05.14

1.28.8

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.28.8.

1.28-2024.04.09

1.28.5

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.28-2024.03.13

1.28.5

1.6.18

1.1.2

1.28-2024.02.13

1.28.5

1.6.18

1.1.2

1.28-2024.01.09

1.28.5

1.6.18

1.1.2

1.28-2023.12.12

1.28.3

1.6.18

1.1.2

1.28-2023.11.14

1.28.3

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.28-2023.10.19

1.28.2

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.28-2023-09.27

1.28.2

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.28-2023.09.12

1.28.1

1.6.6

1.1.2

Kubernetes version 1.27

AMI version kubelet version containerd version csi-proxy version Release notes

1.27-2025-01-01

1.27.16

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.27-2024.12.11

1.27.16

1.7.14

1.1.3

1.27-2024.11.12

1.27.16

1.7.14

1.1.3

1.27-2024.10.08

1.27.16

1.7.14

1.1.3

1.27-2024.09.10

1.27.15

1.7.14

1.1.3

1.27-2024.08.13

1.27.15

1.7.14

1.1.3

1.27-2024.07.10

1.27.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.27-2024.06.17

1.27.12

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.27-2024.05.14

1.27.12

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.27.12.

1.27-2024.04.09

1.27.9

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.27-2024.03.13

1.27.9

1.6.18

1.1.2

1.27-2024.02.13

1.27.9

1.6.18

1.1.2

1.27-2024.01.09

1.27.9

1.6.18

1.1.2

1.27-2023.12.12

1.27.7

1.6.18

1.1.2

1.27-2023.11.14

1.27.7

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.27-2023.10.19

1.27.6

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.27-2023-09.27

1.27.6

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.27-2023.09.12

1.27.4

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.27-2023.08.17

1.27.4

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.27-2023.08.08

1.27.3

1.6.6

1.1.1

1.27-2023.07.11

1.27.3

1.6.6

1.1.1

1.27-2023.06.20

1.27.1

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.27-2023.06.14

1.27.1

1.6.6

1.1.1

Added support for host port mapping in CNI. Merged pull request #93.

1.27-2023.06.06

1.27.1

1.6.6

1.1.1

Fixed containers-roadmap issue #2042, which caused nodes to fail pulling private Amazon ECR images.

11.27-2023.05.18

1.27.1

1.6.6

1.1.1

Kubernetes version 1.26

AMI version kubelet version containerd version csi-proxy version Release notes

1.26-2024.12.11

1.26.15

1.7.14

1.1.3

1.26-2024.11.12

1.26.15

1.7.14

1.1.3

1.26-2024.10.09

1.26.15

1.7.14

1.1.3

1.26-2024.09.10

1.26.15

1.7.14

1.1.3

1.26-2024.08.13

1.26.15

1.7.14

1.1.3

1.26-2024.07.10

1.26.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.26-2024.06.17

1.26.15

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.26-2024.05.14

1.26.15

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.26.15.

1.26-2024.04.09

1.26.12

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.26-2024.03.13

1.26.12

1.6.18

1.1.2

1.26-2024.02.13

1.26.12

1.6.18

1.1.2

1.26-2024.01.09

1.26.12

1.6.18

1.1.2

1.26-2023.12.12

1.26.10

1.6.18

1.1.2

1.26-2023.11.14

1.26.10

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.26-2023.10.19

1.26.9

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.26.9. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.26-2023.09.12

1.26.7

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.26-2023.08.17

1.26.7

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.26-2023.08.08

1.26.6

1.6.6

1.1.1

1.26-2023.07.11

1.26.6

1.6.6

1.1.1

1.26-2023.06.20

1.26.4

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.26-2023.06.14

1.26.4

1.6.6

1.1.1

Upgraded Kubernetes to 1.26.4. Added support for host port mapping in CNI. Merged pull request #93.

1.26-2023.05.09

1.26.2

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.26-2023.04.26

1.26.2

1.6.6

1.1.1

1.26-2023.04.11

1.26.2

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.26-2023.03.24

1.26.2

1.6.6

1.1.1

Kubernetes version 1.25

AMI version kubelet version containerd version csi-proxy version Release notes

1.25-2024.12.13

1.25.16

1.7.14

1.1.3

1.25-2024.11.12

1.25.16

1.7.14

1.1.3

1.25-2024.10.08

1.25.16

1.7.14

1.1.3

1.25-2024.09.10

1.25.16

1.7.14

1.1.3

1.25-2024.08.13

1.25.16

1.7.14

1.1.3

1.25-2024.07.10

1.25.16

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.25-2024.06.17

1.25.16

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.25-2024.05.14

1.25.16

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.25-2024.04.09

1.25.16

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.25-2024.03.13

1.25.16

1.6.18

1.1.2

1.25-2024.02.13

1.25.16

1.6.18

1.1.2

1.25-2024.01.09

1.25.16

1.6.18

1.1.2

1.25-2023.12.12

1.25.15

1.6.18

1.1.2

1.25-2023.11.14

1.25.15

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.25-2023.10.19

1.25.14

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.25.14. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.25-2023.09.12

1.25.12

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.25-2023.08.17

1.25.12

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.25-2023.08.08

1.25.9

1.6.6

1.1.1

1.25-2023.07.11

1.25.9

1.6.6

1.1.1

1.25-2023.06.20

1.25.9

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.25-2023.06.14

1.25.9

1.6.6

1.1.1

Upgraded Kubernetes to 1.25.9. Added support for host port mapping in CNI. Merged pull request #93.

1.25-2023.05.09

1.25.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.25-2023.04.11

1.25.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.25-2023.03.27

1.25.6

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.25-2023.03.20

1.25.6

1.6.6

1.1.1

1.25-2023.02.14

1.25.6

1.6.6

1.1.1

Kubernetes version 1.24

AMI version kubelet version containerd version csi-proxy version Release notes

1.24-2024.12.11

1.24.17

1.7.14

1.1.3

1.24-2024.11.12

1.24.17

1.7.14

1.1.3

1.24-2024.10.08

1.24.17

1.7.14

1.1.3

1.24-2024.09.10

1.24.17

1.7.14

1.1.3

1.24-2024.08.13

1.24.17

1.7.14

1.1.3

1.24-2024.07.10

1.24.17

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.24-2024.06.17

1.24.17

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.24-2024.05.14

1.24.17

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.24-2024.04.09

1.24.17

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.24-2024.03.13

1.24.17

1.6.18

1.1.2

1.24-2024.02.13

1.24.17

1.6.18

1.1.2

1.24-2024.01.09

1.24.17

1.6.18

1.1.2

1.24-2023.12.12

1.24.17

1.6.18

1.1.2

1.24-2023.11.14

1.24.17

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.24-2023.10.19

1.24.17

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.24.17. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.24-2023.09.12

1.24.16

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.24-2023.08.17

1.24.16

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.24-2023.08.08

1.24.13

1.6.6

1.1.1

1.24-2023.07.11

1.24.13

1.6.6

1.1.1

1.24-2023.06.20

1.24.13

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.24-2023.06.14

1.24.13

1.6.6

1.1.1

Upgraded Kubernetes to 1.24.13. Added support for host port mapping in CNI. Merged pull request #93.

1.24-2023.05.09

1.24.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.24-2023.04.11

1.24.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.24-2023.03.27

1.24.7

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.24-2023.03.20

1.24.7

1.6.6

1.1.1

Kubernetes version downgraded to 1.24.7 because 1.24.10 has a reported issue in kube-proxy.

1.24-2023.02.14

1.24.10

1.6.6

1.1.1

1.24-2023.01.23

1.24.7

1.6.6

1.1.1

1.24-2023.01.11

1.24.7

1.6.6

1.1.1

1.24-2022.12.13

1.24.7

1.6.6

1.1.1

1.24-2022.11.08

1.24.7

1.6.6

1.1.1

Amazon EKS optimized `Windows` Server 2019 Full AMI

The following tables list the current and previous versions of the Amazon EKS optimized Windows Server 2019 Full AMI.

Kubernetes version 1.31

AMI version kubelet version containerd version csi-proxy version Release notes

1.31-2025-01-01

1.31.4

1.7.20

1.1.3

Includes patches for CVE-2024-9042.

1.31-2024.12.13

1.31.3

1.7.20

1.1.3

1.31-2024.11.12

1.31.1

1.7.20

1.1.3

1.31-2024.10.08

1.31.1

1.7.20

1.1.3

1.31-2024.10.01

1.31.1

1.7.20

1.1.3

1.31-2024.09.10

1.31.0

1.7.20

1.1.3

Kubernetes version 1.30

AMI version kubelet version containerd version csi-proxy version Release notes

1.30-2025-01-01

1.30.8

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.30-2024.12.11

1.30.7

1.7.14

1.1.3

1.30-2024.11.12

1.30.4

1.7.14

1.1.3

1.30-2024.10.08

1.30.4

1.7.14

1.1.3

1.30-2024.09.10

1.30.2

1.7.14

1.1.3

1.30-2024.08.13

1.30.2

1.7.14

1.1.3

1.30-2024.07.10

1.30.2

1.7.14

1.1.2

Includes patches for CVE-2024-5321.

1.30-2024.06.17

1.30.0

1.7.14

1.1.2

Upgraded containerd to 1.7.14.

1.30-2024.05.15

1.30.0

1.6.28

1.1.2

Kubernetes version 1.29

AMI version kubelet version containerd version csi-proxy version Release notes

1.29-2025.01.01

1.29.12

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.29-2024.12.11

1.29.10

1.7.14

1.1.3

1.29-2024.11.12

1.29.8

1.7.14

1.1.3

1.29-2024.10.08

1.29.8

1.7.14

1.1.3

1.29-2024.09.10

1.29.6

1.7.14

1.1.3

1.29-2024.08.13

1.29.6

1.7.14

1.1.3

1.29-2024.07.10

1.29.6

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.29-2024.06.17

1.29.3

1.7.11

1.1.2

1.29-2024.05.15

1.29.3

1.7.11

1.1.2

Upgraded containerd to 1.7.11. Upgraded kubelet to 1.29.3.

1.29-2024.04.09

1.29.0

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.29-2024.03.13

1.29.0

1.6.25

1.1.2

1.29-2024.02.13

1.29.0

1.6.25

1.1.2

1.29-2024.02.06

1.29.0

1.6.25

1.1.2

Fixed a bug where the pause image was incorrectly deleted by kubelet garbage collection process.

1.29-2024.01.09

1.29.0

1.6.18

1.1.2

Kubernetes version 1.28

AMI version kubelet version containerd version csi-proxy version Release notes

1.28-2025-01-01

1.28.15

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.28-2024.12.11

1.28.15

1.7.14

1.1.3

1.28-2024.11.12

1.28.13

1.7.14

1.1.3

1.28-2024.10.08

1.28.13

1.7.14

1.1.3

1.28-2024.09.10

1.28.11

1.7.14

1.1.3

1.28-2024.08.13

1.28.11

1.7.14

1.1.3

1.28-2024.07.10

1.28.11

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.28-2024.06.17

1.28.8

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.28-2024.05.14

1.28.8

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.28.8.

1.28-2024.04.09

1.28.5

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.28-2024.03.13

1.28.5

1.6.18

1.1.2

1.28-2024.02.13

1.28.5

1.6.18

1.1.2

1.28-2024.01.09

1.28.5

1.6.18

1.1.2

1.28-2023.12.12

1.28.3

1.6.18

1.1.2

1.28-2023.11.14

1.28.3

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.28-2023.10.19

1.28.2

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.28-2023-09.27

1.28.2

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.28-2023.09.12

1.28.1

1.6.6

1.1.2

Kubernetes version 1.27

AMI version kubelet version containerd version csi-proxy version Release notes

1.27-2025-01-01

1.27.16

1.7.14

1.1.3

Includes patches for CVE-2024-9042.

1.27-2024.12.11

1.27.16

1.7.14

1.1.3

1.27-2024.11.12

1.27.16

1.7.14

1.1.3

1.27-2024.10.08

1.27.16

1.7.14

1.1.3

1.27-2024.09.10

1.27.15

1.7.14

1.1.3

1.27-2024.08.13

1.27.15

1.7.14

1.1.3

1.27-2024.07.10

1.27.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.27-2024.06.17

1.27.12

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.27-2024.05.14

1.27.12

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.27.12.

1.27-2024.04.09

1.27.9

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.27-2024.03.13

1.27.9

1.6.18

1.1.2

1.27-2024.02.13

1.27.9

1.6.18

1.1.2

1.27-2024.01.09

1.27.9

1.6.18

1.1.2

1.27-2023.12.12

1.27.7

1.6.18

1.1.2

1.27-2023.11.14

1.27.7

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.27-2023.10.19

1.27.6

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.27-2023-09.27

1.27.6

1.6.6

1.1.2

Fixed a security advisory in kubelet.

1.27-2023.09.12

1.27.4

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.27-2023.08.17

1.27.4

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.27-2023.08.08

1.27.3

1.6.6

1.1.1

1.27-2023.07.11

1.27.3

1.6.6

1.1.1

1.27-2023.06.20

1.27.1

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.27-2023.06.14

1.27.1

1.6.6

1.1.1

Added support for host port mapping in CNI. Merged pull request #93.

1.27-2023.06.06

1.27.1

1.6.6

1.1.1

Fixed containers-roadmap issue #2042, which caused nodes to fail pulling private Amazon ECR images.

1.27-2023.05.17

1.27.1

1.6.6

1.1.1

Kubernetes version 1.26

AMI version kubelet version containerd version csi-proxy version Release notes

1.26-2024.12.11

1.26.15

1.7.14

1.1.3

1.26-2024.11.12

1.26.15

1.7.14

1.1.3

1.26-2024.10.08

1.26.15

1.7.14

1.1.3

1.26-2024.09.10

1.26.15

1.7.14

1.1.3

1.26-2024.08.13

1.26.15

1.7.14

1.1.3

1.26-2024.07.10

1.26.15

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.26-2024.06.17

1.26.15

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.26-2024.05.14

1.26.15

1.6.28

1.1.2

Upgraded containerd to 1.6.28. Upgraded kubelet to 1.26.15.

1.26-2024.04.09

1.26.12

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.26-2024.03.13

1.26.12

1.6.18

1.1.2

1.26-2024.02.13

1.26.12

1.6.18

1.1.2

1.26-2024.01.09

1.26.12

1.6.18

1.1.2

1.26-2023.12.12

1.26.10

1.6.18

1.1.2

1.26-2023.11.14

1.26.10

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.26-2023.10.19

1.26.9

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.26.9. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.26-2023.09.12

1.26.7

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.26-2023.08.17

1.26.7

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.26-2023.08.08

1.26.6

1.6.6

1.1.1

1.26-2023.07.11

1.26.6

1.6.6

1.1.1

1.26-2023.06.20

1.26.4

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.26-2023.06.14

1.26.4

1.6.6

1.1.1

Upgraded Kubernetes to 1.26.4. Added support for host port mapping in CNI. Merged pull request #93.

1.26-2023.05.09

1.26.2

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.26-2023.04.26

1.26.2

1.6.6

1.1.1

1.26-2023.04.11

1.26.2

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.26-2023.03.24

1.26.2

1.6.6

1.1.1

Kubernetes version 1.25

AMI version kubelet version containerd version csi-proxy version Release notes

1.25-2024.12.13

1.25.16

1.7.14

1.1.3

1.25-2024.11.12

1.25.16

1.7.14

1.1.3

1.25-2024.10.08

1.25.16

1.7.14

1.1.3

1.25-2024.09.10

1.25.16

1.7.14

1.1.3

1.25-2024.08.13

1.25.16

1.7.14

1.1.3

1.25-2024.07.10

1.25.16

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.25-2024.06.17

1.25.16

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.25-2024.05.14

1.25.16

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.25-2024.04.09

1.25.16

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.25-2024.03.13

1.25.16

1.6.18

1.1.2

1.25-2024.02.13

1.25.16

1.6.18

1.1.2

1.25-2024.01.09

1.25.16

1.6.18

1.1.2

1.25-2023.12.12

1.25.15

1.6.18

1.1.2

1.25-2023.11.14

1.25.15

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.25-2023.10.19

1.25.14

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.25.14. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.25-2023.09.12

1.25.12

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.25-2023.08.17

1.25.12

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.25-2023.08.08

1.25.9

1.6.6

1.1.1

1.25-2023.07.11

1.25.9

1.6.6

1.1.1

1.25-2023.06.20

1.25.9

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.25-2023.06.14

1.25.9

1.6.6

1.1.1

Upgraded Kubernetes to 1.25.9. Added support for host port mapping in CNI. Merged pull request #93.

1.25-2023.05.09

1.25.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.25-2023.04.11

1.25.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.25-2023.03.27

1.25.6

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.25-2023.03.20

1.25.6

1.6.6

1.1.1

1.25-2023.02.14

1.25.6

1.6.6

1.1.1

Kubernetes version 1.24

AMI version kubelet version containerd version csi-proxy version Release notes

1.24-2024.12.11

1.24.17

1.7.14

1.1.3

1.24-2024.11.12

1.24.17

1.7.14

1.1.3

1.24-2024.10.08

1.24.17

1.7.14

1.1.3

1.24-2024.09.10

1.24.17

1.7.14

1.1.3

1.24-2024.08.13

1.24.17

1.7.14

1.1.3

1.24-2024.07.10

1.24.17

1.7.11

1.1.2

Includes patches for CVE-2024-5321.

1.24-2024.06.17

1.24.17

1.7.11

1.1.2

Upgraded containerd to 1.7.11.

1.24-2024.05.14

1.24.17

1.6.28

1.1.2

Upgraded containerd to 1.6.28.

1.24-2024.04.09

1.24.17

1.6.25

1.1.2

Upgraded containerd to 1.6.25. Rebuilt CNI and csi-proxy using golang 1.22.1.

1.24-2024.03.13

1.24.17

1.6.18

1.1.2

1.24-2024.02.13

1.24.17

1.6.18

1.1.2

1.24-2024.01.09

1.24.17

1.6.18

1.1.2

1.24-2023.12.12

1.24.17

1.6.18

1.1.2

1.24-2023.11.14

1.24.17

1.6.18

1.1.2

Includes patches for CVE-2023-5528.

1.24-2023.10.19

1.24.17

1.6.18

1.1.2

Upgraded containerd to 1.6.18. Upgraded kubelet to 1.24.17. Added new bootstrap script environment variables (SERVICE_IPV4_CIDR and EXCLUDED_SNAT_CIDRS).

1.24-2023.09.12

1.24.16

1.6.6

1.1.2

Upgraded the Amazon VPC CNI plugin to use the Kubernetes connector binary, which gets the Pod IP address from the Kubernetes API server. Merged pull request #100.

1.24-2023.08.17

1.24.16

1.6.6

1.1.2

Includes patches for CVE-2023-3676, CVE-2023-3893, and CVE-2023-3955.

1.24-2023.08.08

1.24.13

1.6.6

1.1.1

1.24-2023.07.11

1.24.13

1.6.6

1.1.1

1.24-2023.06.21

1.24.13

1.6.6

1.1.1

Resolved issue that was causing the DNS suffix search list to be incorrectly populated.

1.24-2023.06.14

1.24.13

1.6.6

1.1.1

Upgraded Kubernetes to 1.24.13. Added support for host port mapping in CNI. Merged pull request #93.

1.24-2023.05.09

1.24.7

1.6.6

1.1.1

Fixed a bug causing network connectivity issue #1126 on pods after node restart. Introduced a new bootstrap script configuration parameter (ExcludedSnatCIDRs).

1.24-2023.04.11

1.24.7

1.6.6

1.1.1

Added recovery mechanism for kubelet and kube-proxy on service crash.

1.24-2023.03.27

1.24.7

1.6.6

1.1.1

Installed a domainless gMSA plugin to facilitate gMSA authentication for Windows containers on Amazon EKS.

1.24-2023.03.20

1.24.7

1.6.6

1.1.1

Kubernetes version downgraded to 1.24.7 because 1.24.10 has a reported issue in kube-proxy.

1.24-2023.02.14

1.24.10

1.6.6

1.1.1

1.24-2023.01.23

1.24.7

1.6.6

1.1.1

1.24-2023.01.11

1.24.7

1.6.6

1.1.1

1.24-2022.12.14

1.24.7

1.6.6

1.1.1

1.24-2022.10.12

1.24.7

1.6.6

1.1.1

Retrieve recommended `Microsoft Windows` AMI IDs

You can programmatically retrieve the Amazon Machine Image (AMI) ID for Amazon EKS optimized AMIs by querying the AWS Systems Manager Parameter Store API.

You can retrieve the image ID of the latest recommended Amazon EKS optimized Windows AMI with the following command, which uses the sub-parameter image_id. Make the following modifications to the command as needed and then run the modified command:

Replace release with one of the following options.
- Use 2022 for Windows Server 2022, but only if you’re using Kubernetes version 1.24 or later.
- Use 2019 for Windows Server 2019.
Replace installation-option with one of the following options. For more information, see What is the Server Core installation option in Windows Server.
- Use Core for a minimal installation with a smaller attack surface.
- Use Full to include the Windows desktop experience.
Replace kubernetes-version with a supported Amazon EKS version.
Replace region-code with an Amazon EKS supported AWS Region for which you want the AMI ID.

aws ssm get-parameter --name /aws/service/ami-windows-latest/Windows_Server-release-English-installation-option-EKS_Optimized-kubernetes-version/image_id \
    --region region-code --query "Parameter.Value" --output text

Here’s an example command after placeholder replacements have been made.

aws ssm get-parameter --name /aws/service/ami-windows-latest/Windows_Server-2022-English-Core-EKS_Optimized-1.31/image_id \
    --region us-west-2 --query "Parameter.Value" --output text

An example output is as follows.

ami-1234567890abcdef0

Build a custom `Windows` AMI with Image Builder

You can use EC2 Image Builder to create custom Amazon EKS optimized Windows AMIs.

You can use EC2 Image Builder to create custom Amazon EKS optimized Windows AMIs with one of the following options:

Using an Amazon EKS optimized Windows AMI as a base
Using the Amazon-managed build component

With both methods, you must create your own Image Builder recipe. For more information, see Create a new version of an image recipe in the Image Builder User Guide.

The following Amazon-managed components for eks include patches for CVE-2024-5321.

1.24.5 and higher
1.25.4 and higher
1.26.4 and higher
1.27.2 and higher
1.28.2 and higher
1.29.2 and higher
1.30.1 and higher

Using an Amazon EKS optimized `Windows` AMI as a base

This option is the recommended way to build your custom Windows AMIs. The Amazon EKS optimized Windows AMIs we provide are more frequently updated than the Amazon-managed build component.

Start a new Image Builder recipe.
1. Open the EC2 Image Builder console at https://console.aws.amazon.com/imagebuilder.
2. In the left navigation pane, choose Image recipes.
3. Choose Create image recipe.
In the Recipe details section, enter a Name and Version.
Specify the ID of the Amazon EKS optimized Windows AMI in the Base image section.
1. Choose Enter custom AMI ID.
2. Retrieve the AMI ID for the Windows OS version that you require. For more information, see retrieve-windows-ami-id.title.
3. Enter the custom AMI ID. If the AMI ID isn’t found, make sure that the AWS Region for the AMI ID matches the AWS Region shown in the upper right of your console.
(Optional) To get the latest security updates, add the update-windows component in the Build components - section.
1. From the dropdown list to the right of the Find components by name search box, choose Amazon-managed.
2. In the Find components by name search box, enter update-windows.
3. Select the check box of the update-windows search result. This component includes the latest Windows patches for the operating system.
Complete the remaining image recipe inputs with your required configurations. For more information, see Create a new image recipe version (console) in the Image Builder User Guide.
Choose Create recipe.
Use the new image recipe in a new or existing image pipeline. Once your image pipeline runs successfully, your custom AMI will be listed as an output image and is ready for use. For more information, see Create an image pipeline using the EC2 Image Builder console wizard.

Using the Amazon-managed build component

When using an Amazon EKS optimized Windows AMI as a base isn’t viable, you can use the Amazon-managed build component instead. This option may lag behind the most recent supported Kubernetes versions.

Start a new Image Builder recipe.
1. Open the EC2 Image Builder console at https://console.aws.amazon.com/imagebuilder.
2. In the left navigation pane, choose Image recipes.
3. Choose Create image recipe.
In the Recipe details section, enter a Name and Version.
Determine which option you will be using to create your custom AMI in the Base image section:
- Select managed images – Choose Windows for your Image Operating System (OS). Then choose one of the following options for Image origin.
  - Quick start (Amazon-managed) – In the Image name dropdown, choose an Amazon EKS supported Windows Server version. For more information, see eks-optimized-windows-ami.title.
  - Images owned by me – For Image name, choose the ARN of your own image with your own license. The image that you provide can’t already have Amazon EKS components installed.
- Enter custom AMI ID – For AMI ID, enter the ID for your AMI with your own license. The image that you provide can’t already have Amazon EKS components installed.
In the Build components - Windows section, do the following:
1. From the dropdown list to the right of the Find components by name search box, choose Amazon-managed.
2. In the Find components by name search box, enter eks.
3. Select the check box of the eks-optimized-ami-windows search result, even though the result returned may not be the version that you want.
4. In the Find components by name search box, enter update-windows .
5. Select the check box of the update-windows search result. This component includes the latest Windows patches for the operating system.
In the Selected components section, do the following:
1. Choose Versioning options for eks-optimized-ami-windows.
2. Choose Specify component version.
3. In the Component Version field, enter version.x, replacing version with a supported Kubernetes version. Entering an x for part of the version number indicates to use the latest component version that also aligns with the part of the version you explicitly define. Pay attention to the console output as it will advise you on whether your desired version is available as a managed component. Keep in mind that the most recent Kubernetes versions may not be available for the build component. For more information about available versions, see custom-windows-ami-component-versions.title.
  
  The following eks-optimized-ami-windows build component versions require eksctl version 0.129 or lower:
  - 1.24.0
Complete the remaining image recipe inputs with your required configurations. For more information, see Create a new image recipe version (console) in the Image Builder User Guide.
Choose Create recipe.
Use the new image recipe in a new or existing image pipeline. Once your image pipeline runs successfully, your custom AMI will be listed as an output image and is ready for use. For more information, see Create an image pipeline using the EC2 Image Builder console wizard.

Retrieving information about `eks-optimized-ami-windows` component versions

You can retrieve specific information regarding what is installed with each component. For example, you can verify what kubelet version is installed. The components go through functional testing on the Amazon EKS supported Windows operating systems versions. For more information, see windows-ami-release-calendar.title. Any other Windows OS versions that aren’t listed as supported or have reached end of support might not be compatible with the component.

Open the EC2 Image Builder console at https://console.aws.amazon.com/imagebuilder.
In the left navigation pane, choose Components.
From the dropdown list to the right of the Find components by name search box, change Owned by me to Quick start (Amazon-managed).
In the Find components by name box, enter eks.
(Optional) If you are using a recent version, sort the Version column in descending order by choosing it twice.
Choose the eks-optimized-ami-windows link with a desired version.

The Description in the resulting page shows the specific information.

9.7. Enable node auto repair and investigate node health issues

9.7.1. View the health status of your nodes

This topic explains the tools and methods available for monitoring node health status in Amazon EKS clusters.

This topic explains the tools and methods available for monitoring node health status in Amazon EKS clusters. The information covers node conditions, events, and detection cases that help you identify and diagnose node-level issues. Use the commands and patterns described here to inspect node health resources, interpret status conditions, and analyze node events for operational troubleshooting.

You can get some node health information with Kubernetes commands for all nodes. And if you use the node monitoring agent through Amazon EKS Auto Mode or the Amazon EKS managed add-on, you will get a wider variety of node signals to help troubleshoot. Descriptions of detected health issues by the node monitoring agent are also made available in the observability dashboard. For more information, see node-health.title.

Node conditions

Node conditions represent terminal issues requiring remediation actions like instance replacement or reboot.

To get conditions for all nodes:

kubectl get nodes -o 'custom-columns=NAME:.metadata.name,CONDITIONS:.status.conditions[*].type,STATUS:.status.conditions[*].status'

To get detailed conditions for a specific node

kubectl describe node node-name

Example condition output of a healthy node:

  - lastHeartbeatTime: "2024-11-21T19:07:40Z"
    lastTransitionTime: "2024-11-08T03:57:40Z"
    message: Monitoring for the Networking system is active
    reason: NetworkingIsReady
    status: "True"
    type: NetworkingReady

Example condition of a unhealthy node with a networking problem:

  - lastHeartbeatTime: "2024-11-21T19:12:29Z"
    lastTransitionTime: "2024-11-08T17:04:17Z"
    message: IPAM-D has failed to connect to API Server which could be an issue with
      IPTable rules or any other network configuration.
    reason: IPAMDNotReady
    status: "False"
    type: NetworkingReady

Node events

Node events indicate temporary issues or sub-optimal configurations.

To get all events reported by the node monitoring agent

When the node monitoring agent is available, you can run the following command.

kubectl get events --field-selector=reportingComponent=eks-node-monitoring-agent

Sample output:

LAST SEEN   TYPE      REASON       OBJECT                                              MESSAGE
4s          Warning   SoftLockup   node/ip-192-168-71-251.us-west-2.compute.internal   CPU stuck for 23s

To get events for all nodes

kubectl get events --field-selector involvedObject.kind=Node

To get events for a specific node

kubectl get events --field-selector involvedObject.kind=Node,involvedObject.name=node-name

To watch events in real-time

kubectl get events -w --field-selector involvedObject.kind=Node

Example event output:

LAST SEEN   TYPE     REASON           OBJECT         MESSAGE
2m          Warning  MemoryPressure   Node/node-1    Node experiencing memory pressure
5m          Normal   NodeReady        Node/node-1    Node became ready

Common troubleshooting commands

# Get comprehensive node status
kubectl get node node-name -o yaml

# Watch node status changes
kubectl get nodes -w

# Get node metrics
kubectl top node

9.7.2. Retrieve node logs for a managed node using kubectl and S3

Learn how to retrieve node logs for an Amazon EKS managed node that has the node monitoring agent.

Learn how to retrieve node logs for an Amazon EKS managed node that has the node monitoring agent.

Prerequisites

Make sure you have the following:

An existing Amazon EKS cluster with the node monitoring agent. For more information, see node-health.title.
The kubectl command-line tool installed and configured to communicate with your cluster.
The AWS CLI installed and logged in with sufficent permissions to create S3 buckets and objects.
A recent version of Python 3 installed
The AWS SDK for Python 3, Boto 3, installed.

Step 1: Create S3 bucket destination (optional)

If you don’t already have an S3 bucket to store the logs, create one. Use the following AWS CLI command. The bucket defaults to the private access control list. Replace bucket-name with your chosen unique bucket name.

aws s3api create-bucket --bucket bucket-name

Step 2: Create pre-signed S3 URL for HTTP Put

Amazon EKS returns the node logs by doing a HTTP PUT operation to a URL you specify. In this tutorial, we will generate a pre-signed S3 HTTP PUT URL.

The logs will be returned as a gzip tarball, with the .tar.gz extension.

You must use the AWS API or a SDK to create the pre-signed S3 upload URL for EKS to upload the log file. You cannot create a pre-signed S3 upload URL using the AWS CLI.

Determine where in the bucket you want to store the logs. For example, you might use 2024-11-12/logs1.tar.gz as the key.

Save the following Python code to the file presign-upload.py. Replace <bucket-name> and <key>. The key should end with .tar.gz.

import boto3; print(boto3.client('s3').generate_presigned_url(
   ClientMethod='put_object',
   Params={'Bucket': '<bucket-name>', 'Key': '<key>'},
   ExpiresIn=1000
))

Run the script with
```
python presign-upload.py
```
Note the URL output. Use this value in the next step as the http-put-destination.

For more information, see Generate a presigned URL to upload a file in the AWS Boto3 SDK for Python Documentation.

Step 3: Create NodeDiagnostic resource

Identify the name of the node you want to collect logs from.

Create a NodeDiagnostic manifest that uses the name of the node as the resource’s name, and providing a HTTP PUT URL destination.

apiVersion: eks.amazonaws.com/v1alpha1
kind: NodeDiagnostic
metadata:
    name: node-name
spec:
    logCapture:
        destination: http-put-destination

Apply the manifest to the cluster.

kubectl apply -f nodediagnostic.yaml

You can check on the Status of the collection by describing the NodeDiagnostic resource:

A status of Success or SuccessWithErrors indicates that the task completed and the logs uploaded to the provided destination (SuccessWithErrors indicates that some logs might be missing)
If the status is Failure, confirm the upload URL is well-formed and not expired.

kubectl describe nodediagnostics.eks.amazonaws.com/node-name

Step 4: Download logs from S3

Wait approximately one minute before attempting to download the logs. Then, use the S3 CLI to download the logs.

# Once NodeDiagnostic shows Success status, download the logs
aws s3 cp s3://bucket-name/key ./node-logs.tar.gz

Step 5: Clean up NodeDiagnostic resource

NodeDiagnostic resources do not get automatically deleted. You should clean these up on your own after you have obtained your log artifacts

# Delete the NodeDiagnostic resource
kubectl delete nodediagnostics.eks.amazonaws.com/node-name

You can use the node monitoring agent to show health issues and use node auto repair to automatically replace nodes when issues are detected.

Node health refers to the operational status and capability of a node to effectively run workloads. A healthy node maintains expected connectivity, has sufficient resources, and can successfully run Pods without disruption. For information on getting details about your nodes, see learn-status-conditions.title and auto-get-logs.title.

To help with maintaining healthy nodes, Amazon EKS offers the node monitoring agent and node auto repair.

9.7.3. Node monitoring agent

The node monitoring agent automatically reads node logs to detect certain health issues. It parses through node logs to detect failures and surfaces various status information about worker nodes. A dedicated NodeCondition is applied on the worker nodes for each category of issues detected, such as storage and networking issues. Descriptions of detected health issues are made available in the observability dashboard. For more information, see observability-node-health-issues.title.

The node monitoring agent is included as a capability for all Amazon EKS Auto Mode clusters. For other cluster types, you can add the monitoring agent as an Amazon EKS add-on. For more information, see creating-an-add-on.title.

9.7.4. Node auto repair

Node auto repair is an additional feature that continuously monitors the health of nodes, automatically reacting to detected problems and replacing nodes when possible. This helps overall availability of the cluster with minimal manual intervention. If a health check fails, the node is automatically cordoned so that no new Pods are scheduled on the node.

By itself, node auto repair can react to the Ready condition of the kubelet and any node objects that are manually deleted. When paired with the node monitoring agent, node auto repair can react to more conditions that wouldn’t be detected otherwise. These additional conditions include KernelReady, NetworkingReady, and StorageReady.

This automated node recovery automatically addresses intermittent node issues such as failures to join the cluster, unresponsive kubelets, and increased accelerator (device) errors. The improved reliability helps reduce application downtime and improve cluster operations. Node auto repair cannot handle certain problems that are reported such as DiskPressure, MemoryPressure, and PIDPressure. Amazon EKS waits 10 minutes before acting on the AcceleratedHardwareReady NodeConditions, and 30 minutes for all other conditions.

Managed node groups will also automatically disable node repairs for safety reasons under two scenarios. Any repair operations that are previously in progress will continue for both situations.

If a zonal shift for your cluster has been triggered through the Application Recovery Controller (ARC), all subsequent repair operations are halted.
If your node group has more than five nodes and more than 20% of the nodes in your node group are in an unhealthy state, repair operations are halted.

You can enable node auto repair when creating or editing a managed node group.

When using the Amazon EKS console, activate the Enable node auto repair checkbox for the managed node group. For more information, see create-managed-node-group.title.
When using the AWS CLI, add the --node-repair-config enabled=true to the eks create nodegroup or eks update-nodegroup-config command.
For an example eksctl ClusterConfig that uses a managed node group with node auto repair, see 44-node-repair.yaml on GitHub.

9.7.5. Node health issues

The following tables describe node health issues that can be detected by the node monitoring agent. There are two types of issues:

Condition – A terminal issue that warrants a remediation action like an instance replacement or reboot. When auto repair is enabled, Amazon EKS will do a repair action, either as a node replacement or reboot. For more information, see status-node-conditions.title.
Event – A temporary issue or sub-optimal node configuration. No auto repair action will take place. For more information, see status-node-events.title.

Kernel node health issues

Name

Severity

Description

ForkFailedOutOfPID

Condition

A fork or exec call has failed due to the system being out of process IDs or memory, which may be caused by zombie processes or physical memory exhaustion.

AppBlocked

Event

The task has been blocked for a long period of time from scheduling, usually caused by being blocked on input or output.

AppCrash

Event

An application on the node has crashed.

ApproachingKernelPidMax

Event

The number of processes is approaching the maximum number of PIDs that are available per the current kernel.pid_max setting, after which no more processes can be launched.

ApproachingMaxOpenFiles

Event

The number of open files is approaching the maximum number of possible open files given the current kernel settings, after which opening new files will fail.

ConntrackExceededKernel

Event

Connection tracking exceeded the maximum for the kernel and new connections could not be established, which can result in packet loss.

ExcessiveZombieProcesses

Event

Processes which can’t be fully reclaimed are accumulating in large numbers, which indicates application issues and may lead to reaching system process limits.

KernelBug

Event

A kernel bug was detected and reported by the Linux kernel itself, though this may sometimes be caused by nodes with high CPU or memory usage leading to delayed event processing.

LargeEnvironment

Event

The number of environment variables for this process is larger than expected, potentially caused by many services with enableServiceLinks set to true, which may cause performance issues.

RapidCron

Event

A cron job is running faster than every five minutes on this node, which may impact performance if the job consumes significant resources.

SoftLockup

Event

The CPU stalled for a given amount of time.

Networking node health issues

Name Severity Description

InterfaceNotRunning

Condition

This interface appears to not be running or there are network issues.

InterfaceNotUp

Condition

This interface appears to not be up or there are network issues.

IPAMDNotReady

Condition

IPAMD fails to connect to the API server.

IPAMDNotRunning

Condition

The aws-k8s-agent process was not found to be running.

MissingLoopbackInterface

Condition

The loopback interface is missing from this instance, causing failure of services depending on local connectivity.

BandwidthInExceeded

Event

Packets have been queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance.

BandwidthOutExceeded

Event

Packets have been queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance.

ConntrackExceeded

Event

Connection tracking exceeded the maximum for the instance and new connections could not be established, which can result in packet loss.

IPAMDNoIPs

Event

IPAM-D is out of IP addresses.

IPAMDRepeatedlyRestart

Event

Multiple restarts in the IPAMD service have occurred.

KubeProxyNotReady

Event

Kube-proxy failed to watch or list resources.

LinkLocalExceeded

Event

Packets were dropped because the PPS of traffic to local proxy services exceeded the network interface maximum.

MissingDefaultRoutes

Event

There are missing default route rules.

MissingIPRules, MissingIPRoutes

Event

There are missing route rules for the following Pod IPs from the route table.

NetworkSysctl

Event

This node’s network sysctl settings are potentially incorrect.

PortConflict

Event

If a Pod uses hostPort, it can write iptables rules that override the host’s already bound ports, potentially preventing API server access to kubelet.

PPSExceeded

Event

Packets have been queued or dropped because the bidirectional PPS exceeded the maximum for the instance.

UnexpectedRejectRule

Event

An unexpected REJECT` or DROP rule was found in the iptables, potentially blocking expected traffic.

Neuron node health issues

Name

Severity

Description

NeuronDMAError

Condition

A DMA engine encountered an unrecoverable error.

NeuronHBMUncorrectableError

Condition

An HBM encountered an uncorrectable error and produced incorrect results.

NeuronNCUncorrectableError

Condition

A Neuron Core uncorrectable memory error was detected.

NeuronSRAMUncorrectableError

Condition

An on-chip SRAM encountered a parity error and produced incorrect results.

NVIDIA node health issues

If auto repair is enabled, the repair actions that are listed start 10 minutes after the issue is detected. For more information on XID errors, see Xid Errors in the NVIDIA GPU Deployment and Management Documentation. For more information on the individual XID messages, see Understanding Xid Messages in the NVIDIA GPU Deployment and Management Documentation.

Name

Severity

Description

Repair action

NvidiaDoubleBitError

Condition

A double bit error was produced by the GPU driver.

Replace

NvidiaNVLinkError

Condition

NVLink errors were reported by the GPU driver.

Replace

NvidiaXID13Error

Condition

There is a graphics engine exception.

Reboot

NvidiaXID31Error

Condition

There are suspected hardware problems.

Reboot

NvidiaXID48Error

Condition

Double bit ECC errors are reported by the driver.

Reboot

NvidiaXID63Error

Condition

There’s a page retirement or row remap.

Reboot

NvidiaXID64Error

Condition

There are failures trying to retire a page or perform a node remap.

Reboot

NvidiaXID74Error

Condition

There is a problem with a connection from the GPU to another GPU or NVSwitch over NVLink. This may indicate a hardware failure with the link itself or may indicate a problem with the device at the remote end of the link.

Replace

NvidiaXID79Error

Condition

The GPU driver attempted to access the GPU over its PCI Express connection and found that the GPU is not accessible.

Replace

NvidiaXID94Error

Condition

There are ECC memory errors.

Reboot

NvidiaXID95Error

Condition

There are ECC memory errors.

Reboot

NvidiaXID119Error

Condition

The GSP timed out responding to RPC requests from other bits in the driver.

Replace

NvidiaXID120Error

Condition

The GSP has responded in time, but with an error.

Replace

NvidiaXID121Error

Condition

C2C is chip interconnect. It enables sharing memory between CPUs, accelerators, and more.

Replace

NvidiaXID140Error

Condition

The GPU driver may have observed uncorrectable errors in GPU memory, in such a way as to interrupt the GPU driver’s ability to mark the pages for dynamic page offlining or row remapping.

Replace

NvidiaPageRetirement

Event

The GPU driver has marked a memory page for retirement. This may occur if there is a single double bit error or two single bit errors are encountered at the same address.

None

NvidiaXID[Code]Warning

Event

Any occurrences of XIDs other than the ones defined in this list result in this event.

None

Runtime node health issues

Name

Severity

Description

PodStuckTerminating

Condition

A Pod is or was stuck terminating for an excessive amount of time, which can be caused by CRI errors preventing pod state progression.

%sRepeatedRestart

Event

Restarts of any systemd service on the node (formatted using the title-cased unit name).

ContainerRuntimeFailed

Event

The container runtime has failed to create a container, likely related to any reported issues if occurring repeatedly.

KubeletFailed

Event

The kubelet entered a failed state.

LivenessProbeFailures

Event

A liveness probe failure was detected, potentially indicating application code issues or insufficient timeout values if occurring repeatedly.

ReadinessProbeFailures

Event

A readiness probe failure was detected, potentially indicating application code issues or insufficient timeout values if occurring repeatedly.

ServiceFailedToStart

Event

A systemd unit failed to start.

Storage node health issues

Name Severity Description

XFSSmallAverageClusterSize

Condition

The XFS Average Cluster size is small, indicating excessive free space fragmentation that can prevent file creation despite available inodes or free space.

EtcHostsMountFailed

Event

Mounting of the kubelet generated /etc/hosts failed due to userdata remounting /var/lib/kubelet/pods during kubelet-container operation.

IODelays

Event

Input or output delay detected in a process, potentially indicating insufficient input-output provisioning if excessive.

KubeletDiskUsageSlow

Event

Kubelet is reporting slow disk usage while trying to access the filesystem, potentially indicating insufficient disk input-output or filesystem issues.

9.8. Amazon EKS Hybrid Nodes overview

Join nodes from your data centers to Amazon EKS Kubernetes clusters with Amazon EKS Hybrid Nodes.

With Amazon EKS Hybrid Nodes, you can use your on-premises and edge infrastructure as nodes in Amazon EKS clusters. AWS manages the AWS-hosted Kubernetes control plane of the Amazon EKS cluster, and you manage the hybrid nodes that run in your on-premises or edge environments. This unifies Kubernetes management across your environments and offloads Kubernetes control plane management to AWS for your on-premises and edge applications.

Amazon EKS Hybrid Nodes works with any on-premises hardware or virtual machines, bringing the efficiency, scalability, and availability of Amazon EKS to wherever your applications need to run. You can use a wide range of Amazon EKS features with Amazon EKS Hybrid Nodes including Amazon EKS add-ons, Amazon EKS Pod Identity, cluster access entries, cluster insights, and extended Kubernetes version support. Amazon EKS Hybrid Nodes natively integrates with AWS-services including AWS Systems Manager, AWS IAM Roles Anywhere, Amazon Managed Service for Prometheus, Amazon CloudWatch, and Amazon GuardDuty for centralized monitoring, logging, and identity management.

With Amazon EKS Hybrid Nodes, there are no upfront commitments or minimum fees, and you are charged per hour for the vCPU resources of your hybrid nodes when they are attached to your Amazon EKS clusters. For more pricing information, see Amazon EKS Pricing.

For an overview of the other Amazon EKS options for on-premises and edge deployments, see eks-deployment-options.title.

9.8.1. General concepts of Amazon EKS Hybrid Nodes

Amazon EKS Hybrid Nodes must have a reliable connection between your on-premises environment and AWS. Amazon EKS Hybrid Nodes aren’t a fit for disconnected, disrupted, intermittent or limited (DDIL) environments. If you are running in a DDIL environment, consider Amazon EKS Anywhere.
Running Amazon EKS Hybrid Nodes on cloud infrastructure, including AWS-Regions, AWS Local Zones, OUTlong, or in other clouds, is not supported. Use Amazon EKS Auto Mode, Karpenter, Amazon EC2 managed node groups, self-managed nodes, or AWS Fargate when running in AWS-Regions. Use Amazon EC2 managed node groups or Amazon EC2 self-managed nodes when running on AWS Local Zones. Only Amazon EC2 self-managed nodes can be used on OUTlong or AWS Wavelength Zones.
A single Amazon EKS cluster can be used to run hybrid nodes and nodes in AWS-Regions, AWS Local Zones, or OUTlong.
Amazon EKS Hybrid Nodes is available in all AWS-Regions, except the AWS GovCloud (US) Regions and the AWS China Regions.
You will be charged the hybrid nodes fee if you run hybrid nodes on Amazon EC2 instances.
Billing for hybrid nodes starts when the nodes join the Amazon EKS cluster and stops when the nodes are removed from the cluster. Be sure to remove your hybrid nodes from your Amazon EKS cluster if you are not using them.

Infrastructure Management

Amazon EKS Hybrid Nodes follows a bring your own infrastructure approach where it is your responsibility to provision and manage the physical or virtual machines and the operating system you use for hybrid nodes.
Amazon EKS Hybrid Nodes are agnostic to the infrastructure they run on. You can run hybrid nodes on physical or virtual machines, and x86 and ARM architectures.

Operating Systems for hybrid nodes

Amazon Linux 2023 (AL2023): You can use Amazon Linux 2023 (AL2023) as the node operating system for hybrid nodes, but only in virtualized environments such as VMWare, KVM, and Hyper-V. AWS supports the integration of hybrid nodes with AL2023, but AL2023 isn’t covered by the AWS Support Plans when you run it outside of Amazon EC2.
Ubuntu: You can use Ubuntu 20.04, Ubuntu 22.04, and Ubuntu 24.04 as the node operating system for hybrid nodes.
Red Hat Enterprise Linux (RHEL): You can use RHEL 8 and RHEL 9 as the node operating system for hybrid nodes.

Kubernetes and platform versions

Amazon EKS Hybrid Nodes supports the same Kubernetes versions and deprecation schedule as Amazon EKS, including standard and extended Kubernetes version support. For more information on Kubernetes versions in Amazon EKS, see kubernetes-versions.title. For more information about Amazon EKS platform versions, see platform-versions.title.
You must create new Amazon EKS clusters to use Amazon EKS Hybrid Nodes. Hybrid nodes can’t be used with existing Amazon EKS clusters.

Networking

The communication between the Amazon EKS control plane and hybrid nodes is routed through the VPC and subnets you pass during cluster creation, which builds on the existing mechanism in Amazon EKS for control plane to node networking.
Amazon EKS Hybrid Nodes is flexible to your preferred method of connecting your on-premises networks to a VPC in AWS. There are several documented options available including AWS Site-to-Site VPN and AWS Direct Connect, and you can choose the method that best fits your use case.
IP address family: Hybrid nodes can be used with Amazon EKS clusters configured with the IPv4 IP address family only. You can’t use Amazon EKS clusters configured with the IPv6 IP address family. Similarly, your on-premises node and Pod CIDRs must be IPv4 RFC1918 CIDR blocks.
You must enable the required domains, protocols, and ports for Amazon EKS Hybrid Nodes in your on-premises environments and firewalls. For more information, including minimum networking requirements, see hybrid-nodes-networking.title.
Cluster endpoint access: You can use “Public” or “Private” cluster endpoint access. You should not use “Public and Private” cluster endpoint access, as the endpoint DNS resolution will always resolve to the public addresses for queries originating from your on-premises environment.
For information and best practices during scenarios where there are network disconnections between hybrid nodes and the AWS-Region, see the hybrid nodes section of the Amazon EKS Best Practices Guide.
Application load balancing: Kubernetes has a Service object to define the names and domain names for your applications and resolve and load balance to them. By default, the type:LoadBalancer type of Service additionally creates an AWS Classic Load Balancer for traffic from outside the cluster. You can change this behavior with add-ons. Specifically, we recommend the AWS Application Load Balancer and AWS Network Load Balancer which are created by the AWS Load Balancer Controller, instead of the AWS Classic Load Balancer. For steps to install the AWS Load Balancer Controller in a hybrid environment, see hybrid-nodes-add-ons-lbc.title.

Security for hybrid nodes

Amazon EKS Hybrid Nodes use temporary IAM credentials to authenticate with your Amazon EKS cluster. You can use either AWS IAM Roles Anywhere or AWS Systems Manager (SSM) hybrid activations for provisioning the on-premises IAM credentials for hybrid nodes. It is recommended to use AWS SSM hybrid activations if you do not have existing Public Key Infrastructure (PKI) with a Certificate Authority (CA) and certificates for your on-premises environments. If you do have existing PKI and certificates on-premises, use AWS IAM Roles Anywhere.
You can use API or API_AND_CONFIG_MAP cluster authentication modes for your hybrid nodes-enabled Amazon EKS clusters. Use the cluster access entry type called HYBRID_LINUX with your hybrid nodes IAM role to enable hybrid nodes to join the Amazon EKS cluster.
OIDC authentication is supported for hybrid nodes-enabled Amazon EKS clusters.
You can use Amazon EKS Pod Identities and IAM Roles for Service Accounts (IRSA) with applications running on hybrid nodes to enable granular access for your Pods running on hybrid nodes with other AWS-services.
You can use Amazon GuardDuty EKS Protection with hybrid nodes-enabled Amazon EKS clusters to analyze activities of users and applications accessing your cluster.

Add-ons for hybrid nodes

For detailed information, see hybrid-nodes-add-ons.title.

Container Networking Interface (CNI): The AWS VPC CNI can’t be used with hybrid nodes. The core capabilities of Cilium and Calico are supported for use with hybrid nodes. You can manage your CNI with your choice of tooling such as Helm. For more information, see hybrid-nodes-cni.title.
kube-proxy and CoreDNS: kube-proxy and CoreDNS are installed automatically when hybrid nodes join the Amazon EKS cluster. These add-ons can be managed as Amazon EKS add-ons after cluster creation.
Ingress and Load Balancing: You can use the AWS Load Balancer Controller and Application Load Balancer (ALB) or Network Load Balancer (NLB) with the target type ip for workloads on hybrid nodes connected with AWS Direct Connect or AWS Site-to-Site VPN. You can alternatively use your choice of Ingress controller or load balancer for application traffic that stays local to your on-premises environment.
Metrics: You can use Amazon Managed Prometheus (AMP) agent-less scrapers, AWS Distro for Open Telemetry (ADOT), and the Amazon CloudWatch Observability Agent with hybrid nodes. To use AMP agent-less scrapers for Pod metrics on hybrid nodes, your Pods must be accessible from the VPC that you use for the Amazon EKS cluster.
Logs: You can enable Amazon EKS control plane logging for hybrid nodes-enabled clusters. You can use the ADOT EKS add-on and the Amazon CloudWatch Observability Agent EKS add-on for hybrid node and Pod logging.

User interfaces

Node management: The Amazon EKS Hybrid Nodes CLI is called nodeadm and is run on each on-premises host to simplify the installation, configuration, registration, and uninstall of the hybrid nodes components. The hybrid nodes nodeadm version is different than the nodeadm version used in the AL2023 Amazon EKS-optimized AMIs. You should not use the hybrid nodes nodeadm version for nodes running in Amazon EC2.
Cluster management: The Amazon EKS user interfaces for cluster management are the same with hybrid nodes-enabled Amazon EKS clusters. This includes the consolelong, AWS API, AWS SDKs, CLI, eksctl CLI, CFN, and Terraform.

9.8.2. Prerequisite setup for hybrid nodes

Learn about the prerequisites and requirements for joining nodes from your data centers to Amazon EKS Kubernetes clusters with Amazon EKS Hybrid Nodes.

To use Amazon EKS Hybrid Nodes, you must have private connectivity from your on-premises environment to/from AWS, bare metal servers or virtual machines with a supported operating system, and AWS IAM Roles Anywhere or AWS Systems Manager (SSM) hybrid activations configured. You are responsible for managing these prerequisites throughout the hybrid nodes lifecycle.

Hybrid network connectivity from your on-premises environment to/from AWS
Infrastructure in the form of physical or virtual machines
Operating system that is compatible with hybrid nodes
On-premises IAM credentials provider configured

Hybrid network connectivity

The communication between the Amazon EKS control plane and hybrid nodes is routed through the VPC and subnets you pass during cluster creation, which builds on the existing mechanism in Amazon EKS for control plane to node networking. There are several documented options available for you to connect your on-premises environment with your VPC including AWS Site-to-Site VPN, AWS Direct Connect, or your own VPN connection. Reference the AWS Site-to-Site VPN and AWS Direct Connect user guides for more information on how to use those solutions for your hybrid network connection.

For an optimal experience, AWS recommends reliable network connectivity of at least 100 Mbps and a maximum of 200ms round trip latency for the hybrid nodes connection to the AWS Region. The bandwidth and latency requirements can vary depending on the number of hybrid nodes and your workload characteristics, such as application image size, application elasticity, monitoring and logging configurations, and application dependencies on accessing data stored in other AWS services. We recommend that you test with your own applications and environments before deploying to production to validate that your networking setup meets the requirements for your workloads.

On-premises network configuration

You must enable inbound network access from the Amazon EKS control plane to your on-premises environment to allow the Amazon EKS control plane to communicate with the kubelet running on hybrid nodes and optionally with webhooks running on your hybrid nodes. Additionally, you must enable outbound network access for your hybrid nodes and components running on them to communicate with the Amazon EKS control plane. You can configure this communication to stay fully private to your AWS Direct Connect, AWS Site-to-Site VPN, or your own VPN connection. For a full list of the required ports and protocols that you must enable in your firewall and on-premises environment, see hybrid-nodes-networking.title.

The Classless Inter-Domain Routing (CIDR) ranges you use for your on-premises node and pod networks must use IPv4 RFC1918 address ranges. When you create your hybrid nodes-enabled Amazon EKS cluster, you pass your on-premises node and optionally pod CIDRs to enable communication from the Amazon EKS control plane to your hybrid nodes and the resources running on them. Your on-premises router must be configured with routes to your on-premises nodes and optionally pods. You can use Border Gateway Protocol (BGP) or static configurations to advertise pod IPs to your router.

EKS cluster configuration

To minimize latency, it is recommended to create your Amazon EKS cluster in the AWS Region closest to your on-premises or edge environment. You pass your on-premises node and pod CIDRs during Amazon EKS cluster creation via two API fields: RemoteNodeNetwork and RemotePodNetwork. You may need to discuss with your on-premises network team to identify your on-premises node and pod CIDRs. The node CIDR is allocated from your on-premises network and the pod CIDR is allocated from the Container Network Interface (CNI) you use if you are using an overlay network for your CNI.

The on-premises node and pod CIDRs are used to configure the Amazon EKS control plane to route traffic through your VPC to the kubelet and the pods running on your hybrid nodes. Your on-premises node and pod CIDRs cannot overlap with each other, the VPC CIDR you pass during cluster creation, or the service IPv4 configuration for your Amazon EKS cluster. The pod CIDR is optional. You must configure your pod CIDR if your CNI does not use Network Address Translation (NAT) or masquerading for pod IP addresses when pod traffic leaves your on-premises hosts. You additionally must configure your pod CIDR if you are running Kubernetes webhooks on hybrid nodes. For example, AWS Distro for Open Telemetry (ADOT) uses webhooks.

It is recommended to use either public or private endpoint access for the Amazon EKS Kubernetes API server endpoint. If you choose “Public and Private”, the Amazon EKS Kubernetes API server endpoint will always resolve to the public IPs for hybrid nodes running outside of your VPC, which can prevent your hybrid nodes from joining the cluster. You can use either public or private endpoint access for the Amazon EKS Kubernetes API server endpoint. You cannot choose “Public and Private”. When you use public endpoint access, the Kubernetes API server endpoint is resolved to public IPs and the communication from hybrid nodes to the Amazon EKS control plane will be routed over the internet. When you choose private endpoint access, the Kubernetes API server endpoint is resolved to private IPs and the communication from hybrid nodes to the Amazon EKS control plane will be routed over your private connectivity link, in most cases AWS Direct Connect or AWS Site-to-Site VPN.

VPC configuration

You must configure the VPC you pass during Amazon EKS cluster creation with routes in its routing table for your on-premises node and optionally pod networks with your virtual private gateway (VGW) or transit gateway (TGW) as the target. An example is shown below. Replace REMOTE_NODE_CIDR and REMOTE_POD_CIDR with the values for your on-premises network.

Destination

Target

Description

10.226.0.0/16

local

Traffic local to the VPC routes within the VPC

REMOTE_NODE_CIDR

tgw-abcdef123456

On-prem node CIDR, route traffic to the TGW

REMODE_POD_CIDR

tgw-abcdef123456

On-prem pod CIDR, route traffic to the TGW

Security group configuration

When you create a cluster, Amazon EKS creates a security group that’s named eks-cluster-sg-<cluster-name>-<uniqueID>. You cannot alter the inbound rules of this Cluster Security Group but you can restrict the outbound rules. You must add an additional security group to your cluster to enable the kubelet and optionally webhooks running on your hybrid nodes to contact the Amazon EKS control plane. The required inbound rules for this additional security group are shown below. Replace REMOTE_NODE_CIDR and REMOTE_POD_CIDR with the values for your on-premises network.

Name

Security group rule ID

IP version

Type

Protocol

Port range

Source

On-prem node inbound

sgr-abcdef123456

IPv4

HTTPS

TCP

443

REMOTE_NODE_CIDR

On-prem pod inbound

sgr-abcdef654321

IPv4

HTTPS

TCP

443

REMOTE_POD_CIDR

Infrastructure

You must have bare metal servers or virtual machines available to use as hybrid nodes. Hybrid nodes are agnostic to the underlying infrastructure and support x86 and ARM architectures. Amazon EKS Hybrid Nodes follows a “bring your own infrastructure” approach, where you are responsible for provisioning and managing the bare metal servers or virtual machines that you use for hybrid nodes. While there is not a strict minimum resource requirement, it is recommended to use hosts with at least 1 vCPU and 1GiB RAM for hybrid nodes.

Operating system

Amazon Linux 2023 (AL2023), Ubuntu, and RHEL are validated on an ongoing basis for use as the node operating system for hybrid nodes. AWS supports the hybrid nodes integration with these operating systems but does not provide support for the operating systems itself. AL2023 is not covered by AWS Support Plans when run outside of Amazon EC2. AL2023 can only be used in on-premises virtualized environments, see the Amazon Linux 2023 User Guide for more information.

You are responsible for operating system provisioning and management. When you are testing hybrid nodes for the first time, it is easiest to run the Amazon EKS Hybrid Nodes CLI (nodeadm) on an already provisioned host. For production deployments, it is recommended to include nodeadm in your golden operating system images with it configured to run as a systemd service to automatically join hosts to Amazon EKS clusters at host startup.

On-premises IAM credentials provider

Amazon EKS Hybrid Nodes use temporary IAM credentials provisioned by AWS SSM hybrid activations or AWS IAM Roles Anywhere to authenticate with the Amazon EKS cluster. You must use either AWS SSM hybrid activations or AWS IAM Roles Anywhere with the Amazon EKS Hybrid Nodes CLI (nodeadm). It is recommended to use AWS SSM hybrid activations if you do not have existing Public Key Infrastructure (PKI) with a Certificate Authority (CA) and certificates for your on-premises environments. If you do have existing PKI and certificates on-premises, use AWS IAM Roles Anywhere.

Similar to the create-node-role.title for nodes running on Amazon EC2, you will create a Hybrid Nodes IAM Role with the required permissions to join hybrid nodes to Amazon EKS clusters. If you are using AWS IAM Roles Anywhere, configure a trust policy that allows AWS IAM Roles Anywhere to assume the Hybrid Nodes IAM Role and configure your AWS IAM Roles Anywhere profile with the Hybrid Nodes IAM Role as an assumable role. If you are using AWS SSM, configure a trust policy that allows AWS SSM to assume the Hybrid Nodes IAM Role and create the hybrid activation with the Hybrid Nodes IAM Role. See hybrid-nodes-creds.title for how to create the Hybrid Nodes IAM Role with the required permissions.

Prepare networking for hybrid nodes

Learn about and configure the VPC and on-premises networking for joining nodes from your data centers to Amazon EKS Kubernetes clusters with Amazon EKS Hybrid Nodes.

This topic provides an overview of the networking setup you must have configured before creating your Amazon EKS cluster and attaching hybrid nodes. This guide assumes you have met the prerequisite requirements for hybrid network connectivity using AWS Site-to-Site VPN, AWS Direct Connect, or your own VPN solution.

On-premises networking configuration

Minimum network requirements

For an optimal experience, AWS recommends reliable network connectivity of at least 100 Mbps and a maximum of 200ms round trip latency for the hybrid nodes connection to the AWS-Region. The bandwidth and latency requirements can vary depending on the number of hybrid nodes and your workload characteristics such as application image size, application elasticity, monitoring and logging configurations, and application dependencies on accessing data stored in other AWS-services.

On-premises node and pod CIDRs

Identify the node and pod CIDRs you will use for your hybrid nodes and the workloads running on them. The node CIDR is allocated from your on-premises network and the pod CIDR is allocated from your Container Network Interface (CNI) if you are using an overlay network for your CNI. You pass your on-premises node CIDRs and optionally pod CIDRs as inputs when you create your Amazon EKS cluster with the RemoteNodeNetwork and RemotePodNetwork fields.

The on-premises node and pod CIDR blocks must meet the following requirements:

Be within one of the following IPv4 RFC-1918 ranges: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
Not overlap with each other, the VPC CIDR for your Amazon EKS cluster, or your Kubernetes service IPv4 CIDR.

If your CNI performs Network Address Translation (NAT) for pod traffic as it leaves your on-premises hosts, you do not need to advertise your pod CIDR to your on-premises network or configure your Amazon EKS cluster with your remote pod network for hybrid nodes to become ready to workloads. If your CNI does not use NAT for pod traffic as it leaves your on-premises hosts, you must advertise your pod CIDR with your on-premises network and you must configure your Amazon EKS cluster with your remote pod network for hybrid nodes to become ready to workloads. If you are running webhooks on your hybrid nodes, you must advertise your pod CIDR to your on-premises network and configure your Amazon EKS cluster with your remote pod network so the Amazon EKS control plane can directly connect to the webhooks running on hybrid nodes.

Access required during hybrid node installation and upgrade

You must have access to the following domains during the installation process where you install the hybrid nodes dependencies on your hosts. This process can be done once when you are building your operating system images or it can be done on each host at runtime. This includes initial installation and when you upgrade the Kubernetes version of your hybrid nodes.

Component URL Protocol Port

EKS node artifacts (S3)

https://hybrid-assets.eks.amazonaws.com

HTTPS

443

EKS service endpoints

https://eks.region.amazonaws.com

HTTPS

443

EKS ECR endpoints

See add-ons-images.title for regional endpoints.

HTTPS

443

SSM binary endpoint ¹

https://amazon-ssm-region.s3.region.amazonaws.com

HTTPS

443

SSM service endpoint ¹

https://ssm.region.amazonaws.com

HTTPS

443

IAM Anywhere binary endpoint ²

https://rolesanywhere.amazonaws.com

HTTPS

443

IAM Anywhere service endpoint ²

https://rolesanywhere.region.amazonaws.com

HTTPS

443

¹ Access to the AWS SSM endpoints are only required if you are using AWS SSM hybrid activations for your on-premises IAM credential provider.

² Access to the AWS IAM endpoints are only required if you are using AWS IAM Roles Anywhere for your on-premises IAM credential provider.

Access required for ongoing cluster operations

The following network access for your on-premises firewall is required for ongoing cluster operations.

Depending on your choice of CNI, you need to configure additional network access rules for the CNI ports. See the Cilium documentation and the Calico documentation for details.

Type Protocol Direction Port Source Destination Usage

HTTPS

TCP

Outbound

443

Remote Node CIDR(s)

EKS cluster IPs ¹

kubelet to Kubernetes API server

HTTPS

TCP

Outbound

443

Remote Pod CIDR(s)

EKS cluster IPs ¹

Pod to Kubernetes API server

HTTPS

TCP

Outbound

443

Remote Node CIDR(s)

SSM service endpoint

SSM hybrid activations credential refresh and SSM heartbeats every 5 minutes

HTTPS

TCP

Outbound

443

Remote Node CIDR(s)

IAM Anywhere service endpoint

IAM Roles Anywhere credential refresh

HTTPS

TCP

Outbound

443

Remote Pod CIDR(s)

STS Regional Endpoint

Pod to STS endpoint, only required for IRSA

HTTPS

TCP

Outbound

443

Remote Node CIDR(s)

Amazon EKS Auth service endpoint

Node to Amazon EKS Auth endpoint, only required for Amazon EKS Pod Identity

HTTPS

TCP

Inbound

10250

EKS cluster IPs ¹

Remote Node CIDR(s)

kubelet to Kubernetes API server

HTTPS

TCP

Inbound

Webhook ports

EKS cluster IPs ¹

Remote Pod CIDR(s)

Kubernetes API server to webhooks

HTTPS

TCP,UDP

Inbound,Outbound

Remote Pod CIDR(s)

Pod to CoreDNS. If you run at least 1 replica of CoreDNS in the cloud, you must allow DNS traffic to the VPC where CoreDNS is running.

User-defined

Inbound,Outbound

App ports

Remote Pod CIDR(s)

Pod to Pod

¹ The IPs of the Amazon EKS cluster. See the following section on Amazon EKS elastic network interfaces.

Amazon EKS network interfaces

Amazon EKS attaches network interfaces to the subnets in the VPC you pass during cluster creation to enable the communication between the Amazon EKS control plane and your VPC. The network interfaces that Amazon EKS creates can be found after cluster creation in the Amazon EC2 console or with the CLI. The original network interfaces are deleted and new network interfaces are created when changes are applied on your Amazon EKS cluster, such as Kubernetes version upgrades. You can restrict the IP range for the Amazon EKS network interfaces by using constrained subnet sizes for the subnets you pass during cluster creation, which makes it easier to configure your on-premises firewall to allow inbound/outbound connectivity to this known, constrained set of IPs. To control which subnets network interfaces are created in, you can limit the number of subnets you specify when you create a cluster or you can update the subnets after creating the cluster.

The network interfaces provisioned by Amazon EKS have a description of the format Amazon EKS your-cluster-name. See the example below for an CLI command you can use to find the IP addresses of the network interfaces that Amazon EKS provisions. Replace VPC_ID with the ID of the VPC you pass during cluster creation.

aws ec2 describe-network-interfaces \
--query 'NetworkInterfaces[?(VpcId == VPC_ID && contains(Description,Amazon EKS))].PrivateIpAddress'

`AWS` VPC and subnet setup

The existing VPC and subnet requirements for Amazon EKS apply to clusters with hybrid nodes. Additionally, your VPC CIDR can’t overlap with your on-premises node and pod CIDRs. You must configure routes in your VPC routing table for your on-premises node and optionally pod CIDRs. These routes must be setup to route traffic to the gateway you are using for your hybrid network connectivity, which is commonly a virtual private gateway (VGW) or transit gateway (TGW). If you are using TGW or VGW to connect your VPC with your on-premises environment, you must create a TGW or VGW attachment for your VPC. Your VPC must have DNS hostname and DNS resolution support.

The following steps use the CLI. You can also create these resources in the consolelong or with other interfaces such as AWS CloudFormation, AWS CDK, or Terraform.

Step 1: Create VPC

Run the following command to create a VPC. Replace VPC_CIDR with an IPv4 RFC-1918 (private) or non-RFC-1918 (public) CIDR range (for example 10.0.0.0/16). Note: DNS resolution, which is an EKS requirement, is enabled for the VPC by default.
```
aws ec2 create-vpc --cidr-block VPC_CIDR
```
Enable DNS hostnames for your VPC. Note, DNS resolution is enabled for the VPC by default. Replace VPC_ID with the ID of the VPC you created in the previous step.
```
aws ec2 modify-vpc-attribute --vpc-id VPC_ID --enable-dns-hostnames
```

Step 2: Create subnets

Create at least 2 subnets. Amazon EKS uses these subnets for the cluster network interfaces. For more information, see the Subnets requirements and considerations.

You can find the availability zones for an AWS-Region with the following command. Replace us-west-2 with your region.
```
aws ec2 describe-availability-zones \
     --query 'AvailabilityZones[?(RegionName == us-west-2)].ZoneName'
```
Create a subnet. Replace VPC_ID with the ID of the VPC. Replace SUBNET_CIDR with the CIDR block for your subnet (for example 10.0.1.0/24 ). Replace AZ with the availability zone where the subnet will be created (for example us-west-2a). The subnets you create must be in at least 2 different availability zones.
```
aws ec2 create-subnet \
    --vpc-id VPC_ID \
    --cidr-block SUBNET_CIDR \
    --availability-zone AZ
```

(Optional) Step 3: Attach VPC with VPC Transit Gateway (TGW) or AWS-DC virtual private gateway (VGW)

If you are using a TGW or VGW, attach your VPC to the TGW or VGW. For more information, see VPC attachments in VPC Transit Gateways or AWS Direct Connect virtual private gateway associations.

Transit Gateway

Run the following command to attach a Transit Gateway. Replace VPC_ID with the ID of the VPC. Replace SUBNET_ID1 and SUBNET_ID2 with the IDs of the subnets you created in the previous step. Replace TGW_ID with the ID of your TGW.

aws ec2 create-transit-gateway-vpc-attachment \
    --vpc-id VPC_ID \
    --subnet-ids SUBNET_ID1 SUBNET_ID2 \
    --transit-gateway-id TGW_ID

Virtual Private Gateway

Run the following command to attach a Transit Gateway. Replace VPN_ID with the ID of your VGW. Replace VPC_ID with the ID of the VPC.

aws ec2 attach-vpn-gateway \
    --vpn-gateway-id VPN_ID \
    --vpc-id VPC_ID

(Optional) Step 4: Create route table

You can modify the main route table for the VPC or you can create a custom route table. The following steps create a custom route table with the routes to on-premises node and pod CIDRs. For more information, see Subnet route tables. Replace VPC_ID with the ID of the VPC.

aws ec2 create-route-table --vpc-id VPC_ID

Step 5: Create routes for on-premises nodes and pods

Create routes in the route table for each of your on-premises remote nodes. You can modify the main route table for the VPC or use the custom route table you created in the previous step.

The examples below show how to create routes for your on-premises node and pod CIDRs. In the examples, a transit gateway (TGW) is used to connect the VPC with the on-premises environment. If you have multiple on-premises node and pods CIDRs, repeat the steps for each CIDR.

If you are using an internet gateway or a virtual private gateway (VGW) replace --transit-gateway-id with --gateway-id.
Replace RT_ID with the ID of the route table you created in the previous step.
Replace REMOTE_NODE_CIDR with the CIDR range you will use for your hybrid nodes.
Replace REMOTE_POD_CIDR with the CIDR range you will use for the pods running on hybrid nodes. The pod CIDR range corresponds to the Container Networking Interface (CNI) configuration, which most commonly uses an overlay network on-premises. For more information, see hybrid-nodes-cni.title.
Replace TGW_ID with the ID of your TGW.

Remote node network

aws ec2 create-route \
    --route-table-id RT_ID \
    --destination-cidr-block REMOTE_NODE_CIDR \
    --transit-gateway-id TGW_ID

Remote Pod network

aws ec2 create-route \
    --route-table-id RT_ID \
    --destination-cidr-block REMOTE_POD_CIDR \
    --transit-gateway-id TGW_ID

(Optional) Step 6: Associate subnets with route table

If you created a custom route table in the previous step, associate each of the subnets you created in the previous step with your custom route table. If you are modifying the VPC main route table, the subnets are automatically associated with the main route table of the VPC and you can skip this step.

Run the following command for each of the subnets you created in the previous steps. Replace RT_ID with the route table you created in the previous step. Replace SUBNET_ID with the ID of a subnet.

aws ec2 associate-route-table --route-table-id RT_ID --subnet-id SUBNET_ID

Cluster security group configuration

The following access for your Amazon EKS cluster security group is required for ongoing cluster operations.

Type

Protocol

Direction

Port

Source

Destination

Usage

HTTPS

TCP

Inbound

443

Remote Node CIDR(s)

N/A

Kubelet to Kubernetes API server

HTTPS

TCP

Inbound

443

Remote Pod CIDR(s)

N/A

Pods requiring access to K8s API server when the CNI is not using NAT for the pod traffic.

HTTPS

TCP

Outbound

10250

N/A

Remote Node CIDR(s)

Kubernetes API server to Kubelet

HTTPS

TCP

Outbound

Webhook ports

N/A

Remote Pod CIDR(s)

Kubernetes API server to webhook (if running webhooks on hybrid nodes)

To create a security group with the inbound access rules, run the following commands. This security group must be passed when you create your Amazon EKS cluster. By default, the command below creates a security group that allows all outbound access. You can restrict outbound access to include only the rules above. If you’re considering limiting the outbound rules, we recommend that you thoroughly test all of your applications and pod connectivity before you apply your changed rules to a production cluster.

In the first command, replace SG_NAME with a name for your security group
In the first command, replace VPC_ID with the ID of the VPC you created in the previous step
In the second command, replace SG_ID with the ID of the security group you create in the first command
In the second command, replace REMOTE_NODE_CIDR and REMOTE_POD_CIDR with the values for your hybrid nodes and on-premises network.

aws ec2 create-security-group \
    --group-name SG_NAME \
    --description "security group for hybrid nodes" \
    --vpc-id VPC_ID

aws ec2 authorize-security-group-ingress \
    --group-id SG_ID \
    --ip-permissions '[{"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "REMOTE_NODE_CIDR"}, {"CidrIp": "REMOTE_POD_CIDR"}]}]'

Prepare operating system for hybrid nodes

Prepare operating system for use with Hybrid Nodes

Amazon Linux 2023 (AL2023), Ubuntu, and Red Hat Enterprise Linux (RHEL) are validated on an ongoing basis for use as the node operating system for hybrid nodes. AWS supports the hybrid nodes integration with these operating systems but does not provide support for the operating systems itself. AL2023 is not covered by AWS Support Plans when run outside of Amazon EC2. AL2023 can only be used in on-premises virtualized environments, reference the Amazon Linux 2023 User Guide for more information.

You are responsible for operating system provisioning and management. When you are testing hybrid nodes for the first time, it is easiest to run the Amazon EKS Hybrid Nodes CLI (nodeadm) on an already provisioned host. For production deployments, it is recommended to include nodeadm in your operating system images with it configured to run as a systemd service to automatically join hosts to Amazon EKS clusters at host startup.

Version compatibility

The table below represents the operating system versions that are compatible and validated to use as the node operating system for hybrid nodes. If you are using other operating system variants or versions that are not included in this table, then the compatibility of hybrid nodes with your operating system variant or version is not covered by AWS Support. Hybrid nodes are agnostic to the underlying infrastructure and support x86 and ARM architectures.

Operating System

Versions

Amazon Linux

Amazon Linux 2023 (AL2023)

Ubuntu

Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04

Red Hat Enterprise Linux

RHEL 8, RHEL 9

Operating system considerations

General

The Amazon EKS Hybrid Nodes CLI (nodeadm) can be used to simplify the installation and configuration of the hybrid nodes components and dependencies. You can run the nodeadm install process during your operating system image build pipelines or at runtime on each on-premises host. For more information on the components that nodeadm installs, see the hybrid-nodes-nodeadm.title.
If you are using a proxy in your on-premises environment to reach the internet, there is additional operating system configuration required for the install and upgrade processes to configure your package manager to use the proxy. See hybrid-nodes-proxy.title for instructions.

Containerd

Containerd is the standard Kubernetes container runtime and is a dependency for hybrid nodes, as well as all Amazon EKS node compute types. The Amazon EKS Hybrid Nodes CLI (nodeadm) attempts to install containerd during the nodeadm install process. You can configure the containerd installation at nodeadm install runtime with the --containerd-source command line option. Valid options are none, distro, and docker. If you are using RHEL, distro is not a valid option and you can either configure nodeadm to install the containerd build from Docker’s repos or you can manually install containerd. When using AL2023 or Ubuntu, nodeadm defaults to installing containerd from the operating system distribution. If you do not want nodeadm to install containerd, use the --containerd-source none option.

Ubuntu

If you are using Ubuntu 20.04, you must use AWS Systems Manager hybrid activations as your credential provider. AWS IAM Roles Anywhere is not supported on Ubuntu 20.04.
If you are using Ubuntu 24.04, you may need to update your version of containerd or change your AppArmor configuration to adopt a fix that allows pods to properly terminate, see Ubuntu #2065423. A reboot is required to apply changes to the AppArmor profile. The latest version of Ubuntu 24.04 has an updated containerd version in its package manager with the fix (containerd version 1.7.19+).

RHEL

If you are using RHEL 8, you must use AWS Systems Manager hybrid activations as your credential provider. AWS IAM Roles Anywhere isn’t supported on RHEL 8.

Building operating system images

Amazon EKS provides example Packer templates you can use to create operating system images that include nodeadm and configure it to run at host-startup. This process is recommended to avoid pulling the hybrid nodes dependencies individually on each host and to automate the hybrid nodes bootstrap process. You can use the example Packer templates with an Ubuntu 22.04, Ubuntu 24.04, RHEL 8 or RHEL 9 ISO image and can output images with these formats: OVA, Qcow2, or raw.

Prerequisites

Before using the example Packer templates, you must have the following installed on the machine from where you are running Packer.

Packer version 1.11.0 or higher. For instructions on installing Packer, see Install Packer in the Packer documentation.
If building OVAs, VMware vSphere plugin 1.4.0 or higher
If building Qcow2 or raw images, QEMU plugin version 1.x

Set Environment Variables

Before running the Packer build, set the following environment variables on the machine from where you are running Packer.

General

The following environment variables must be set for building images with all operating systems and output formats.

Environment Variable Type Description

PKR_SSH_PASSWORD

String

Packer uses the ssh_username and ssh_password variables to SSH into the created machine when provisioning. This needs to match the passwords used to create the initial user within the respective OS’s kickstart or user-data files. The default is set as "builder" or "ubuntu" depending on the OS. When setting your password, make sure to change it within the corresponding ks.cfg or user-data file to match.

ISO_URL

String

URL of the ISO to use. Can be a web link to download from a server, or an absolute path to a local file

ISO_CHECKSUM

String

Associated checksum for the supplied ISO.

CREDENTIAL_PROVIDER

String

Credential provider for hybrid nodes. Valid values are ssm (default) for SSM hybrid activations and iam for IAM Roles Anywhere

K8S_VERSION

String

Kubernetes version for hybrid nodes (for example 1.31). For supported Kubernetes versions, see kubernetes-versions.title.

NODEADM_ARCH

String

Architecture for nodeadm install. Select amd or arm.

RHEL

If you are using RHEL, the following environment variables must be set.

Environment Variable Type Description

RH_USERNAME

String

RHEL subscription manager username

RH_PASSWORD

String

RHEL subscription manager password

RHEL_VERSION

String

Rhel iso version being used. Valid values are 8 or 9.

Ubuntu

There are no Ubuntu-specific environment variables required.

vSphere

If you are building a VMware vSphere OVA, the following environment variables must be set.

Environment Variable

Type

Description

VSPHERE_SERVER

String

vSphere server address

VSPHERE_USER

String

vSphere username

VSPHERE_PASSWORD

String

vSphere password

VSPHERE_DATACENTER

String

vSphere datacenter name

VSPHERE_CLUSTER

String

vSphere cluster name

VSPHERE_DATASTORE

String

vSphere datastore name

VSPHERE_NETWORK

String

vSphere network name

VSPHERE_OUTPUT_FOLDER

String

vSphere output folder for the templates

QEMU

Environment Variable Type Description

PACKER_OUTPUT_FORMAT

String

Output format for the QEMU builder. Valid values are qcow2 and raw.

Validate template

Before running your build, validate your template with the following command after setting your environment variables. Replace template.pkr.hcl if you are using a different name for your template.

packer validate template.pkr.hcl

Build images

Build your images with the following commands and use the -only flag to specify the target and operating system for your images. Replace template.pkr.hcl if you are using a different name for your template.

vSphere OVAs

If you are using RHEL with vSphere you need to convert the kickstart files to an OEMDRV image and pass it as an ISO to boot from. For more information, see the Packer Readme in the EKS Hybrid Nodes GitHub Repository.

Ubuntu 22.04 OVA

packer build -only=general-build.vsphere-iso.ubuntu22 template.pkr.hcl

Ubuntu 24.04 OVA

packer build -only=general-build.vsphere-iso.ubuntu24 template.pkr.hcl

RHEL 8 OVA

packer build -only=general-build.vsphere-iso.rhel8 template.pkr.hcl

RHEL 9 OVA

packer build -only=general-build.vsphere-iso.rhel9 template.pkr.hcl

QEMU

If you are building an image for a specific host CPU that does not match your builder host, see the QEMU documentation for the name that matches your host CPU and use the -cpu flag with the name of the host CPU when you run the following commands.

Ubuntu 22.04 Qcow2 / Raw

packer build -only=general-build.qemu.ubuntu22 template.pkr.hcl

Ubuntu 24.04 Qcow2 / Raw

packer build -only=general-build.qemu.ubuntu24 template.pkr.hcl

RHEL 8 Qcow2 / Raw

packer build -only=general-build.qemu.rhel8 template.pkr.hcl

RHEL 9 Qcow2 / Raw

packer build -only=general-build.qemu.rhel9 template.pkr.hcl

Pass nodeadm configuration through user-data

You can pass configuration for nodeadm in your user-data through cloud-init to configure and automatically connect hybrid nodes to your EKS cluster at host startup. Below is an example for how to accomplish this when using VMware vSphere as the infrastructure for your hybrid nodes.

Install the the govc CLI following the instructions in the govc readme on GitHub.
After running the Packer build in the previous section and provisioning your template, you can clone your template to create multiple different nodes using the following. You must clone the template for each new VM you are creating that will be used for hybrid nodes. Replace the variables in the command below with the values for your environment. The VM_NAME in the command below is used as your NODE_NAME when you inject the names for your VMs via your metadata.yaml file.
```
govc vm.clone -vm "/PATH/TO/TEMPLATE" -ds="YOUR_DATASTORE" \
    -on=false -template=false -folder=/FOLDER/TO/SAVE/VM "VM_NAME"
```

After cloning the template for each of your new VMs, create a userdata.yaml and metadata.yaml for your VMs. Your VMs can share the same userdata.yaml and metadata.yaml and you will populate these on a per VM basis in the steps below. The nodeadm configuration is created and defined in the write_files section of your userdata.yaml. The example below uses AWS SSM hybrid activations as the on-premises credential provider for hybrid nodes. For more information on nodeadm configuration, see the hybrid-nodes-nodeadm.title.

userdata.yaml:

#cloud-config
users:
  - name: # username for login. Use 'builder' for RHEL or 'ubuntu' for Ubuntu.
    passwd: # password to login. Default is 'builder' for RHEL.
    groups: [adm, cdrom, dip, plugdev, lxd, sudo]
    lock-passwd: false
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash

write_files:
  - path: /usr/local/bin/nodeConfig.yaml
    permissions: '0644'
    content: |
      apiVersion: node.eks.aws/v1alpha1
      kind: NodeConfig
      spec:
          cluster:
              name: # Cluster Name
              region: # AWS region
          hybrid:
              ssm:
                  activationCode: # Your ssm activation code
                  activationId: # Your ssm activation id

runcmd:
  - /usr/local/bin/nodeadm init -c file:///usr/local/bin/nodeConfig.yaml >> /var/log/nodeadm-init.log 2>&1

metadata.yaml:

Create a metadata.yaml for your environment. Keep the "$NODE_NAME" variable format in the file as this will be populated with values in a subsequent step.

instance-id: "$NODE_NAME"
local-hostname: "$NODE_NAME"
network:
  version: 2
  ethernets:
    nics:
      match:
        name: ens*
      dhcp4: yes

Add the userdata.yaml and metadata.yaml files as gzip+base64 strings with the following commands. The following commands should be run for each of the VMs you are creating. Replace VM_NAME with the name of the VM you are updating.

export NODE_NAME="VM_NAME"
export USER_DATA=$(gzip -c9 <userdata.yaml | base64)

govc vm.change -dc="YOUR_DATASTORE" -vm "$NODE_NAME" -e guestinfo.userdata="${USER_DATA}"
govc vm.change -dc="YOUR_DATASTORE" -vm "$NODE_NAME" -e guestinfo.userdata.encoding=gzip+base64

envsubst '$NODE_NAME' < metadata.yaml > metadata.yaml.tmp
export METADATA=$(gzip -c9 <metadata.yaml.tmp | base64)

govc vm.change -dc="YOUR_DATASTORE" -vm "$NODE_NAME" -e guestinfo.metadata="${METADATA}"
govc vm.change -dc="YOUR_DATASTORE" -vm "$NODE_NAME" -e guestinfo.metadata.encoding=gzip+base64

Power on your new VMs, which should automatically connect to the EKS cluster you configured.
```
govc vm.power -on "${NODE_NAME}"
```

Prepare credentials for hybrid nodes

Prepare credentials to authenticate hybrid nodes with Amazon EKS clusters

Amazon EKS Hybrid Nodes use temporary IAM credentials provisioned by AWS SSM hybrid activations or AWS IAM Roles Anywhere to authenticate with the Amazon EKS cluster. You must use either AWS SSM hybrid activations or AWS IAM Roles Anywhere with the Amazon EKS Hybrid Nodes CLI (nodeadm). You should not use both AWS SSM hybrid activations and AWS IAM Roles Anywhere. It is recommended to use AWS SSM hybrid activations if you do not have existing Public Key Infrastructure (PKI) with a Certificate Authority (CA) and certificates for your on-premises environments. If you do have existing PKI and certificates on-premises, use AWS IAM Roles Anywhere.

Hybrid Nodes IAM Role

Before you can connect hybrid nodes to your Amazon EKS cluster, you must create an IAM role that will be used with AWS SSM hybrid activations or AWS IAM Roles Anywhere for your hybrid nodes credentials. After cluster creation, you will use this role with an Amazon EKS access entry or aws-auth ConfigMap entry to map the IAM role to Kubernetes Role-Based Access Control (RBAC). For more information on associating the Hybrid Nodes IAM role with Kubernetes RBAC, see hybrid-nodes-cluster-prep.title.

The Hybrid Nodes IAM role must have the following permissions.

Permissions for nodeadm to use the eks:DescribeCluster action to gather information about the cluster used for connecting hybrid nodes to the cluster. If you do not enable the eks:DescribeCluster action, then you must pass your Kubernetes API endpoint, cluster CA bundle, and service IPv4 CIDR in the node configuration you pass to nodeadm when you run nodeadm init.
Permissions for the kubelet to use container images from Amazon Elastic Container Registry (Amazon ECR) as defined in the AmazonEC2ContainerRegistryPullOnly policy.
If using AWS SSM, permissions for nodeadm init to use AWS SSM hybrid activations as defined in the aws-managed-policy/latest/reference/AmazonSSMManagedInstanceCore.html policy.
If using AWS SSM, permissions to use the ssm:DeregisterManagedInstance action and ssm:DescribeInstanceInformation action for nodeadm uninstall to deregister instances.
(Optional) Permissions for the Amazon EKS Pod Identity Agent to use the eks-auth:AssumeRoleForPodIdentity action to retrieve credentials for pods.

Setup `AWS` SSM hybrid activations

Before setting up AWS SSM hybrid activations, you must have a Hybrid Nodes IAM role created and configured. For more information, see hybrid-nodes-creds.html. Follow the instructions at Create a hybrid activation to register nodes with Systems Manager in the AWS Systems Manager User Guide to create an AWS SSM hybrid activation for your hybrid nodes. The Activation Code and ID you receive is used with nodeadm when you register your hosts as hybrid nodes with your Amazon EKS cluster. You can come back to this step at a later point after you have created and prepared your Amazon EKS clusters for hybrid nodes.

Systems Manager immediately returns the Activation Code and ID to the console or the command window, depending on how you created the activation. Copy this information and store it in a safe place. If you navigate away from the console or close the command window, you might lose this information. If you lose it, you must create a new activation.

By default, AWS SSM hybrid activations are active for 24 hours. You can alternatively specify an --expiration-date when you create your hybrid activation in timestamp format, such as 2024-08-01T00:00:00. When you use AWS SSM as your credential provider, the node name for your hybrid nodes is not configurable, and is auto-generated by AWS SSM. You can view and manage the AWS SSM Managed Instances in the AWS Systems Manager console under Fleet Manager. You can register up to 1,000 standard hybrid-activated nodes per account per AWS Region at no additional cost. However, registering more than 1,000 hybrid nodes requires that you activate the advanced-instances tier. There is a charge to use the advanced-instances tier that is not included in the Amazon EKS Hybrid Nodes pricing. For more information, see AWS Systems Manager Pricing.

See the example below for how to create an AWS SSM hybrid activation with your Hybrid Nodes IAM role. When you use AWS SSM hybrid activations for your hybrid nodes credentials, the names of your hybrid nodes will have the format mi-012345678abcdefgh and the temporary credentials provisioned by AWS SSM are valid for 1 hour. You cannot alter the node name or credential duration when using AWS SSM as your credential provider. The temporary credentials are automatically rotated by AWS SSM and the rotation does not impact the status of your nodes or applications.

It is recommended to use one AWS SSM hybrid activation per EKS cluster to scope the AWS SSM ssm:DeregisterManagedInstance permission of the Hybrid Nodes IAM role to only be able to deregister instances that are associated with your AWS SSM hybrid activation. In the example on this page, a tag with the EKS cluster ARN is used, which can be used to map your AWS SSM hybrid activation to the EKS cluster. You can alternatively use your preferred tag and method of scoping the AWS SSM permissions based on your permission boundaries and requirements. The REGISTRATION_LIMIT option in the command below is an integer used to limit the number of machines that can use the AWS SSM hybrid activation (for example 10)

aws ssm create-activation \
     --region AWS_REGION \
     --default-instance-name eks-hybrid-nodes \
     --description "Activation for EKS hybrid nodes" \
     --iam-role AmazonEKSHybridNodesRole \
     --tags Key=EKSClusterARN,Value=arn:aws:eks:AWS_REGION:AWS_ACCOUNT_ID:cluster/CLUSTER_NAME \
     --registration-limit REGISTRATION_LIMIT

Review the instructions on Create a hybrid activation to register nodes with Systems Manager for more information about the available configuration settings for AWS SSM hybrid activations.

Setup `AWS` IAM Roles Anywhere

Follow the instructions at Getting started with IAM Roles Anywhere in the IAM Roles Anywhere User Guide to set up the trust anchor and profile you will use for temporary IAM credentials for your Hybrid Nodes IAM role. When you create your profile, you can create it without adding any roles. You can create this profile, return to these steps to create your Hybrid Nodes IAM role, and then add your role to your profile after it is created. You can alternatively use the AWS CloudFormation steps later on this page to complete your IAM Roles Anywhere setup for hybrid nodes.

When you add the Hybrid Nodes IAM role to your profile, select Accept custom role session name in the Custom role session name panel at the bottom of the Edit profile page in the AWS IAM Roles Anywhere console. This corresponds to the acceptRoleSessionName field of the CreateProfile API. This allows you to supply a custom node name for your hybrid nodes in the configuration you pass to nodeadm during the bootstrap process. Passing a custom node name during the nodeadm init process is required. You can update your profile to accept a custom role session name after creating your profile.

You can configure the credential validity duration with AWS IAM Roles Anywhere through the durationSeconds field of your AWS IAM Roles Anywhere profile. The default duration is 1 hour with a maximum of 12 hours. The MaxSessionDuration setting on your Hybrid Nodes IAM role must be greater than the durationSeconds setting on your AWS IAM Roles Anywhere profile. For more information on MaxSessionDuration, see UpdateRole API documentation.

The per-machine certificates and keys you generate from your certificate authority (CA) must be placed in the /etc/iam/pki directory on each hybrid node with the file names server.pem for the certificate and server.key for the key.

Create the Hybrid Nodes IAM role

To run the steps in this section, the IAM principal using the AWS console or AWS CLI must have the following permissions.

iam:CreatePolicy
iam:CreateRole
iam:AttachRolePolicy
If using AWS IAM Roles Anywhere
- rolesanywhere:CreateTrustAnchor
- rolesanywhere:CreateProfile
- iam:PassRole

AWS CloudFormation

Install and configure the AWS CLI, if you haven’t already. See Installing or updating to the last version of the AWS CLI.

Steps for AWS SSM hybrid activations

The CloudFormation stack creates the Hybrid Nodes IAM Role with the permissions outlined above. The CloudFormation template does not create the AWS SSM hybrid activation.

Download the AWS SSM CloudFormation template for hybrid nodes:

curl -OL 'https://raw.githubusercontent.com/aws/eks-hybrid/refs/heads/main/example/hybrid-ssm-cfn.yaml'

Create a cfn-ssm-parameters.json with the following options:
1. Replace ROLE_NAME with the name for your Hybrid Nodes IAM role. By default, the CloudFormation template uses AmazonEKSHybridNodesRole as the name of the role it creates if you do not specify a name.
2. Replace TAG_KEY with the AWS SSM resource tag key you used when creating your AWS SSM hybrid activation. The combination of the tag key and tag value is used in the condition for the ssm:DeregisterManagedInstance to only allow the Hybrid Nodes IAM role to deregister the AWS SSM managed instances that are associated with your AWS SSM hybrid activation. In the CloudFormation template, TAG_KEY defaults to EKSClusterARN.
3. Replace TAG_VALUE with the AWS SSM resource tag value you used when creating your AWS SSM hybrid activation. The combination of the tag key and tag value is used in the condition for the ssm:DeregisterManagedInstance to only allow the Hybrid Nodes IAM role to deregister the AWS SSM managed instances that are associated with your AWS SSM hybrid activation. If you are using the default TAG_KEY of EKSClusterARN, then pass your EKS cluster ARN as the TAG_VALUE. EKS cluster ARNs have the format arn:aws:eks:AWS_REGION:AWS_ACCOUNT_ID:cluster/CLUSTER_NAME.
  { "Parameters": { "RoleName": "ROLE_NAME", "SSMDeregisterConditionTagKey": "TAG_KEY", "SSMDeregisterConditionTagValue": "TAG_VALUE" } }

Deploy the CloudFormation stack. Replace STACK_NAME with your name for the CloudFormation stack.

aws cloudformation deploy \
    --stack-name STACK_NAME \
    --template-file hybrid-ssm-cfn.yaml \
    --parameter-overrides file://cfn-ssm-parameters.json \
    --capabilities CAPABILITY_NAMED_IAM

Steps for AWS IAM Roles Anywhere

The CloudFormation stack creates the AWS IAM Roles Anywhere trust anchor with the certificate authority (CA) you configure, creates the AWS IAM Roles Anywhere profile, and creates the Hybrid Nodes IAM role with the permissions outlined previously.

To set up a certificate authority (CA)
1. To use an AWS Private CA resource, open the AWS Private Certificate Authority console. Follow the instructions in the AWS Private CA User Guide.
2. To use an external CA, follow the instructions provided by the CA. You provide the certificate body in a later step.
3. Certificates issued from public CAs cannot be used as trust anchors.

Download the AWS IAM Roles Anywhere CloudFormation template for hybrid nodes

curl -OL 'https://raw.githubusercontent.com/aws/eks-hybrid/refs/heads/main/example/hybrid-ira-cfn.yaml'

Create a cfn-iamra-parameters.json with the following options:
1. Replace ROLE_NAME with the name for your Hybrid Nodes IAM role. By default, the CloudFormation template uses AmazonEKSHybridNodesRole as the name of the role it creates if you do not specify a name.
2. Replace CERT_ATTRIBUTE with the per-machine certificate attribute that uniquely identifies your host. The certificate attribute you use must match the nodeName you use for the nodeadm configuration when you connect hybrid nodes to your cluster. For more information, see the hybrid-nodes-nodeadm.title. By default, the CloudFormation template uses ${aws:PrincipalTag/x509Subject/CN} as the CERT_ATTRIBUTE, which corresponds to the CN field of your per-machine certificates. You can alternatively pass $(aws:PrincipalTag/x509SAN/Name/CN} as your CERT_ATTRIBUTE.
3. Replace CA_CERT_BODY with the certificate body of your CA without line breaks. The CA_CERT_BODY must be in Privacy Enhanced Mail (PEM) format. If you have a CA certificate in PEM format, remove the line breaks and BEGIN CERTIFICATE and END CERTIFICATE lines before placing the CA certificate body in your cfn-iamra-parameters.json file.
  { "Parameters": { "RoleName": "ROLE_NAME", "CertAttributeTrustPolicy": "CERT_ATTRIBUTE", "CABundleCert": "CA_CERT_BODY" } }

Deploy the CloudFormation template. Replace STACK_NAME with your name for the CloudFormation stack.

aws cloudformation deploy \
    --stack-name STACK_NAME \
    --template-file hybrid-ira-cfn.yaml \
    --parameter-overrides file://cfn-iamra-parameters.json
    --capabilities CAPABILITY_NAMED_IAM

AWS CLI

Install and configure the AWS CLI, if you haven’t already. See Installing or updating to the last version of the AWS CLI.

Create EKS Describe Cluster Policy

Create a file named eks-describe-cluster-policy.json with the following contents:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster"
            ],
            "Resource": "*"
        }
    ]
}

Create the policy with the following command:

aws iam create-policy \
    --policy-name EKSDescribeClusterPolicy \
    --policy-document file://eks-describe-cluster-policy.json

Steps for AWS SSM hybrid activations

Create a file named eks-hybrid-ssm-policy.json with the following contents. The policy grants permission for two actions ssm:DescribeInstanceInformation and ssm:DeregisterManagedInstance. The policy restricts the ssm:DeregisterManagedInstance permission to AWS SSM managed instances associated with your AWS SSM hybrid activation based on the resource tag you specify in your trust policy.
1. Replace AWS_REGION with the AWS Region for your AWS SSM hybrid activation.
2. Replace AWS_ACCOUNT_ID with your AWS account ID.
3. Replace TAG_KEY with the AWS SSM resource tag key you used when creating your AWS SSM hybrid activation. The combination of the tag key and tag value is used in the condition for the ssm:DeregisterManagedInstance to only allow the Hybrid Nodes IAM role to deregister the AWS SSM managed instances that are associated with your AWS SSM hybrid activation. In the CloudFormation template, TAG_KEY defaults to EKSClusterARN.
4. Replace TAG_VALUE with the AWS SSM resource tag value you used when creating your AWS SSM hybrid activation. The combination of the tag key and tag value is used in the condition for the ssm:DeregisterManagedInstance to only allow the Hybrid Nodes IAM role to deregister the AWS SSM managed instances that are associated with your AWS SSM hybrid activation. If you are using the default TAG_KEY of EKSClusterARN, then pass your EKS cluster ARN as the TAG_VALUE. EKS cluster ARNs have the format arn:aws:eks:AWS_REGION:AWS_ACCOUNT_ID:cluster/CLUSTER_NAME.
  { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "ssm:DescribeInstanceInformation", "Resource": "" }, { "Effect": "Allow", "Action": "ssm:DeregisterManagedInstance", "Resource": "arn:aws:ssm:AWS_REGION:AWS_ACCOUNT_ID:managed-instance/", "Condition": { "StringEquals": { "ssm:resourceTag/TAG_KEY": "TAG_VALUE" } } } ] }

Create the policy with the following command

aws iam create-policy \
    --policy-name EKSHybridSSMPolicy \
    --policy-document file://eks-hybrid-ssm-policy.json

Create a file named eks-hybrid-ssm-trust.json. Replace AWS_REGION with the AWS Region of your AWS SSM hybrid activation and AWS_ACCOUNT_ID with your AWS account ID.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"",
         "Effect":"Allow",
         "Principal":{
            "Service":"ssm.amazonaws.com"
         },
         "Action":"sts:AssumeRole",
         "Condition":{
            "StringEquals":{
               "aws:SourceAccount":"AWS_ACCOUNT_ID"
            },
            "ArnEquals":{
               "aws:SourceArn":"arn:aws:ssm:AWS_REGION:AWS_ACCOUNT_ID:*"
            }
         }
      }
   ]
}

Create the role with the following command.

aws iam create-role \
    --role-name AmazonEKSHybridNodesRole \
    --assume-role-policy-document file://eks-hybrid-ssm-trust.json

Attach the EKSDescribeClusterPolicy and the EKSHybridSSMPolicy you created in the previous steps. Replace AWS_ACCOUNT_ID with your AWS account ID.

aws iam attach-role-policy \
    --role-name AmazonEKSHybridNodesRole \
    --policy-arn arn:aws:iam::AWS_ACCOUNT_ID:policy/EKSDescribeClusterPolicy

aws iam attach-role-policy \
    --role-name AmazonEKSHybridNodesRole \
    --policy-arn arn:aws:iam::AWS_ACCOUNT_ID:policy/EKSHybridSSMPolicy

Attach the AmazonEC2ContainerRegistryPullOnly and AmazonSSMManagedInstanceCore AWS managed policies.

aws iam attach-role-policy \
    --role-name AmazonEKSHybridNodesRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly

aws iam attach-role-policy \
    --role-name AmazonEKSHybridNodesRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Steps for AWS IAM Roles Anywhere

To use AWS IAM Roles Anywhere, you must set up your AWS IAM Roles Anywhere trust anchor before creating the Hybrid Nodes IAM Role. See hybrid-nodes-creds.html for instructions.

Create a file named eks-hybrid-iamra-trust.json. Replace TRUST_ANCHOR ARN with the ARN of the trust anchor you created in the hybrid-nodes-creds.html steps. The condition in this trust policy restricts the ability of AWS IAM Roles Anywhere to assume the Hybrid Nodes IAM role to exchange temporary IAM credentials only when the role session name matches the CN in the x509 certificate installed on your hybrid nodes. You can alternatively use other certificate attributes to uniquely identify your node. The certificate attribute that you use in the trust policy must correspond to the nodeName you set in your nodeadm configuration. For more information, see the hybrid-nodes-nodeadm.title.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": [
                "sts:TagSession",
                "sts:SetSourceIdentity"
            ],
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": "TRUST_ANCHOR_ARN"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:RoleSessionName": "${aws:PrincipalTag/x509Subject/CN}"
                },
                "ArnEquals": {
                    "aws:SourceArn": "TRUST_ANCHOR_ARN"
                }
            }
        }
    ]
}

Create the role with the following command.

aws iam create-role \
    --role-name AmazonEKSHybridNodesRole \
    --assume-role-policy-document file://eks-hybrid-iamra-trust.json

Attach the EKSDescribeClusterPolicy you created in the previous steps. Replace AWS_ACCOUNT_ID with your AWS account ID.

aws iam attach-role-policy \
    --role-name AmazonEKSHybridNodesRole \
    --policy-arn arn:aws:iam::AWS_ACCOUNT_ID:policy/EKSDescribeClusterPolicy

Attach the AmazonEC2ContainerRegistryPullOnly AWS managed policy

aws iam attach-role-policy \
    --role-name AmazonEKSHybridNodesRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly

AWS Management Console

Create EKS Describe Cluster Policy

Open the Amazon IAM console
In the left navigation pane, choose Policies.
On the Policies page, choose Create policy.
On the Specify permissions page, in the Select a service panel, choose EKS.
1. Filter actions for DescribeCluster and select the DescribeCluster Read action.
2. Choose Next.
On the Review and create page
1. Enter a Policy name for your policy such as EKSDescribeClusterPolicy.
2. Choose Create policy.

Steps for AWS SSM hybrid activations

Open the Amazon IAM console
In the left navigation pane, choose Policies.
On the Policies page, choose Create policy.

On the Specify permissions page, in the Policy editor top right navigation, choose JSON. Paste the following snippet. Replace AWS_REGION with the AWS Region of your AWS SSM hybrid activation and replace AWS_ACCOUNT_ID with your AWS account ID. Replace TAG_KEY and TAG_VALUE with the AWS SSM resource tag key you used when creating your AWS SSM hybrid activation.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:DescribeInstanceInformation",
            "Resource": ""
        },
        {
            "Effect": "Allow",
            "Action": "ssm:DeregisterManagedInstance",
            "Resource": "arn:aws:ssm:AWS_REGION:AWS_ACCOUNT_ID:managed-instance/",
            "Condition": {
                "StringEquals": {
                    "ssm:resourceTag/TAG_KEY": "TAG_VALUE"
                }
            }
        }
    ]
}

Choose Next.

On the Review and Create page.
1. Enter a Policy name for your policy such as EKSHybridSSMPolicy
2. Choose Create Policy.
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.

On the Select trusted entity page, do the following:

In the Trusted entity type section, choose Custom trust policy. Paste the following into the Custom trust policy editor. Replace AWS_REGION with the AWS Region of your AWS SSM hybrid activation and AWS_ACCOUNT_ID with your AWS account ID.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"",
         "Effect":"Allow",
         "Principal":{
            "Service":"ssm.amazonaws.com"
         },
         "Action":"sts:AssumeRole",
         "Condition":{
            "StringEquals":{
               "aws:SourceAccount":"AWS_ACCOUNT_ID"
            },
            "ArnEquals":{
               "aws:SourceArn":"arn:aws:ssm:AWS_REGION:AWS_ACCOUNT_ID:*"
            }
         }
      }
   ]
}

Choose Next.

On the Add permissions page, attach a custom policy or do the following:
1. In the Filter policies box, enter EKSDescribeClusterPolicy, or the name of the policy you created above. Select the check box to the left of your policy name in the search results.
2. In the Filter policies box, enter EKSHybridSSMPolicy, or the name of the policy you created above. Select the check box to the left of your policy name in the search results.
3. In the Filter policies box, enter AmazonEC2ContainerRegistryPullOnly. Select the check box to the left of AmazonEC2ContainerRegistryPullOnly in the search results.
4. In the Filter policies box, enter AmazonSSMManagedInstanceCore. Select the check box to the left of AmazonSSMManagedInstanceCore in the search results.
5. Choose Next.
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKSHybridNodesRole.
2. For Description, replace the current text with descriptive text such as Amazon EKS - Hybrid Nodes role.
3. Choose Create role.

Steps for AWS IAM Roles Anywhere

To use AWS IAM Roles Anywhere, you must set up your AWS IAM Roles Anywhere trust anchor before creating the Hybrid Nodes IAM Role. See hybrid-nodes-creds.html for instructions.

Open the Amazon IAM console
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.

On the Select trusted entity page, do the following:

In the Trusted entity type section, choose Custom trust policy. Paste the following into the Custom trust policy editor. Replace TRUST_ANCHOR ARN with the ARN of the trust anchor you created in the hybrid-nodes-creds.html steps. The condition in this trust policy restricts the ability of AWS IAM Roles Anywhere to assume the Hybrid Nodes IAM role to exchange temporary IAM credentials only when the role session name matches the CN in the x509 certificate installed on your hybrid nodes. You can alternatively use other certificate attributes to uniquely identify your node. The certificate attribute that you use in the trust policy must correspond to the nodeName you set in your nodeadm configuration. For more information, see the hybrid-nodes-nodeadm.title.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": [
                "sts:TagSession",
                "sts:SetSourceIdentity"
            ],
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": "TRUST_ANCHOR_ARN"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:RoleSessionName": "${aws:PrincipalTag/x509Subject/CN}"
                },
                "ArnEquals": {
                    "aws:SourceArn": "TRUST_ANCHOR_ARN"
                }
            }
        }
    ]
}

Choose Next.

On the Add permissions page, attach a custom policy or do the following:
1. In the Filter policies box, enter EKSDescribeClusterPolicy, or the name of the policy you created above. Select the check box to the left of your policy name in the search results.
2. In the Filter policies box, enter AmazonEC2ContainerRegistryPullOnly. Select the check box to the left of AmazonEC2ContainerRegistryPullOnly in the search results.
3. Choose Next.
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKSHybridNodesRole.
2. For Description, replace the current text with descriptive text such as Amazon EKS - Hybrid Nodes role.
3. Choose Create role.

Create an Amazon EKS cluster with hybrid nodes

Create hybrid nodes cluster

This topic provides an overview of the available options and describes what to consider when you create a hybrid nodes-enabled Amazon EKS cluster. If you are not planning to use hybrid nodes, see create-cluster.title.

Prerequisites

The hybrid-nodes-prereqs.title completed. Before you create your hybrid nodes-enabled cluster, you must have your on-premises node and optionally pod CIDRs identified, your VPC and subnets created according to the EKS requirements, and hybrid nodes requirements, and your security group with inbound rules for your on-premises and optionally pod CIDRs. For more information on these prerequisites, see hybrid-nodes-networking.title.
The latest version of the AWS Command Line Interface (AWS CLI) installed and configured on your device. To check your current version, use aws --version. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing or updating to the last version of the AWS CLI and Configuring settings for the AWS CLI in the AWS Command Line Interface User Guide.
An IAM principal with permissions to create IAM roles and attach policies, and create and describe EKS clusters

Considerations

Your cluster must use either API or API_AND_CONFIG_MAP for the cluster authentication mode.
Your cluster must use IPv4 address family.
Your cluster must use either Public or Private cluster endpoint connectivity. Your cluster cannot use “Public and Private” cluster endpoint connectivity, because the Amazon EKS Kubernetes API server endpoint will resolve to the public IPs for hybrid nodes running outside of your VPC.
Currently, hybrid nodes must be enabled during cluster creation. You cannot change your RemoteNodeNetwork or RemotePodNetwork after cluster creation.

Step 1: Create cluster IAM role

If you already have a cluster IAM role, or you’re going to create your cluster with eksctl or AWS CloudFormation, then you can skip this step. By default, eksctl and the AWS CloudFormation template create the cluster IAM role for you.

Run the following command to create an IAM trust policy JSON file.

cat >eks-cluster-role-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Create the Amazon EKS cluster IAM role. If necessary, preface eks-cluster-role-trust-policy.json with the path on your computer that you wrote the file to in the previous step. The command associates the trust policy that you created in the previous step to the role. To create an IAM role, the IAM principal that is creating the role must be assigned the iam:CreateRole action (permission).
```
aws iam create-role \
    --role-name myAmazonEKSClusterRole \
    --assume-role-policy-document file://"eks-cluster-role-trust-policy.json"
```
You can assign either the Amazon EKS managed policy or create your own custom policy. For the minimum permissions that you must use in your custom policy, see create-node-role.title. Attach the Amazon EKS managed policy named AmazonEKSClusterPolicy to the role. To attach an IAM policy to an IAM principal, the principal that is attaching the policy must be assigned one of the following IAM actions (permissions): iam:AttachUserPolicy or iam:AttachRolePolicy.
```
aws iam attach-role-policy \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy \
    --role-name myAmazonEKSClusterRole
```

Step 2: Create hybrid nodes-enabled cluster

You can create a cluster by using:

eksctl
AWS CloudFormation
CLI
consolelong

Create hybrid nodes-enabled cluster - eksctl

You need to install the latest version of the eksctl command line tool. To install or update eksctl, see Installation in the eksctl documentation.

Create cluster-config.yaml to define a hybrid nodes-enabled Amazon EKS IPv4 cluster. Make the following replacements in your cluster-config.yaml. For a full list of settings, see the eksctl documentation.
1. Replace CLUSTER_NAME with a name for your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
2. Replace AWS_REGION with the AWS Region that you want to create your cluster in.
3. Replace K8S_VERSION with any Amazon EKS supported version.
4. Replace CREDS_PROVIDER with ssm or ira based on the credential provider you configured in the steps for hybrid-nodes-creds.title.
5. Replace CA_BUNDLE_CERT if your credential provider is set to ira, which uses AWS IAM Roles Anywhere as the credential provider. The CA_BUNDLE_CERT is the certificate authority (CA) certificate body and depends on your choice of CA. The certificate must be in Privacy Enhanced Mail (PEM) format.
6. Replace GATEWAY_ID with the ID of your virtual private gateway or transit gateway to be attached to your VPC.
7. Replace REMOTE_NODE_CIDRS with the on-premises node CIDR for your hybrid nodes.
8. Replace REMOTE_POD_CIDRS with the on-premises pod CIDR for workloads running on hybrid nodes or remove the line from your configuration if you are not running webhooks on hybrid nodes. You must configure your REMOTE_POD_CIDRS if your CNI does not use Network Address Translation (NAT) or masquerading for pod IP addresses when pod traffic leaves your on-premises hosts. You must configure REMOTE_POD_CIDRS if you are running webhooks on hybrid nodes.
9. Your on-premises node and pod CIDR blocks must meet the following requirements:
  1. Be within one of the IPv4 RFC-1918 ranges: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
  2. Not overlap with each other, the VPC CIDR for your cluster, or your Kubernetes service IPv4 CIDR
    
    apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: CLUSTER_NAME region: AWS_REGION version: "K8S_VERSION" remoteNetworkConfig: iam: provider: CREDS_PROVIDER # default SSM, can also be set to IRA # caBundleCert: CA_BUNDLE_CERT vpcGatewayID: GATEWAY_ID remoteNodeNetworks: - cidrs: ["REMOTE_NODE_CIDRS"] remotePodNetworks: - cidrs: ["REMOTE_POD_CIDRS"]
Run the following command:
```
eksctl create cluster -f cluster-config.yaml
```
Cluster provisioning takes several minutes. While the cluster is being created, several lines of output appear. The last line of output is similar to the following example line.
```
[✓]  EKS cluster "CLUSTER_NAME" in "REGION" region is ready
```
Continue with hybrid-nodes-cluster-create-kubeconfig.title.

Create hybrid nodes-enabled cluster - AWS CloudFormation

The CloudFormation stack creates the EKS cluster IAM role and an EKS cluster with the RemoteNodeNetwork and RemotePodNetwork you specify. Modify the CloudFormation template If you need to customize settings for your EKS cluster that are not exposed in the CloudFormation template.

Download the CloudFormation template.

curl -OL 'https://raw.githubusercontent.com/aws/eks-hybrid/refs/heads/main/example/hybrid-eks-cfn.yaml'

Create a cfn-eks-parameters.json and specify your configuration for each value.
1. CLUSTER_NAME: name of the EKS cluster to be created
2. CLUSTER_ROLE_NAME: name of the EKS cluster IAM role to be created. The default in the template is “EKSClusterRole”.
3. SUBNET1_ID: the ID of the first subnet you created in the prerequisite steps
4. SUBNET2_ID: the ID of the second subnet you created in the prerequisite steps
5. SG_ID: the security group ID you created in the prerequisite steps
6. REMOTE_NODE_CIDRS: the on-premises node CIDR for your hybrid nodes
7. REMOTE_POD_CIDRS: the on-premises pod CIDR for workloads running on hybrid nodes. You must configure your REMOTE_POD_CIDRS if your CNI does not use Network Address Translation (NAT) or masquerading for pod IP addresses when pod traffic leaves your on-premises hosts. You must configure REMOTE_POD_CIDRS if you are running webhooks on hybrid nodes.
8. Your on-premises node and pod CIDR blocks must meet the following requirements:
  1. Be within one of the IPv4 RFC-1918 ranges: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
  2. Not overlap with each other, the VPC CIDR for your cluster, or your Kubernetes service IPv4 CIDR.
9. CLUSTER_AUTH: the cluster authentication mode for your cluster. Valid values are API and API_AND_CONFIG_MAP. The default in the template is API_AND_CONFIG_MAP.
10. CLUSTER_ENDPOINT: the cluster endpoint connectivity for your cluster. Valid values are “Public” and “Private”. The default in the template is Private, which means you will only be able to connect to the Kubernetes API endpoint from within your VPC.
11. K8S_VERSION: the Kubernetes version to use for your cluster. See Amazon EKS supported versions.
  { "Parameters": { "ClusterName": "CLUSTER_NAME", "ClusterRoleName": "CLUSTER_ROLE_NAME", "SubnetId1": "SUBNET1_ID", "SubnetId2": "SUBNET2_ID", "SecurityGroupId" "SG_ID", "RemoteNodeCIDR": "REMOTE_NODE_CIDRS", "RemotePodCIDR": "REMOTE_POD_CIDRS", "ClusterAuthMode": "CLUSTER_AUTH", "ClusterEndpointConnectivity": "CLUSTER_ENDPOINT", "K8sVersion": "K8S_VERSION" } }
Deploy the CloudFormation stack. Replace STACK_NAME with your name for the CloudFormation stack and AWS_REGION with your desired AWS Region where the cluster will be created.
```
aws cloudformation deploy \
    --stack-name STACK_NAME \
    --region AWS_REGION \
    --template-file hybrid-eks-cfn.yaml \
    --parameter-overrides file://cfn-eks-parameters.json \
    --capabilities CAPABILITY_NAMED_IAM
```
Cluster provisioning takes several minutes. You can check the status of your stack with the following command. Replace STACK_NAME with your name for the CloudFormation stack and AWS_REGION with your desired AWS Region where the cluster will be created.
```
aws cloudformation describe-stacks \
    --stack-name STACK_NAME \
    --region AWS_REGION \
    --query 'Stacks[].StackStatus'
```
Continue with hybrid-nodes-cluster-create-kubeconfig.title.

Create hybrid nodes-enabled cluster - AWS CLI

Run the following command to create a hybrid nodes-enabled EKS cluster. Before running the command, replace the following with your desired settings. For a full list of settings, see the create-cluster.title documentation.
1. CLUSTER_NAME: name of the EKS cluster to be created
2. AWS_REGION: AWS Region where the cluster will be created.
3. K8S_VERSION: the Kubernetes version to use for your cluster. See Amazon EKS supported versions.
4. ROLE_ARN: the Amazon EKS cluster role you configured for your cluster. See Amazon EKS cluster IAM role for more information.
5. SUBNET1_ID: the ID of the first subnet you created in the prerequisite steps
6. SUBNET2_ID: the ID of the second subnet you created in the prerequisite steps
7. SG_ID: the security group ID you created in the prerequisite steps
8. You can use API and API_AND_CONFIG_MAP for your cluster access authentication mode. In the command below, the cluster access authentication mode is set to API_AND_CONFIG_MAP.
9. You can use the endpointPublicAccess and endpointPrivateAccess parameters to enable or disable public and private access to your cluster’s Kubernetes API server endpoint. In the command below endpointPublicAccess is set to false and endpointPrivateAccess is set to true.
10. REMOTE_NODE_CIDRS: the on-premises node CIDR for your hybrid nodes.
11. REMOTE_POD_CIDRS (optional): the on-premises pod CIDR for workloads running on hybrid nodes.
12. Your on-premises node and pod CIDR blocks must meet the following requirements:
  1. Be within one of the IPv4 RFC-1918 ranges: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
  2. Not overlap with each other, the VPC CIDR for your Amazon EKS cluster, or your Kubernetes service IPv4 CIDR.
    
    aws eks create-cluster \ --name CLUSTER_NAME \ --region AWS_REGION \ --kubernetes-version K8S_VERSION \ --role-arn ROLE_ARN \ --resources-vpc-config subnetIds=SUBNET1_ID,SUBNET2_ID,securityGroupIds=SG_ID,endpointPrivateAccess=true,endpointPublicAccess=false \ --access-config authenticationMode=API_AND_CONFIG_MAP \ --remote-network-config '{"remoteNodeNetworks":[{"cidrs":["REMOTE_NODE_CIDRS"]}],"remotePodNetworks":[{"cidrs":["REMOTE_POD_CIDRS"]}]}'
It takes several minutes to provision the cluster. You can query the status of your cluster with the following command. Replace CLUSTER_NAME with the name of the cluster you are creating and AWS_REGION with the AWS Region where the cluster is creating. Don’t proceed to the next step until the output returned is ACTIVE.
```
aws eks describe-cluster \
    --name CLUSTER_NAME \
    --region AWS_REGION \
    --query "cluster.status"
```
Continue with hybrid-nodes-cluster-create-kubeconfig.title.

Create hybrid nodes-enabled cluster - AWS Management Console

Open the Amazon EKS console at Amazon EKS console.
Choose Add cluster and then choose Create.
On the Configure cluster page, enter the following fields:
1. Name – A name for your cluster. The name can contain only alphanumeric characters (case-sensitive), hyphens, and underscores. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
2. Cluster IAM role – Choose the Amazon EKS cluster IAM role that you created to allow the Kubernetes control plane to manage AWS resources on your behalf.
3. Kubernetes version – The version of Kubernetes to use for your cluster. We recommend selecting the latest version, unless you need an earlier version.
4. Upgrade policy - Choose either Extended or Standard.
  1. Extended: This option supports the Kubernetes version for 26 months after the release date. The extended support period has an additional hourly cost that begins after the standard support period ends. When extended support ends, your cluster will be auto upgraded to the next version.
  2. Standard: This option supports the Kubernetes version for 14 months after the release date. There is no additional cost. When standard support ends, your cluster will be auto upgraded to the next version.
5. Cluster access - choose to allow or disallow cluster administrator access and select an authentication mode. The following authentication modes are supported for hybrid nodes-enabled clusters.
  1. EKS API: The cluster will source authenticated IAM principals only from EKS access entry APIs.
  2. EKS API and ConfigMap: The cluster will source authenticated IAM principals from both EKS access entry APIs and the aws-auth ConfigMap.
6. Secrets encryption – (Optional) Choose to enable secrets encryption of Kubernetes secrets using a KMS key. You can also enable this after you create your cluster. Before you enable this capability, make sure that you’re familiar with the information in enable-kms.title.
7. ARC Zonal Shift - If enabled, EKS will register your cluster with ARC zonal shift to enable you to use zonal shift to shift application traffic away from an AZ.
8. Tags – (Optional) Add any tags to your cluster. For more information, see eks-using-tags.title.
9. When you’re done with this page, choose Next.
On the Specify networking page, select values for the following fields:
1. VPC – Choose an existing VPC that meets network-reqs.title and Amazon EKS Hybrid Nodes requirements. Before choosing a VPC, we recommend that you’re familiar with all of the requirements and considerations in View Amazon EKS networking requirements for VPC, subnets, and hybrid nodes. You can’t change which VPC you want to use after cluster creation. If no VPCs are listed, then you need to create one first. For more information, see creating-a-vpc.title and the Amazon EKS Hybrid Nodes networking requirements.
2. Subnets – By default, all available subnets in the VPC specified in the previous field are preselected. You must select at least two.
3. Security groups – (Optional) Specify one or more security groups that you want Amazon EKS to associate to the network interfaces that it creates. At least one of the security groups you specify must have inbound rules for your on-premises node and optionally pod CIDRs. See the Amazon EKS Hybrid Nodes networking requirements for more information. Whether you choose any security groups or not, Amazon EKS creates a security group that enables communication between your cluster and your VPC. Amazon EKS associates this security group, and any that you choose, to the network interfaces that it creates. For more information about the cluster security group that Amazon EKS creates, see sec-group-reqs.title You can modify the rules in the cluster security group that Amazon EKS creates.
4. Choose cluster IP address family – You must choose IPv4 for hybrid nodes-enabled clusters.
5. (Optional) Choose Configure Kubernetes Service IP address range and specify a Service IPv4 range.
6. Choose Configure remote networks to enable hybrid nodes and specify your on-premises node and pod CIDRs for hybrid nodes.
7. You must configure your remote pod CIDR if your CNI does not use Network Address Translation (NAT) or masquerading for pod IP addresses when pod traffic leaves your on-premises hosts. You must configure the remote pod CIDR if you are running webhooks on hybrid nodes.
8. Your on-premises node and pod CIDR blocks must meet the following requirements:
  1. Be within one of the IPv4 RFC-1918 ranges: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
  2. Not overlap with each other, the VPC CIDR for your cluster, or your Kubernetes service IPv4 CIDR
9. For Cluster endpoint access, select an option. After your cluster is created, you can change this option. For hybrid nodes-enabled clusters, you must choose either Public or Private. Before selecting a non-default option, make sure to familiarize yourself with the options and their implications. For more information, see cluster-endpoint.title.
10. When you’re done with this page, choose Next.
(Optional) On the Configure observability page, choose which Metrics and Control plane logging options to turn on. By default, each log type is turned off.
1. For more information about the Prometheus metrics option, see prometheus.title.
2. For more information about the EKS control logging options, see control-plane-logs.title.
3. When you’re done with this page, choose Next.
On the Select add-ons page, choose the add-ons that you want to add to your cluster.
1. You can choose as many Amazon EKS add-ons and AWS Marketplace add-ons as you require. Amazon EKS add-ons that are not compatible with hybrid nodes are marked with “Not compatible with Hybrid Nodes” and the add-ons have an anti-affinity rule to prevent them from running on hybrid nodes. See Configuring add-ons for hybrid nodes for more information. If the AWS Marketplace add-ons that you want to install isn’t listed, you can search for available AWS Marketplace add-ons by entering text in the search box. You can also search by category, vendor, or pricing model and then choose the add-ons from the search results.
2. Some add-ons, such as CoreDNS and kube-proxy, are installed by default. If you disable any of the default add-ons, this may affect your ability to run Kubernetes applications.
3. When you’re done with this page, choose Next.
On the Configure selected add-ons settings page, select the version that you want to install.
1. You can always update to a later version after cluster creation. You can update the configuration of each add-on after cluster creation. For more information about configuring add-ons, see updating-an-add-on.title. For the add-ons versions that are compatible with hybrid nodes, see hybrid-nodes-add-ons.title.
2. When you’re done with this page, choose Next.
On the Review and create page, review the information that you entered or selected on the previous pages. If you need to make changes, choose Edit. When you’re satisfied, choose Create. The Status field shows CREATING while the cluster is provisioned. Cluster provisioning takes several minutes.
Continue with hybrid-nodes-cluster-create-kubeconfig.title.

Step 3: Update kubeconfig

If you created your cluster using eksctl, then you can skip this step. This is because eksctl already completed this step for you. Enable kubectl to communicate with your cluster by adding a new context to the kubectl config file. For more information about how to create and update the file, see create-kubeconfig.title.

aws eks update-kubeconfig --name CLUSTER_NAME --region AWS_REGION

An example output is as follows.

Added new context arn:aws:eks:AWS_REGION:111122223333:cluster/CLUSTER_NAME to /home/username/.kube/config

Confirm communication with your cluster by running the following command.

kubectl get svc

An example output is as follows.

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   28h

Step 4: Cluster setup

As a next step, see hybrid-nodes-cluster-prep.title to enable access for your hybrid nodes to join your cluster.

Prepare cluster access for hybrid nodes

Prepare cluster access for Amazon EKS hybrid nodes

Before connecting hybrid nodes to your Amazon EKS cluster, you must enable your Hybrid Nodes IAM Role with Kubernetes permissions to join the cluster. See hybrid-nodes-creds.title for information on how to create the Hybrid Nodes IAM role. Amazon EKS supports two ways to associate IAM principals with Kubernetes Role-Based Access Control (RBAC), Amazon EKS access entries and the aws-auth ConfigMap. For more information on Amazon EKS access management, see grant-k8s-access.title.

Use the procedures below to associate your Hybrid Nodes IAM role with Kubernetes permissions. To use Amazon EKS access entries, your cluster must have been created with the API or API_AND_CONFIG_MAP authentication modes. To use the aws-auth ConfigMap, your cluster must have been created with the API_AND_CONFIG_MAP authentication mode. The CONFIG_MAP-only authentication mode is not supported for hybrid nodes-enabled Amazon EKS clusters.

Using Amazon EKS access entries for Hybrid Nodes IAM role

There is an Amazon EKS access entry type for hybrid nodes named HYBRID_LINUX that can be used with an IAM role. With this access entry type, the username is automatically set to system:node:{{SessionName}}. For more information on creating access entries, see creating-access-entries.title.

AWS CLI

You must have the latest version of the AWS CLI installed and configured on your device. To check your current version, use aws --version. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide.
Create your access entry with the following command. Replace CLUSTER_NAME with the name of your cluster and HYBRID_NODES_ROLE_ARN with the ARN of the role you created in the steps for hybrid-nodes-creds.title.
```
aws eks create-access-entry --cluster-name CLUSTER_NAME \
    --principal-arn HYBRID_NODES_ROLE_ARN \
    --type HYBRID_LINUX
```

AWS Management Console

Open the Amazon EKS console at Amazon EKS console.
Choose the name of your hybrid nodes-enabled cluster.
Choose the Access tab.
Choose Create access entry.
For IAM principal, select the Hybrid Nodes IAM role you created in the steps for hybrid-nodes-creds.title.
For Type, select Hybrid Linux.
(Optional) For Tags, assign labels to the access entry. For example, to make it easier to find all resources with the same tag.
Choose Skip to review and create. You cannot add policies to the Hybrid Linux access entry or change its access scope.
Review the configuration for your access entry. If anything looks incorrect, choose Previous to go back through the steps and correct the error. If the configuration is correct, choose Create.

Using aws-auth ConfigMap for Hybrid Nodes IAM role

In the following steps, you will create or update the aws-auth ConfigMap with the ARN of the Hybrid Nodes IAM Role you created in the steps for hybrid-nodes-creds.title.

Check to see if you have an existing aws-auth ConfigMap for your cluster. Note that if you are using a specific kubeconfig file, use the --kubeconfig flag.
```
kubectl describe configmap -n kube-system aws-auth
```
If you are shown an aws-auth ConfigMap, then update it as needed.
1. Open the ConfigMap for editing.
  kubectl edit -n kube-system configmap/aws-auth
2. Add a new mapRoles entry as needed. Replace HYBRID_NODES_ROLE_ARN with the ARN of your Hybrid Nodes IAM role. Note, {{SessionName}} is the correct template format to save in the ConfigMap. Do not replace it with other values.
  data: mapRoles: | - groups: - system:bootstrappers - system:nodes rolearn: HYBRID_NODES_ROLE_ARN username: system:node:{{SessionName}}
3. Save the file and exit your text editor.
If there is not an existing aws-auth ConfigMap for your cluster, create it with the following command. Replace HYBRID_NODES_ROLE_ARN with the ARN of your Hybrid Nodes IAM role. Note that {{SessionName}} is the correct template format to save in the ConfigMap. Do not replace it with other values.
```
kubectl apply -f=/dev/stdin <<-EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
  - groups:
    - system:bootstrappers
    - system:nodes
    rolearn: HYBRID_NODES_ROLE_ARN
    username: system:node:{{SessionName}}
EOF
```

9.8.3. Run on-premises workloads on hybrid nodes

Join nodes from your data centers to Amazon EKS Kubernetes clusters with Amazon EKS Hybrid Nodes.

In an EKS cluster with hybrid nodes enabled, you can run on-premises and edge applications on your own infrastructure with the same Amazon EKS clusters, features, and tools that you use in AWS Cloud.

The following sections contain step-by-step instructions for using hybrid nodes.

[[Topic List]]

Connect hybrid nodes

Connect hybrid nodes to Amazon EKS cluster.

This topic describes how to connect hybrid nodes to an Amazon EKS cluster. After your hybrid nodes join the cluster, they will appear with status Not Ready in the Amazon EKS console and in Kubernetes-compatible tooling such as kubectl. After completing the steps on this page, proceed to hybrid-nodes-cni.title to make your hybrid nodes ready to run applications.

Prerequisites

Before connecting hybrid nodes to your Amazon EKS cluster, make sure you have completed the prerequisite steps.

You have network connectivity from your on-premises environment to the AWS Region hosting your Amazon EKS cluster. See hybrid-nodes-networking.title for more information.
You have a compatible operating system for hybrid nodes installed on your on-premises hosts. See hybrid-nodes-os.title for more information.
You have created your Hybrid Nodes IAM role and set up your on-premises credential provider (AWS Systems Manager hybrid activations or AWS IAM Roles Anywhere). See hybrid-nodes-creds.title for more information.
You have created your hybrid nodes-enabled Amazon EKS cluster. See hybrid-nodes-cluster-create.title for more information.
You have associated your Hybrid Nodes IAM role with Kubernetes Role-Based Access Control (RBAC) permissions. See hybrid-nodes-cluster-prep.title for more information.

Step 1: Install the hybrid nodes CLI (`nodeadm`) on each on-premises host

If you are including the Amazon EKS Hybrid Nodes CLI (nodeadm) in your pre-built operating system images, you can skip this step. For more information on the hybrid nodes version of nodeadm, see hybrid-nodes-nodeadm.title.

The hybrid nodes version of nodeadm is hosted in Amazon S3 fronted by Amazon CloudFront. To install nodeadm on each on-premises host, you can run the following command from your on-premises hosts.

For x86_64 hosts:

curl -OL 'https://hybrid-assets.eks.amazonaws.com/releases/latest/bin/linux/amd64/nodeadm'

For ARM hosts

curl -OL 'https://hybrid-assets.eks.amazonaws.com/releases/latest/bin/linux/arm64/nodeadm'

Add executable file permission to the downloaded binary on each host.

chmod +x nodeadm

Step 2: Install the hybrid nodes dependencies with `nodeadm`

If you are installing the hybrid nodes dependencies in pre-built operating system images, you can skip this step. The nodeadm install command can be used to install all dependencies required for hybrid nodes. The hybrid nodes dependencies include containerd, kubelet, kubectl, and AWS SSM or AWS IAM Roles Anywhere components. See hybrid-nodes-nodeadm.title for more information on the components and file locations installed by nodeadm install. See hybrid-nodes-networking.title for hybrid nodes for more information on the domains that must be allowed in your on-premises firewall for the nodeadm install process.

Run the command below to install the hybrid nodes dependencies on your on-premises host. The command below must be run with a user that has sudo/root access on your host.

The hybrid nodes CLI (nodeadm) must be run with a user that has sudo/root access on your host.

Replace K8S_VERSION with the Kubernetes minor version of your Amazon EKS cluster, for example 1.31. See Amazon EKS Kubernetes versions for a list of the supported Kubernetes versions.
Replace CREDS_PROVIDER with the on-premises credential provider you are using. Valid values are ssm for AWS SSM and iam-ra for AWS IAM Roles Anywhere.

nodeadm install K8S_VERSION --credential-provider CREDS_PROVIDER

Step 3: Connect hybrid nodes to your cluster

Before connecting your hybrid nodes to your cluster, make sure you have allowed the required access in your on-premises firewall and in the security group for your cluster for the Amazon EKS control plane to/from hybrid node communication. Most issues at this step are related to the firewall configuration, security group configuration, or Hybrid Nodes IAM role configuration.

The hybrid nodes CLI (nodeadm) must be run with a user that has sudo/root access on your host.

Create a nodeConfig.yaml file on each host with the values for your deployment. For a full description of the available configuration settings, see hybrid-nodes-nodeadm.title. If your Hybrid Nodes IAM role does not have permission for the eks:DescribeCluster action, you must pass your Kubernetes API endpoint, cluster CA bundle, and Kubernetes service IPv4 CIDR in the cluster section of your nodeConfig.yaml.
1. Use the nodeConfig.yaml example below if you are using AWS SSM hybrid activations for your on-premises credentials provider.
  1. Replace CLUSTER_NAME with the name of your cluster.
  2. Replace AWS_REGION with the AWS Region hosting your cluster. For example, us-west-2.
  3. Replace ACTIVATION_CODE with the activation code you received when creating your AWS SSM hybrid activation. See hybrid-nodes-creds.title for more information.
  4. Replace ACTIVATION_ID with the activation ID you received when creating your AWS SSM hybrid activation. You can retrieve this information from the AWS Systems Manager console or from the AWS CLI aws ssm describe-activations command.
    
    apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: cluster: name: CLUSTER_NAME region: AWS_REGION hybrid: ssm: activationCode: ACTIVATION_CODE activationId: ACTIVATION_ID
2. Use the nodeConfig.yaml example below if you are using AWS IAM Roles Anywhere for your on-premises credentials provider.
  1. Replace CLUSTER_NAME with the name of your cluster.
  2. Replace AWS_REGION with the AWS Region hosting your cluster. For example, us-west-2.
  3. Replace NODE_NAME with the name of your node. The node name must match the CN of the certificate on the host if you configured the trust policy of your Hybrid Nodes IAM role with the "sts:RoleSessionName": "${aws:PrincipalTag/x509Subject/CN}" resource condition. The nodeName you use must not be longer than 64 characters.
  4. Replace TRUST_ANCHOR_ARN with the ARN of the trust anchor you configured in the steps for Prepare credentials for hybrid nodes.
  5. Replace PROFILE_ARN with the ARN of the trust anchor you configured in the steps for hybrid-nodes-creds.title.
  6. Replace ROLE_ARN with the ARN of your Hybrid Nodes IAM role.
  7. Replace CERTIFICATE_PATH with the path in disk to your node certificate. If you don’t specify it, the default is /etc/iam/pki/server.pem.
  8. Replace KEY_PATH with the path in disk to your certificate private key. If you don’t specify it, the default is /etc/iam/pki/server.key.
    
    apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: cluster: name: CLUSTER_NAME region: AWS_REGION hybrid: iamRolesAnywhere: nodeName: NODE_NAME trustAnchorArn: TRUST_ANCHOR_ARN profileArn: PROFILE_ARN roleArn: ROLE_ARN certificatePath: CERTIFICATE_PATH privateKeyPath: KEY_PATH
Run the nodeadm init command with your nodeConfig.yaml to connect your hybrid nodes to your Amazon EKS cluster.
```
nodeadm init -c file://nodeConfig.yaml
```

If the above command completes successfully, your hybrid node has joined your Amazon EKS cluster. You can verify this in the Amazon EKS console by navigating to the Compute tab for your cluster (ensure IAM principal has permissions to view) or with kubectl get nodes.

Your nodes will have status Not Ready, which is expected and is due to the lack of a CNI running on your hybrid nodes. If your nodes did not join the cluster, see hybrid-nodes-troubleshooting.title.

Step 4: Configure a CNI for hybrid nodes

To make your hybrid nodes ready to run applications, continue with the steps on hybrid-nodes-cni.title.

Upgrade hybrid nodes for your cluster

Upgrade Kubernetes versions on hybrid nodes

The guidance for upgrading hybrid nodes is similar to self-managed Amazon EKS nodes that run in Amazon EC2. It is recommended to can create new hybrid nodes on your target Kubernetes version, gracefully migrate your existing applications to the hybrid nodes on the new Kubernetes version, and remove the hybrid nodes on the old Kubernetes version from your cluster. Be sure to review the Amazon EKS Best Practices for upgrades before initiating an upgrade. Amazon EKS Hybrid Nodes have the same Kubernetes version support for Amazon EKS clusters with cloud nodes, including standard and extended support.

Amazon EKS Hybrid Nodes follow the same version skew policy for nodes as upstream Kubernetes. Amazon EKS Hybrid Nodes cannot be on a newer version than the Amazon EKS control plane, and hybrid nodes may be up to three Kubernetes minor versions older than the Amazon EKS control plane minor version.

If you do not have spare capacity to create new hybrid nodes on your target Kubernetes version for a cutover migration upgrade strategy, you can alternatively use the Amazon EKS Hybrid Nodes CLI (nodeadm) to upgrade the Kubernetes version of your hybrid nodes in-place.

If you are upgrading your hybrid nodes in-place with nodeadm, there is downtime for the node during the process where the older version of the Kubernetes components are shut down and the new Kubernetes version components are installed and started.

Prerequisites

Before upgrading, make sure you have completed the following prerequisites.

The target Kubernetes version for your hybrid nodes upgrade must be equal to or less than the Amazon EKS control plane version.
If you are following a cutover migration upgrade strategy, the new hybrid nodes you are installing on your target Kubernetes version must meet the hybrid-nodes-prereqs.title requirements. This includes having IP addresses within the Remote Node Network CIDR you passed during Amazon EKS cluster creation.
For both cutover migration and in-place upgrades, the hybrid nodes must have access to the required domains to pull the new versions of the hybrid nodes dependencies.
You must have kubectl installed on your local machine or instance you are using to interact with your Amazon EKS Kubernetes API endpoint.
The version of your CNI must support the Kubernetes version you are upgrading to. If it does not, upgrade your CNI version before upgrading your hybrid nodes. See hybrid-nodes-cni.title for more information.

Cutover migration upgrades

Cutover migration upgrades refer to the process of creating new hybrid nodes on new hosts with your target Kubernetes version, gracefully migrating your existing applications to the new hybrid nodes on your target Kubernetes version, and removing the hybrid nodes on the old Kubernetes version from your cluster.

Connect your new hosts as hybrid nodes following the hybrid-nodes-join.title steps. When running the nodeadm install command, use your target Kubernetes version.
Enable communication between the new hybrid nodes on the target Kubernetes version and your hybrid nodes on the old Kubernetes version. This configuration allows pods to communicate with each other while you are migrating your workload to the hybrid nodes on the target Kubernetes version.
Confirm your hybrid nodes on your target Kubernetes version successfully joined your cluster and have status Ready.
Use the following command to taint each of the nodes that you want to remove with NoSchedule. This is so that new pods aren’t scheduled or rescheduled on the nodes that you are replacing. For more information, see Taints and Tolerations in the Kubernetes documentation. Replace NODE_NAME with the name of the hybrid nodes on the old Kubernetes version.
```
kubectl taint nodes NODE_NAME key=value:NoSchedule
```
You can identify and taint all of the nodes of a particular Kubernetes version (in this case, 1.28) with the following code snippet.
```
K8S_VERSION=1.28
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
    echo "Tainting $node"
    kubectl taint nodes $node key=value:NoSchedule
done
```
If your current deployment is running fewer than two CoreDNS replicas on your hybrid nodes, scale out the deployment to at least two replicas. It is recommended to run at least two CoreDNS replicas on hybrid nodes for resiliency during normal operations.
```
kubectl scale deployments/coredns --replicas=2 -n kube-system
```
Drain each of the hybrid nodes on the old Kubernetes version that you want to remove from your cluster with the following command. For more information on draining nodes, see Safely Drain a Node in the Kubernetes documentation. Replace NODE_NAME with the name of the hybrid nodes on the old Kubernetes version.
```
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data
```
You can identify and drain all of the nodes of a particular Kubernetes version (in this case, 1.28) with the following code snippet.
```
K8S_VERSION=1.28
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
    echo "Draining $node"
    kubectl drain $node --ignore-daemonsets --delete-emptydir-data
done
```
You can use nodeadm to stop and remove the hybrid nodes artifacts from the host. You must run nodeadm with a user that has root/sudo privileges. By default, nodeadm uninstall will not proceed if there are pods remaining on the node. For more information see hybrid-nodes-nodeadm.title.
```
nodeadm uninstall
```

With the hybrid nodes artifacts stopped and uninstalled, remove the node resource from your cluster.

kubectl delete node node-name

You can identify and delete all of the nodes of a particular Kubernetes version (in this case, 1.28) with the following code snippet.

K8S_VERSION=1.28
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
    echo "Deleting $node"
    kubectl delete node $node
done

Depending on your choice of CNI, there may be artifacts remaining on your hybrid nodes after running the above steps. See hybrid-nodes-cni.title for more information.

In-place upgrades

The in-place upgrade process refers to using nodeadm upgrade to upgrade the Kubernetes version for hybrid nodes without using new physical or virtual hosts and a cutover migration strategy. The nodeadm upgrade process shuts down the existing older Kubernetes components running on the hybrid node, uninstalls the existing older Kubernetes components, installs the new target Kubernetes components, and starts the new target Kubernetes components. It is strongly recommend to upgrade one node at a time to minimize impact to applications running on the hybrid nodes. The duration of this process depends on your network bandwidth and latency.

Use the following command to taint the node you are upgrading with NoSchedule. This is so that new pods aren’t scheduled or rescheduled on the node that you are upgrading. For more information, see Taints and Tolerations in the Kubernetes documentation. Replace NODE_NAME with the name of the hybrid node you are upgrading
```
kubectl taint nodes NODE_NAME key=value:NoSchedule
```
Drain the node you are upgrading with the following command. For more information on draining nodes, see Safely Drain a Node in the Kubernetes documentation. Replace NODE_NAME with the name of the hybrid node you are upgrading.
```
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data
```
Run nodeadm upgrade on the hybrid node you are upgrading. You must run nodeadm with a user that has root/sudo privileges. The name of the node is preserved through upgrade for both AWS SSM and AWS IAM Roles Anywhere credential providers. You cannot change credentials providers during the upgrade process. See hybrid-nodes-nodeadm.title for configuration values for nodeConfig.yaml. Replace K8S_VERSION with the target Kubernetes version you upgrading to.
```
nodeadm upgrade K8S_VERSION -c file://nodeConfig.yaml
```
Watch the status of your hybrid nodes and wait for your nodes to shutdown and restart on the new Kubernetes version with the Ready status.
```
kubectl get nodes -o -w
```

Remove hybrid nodes

Delete hybrid nodes from your EKS cluster

This topic describes how to delete hybrid nodes from your Amazon EKS cluster. You must delete your hybrid nodes with your choice of Kubernetes-compatible tooling such as kubectl. Charges for hybrid nodes stop when the node object is removed from the Amazon EKS cluster. For more information on hybrid nodes pricing, see Amazon EKS Pricing.

Removing nodes is disruptive to workloads running on the node. Before deleting hybrid nodes, it is recommended to first drain the node to move pods to another active node. For more information on draining nodes, see Safely Drain a Node in the Kubernetes documentation.

Run the kubectl steps below from your local machine or instance that you use to interact with the Amazon EKS cluster’s Kubernetes API endpoint. If you are using a specific kubeconfig file, use the --kubeconfig flag.

Step 1: List your nodes

kubectl get nodes

Step 2: Drain your node

See kubectl drain in the Kubernetes documentation for more information on the kubectl drain command.

kubectl drain --ignore-daemonsets <node-name>

Step 3: Stop and uninstall hybrid nodes artifacts

You can use the Amazon EKS Hybrid Nodes CLI (nodeadm) to stop and remove the hybrid nodes artifacts from the host. You must run nodeadm with a user that has root/sudo privileges. By default, nodeadm uninstall will not proceed if there are pods remaining on the node. If you are using AWS Systems Manager (SSM) as your credentials provider, the nodeadm uninstall command deregisters the host as an AWS SSM managed instance. For more information, see hybrid-nodes-nodeadm.title.

nodeadm uninstall

Step 4: Delete your node from the cluster

With the hybrid nodes artifacts stopped and uninstalled, remove the node resource from your cluster.

kubectl delete node <node-name>

Step 5: Check for remaining artifacts

Depending on your choice of CNI, there may be artifacts remaining on your hybrid nodes after running the above steps. See hybrid-nodes-cni.title for more information.

9.8.4. Configure a CNI for hybrid nodes

Configure a CNI for Amazon EKS hybrid nodes

Cilium and Calico are supported as the Container Networking Interfaces (CNIs) for Amazon EKS Hybrid Nodes. You must install a CNI for hybrid nodes to become ready to serve workloads. Hybrid nodes appear with status Not Ready until a CNI is running. You can manage these CNIs with your choice of tooling such as Helm. The Amazon VPC CNI is not compatible with hybrid nodes and the VPC CNI is configured with anti-affinity for the eks.amazonaws.com/compute-type: hybrid label.

Version compatibility

The table below represents the Cilium and Calico versions that are compatible and validated for each Kubernetes version supported in Amazon EKS.

Kubernetes version

Cilium version

Calico version

1.31

1.16.x

3.29.x

1.30

1.16.x

3.29.x

1.29

1.16.x

3.29.x

1.28

1.16.x

3.29.x

1.27

1.16.x

3.29.x

1.26

1.16.x

3.29.x

1.25

1.16.x

3.29.x

Supported capabilities

AWS supports the following capabilities of Cilium and Calico for use with hybrid nodes. If you plan to use functionality outside the scope of AWS support, we recommend that you obtain commercial support for the plugin or have the in-house expertise to troubleshoot and contribute fixes to the CNI plugin project.

Feature

Cilium

Calico

Kubernetes network conformance

Yes

Control plane to node connectivity

Yes

Control plane to pod connectivity

Yes

Lifecycle Management

Install, Upgrade, Delete

Networking Mode

VXLAN

IP Address Management (IPAM)

Cluster Scope (Cilium IPAM)

Calico IPAM

IP family

IPv4

BGP

Yes (Cilium Control Plane)

Yes

Install Cilium on hybrid nodes

Ensure that you have installed the helm CLI on your command-line environment. See the Helm documentation for installation instructions.

Install the Cilium Helm repo.

helm repo add cilium https://helm.cilium.io/

Create a yaml file called cilium-values.yaml. If you configured at least one remote pod network, configure the same pod CIDRs for your clusterPoolIPv4PodCIDRList. You shouldn’t change your clusterPoolIPv4PodCIDRList after deploying Cilium on your cluster. You can configure clusterPoolIPv4MaskSize based on your required pods per node, see Expanding the cluster pool in the Cilium documentation. For a full list of Helm values for Cilium, see the the Helm reference in the Cilium documentation. The following example configures all of the Cilium components to run on only the hybrid nodes, since they have the the eks.amazonaws.com/compute-type: hybrid label.

By default, Cilium masquerades the source IP address of all pod traffic leaving the cluster to the IP address of the node. This makes it possible for Cilium to run with Amazon EKS clusters that have remote pod networks configured and with clusters that don’t have remote pod networks configured. If you disable masquerading for your Cilium deployment, then you must configure your Amazon EKS cluster with your remote pod networks and you must advertise your pod addresses with your on-premises network. If you are running webhooks on your hybrid nodes, you must configure your cluster with your remote pod networks and you must advertise your pod addresses with your on-premises network.

A common way to advertise pod addresses with your on-premises network is by using BGP. To use BGP with Cilium, you must set bgpControlPlane.enabled: true. For more information on Cilium’s BGP support, see Cilium BGP Control Plane in the Cilium documentation.
```
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: In
          values:
          - hybrid
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4MaskSize: 25
    clusterPoolIPv4PodCIDRList:
    - POD_CIDR
operator:
  unmanagedPodWatcher:
    restart: false
```
Install Cilium on your cluster. Replace CILIUM_VERSION with your desired Cilium version. It is recommended to run the latest patch version for your Cilium minor version. You can find the latest patch release for a given minor Cilium release in the Stable Releases section of the Cilium documentation. If you are enabling BGP for your deployment, add the --set bgpControlPlane.enabled=true flag in the command below. If you are using a specific kubeconfig file, use the --kubeconfig flag with the Helm install command.
```
helm install cilium cilium/cilium \
    --version CILIUM_VERSION \
    --namespace kube-system \
    --values cilium-values.yaml
```

You can confirm your Cilium installation was successful with the following commands. You should see the cilium-operator deployment and the cilium-agent running on each of your hybrid nodes. Additionally, your hybrid nodes should now have status Ready. For information on how to configure BGP for Cilium, proceed to the next step.

kubectl get pods -n kube-system

NAME                              READY   STATUS    RESTARTS   AGE
cilium-jjjn8                      1/1     Running   0          11m
cilium-operator-d4f4d7fcb-sc5xn   1/1     Running   0          11m

kubectl get nodes

NAME                   STATUS   ROLES    AGE   VERSION
mi-04a2cf999b7112233   Ready    <none>   19m   v1.31.0-eks-a737599

To use BGP with Cilium to advertise your pod addresses with your on-premises network, you must have installed Cilium with bgpControlPlane.enabled: true. To configure BGP in Cilium, first create a file called cilium-bgp-cluster.yaml with a CiliumBGPClusterConfig with the peerAddress set to your on-premises router IP that you are peering with. Configure the localASN and peerASN based on your on-premises router configuration.

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchExpressions:
    - key: eks.amazonaws.com/compute-type
      operator: In
      values:
      - hybrid
  bgpInstances:
  - name: "rack0"
    localASN: ONPREM_ROUTER_ASN
    peers:
    - name: "onprem-router"
      peerASN: PEER_ASN
      peerAddress: ONPREM_ROUTER_IP
      peerConfigRef:
        name: "cilium-peer"

Apply the Cilium BGP Cluster configuration to your cluster.
```
kubectl apply -f cilium-bgp-cluster.yaml
```
The CiliumBGPPeerConfig resource is used to define a BGP peer configuration. Multiple peers can share the same configuration and provide reference to the common CiliumBGPPeerConfig resource. Create a file named cilium-bgp-peer.yaml to configure the peer configuration for your on-premises network. See the BGP Peer Configuration in the Cilium documentation for a full list of configuration options.
```
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  timers:
    holdTimeSeconds: 30
    keepAliveTimeSeconds: 10
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "bgp"
```
Apply the Cilium BGP Peer configuration to your cluster.
```
kubectl apply -f cilium-bgp-peer.yaml
```

The CiliumBGPAdvertisement resource is used to define various advertisement types and attributes associated with them. Create a file named cilium-bgp-advertisement.yaml and configure the CiliumBGPAdvertisement resource with your desired settings.

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisements
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: "PodCIDR"
    - advertisementType: "Service"
      service:
        addresses:
          - ClusterIP
          - ExternalIP
          - LoadBalancerIP

Apply the Cilium BGP Advertisement configuration to your cluster.
```
kubectl apply -f cilium-bgp-advertisement.yaml
```
You can confirm the BGP peering worked with the Cilium CLI by using the cilium bgp peers command. You should see the correct values in the output for your environment and the Session State as established. See the Troubleshooting and Operations Guide in the Cilium documentation for more information on troubleshooting.

Upgrade Cilium on hybrid nodes

Before upgrading your Cilium deployment, carefully review the Cilium upgrade documentation and the upgrade notes to understand the changes in the target Cilium version.

Ensure that you have installed the helm CLI on your command-line environment. See the Helm documentation for installation instructions.

Install the Cilium Helm repo.

helm repo add cilium https://helm.cilium.io/

Run the Cilium upgrade pre-flight check. Replace CILIUM_VERSION with your target Cilium version. It is recommended to run the latest patch version for your Cilium minor version. You can find the latest patch release for a given minor Cilium release in the Stable Releases section of the Cilium documentation.
```
helm install cilium-preflight cilium/cilium --version CILIUM_VERSION \
  --namespace=kube-system \
  --set preflight.enabled=true \
  --set agent=false \
  --set operator.enabled=false
```

After applying the cilium-preflight.yaml, ensure that the number of READY pods is the same number of Cilium pods running.

kubectl get ds -n kube-system | sed -n '1p;/cilium/p'

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
cilium                    2         2         2       2            2           <none>          1h20m
cilium-pre-flight-check   2         2         2       2            2           <none>          7m15s

Once the number of READY pods are equal, make sure the Cilium pre-flight deployment is also marked as READY 1/1. If it shows READY 0/1, consult the CNP Validation section and resolve issues with the deployment before continuing with the upgrade.
```
kubectl get deployment -n kube-system cilium-pre-flight-check -w
```
```
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cilium-pre-flight-check   1/1     1            0           12s
```

Delete the preflight

helm uninstall cilium-preflight --namespace kube-system

During normal cluster operations, all Cilium components should run the same version. The following steps describe how to upgrade all of the components from one stable release to a later stable release. When upgrading from one minor release to another minor release, it is recommended to upgrade to the latest patch release for the existing Cilium minor version first. To minimize disruption, the upgradeCompatibility option should be set to the initial Cilium version which was installed in this cluster.

Before running the helm upgrade command, preserve the values for your deployment in a cilium-values.yaml or use --set command line options for your settings. The upgrade operation overwrites the Cilium ConfigMap, so it is critical that your configuration values are passed when you upgrade. If you are using BGP, it is recommended to use the --set bgpControlPlane=true command line option instead of supplying this information in your values file.
```
helm upgrade cilium cilium/cilium --version CILIUM_VERSION \
  --namespace kube-system \
  --set upgradeCompatibility=1.X \
  -f cilium-values.yaml
```

(Optional) If you need to rollback your upgrade due to issues, run the following commands.

helm history cilium --namespace kube-system
helm rollback cilium [REVISION] --namespace kube-system

Delete Cilium from hybrid nodes

Run the following command to uninstall all Cilium components from your cluster. Note, uninstalling the CNI may impact the health of nodes and pods and shouldn’t be performed on production clusters.
```
helm uninstall cilium --namespace kube-system
```
The interfaces and routes configured by Cilium are not removed by default when the CNI is removed from the cluster, see the GitHub issue for more information.
To clean up the on-disk configuration files and resources, if you are using the standard configuration directories, you can remove the files as shown by the cni-uninstall.sh script in the Cilium repository on GitHub.
To remove the Cilium Custom Resource Definitions (CRDs) from your cluster, you can run the following commands.
```
kubectl get crds -oname | grep "cilium" | xargs kubectl delete
```

Install Calico on hybrid nodes

Ensure that you have installed the helm CLI on your command-line environment. See the Helm documentation for installation instructions.

Install the Cilium Helm repo.

helm repo add projectcalico https://docs.tigera.io/calico/charts

Create a yaml file called calico-values.yaml that configures Calico with affinity to run on hybrid nodes. For more information on the different Calico networking modes, see Determining the best networking option in the Calico documentation.
1. Replace POD_CIDR with the CIDR ranges for your pods. If you configured your Amazon EKS cluster with remote pod networks, the POD_CIDR that you specify for Calico should be the same as the remote pod networks. For example, 10.100.0.0/24.
2. Replace CIDR_SIZE with the size of the CIDR segment you wish to allocate to each node. For example, 25 for a /25 segment size. For more information on CIDR blockSize and changing the blockSize, see Change IP pool block size in the Calico documentation.
3. In the example below, natOutgoing is enabled and bgp is disabled. In this configuration, Calico can run on Amazon EKS clusters that have Remote Pod Network configured and can run on clusters that do not have Remote Pod Network configured. If you have natOutgoing set to disabled, you must configure your cluster with your remote pod networks and your on-premises network must be able to properly route traffic destined for your pod CIDRs. A common way to advertise pod addresses with your on-premises network is by using BGP. To use BGP with Calico, you must enable bgp. The example below configures all of the Calico components to run on only the hybrid nodes, since they have the eks.amazonaws.com/compute-type: hybrid label. If you are running webhooks on your hybrid nodes, you must configure your cluster with your Remote Pod Networks and you must advertise your pod addresses with your on-premises network. The example below configures controlPlaneReplicas: 1, increase the value if you have multiple hybrid nodes and want to run the Calico control plane components in a highly available fashion.
  installation: enabled: true cni: type: Calico ipam: type: Calico calicoNetwork: bgp: Disabled ipPools: - cidr: POD_CIDR blockSize: CIDR_SIZE encapsulation: VXLAN natOutgoing: Enabled nodeSelector: eks.amazonaws.com/compute-type == "hybrid" controlPlaneReplicas: 1 controlPlaneNodeSelector: eks.amazonaws.com/compute-type: hybrid calicoNodeDaemonSet: spec: template: spec: nodeSelector: eks.amazonaws.com/compute-type: hybrid csiNodeDriverDaemonSet: spec: template: spec: nodeSelector: eks.amazonaws.com/compute-type: hybrid calicoKubeControllersDeployment: spec: template: spec: nodeSelector: eks.amazonaws.com/compute-type: hybrid typhaDeployment: spec: template: spec: nodeSelector: eks.amazonaws.com/compute-type: hybrid
Install Calico on your cluster. Replace CALICO_VERSION with your desired Calico version (for example 3.29.0), see the Calico releases to find the latest patch release for your Calico minor version. It is recommended to run the latest patch version for the Calico minor version. If you are using a specific kubeconfig file, use the --kubeconfig flag.
```
helm install calico projectcalico/tigera-operator \
    --version CALICO_VERSION \
    --namespace kube-system \
    -f calico-values.yaml
```

You can confirm your Calico installation was successful with the following commands. You should see the tigera-operator deployment, the calico-node agent running on each of your hybrid nodes, as well as the calico-apiserver, csi-node-driver, and calico-kube-controllers deployed. Additionally, your hybrid nodes should now have status Ready. If you are using natOutgoing: Disabled, then all of the Calico components will not be able to start successfully until you advertise your pod addresses with your on-premises network. For information on how to configure BGP for Calico, proceed to the next step.

kubectl get pods -A

NAMESPACE          NAME                                       READY   STATUS    RESTARTS   AGE
calico-apiserver   calico-apiserver-6c77bb6d46-2n8mq          1/1     Running   0          69s
calico-system      calico-kube-controllers-7c5f8556b5-7h267   1/1     Running   0          68s
calico-system      calico-node-s5nnk                          1/1     Running   0          68s
calico-system      calico-typha-6487cc9d8c-wc5jm              1/1     Running   0          69s
calico-system      csi-node-driver-cv42d                      2/2     Running   0          68s
kube-system        coredns-7bb495d866-2lc9v                   1/1     Running   0          6m27s
kube-system        coredns-7bb495d866-2t8ln                   1/1     Running   0          157m
kube-system        kube-proxy-lxzxh                           1/1     Running   0          18m
kube-system        tigera-operator-f8bc97d4c-28b4d            1/1     Running   0          90s

kubectl get nodes

NAME                  STATUS   ROLES    AGE    VERSION
mi-0c6ec2f6f79176565  Ready    <none>   5h13m  v1.31.0-eks-a737599

If you installed Calico without BGP, skip this step. To configure BGP, create a file called calico-bgp.yaml with a BGPPeer configuration and a BGPConfiguration. It is important to distinguish BGPPeer and BGPConfiguration. The BGPPeer is the BGP-enabled router or remote resource with which the nodes in a Calico cluster will peer. The asNumber in the BGPPeer configuration is similar to the Cilium setting peerASN . The BGPConfiguration is applied to each Calico node and the asNumber for the BGPConfiguration is equivalent to the Cilium setting localASN. Replace ONPREM_ROUTER_IP, ONPREM_ROUTER_ASN, and LOCAL_ASN in the example below with the values for your on-premises environment. The keepOriginalNextHop: true setting is used to ensure each node advertises only the pod network CIDR that it owns.
```
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: calico-hybrid-nodes
spec:
  peerIP: ONPREM_ROUTER_IP
  asNumber: ONPREM_ROUTER_ASN
  keepOriginalNextHop: true
---
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  nodeToNodeMeshEnabled: false
  asNumber: LOCAL_ASN
```
Apply the file to your cluster.
```
kubectl apply -f calico-bgp.yaml
```

Confirm the Calico pods are running with the following command.

kubectl get pods -n calico-system -w

NAMESPACE          NAME                                       READY   STATUS    RESTARTS       AGE
calico-apiserver   calico-apiserver-598bf99b6c-2vltk          1/1     Running   0              3h24m
calico-system      calico-kube-controllers-75f84bbfd6-zwmnx   1/1     Running   31 (59m ago)   3h20m
calico-system      calico-node-9b2pg                          1/1     Running   0              5h17m
calico-system      calico-typha-7d55c76584-kxtnq              1/1     Running   0              5h18m
calico-system      csi-node-driver-dmnmm                      2/2     Running   0              5h18m
kube-system        coredns-7bb495d866-dtn4z                   1/1     Running   0              6h23m
kube-system        coredns-7bb495d866-mk7j4                   1/1     Running   0              6h19m
kube-system        kube-proxy-vms28                           1/1     Running   0              6h12m
kube-system        tigera-operator-55f9d9d565-jj9bg           1/1     Running   0              73m

If you encountered issues during these steps, see the troubleshooting guidance in the Calico documentation.

Upgrade Calico on hybrid nodes

Before upgrading your Calico deployment, carefully review the Calico upgrade documentation and the release notes to understand the changes in the target Calico version. The upgrade steps vary based on whether you are using Helm, the Calico operator, and the type of datastore. The steps below assume use of Helm.

Download the operator manifest for the version of Calico you are upgrading to. Replace CALICO_VERSION with the version you are upgrading to, for example v3.29.0. Make sure to prepend the v to the major.minor.patch.
```
kubectl apply --server-side --force-conflicts \
    -f https://raw.githubusercontent.com/projectcalico/calico/CALICO_VERSION/manifests/operator-crds.yaml
```
Run helm upgrade to upgrade your Calico deployment. Replace CALICO_VERSION with the version you are upgrading to, for example v3.29.0. Create the calico-values.yaml file from the configuration values that you used to install Calico.
```
helm upgrade calico projectcalico/tigera-operator \
    --version CALICO_VERSION \
    --namespace kube-system \
    -f calico-values.yaml
```

Delete Calico from hybrid nodes

Run the following command to uninstall Calico components from your cluster. Note that uninstalling the CNI may impact the health of nodes and pods and should not be performed on production clusters. If you installed Calico in a namespace other than kube-system change the namespace in the command below.
```
helm uninstall calico --namespace kube-system
```
Note that the interfaces and routes configured by Calico are not removed by default when the CNI is removed from the cluster.
To clean up the on-disk configuration files and resources, remove the Calico files from the /opt/cni and /etc/cni directories.

To remove the Calico CRDs from your cluster, run the following commands.

kubectl get crds -oname | grep "calico" | xargs kubectl delete

kubectl get crds -oname | grep "tigera" | xargs kubectl delete

9.8.5. Configure add-ons for hybrid nodes

Configure common add-ons for hybrid nodes

This page describes considerations for running Amazon EKS add-ons from AWS on Amazon EKS Hybrid Nodes. To learn more about the Amazon EKS add-ons from AWS and the processes for creating, upgrading, and removing add-ons from your cluster, see eks-add-ons.title. The processes for creating, upgrading, and removing Amazon EKS add-ons is the same for Amazon EKS clusters with hybrid nodes as it is for Amazon EKS clusters with nodes running in AWS Cloud unless otherwise noted on this page.

The following Amazon EKS add-ons from AWS are compatible with Amazon EKS Hybrid Nodes.

EKS add-on Compatible add-on versions

kube-proxy

v1.25.14-eksbuild.2 and above

CoreDNS

v1.9.3-eksbuild.7 and above

AWS Distro for OpenTelemetry (ADOT)

v0.102.1-eksbuild.2 and above

CloudWatch Observability Agent

v2.2.1-eksbuild.1 and above

EKS Pod Identity Agent

v1.3.3-eksbuild.1 and above

CSI snapshot controller

v8.1.0-eksbuild.1 and above

In addition to the Amazon EKS add-ons in the table above, the Amazon Managed Service for Prometheus Collector, and the AWS Load Balancer Controller for application ingress (HTTP) and load balancing (TCP/UDP) are compatible with hybrid nodes.

Amazon EKS add-ons from AWS that are not compatible with Amazon EKS Hybrid Nodes have been updated with an affinity rule for the default eks.amazonaws.com/compute-type: hybrid label applied to hybrid nodes. This prevents them from running on hybrid nodes when deployed in your clusters. If you have clusters with both hybrid nodes and nodes running in AWS Cloud, Amazon EKS add-ons that are not compatible with hybrid nodes can still be deployed in your cluster to nodes running in AWS Cloud. The Amazon VPC CNI is not compatible with hybrid nodes, and Cilium and Calico are supported as the Container Networking Interfaces (CNIs) for Amazon EKS Hybrid Nodes. See hybrid-nodes-cni.title for more information.

The rest of this page describes differences between running compatible Amazon EKS add-ons from AWS on hybrid nodes, compared to the other Amazon EKS compute types.

kube-proxy and CoreDNS

Kube-proxy and CoreDNS are installed as unmanaged add-ons by default when an EKS cluster is created. These add-ons can be managed as Amazon EKS add-ons after cluster creation. Reference the EKS documentation for details on managing-kube-proxy.title and managing-coredns.title. If you are running a cluster with hybrid nodes and nodes in AWS Cloud, it is recommended to have at least one CoreDNS replica on hybrid nodes and at least one CoreDNS replica on your nodes in AWS Cloud.

CloudWatch Observability Agent add-on

Node-level metrics are not available for hybrid nodes because CloudWatch Container Insights depends on the availability of Instance Metadata Service (IMDS) for node-level metrics. Cluster, workload, pod, and container-level metrics are available for hybrid nodes.

After installing the add-on by following the steps described in Install the CloudWatch agent with the Amazon CloudWatch Observability, the add-on manifest must be updated before the agent can run successfully on hybrid nodes. Edit the amazoncloudwatchagents resource on the cluster to add the RUN_WITH_IRSA environment variable as shown below.

kubectl edit amazoncloudwatchagents -n amazon-cloudwatch cloudwatch-agent

apiVersion: v1
items:
- apiVersion: cloudwatch.aws.amazon.com/v1alpha1
  kind: AmazonCloudWatchAgent
  metadata:
    ...
    name: cloudwatch-agent
    namespace: amazon-cloudwatch
    ...
  spec:
    ...
    env:
    - name: RUN_WITH_IRSA # <-- Add this
      value: "True" # <-- Add this
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
          ...

Amazon Managed Prometheus managed collector for hybrid nodes

An Amazon Managed Service for Prometheus (AMP) managed collector consists of a scraper that discovers and collects metrics from the resources in an Amazon EKS cluster. AMP manages the scraper for you, removing the need to manage any instances, agents, or scrapers yourself.

You can use AMP managed collectors without any additional configuration specific to hybrid nodes. However the metric endpoints for your applications on the hybrid nodes must be reachable from the VPC, including routes from the VPC to remote pod network CIDRs and the ports open in your on-premises firewall. Additionally, your cluster must have private cluster endpoint access.

Follow the steps in Using an AWS managed collector in the Amazon Managed Service for Prometheus User Guide.

`AWS` Distro for OpenTelemetry (ADOT) add-on

You can use the AWS Distro for OpenTelemetry (ADOT) Amazon EKS add-on to collect metrics, logs, and tracing data from your applications running on hybrid nodes. Note, ADOT uses admission webhooks to mutate and validate the Collector Custom Resource requests and you must configure your remote pod network when creating your Amazon EKS cluster.

Follow the steps in Getting Started with AWS Distro for OpenTelemetry using EKS Add-Ons in the AWS Distro for OpenTelemetry documentation.

`AWS` Load Balancer Controller

You can use the AWS Load Balancer Controller and Application Load Balancer (ALB) or Network Load Balancer (NLB) with the target type ip for workloads on hybrid nodes connected with AWS Direct Connect or AWS Site-to-Site VPN. As the AWS Load Balancer Controller uses webhooks, you must configure your remote pod network when creating your Amazon EKS cluster.

To install the AWS Load Balancer Controller, follow the steps at lbc-helm.title or lbc-manifest.title.

For ingress with ALB, you must specify the annotations below. See alb-ingress.title for instructions.

alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip

For load balancing with NLB, you must specify the annotations below. See network-load-balancing.title for instructions.

service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

EKS Pod Identity Agent add-on

The original Amazon EKS Pod Identity Agent DaemonSet relies on the availability of EC2 IMDS on the node to obtain the required AWS credentials. As IMDS isn’t available on hybrid nodes, starting in add-on version 1.3.3-eksbuild.1, the Pod Identity Agent add-on optionally deploys a second DaemonSet that specifically targets hybrid nodes. This DaemonSet mounts the required credentials to the pods created by the Pod Identity Agent add-on.

To use the Pod Identity agent on hybrid nodes, set enableCredentialsFile: true in the hybrid section of nodeadm config as shown below:
```
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
    hybrid:
        enableCredentialsFile: true # <-- Add this
```
This will configure nodeadm to create a credentials file to be configured on the node under /eks-hybrid/.aws/credentials, which will be used by eks-pod-identity-agent pods. This credentials file will contain temporary AWS credentials that will be refreshed periodically.
After you update the nodeadm config on each node, run the following nodeadm init command with your nodeConfig.yaml to join your hybrid nodes to your Amazon EKS cluster. If your nodes have joined the cluster previous, still run the init command again.
```
nodeadm init -c file://nodeConfig.yaml
```
Install eks-pod-identity-agent with support for hybrid nodes enabled, by either using the CLI or consolelong.
1. CLI: From the machine that you’re using to administer the cluster, run the following command to install eks-pod-identity-agent with support for hybrid nodes enabled.
  aws eks create-addon \ --cluster-name cluster-name \ --addon-name eks-pod-identity-agent \ --configuration-values '{"daemonsets":{"hybrid":{"create": true}}}'
2. consolelong: If you are installing the Pod Identity Agent add-on through the AWS console, add the following to the optional configuration to deploy the daemonset that targets hybrid nodes.
  {"daemonsets":{"hybrid":{"create": true}}}

CSI snapshot controller add-on

Starting with version v8.1.0-eksbuild.2, the CSI snapshot controller add-on applies a soft anti-affinity rule for hybrid nodes, preferring the controller deployment to run on EC2 in the same AWS Region as the Amazon EKS control plane. Co-locating the deployment in the same AWS Region as the Amazon EKS control plane improves latency.

9.8.6. Configure proxy for hybrid nodes

Configure HTTP/S proxies for Amazon EKS hybrid nodes

If you are using a proxy server in your on-premises environment for traffic leaving your data center or edge environment, you need to configure your operating system, containerd, kubelet, and kube-proxy to use your proxy server. You must configure kube-proxy after creating your Amazon EKS cluster. You can make the changes for your operating system, containerd, and the kubelet during the build process for your operating system images or before you run nodeadm init on each hybrid node.

Node-level configuration

The configurations in this section must be applied in your operating system images or before running nodeadm init on each hybrid node.

`containerd` proxy configuration

containerd is the default container management runtime for Kubernetes. If you are using a proxy for internet access, you must configure containerd so it can pull the container images required by Kubernetes and Amazon EKS.

Create a file on each hybrid node called http-proxy.conf in the /etc/systemd/system/containerd.service.d directory with the following contents. Replace proxy-domain and port with the values for your environment.

[Service]
Environment="HTTP_PROXY=http://proxy-domain:port"
Environment="HTTPS_PROXY=http://proxy-domain:port"
Environment="NO_PROXY=localhost"

Kubelet proxy configuration

kubelet is the Kubernetes node agent that runs on each Kubernetes node and is responsible for managing the node and pods running on it. If you are using a proxy in your on-premises environment, you must configure the kubelet so it can communicate with your Amazon EKS cluster’s public or private endpoints.

Create a file on each hybrid node called http-proxy.conf in the /etc/systemd/system/kubelet.service.d/ directory with the following content. Replace proxy-domain and port with the values for your environment.

[Service]
Environment="HTTP_PROXY=http://proxy-domain:port"
Environment="HTTPS_PROXY=http://proxy-domain:port"
Environment="NO_PROXY=localhost"

Operating system proxy configuration

If you are using a proxy for internet access, you must configure your operating system to be able to pull the hybrid nodes dependencies from your operating systems' package manager.

Ubuntu

Configure snap to use your proxy with the following commands:

sudo snap set system proxy.https=http://proxy-domain:port
sudo snap set system proxy.http=http://proxy-domain:port

To enable proxy for apt, create a file called apt.conf in the /etc/apt/ directory. Replace proxy-domain and port with the values for your environment.
```
Acquire::http::Proxy "http://proxy-domain:port";
Acquire::https::Proxy "http://proxy-domain:port";
```

Amazon Linux 2023 and Red Hat Enterprise Linux

Configure yum to use your proxy. Create a file /etc/yum.conf with the proxy-domain and port values for your environment.
```
proxy=http://proxy-domain:port
```

Cluster wide configuration

The configurations in this section must be applied after you create your Amazon EKS cluster and before running nodeadm init on each hybrid node.

kube-proxy proxy configuration

Amazon EKS automatically installs kube-proxy on each hybrid node as a DaemonSet when your hybrid nodes join the cluster. kube-proxy enables routing across services that are backed by pods on Amazon EKS clusters. To configure each host, kube-proxy requires DNS resolution for your Amazon EKS cluster endpoint.

Edit the kube-proxy DaemonSet with the following command
```
kubectl -n kube-system edit ds kube-proxy
```
This will open the kube-proxy DaemonSet definition on your configured editor.

Add the environment variables for HTTP_PROXY and HTTPS_PROXY. Note the NODE_NAME environment variable should already exist in your configuration. Replace proxy-domain and port with values for your environment.

containers:
  - command:
    - kube-proxy
    - --v=2
    - --config=/var/lib/kube-proxy-config/config - --hostname-override=$(NODE_NAME)
    env:
    - name: HTTP_PROXY
      value: http://proxy-domain:port
    - name: HTTPS_PROXY
      value: http://proxy-domain:port
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName

9.8.7. Hybrid nodes `nodeadm` reference

Hybrid nodes nodeadm reference

The Amazon EKS Hybrid Nodes CLI (nodeadm) used for hybrid nodes lifecycle management differs from the nodeadm version used for bootstrapping Amazon EC2 instances as nodes in Amazon EKS clusters. Follow the documentation and references for the appropriate nodeadm version. This documentation page is for the hybrid nodes nodeadm version and the hybrid nodes nodeadm version is available in the eks-hybrid repository on GitHub. See the nodeadm - Amazon EKS AMI documentation for the nodeadm version used for Amazon EC2 instances.

Download `nodeadm`

For x86_64 hosts:

curl -OL 'https://hybrid-assets.eks.amazonaws.com/releases/latest/bin/linux/amd64/nodeadm'

For ARM hosts

curl -OL 'https://hybrid-assets.eks.amazonaws.com/releases/latest/bin/linux/arm64/nodeadm'

Add executable file permission to the downloaded binary on each host.

chmod +x nodeadm

Commands

You must run nodeadm with a user that has root/sudo privileges.

Install

The install command is used to install the artifacts and dependencies required to run and join hybrid nodes to an Amazon EKS cluster. The install command can be run individually on each hybrid node or can be run during image build pipelines to preinstall the hybrid nodes dependencies in operating system images.

Usage

nodeadm install [KUBERNETES_VERSION] [flags]

Positional Arguments

(Required) KUBERNETES_VERSION The major.minor version of EKS Kubernetes to install, for example 1.31

Flags

Name Required Description

-p,

--credential-provider

TRUE

Credential provider to install. Supported values are iam-ra and ssm. See hybrid-nodes-creds.title for more information.

-s,

--containerd-source

FALSE

Source for containerd. nodeadm supports installing containerd from the OS distro, Docker packages, and skipping containerd install.

Values

distro - This is the default value. nodeadm will install containerd package distributed by the node OS. distro is not a supported value for Red Hat Enterprise Linux (RHEL) operating systems.

docker - nodeadm will install containerd package built and distributed by Docker. docker is not a supported value for Amazon Linux 2023

none - nodeadm will not install containerd package. You must manually install containerd before running nodeadm init.

-t,

--timeout

FALSE

Maximum install command duration. The input follows duration format. For example 1h23m. Default download timeout for install command is set to 20 minutes.

-h, --help

FALSE

Displays help message with available flag, subcommand and positional value parameters.

Examples

Install Kubernetes version 1.31 with AWS Systems Manager (SSM) as the credential provider

nodeadm install 1.31 --credential-provider ssm

Install Kubernetes version 1.31 with AWS Systems Manager (SSM) as the credential provider, Docker as the containerd source, with a download timeout of 20 minutes.

nodeadm install 1.31 --credential-provider ssm --containerd-source docker --timeout 20m

Install Kubernetes version 1.31 with AWS IAM Roles Anywhere as the credential provider

nodeadm install 1.31 --credential-provider iam-ra

Files installed

Artifact Path

IAM Roles Anywhere CLI

/usr/local/bin/aws_signing_helper

Kubelet binary

/usr/bin/kubelet

Kubectl binary

usr/local/bin/kubectl

ECR Credentials Provider

/etc/eks/image-credential-provider/ecr-credential-provider

AWS IAM Authenticator

/usr/local/bin/aws-iam-authenticator

SSM Setup CLI

/opt/ssm/ssm-setup-cli

SSM Agent

On Ubuntu - /snap/amazon-ssm-agent/current/amazon-ssm-agent

On RHEL & AL2023 - /usr/bin/amazon-ssm-agent

Containerd

On Ubuntu & AL2023 - /usr/bin/containerd

On RHEL - /bin/containerd

Iptables

On Ubuntu & AL2023 - /usr/sbin/iptables

On RHEL - /sbin/iptables

CNI plugins

/opt/cni/bin

installed artifacts tracker

/opt/nodeadm/tracker

Config check

The config check command checks the provided node configuration for errors. This command can be used to verify and validate the correctness of a hybrid node configuration file.

Usage

nodeadm config check [flags]

Flags

Name Required Description

-c,

--config-source

TRUE

Source of nodeadm configuration. For hybrid nodes the input should follow a URI with file scheme.

-h, --help

FALSE

Displays help message with available flag, subcommand and positional value parameters.

Examples

nodeadm config check --config-source file:///root/nodeConfig.yaml

Init

The init command starts and connects the hybrid node with the configured Amazon EKS cluster.

Usage

nodeadm init [flags]

Flags

Name Required Description

-c,

--config-source

TRUE

Source of nodeadm configuration. For hybrid nodes the input should follow a URI with file scheme.

-s,

--skip

FALSE

Phases of init to be skipped. It is not recommended to skip any of the phases unless it helps to fix an issue.

Values

install-validation skips checking if the proceding install command ran successfully.

-h, `--help

FALSE

Displays help message with available flag, subcommand and positional value parameters.

Examples

nodeadm init --config-source file://root/nodeConfig.yaml

Files installed

Name Path

Kubelet kubeconfig

/var/lib/kubelet/kubeconfig

Kubelet config

/etc/kubernetes/kubelet/config.json

Kubelet systemd unit

/etc/systemd/system/kubelet.service

Image credentials provider config

/etc/eks/image-credential-provider/config.json

Kubelet env file

/etc/eks/kubelet/environment

Kubelet Certs

/etc/kubernetes/pki/ca.crt

Containerd config

/etc/containerd/config.toml

Containerd kernel modules config

/etc/modules-load.d/contianerd.conf

AWS config file

/etc/aws/hybrid/config

AWS credentials file (if enable credentials file)

/eks-hybrid/.aws/credentials

AWS signing helper system unit

/etc/systemd/system/aws_signing_helper_update.service

Sysctl conf file

/etc/sysctl.d/99-nodeadm.conf

Apt manager files for docker repo (if containerd source is docker)

Ca-certificates

/etc/ssl/certs/ca-certificates.crt

Gpg key file

/etc/apt/keyrings/docker.asc

Docker repo source file

/etc/apt/sources.list.d/docker.list

Upgrade

The nodeadm upgrade command upgrades all the installed artifacts to the latest version and bootstraps the node to configure the upgraded artifacts and join the EKS cluster on AWS. Upgrade is a disruptive command to the workloads running on the node. Please move your workloads to another node before running upgrade.

Usage

nodeadm upgrade [KUBERNETES_VERSION] [flags]

Positional Arguments

(Required) KUBERNETES_VERSION The major.minor version of EKS Kubernetes to install, for example 1.31

Flags

Name Required Description

-c,

--config-source

TRUE

Source of nodeadm configuration. For hybrid nodes the input should follow a URI with file scheme.

-t,

--timeout

FALSE

Timeout for downloading artifacts. The input follows duration format. For example 1h23m. Default download timeout for upgrade command is set to 10 minutes.

-s,

--skip

FALSE

Phases of upgrade to be skipped. It is not recommended to skip any of the phase unless it helps to fix an issue.

Values

pod-validation skips checking if all the no pods are running on the node, except daemon sets and static pods.

node-validation skips checking if the node has been cordoned.

init-validation skips checking if the node has been initialized successfully before running upgrade.

-h, --help

FALSE

Displays help message with available flag, subcommand and positional value parameters.

Examples

nodeadm upgrade 1.31 --config-source file:///root/nodeConfig.yaml

nodeadm upgrade 1.31 --config-source file:///root/nodeConfig.yaml --timeout 20m

Uninstall

The nodeadm uninstall command stops and removes the artifacts nodeadm installs during nodeadm install, including the kubelet and containerd. Note, the uninstall command does not drain or delete your hybrid nodes from your cluster. You must run the drain and delete operations separately, see hybrid-nodes-remove.title for more information. By default, nodeadm uninstall will not proceed if there are pods remaining on the node. Similarly, nodeadm uninstall does not remove CNI dependencies or dependencies of other Kubernetes add-ons you run on your cluster. To fully remove the CNI installation from your host, see the instructions at hybrid-nodes-cni.title. If you are using AWS SSM hybrid activations as your on-premises credentials provider, the nodeadm uninstall command deregisters your hosts as AWS SSM managed instances.

Usage

nodeadm uninstall [flags]

Flags

Name Required Description

-s,

--skip

FALSE

Phases of upgrade to be skipped. It is not recommended to skip any of the phase unless it helps to fix an issue.

Values

pod-validation skips checking if all the no pods are running on the node, except daemon sets and static pods.

node-validation skips checking if the node has been cordoned.

init-validation skips checking if the node has been initialized successfully before running upgrade.

-h,

--help

FALSE

Displays help message with available flag, subcommand and positional value parameters.

Examples

nodeadm uninstall

nodeadm uninstall --skip node-validation,pod-validation

Debug

The nodeadm debug command can be used to troubleshoot unhealthy or misconfigured hybrid nodes. It validates the following requirements are in-place.

The node has network access to the required AWS APIs for obtaining credentials,
The node is able to get AWS credentials for the configured Hybrid Nodes IAM role,
The node has network access to the EKS Kubernetes API endpoint and the validity of the EKS Kubernetes API endpoint certificate,
The node is able to authenticate with the EKS cluster, its identity in the cluster is valid, and that the node has access to the EKS cluster through the VPC configured for the EKS cluster.

If errors are found, the command’s output suggests troubleshooting steps. Certain validation steps show child processes. If these fail, the output is showed in a stderr section under the validation error.

Usage

nodeadm debug [flags]

Flags

Name Required Description

-c, --config-source

TRUE

Source of nodeadm configuration. For hybrid nodes the input should follow a URI with file scheme.

-h, --help

FALSE

Displays help message with available flag, subcommand and positional value parameters.

Examples

nodeadm debug --config-source file://nodeConfig.yaml

Node Config API Reference

AWS SSM hybrid activations

The following is a sample nodeConfig.yaml when using AWS SSM hybrid activations for hybrid nodes credentials.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name:             # Name of the EKS cluster
    region:           # AWS Region where the EKS cluster resides
  hybrid:
    ssm:
      activationCode: # SSM hybrid activation code
      activationId:   # SSM hybrid activation id

AWS IAM Roles Anywhere

The following is a sample nodeConfig.yaml for AWS IAM Roles Anywhere for hybrid nodes credentials.

When using AWS IAM Roles Anywhere as your on-premises credentials provider, the nodeName you use in your nodeadm configuration must align with the permissions you scoped for your Hybrid Nodes IAM role. For example, if your permissions for the Hybrid Nodes IAM role only allow AWS IAM Roles Anywhere to assume the role when the role session name is equal to the CN of the host certificate, then the nodeName in your nodeadm configuration must be the same as the CN of your certificates. The nodeName that you use can’t be longer than 64 characters. For more information, see hybrid-nodes-creds.title.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name:              # Name of the EKS cluster
    region:            # AWS Region where the EKS cluster resides
  hybrid:
    iamRolesAnywhere:
      nodeName:        # Name of the node
      trustAnchorArn:  # ARN of the IAM Roles Anywhere trust anchor
      profileArn:      # ARN of the IAM Roles Anywhere profile
      roleArn:         # ARN of the Hybrid Nodes IAM role
      certificatePath: # Path to the certificate file to authenticate with the IAM Roles Anywhere trust anchor
      privateKeyPath:  # Path to the private key file for the certificate

(Optional) Kubelet configuration

You can pass kubelet configuration and flags in your nodeadm configuration. See the example below for how to add an additional node label abc.amazonaws.com/test-label and config for setting shutdownGracePeriod to 30 seconds.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name:             # Name of the EKS cluster
    region:           # AWS Region where the EKS cluster resides
  kubelet:
    config:           # Map of kubelet config and values
       shutdownGracePeriod: 30s
    flags:            # List of kubelet flags
       - --node-labels=abc.company.com/test-label=true
  hybrid:
    ssm:
      activationCode: # SSM hybrid activation code
      activationId:   # SSM hybrid activation id

(Optional) Containerd configuration

You can pass custom containerd configuration in your nodeadm configuration. The containerd configuration for nodeadm accepts in-line TOML. See the example below for how to configure containerd to disable deletion of unpacked image layers in the containerd content store.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name:             # Name of the EKS cluster
    region:           # AWS Region where the EKS cluster resides
  containerd:
    config: |         # Inline TOML containerd additional configuration
       [plugins."io.containerd.grpc.v1.cri".containerd]
       discard_unpacked_layers = false
  hybrid:
    ssm:
      activationCode: # SSM hybrid activation code
      activationId:   # SSM hybrid activation id

You can also use the containerd configuration to enable SELinux support. With SELinux enabled on containerd, ensure pods scheduled on the node have the proper securityContext and seLinuxOptions enabled. More information on configuring a security context can be found on the Kubernetes documentation.

Red Hat Enterprise Linux (RHEL) 8 and RHEL 9 have SELinux enabled by default and set to strict on the host. Amazon Linux 2023 has SELinux enabled by default and set to permissive mode. When SELinux is set to permissive mode on the host, enabling it on containerd will not block requests but will log it according to the SELinux configuration on the host.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name:             # Name of the EKS cluster
    region:           # AWS Region where the EKS cluster resides
  containerd:
    config: |         # Inline TOML containerd additional configuration
       [plugins."io.containerd.grpc.v1.cri"]
       enable_selinux = true
  hybrid:
    ssm:
      activationCode: # SSM hybrid activation code
      activationId:   # SSM hybrid activation id

9.8.8. Troubleshooting hybrid nodes

Troubleshoot, diagnose, and repair hybrid nodes from your data centers to Amazon EKS Kubernetes clusters.

This topic covers some common errors that you may see while using Amazon EKS Hybrid Nodes and how to fix them. For other troubleshooting information, see troubleshooting.title and Knowledge Center tag for Amazon EKS on AWS re:Post. If you cannot resolve the issue, contact AWS Support.

You can run the nodeadm debug command from your hybrid nodes to validate networking and credential requirements are met. For more information on the nodeadm debug command, see hybrid-nodes-nodeadm.title.

Installing hybrid nodes troubleshooting

The troubleshooting topics in this section are related to installing the hybrid nodes dependencies on hosts with the nodeadm install command.

nodeadm command failed must run as root

The nodeadm install command must be run with a user that has root/sudo privileges on your host. If you run nodeadm install with a user that does not have root/sudo privileges, you will see the following error in the nodeadm output.

"msg":"Command failed","error":"must run as root"

Unable to connect to dependencies

The nodeadm install command installs the dependencies required for hybrid nodes. The hybrid nodes dependencies include containerd, kubelet, kubectl, and AWS SSM or AWS IAM Roles Anywhere components. You must have access from where you are running nodeadm install to download these dependencies. For more information on the list of locations that you must be able to access, see hybrid-nodes-networking.title. If you do not have access, you will see errors similar to the following in the nodeadm install output.

"msg":"Command failed","error":"failed reading file from url: ...: max retries achieved for http request"

Failed to update package manager

The nodeadm install command runs apt update or yum/dnf update before installing the hybrid nodes dependencies. If this step does not succeed you may see errors similar to the following. To remediate, you can run apt update or yum/dnf update before running nodeadm install or you can attempt to re-run nodeadm install.

failed to run update using package manager

Timeout or context deadline exceeded

When running nodeadm install, if you see issues at various stages of the install process with errors that indicate there was a timeout or context deadline exceeded, you may have a slow connection that is preventing the installation of the hybrid nodes dependencies before timeouts are met. To work around these issues, you can attempt to use the --timeout flag in nodeadm to extend the duration of the timeouts for downloading the dependencies.

nodeadm install K8S_VERSION --credential-provider CREDS_PROVIDER --timeout 20m0s

Connecting hybrid nodes troubleshooting

The troubleshooting topics in this section are related to the process of connecting hybrid nodes to EKS clusters with the nodeadm init command.

Operation errors or unsupported scheme

When running nodeadm init, if you see errors related to operation error or unsupported scheme, check your nodeConfig.yaml to make sure it is properly formatted and passed to nodeadm. For more information on the format and options for nodeConfig.yaml, see hybrid-nodes-nodeadm.title.

"msg":"Command failed","error":"operation error ec2imds: GetRegion, request canceled, context deadline exceeded"

Hybrid Nodes IAM role missing permissions for the eks:DescribeCluster action

When running nodeadm init, nodeadm attempts to gather information about your EKS cluster by calling Describe Cluster. If your Hybrid Nodes IAM role does not have permission for the eks:DescribeCluster action. For more information on the required permissions for the Hybrid Nodes IAM role, see hybrid-nodes-creds.title.

"msg":"Command failed","error":"operation error EKS: DescribeCluster, https response error StatusCode: 403 ... AccessDeniedException"

Hybrid nodes are not appearing in EKS cluster

If you ran nodeadm init and it completed but your hybrid nodes do not appear in your cluster, there may be issues with the network connection between your hybrid nodes and the EKS control plane, you may not have the required security group permissions configured, or you may not have the required mapping of your Hybrid Nodes IAM role to Kubernetes Role-Based Access Control (RBAC). You can start the debugging process by checking the status of kubelet and the kubelet logs with the following commands. Run the following commands from the hybrid nodes that failed to join your cluster.

systemctl status kubelet

journalctl -u kubelet -f

Unable to communicate with cluster

If your hybrid node was unable to communicate with the cluster control plane, you may see logs similar to the following.

"Failed to ensure lease exists, will retry" err="Get ..."

"Unable to register node with API server" err="Post ..."

Failed to contact API server when waiting for CSINode publishing ... dial tcp <ip address>: i/o timeout

If you see these messages, check the following to ensure it meets the hybrid nodes requirements detailed in hybrid-nodes-networking.title.

Confirm the VPC passed to EKS cluster has routes to your Transit Gateway (TGW) or Virtual Private Gateway (VGW) for your on-premises node and optionally pod CIDRs.
Confirm you have an additional security group for your EKS cluster has inbound rules for your on-premises node CIDRs and optionally pod CIDRs.
Confirm your on-premises router is configured to allow traffic to and from the EKS control plane.

Unauthorized

If your hybrid node was able to communicate with the EKS control plane but was not able to register, you may see logs similar to the following. Note the key difference in the log messages below is the Unauthorized error. This signals that the node was not able to perform its tasks because it does not have the required permissions.

"Failed to ensure lease exists, will retry" err="Unauthorized"

"Unable to register node with API server" err="Unauthorized"

Failed to contact API server when waiting for CSINode publishing: Unauthorized

If you see these messages, check the following to ensure it meets the hybrid nodes requirements details in hybrid-nodes-creds.title and hybrid-nodes-cluster-prep.title.

Confirm the identity of the hybrid nodes matches your expected Hybrid Nodes IAM role. This can be done by running sudo aws sts get-caller-identity from your hybrid nodes.
Confirm your Hybrid Nodes IAM role has the required permissions.
Confirm that in your cluster you have an EKS access entry for your Hybrid Nodes IAM role or confirm that your aws-auth ConfigMap has an entry for your Hybrid Nodes IAM role. If you are using EKS access entries, confirm your access entry for your Hybrid Nodes IAM role has the HYBRID_LINUX access type. If you are using the aws-auth ConfigMap, confirm your entry for the Hybrid Nodes IAM role meets the requirements and formatting detailed in hybrid-nodes-cluster-prep.title.

Hybrid nodes registered with EKS cluster but show status `Not Ready`

If your hybrid nodes successfully registered with your EKS cluster, but the hybrid nodes show status Not Ready, the first thing to check is your Container Networking Interface (CNI) status. If you have not installed a CNI, then it is expected that your hybrid nodes have status Not Ready. Once a CNI is installed and running successfully, nodes transition to have the status Ready. If you attempted to install a CNI but it is not running successfully, see hybrid-nodes-troubleshooting-cni.title on this page.

Certificate Signing Requests (CSRs) are stuck Pending

After connecting hybrid nodes to your EKS cluster, if you see that there are pending CSRs for your hybrid nodes, your hybrid nodes are not meeting the requirements for automatic approval. CSRs for hybrid nodes are automatically approved and issued if the CSRs for hybrid nodes were created by a node with eks.amazonaws.com/compute-type: hybrid label, and the CSR has the following Subject Alternative Names (SANs): at least one DNS SAN equal to the node name and the IP SANs belong to the remote node network CIDRs.

Hybrid profile already exists

If you changed your nodeadm configuration and attempt to re-register the node with the new configuration, you may see an error that states that the hybrid profile already exists but its contents have changed. Instead of running nodeadm init in between configuration changes, run nodeadm uninstall followed by a nodeadm install and nodeadm init. This ensures a proper clean up with the changes in configuration.

"msg":"Command failed","error":"hybrid profile already exists at /etc/aws/hybrid/config but its contents do not align with the expected configuration"

Hybrid node failed to resolve Private API

After running nodeadm init, if you see an error in the kubelet logs that shows failures to contact the EKS Kubernetes API server because there is no such host, you may have to change your DNS entry for the EKS Kubernetes API endpoint in your on-premises network or at the host level. See Forwarding inbound DNS queries to your VPC in the AWS Route53 documentation.

Failed to contact API server when waiting for CSINode publishing: Get ... no such host

Can’t view hybrid nodes in the EKS console

If you have registered your hybrid nodes but are unable to view them in the EKS console, check the permissions of the IAM principal you are using to view the console. The IAM principal you’re using must have specific minimum IAM and Kubernetes permissions to view resources in the console. For more information, see view-kubernetes-resources.title.

Running hybrid nodes troubleshooting

If your hybrid nodes registered with your EKS cluster, had status Ready, and then transitioned to status Not Ready, there are a wide range of issues that may have contributed to the unhealthy status such as the node lacking sufficient resources for CPU, memory, or available disk space, or the node is disconnected from the EKS control plane. You can use the steps below to troubleshoot your nodes, and if you cannot resolve the issue, contact AWS Support.

Run nodeadm debug from your hybrid nodes to validate networking and credential requirements are met. For more information on the nodeadm debug command, see hybrid-nodes-nodeadm.title.

Get node status

kubectl get nodes -o wide

Check node conditions and events

kubectl describe node NODE_NAME

Get pod status

kubectl get pods -A -o wide

Check pod conditions and events

kubectl describe pod POD_NAME

Check pod logs

kubectl logs POD_NAME

Check kubectl logs

systemctl status kubelet

journalctl -u kubelet -f

Pod liveness probes failing or webhooks are not working

If applications, add-ons, or webhooks running on your hybrid nodes are not starting properly, you may have networking issues that block the communication to the pods. For the EKS control plane to contact webhooks running on hybrid nodes, you must configure your EKS cluster with a remote pod network and have routes for your on-premises pod CIDR in your VPC routing table with the target as your Transit Gateway (TGW), virtual private gateway (VPW), or other gateway you are using to connect your VPC with your on-premises network. For more information on the networking requirements for hybrid nodes, see hybrid-nodes-networking.title. You additionally must allow this traffic in your on-premises firewall and ensure your router can properly route to your pods.

A common pod log message for this scenario is shown below the following where ip-address is the Cluster IP for the Kubernetes service.

dial tcp <ip-address>:443: connect: no route to host

Hybrid nodes CNI troubleshooting

If you run into issues with initially starting Cilium or Calico with hybrid nodes, it is most often due to networking issues between hybrid nodes or the CNI pods running on hybrid nodes, and the EKS control plane. Make sure your environment meets the requirements in Prepare networking for hybrid nodes. It’s useful to break down the problem into parts.

EKS cluster configuration: Are the RemoteNodeNetwork and RemotePodNetwork configurations correct?
VPC configuration: Are there routes for the RemoteNodeNetwork and RemotePodNetwork in the VPC routing table with the target of the Transit Gateway or Virtual Private Gateway?
Security group configuration: Are there inbound and outbound rules for the RemoteNodeNetwork and RemotePodNetwork ?
On-premises network: Are there routes and access to/from the EKS control plane and to/from the hybrid nodes and the pods running on hybrid nodes?
CNI configuration: If using an overlay network, does the IP pool configuration for the CNI match the RemotePodNetwork configured for the EKS cluster if using webhooks?

Hybrid node has status Ready without a CNI installed

If your hybrid nodes are showing status Ready, but you have not installed a CNI on your cluster, it is possible that there are old CNI artifacts on your hybrid nodes. By default, when you uninstall Cilium and Calico with tools such as Helm, the on-disk resources are not removed from your physical or virtual machines. Additionally, the Custom Resource Definitions (CRDs) for these CNIs may still be present on your cluster from an old installation. For more information, see the Delete Cilium and Delete Calico sections of hybrid-nodes-cni.title.

Cilium troubleshooting

If you are having issues running Cilium on hybrid nodes, see the troubleshooting steps in the Cilium documentation. The sections below cover issues that may be unique to deploying Cilium on hybrid nodes.

Cilium isn’t starting

If the Cilium agents that run on each hybrid node are not starting, check the logs of the Cilium agent pods for errors. The Cilium agent requires connectivity to the EKS Kubernetes API endpoint to start. Cilium agent startup will fail if this connectivity is not correctly configured. In this case, you will see log messages similar to the following in the Cilium agent pod logs.

msg="Unable to contact k8s api-server"
level=fatal msg="failed to start: Get \"https://<k8s-cluster-ip>:443/api/v1/namespaces/kube-system\": dial tcp <k8s-cluster-ip>:443: i/o timeout"

The Cilium agent runs on the host network. Your EKS cluster must be configured with RemoteNodeNetwork for the Cilium connectivity. Confirm you have an additional security group for your EKS cluster with an inbound rule for your RemoteNodeNetwork, that you have routes in your VPC for your RemodeNodeNetwork, and that your on-premises network is configured correctly to allow connectivity to the EKS control plane.

If the Cilium operator is running and some of your Cilium agents are running but not all, confirm that you have available pod IPs to allocate for all nodes in your cluster. You configure the size of your allocatable Pod CIDRs when using cluster pool IPAM with clusterPoolIPv4PodCIDRList in your Cilium configuration. The per-node CIDR size is configured with the clusterPoolIPv4MaskSize setting in your Cilium configuration. See Expanding the cluster pool in the Cilium documentation for more information.

Cilium BGP is not working

If you are using Cilium BGP Control Plane to advertise your pod or service addresses to your on-premises network, you can use the following Cilium CLI commands to check if BGP is advertising the routes to your resources. For steps to install the Cilium CLI, see Install the Cilium CLI in the Cilium documentation.

If BGP is working correctly, you should your hybrid nodes with Session State established in the output. You may need to work with your networking team to identify the correct values for your environment’s Local AS, Peer AS, and Peer Address.

cilium bgp peers

cilium bgp routes

If you are using Cilium BGP to advertise the IPs of Services with type LoadBalancer, you must have the same label on both your CiliumLoadBalancerIPPool and Service, which should be used in the selector of your CiliumBGPAdvertisement. An example is shown below. Note, if you are using Cilium BGP to advertise the IPs of Services with type LoadBalancer, the BGP routes may be disrupted during Cilium agent restart. For more information, see Failure Scenarios in the Cilium documentation.

Service

kind: Service
apiVersion: v1
metadata:
  name: guestbook
  labels:
    app: guestbook
spec:
  ports:
  - port: 3000
    targetPort: http-server
  selector:
    app: guestbook
  type: LoadBalancer

CiliumLoadBalancerIPPool

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: guestbook-pool
  labels:
    app: guestbook
spec:
  blocks:
  - cidr: <CIDR to advertise>
  serviceSelector:
    matchExpressions:
      - { key: app, operator: In, values: [ guestbook ] }

CiliumBGPAdvertisement

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisements-guestbook
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: "Service"
      service:
        addresses:
          - ExternalIP
          - LoadBalancerIP
      selector:
        matchExpressions:
          - { key: app, operator: In, values: [ guestbook ] }

Calico troubleshooting

If you are having issues running Calico on hybrid nodes, see the troubleshooting steps in the Calico documentation. The sections below cover issues that may be unique to deploying Calico on hybrid nodes.

The table below summarizes the Calico components and whether they run on the node or pod network by default. If you configured Calico to use NAT for outgoing pod traffic, your on-premises network must be configured to route traffic to your on-premises node CIDR and your VPC routing tables must be configured with a route for your on-premises node CIDR with your transit gateway (TGW) or virtual private gateway (VGW) as the target. If you are not configuring Calico to use NAT for outgoing pod traffic, your on-premises network must be configured to route traffic to your on-premises pod CIDR and your VPC routing tables must be configured with a route for your on-premises pod CIDR with your transit gateway (TGW) or virtual private gateway (VGW) as the target.

Component

Network

Calico API server

Node

Calico kube controllers

Pod

Calico node agent

Node

Calico typha

Node

Calico CSI node driver

Pod

Calico operator

Node

Calico resources are scheduled or running on cordoned nodes

The Calico resources that don’t run as a DaemonSet have flexible tolerations by default that enable them to be scheduled on cordoned nodes that are not ready for scheduling or running pods. You can tighten the tolerations for the non-DaemonSet Calico resources by changing your operator installation to include the following.

installation:
  ...
  controlPlaneTolerations:
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  calicoKubeControllersDeployment:
    spec:
      template:
        spec:
          tolerations:
          - effect: NoExecute
            key: node.kubernetes.io/unreachable
            operator: Exists
            tolerationSeconds: 300
          - effect: NoExecute
            key: node.kubernetes.io/not-ready
            operator: Exists
            tolerationSeconds: 300
  typhaDeployment:
    spec:
      template:
        spec:
          tolerations:
          - effect: NoExecute
            key: node.kubernetes.io/unreachable
            operator: Exists
            tolerationSeconds: 300
          - effect: NoExecute
            key: node.kubernetes.io/not-ready
            operator: Exists
            tolerationSeconds: 300

Credentials troubleshooting

For both AWS SSM hybrid activations and AWS IAM Roles Anywhere, you can validate that credentials for the Hybrid Nodes IAM role are correctly configured on your hybrid nodes by running the following command from your hybrid nodes. Confirm the node name and Hybrid Nodes IAM Role name are what you expect.

sudo aws sts get-caller-identity

{
    "UserId": "ABCDEFGHIJKLM12345678910:<node-name>",
    "Account": "<aws-account-id>",
    "Arn": "arn:aws:sts::<aws-account-id>:assumed-role/<hybrid-nodes-iam-role/<node-name>"
}

AWS Systems Manager (SSM) troubleshooting

If you are using AWS SSM hybrid activations for your hybrid nodes credentials, be aware of the following SSM directories and artifacts that are installed on your hybrid nodes by nodeadm. For more information on the SSM agent, see Working with the SSM agent in the AWS Systems Manager User Guide.

Description Location

SSM agent

Ubuntu - /snap/amazon-ssm-agent/current/amazon-ssm-agent RHEL & AL2023 - /usr/bin/amazon-ssm-agent

SSM agent logs

/var/log/amazon/ssm

AWS credentials

/root/.aws/credentials

SSM Setup CLI

/opt/ssm/ssm-setup-cli

Restarting the SSM agent

Some issues can be resolved by restarting the SSM agent. You can use the commands below to restart it.

AL2023 and other operating systems

systemctl restart amazon-ssm-agent

Ubuntu

systemctl restart snap.amazon-ssm-agent.amazon-ssm-agent

Check connectivity to SSM endpoints

Confirm you can connect to the SSM endpoints from your hybrid nodes. For a list of the SSM endpoints, see AWS Systems Manager endpoints and quotas. Replace us-west-2 in the command below with the AWS-Region for your AWS SSM hybrid activation.

ping ssm.us-west-2.amazonaws.com

View connection status of registered SSM instances

You can check the connection status of the instances that are registered with SSM hybrid activations with the following CLI command. Replace the machine ID with the machine ID of your instance.

aws ssm get-connection-status --target mi-012345678abcdefgh

SSM Setup CLI checksum mismatch

When running nodeadm install if you see an issue with the ssm-setup-cli checksum mismatch you should confirm there are not older existing SSM installations on your host. If there are older SSM installations on your host, remove them and re-run nodeadm install to resolve the issue.

Failed to perform agent-installation/on-prem registration: error while verifying installed ssm-setup-cli checksum: checksum mismatch with latest ssm-setup-cli.

SSM InvalidActivation

If you see an error registering your instance with AWS SSM, confirm the region, activationCode, and activationId in your nodeConfig.yaml are correct. The AWS-Region for your EKS cluster must match the region of your SSM hybrid activation. If these values are misconfigured, you may see an error similar to the following.

ERROR Registration failed due to error registering the instance with AWS SSM. InvalidActivation

SSM ExpiredTokenException: The security token included in the request is expired

If the SSM agent is not able to refresh credentials, you may see an ExpiredTokenException. In this scenario, if you are able to connect to the SSM endpoints from your hybrid nodes, you may need to restart the SSM agent to force a credential refresh.

"msg":"Command failed","error":"operation error SSM: DescribeInstanceInformation, https response error StatusCode: 400, RequestID: eee03a9e-f7cc-470a-9647-73d47e4cf0be, api error ExpiredTokenException: The security token included in the request is expired"

SSM error in running register machine command

If you see an error registering the machine with SSM, you may need to re-run nodeadm install to make sure all of the SSM dependencies are properly installed.

"error":"running register machine command: , error: fork/exec /opt/aws/ssm-setup-cli: no such file or directory"

SSM ActivationExpired

When running nodeadm init, if you see an error registering the instance with SSM due to an expired activation, you need to create a new SSM hybrid activation, update your nodeConfig.yaml with the activationCode and activationId of your new SSM hybrid activation, and re-run nodeadm init.

"msg":"Command failed","error":"SSM activation expired. Please use a valid activation"

ERROR Registration failed due to error registering the instance with AWS SSM. ActivationExpired

SSM failed to refresh cached credentials

If you see a failure to refresh cached credentials, the /root/.aws/credentials file may have been deleted on your host. First check your SSM hybrid activation and ensure it is active and your hybrid nodes are configured correctly to use the activation. Check the SSM agent logs at /var/log/amazon/ssm and re-run the nodeadm init command once you have resolved the issue on the SSM side.

"Command failed","error":"operation error SSM: DescribeInstanceInformation, get identity: get credentials: failed to refresh cached credentials"

Clean up SSM

To remove the SSM agent from your host, you can run the following commands.

dnf remove -y amazon-ssm-agent
sudo apt remove --purge amazon-ssm-agent
snap remove amazon-ssm-agent
rm -rf /var/lib/amazon/ssm/Vault/Store/RegistrationKey

AWS IAM Roles Anywhere troubleshooting

If you are using AWS IAM Roles Anywhere for your hybrid nodes credentials, be aware of the following directories and artifacts that are installed on your hybrid nodes by nodeadm. For more information on the troubleshooting IAM Roles Anywhere, see Troubleshooting AWS IAM Roles Anywhere identity and access in the AWS IAM Roles Anywhere User Guide.

Description Location

IAM Roles Anywhere CLI

/usr/local/bin/aws_signing_helper

Default certificate location and name

/etc/iam/pki/server.pem

Default key location and name

/etc/iam/pki/server.key

IAM Roles Anywhere failed to refresh cached credentials

If you see a failure to refresh cached credentials, review the contents of /etc/aws/hybrid/config and confirm that IAM Roles Anywhere was configured correctly in your nodeadm configuration. Confirm that /etc/iam/pki exists. Each node must have a unique certificate and key. By default, when using IAM Roles Anywhere as the credential provider, nodeadm uses /etc/iam/pki/server.pem for the certificate location and name, and /etc/iam/pki/server.key for the private key. You may need to create the directories before placing the certificates and keys in the directories with sudo mkdir -p /etc/iam/pki. You can verify the content of your certificate with the command below.

openssl x509 -text -noout -in server.pem

open /etc/iam/pki/server.pem: no such file or directory
could not parse PEM data
Command failed {"error": "... get identity: get credentials: failed to refresh cached credentials, process provider error: error in credential_process: exit status 1"}

IAM Roles Anywhere not authorized to perform sts:AssumeRole

In the kubelet logs, if you see an access denied issue for the sts:AssumeRole operation when using IAM Roles Anywhere, check the trust policy of your Hybrid Nodes IAM role to confirm the IAM Roles Anywhere service principal is allowed to assume the Hybrid Nodes IAM Role. Additionally confirm that the trust anchor ARN is configured properly in your Hybrid Nodes IAM role trust policy and that your Hybrid Nodes IAM role is added to your IAM Roles Anywhere profile.

could not get token: AccessDenied: User: ... is not authorized to perform: sts:AssumeRole on resource: ...

IAM Roles Anywhere not authorized to set roleSessionName

In the kubelet logs, if you see an access denied issue for setting the roleSessionName, confirm you have set acceptRoleSessionName to true for your IAM Roles Anywhere profile.

AccessDeniedException: Not authorized to set roleSessionName

Operating system troubleshooting

RHEL

Entitlement or subscription manager registration failures

If you are running nodeadm install and encounter a failure to install the hybrid nodes dependencies due to entitlement registration issues, ensure you have properly set your Red Hat username and password on your host.

This system is not registered with an entitlement server

GLIBC not found

If you are using Ubuntu for your operating system and IAM Roles Anywhere for your credential provider with hybrid nodes and see an issue with GLIBC not found, you can install that dependency manually to resolve the issue.

GLIBC_2.32 not found (required by /usr/local/bin/aws_signing_helper)

Run the following commands to install the dependency:

ldd --version
sudo apt update && apt install libc6
sudo apt install glibc-source

10. Store application data for your cluster

This chapter covers storage options for Amazon EKS clusters.

This chapter covers storage options for Amazon EKS clusters.

[[Topic List]]

10.1. Store `Kubernetes` volumes with Amazon EBS

The Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver manages the lifecycle of Amazon EBS volumes as storage for Kubernetes Volumes.

New: Amazon EKS Auto Mode automates routine tasks for block storage. Learn how to sample-storage-workload.title.

The Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver manages the lifecycle of Amazon EBS volumes as storage for the Kubernetes Volumes that you create. The Amazon EBS CSI driver makes Amazon EBS volumes for these types of Kubernetes volumes: generic ephemeral volumes and persistent volumes.

10.1.1. Considerations

You do not need to install the Amazon EBS CSI controller on EKS Auto Mode clusters.
You can’t mount Amazon EBS volumes to Fargate Pods.
You can run the Amazon EBS CSI controller on Fargate nodes, but the Amazon EBS CSI node DaemonSet can only run on Amazon EC2 instances.
Amazon EBS volumes and the Amazon EBS CSI driver are not compatible with Amazon EKS Hybrid Nodes.
Support will be provided for the latest add-on version and one prior version. Bugs or vulnerabilities found in the latest version will be backported to the previous release in a new minor version.

To use the snapshot functionality of the Amazon EBS CSI driver, you must first install the CSI snapshot controller. For more information, see csi-snapshot-controller.title.

10.1.2. Prerequisites

An existing cluster. To see the required platform version, run the following command.
```
aws eks describe-addon-versions --addon-name aws-ebs-csi-driver
```
An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.
If you’re using a cluster wide restricted PodSecurityPolicy, make sure that the add-on is granted sufficient permissions to be deployed. For the permissions required by each add-on Pod, see the relevant add-on manifest definition on GitHub.

10.1.3. Step 1: Create an IAM role

The Amazon EBS CSI plugin requires IAM permissions to make calls to AWS APIs on your behalf. If you don’t do these steps, attempting to install the add-on and running kubectl describe pvc will show failed to provision volume with StorageClass along with a could not create volume in EC2: UnauthorizedOperation error. For more information, see Set up driver permission on GitHub.

Pods will have access to the permissions that are assigned to the IAM role unless you block access to IMDS. For more information, see security-best-practices.title.

The following procedure shows you how to create an IAM role and attach the AWS managed policy to it. To implement this procedure, you can use one of these tools:

eksctl
consolelong
AWS CLI

The specific steps in this procedure are written for using the driver as an Amazon EKS add-on. Different steps are needed to use the driver as a self-managed add-on. For more information, see Set up driver permissions on GitHub.

`eksctl`

Create an IAM role and attach a policy. AWS maintains an AWS managed policy or you can create your own custom policy. You can create an IAM role and attach the AWS managed policy with the following command. Replace my-cluster with the name of your cluster. The command deploys an AWS CloudFormation stack that creates an IAM role and attaches the IAM policy to it. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.
```
eksctl create iamserviceaccount \
        --name ebs-csi-controller-sa \
        --namespace kube-system \
        --cluster my-cluster \
        --role-name AmazonEKS_EBS_CSI_DriverRole \
        --role-only \
        --attach-policy-arn region.arniam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
        --approve
```

If you use a custom KMS key for encryption on your Amazon EBS volumes, customize the IAM role as needed. For example, do the following:

Copy and paste the following code into a new kms-key-for-encryption-on-ebs.json file. Replace custom-key-arn with the custom KMS key ARN.

{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
          ],
          "Resource": ["custom-key-arn"],
          "Condition": {
            "Bool": {
              "kms:GrantIsForAWSResource": "true"
            }
          }
        },
        {
          "Effect": "Allow",
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
          ],
          "Resource": ["custom-key-arn"]
        }
      ]
    }

Create the policy. You can change KMS_Key_For_Encryption_On_EBS_Policy to a different name. However, if you do, make sure to change it in later steps, too.
```
aws iam create-policy \
      --policy-name KMS_Key_For_Encryption_On_EBS_Policy \
      --policy-document file://kms-key-for-encryption-on-ebs.json
```
Attach the IAM policy to the role with the following command. Replace 111122223333 with your account ID. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.
```
aws iam attach-role-policy \
      --policy-arn region.arniam::111122223333:policy/KMS_Key_For_Encryption_On_EBS_Policy \
      --role-name AmazonEKS_EBS_CSI_DriverRole
```

`consolelong`

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.
On the Select trusted entity page, do the following:
1. In the Trusted entity type section, choose Web identity.
2. For Identity provider, choose the OpenID Connect provider URL for your cluster (as shown under Overview in Amazon EKS).
3. For Audience, choose sts.amazonaws.com.
4. Choose Next.
On the Add permissions page, do the following:
1. In the Filter policies box, enter AmazonEBSCSIDriverPolicy.
2. Select the check box to the left of the AmazonEBSCSIDriverPolicy returned in the search.
3. Choose Next.
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKS_EBS_CSI_DriverRole.
2. Under Add tags (Optional), add metadata to the role by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM resources in the IAM User Guide.
3. Choose Create role.
After the role is created, choose the role in the console to open it for editing.
Choose the Trust relationships tab, and then choose Edit trust policy.
Find the line that looks similar to the following line:
```
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
```
Add a comma to the end of the previous line, and then add the following line after the previous line. Replace region-code with the AWS Region that your cluster is in. Replace EXAMPLED539D4633E53DE1B71EXAMPLE with your cluster’s OIDC provider ID.
```
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
```
Choose Update policy to finish.
If you use a custom KMS key for encryption on your Amazon EBS volumes, customize the IAM role as needed. For example, do the following:
1. In the left navigation pane, choose Policies.
2. On the Policies page, choose Create Policy.
3. On the Create policy page, choose the JSON tab.
4. Copy and paste the following code into the editor, replacing custom-key-arn with the custom KMS key ARN.
  { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant" ], "Resource": ["custom-key-arn"], "Condition": { "Bool": { "kms:GrantIsForAWSResource": "true" } } }, { "Effect": "Allow", "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ], "Resource": ["custom-key-arn"] } ] }
5. Choose Next: Tags.
6. On the Add tags (Optional) page, choose Next: Review.
7. For Name, enter a unique name for your policy (for example, KMS_Key_For_Encryption_On_EBS_Policy).
8. Choose Create policy.
9. In the left navigation pane, choose Roles.
10. Choose the AmazonEKS_EBS_CSI_DriverRole in the console to open it for editing.
11. From the Add permissions dropdown list, choose Attach policies.
12. In the Filter policies box, enter KMS_Key_For_Encryption_On_EBS_Policy.
13. Select the check box to the left of the KMS_Key_For_Encryption_On_EBS_Policy that was returned in the search.
14. Choose Attach policies.

`AWS` CLI

View your cluster’s OIDC provider URL. Replace my-cluster with your cluster name. If the output from the command is None, review the Prerequisites.
```
aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text
```
An example output is as follows.
```
https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
```

Create the IAM role, granting the AssumeRoleWithWebIdentity action.

Copy the following contents to a file that’s named aws-ebs-csi-driver-trust-policy.json. Replace 111122223333 with your account ID. Replace EXAMPLED539D4633E53DE1B71EXAMPLE and region-code with the values returned in the previous step. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.

{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "region.arniam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
              "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
            }
          }
        }
      ]
    }

Create the role. You can change AmazonEKS_EBS_CSI_DriverRole to a different name. If you change it, make sure to change it in later steps.

aws iam create-role \
      --role-name AmazonEKS_EBS_CSI_DriverRole \
      --assume-role-policy-document file://"aws-ebs-csi-driver-trust-policy.json"

Attach a policy. AWS maintains an AWS managed policy or you can create your own custom policy. Attach the AWS managed policy to the role with the following command. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.
```
aws iam attach-role-policy \
      --policy-arn region.arniam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
      --role-name AmazonEKS_EBS_CSI_DriverRole
```

If you use a custom KMS key for encryption on your Amazon EBS volumes, customize the IAM role as needed. For example, do the following:

Copy and paste the following code into a new kms-key-for-encryption-on-ebs.json file. Replace custom-key-arn with the custom KMS key ARN.

{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
          ],
          "Resource": ["custom-key-arn"],
          "Condition": {
            "Bool": {
              "kms:GrantIsForAWSResource": "true"
            }
          }
        },
        {
          "Effect": "Allow",
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
          ],
          "Resource": ["custom-key-arn"]
        }
      ]
    }

Create the policy. You can change KMS_Key_For_Encryption_On_EBS_Policy to a different name. However, if you do, make sure to change it in later steps, too.
```
aws iam create-policy \
      --policy-name KMS_Key_For_Encryption_On_EBS_Policy \
      --policy-document file://kms-key-for-encryption-on-ebs.json
```
Attach the IAM policy to the role with the following command. Replace 111122223333 with your account ID. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.
```
aws iam attach-role-policy \
      --policy-arn region.arniam::111122223333:policy/KMS_Key_For_Encryption_On_EBS_Policy \
      --role-name AmazonEKS_EBS_CSI_DriverRole
```

Now that you have created the Amazon EBS CSI driver IAM role, you can continue to the next section. When you deploy the add-on with this IAM role, it creates and is configured to use a service account that’s named ebs-csi-controller-sa. The service account is bound to a Kubernetes clusterrole that’s assigned the required Kubernetes permissions.

10.1.4. Step 2: Get the Amazon EBS CSI driver

We recommend that you install the Amazon EBS CSI driver through the Amazon EKS add-on to improve security and reduce the amount of work. To add an Amazon EKS add-on to your cluster, see creating-an-add-on.title. For more information about add-ons, see eks-add-ons.title.

Before adding the Amazon EBS driver as an Amazon EKS add-on, confirm that you don’t have a self-managed version of the driver installed on your cluster. If so, see Uninstalling a self-managed Amazon EBS CSI driver on GitHub.

Alternatively, if you want a self-managed installation of the Amazon EBS CSI driver, see Installation on GitHub.

10.1.5. Step 3: Deploy a sample application

You can deploy a variety of sample apps and modify them as needed. For more information, see Kubernetes Examples on GitHub.

10.2. Amazon EBS CSI migration frequently asked questions

The Amazon EBS container storage interface migration feature is enabled by default on Amazon EKS 1.23 and later clusters. Learn answers to frequently asked questions about the feature and how it works with 1.23 and later clusters.

If you have Pods running on a version 1.22 or earlier cluster, then you must install the Amazon EBS CSI driver (see ebs-csi.title) before updating your cluster to version 1.23 to avoid service interruption.

The Amazon EBS container storage interface (CSI) migration feature moves responsibility for handling storage operations from the Amazon EBS in-tree EBS storage provisioner to the Amazon EBS CSI driver (see ebs-csi.title).

10.2.1. What are CSI drivers?

CSI drivers:

Replace the Kubernetes "in-tree" storage drivers that exist in the Kubernetes project source code.
Work with storage providers, such as Amazon EBS.
Provide a simplified plugin model that make it easier for storage providers like AWS to release features and maintain support without depending on the Kubernetes release cycle.

For more information, see Introduction in the Kubernetes CSI documentation.

10.2.2. What is CSI migration?

The Kubernetes CSI Migration feature moves responsibility for handling storage operations from the existing in-tree storage plugins, such as kubernetes.io/aws-ebs, to corresponding CSI drivers. Existing StorageClass, PersistentVolume and PersistentVolumeClaim (PVC) objects continue to work, as long as the corresponding CSI driver is installed. When the feature is enabled:

Existing workloads that utilize PVCs continue to function as they always have.
Kubernetes passes control of all storage management operations to CSI drivers.

For more information, see Kubernetes1.23: Kubernetes In-Tree to CSI Volume Migration Status Update on the Kubernetes blog.

To help you migrate from the in-tree plugin to CSI drivers, the CSIMigration and CSIMigrationAWS flags are enabled by default on Amazon EKS version 1.23 and later clusters. These flags enable your cluster to translate the in-tree APIs to their equivalent CSI APIs. These flags are set on the Kubernetes control plane managed by Amazon EKS and in the kubelet settings configured in Amazon EKS optimized AMIs. If you have Pods using Amazon EBS volumes in your cluster, you must install the Amazon EBS CSI driver before updating your cluster to version 1.23. If you don’t, volume operations such as provisioning and mounting might not work as expected. For more information, see ebs-csi.title.

The in-tree StorageClass provisioner is named kubernetes.io/aws-ebs. The Amazon EBS CSI StorageClass provisioner is named ebs.csi.aws.com.

10.2.3. Can I mount `kubernetes.io/aws-ebs StorageClass` volumes in version `1.23` and later clusters?

Yes, as long as the Amazon EBS CSI driver is installed. For newly created version 1.23 and later clusters, we recommend installing the Amazon EBS CSI driver as part of your cluster creation process. We also recommend only using StorageClasses based on the ebs.csi.aws.com provisioner.

If you’ve updated your cluster control plane to version 1.23 and haven’t yet updated your nodes to 1.23, then the CSIMigration and CSIMigrationAWS kubelet flags aren’t enabled. In this case, the in-tree driver is used to mount kubernetes.io/aws-ebs based volumes. The Amazon EBS CSI driver must still be installed however, to ensure that Pods using kubernetes.io/aws-ebs based volumes can be scheduled. The driver is also required for other volume operations to succeed.

10.2.4. Can I provision `kubernetes.io/aws-ebs StorageClass` volumes on Amazon EKS `1.23` and later clusters?

Yes, as long as the Amazon EBS CSI driver is installed.

10.2.5. Will the `kubernetes.io/aws-ebs StorageClass` provisioner ever be removed from Amazon EKS?

The kubernetes.io/aws-ebs StorageClass provisioner and awsElasticBlockStore volume type are no longer supported, but there are no plans to remove them. These resources are treated as a part of the Kubernetes API.

10.2.6. How do I install the Amazon EBS CSI driver?

We recommend installing the Amazon EBS CSI driver Amazon EKS add-on. When an update is required to the Amazon EKS add-on, you initiate the update and Amazon EKS updates the add-on for you. If you want to manage the driver yourself, you can install it using the open source Helm chart.

The Kubernetes in-tree Amazon EBS driver runs on the Kubernetes control plane. It uses IAM permissions assigned to the Amazon EKS cluster IAM role to provision Amazon EBS volumes. The Amazon EBS CSI driver runs on nodes. The driver needs IAM permissions to provision volumes. For more information, see csi-iam-role.title.

10.2.7. How can I check whether the Amazon EBS CSI driver is installed in my cluster?

To determine whether the driver is installed on your cluster, run the following command:

kubectl get csidriver ebs.csi.aws.com

To check if that installation is managed by Amazon EKS, run the following command:

aws eks list-addons --cluster-name my-cluster

10.2.8. Will Amazon EKS prevent a cluster update to version `1.23` if I haven’t already installed the Amazon EBS CSI driver?

No.

10.2.9. What if I forget to install the Amazon EBS CSI driver before I update my cluster to version 1.23? Can I install the driver after updating my cluster?

Yes, but volume operations requiring the Amazon EBS CSI driver will fail after your cluster update until the driver is installed.

10.2.10. What is the default `StorageClass` applied in newly created Amazon EKS version `1.23` and later clusters?

The default StorageClass behavior remains unchanged. With each new cluster, Amazon EKS applies a kubernetes.io/aws-ebs based StorageClass named gp2. We don’t plan to ever remove this StorageClass from newly created clusters. Separate from the cluster default StorageClass, if you create an ebs.csi.aws.com based StorageClass without specifying a volume type, the Amazon EBS CSI driver will default to using gp3.

10.2.11. Will Amazon EKS make any changes to `StorageClasses` already present in my existing cluster when I update my cluster to version `1.23`?

No.

10.2.12. How do I migrate a persistent volume from the `kubernetes.io/aws-ebs` `StorageClass` to `ebs.csi.aws.com` using snapshots?

To migrate a persistent volume, see Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes on the AWS blog.

10.2.13. How do I modify an Amazon EBS volume using annotations?

Starting with aws-ebs-csi-driver v1.19.0-eksbuild.2, you can modify Amazon EBS volumes using annotations within each PersistentVolumeClaim (PVC). The new volume modification feature is implemented as an additional sidecar, called volumemodifier. For more information, see Simplifying Amazon EBS volume migration and modification on Kubernetes using the EBS CSI Driver on the AWS blog.

10.2.14. Is migration supported for Windows workloads?

Yes. If you’re installing the Amazon EBS CSI driver using the open source Helm chart, set node.enableWindows to true. This is set by default if installing the Amazon EBS CSI driver as an Amazon EKS add-on. When creating StorageClasses, set the fsType to a Windows file system, such as ntfs. Volume operations for Windows workloads are then migrated to the Amazon EBS CSI driver the same as they are for Linux workloads.

10.3. Store an elastic file system with Amazon EFS

The Amazon EFS Container Storage Interface (CSI) driver provides a CSI interface that allows Kubernetes clusters running on AWS to manage the lifecycle of Amazon EFS file systems.

Amazon Elastic File System (Amazon EFS) provides serverless, fully elastic file storage so that you can share file data without provisioning or managing storage capacity and performance. The Amazon EFS Container Storage Interface (CSI) driver provides a CSI interface that allows Kubernetes clusters running on AWS to manage the lifecycle of Amazon EFS file systems. This topic shows you how to deploy the Amazon EFS CSI driver to your Amazon EKS cluster.

10.3.1. Considerations

The Amazon EFS CSI driver isn’t compatible with Windows-based container images.
You can’t use dynamic provisioning for persistent volumes with Fargate nodes, but you can use static provisioning.
Dynamic provisioning requires 1.2 or later of the driver. You can use static provisioning for persistent volumes using version 1.1 of the driver on any supported Amazon EKS cluster version (see kubernetes-versions.title).
Version 1.3.2 or later of this driver supports the Arm64 architecture, including Amazon EC2 Graviton-based instances.
Version 1.4.2 or later of this driver supports using FIPS for mounting file systems.
Take note of the resource quotas for Amazon EFS. For example, there’s a quota of 1000 access points that can be created for each Amazon EFS file system. For more information, see Amazon EFS resource quotas that you cannot change.
Starting in version 2.0.0, this driver switched from using stunnel to efs-proxy for TLS connections. When efs-proxy is used, it will open a number of threads equal to one plus the number of cores for the node it’s running on.
The Amazon EFS CSI driver isn’t compatible with Amazon EKS Hybrid Nodes.

10.3.2. Prerequisites

An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.

10.3.3. Step 1: Create an IAM role

The Amazon EFS CSI driver requires IAM permissions to interact with your file system. Create an IAM role and attach the required AWS managed policy to it. To implement this procedure, you can use one of these tools:

eksctl
consolelong
AWS CLI

The specific steps in this procedure are written for using the driver as an Amazon EKS add-on. For details on self-managed installations, see Set up driver permission on GitHub.

`eksctl`

Run the following commands to create an IAM role with eksctl. Replace my-cluster with your cluster name and AmazonEKS_EFS_CSI_DriverRole with the name for your role.

export cluster_name=my-cluster
export role_name=AmazonEKS_EFS_CSI_DriverRole
eksctl create iamserviceaccount \
    --name efs-csi-controller-sa \
    --namespace kube-system \
    --cluster $cluster_name \
    --role-name $role_name \
    --role-only \
    --attach-policy-arn region.arniam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
    --approve
TRUST_POLICY=$(aws iam get-role --role-name $role_name --query 'Role.AssumeRolePolicyDocument' | \
    sed -e 's/efs-csi-controller-sa/efs-csi-*/' -e 's/StringEquals/StringLike/')
aws iam update-assume-role-policy --role-name $role_name --policy-document "$TRUST_POLICY"

`consolelong`

Run the following to create an IAM role with consolelong.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.
On the Select trusted entity page, do the following:
1. In the Trusted entity type section, choose Web identity.
2. For Identity provider, choose the OpenID Connect provider URL for your cluster (as shown under Overview in Amazon EKS).
3. For Audience, choose sts.amazonaws.com.
4. Choose Next.
On the Add permissions page, do the following:
1. In the Filter policies box, enter AmazonEFSCSIDriverPolicy.
2. Select the check box to the left of the AmazonEFSCSIDriverPolicy returned in the search.
3. Choose Next.
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKS_EFS_CSI_DriverRole.
2. Under Add tags (Optional), add metadata to the role by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM resources in the IAM User Guide.
3. Choose Create role.
After the role is created, choose the role in the console to open it for editing.
Choose the Trust relationships tab, and then choose Edit trust policy.
Find the line that looks similar to the following line:
```
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
```
Add the following line above the previous line. Replace region-code with the AWS Region that your cluster is in. Replace EXAMPLED539D4633E53DE1B71EXAMPLE with your cluster’s OIDC provider ID.
```
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:efs-csi-*",
```
Modify the Condition operator from "StringEquals" to "StringLike".
Choose Update policy to finish.

`AWS` CLI

Run the following commands to create an IAM role with AWS CLI.

View your cluster’s OIDC provider URL. Replace my-cluster with your cluster name. If the output from the command is None, review the Prerequisites.
```
aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text
```
An example output is as follows.
```
https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
```

Create the IAM role that grants the AssumeRoleWithWebIdentity action.

Copy the following contents to a file named aws-efs-csi-driver-trust-policy.json``. Replace 111122223333 with your account ID. Replace EXAMPLED539D4633E53DE1B71EXAMPLE and region-code with the values returned in the previous step. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "region.arniam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringLike": {
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:efs-csi-*",
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Create the role. You can change AmazonEKS_EFS_CSI_DriverRole to a different name, but if you do, make sure to change it in later steps too.

aws iam create-role \
  --role-name AmazonEKS_EFS_CSI_DriverRole \
  --assume-role-policy-document file://"aws-efs-csi-driver-trust-policy.json"

Attach the required AWS managed policy to the role with the following command. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.
```
aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
  --role-name AmazonEKS_EFS_CSI_DriverRole
```

10.3.4. Step 2: Get the Amazon EFS CSI driver

We recommend that you install the Amazon EFS CSI driver through the Amazon EKS add-on. To add an Amazon EKS add-on to your cluster, see creating-an-add-on.title. For more information about add-ons, see eks-add-ons.title. If you’re unable to use the Amazon EKS add-on, we encourage you to submit an issue about why you can’t to the Containers roadmap GitHub repository.

Alternatively, if you want a self-managed installation of the Amazon EFS CSI driver, see Installation on GitHub.

10.3.5. Step 3: Create an Amazon EFS file system

This step isn’t needed for AWS Fargate. A Pod running on Fargate automatically mounts an Amazon EFS file system, without needing manual driver installation steps.

To create an Amazon EFS file system, see Create an Amazon EFS file system for Amazon EKS on GitHub.

10.3.6. Step 4: Deploy a sample application

You can deploy a variety of sample apps and modify them as needed. For more information, see Examples on GitHub.

10.4. Store high-performance apps with FSx for Lustre

The FSx for Lustre Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the lifecycle of FSx for Lustre file systems.

The FSx for Lustre Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the lifecycle of FSx for Lustre file systems. For more information, see the FSx for Lustre User Guide.

This topic shows you how to deploy the FSx for Lustre CSI driver to your Amazon EKS cluster and verify that it works. We recommend using the latest version of the driver. For available versions, see CSI Specification Compatibility Matrix on GitHub.

The driver isn’t supported on Fargate or Amazon EKS Hybrid Nodes.

For detailed descriptions of the available parameters and complete examples that demonstrate the driver’s features, see the FSx for Lustre Container Storage Interface (CSI) driver project on GitHub.

You must have:

Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
Version 0.199.0 or later of the eksctl command line tool installed on your device or AWS CloudShell. To install or update eksctl, see Installation in the eksctl documentation.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.

The following procedures help you create a simple test cluster with the FSx for Lustre CSI driver so that you can see how it works. We don’t recommend using the testing cluster for production workloads. For this tutorial, we recommend using the example values, except where it’s noted to replace them. You can replace any example value when completing the steps for your production cluster. We recommend completing all steps in the same terminal because variables are set and used throughout the steps and won’t exist in different terminals.

Set a few variables to use in the remaining steps. Replace my-csi-fsx-cluster with the name of the test cluster you want to createand region-code with the AWS Region that you want to create your test cluster in.
```
export cluster_name=my-csi-fsx-cluster
export region_code=region-code
```

Create a test cluster.

eksctl create cluster \
  --name $cluster_name \
  --region $region_code \
  --with-oidc \
  --ssh-access \
  --ssh-public-key my-key

Cluster provisioning takes several minutes. During cluster creation, you’ll see several lines of output. The last line of output is similar to the following example line.

[✓]  EKS cluster "my-csi-fsx-cluster" in "region-code" region is ready

Create a Kubernetes service account for the driver and attach the AmazonFSxFullAccess AWS-managed policy to the service account with the following command. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.

eksctl create iamserviceaccount \
  --name fsx-csi-controller-sa \
  --namespace kube-system \
  --cluster $cluster_name \
  --attach-policy-arn region.arniam::aws:policy/AmazonFSxFullAccess \
  --approve \
  --role-name AmazonEKSFSxLustreCSIDriverFullAccess \
  --region $region_code

You’ll see several lines of output as the service account is created. The last lines of output are similar to the following.

[ℹ]  1 task: {
    2 sequential sub-tasks: {
        create IAM role for serviceaccount "kube-system/fsx-csi-controller-sa",
        create serviceaccount "kube-system/fsx-csi-controller-sa",
    } }
[ℹ]  building iamserviceaccount stack "eksctl-my-csi-fsx-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa"
[ℹ]  deploying stack "eksctl-my-csi-fsx-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa"
[ℹ]  waiting for CloudFormation stack "eksctl-my-csi-fsx-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa"
[ℹ]  created serviceaccount "kube-system/fsx-csi-controller-sa"

Note the name of the AWS CloudFormation stack that was deployed. In the previous example output, the stack is named eksctl-my-csi-fsx-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa.

Deploy the driver with the following command. Replace release-X.XX with your desired branch. The master branch isn’t supported because it may contain upcoming features incompatible with the currently released stable version of the driver. We recommend using the latest released version. For a list of branches, see aws-fsx-csi-driver Branches on GitHub.

You can view the content being applied in aws-fsx-csi-driver/deploy/kubernetes/overlays/stable on GitHub.

kubectl apply -k "github.com/kubernetes-sigs/aws-fsx-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-X.XX"

An example output is as follows.

serviceaccount/fsx-csi-controller-sa created
serviceaccount/fsx-csi-node-sa created
clusterrole.rbac.authorization.k8s.io/fsx-csi-external-provisioner-role created
clusterrole.rbac.authorization.k8s.io/fsx-external-resizer-role created
clusterrolebinding.rbac.authorization.k8s.io/fsx-csi-external-provisioner-binding created
clusterrolebinding.rbac.authorization.k8s.io/fsx-csi-resizer-binding created
deployment.apps/fsx-csi-controller created
daemonset.apps/fsx-csi-node created
csidriver.storage.k8s.io/fsx.csi.aws.com created

Note the ARN for the role that was created. If you didn’t note it earlier and don’t have it available anymore in the AWS CLI output, you can do the following to see it in the consolelong.
1. Open the AWS CloudFormation console.
2. Ensure that the console is set to the AWS Region that you created your IAM role in and then select Stacks.
3. Select the stack named eksctl-my-csi-fsx-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa.
4. Select the Outputs tab. The Role1 ARN is listed on the Outputs (1) page.
Patch the driver deployment to add the service account that you created earlier with the following command. Replace the ARN with the ARN that you noted. Replace 111122223333 with your account ID. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.
```
kubectl annotate serviceaccount -n kube-system fsx-csi-controller-sa \
  eks.amazonaws.com/role-arn=region.arniam::111122223333:role/AmazonEKSFSxLustreCSIDriverFullAccess --overwrite=true
```
An example output is as follows.
```
serviceaccount/fsx-csi-controller-sa annotated
```

This procedure uses the FSx for Lustre Container Storage Interface (CSI) driverGitHub repository to consume a dynamically-provisioned FSx for Lustre volume.

Note the security group for your cluster. You can see it in the consolelong under the Networking section or by using the following AWS CLI command.
```
aws eks describe-cluster --name $cluster_name --query cluster.resourcesVpcConfig.clusterSecurityGroupId
```
Create a security group for your Amazon FSx file system according to the criteria shown in Amazon VPC Security Groups in the Amazon FSx for Lustre User Guide. For the VPC, select the VPC of your cluster as shown under the Networking section. For "the security groups associated with your Lustre clients", use your cluster security group. You can leave the outbound rules alone to allow All traffic.

Download the storage class manifest with the following command.

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/storageclass.yaml

Edit the parameters section of the storageclass.yaml file. Replace every example value with your own values.
```
parameters:
  subnetId: subnet-0eabfaa81fb22bcaf
  securityGroupIds: sg-068000ccf82dfba88
  deploymentType: PERSISTENT_1
  automaticBackupRetentionDays: "1"
  dailyAutomaticBackupStartTime: "00:00"
  copyTagsToBackups: "true"
  perUnitStorageThroughput: "200"
  dataCompressionType: "NONE"
  weeklyMaintenanceStartTime: "7:09:00"
  fileSystemTypeVersion: "2.12"
```
- subnetId – The subnet ID that the Amazon FSx for Lustre file system should be created in. Amazon FSx for Lustre isn’t supported in all Availability Zones. Open the Amazon FSx for Lustre console at https://console.aws.amazon.com/fsx/ to confirm that the subnet that you want to use is in a supported Availability Zone. The subnet can include your nodes, or can be a different subnet or VPC:
  - You can check for the node subnets in the consolelong by selecting the node group under the Compute section.
  - If the subnet that you specify isn’t the same subnet that you have nodes in, then your VPCs must be connected, and you must ensure that you have the necessary ports open in your security groups.
- securityGroupIds – The ID of the security group you created for the file system.
- deploymentType (optional) – The file system deployment type. Valid values are SCRATCH_1, SCRATCH_2, PERSISTENT_1, and PERSISTENT_2. For more information about deployment types, see Create your Amazon FSx for Lustre file system.
- other parameters (optional) – For information about the other parameters, see Edit StorageClass on GitHub.

Create the storage class manifest.

kubectl apply -f storageclass.yaml

An example output is as follows.

storageclass.storage.k8s.io/fsx-sc created

Download the persistent volume claim manifest.

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/claim.yaml

(Optional) Edit the claim.yaml file. Change 1200Gi to one of the following increment values, based on your storage requirements and the deploymentType that you selected in a previous step.
```
storage: 1200Gi
```
- SCRATCH_2 and PERSISTENT – 1.2 TiB, 2.4 TiB, or increments of 2.4 TiB over 2.4 TiB.
- SCRATCH_1 – 1.2 TiB, 2.4 TiB, 3.6 TiB, or increments of 3.6 TiB over 3.6 TiB.
Create the persistent volume claim.
```
kubectl apply -f claim.yaml
```
An example output is as follows.
```
persistentvolumeclaim/fsx-claim created
```

Confirm that the file system is provisioned.

kubectl describe pvc

An example output is as follows.

Name:          fsx-claim
Namespace:     default
StorageClass:  fsx-sc
Status:        Bound
[...]

The Status may show as Pending for 5-10 minutes, before changing to Bound. Don’t continue with the next step until the Status is Bound. If the Status shows Pending for more than 10 minutes, use warning messages in the Events as reference for addressing any problems.

Deploy the sample application.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/pod.yaml

Verify that the sample application is running.

kubectl get pods

An example output is as follows.

NAME      READY   STATUS              RESTARTS   AGE
fsx-app   1/1     Running             0          8s

Verify that the file system is mounted correctly by the application.

kubectl exec -ti fsx-app -- df -h

An example output is as follows.

Filesystem                   Size  Used Avail Use% Mounted on
overlay                       80G  4.0G   77G   5% /
tmpfs                         64M     0   64M   0% /dev
tmpfs                        3.8G     0  3.8G   0% /sys/fs/cgroup
192.0.2.0@tcp:/abcdef01      1.1T  7.8M  1.1T   1% /data
/dev/nvme0n1p1                80G  4.0G   77G   5% /etc/hosts
shm                           64M     0   64M   0% /dev/shm
tmpfs                        6.9G   12K  6.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                        3.8G     0  3.8G   0% /proc/acpi
tmpfs                        3.8G     0  3.8G   0% /sys/firmware

Verify that data was written to the FSx for Lustre file system by the sample app.
```
kubectl exec -it fsx-app -- ls /data
```
An example output is as follows.
```
out.txt
```
This example output shows that the sample app successfully wrote the out.txt file to the file system.

Before deleting the cluster, make sure to delete the FSx for Lustre file system. For more information, see Clean up resources in the FSx for Lustre User Guide.

10.5. Store high-performance apps with FSx for NetApp ONTAP

The NetApp Trident allows Amazon EKS clusters to manage the lifecycle of persistent volumes (PVs) backed by Amazon FSx for NetApp ONTAP file systems.

The NetApp Trident provides dynamic storage orchestration using a Container Storage Interface (CSI) compliant driver. This allows Amazon EKS clusters to manage the lifecycle of persistent volumes (PVs) backed by Amazon FSx for NetApp ONTAP file systems. Note that the Amazon FSx for NetApp ONTAP CSI driver is not compatible with Amazon EKS Hybrid Nodes. To get started, see Use Trident with Amazon FSx for NetApp ONTAP in the NetApp Trident documentation.

Amazon FSx for NetApp ONTAP is a storage service that allows you to launch and run fully managed ONTAP file systems in the cloud. ONTAP is NetApp’s file system technology that provides a widely adopted set of data access and data management capabilities. FSx for ONTAP provides the features, performance, and APIs of on-premises NetApp file systems with the agility, scalability, and simplicity of a fully managed AWS service. For more information, see the FSx for ONTAP User Guide.

10.6. Store data using Amazon FSx for OpenZFS

The Amazon FSx for OpenZFS Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the life cycle of Amazon FSx for OpenZFS volumes.

Amazon FSx for OpenZFS is a fully managed file storage service that makes it easy to move data to AWS from on-premises ZFS or other Linux-based file servers. You can do this without changing your application code or how you manage data. It offers highly reliable, scalable, efficient, and feature-rich file storage built on the open-source OpenZFS file system. It combines these capabilities with the agility, scalability, and simplicity of a fully managed AWS service. For more information, see the Amazon FSx for OpenZFS User Guide.

The FSx for OpenZFS Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the life cycle of FSx for OpenZFS volumes. Note that the Amazon FSx for OpenZFS CSI driver is not compatible with Amazon EKS Hybrid Nodes. To deploy the FSx for OpenZFS CSI driver to your Amazon EKS cluster, see aws-fsx-openzfs-csi-driver on GitHub.

10.7. Minimize latency with Amazon File Cache

The Amazon File Cache Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the life cycle of Amazon file caches.

Amazon File Cache is a fully managed, high-speed cache on AWS that’s used to process file data, regardless of where the data is stored. Amazon File Cache automatically loads data into the cache when it’s accessed for the first time and releases data when it’s not used. For more information, see the Amazon File Cache User Guide.

The Amazon File Cache Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the life cycle of Amazon file caches. Note that the Amazon File Cache CSI driver is not compatible with Amazon EKS Hybrid Nodes. To deploy the Amazon File Cache CSI driver to your Amazon EKS cluster, see aws-file-cache-csi-driver on GitHub.

10.8. Access Amazon S3 objects with Mountpoint for Amazon S3 CSI driver

Learn about the Amazon S3 Container Storage Interface (CSI) driver, which provides a CSI interface for managing Amazon S3 files and buckets.

With the Mountpoint for Amazon S3 Container Storage Interface (CSI) driver, your Kubernetes applications can access Amazon S3 objects through a file system interface, achieving high aggregate throughput without changing any application code. Built on Mountpoint for Amazon S3, the CSI driver presents an Amazon S3 bucket as a volume that can be accessed by containers in Amazon EKS and self-managed Kubernetes clusters. This topic shows you how to deploy the Mountpoint for Amazon S3 CSI driver to your Amazon EKS cluster.

10.8.1. Considerations

The Mountpoint for Amazon S3 CSI driver isn’t presently compatible with Windows-based container images.
The Mountpoint for Amazon S3 CSI driver isn’t presently compatible with Amazon EKS Hybrid Nodes.
The Mountpoint for Amazon S3 CSI driver doesn’t support AWS Fargate. However, containers that are running in Amazon EC2 (either with Amazon EKS or a custom Kubernetes installation) are supported.

The Mountpoint for Amazon S3 CSI driver supports only static provisioning. Dynamic provisioning, or creation of new buckets, isn’t supported.

Static provisioning refers to using an existing Amazon S3 bucket that is specified as the bucketName in the volumeAttributes in the PersistentVolume object. For more information, see Static Provisioning on GitHub.

Volumes mounted with the Mountpoint for Amazon S3 CSI driver don’t support all POSIX file-system features. For details about file-system behavior, see Mountpoint for Amazon S3 file system behavior on GitHub.

10.8.2. Prerequisites

An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.
Version 2.12.3 or later of the AWS CLI installed and configured on your device or AWS CloudShell.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.

10.8.3. Create an IAM policy

The Mountpoint for Amazon S3 CSI driver requires Amazon S3 permissions to interact with your file system. This section shows how to create an IAM policy that grants the necessary permissions.

The following example policy follows the IAM permission recommendations for Mountpoint. Alternatively, you can use the AWS managed policy AmazonS3FullAccess, but this managed policy grants more permissions than are needed for Mountpoint.

For more information about the recommended permissions for Mountpoint, see Mountpoint IAM permissions on GitHub.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Policies.
On the Policies page, choose Create policy.
For Policy editor, choose JSON.

Under Policy editor, copy and paste the following:

Replace amzn-s3-demo-bucket1 with your own Amazon S3 bucket name.

{
   "Version": "2012-10-17",
   "Statement": [
        {
            "Sid": "MountpointFullBucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "region.arns3:::amzn-s3-demo-bucket1"
            ]
        },
        {
            "Sid": "MountpointFullObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": [
                "region.arns3:::amzn-s3-demo-bucket1/*"
            ]
        }
   ]
}

Directory buckets, introduced with the Amazon S3 Express One Zone storage class, use a different authentication mechanism from general purpose buckets. Instead of using s3:* actions, you should use the s3express:CreateSession action. For information about directory buckets, see Directory buckets in the Amazon S3 User Guide.

Below is an example of least-privilege policy that you would use for a directory bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3express:CreateSession",
            "Resource": "region.arns3express:us-west-2:111122223333:bucket/amzn-s3-demo-bucket1--usw2-az1--x-s3"
        }
    ]
}

Choose Next.
On the Review and create page, name your policy. This example walkthrough uses the name AmazonS3CSIDriverPolicy.
Choose Create policy.

10.8.4. Create an IAM role

The Mountpoint for Amazon S3 CSI driver requires Amazon S3 permissions to interact with your file system. This section shows how to create an IAM role to delegate these permissions. To create this role, you can use one of these tools:

eksctl
consolelong
AWS CLI

The IAM policy AmazonS3CSIDriverPolicy was created in the previous section.

`eksctl`

To create your Mountpoint for Amazon S3 CSI driver IAM role with eksctl

To create the IAM role and the Kubernetes service account, run the following commands. These commands also attach the AmazonS3CSIDriverPolicy IAM policy to the role, annotate the Kubernetes service account (s3-csi-controller-sa) with the IAM role’s Amazon Resource Name (ARN), and add the Kubernetes service account name to the trust policy for the IAM role.

CLUSTER_NAME=my-cluster
REGION=region-code
ROLE_NAME=AmazonEKS_S3_CSI_DriverRole
POLICY_ARN=AmazonEKS_S3_CSI_DriverRole_ARN
eksctl create iamserviceaccount \
    --name s3-csi-driver-sa \
    --namespace kube-system \
    --cluster $CLUSTER_NAME \
    --attach-policy-arn $POLICY_ARN \
    --approve \
    --role-name $ROLE_NAME \
    --region $REGION \
    --role-only

`consolelong`

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.
On the Select trusted entity page, do the following:
1. In the Trusted entity type section, choose Web identity.
2. For Identity provider, choose the OpenID Connect provider URL for your cluster (as shown under Overview in Amazon EKS).
  
  If no URLs are shown, review the Prerequisites.
3. For Audience, choose sts.amazonaws.com.
4. Choose Next.
On the Add permissions page, do the following:
1. In the Filter policies box, enter AmazonS3CSIDriverPolicy.
  
  This policy was created in the previous section.
2. Select the check box to the left of the AmazonS3CSIDriverPolicy result that was returned in the search.
3. Choose Next.
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKS_S3_CSI_DriverRole.
2. Under Add tags (Optional), add metadata to the role by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM resources in the IAM User Guide.
3. Choose Create role.
After the role is created, choose the role in the console to open it for editing.
Choose the Trust relationships tab, and then choose Edit trust policy.
Find the line that looks similar to the following:
```
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
```
Add a comma to the end of the previous line, and then add the following line after it. Replace region-code with the AWS Region that your cluster is in. Replace EXAMPLED539D4633E53DE1B71EXAMPLE with your cluster’s OIDC provider ID.
```
"oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:s3-csi-driver-sa"
```
Ensure that the Condition operator is set to "StringEquals".
Choose Update policy to finish.

`AWS` CLI

View the OIDC provider URL for your cluster. Replace my-cluster with the name of your cluster. If the output from the command is None, review the Prerequisites.
```
aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text
```
An example output is as follows.
```
https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
```

Create the IAM role, granting the Kubernetes service account the AssumeRoleWithWebIdentity action.

Copy the following contents to a file named aws-s3-csi-driver-trust-policy.json. Replace 111122223333 with your account ID. Replace EXAMPLED539D4633E53DE1B71EXAMPLE and region-code with the values returned in the previous step.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "region.arniam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:s3-csi-driver-sa",
          "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Create the role. You can change AmazonEKS_S3_CSI_DriverRole to a different name, but if you do, make sure to change it in later steps too.

aws iam create-role \
  --role-name AmazonEKS_S3_CSI_DriverRole \
  --assume-role-policy-document file://"aws-s3-csi-driver-trust-policy.json"

Attach the previously created IAM policy to the role with the following command.
```
aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonS3CSIDriverPolicy \
  --role-name AmazonEKS_S3_CSI_DriverRole
```
The IAM policy AmazonS3CSIDriverPolicy was created in the previous section.
Skip this step if you’re installing the driver as an Amazon EKS add-on. For self-managed installations of the driver, create Kubernetes service accounts that are annotated with the ARN of the IAM role that you created.
1. Save the following contents to a file named mountpoint-s3-service-account.yaml. Replace 111122223333 with your account ID.
  --- apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/name: aws-mountpoint-s3-csi-driver name: mountpoint-s3-csi-controller-sa namespace: kube-system annotations: eks.amazonaws.com/role-arn: region.arniam::111122223333:role/AmazonEKS_S3_CSI_DriverRole
2. Create the Kubernetes service account on your cluster. The Kubernetes service account (mountpoint-s3-csi-controller-sa) is annotated with the IAM role that you created named AmazonEKS_S3_CSI_DriverRole.
  kubectl apply -f mountpoint-s3-service-account.yaml
  When you deploy the plugin in this procedure, it creates and is configured to use a service account named s3-csi-driver-sa.

10.8.5. Install the `Mountpoint` for Amazon S3 CSI driver

You may install the Mountpoint for Amazon S3 CSI driver through the Amazon EKS add-on. You can use the following tools to add the add-on to your cluster:

eksctl
consolelong
AWS CLI

Alternatively, you may install Mountpoint for Amazon S3 CSI driver as a self-managed installation. For instructions on doing a self-managed installation, see Installation on GitHub.

Starting from v1.8.0, you can configure taints to tolerate for the CSI driver’s Pods. To do this, either specify a custom set of taints to tolerate with node.tolerations or tolorate all taints with node.tolerateAllTaints. For more information, see Taints and Tolerations in the Kubernetes documentation.

`eksctl`

To add the Amazon S3 CSI add-on using eksctl

Run the following command. Replace my-cluster with the name of your cluster, 111122223333 with your account ID, and AmazonEKS_S3_CSI_DriverRole with the name of the IAM role created earlier.

eksctl create addon --name aws-mountpoint-s3-csi-driver --cluster my-cluster \
  --service-account-role-arn region.arniam::111122223333:role/AmazonEKS_S3_CSI_DriverRole --force

If you remove the --force option and any of the Amazon EKS add-on settings conflict with your existing settings, then updating the Amazon EKS add-on fails, and you receive an error message to help you resolve the conflict. Before specifying this option, make sure that the Amazon EKS add-on doesn’t manage settings that you need to manage, because those settings are overwritten with this option. For more information about other options for this setting, see Addons in the eksctl documentation. For more information about Amazon EKS Kubernetes field management, see kubernetes-field-management.title.

You can customize eksctl through configuration files. For more information, see Working with configuration values in the eksctl documentation. The following example shows how to tolerate all taints.

# config.yaml
...

addons:
- name: aws-mountpoint-s3-csi-driver
  serviceAccountRoleARN: region.arniam::111122223333:role/AmazonEKS_S3_CSI_DriverRole
  configurationValues: |-
    node:
      tolerateAllTaints: true

`consolelong`

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
Choose the name of the cluster that you want to configure the Mountpoint for Amazon S3 CSI add-on for.
Choose the Add-ons tab.
Choose Get more add-ons.
On the Select add-ons page, do the following:
1. In the Amazon EKS-addons section, select the Mountpoint for Amazon S3 CSI Driver check box.
2. Choose Next.
On the Configure selected add-ons settings page, do the following:
1. Select the Version you’d like to use.
2. For Select IAM role, select the name of an IAM role that you attached the Mountpoint for Amazon S3 CSI driver IAM policy to.
3. (Optional) Update the Conflict resolution method after expanding the Optional configuration settings. If you select Override, one or more of the settings for the existing add-on can be overwritten with the Amazon EKS add-on settings. If you don’t enable this option and there’s a conflict with your existing settings, the operation fails. You can use the resulting error message to troubleshoot the conflict. Before selecting this option, make sure that the Amazon EKS add-on doesn’t manage settings that you need to self-manage.
4. (Optional) Configure tolerations in the Configuration values field after expanding the Optional configuration settings.
5. Choose Next.
On the Review and add page, choose Create. After the add-on installation is complete, you see your installed add-on.

`AWS` CLI

To add the Mountpoint for Amazon S3 CSI add-on using the AWS CLI

Run the following command. Replace my-cluster with the name of your cluster, 111122223333 with your account ID, and AmazonEKS_S3_CSI_DriverRole with the name of the role that was created earlier.

aws eks create-addon --cluster-name my-cluster --addon-name aws-mountpoint-s3-csi-driver \
  --service-account-role-arn region.arniam::111122223333:role/AmazonEKS_S3_CSI_DriverRole

You can customize the command with the --configuration-values flag. The following alternative example shows how to tolerate all taints.

aws eks create-addon --cluster-name my-cluster --addon-name aws-mountpoint-s3-csi-driver \
  --service-account-role-arn region.arniam::111122223333:role/AmazonEKS_S3_CSI_DriverRole \
  --configuration-values '{"node":{"tolerateAllTaints":true}}'

10.8.6. Configure `Mountpoint` for Amazon S3

In most cases, you can configure Mountpoint for Amazon S3 with only a bucket name. For instructions on configuring Mountpoint for Amazon S3, see Configuring Mountpoint for Amazon S3 on GitHub.

10.8.7. Deploy a sample application

You can deploy static provisioning to the driver on an existing Amazon S3 bucket. For more information, see Static provisioning on GitHub.

10.8.8. Remove `Mountpoint` for Amazon S3 CSI Driver

You have two options for removing an Amazon EKS add-on.

Preserve add-on software on your cluster – This option removes Amazon EKS management of any settings. It also removes the ability for Amazon EKS to notify you of updates and automatically update the Amazon EKS add-on after you initiate an update. However, it preserves the add-on software on your cluster. This option makes the add-on a self-managed installation, rather than an Amazon EKS add-on. With this option, there’s no downtime for the add-on. The commands in this procedure use this option.
Remove add-on software entirely from your cluster – We recommend that you remove the Amazon EKS add-on from your cluster only if there are no resources on your cluster that are dependent on it. To do this option, delete --preserve from the command you use in this procedure.

If the add-on has an IAM account associated with it, the IAM account isn’t removed.

You can use the following tools to remove the Amazon S3 CSI add-on:

eksctl
consolelong
AWS CLI

`eksctl`

To remove the Amazon S3 CSI add-on using eksctl

Replace my-cluster with the name of your cluster, and then run the following command.

eksctl delete addon --cluster my-cluster --name aws-mountpoint-s3-csi-driver --preserve

`consolelong`

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
Choose the name of the cluster that you want to remove the Amazon EBS CSI add-on for.
Choose the Add-ons tab.
Choose Mountpoint for Amazon S3 CSI Driver.
Choose Remove.
In the Remove: aws-mountpoint-s3-csi-driver confirmation dialog box, do the following:
1. If you want Amazon EKS to stop managing settings for the add-on, select Preserve on cluster. Do this if you want to retain the add-on software on your cluster. This is so that you can manage all of the settings of the add-on on your own.
2. Enter aws-mountpoint-s3-csi-driver.
3. Select Remove.

`AWS` CLI

To remove the Amazon S3 CSI add-on using the AWS CLI

Replace my-cluster with the name of your cluster, and then run the following command.

aws eks delete-addon --cluster-name my-cluster --addon-name aws-mountpoint-s3-csi-driver --preserve

10.9. Enable snapshot functionality for CSI volumes

The Container Storage Interface (CSI) snapshot controller enables the use of snapshot functionality in compatible CSI drivers, such as the Amazon EBS CSI driver.

Snapshot functionality allows for point-in-time copies of your data. For this capability to work in Kubernetes, you need both a CSI driver with snapshot support (such as the Amazon EBS CSI driver) and a CSI snapshot controller. The snapshot controller is available either as an Amazon EKS managed add-on or as a self-managed installation.

Here are some things to consider when using the CSI snapshot controller.

The snapshot controller must be installed alongside a CSI driver with snapshot functionality. For installation instructions of the Amazon EBS CSI driver, see ebs-csi.title.
Kubernetes doesn’t support snapshots of volumes being served via CSI migration, such as Amazon EBS volumes using a StorageClass with provisioner kubernetes.io/aws-ebs. Volumes must be created with a StorageClass that references the CSI driver provisioner, ebs.csi.aws.com. For more information about CSI migration, see ebs-csi-migration-faq.title.
Amazon EKS Auto Mode does not include the snapshot controller. The storage capability of EKS Auto Mode is compatible with the snapshot controller.

We recommend that you install the CSI snapshot controller through the Amazon EKS managed add-on. This add-on includes the custom resource definitions (CRDs) that are needed to create and manage snapshots on Amazon EKS. To add an Amazon EKS add-on to your cluster, see creating-an-add-on.title. For more information about add-ons, see eks-add-ons.title.

Alternatively, if you want a self-managed installation of the CSI snapshot controller, see Usage in the upstream Kubernetes external-snapshotter on GitHub.

11. Configure networking for Amazon EKS clusters

11.1. View Amazon EKS networking requirements for VPC and subnets

Learn how to configure the VPC and subnets to meet networking requirements for creating Amazon EKS clusters with sufficient IP addresses, subnet types, and availability zones. Understand IP family usage by component and shared subnet considerations.

When you create a cluster, you specify a VPC and at least two subnets that are in different Availability Zones. This topic provides an overview of Amazon EKS specific requirements and considerations for the VPC and subnets that you use with your cluster. If you don’t have a VPC to use with Amazon EKS, see creating-a-vpc.title. If you’re creating a local or extended cluster on AWS Outposts, see eks-outposts-vpc-subnet-requirements.title instead of this topic. The content in this topic applies for Amazon EKS clusters with hybrid nodes. For additional networking requirements for hybrid nodes, see hybrid-nodes-networking.title.

11.1.1. VPC requirements and considerations

When you create a cluster, the VPC that you specify must meet the following requirements and considerations:

The VPC must have a sufficient number of IP addresses available for the cluster, any nodes, and other Kubernetes resources that you want to create. If the VPC that you want to use doesn’t have a sufficient number of IP addresses, try to increase the number of available IP addresses.

You can do this by updating the cluster configuration to change which subnets and security groups the cluster uses. You can update from the consolelong, the latest version of the AWS CLI, AWS CloudFormation, and eksctl version v0.164.0-rc.0 or later. You might need to do this to provide subnets with more available IP addresses to successfully upgrade a cluster version.

All subnets that you add must be in the same set of AZs as originally provided when you created the cluster. New subnets must satisfy all of the other requirements, for example they must have sufficient IP addresses.

For example, assume that you made a cluster and specified four subnets. In the order that you specified them, the first subnet is in the us-west-2a Availability Zone, the second and third subnets are in us-west-2b Availability Zone, and the fourth subnet is in us-west-2c Availability Zone. If you want to change the subnets, you must provide at least one subnet in each of the three Availability Zones, and the subnets must be in the same VPC as the original subnets.

If you need more IP addresses than the CIDR blocks in the VPC have, you can add additional CIDR blocks by associating additional Classless Inter-Domain Routing (CIDR) blocks with your VPC. You can associate private (RFC 1918) and public (non-RFC 1918) CIDR blocks to your VPC either before or after you create your cluster. It can take a cluster up to five hours for a CIDR block that you associated with a VPC to be recognized.

You can conserve IP address utilization by using a transit gateway with a shared services VPC. For more information, see Isolated VPCs with shared services and Amazon EKS VPC routable IP address conservation patterns in a hybrid network.

If you want Kubernetes to assign IPv6 addresses to Pods and services, associate an IPv6 CIDR block with your VPC. For more information, see Associate an IPv6 CIDR block with your VPC in the Amazon VPC User Guide. You cannot use IPv6 addresses with Pods and services running on hybrid nodes and you cannot use hybrid nodes with clusters configured with the IPv6 IP address family.
The VPC must have DNS hostname and DNS resolution support. Otherwise, nodes can’t register to your cluster. For more information, see DNS attributes for your VPC in the Amazon VPC User Guide.
The VPC might require VPC endpoints using AWS PrivateLink. For more information, see network-requirements-subnets.title.

If you created a cluster with Kubernetes 1.14 or earlier, Amazon EKS added the following tag to your VPC:

Key Value

kubernetes.io/cluster/my-cluster

owned

This tag was only used by Amazon EKS. You can remove the tag without impacting your services. It’s not used with clusters that are version 1.15 or later.

11.1.2. Subnet requirements and considerations

When you create a cluster, Amazon EKS creates 2–4 elastic network interfaces in the subnets that you specify. These network interfaces enable communication between your cluster and your VPC. These network interfaces also enable Kubernetes features such as kubectl exec and kubectl logs. Each Amazon EKS created network interface has the text Amazon EKS cluster-name in its description.

Amazon EKS can create its network interfaces in any subnet that you specify when you create a cluster. You can change which subnets Amazon EKS creates its network interfaces in after your cluster is created. When you update the Kubernetes version of a cluster, Amazon EKS deletes the original network interfaces that it created, and creates new network interfaces. These network interfaces might be created in the same subnets as the original network interfaces or in different subnets than the original network interfaces. To control which subnets network interfaces are created in, you can limit the number of subnets you specify to only two when you create a cluster or update the subnets after creating the cluster.

Subnet requirements for clusters

The subnets that you specify when you create or update a cluster must meet the following requirements:

The subnets must each have at least six IP addresses for use by Amazon EKS. However, we recommend at least 16 IP addresses.
The subnets must be in at least two different Availability Zones.
The subnets can’t reside in AWS Outposts or AWS Wavelength. However, if you have them in your VPC, you can deploy self-managed nodes and Kubernetes resources to these types of subnets. For more information about self-managed nodes, see worker.title.
The subnets can be a public or private. However, we recommend that you specify private subnets, if possible. A public subnet is a subnet with a route table that includes a route to an internet gateway, whereas a private subnet is a subnet with a route table that doesn’t include a route to an internet gateway.
The subnets can’t reside in the following Availability Zones:

AWS Region Region name Disallowed Availability Zone IDs

us-east-1

US East (N. Virginia)

use1-az3

us-west-1

US West (N. California)

usw1-az2

ca-central-1

Canada (Central)

cac1-az3

IP address family usage by component

The following table contains the IP address family used by each component of Amazon EKS. You can use a network address translation (NAT) or other compatibility system to connect to these components from source IP addresses in families with the "No" value for a table entry.

Functionality can differ depending on the IP family (ipFamily) setting of the cluster. This setting changes the type of IP addresses used for the CIDR block that Kubernetes assigns to Services. A cluster with the setting value of IPv4 is referred to as an IPv4 cluster, and a cluster with the setting value of IPv6 is referred to as an IPv6 cluster.

Component IPv4 addresses IPv6 addresses Dual stack addresses

EKS API public endpoint

Yes^1,3

EKS API VPC endpoint

Yes

EKS Auth API public endpoint (EKS Pod Identity)

Yes¹

EKS Auth API VPC endpoint (EKS Pod Identity)

Yes¹

IPv4 Kubernetes cluster public endpoint²

Yes

IPv4 Kubernetes cluster private endpoint²

Yes

IPv6 Kubernetes cluster public endpoint²

Yes^1,4

Yes⁴

IPv6 Kubernetes cluster private endpoint²

Yes^1,4

Yes⁴

Kubernetes cluster subnets

Yes²

Node Primary IP addresses

Yes²

Cluster CIDR range for Service IP addresses

Yes²

Pod IP addresses from the VPC CNI

Yes²

IRSA OIDC Issuer URLs

Yes^1,3

¹ The endpoint is dual stack with both IPv4 and IPv6 addresses. Your applications outside of AWS, your nodes for the cluster, and your pods inside the cluster can reach this endpoint by either IPv4 or IPv6.

² You choose between an IPv4 cluster and IPv6 cluster in the IP family (ipFamily) setting of the cluster when you create a cluster and this can’t be changed. Instead, you must choose a different setting when you create another cluster and migrate your workloads.

³ The dual-stack endpoint was introduced in August 2024. To use the dual-stack endpoints with the AWS CLI, see the Dual-stack and FIPS endpoints configuration in the AWS SDKs and Tools Reference Guide. The following lists the new endpoints:

EKS API public endpoint: eks.region.api.aws
IRSA OIDC Issuer URLs: oidc-eks.region.api.aws

⁴ The dual-stack cluster endpoint was introduced in October 2024. EKS creates the following endpoint for new clusters that are made after this date and that select IPv6 in the IP family (ipFamily) setting of the cluster:

EKS cluster public/private endpoint: eks-cluster.region.api.aws

Subnet requirements for nodes

You can deploy nodes and Kubernetes resources to the same subnets that you specify when you create your cluster. However, this isn’t necessary. This is because you can also deploy nodes and Kubernetes resources to subnets that you didn’t specify when you created the cluster. If you deploy nodes to different subnets, Amazon EKS doesn’t create cluster network interfaces in those subnets. Any subnet that you deploy nodes and Kubernetes resources to must meet the following requirements:

The subnets must have enough available IP addresses to deploy all of your nodes and Kubernetes resources to.
If you want Kubernetes to assign IPv6 addresses to Pods and services, then you must have one IPv6 CIDR block and one IPv4 CIDR block that are associated with your subnet. For more information, see Associate an IPv6 CIDR block with your subnet in the Amazon VPC User Guide. The route tables that are associated with the subnets must include routes to IPv4 and IPv6 addresses. For more information, see Routes in the Amazon VPC User Guide. Pods are assigned only an IPv6 address. However the network interfaces that Amazon EKS creates for your cluster and your nodes are assigned an IPv4 and an IPv6 address.
If you need inbound access from the internet to your Pods, make sure to have at least one public subnet with enough available IP addresses to deploy load balancers and ingresses to. You can deploy load balancers to public subnets. Load balancers can load balance to Pods in private or public subnets. We recommend deploying your nodes to private subnets, if possible.
If you plan to deploy nodes to a public subnet, the subnet must auto-assign IPv4 public addresses or IPv6 addresses. If you deploy nodes to a private subnet that has an associated IPv6 CIDR block, the private subnet must also auto-assign IPv6 addresses. If you used the AWS CloudFormation template provided by Amazon EKS to deploy your VPC after March 26, 2020, this setting is enabled. If you used the templates to deploy your VPC before this date or you use your own VPC, you must enable this setting manually. For the template, see creating-a-vpc.title. For more information, see Modify the public IPv4 addressing attribute for your subnet and Modify the IPv6 addressing attribute for your subnet in the Amazon VPC User Guide.
If the subnet that you deploy a node to is a private subnet and its route table doesn’t include a route to a network address translation (NAT) device (IPv4) or an egress-only gateway (IPv6), add VPC endpoints using AWS PrivateLink to your VPC. VPC endpoints are needed for all the AWS services that your nodes and Pods need to communicate with. Examples include Amazon ECR, Elastic Load Balancing, Amazon CloudWatch, AWS Security Token Service, and Amazon Simple Storage Service (Amazon S3). The endpoint must include the subnet that the nodes are in. Not all AWS services support VPC endpoints. For more information, see What is AWS PrivateLink? and AWS services that integrate with AWS PrivateLink. For a list of more Amazon EKS requirements, see private-clusters.title.
If you want to deploy load balancers to a subnet, the subnet must have the following tag:
- Private subnets
  
  Key Value
  
  kubernetes.io/role/internal-elb
  
  1
- Public subnets
  
  Key Value
  
  kubernetes.io/role/elb
  
  1

When a Kubernetes cluster that’s version 1.18 and earlier was created, Amazon EKS added the following tag to all of the subnets that were specified.

Key Value

kubernetes.io/cluster/my-cluster

shared

When you create a new Kubernetes cluster now, Amazon EKS doesn’t add the tag to your subnets. If the tag was on subnets that were used by a cluster that was previously a version earlier than 1.19, the tag wasn’t automatically removed from the subnets when the cluster was updated to a newer version. Version 2.1.1 or earlier of the AWS Load Balancer Controller requires this tag. If you are using a newer version of the Load Balancer Controller, you can remove the tag without interrupting your services. For more information about the controller, see aws-load-balancer-controller.title.

If you deployed a VPC by using eksctl or any of the Amazon EKS AWS CloudFormation VPC templates, the following applies:

On or after March 26, 2020 – Public IPv4 addresses are automatically assigned by public subnets to new nodes that are deployed to public subnets.
Before March 26, 2020 – Public IPv4 addresses aren’t automatically assigned by public subnets to new nodes that are deployed to public subnets.

This change impacts new node groups that are deployed to public subnets in the following ways:

Managed node groups – If the node group is deployed to a public subnet on or after April 22, 2020, automatic assignment of public IP addresses must be enabled for the public subnet. For more information, see Modifying the public IPv4 addressing attribute for your subnet.
Linux, Windows, or Arm self-managed node groups – If the node group is deployed to a public subnet on or after March 26, 2020, automatic assignment of public IP addresses must be enabled for the public subnet. Otherwise, the nodes must be launched with a public IP address instead. For more information, see Modifying the public IPv4 addressing attribute for your subnet or Assigning a public IPv4 address during instance launch.

11.1.3. Shared subnet requirements and considerations

You can use VPC sharing to share subnets with other AWS accounts within the same AWS Organizations. You can create Amazon EKS clusters in shared subnets, with the following considerations:

The owner of the VPC subnet must share a subnet with a participant account before that account can create an Amazon EKS cluster in it.
You can’t launch resources using the default security group for the VPC because it belongs to the owner. Additionally, participants can’t launch resources using security groups that are owned by other participants or the owner.
In a shared subnet, the participant and the owner separately controls the security groups within each respective account. The subnet owner can see security groups that are created by the participants but cannot perform any actions on them. If the subnet owner wants to remove or modify these security groups, the participant that created the security group must take the action.
If a cluster is created by a participant, the following considerations apply:
- Cluster IAM role and Node IAM roles must be created in that account. For more information, see cluster-iam-role.title and create-node-role.title.
- All nodes must be made by the same participant, including managed node groups.
The shared VPC owner cannot view, update or delete a cluster that a participant creates in the shared subnet. This is in addition to the VPC resources that each account has different access to. For more information, see Responsibilities and permissions for owners and participants in the Amazon VPC User Guide.
If you use the custom networking feature of the Amazon VPC CNI plugin for Kubernetes, you need to use the Availability Zone ID mappings listed in the owner account to create each ENIConfig. For more information, see cni-custom-network.title.

For more information about VPC subnet sharing, see Share your VPC with other accounts in the Amazon VPC User Guide.

11.2. Create an Amazon VPC for your Amazon EKS cluster

Learn how to create an Amazon VPC for your cluster using an Amazon EKS provided AWS CloudFormation template.

You can use Amazon Virtual Private Cloud (Amazon VPC) to launch AWS resources into a virtual network that you’ve defined. This virtual network closely resembles a traditional network that you might operate in your own data center. However, it comes with the benefits of using the scalable infrastructure of Amazon Web Services. We recommend that you have a thorough understanding of the Amazon VPC service before deploying production Amazon EKS clusters. For more information, see the Amazon VPC User Guide.

An Amazon EKS cluster, nodes, and Kubernetes resources are deployed to a VPC. If you want to use an existing VPC with Amazon EKS, that VPC must meet the requirements that are described in network-reqs.title. This topic describes how to create a VPC that meets Amazon EKS requirements using an Amazon EKS provided AWS CloudFormation template. Once you’ve deployed a template, you can view the resources created by the template to know exactly what resources it created, and the configuration of those resources. If you are using hybrid nodes, your VPC must have routes in its route table for your on-premises network. For more information about the network requirements for hybrid nodes, see hybrid-nodes-networking.title.

11.2.1. Prerequisites

To create a VPC for Amazon EKS, you must have the necessary IAM permissions to create Amazon VPC resources. These resources are VPCs, subnets, security groups, route tables and routes, and internet and NAT gateways. For more information, see Create a VPC with a public subnet example policy in the Amazon VPC User Guide and the full list of Actions in the Service Authorization Reference.

You can create a VPC with public and private subnets, only public subnets, or only private subnets.

11.2.2. Public and private subnets

This VPC has two public and two private subnets. A public subnet’s associated route table has a route to an internet gateway. However, the route table of a private subnet doesn’t have a route to an internet gateway. One public and one private subnet are deployed to the same Availability Zone. The other public and private subnets are deployed to a second Availability Zone in the same AWS Region. We recommend this option for most deployments.

With this option, you can deploy your nodes to private subnets. This option allows Kubernetes to deploy load balancers to the public subnets that can load balance traffic to Pods that run on nodes in the private subnets. Public IPv4 addresses are automatically assigned to nodes that are deployed to public subnets, but public IPv4 addresses aren’t assigned to nodes deployed to private subnets.

You can also assign IPv6 addresses to nodes in public and private subnets. The nodes in private subnets can communicate with the cluster and other AWS services. Pods can communicate to the internet through a NAT gateway using IPv4 addresses or outbound-only Internet gateway using IPv6 addresses deployed in each Availability Zone. A security group is deployed that has rules that deny all inbound traffic from sources other than the cluster or nodes but allows all outbound traffic. The subnets are tagged so that Kubernetes can deploy load balancers to them.

Open the AWS CloudFormation console.
From the navigation bar, select an AWS Region that supports Amazon EKS.
Choose Create stack, With new resources (standard).
Under Prerequisite - Prepare template, make sure that Template is ready is selected and then under Specify template, select Amazon S3 URL.
You can create a VPC that supports only IPv4, or a VPC that supports IPv4 and IPv6. Paste one of the following URLs into the text area under Amazon S3 URL and choose Next:
- IPv4

https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-vpc-private-subnets.yaml

IPv4 and IPv6

https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-ipv6-vpc-public-private-subnets.yaml

On the Specify stack details page, enter the parameters, and then choose Next.
- Stack name: Choose a stack name for your AWS CloudFormation stack. For example, you can use the template name you used in the previous step. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
- VpcBlock: Choose an IPv4 CIDR range for your VPC. Each node, Pod, and load balancer that you deploy is assigned an IPv4 address from this block. The default IPv4 values provide enough IP addresses for most implementations, but if it doesn’t, then you can change it. For more information, see VPC and subnet sizing in the Amazon VPC User Guide. You can also add additional CIDR blocks to the VPC once it’s created. If you’re creating an IPv6 VPC, IPv6 CIDR ranges are automatically assigned for you from Amazon’s Global Unicast Address space.
- PublicSubnet01Block: Specify an IPv4 CIDR block for public subnet 1. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it. If you’re creating an IPv6 VPC, this block is specified for you within the template.
- PublicSubnet02Block: Specify an IPv4 CIDR block for public subnet 2. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it. If you’re creating an IPv6 VPC, this block is specified for you within the template.
- PrivateSubnet01Block: Specify an IPv4 CIDR block for private subnet 1. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it. If you’re creating an IPv6 VPC, this block is specified for you within the template.
- PrivateSubnet02Block: Specify an IPv4 CIDR block for private subnet 2. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it. If you’re creating an IPv6 VPC, this block is specified for you within the template.
(Optional) On the Configure stack options page, tag your stack resources and then choose Next.
On the Review page, choose Create stack.
When your stack is created, select it in the console and choose Outputs.
Record the VpcId for the VPC that was created. You need this when you create your cluster and nodes.
Record the SubnetIds for the subnets that were created and whether you created them as public or private subnets. You need at least two of these when you create your cluster and nodes.
If you created an IPv4 VPC, skip this step. If you created an IPv6 VPC, you must enable the auto-assign IPv6 address option for the public subnets that were created by the template. That setting is already enabled for the private subnets. To enable the setting, complete the following steps:
1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
2. In the left navigation pane, choose Subnets
3. Select one of your public subnets (stack-name/SubnetPublic01 or stack-name/SubnetPublic02 contains the word public) and choose Actions, Edit subnet settings.
4. Choose the Enable auto-assign *IPv6 address* check box and then choose Save.
5. Complete the previous steps again for your other public subnet.

11.2.3. Only public subnets

This VPC has three public subnets that are deployed into different Availability Zones in an AWS Region. All nodes are automatically assigned public IPv4 addresses and can send and receive internet traffic through an internet gateway. A security group is deployed that denies all inbound traffic and allows all outbound traffic. The subnets are tagged so that Kubernetes can deploy load balancers to them.

Open the AWS CloudFormation console.
From the navigation bar, select an AWS Region that supports Amazon EKS.
Choose Create stack, With new resources (standard).
Under Prepare template, make sure that Template is ready is selected and then under Template source, select Amazon S3 URL.
Paste the following URL into the text area under Amazon S3 URL and choose Next:

https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-vpc-sample.yaml

On the Specify Details page, enter the parameters, and then choose Next.
- Stack name: Choose a stack name for your AWS CloudFormation stack. For example, you can call it amazon-eks-vpc-sample. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
- VpcBlock: Choose a CIDR block for your VPC. Each node, Pod, and load balancer that you deploy is assigned an IPv4 address from this block. The default IPv4 values provide enough IP addresses for most implementations, but if it doesn’t, then you can change it. For more information, see VPC and subnet sizing in the Amazon VPC User Guide. You can also add additional CIDR blocks to the VPC once it’s created.
- Subnet01Block: Specify a CIDR block for subnet 1. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it.
- Subnet02Block: Specify a CIDR block for subnet 2. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it.
- Subnet03Block: Specify a CIDR block for subnet 3. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it.
(Optional) On the Options page, tag your stack resources. Choose Next.
On the Review page, choose Create.
When your stack is created, select it in the console and choose Outputs.
Record the VpcId for the VPC that was created. You need this when you create your cluster and nodes.
Record the SubnetIds for the subnets that were created. You need at least two of these when you create your cluster and nodes.
(Optional) Any cluster that you deploy to this VPC can assign private IPv4 addresses to your Pods and services. If you want to deploy clusters to this VPC to assign private IPv6 addresses to your Pods and services, make updates to your VPC, subnet, route tables, and security groups. For more information, see Migrate existing VPCs from IPv4 to IPv6 in the Amazon VPC User Guide. Amazon EKS requires that your subnets have the Auto-assign IPv6 addresses option enabled. By default, it’s disabled.

11.2.4. Only private subnets

This VPC has three private subnets that are deployed into different Availability Zones in the AWS Region. Resources that are deployed to the subnets can’t access the internet, nor can the internet access resources in the subnets. The template creates VPC endpoints using AWS PrivateLink for several AWS services that nodes typically need to access. If your nodes need outbound internet access, you can add a public NAT gateway in the Availability Zone of each subnet after the VPC is created. A security group is created that denies all inbound traffic, except from resources deployed into the subnets. A security group also allows all outbound traffic. The subnets are tagged so that Kubernetes can deploy internal load balancers to them. If you’re creating a VPC with this configuration, see private-clusters.title for additional requirements and considerations.

Open the AWS CloudFormation console.
From the navigation bar, select an AWS Region that supports Amazon EKS.
Choose Create stack, With new resources (standard).
Under Prepare template, make sure that Template is ready is selected and then under Template source, select Amazon S3 URL.
Paste the following URL into the text area under Amazon S3 URL and choose Next:

https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-fully-private-vpc.yaml

On the Specify Details page, enter the parameters and then choose Next.
- Stack name: Choose a stack name for your AWS CloudFormation stack. For example, you can call it amazon-eks-fully-private-vpc. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
- VpcBlock: Choose a CIDR block for your VPC. Each node, Pod, and load balancer that you deploy is assigned an IPv4 address from this block. The default IPv4 values provide enough IP addresses for most implementations, but if it doesn’t, then you can change it. For more information, see VPC and subnet sizing in the Amazon VPC User Guide. You can also add additional CIDR blocks to the VPC once it’s created.
- PrivateSubnet01Block: Specify a CIDR block for subnet 1. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it.
- PrivateSubnet02Block: Specify a CIDR block for subnet 2. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it.
- PrivateSubnet03Block: Specify a CIDR block for subnet 3. The default value provides enough IP addresses for most implementations, but if it doesn’t, then you can change it.
(Optional) On the Options page, tag your stack resources. Choose Next.
On the Review page, choose Create.
When your stack is created, select it in the console and choose Outputs.
Record the VpcId for the VPC that was created. You need this when you create your cluster and nodes.
Record the SubnetIds for the subnets that were created. You need at least two of these when you create your cluster and nodes.
(Optional) Any cluster that you deploy to this VPC can assign private IPv4 addresses to your Pods and services. If you want deploy clusters to this VPC to assign private IPv6 addresses to your Pods and services, make updates to your VPC, subnet, route tables, and security groups. For more information, see Migrate existing VPCs from IPv4 to IPv6 in the Amazon VPC User Guide. Amazon EKS requires that your subnets have the Auto-assign IPv6 addresses option enabled (it’s disabled by default).

11.3. View Amazon EKS security group requirements for clusters

Learn how to manage security groups for Amazon EKS clusters, including default rules, restricting traffic, and required outbound access for nodes to function properly with your cluster. Understand key security group considerations for secure operation of your Kubernetes cluster on AWS.

This topic describes the security group requirements of an Amazon EKS cluster.

11.3.1. Default cluster security group

When you create a cluster, Amazon EKS creates a security group that’s named eks-cluster-sg-my-cluster-uniqueID. This security group has the following default rules:

Rule type Protocol Ports Source Destination

Inbound

All

Self

Outbound

All

0.0.0.0/0(IPv4) or ::/0 (IPv6)

If your cluster doesn’t need the outbound rule, you can remove it. If you remove it, you must still have the minimum rules listed in Restricting cluster traffic. If you remove the inbound rule, Amazon EKS recreates it whenever the cluster is updated.

Amazon EKS adds the following tags to the security group. If you remove the tags, Amazon EKS adds them back to the security group whenever your cluster is updated.

Key Value

kubernetes.io/cluster/my-cluster

owned

aws:eks:cluster-name

my-cluster

Name

eks-cluster-sg-my-cluster-uniqueid

Amazon EKS automatically associates this security group to the following resources that it also creates:

2–4 elastic network interfaces (referred to for the rest of this document as network interface) that are created when you create your cluster.
Network interfaces of the nodes in any managed node group that you create.

The default rules allow all traffic to flow freely between your cluster and nodes, and allows all outbound traffic to any destination. When you create a cluster, you can (optionally) specify your own security groups. If you do, then Amazon EKS also associates the security groups that you specify to the network interfaces that it creates for your cluster. However, it doesn’t associate them to any node groups that you create.

You can determine the ID of your cluster security group in the consolelong under the cluster’s Networking section. Or, you can do so by running the following AWS CLI command.

aws eks describe-cluster --name my-cluster --query cluster.resourcesVpcConfig.clusterSecurityGroupId

11.3.2. Restricting cluster traffic

If you need to limit the open ports between the cluster and nodes, you can remove the default outbound rule and add the following minimum rules that are required for the cluster. If you remove the default inbound rule, Amazon EKS recreates it whenever the cluster is updated.

Rule type

Protocol

Port

Destination

Outbound

TCP

443

Cluster security group

Outbound

TCP

10250

Cluster security group

Outbound (DNS)

TCP and UDP

Cluster security group

You must also add rules for the following traffic:

Any protocol and ports that you expect your nodes to use for inter-node communication.
Outbound internet access so that nodes can access the Amazon EKS APIs for cluster introspection and node registration at launch time. If your nodes don’t have internet access, review Deploy private clusters with limited internet access for additional considerations.
Node access to pull container images from Amazon ECR or other container registries APIs that they need to pull images from, such as DockerHub. For more information, see AWS IP address ranges in the AWS General Reference.
Node access to Amazon S3.
Separate rules are required for IPv4 and IPv6 addresses.
If you are using hybrid nodes, you must add an additional security group to your cluster to allow communication with your on-premises nodes and pods. For more information, see hybrid-nodes-networking.title.

If you’re considering limiting the rules, we recommend that you thoroughly test all of your Pods before you apply your changed rules to a production cluster.

If you originally deployed a cluster with Kubernetes 1.14 and a platform version of eks.3 or earlier, then consider the following:

You might also have control plane and node security groups. When these groups were created, they included the restricted rules listed in the previous table. These security groups are no longer required and can be removed. However, you need to make sure your cluster security group contains the rules that those groups contain.
If you deployed the cluster using the API directly or you used a tool such as the AWS CLI or AWS CloudFormation to create the cluster and you didn’t specify a security group at cluster creation, then the default security group for the VPC was applied to the cluster network interfaces that Amazon EKS created.

11.3.3. Shared security groups

Amazon EKS supports shared security groups.

Security Group VPC Associations associate security groups with multiple VPCs in the same account and region.
- Learn how to Associate security groups with multiple VPCs in the Amazon VPC User Guide.
Shared security groups enable you to share security groups with other AWS accounts. The accounts must be in the same AWS organization.
- Learn how to Share security groups with organizations in the Amazon VPC User Guide.
Security groups are always limited to a single AWS region.

Considerations for Amazon EKS

EKS has the same requirements of shared or multi-VPC security groups as standard security groups.

11.4. Manage networking add-ons for Amazon EKS clusters

11.4.1. Assign IPs to `Pods` with the Amazon VPC CNI

Create the Amazon VPC CNI (Amazon EKS add-on)

Use the following steps to create the Amazon VPC CNI plugin for Kubernetes Amazon EKS add-on.

Before you begin, review the considerations. For more information, see manage-vpc-cni-add-on-on-considerations.title.

Prerequisites

The following are prerequisites for the Amazon VPC CNI plugin for Kubernetes Amazon EKS add-on.

An existing Amazon EKS cluster. To deploy one, see getting-started.title.
An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.
An IAM role with the AmazonEKS_CNI_Policy IAM policy (if your cluster uses the IPv4 family) or an IPv6 policy (if your cluster uses the IPv6 family) attached to it. For more information about the VPC CNI role, see cni-iam-role.title. For information about the IPv6 policy, see cni-iam-role-create-ipv6-policy.title.
If you’re using version 1.7.0 or later of the Amazon VPC CNI plugin for Kubernetes and you use custom Pod security policies, see psp-delete-default.title and pod-security-policy.title.

Amazon VPC CNI plugin for Kubernetes versions v1.16.0 to v1.16.1 removed compatibility with Kubernetes versions 1.23 and earlier. VPC CNI version v1.16.2 restores compatibility with Kubernetes versions 1.23 and earlier and CNI spec v0.4.0.

Amazon VPC CNI plugin for Kubernetes versions v1.16.0 to v1.16.1 implement CNI specification version v1.0.0. CNI spec v1.0.0 is supported on EKS clusters that run the Kubernetes versions v1.24 or later. VPC CNI version v1.16.0 to v1.16.1 and CNI spec v1.0.0 aren’t supported on Kubernetes version v1.23 or earlier. For more information about v1.0.0 of the CNI spec, see Container Network Interface (CNI) Specification on GitHub.

Procedure

After you complete the prerequisites, use the following steps to create the add-on.

See which version of the add-on is installed on your cluster.

kubectl describe daemonset aws-node --namespace kube-system | grep amazon-k8s-cni: | cut -d : -f 3

An example output is as follows.

v1.16.4-eksbuild.2

See which type of the add-on is installed on your cluster. Depending on the tool that you created your cluster with, you might not currently have the Amazon EKS add-on type installed on your cluster. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni --query addon.addonVersion --output text
```
If a version number is returned, you have the Amazon EKS type of the add-on installed on your cluster and don’t need to complete the remaining steps in this procedure. If an error is returned, you don’t have the Amazon EKS type of the add-on installed on your cluster. Complete the remaining steps of this procedure to install it.

Save the configuration of your currently installed add-on.

kubectl get daemonset aws-node -n kube-system -o yaml > aws-k8s-cni-old.yaml

Create the add-on using the AWS CLI. If you want to use the consolelong or eksctl to create the add-on, see creating-an-add-on.title and specify vpc-cni for the add-on name. Copy the command that follows to your device. Make the following modifications to the command, as needed, and then run the modified command.
- Replace my-cluster with the name of your cluster.
- Replace v1.19.0-eksbuild.1 with the latest version listed in the latest version table for your cluster version. For the latest version table, see vpc-cni-latest-available-version.title.
- Replace 111122223333 with your account ID and AmazonEKSVPCCNIRole with the name of an existing IAM role that you’ve created. Specifying a role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.
  aws eks create-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version v1.19.0-eksbuild.1 \ --service-account-role-arn region.arniam::111122223333:role/AmazonEKSVPCCNIRole
  If you’ve applied custom settings to your current add-on that conflict with the default settings of the Amazon EKS add-on, creation might fail. If creation fails, you receive an error that can help you resolve the issue. Alternatively, you can add --resolve-conflicts OVERWRITE to the previous command. This allows the add-on to overwrite any existing custom settings. Once you’ve created the add-on, you can update it with your custom settings.
Confirm that the latest version of the add-on for your cluster’s Kubernetes version was added to your cluster. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni --query addon.addonVersion --output text
```
It might take several seconds for add-on creation to complete.

An example output is as follows.
```
v1.19.0-eksbuild.1
```
If you made custom settings to your original add-on, before you created the Amazon EKS add-on, use the configuration that you saved in a previous step to update the EKS add-on with your custom settings. Follow the steps in vpc-add-on-update.title.
(Optional) Install the cni-metrics-helper to your cluster. It scrapes elastic network interface and IP address information, aggregates it at a cluster level, and publishes the metrics to Amazon CloudWatch. For more information, see cni-metrics-helper on GitHub.

Update the Amazon VPC CNI (Amazon EKS add-on)

Update the Amazon EKS type of the Amazon VPC CNI plugin for Kubernetes add-on. If you haven’t added the Amazon EKS type of the add-on to your cluster, you can install it by following vpc-add-on-create.title. Or, update the other type of VPC CNI installation by following vpc-add-on-self-managed-update.title.

See which version of the add-on is installed on your cluster. Replace my-cluster with your cluster name.
```
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni --query "addon.addonVersion" --output text
```
An example output is as follows.
```
v1.16.4-eksbuild.2
```
Compare the version with the table of latest versions at vpc-cni-latest-available-version.title. If the version returned is the same as the version for your cluster’s Kubernetes version in the latest version table, then you already have the latest version installed on your cluster and don’t need to complete the rest of this procedure. If you receive an error, instead of a version number in your output, then you don’t have the Amazon EKS type of the add-on installed on your cluster. You need to create the add-on before you can update it with this procedure. To create the Amazon EKS type of the VPC CNI add-on, you can follow vpc-add-on-create.title.

Save the configuration of your currently installed add-on.

kubectl get daemonset aws-node -n kube-system -o yaml > aws-k8s-cni-old.yaml

Update your add-on using the AWS CLI. If you want to use the consolelong or eksctl to update the add-on, see updating-an-add-on.title. Copy the command that follows to your device. Make the following modifications to the command, as needed, and then run the modified command.
- Replace my-cluster with the name of your cluster.
- Replace v1.19.0-eksbuild.1 with the latest version listed in the latest version table for your cluster version.
- Replace 111122223333 with your account ID and AmazonEKSVPCCNIRole with the name of an existing IAM role that you’ve created. To create an IAM role for the VPC CNI, see cni-iam-role-create-role.title. Specifying a role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.
- The --resolve-conflicts PRESERVE option preserves existing configuration values for the add-on. If you’ve set custom values for add-on settings, and you don’t use this option, Amazon EKS overwrites your values with its default values. If you use this option, then we recommend testing any field and value changes on a non-production cluster before updating the add-on on your production cluster. If you change this value to OVERWRITE, all settings are changed to Amazon EKS default values. If you’ve set custom values for any settings, they might be overwritten with Amazon EKS default values. If you change this value to none, Amazon EKS doesn’t change the value of any settings, but the update might fail. If the update fails, you receive an error message to help you resolve the conflict.
- If you’re not updating a configuration setting, remove --configuration-values '{"env":{"AWS_VPC_K8S_CNI_EXTERNALSNAT":"true"}}' from the command. If you’re updating a configuration setting, replace "env":{"AWS_VPC_K8S_CNI_EXTERNALSNAT":"true"} with the setting that you want to set. In this example, the AWS_VPC_K8S_CNI_EXTERNALSNAT environment variable is set to true. The value that you specify must be valid for the configuration schema. If you don’t know the configuration schema, run aws eks describe-addon-configuration --addon-name vpc-cni --addon-version v1.19.0-eksbuild.1, replacing v1.19.0-eksbuild.1 with the version number of the add-on that you want to see the configuration for. The schema is returned in the output. If you have any existing custom configuration, want to remove it all, and set the values for all settings back to Amazon EKS defaults, remove "env":{"AWS_VPC_K8S_CNI_EXTERNALSNAT":"true"} from the command, so that you have empty {}. For an explanation of each setting, see CNI Configuration Variables on GitHub.
  aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version v1.19.0-eksbuild.1 \ --service-account-role-arn region.arniam::111122223333:role/AmazonEKSVPCCNIRole \ --resolve-conflicts PRESERVE --configuration-values '{"env":{"AWS_VPC_K8S_CNI_EXTERNALSNAT":"true"}}'
  It might take several seconds for the update to complete.

Confirm that the add-on version was updated. Replace my-cluster with the name of your cluster.

aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni

It might take several seconds for the update to complete.

An example output is as follows.

{
    "addon": {
        "addonName": "vpc-cni",
        "clusterName": "my-cluster",
        "status": "ACTIVE",
        "addonVersion": "v1.19.0-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "region.arneks:region:111122223333:addon/my-cluster/vpc-cni/74c33d2f-b4dc-8718-56e7-9fdfa65d14a9",
        "createdAt": "2023-04-12T18:25:19.319000+00:00",
        "modifiedAt": "2023-04-12T18:40:28.683000+00:00",
        "serviceAccountRoleArn": "region.arniam::111122223333:role/AmazonEKSVPCCNIRole",
        "tags": {},
        "configurationValues": "{\"env\":{\"AWS_VPC_K8S_CNI_EXTERNALSNAT\":\"true\"}}"
    }
}

Update the Amazon VPC CNI (self-managed add-on)

We recommend adding the Amazon EKS type of the add-on to your cluster instead of using the self-managed type of the add-on. If you’re not familiar with the difference between the types, see eks-add-ons.title. For more information about adding an Amazon EKS add-on to your cluster, see creating-an-add-on.title. If you’re unable to use the Amazon EKS add-on, we encourage you to submit an issue about why you can’t to the Containers roadmap GitHub repository.

Confirm that you don’t have the Amazon EKS type of the add-on installed on your cluster. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni --query addon.addonVersion --output text
```
If an error message is returned, you don’t have the Amazon EKS type of the add-on installed on your cluster. To self-manage the add-on, complete the remaining steps in this procedure to update the add-on. If a version number is returned, you have the Amazon EKS type of the add-on installed on your cluster. To update it, use the procedure in updating-an-add-on.title, rather than using this procedure. If you’re not familiar with the differences between the add-on types, see eks-add-ons.title.
See which version of the container image is currently installed on your cluster.
```
kubectl describe daemonset aws-node --namespace kube-system | grep amazon-k8s-cni: | cut -d : -f 3
```
An example output is as follows.
```
v1.16.4-eksbuild.2
```
Your output might not include the build number.
Backup your current settings so you can configure the same settings once you’ve updated your version.
```
kubectl get daemonset aws-node -n kube-system -o yaml > aws-k8s-cni-old.yaml
```
To review the available versions and familiarize yourself with the changes in the version that you want to update to, see releases on GitHub. Note that we recommend updating to the same major.minor.patch version listed in the latest available versions table, even if later versions are available on GitHub. For the latest available version table, see vpc-cni-latest-available-version.title. The build versions listed in the table aren’t specified in the self-managed versions listed on GitHub. Update your version by completing the tasks in one of the following options:
- If you don’t have any custom settings for the add-on, then run the command under the To apply this release: heading on GitHub for the release that you’re updating to.
- If you have custom settings, download the manifest file with the following command. Change https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.19.0/config/master/aws-k8s-cni.yaml to the URL for the release on GitHub that you’re updating to.
  curl -O https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.19.0/config/master/aws-k8s-cni.yaml
  If necessary, modify the manifest with the custom settings from the backup you made in a previous step and then apply the modified manifest to your cluster. If your nodes don’t have access to the private Amazon EKS Amazon ECR repositories that the images are pulled from (see the lines that start with image: in the manifest), then you’ll have to download the images, copy them to your own repository, and modify the manifest to pull the images from your repository. For more information, see copy-image-to-repository.title.
  kubectl apply -f aws-k8s-cni.yaml

Confirm that the new version is now installed on your cluster.

kubectl describe daemonset aws-node --namespace kube-system | grep amazon-k8s-cni: | cut -d : -f 3

An example output is as follows.

v1.19.0

(Optional) Install the cni-metrics-helper to your cluster. It scrapes elastic network interface and IP address information, aggregates it at a cluster level, and publishes the metrics to Amazon CloudWatch. For more information, see cni-metrics-helper on GitHub.

Configure Amazon VPC CNI plugin to use IRSA

Learn how to configure the Amazon VPC CNI plugin for Kubernetes to use IAM roles for service accounts (IRSA) for Pod networking in Amazon EKS clusters.

The Amazon VPC CNI plugin for Kubernetes is the networking plugin for Pod networking in Amazon EKS clusters. The plugin is responsible for allocating VPC IP addresses to Kubernetes nodes and configuring the necessary networking for Pods on each node. The plugin:

Requires AWS Identity and Access Management (IAM) permissions. If your cluster uses the IPv4 family, the permissions are specified in the ` AmazonEKS_CNI_Policy` AWS managed policy.If your cluster uses the IPv6 family, then the permissions must be added to an IAM policy that you create; for instructions, see cni-iam-role-create-ipv6-policy.title. You can attach the policy to the Amazon EKS node IAM role, or to a separate IAM role. For instructions to attach the policy to the Amazon EKS node IAM role, see create-node-role.title. We recommend that you assign it to a separate role, as detailed in this topic.
Creates and is configured to use a Kubernetes service account named aws-node when it’s deployed. The service account is bound to a Kubernetes clusterrole named aws-node, which is assigned the required Kubernetes permissions.

The Pods for the Amazon VPC CNI plugin for Kubernetes have access to the permissions assigned to the Amazon EKS node IAM role, unless you block access to IMDS. For more information, see Restrict access to the instance profile assigned to the worker node.

An existing Amazon EKS cluster. To deploy one, see getting-started.title.
An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.

Step 1: Create the `Amazon VPC CNI plugin for Kubernetes` IAM role

Determine the IP family of your cluster.
```
aws eks describe-cluster --name my-cluster | grep ipFamily
```
An example output is as follows.
```
"ipFamily": "ipv4"
```
The output may return ipv6 instead.

Create the IAM role. You can use eksctl or kubectl and the AWS CLI to create your IAM role.

eksctl

Create an IAM role and attach the IAM policy to the role with the command that matches the IP family of your cluster. The command creates and deploys an AWS CloudFormation stack that creates an IAM role, attaches the policy that you specify to it, and annotates the existing aws-node Kubernetes service account with the ARN of the IAM role that is created.
- IPv4
  
  Replace my-cluster with your own value.
  eksctl create iamserviceaccount \ --name aws-node \ --namespace kube-system \ --cluster my-cluster \ --role-name AmazonEKSVPCCNIRole \ --attach-policy-arn region.arniam::aws:policy/AmazonEKS_CNI_Policy \ --override-existing-serviceaccounts \ --approve
- IPv6
  
  Replace my-cluster with your own value. Replace 111122223333 with your account ID and replace AmazonEKS_CNI_IPv6_Policy with the name of your IPv6 policy. If you don’t have an IPv6 policy, see cni-iam-role-create-ipv6-policy.title to create one. To use IPv6 with your cluster, it must meet several requirements. For more information, see cni-ipv6.title.
  eksctl create iamserviceaccount \ --name aws-node \ --namespace kube-system \ --cluster my-cluster \ --role-name AmazonEKSVPCCNIRole \ --attach-policy-arn region.arniam::111122223333:policy/AmazonEKS_CNI_IPv6_Policy \ --override-existing-serviceaccounts \ --approve

kubectl and the AWS CLI

View your cluster’s OIDC provider URL.

aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text

An example output is as follows.

https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE

If no output is returned, then you must create an IAM OIDC provider for your cluster.

Copy the following contents to a file named vpc-cni-trust-policy.json. Replace 111122223333 with your account ID and EXAMPLED539D4633E53DE1B71EXAMPLE with the output returned in the previous step. Replace region-code with the AWS Region that your cluster is in.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "region.arniam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:aws-node"
                }
            }
        }
    ]
}

Create the role. You can replace AmazonEKSVPCCNIRole with any name that you choose.

aws iam create-role \
  --role-name AmazonEKSVPCCNIRole \
  --assume-role-policy-document file://"vpc-cni-trust-policy.json"

Attach the required IAM policy to the role. Run the command that matches the IP family of your cluster.
- IPv4
  aws iam attach-role-policy \ --policy-arn region.arniam::aws:policy/AmazonEKS_CNI_Policy \ --role-name AmazonEKSVPCCNIRole
- IPv6
  
  Replace 111122223333 with your account ID and AmazonEKS_CNI_IPv6_Policy with the name of your IPv6 policy. If you don’t have an IPv6 policy, see cni-iam-role-create-ipv6-policy.title to create one. To use IPv6 with your cluster, it must meet several requirements. For more information, see cni-ipv6.title.
  aws iam attach-role-policy \ --policy-arn region.arniam::111122223333:policy/AmazonEKS_CNI_IPv6_Policy \ --role-name AmazonEKSVPCCNIRole
Run the following command to annotate the aws-node service account with the ARN of the IAM role that you created previously. Replace the example values with your own values.
```
kubectl annotate serviceaccount \
    -n kube-system aws-node \
    eks.amazonaws.com/role-arn=region.arniam::111122223333:role/AmazonEKSVPCCNIRole
```

(Optional) Configure the AWS Security Token Service endpoint type used by your Kubernetes service account. For more information, see configure-sts-endpoint.title.

Step 2: Re-deploy `Amazon VPC CNI plugin for Kubernetes` `Pods`

Delete and re-create any existing Pods that are associated with the service account to apply the credential environment variables. The annotation is not applied to Pods that are currently running without the annotation. The following command deletes the existing aws-node DaemonSet Pods and deploys them with the service account annotation.
```
kubectl delete Pods -n kube-system -l k8s-app=aws-node
```

Confirm that the Pods all restarted.

kubectl get pods -n kube-system -l k8s-app=aws-node

Describe one of the Pods and verify that the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables exist. Replace cpjw7 with the name of one of your Pods returned in the output of the previous step.

kubectl describe pod -n kube-system aws-node-cpjw7 | grep 'AWS_ROLE_ARN:\|AWS_WEB_IDENTITY_TOKEN_FILE:'

An example output is as follows.

AWS_ROLE_ARN:                 region.arniam::111122223333:role/AmazonEKSVPCCNIRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      AWS_ROLE_ARN:                           region.arniam::111122223333:role/AmazonEKSVPCCNIRole
      AWS_WEB_IDENTITY_TOKEN_FILE:            /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Two sets of duplicate results are returned because the Pod contains two containers. Both containers have the same values.

If your Pod is using the AWS Regional endpoint, then the following line is also returned in the previous output.

AWS_STS_REGIONAL_ENDPOINTS=regional

Step 3: Remove the CNI policy from the node IAM role

If your Amazon EKS node IAM role currently has the AmazonEKS_CNI_Policy IAM (IPv4) policyor an IPv6 policyattached to it, and you’ve created a separate IAM role, attached the policy to it instead, and assigned it to the aws-node Kubernetes service account, then we recommend that you remove the policy from your node role with the AWS CLI command that matches the IP family of your cluster. Replace AmazonEKSNodeRole with the name of your node role.

IPv4

aws iam detach-role-policy --role-name AmazonEKSNodeRole --policy-arn region.arniam::aws:policy/AmazonEKS_CNI_Policy

IPv6

Replace 111122223333 with your account ID and AmazonEKS_CNI_IPv6_Policy with the name of your IPv6 policy.

aws iam detach-role-policy --role-name AmazonEKSNodeRole --policy-arn region.arniam::111122223333:policy/AmazonEKS_CNI_IPv6_Policy

Create IAM policy for clusters that use the `IPv6` family

If you created a cluster that uses the IPv6 family and the cluster has version 1.10.1 or later of the Amazon VPC CNI plugin for Kubernetes add-on configured, then you need to create an IAM policy that you can assign to an IAM role. If you have an existing cluster that you didn’t configure with the IPv6 family when you created it, then to use IPv6, you must create a new cluster. For more information about using IPv6 with your cluster, see cni-ipv6.title.

Copy the following text and save it to a file named vpc-cni-ipv6-policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AssignIpv6Addresses",
                "ec2:DescribeInstances",
                "ec2:DescribeTags",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInstanceTypes"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": [
                "region.arnec2:*:*:network-interface/*"
            ]
        }
    ]
}

Create the IAM policy.

aws iam create-policy --policy-name AmazonEKS_CNI_IPv6_Policy --policy-document file://vpc-cni-ipv6-policy.json

Learn about VPC CNI modes and configuration

Learn about IPv6 addresses to clusters, `Pods`, and services

Deploying an Amazon EKS IPv6 cluster and managed Amazon Linux nodes

In this tutorial, you deploy an IPv6 Amazon VPC, an Amazon EKS cluster with the IPv6 family, and a managed node group with Amazon EC2 Amazon Linux nodes. You can’t deploy Amazon EC2 Windows nodes in an IPv6 cluster. You can also deploy Fargate nodes to your cluster, though those instructions aren’t provided in this topic for simplicity.

Prerequisites

Complete the following before you start the tutorial:

Install and configure the following tools and resources that you need to create and manage an Amazon EKS cluster.

We recommend that you familiarize yourself with all settings and deploy a cluster with the settings that meet your requirements. For more information, see create-cluster.title, managed-node-groups.title, and the Considerations for this topic. You can only enable some settings when creating your cluster.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.
The IAM security principal that you’re using must have permissions to work with Amazon EKS IAM roles, service linked roles, AWS CloudFormation, a VPC, and related resources. For more information, see Actions and Using service-linked roles in the IAM User Guide.
If you use the eksctl, install version 0.199.0 or later on your computer. To install or update to it, see Installation in the eksctl documentation.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide. If you use the AWS CloudShell, you may need to install version 2.12.3 or later or 1.27.160 or later of the AWS CLI, because the default AWS CLI version installed in the AWS CloudShell may be an earlier version.

You can use the eksctl or CLI to deploy an IPv6 cluster.

Deploy an IPv6 cluster with eksctl

Create the ipv6-cluster.yaml file. Copy the command that follows to your device. Make the following modifications to the command as needed and then run the modified command:
- Replace my-cluster with a name for your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
- Replace region-code with any AWS Region that is supported by Amazon EKS. For a list of AWS Regions, see Amazon EKS endpoints and quotas in the AWS General Reference guide.
- The value for version with the version of your cluster. For more information, see kubernetes-versions.title.
- Replace my-nodegroup with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
- Replace t3.medium with any AWS Nitro System instance type.
  cat >ipv6-cluster.yaml <<EOF --- apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: my-cluster region: region-code version: "X.XX" kubernetesNetworkConfig: ipFamily: IPv6 addons: - name: vpc-cni version: latest - name: coredns version: latest - name: kube-proxy version: latest iam: withOIDC: true managedNodeGroups: - name: my-nodegroup instanceType: t3.medium EOF
Create your cluster.
```
eksctl create cluster -f ipv6-cluster.yaml
```
Cluster creation takes several minutes. Don’t proceed until you see the last line of output, which looks similar to the following output.
```
[...]
[✓]  EKS cluster "my-cluster" in "region-code" region is ready
```

Confirm that default Pods are assigned IPv6 addresses.

kubectl get pods -n kube-system -o wide

An example output is as follows.

NAME                       READY   STATUS    RESTARTS   AGE     IP                                       NODE                                            NOMINATED NODE   READINESS GATES
aws-node-rslts             1/1     Running   1          5m36s   2600:1f13:b66:8200:11a5:ade0:c590:6ac8   ip-192-168-34-75.region-code.compute.internal   <none>           <none>
aws-node-t74jh             1/1     Running   0          5m32s   2600:1f13:b66:8203:4516:2080:8ced:1ca9   ip-192-168-253-70.region-code.compute.internal  <none>           <none>
coredns-85d5b4454c-cw7w2   1/1     Running   0          56m     2600:1f13:b66:8203:34e5::                ip-192-168-253-70.region-code.compute.internal  <none>           <none>
coredns-85d5b4454c-tx6n8   1/1     Running   0          56m     2600:1f13:b66:8203:34e5::1               ip-192-168-253-70.region-code.compute.internal  <none>           <none>
kube-proxy-btpbk           1/1     Running   0          5m36s   2600:1f13:b66:8200:11a5:ade0:c590:6ac8   ip-192-168-34-75.region-code.compute.internal   <none>           <none>
kube-proxy-jjk2g           1/1     Running   0          5m33s   2600:1f13:b66:8203:4516:2080:8ced:1ca9   ip-192-168-253-70.region-code.compute.internal  <none>           <none>

Confirm that default services are assigned IPv6 addresses.

kubectl get services -n kube-system -o wide

An example output is as follows.

NAME       TYPE        CLUSTER-IP          EXTERNAL-IP   PORT(S)         AGE   SELECTOR
kube-dns   ClusterIP   fd30:3087:b6c2::a   <none>        53/UDP,53/TCP   57m   k8s-app=kube-dns

(Optional) Deploy a sample application or deploy the AWS Load Balancer Controller and a sample application to load balance HTTP applications with alb-ingress.title or network traffic with network-load-balancing.title to IPv6 Pods.
After you’ve finished with the cluster and nodes that you created for this tutorial, you should clean up the resources that you created with the following command.
```
eksctl delete cluster my-cluster
```

Deploy an IPv6 cluster with AWS CLI

You must complete all steps in this procedure as the same user. To check the current user, run the following command:
```
aws sts get-caller-identity
```
You must complete all steps in this procedure in the same shell. Several steps use variables set in previous steps. Steps that use variables won’t function properly if the variable values are set in a different shell. If you use the AWS CloudShell to complete the following procedure, remember that if you don’t interact with it using your keyboard or pointer for approximately 20–30 minutes, your shell session ends. Running processes do not count as interactions.
The instructions are written for the Bash shell, and may need adjusting in other shells.

Replace all example values in the steps of this procedure with your own values.

Run the following commands to set some variables used in later steps. Replace region-code with the AWS Region that you want to deploy your resources in. The value can be any AWS Region that is supported by Amazon EKS. For a list of AWS Regions, see Amazon EKS endpoints and quotas in the AWS General Reference guide. Replace my-cluster with a name for your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in. Replace my-nodegroup with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. Replace 111122223333 with your account ID.
```
export region_code=region-code
export cluster_name=my-cluster
export nodegroup_name=my-nodegroup
export account_id=111122223333
```

Create an Amazon VPC with public and private subnets that meets Amazon EKS and IPv6 requirements.

Run the following command to set a variable for your AWS CloudFormation stack name. You can replace my-eks-ipv6-vpc with any name you choose.
```
export vpc_stack_name=my-eks-ipv6-vpc
```

Create an IPv6 VPC using an AWS CloudFormation template.

aws cloudformation create-stack --region $region_code --stack-name $vpc_stack_name \
  --template-url https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-ipv6-vpc-public-private-subnets.yaml

The stack takes a few minutes to create. Run the following command. Don’t continue to the next step until the output of the command is CREATE_COMPLETE.

aws cloudformation describe-stacks --region $region_code --stack-name $vpc_stack_name --query Stacks[].StackStatus --output text

Retrieve the IDs of the public subnets that were created.

aws cloudformation describe-stacks --region $region_code --stack-name $vpc_stack_name \
    --query='Stacks[].Outputs[?OutputKey==`SubnetsPublic`].OutputValue' --output text

An example output is as follows.

subnet-0a1a56c486EXAMPLE,subnet-099e6ca77aEXAMPLE

Enable the auto-assign IPv6 address option for the public subnets that were created.

aws ec2 modify-subnet-attribute --region $region_code --subnet-id subnet-0a1a56c486EXAMPLE --assign-ipv6-address-on-creation
aws ec2 modify-subnet-attribute --region $region_code --subnet-id subnet-099e6ca77aEXAMPLE --assign-ipv6-address-on-creation

Retrieve the names of the subnets and security groups created by the template from the deployed AWS CloudFormation stack and store them in variables for use in a later step.

security_groups=$(aws cloudformation describe-stacks --region $region_code --stack-name $vpc_stack_name \
    --query='Stacks[].Outputs[?OutputKey==`SecurityGroups`].OutputValue' --output text)

public_subnets=$(aws cloudformation describe-stacks --region $region_code --stack-name $vpc_stack_name \
    --query='Stacks[].Outputs[?OutputKey==`SubnetsPublic`].OutputValue' --output text)

private_subnets=$(aws cloudformation describe-stacks --region $region_code --stack-name $vpc_stack_name \
    --query='Stacks[].Outputs[?OutputKey==`SubnetsPrivate`].OutputValue' --output text)

subnets=${public_subnets},${private_subnets}

Run the following command to create the eks-cluster-role-trust-policy.json file.

cat >eks-cluster-role-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Run the following command to set a variable for your role name. You can replace myAmazonEKSClusterRole with any name you choose.
```
export cluster_role_name=myAmazonEKSClusterRole
```

Create the role.

aws iam create-role --role-name $cluster_role_name --assume-role-policy-document file://"eks-cluster-role-trust-policy.json"

Retrieve the ARN of the IAM role and store it in a variable for a later step.

CLUSTER_IAM_ROLE=$(aws iam get-role --role-name $cluster_role_name --query="Role.Arn" --output text)

Attach the required Amazon EKS managed IAM policy to the role.

aws iam attach-role-policy --policy-arn region.arniam::aws:policy/AmazonEKSClusterPolicy --role-name $cluster_role_name

Create your cluster.
```
aws eks create-cluster --region $region_code --name $cluster_name --kubernetes-version 1.XX \
   --role-arn $CLUSTER_IAM_ROLE --resources-vpc-config subnetIds=$subnets,securityGroupIds=$security_groups \
   --kubernetes-network-config ipFamily=ipv6
```
1. NOTE: You might receive an error that one of the Availability Zones in your request doesn’t have sufficient capacity to create an Amazon EKS cluster. If this happens, the error output contains the Availability Zones that can support a new cluster. Retry creating your cluster with at least two subnets that are located in the supported Availability Zones for your account. For more information, see ice.title.
  
  The cluster takes several minutes to create. Run the following command. Don’t continue to the next step until the output from the command is ACTIVE.
  aws eks describe-cluster --region $region_code --name $cluster_name --query cluster.status
Create or update a kubeconfig file for your cluster so that you can communicate with your cluster.
```
aws eks update-kubeconfig --region $region_code --name $cluster_name
```
By default, the config file is created in ~/.kube or the new cluster’s configuration is added to an existing config file in ~/.kube.

Create a node IAM role.

Run the following command to create the vpc-cni-ipv6-policy.json file.

cat >vpc-cni-ipv6-policy <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AssignIpv6Addresses",
                "ec2:DescribeInstances",
                "ec2:DescribeTags",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInstanceTypes"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": [
                "region.arnec2:*:*:network-interface/*"
            ]
        }
    ]
}
EOF

Create the IAM policy.

aws iam create-policy --policy-name AmazonEKS_CNI_IPv6_Policy --policy-document file://vpc-cni-ipv6-policy.json

Run the following command to create the node-role-trust-relationship.json file.

cat >node-role-trust-relationship.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Run the following command to set a variable for your role name. You can replace AmazonEKSNodeRole with any name you choose.
```
export node_role_name=AmazonEKSNodeRole
```

Create the IAM role.

aws iam create-role --role-name $node_role_name --assume-role-policy-document file://"node-role-trust-relationship.json"

Attach the IAM policy to the IAM role.

aws iam attach-role-policy --policy-arn region.arniam::$account_id:policy/AmazonEKS_CNI_IPv6_Policy \
    --role-name $node_role_name

For simplicity in this tutorial, the policy is attached to this IAM role. In a production cluster however, we recommend attaching the policy to a separate IAM role. For more information, see cni-iam-role.title.

Attach two required IAM managed policies to the IAM role.

aws iam attach-role-policy --policy-arn region.arniam::aws:policy/AmazonEKSWorkerNodePolicy \
  --role-name $node_role_name
aws iam attach-role-policy --policy-arn region.arniam::aws:policy/AmazonEC2ContainerRegistryReadOnly \
  --role-name $node_role_name

Retrieve the ARN of the IAM role and store it in a variable for a later step.

node_iam_role=$(aws iam get-role --role-name $node_role_name --query="Role.Arn" --output text)

Create a managed node group.
1. View the IDs of the subnets that you created in a previous step.
  echo $subnets
  An example output is as follows.
  subnet-0a1a56c486EXAMPLE,subnet-099e6ca77aEXAMPLE,subnet-0377963d69EXAMPLE,subnet-0c05f819d5EXAMPLE
2. Create the node group. Replace 0a1a56c486EXAMPLE, 099e6ca77aEXAMPLE, 0377963d69EXAMPLE, and 0c05f819d5EXAMPLE with the values returned in the output of the previous step. Be sure to remove the commas between subnet IDs from the previous output in the following command. You can replace t3.medium with any AWS Nitro System instance type.
  aws eks create-nodegroup --region $region_code --cluster-name $cluster_name --nodegroup-name $nodegroup_name \ --subnets subnet-0a1a56c486EXAMPLE subnet-099e6ca77aEXAMPLE subnet-0377963d69EXAMPLE subnet-0c05f819d5EXAMPLE \ --instance-types t3.medium --node-role $node_iam_role
  The node group takes a few minutes to create. Run the following command. Don’t proceed to the next step until the output returned is ACTIVE.
  aws eks describe-nodegroup --region $region_code --cluster-name $cluster_name --nodegroup-name $nodegroup_name \ --query nodegroup.status --output text

Confirm that the default Pods are assigned IPv6 addresses in the IP column.

kubectl get pods -n kube-system -o wide

An example output is as follows.

NAME                       READY   STATUS    RESTARTS   AGE     IP                                       NODE                                            NOMINATED NODE   READINESS GATES
aws-node-rslts             1/1     Running   1          5m36s   2600:1f13:b66:8200:11a5:ade0:c590:6ac8   ip-192-168-34-75.region-code.compute.internal   <none>           <none>
aws-node-t74jh             1/1     Running   0          5m32s   2600:1f13:b66:8203:4516:2080:8ced:1ca9   ip-192-168-253-70.region-code.compute.internal  <none>           <none>
coredns-85d5b4454c-cw7w2   1/1     Running   0          56m     2600:1f13:b66:8203:34e5::                ip-192-168-253-70.region-code.compute.internal  <none>           <none>
coredns-85d5b4454c-tx6n8   1/1     Running   0          56m     2600:1f13:b66:8203:34e5::1               ip-192-168-253-70.region-code.compute.internal  <none>           <none>
kube-proxy-btpbk           1/1     Running   0          5m36s   2600:1f13:b66:8200:11a5:ade0:c590:6ac8   ip-192-168-34-75.region-code.compute.internal   <none>           <none>
kube-proxy-jjk2g           1/1     Running   0          5m33s   2600:1f13:b66:8203:4516:2080:8ced:1ca9   ip-192-168-253-70.region-code.compute.internal  <none>           <none>

Confirm that the default services are assigned IPv6 addresses in the IP column.

kubectl get services -n kube-system -o wide

An example output is as follows.

NAME       TYPE        CLUSTER-IP          EXTERNAL-IP   PORT(S)         AGE   SELECTOR
kube-dns   ClusterIP   fd30:3087:b6c2::a   <none>        53/UDP,53/TCP   57m   k8s-app=kube-dns

(Optional) Deploy a sample application or deploy the AWS Load Balancer Controller and a sample application to load balance HTTP applications with alb-ingress.title or network traffic with network-load-balancing.title to IPv6 Pods.

After you’ve finished with the cluster and nodes that you created for this tutorial, you should clean up the resources that you created with the following commands. Make sure that you’re not using any of the resources outside of this tutorial before deleting them.

If you’re completing this step in a different shell than you completed the previous steps in, set the values of all the variables used in previous steps, replacing the example values with the values you specified when you completed the previous steps. If you’re completing this step in the same shell that you completed the previous steps in, skip to the next step.
```
export region_code=region-code
export vpc_stack_name=my-eks-ipv6-vpc
export cluster_name=my-cluster
export nodegroup_name=my-nodegroup
export account_id=111122223333
export node_role_name=AmazonEKSNodeRole
export cluster_role_name=myAmazonEKSClusterRole
```

Delete your node group.

aws eks delete-nodegroup --region $region_code --cluster-name $cluster_name --nodegroup-name $nodegroup_name

Deletion takes a few minutes. Run the following command. Don’t proceed to the next step if any output is returned.

aws eks list-nodegroups --region $region_code --cluster-name $cluster_name --query nodegroups --output text

Delete the cluster.

aws eks delete-cluster --region $region_code --name $cluster_name

The cluster takes a few minutes to delete. Before continuing make sure that the cluster is deleted with the following command.

aws eks describe-cluster --region $region_code --name $cluster_name

Don’t proceed to the next step until your output is similar to the following output.

An error occurred (ResourceNotFoundException) when calling the DescribeCluster operation: No cluster found for name: my-cluster.

Delete the IAM resources that you created. Replace AmazonEKS_CNI_IPv6_Policy with the name you chose, if you chose a different name than the one used in previous steps.

aws iam detach-role-policy --role-name $cluster_role_name --policy-arn region.arniam::aws:policy/AmazonEKSClusterPolicy
aws iam detach-role-policy --role-name $node_role_name --policy-arn region.arniam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam detach-role-policy --role-name $node_role_name --policy-arn region.arniam::aws:policy/AmazonEC2ContainerRegistryReadOnly
aws iam detach-role-policy --role-name $node_role_name --policy-arn region.arniam::$account_id:policy/AmazonEKS_CNI_IPv6_Policy
aws iam delete-policy --policy-arn region.arniam::$account_id:policy/AmazonEKS_CNI_IPv6_Policy
aws iam delete-role --role-name $cluster_role_name
aws iam delete-role --role-name $node_role_name

Delete the AWS CloudFormation stack that created the VPC.

aws cloudformation delete-stack --region $region_code --stack-name $vpc_stack_name

Learn how to deploy an IPv6 cluster and nodes with Amazon EKS for assigning IPv6 addresses to Pods and services instead of IPv4, leveraging IP prefix delegation and the latest Amazon VPC CNI plugin.

Applies to: Pods with Amazon EC2 instances and Fargate Pods

By default, Kubernetes assigns IPv4 addresses to your Pods and services. Instead of assigning IPv4 addresses to your Pods and services, you can configure your cluster to assign IPv6 addresses to them. Amazon EKS doesn’t support dual-stacked Pods or services, even though Kubernetes does in version 1.23 and later. As a result, you can’t assign both IPv4 and IPv6 addresses to your Pods and services.

You select which IP family you want to use for your cluster when you create it. You can’t change the family after you create the cluster.

For a tutorial to deploy an Amazon EKS IPv6 cluster, see deploy-ipv6-cluster.title.

The following are considerations for using the feature:

IPv6 Feature support

No Windows support: Windows Pods and services aren’t supported.
Nitro-based EC2 nodes required: You can only use IPv6 with AWS Nitro-based Amazon EC2 or Fargate nodes.
EC2 and Fargate nodes supported: You can use IPv6 with security-groups-for-pods.title with Amazon EC2 nodes and Fargate nodes.
Outposts not supported: You can’t use IPv6 with eks-outposts.title.
FSx for Lustre is not supported: The fsx-csi.title is not supported.
Instance Metadata Service not supported: Use of the Amazon EC2 Instance Metadata Service IPv6 endpoint is not supported with Amazon EKS.
Custom networking not supported: If you previously used cni-custom-network.title to help alleviate IP address exhaustion, you can use IPv6 instead. You can’t use custom networking with IPv6. If you use custom networking for network isolation, then you might need to continue to use custom networking and the IPv4 family for your clusters.

IP address assignments

Kubernetes services: Kubernetes services are only assigned an IPv6 addresses. They aren’t assigned IPv4 addresses.
Pods: Pods are assigned an IPv6 address and a host-local IPv4 address. The host-local IPv4 address is assigned by using a host-local CNI plugin chained with VPC CNI and the address is not reported to the Kubernetes control plane. It is only used when a pod needs to communicate with an external IPv4 resources in another Amazon VPC or the internet. The host-local IPv4 address gets SNATed (by VPC CNI) to the primary IPv4 address of the primary ENI of the worker node.
Pods and services: Pods and services are only assigned an IPv6 address. They aren’t assigned an IPv4 address. Because Pods are able to communicate to IPv4 endpoints through NAT on the instance itself, DNS64 and NAT64 aren’t needed. If the traffic needs a public IP address, the traffic is then source network address translated to a public IP.
Routing addresses: The source IPv6 address of a Pod isn’t source network address translated to the IPv6 address of the node when communicating outside of the VPC. It is routed using an internet gateway or egress-only internet gateway.
Nodes: All nodes are assigned an IPv4 and IPv6 address.
Fargate Pods: Each Fargate Pod receives an IPv6 address from the CIDR that’s specified for the subnet that it’s deployed in. The underlying hardware unit that runs Fargate Pods gets a unique IPv4 and IPv6 address from the CIDRs that are assigned to the subnet that the hardware unit is deployed in.

How to use IPv6 with EKS

Create new cluster: You must create a new cluster and specify that you want to use the IPv6 family for that cluster. You can’t enable the IPv6 family for a cluster that you updated from a previous version. For instructions on how to create a new cluster, see Considerations .
Use recent VPC CNI: Deploy Amazon VPC CNI version 1.10.1 or later. This version or later is deployed by default. After you deploy the add-on, you can’t downgrade your Amazon VPC CNI add-on to a version lower than 1.10.1 without first removing all nodes in all node groups in your cluster.
Configure VPC CNI for IPv6: If you use Amazon EC2 nodes, you must configure the Amazon VPC CNI add-on with IP prefix delegation and IPv6. If you choose the IPv6 family when creating your cluster, the 1.10.1 version of the add-on defaults to this configuration. This is the case for both a self-managed or Amazon EKS add-on. For more information about IP prefix delegation, see cni-increase-ip-addresses.title.
Configure IPv4 and IPv6 addresses: When you create a cluster, the VPC and subnets that you specify must have an IPv6 CIDR block that’s assigned to the VPC and subnets that you specify. They must also have an IPv4 CIDR block assigned to them. This is because, even if you only want to use IPv6, a VPC still requires an IPv4 CIDR block to function. For more information, see Associate an IPv6 CIDR block with your VPC in the Amazon VPC User Guide.
Auto-assign IPv6 addresses to nodes: When you create your nodes, you must specify subnets that are configured to auto-assign IPv6 addresses. Otherwise, you can’t deploy your nodes. By default, this configuration is disabled. For more information, see Modify the IPv6 addressing attribute for your subnet in the Amazon VPC User Guide.
Set route tables to use IPv6: The route tables that are assigned to your subnets must have routes for IPv6 addresses. For more information, see Migrate to IPv6 in the Amazon VPC User Guide.
Set security groups for IPv6: Your security groups must allow IPv6 addresses. For more information, see Migrate to IPv6 in the Amazon VPC User Guide.
Set up load balancer: Use version 2.3.1 or later of the AWS Load Balancer Controller to load balance HTTP applications using the alb-ingress.title or network traffic using the network-load-balancing.title to IPv6 Pods with either load balancer in IP mode, but not instance mode. For more information, see aws-load-balancer-controller.title.
Add IPv6 IAM policy: You must attach an IPv6 IAM policy to your node IAM or CNI IAM role. Between the two, we recommend that you attach it to a CNI IAM role. For more information, see cni-iam-role-create-ipv6-policy.title and cni-iam-role-create-role.title.
Evaluate all components: Perform a thorough evaluation of your applications, Amazon EKS add-ons, and AWS services that you integrate with before deploying IPv6 clusters. This is to ensure that everything works as expected with IPv6.
Add BootstrapArguments self-managed node groups: When creating a self-managed node group in a cluster that uses the IPv6 family, user-data must include the following BootstrapArguments for the bootstrap.sh file that runs at node start up. Replace your-cidr with the IPv6 CIDR range of your cluster’s VPC.
```
--ip-family ipv6 --service-ipv6-cidr your-cidr
```
If you don’t know the IPv6 CIDR range for your cluster, you can see it with the following command (requires the AWS CLI version 2.4.9 or later).
```
aws eks describe-cluster --name my-cluster --query cluster.kubernetesNetworkConfig.serviceIpv6Cidr --output text
```

Enable outbound internet access for `Pods`

Learn how Amazon EKS manages external communication for Pods using Source Network Address Translation (SNAT), allowing Pods to access internet resources or networks connected via VPC peering, Transit Gateway, or AWS Direct Connect.

Applies to: Linux IPv4 Fargate nodes, Linux nodes with Amazon EC2 instances

If you deployed your cluster using the IPv6 family, then the information in this topic isn’t applicable to your cluster, because IPv6 addresses are not network translated. For more information about using IPv6 with your cluster, see cni-ipv6.title.

By default, each Pod in your cluster is assigned a link:AWSEC2/latest/UserGuide/using-instance-addressing.html#concepts-private-addressesIPv4 address from a classless inter-domain routing (CIDR) block that is associated with the VPC that the Pod is deployed in. Pods in the same VPC communicate with each other using these private IP addresses as end points. When a Pod communicates to any IPv4 address that isn’t within a CIDR block that’s associated to your VPC, the Amazon VPC CNI plugin (for both Linux or Windows) translates the Pod’s IPv4 address to the primary private IPv4 address of the primary elastic network interface of the node that the Pod is running on, by default ^{^*}^.

For Windows nodes, there are additional details to consider. By default, the VPC CNI plugin for Windows is defined with a networking configuration in which the traffic to a destination within the same VPC is excluded for SNAT. This means that internal VPC communication has SNAT disabled and the IP address allocated to a Pod is routable inside the VPC. But traffic to a destination outside of the VPC has the source Pod IP SNAT’ed to the instance ENI’s primary IP address. This default configuration for Windows ensures that the pod can access networks outside of your VPC in the same way as the host instance.

Due to this behavior:

Your Pods can communicate with internet resources only if the node that they’re running on has a public or elastic IP address assigned to it and is in a public subnet. A public subnet’s associated route table has a route to an internet gateway. We recommend deploying nodes to private subnets, whenever possible.
For versions of the plugin earlier than 1.8.0, resources that are in networks or VPCs that are connected to your cluster VPC using VPC peering, a transit VPC, or AWS Direct Connect can’t initiate communication to your Pods behind secondary elastic network interfaces. Your Pods can initiate communication to those resources and receive responses from them, though.

If either of the following statements are true in your environment, then change the default configuration with the command that follows.

You have resources in networks or VPCs that are connected to your cluster VPC using VPC peering, a transit VPC, or AWS Direct Connect that need to initiate communication with your Pods using an IPv4 address and your plugin version is earlier than 1.8.0.
Your Pods are in a private subnet and need to communicate outbound to the internet. The subnet has a route to a NAT gateway.

kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true

The AWS_VPC_K8S_CNI_EXTERNALSNAT and AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS CNI configuration variables aren’t applicable to Windows nodes. Disabling SNAT isn’t supported for Windows. As for excluding a list of IPv4 CIDRs from SNAT, you can define this by specifying the ExcludedSnatCIDRs parameter in the Windows bootstrap script. For more information on using this parameter, see bootstrap-script-configuration-parameters.title.

Host networking

^{^*}^If a Pod’s spec contains hostNetwork=true (default is false), then its IP address isn’t translated to a different address. This is the case for the kube-proxy and Amazon VPC CNI plugin for Kubernetes Pods that run on your cluster, by default. For these Pods, the IP address is the same as the node’s primary IP address, so the Pod’s IP address isn’t translated. For more information about a Pod’s hostNetwork setting, see PodSpec v1 core in the Kubernetes API reference.

Limit `Pod` traffic with `Kubernetes` network policies

Restrict Pod network traffic with Kubernetes network policies

Learn how to deploy Kubernetes network policies on your Amazon EKS cluster.

You can use a Kubernetes network policy to restrict network traffic to and from your Pods. For more information, see Network Policies in the Kubernetes documentation.

You must configure the following in order to use this feature:

Set up policy enforcement at Pod startup. You do this in the aws-node container of the VPC CNI DaemonSet.
Enable the network policy parameter for the add-on.
Configure your cluster to use the Kubernetes network policy

Before you begin, review the considerations. For more information, see cni-network-policy-considerations.title.

Prerequisites

The following are prerequisites for the feature:

* .Minimum cluster version An existing Amazon EKS cluster. To deploy one, see getting-started.title. The cluster must be Kubernetes version 1.25 or later. The cluster must be running one of the Kubernetes versions and platform versions listed in the following table. Note that any Kubernetes and platform versions later than those listed are also supported. You can check your current Kubernetes version by replacing my-cluster in the following command with the name of your cluster and then running the modified command:

aws eks describe-cluster
              --name my-cluster --query cluster.version --output
              text

Kubernetes version Platform version

1.27.4

eks.5

1.26.7

eks.6

1.25.12

eks.7

* .Minimum VPC CNI version Version 1.14 or later of the Amazon VPC CNI plugin for Kubernetes on your cluster. You can see which version that you currently have with the following command.

kubectl describe daemonset aws-node --namespace kube-system | grep amazon-k8s-cni: | cut -d : -f 3

+ If your version is earlier than 1.14, see vpc-add-on-update.title to upgrade to version 1.14 or later. * .Minimum Linux kernel version Your nodes must have Linux kernel version 5.10 or later. You can check your kernel version with uname -r. If you’re using the latest versions of the Amazon EKS optimized Amazon Linux, Amazon EKS optimized accelerated Amazon Linux AMIs, and Bottlerocket AMIs, they already have the required kernel version.

+ The Amazon EKS optimized accelerated Amazon Linux AMI version v20231116 or later have kernel version 5.10.

Step 1: Set up policy enforcement at Pod startup

The Amazon VPC CNI plugin for Kubernetes configures network policies for pods in parallel with the pod provisioning. Until all of the policies are configured for the new pod, containers in the new pod will start with a default allow policy. This is called standard mode. A default allow policy means that all ingress and egress traffic is allowed to and from the new pods. For example, the pods will not have any firewall rules enforced (all traffic is allowed) until the new pod is updated with the active policies.

With the NETWORK_POLICY_ENFORCING_MODE variable set to strict, pods that use the VPC CNI start with a default deny policy, then policies are configured. This is called strict mode. In strict mode, you must have a network policy for every endpoint that your pods need to access in your cluster. Note that this requirement applies to the CoreDNS pods. The default deny policy isn’t configured for pods with Host networking.

You can change the default network policy by setting the environment variable NETWORK_POLICY_ENFORCING_MODE to strict in the aws-node container of the VPC CNI DaemonSet.

env:
  - name: NETWORK_POLICY_ENFORCING_MODE
    value: "strict"

Step 2: Enable the network policy parameter for the add-on

The network policy feature uses port 8162 on the node for metrics by default. Also, the feature used port 8163 for health probes. If you run another application on the nodes or inside pods that needs to use these ports, the app fails to run. In VPC CNI version v1.14.1 or later, you can change these ports.

Use the following procedure to enable the network policy parameter for the add-on.

consolelong

Open the Amazon EKS console.
In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the Amazon VPC CNI add-on for.
Choose the Add-ons tab.
Select the box in the top right of the add-on box and then choose Edit.
On the Configure name of add-on page:
1. Select a v1.14.0-eksbuild.3 or later version in the Version list.
2. Expand the Optional configuration settings.
3. Enter the JSON key "enableNetworkPolicy": and value "true" in Configuration values. The resulting text must be a valid JSON object. If this key and value are the only data in the text box, surround the key and value with curly braces { }.
  
  The following example has network policy feature enabled and metrics and health probes are set to the default port numbers:
  { "enableNetworkPolicy": "true", "nodeAgent": { "healthProbeBindAddr": "8163", "metricsBindAddr": "8162" } }

Helm

If you have installed the Amazon VPC CNI plugin for Kubernetes through helm, you can update the configuration to change the ports.

Run the following command to change the ports. Set the port number in the value for either key nodeAgent.metricsBindAddr or key nodeAgent.healthProbeBindAddr, respectively.
```
helm upgrade --set nodeAgent.metricsBindAddr=8162 --set nodeAgent.healthProbeBindAddr=8163 aws-vpc-cni --namespace kube-system eks/aws-vpc-cni
```

kubectl

Open the aws-node DaemonSet in your editor.

kubectl edit daemonset -n kube-system aws-node

Replace the port numbers in the following command arguments in the args: in the aws-network-policy-agent container in the VPC CNI aws-node daemonset manifest.
```
    - args:
            - --metrics-bind-addr=:8162
            - --health-probe-bind-addr=:8163
```

Step 3: Mount the Berkeley Packet Filter (BPF) file system on your nodes

You must mount the Berkeley Packet Filter (BPF) file system on each of your nodes.

If your cluster is version 1.27 or later, you can skip this step as all Amazon EKS optimized Amazon Linux and Bottlerocket AMIs for 1.27 or later have this feature already.

For all other cluster versions, if you upgrade the Amazon EKS optimized Amazon Linux to version v20230703 or later or you upgrade the Bottlerocket AMI to version v1.0.2 or later, you can skip this step.

Mount the Berkeley Packet Filter (BPF) file system on each of your nodes.
```
sudo mount -t bpf bpffs /sys/fs/bpf
```
Then, add the same command to your user data in your launch template for your Amazon EC2 Auto Scaling Groups.

Step 4: Configure your cluster to use Kubernetes network policies

Configure the cluster to use Kubernetes network policies. You can set this for an Amazon EKS add-on or self-managed add-on.

Amazon EKS add-on

consolelong

Open the Amazon EKS console.
In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the Amazon VPC CNI add-on for.
Choose the Add-ons tab.
Select the box in the top right of the add-on box and then choose Edit.
On the Configure name of addon page:
1. Select a v1.14.0-eksbuild.3 or later version in the Version list.
2. Expand the Optional configuration settings.
3. Enter the JSON key "enableNetworkPolicy": and value "true" in Configuration values. The resulting text must be a valid JSON object. If this key and value are the only data in the text box, surround the key and value with curly braces { }. The following example shows network policy is enabled:
  { "enableNetworkPolicy": "true" }
  The following screenshot shows an example of this scenario.

AWS CLI

Run the following AWS CLI command. Replace my-cluster with the name of your cluster and the IAM role ARN with the role that you are using.

aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version v1.14.0-eksbuild.3 \
    --service-account-role-arn region.arniam::123456789012:role/AmazonEKSVPCCNIRole \
    --resolve-conflicts PRESERVE --configuration-values '{"enableNetworkPolicy": "true"}'

Self-managed add-on

Helm

If you have installed the Amazon VPC CNI plugin for Kubernetes through helm, you can update the configuration to enable network policy.

Run the following command to enable network policy.

helm upgrade --set enableNetworkPolicy=true aws-vpc-cni --namespace kube-system eks/aws-vpc-cni

kubectl

Open the amazon-vpc-cni ConfigMap in your editor.

kubectl edit configmap -n kube-system amazon-vpc-cni -o yaml

Add the following line to the data in the ConfigMap.

enable-network-policy-controller: "true"

Once you’ve added the line, your ConfigMap should look like the following example.

apiVersion: v1
 kind: ConfigMap
 metadata:
  name: amazon-vpc-cni
  namespace: kube-system
 data:
  enable-network-policy-controller: "true"

Open the aws-node DaemonSet in your editor.

kubectl edit daemonset -n kube-system aws-node

Replace the false with true in the command argument --enable-network-policy=false in the args: in the aws-network-policy-agent container in the VPC CNI aws-node daemonset manifest.
```
     - args:
        - --enable-network-policy=true
```

Step 5. Next steps

After you complete the configuration, confirm that the aws-node pods are running on your cluster.

kubectl get pods -n kube-system | grep 'aws-node\|amazon'

An example output is as follows.

aws-node-gmqp7                                          2/2     Running   1 (24h ago)   24h
aws-node-prnsh                                          2/2     Running   1 (24h ago)   24h

There are 2 containers in the aws-node pods in versions 1.14 and later. In previous versions and if network policy is disabled, there is only a single container in the aws-node pods.

You can now deploy Kubernetes network policies to your cluster.

To implement Kubernetes network policies you create Kubernetes NetworkPolicy objects and deploy them to your cluster. NetworkPolicy objects are scoped to a namespace. You implement policies to allow or deny traffic between Pods based on label selectors, namespaces, and IP address ranges. For more information about creating NetworkPolicy objects, see Network Policies in the Kubernetes documentation.

Enforcement of Kubernetes NetworkPolicy objects is implemented using the Extended Berkeley Packet Filter (eBPF). Relative to iptables based implementations, it offers lower latency and performance characteristics, including reduced CPU utilization and avoiding sequential lookups. Additionally, eBPF probes provide access to context rich data that helps debug complex kernel level issues and improve observability. Amazon EKS supports an eBPF-based exporter that leverages the probes to log policy results on each node and export the data to external log collectors to aid in debugging. For more information, see the eBPF documentation.

Disable Kubernetes network policies for Amazon EKS Pod network traffic

Learn how to disable Kubernetes network policies for Amazon EKS Pod network traffic.

Disable Kubernetes network policies to stop restricting Amazon EKS Pod network traffic

List all Kubernetes network policies.
```
kubectl get netpol -A
```
Delete each Kubernetes network policy. You must delete all network policies before disabling network policies.
```
kubectl delete netpol <policy-name>
```

Open the aws-node DaemonSet in your editor.

kubectl edit daemonset -n kube-system aws-node

Replace the true with false in the command argument --enable-network-policy=true in the args: in the aws-network-policy-agent container in the VPC CNI aws-node daemonset manifest.
```
     - args:
        - --enable-network-policy=true
```

Troubleshooting Kubernetes network policies For Amazon EKS

Learn how to troubleshoot and investigate network connections that use network policies.

You can troubleshoot and investigate network connections that use network policies by reading the Network policy logs and by running tools from the eBPF SDK.

Network policy logs

Whether connections are allowed or denied by a network policies is logged in flow logs. The network policy logs on each node include the flow logs for every pod that has a network policy. Network policy logs are stored at /var/log/aws-routed-eni/network-policy-agent.log. The following example is from a network-policy-agent.log file:

{"level":"info","timestamp":"2023-05-30T16:05:32.573Z","logger":"ebpf-client","msg":"Flow Info: ","Src
IP":"192.168.87.155","Src Port":38971,"Dest IP":"64.6.160","Dest
Port":53,"Proto":"UDP","Verdict":"ACCEPT"}

Network policy logs are disabled by default. To enable the network policy logs, follow these steps:

Network policy logs require an additional 1 vCPU for the aws-network-policy-agent container in the VPC CNI aws-node daemonset manifest.

Amazon EKS add-on

consolelong

Open the Amazon EKS console.
In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the Amazon VPC CNI add-on for.
Choose the Add-ons tab.
Select the box in the top right of the add-on box and then choose Edit.
On the Configure name of addon page:
1. Select a v1.14.0-eksbuild.3 or later version in the Version dropdown list.
2. Expand the Optional configuration settings.
3. Enter the top-level JSON key "nodeAgent": and value is an object with a key "enablePolicyEventLogs": and value of "true" in Configuration values. The resulting text must be a valid JSON object. The following example shows network policy and the network policy logs are enabled, and the network policy logs are sent to CloudWatch Logs:
  { "enableNetworkPolicy": "true", "nodeAgent": { "enablePolicyEventLogs": "true" } }

The following screenshot shows an example of this scenario.

<code class="shared">consolelong</code> showing the VPC CNI add-on with network policy and CloudWatch Logs in the optional configuration.

AWS CLI

Run the following AWS CLI command. Replace my-cluster with the name of your cluster and replace the IAM role ARN with the role that you are using.

aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version v1.14.0-eksbuild.3 \
    --service-account-role-arn region.arniam::123456789012:role/AmazonEKSVPCCNIRole \
    --resolve-conflicts PRESERVE --configuration-values '{"nodeAgent": {"enablePolicyEventLogs": "true"}}'

Self-managed add-on

Helm

If you have installed the Amazon VPC CNI plugin for Kubernetes through helm, you can update the configuration to write the network policy logs.

Run the following command to enable network policy.

helm upgrade --set nodeAgent.enablePolicyEventLogs=true aws-vpc-cni --namespace kube-system eks/aws-vpc-cni

kubectl

If you have installed the Amazon VPC CNI plugin for Kubernetes through kubectl, you can update the configuration to write the network policy logs.

Open the aws-node DaemonSet in your editor.

kubectl edit daemonset -n kube-system aws-node

Replace the false with true in the command argument --enable-policy-event-logs=false in the args: in the aws-network-policy-agent container in the VPC CNI aws-node daemonset manifest.
```
     - args:
        - --enable-policy-event-logs=true
```

Send network policy logs to Amazon CloudWatch Logs

You can monitor the network policy logs using services such as Amazon CloudWatch Logs. You can use the following methods to send the network policy logs to CloudWatch Logs.

For EKS clusters, the policy logs will be located under /aws/eks/cluster-name/cluster/ and for self-managed K8S clusters, the logs will be placed under /aws/k8s-cluster/cluster/.

Send network policy logs with Amazon VPC CNI plugin for Kubernetes

If you enable network policy, a second container is add to the aws-node pods for a node agent. This node agent can send the network policy logs to CloudWatch Logs.

Only the network policy logs are sent by the node agent. Other logs made by the VPC CNI aren’t included.

Prerequisites

Add the following permissions as a stanza or separate policy to the IAM role that you are using for the VPC CNI.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Amazon EKS add-on

consolelong

Open the Amazon EKS console.
In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the Amazon VPC CNI add-on for.
Choose the Add-ons tab.
Select the box in the top right of the add-on box and then choose Edit.
On the Configure name of addon page:
1. Select a v1.14.0-eksbuild.3 or later version in the Version dropdown list.
2. Expand the Optional configuration settings.
3. Enter the top-level JSON key "nodeAgent": and value is an object with a key "enableCloudWatchLogs": and value of "true" in Configuration values. The resulting text must be a valid JSON object. The following example shows network policy and the network policy logs are enabled, and the logs are sent to CloudWatch Logs:
  { "enableNetworkPolicy": "true", "nodeAgent": { "enablePolicyEventLogs": "true", "enableCloudWatchLogs": "true", } }

The following screenshot shows an example of this scenario.

+ image::images/console-cni-config-network-policy-logs-cwl.png[consolelong showing the VPC CNI add-on with network policy and CloudWatch Logs in the optional configuration.,scaledwidth=80%]

AWS CLI

Run the following AWS CLI command. Replace my-cluster with the name of your cluster and replace the IAM role ARN with the role that you are using.

aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version v1.14.0-eksbuild.3 \
    --service-account-role-arn region.arniam::123456789012:role/AmazonEKSVPCCNIRole \
    --resolve-conflicts PRESERVE --configuration-values '{"nodeAgent": {"enablePolicyEventLogs": "true", "enableCloudWatchLogs": "true"}}'

Self-managed add-on

Helm

If you have installed the Amazon VPC CNI plugin for Kubernetes through helm, you can update the configuration to send network policy logs to CloudWatch Logs.

Run the following command to enable network policy logs and send them to CloudWatch Logs.

helm upgrade --set nodeAgent.enablePolicyEventLogs=true --set nodeAgent.enableCloudWatchLogs=true aws-vpc-cni --namespace kube-system eks/aws-vpc-cni

kubectl

Open the aws-node DaemonSet in your editor.

kubectl edit daemonset -n kube-system aws-node

Replace the false with true in two command arguments --enable-policy-event-logs=false and --enable-cloudwatch-logs=false in the args: in the aws-network-policy-agent container in the VPC CNI aws-node daemonset manifest.
```
     - args:
        - --enable-policy-event-logs=true
        - --enable-cloudwatch-logs=true
```

Send network policy logs with a Fluent Bit daemonset

If you are using Fluent Bit in a daemonset to send logs from your nodes, you can add configuration to include the network policy logs from network policies. You can use the following example configuration:

    [INPUT]
        Name              tail
        Tag               eksnp.*
        Path              /var/log/aws-routed-eni/network-policy-agent*.log
        Parser            json
        DB                /var/log/aws-routed-eni/flb_npagent.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

Included eBPF SDK

The Amazon VPC CNI plugin for Kubernetes installs eBPF SDK collection of tools on the nodes. You can use the eBPF SDK tools to identify issues with network policies. For example, the following command lists the programs that are running on the node.

sudo /opt/cni/bin/aws-eks-na-cli ebpf progs

To run this command, you can use any method to connect to the node.

Stars demo of network policy for Amazon EKS

This demo creates a front-end, back-end, and client service on your Amazon EKS cluster. The demo also creates a management graphical user interface that shows the available ingress and egress paths between each service.

This demo creates a front-end, back-end, and client service on your Amazon EKS cluster. The demo also creates a management graphical user interface that shows the available ingress and egress paths between each service. We recommend that you complete the demo on a cluster that you don’t run production workloads on.

Before you create any network policies, all services can communicate bidirectionally. After you apply the network policies, you can see that the client can only communicate with the front-end service, and the back-end only accepts traffic from the front-end.

Apply the front-end, back-end, client, and management user interface services:

kubectl apply -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/namespace.yaml
kubectl apply -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/management-ui.yaml
kubectl apply -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/backend.yaml
kubectl apply -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/frontend.yaml
kubectl apply -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/client.yaml

View all Pods on the cluster.

kubectl get pods -A

An example output is as follows.

In your output, you should see pods in the namespaces shown in the following output. The NAMES of your pods and the number of pods in the READY column are different than those in the following output. Don’t continue until you see pods with similar names and they all have Running in the STATUS column.

NAMESPACE         NAME                                       READY   STATUS    RESTARTS   AGE
[...]
client            client-xlffc                               1/1     Running   0          5m19s
[...]
management-ui     management-ui-qrb2g                        1/1     Running   0          5m24s
stars             backend-sz87q                              1/1     Running   0          5m23s
stars             frontend-cscnf                             1/1     Running   0          5m21s
[...]

To connect to the management user interface, connect to the EXTERNAL-IP of the service running on your cluster:
```
kubectl get service/management-ui -n management-ui
```
Open the a browser to the location from the previous step. You should see the management user interface. The C node is the client service, the F node is the front-end service, and the B node is the back-end service. Each node has full communication access to all other nodes, as indicated by the bold, colored lines.

Apply the following network policy in both the stars and client namespaces to isolate the services from each other:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny
spec:
  podSelector:
    matchLabels: {}

You can use the following commands to apply the policy to both namespaces:

kubectl apply -n stars -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/apply_network_policies.files/default-deny.yaml
kubectl apply -n client -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/apply_network_policies.files/default-deny.yaml

Refresh your browser. You see that the management user interface can no longer reach any of the nodes, so they don’t show up in the user interface.

Apply the following different network policies to allow the management user interface to access the services. Apply this policy to allow the UI:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: stars
  name: allow-ui
spec:
  podSelector:
    matchLabels: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              role: management-ui

Apply this policy to allow the client:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: client
  name: allow-ui
spec:
  podSelector:
    matchLabels: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              role: management-ui

You can use the following commands to apply both policies:

kubectl apply -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/apply_network_policies.files/allow-ui.yaml
kubectl apply -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/apply_network_policies.files/allow-ui-client.yaml

Refresh your browser. You see that the management user interface can reach the nodes again, but the nodes cannot communicate with each other.

Apply the following network policy to allow traffic from the front-end service to the back-end service:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: stars
  name: backend-policy
spec:
  podSelector:
    matchLabels:
      role: backend
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: frontend
      ports:
        - protocol: TCP
          port: 6379

Refresh your browser. You see that the front-end can communicate with the back-end.

Apply the following network policy to allow traffic from the client to the front-end service:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: stars
  name: frontend-policy
spec:
  podSelector:
    matchLabels:
      role: frontend
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              role: client
      ports:
        - protocol: TCP
          port: 80

Refresh your browser. You see that the client can communicate to the front-end service. The front-end service can still communicate to the back-end service.

(Optional) When you are done with the demo, you can delete its resources.

kubectl delete -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/client.yaml
kubectl delete -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/frontend.yaml
kubectl delete -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/backend.yaml
kubectl delete -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/management-ui.yaml
kubectl delete -f https://eksworkshop.com/beginner/120_network-policies/calico/stars_policy_demo/create_resources.files/namespace.yaml

Even after deleting the resources, there can still be network policy endpoints on the nodes that might interfere in unexpected ways with networking in your cluster. The only sure way to remove these rules is to reboot the nodes or terminate all of the nodes and recycle them. To terminate all nodes, either set the Auto Scaling Group desired count to 0, then back up to the desired number, or just terminate the nodes.

Learn how to configure your Amazon EKS cluster to use Kubernetes network policies with the Amazon VPC CNI plugin. Control network traffic to and from pods using network policies for enhanced security. Covers network policy considerations, requirements, setup instructions, and troubleshooting tips.

By default, there are no restrictions in Kubernetes for IP addresses, ports, or connections between any Pods in your cluster or between your Pods and resources in any other network. You can use Kubernetes network policy to restrict network traffic to and from your Pods. For more information, see Network Policies in the Kubernetes documentation.

If you have version 1.13 or earlier of the Amazon VPC CNI plugin for Kubernetes on your cluster, you need to implement a third party solution to apply Kubernetes network policies to your cluster. Version 1.14 or later of the plugin can implement network policies, so you don’t need to use a third party solution. In this topic, you learn how to configure your cluster to use Kubernetes network policy on your cluster without using a third party add-on.

Network policies in the Amazon VPC CNI plugin for Kubernetes are supported in the following configurations.

Amazon EKS clusters of version 1.25 and later.
Version 1.14 or later of the Amazon VPC CNI plugin for Kubernetes on your cluster.
Cluster configured for IPv4 or IPv6 addresses.
You can use network policies with security groups for Pods. With network policies, you can control all in-cluster communication. With security groups for Pods, you can control access to AWS services from applications within a Pod.
You can use network policies with custom networking and prefix delegation.

Considerations

Architecture

When applying Amazon VPC CNI plugin for Kubernetes network policies to your cluster with the Amazon VPC CNI plugin for Kubernetes , you can apply the policies to Amazon EC2 Linux nodes only. You can’t apply the policies to Fargate or Windows nodes.
Network policies only apply either IPv4 or IPv6 addresses, but not both. In an IPv4 cluster, the VPC CNI assigns IPv4 address to pods and applies IPv4 policies. In an IPv6 cluster, the VPC CNI assigns IPv6 address to pods and applies IPv6 policies. Any IPv4 network policy rules applied to an IPv6 cluster are ignored. Any IPv6 network policy rules applied to an IPv4 cluster are ignored.

Network Policies

Network Policies are only applied to Pods that are part of a Deployment. Standalone Pods that don’t have a metadata.ownerReferences set can’t have network policies applied to them.
You can apply multiple network policies to the same Pod. When two or more policies that select the same Pod are configured, all policies are applied to the Pod.
The maximum number of unique combinations of ports for each protocol in each ingress: or egress: selector in a network policy is 24.
For any of your Kubernetes services, the service port must be the same as the container port. If you’re using named ports, use the same name in the service spec too.

Migration

If your cluster is currently using a third party solution to manage Kubernetes network policies, you can use those same policies with the Amazon VPC CNI plugin for Kubernetes. However you must remove your existing solution so that it isn’t managing the same policies.

Installation

The network policy feature creates and requires a PolicyEndpoint Custom Resource Definition (CRD) called policyendpoints.networking.k8s.aws. PolicyEndpoint objects of the Custom Resource are managed by Amazon EKS. You shouldn’t modify or delete these resources.
If you run pods that use the instance role IAM credentials or connect to the EC2 IMDS, be careful to check for network policies that would block access to the EC2 IMDS. You may need to add a network policy to allow access to EC2 IMDS. For more information, see Instance metadata and user data in the Amazon EC2 User Guide.

Pods that use IAM roles for service accounts or EKS Pod Identity don’t access EC2 IMDS.
The Amazon VPC CNI plugin for Kubernetes doesn’t apply network policies to additional network interfaces for each pod, only the primary interface for each pod (eth0). This affects the following architectures:
- IPv6 pods with the ENABLE_V4_EGRESS variable set to true. This variable enables the IPv4 egress feature to connect the IPv6 pods to IPv4 endpoints such as those outside the cluster. The IPv4 egress feature works by creating an additional network interface with a local loopback IPv4 address.
- When using chained network plugins such as Multus. Because these plugins add network interfaces to each pod, network policies aren’t applied to the chained network plugins.

Discover how Amazon VPC CNI plugin for Kubernetes provides pod networking capabilities and settings for different Amazon EKS node types and use cases, including security groups, Kubernetes network policies, custom networking, IPv4, and IPv6 support.

The Amazon VPC CNI plugin for Kubernetes provides networking for Pods. Use the following table to learn more about the available networking features.

Networking feature Learn more

Configure your cluster to assign IPv6 addresses to clusters, Pods, and services

cni-ipv6.title

Use IPv4 Source Network Address Translation for Pods

external-snat.title

Restrict network traffic to and from your Pods

cni-network-policy-configure.title

Customize the secondary network interface in nodes

cni-custom-network.title

Increase IP addresses for your node

cni-increase-ip-addresses.title

Use security groups for Pod network traffic

security-groups-for-pods.title

Use multiple network interfaces for Pods

pod-multiple-network-interfaces.title

Deploy `Pods` in alternate subnets with custom networking

Customize the secondary network interface in Amazon EKS nodes

Learn how your Pods can use different security groups and subnets than the primary elastic network interface of the Amazon EC2 node that they run on.

Complete the following before you start the tutorial:

Review the considerations
Familiarity with how the Amazon VPC CNI plugin for Kubernetes creates secondary network interfaces and assigns IP addresses to Pods. For more information, see ENI Allocation on GitHub.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.
We recommend that you complete the steps in this topic in a Bash shell. If you aren’t using a Bash shell, some script commands such as line continuation characters and the way variables are set and used require adjustment for your shell. Additionally, the quoting and escaping rules for your shell might be different. For more information, see Using quotation marks with strings in the AWS CLI in the AWS Command Line Interface User Guide.

For this tutorial, we recommend using the example values, except where it’s noted to replace them. You can replace any example value when completing the steps for a production cluster. We recommend completing all steps in the same terminal. This is because variables are set and used throughout the steps and won’t exist in different terminals.

The commands in this topic are formatted using the conventions listed in Using the AWS CLI examples. If you’re running commands from the command line against resources that are in a different AWS Region than the default AWS Region defined in the AWS CLI profile that you’re using, then you need to add --region region-code to the commands.

When you want to deploy custom networking to your production cluster, skip to Step 2: Configure your VPC.

Step 1: Create a test VPC and cluster

The following procedures help you create a test VPC and cluster and configure custom networking for that cluster. We don’t recommend using the test cluster for production workloads because several unrelated features that you might use on your production cluster aren’t covered in this topic. For more information, see create-cluster.title.

Define the cluster_name and account_id variables..

export cluster_name=my-custom-networking-cluster
account_id=$(aws sts get-caller-identity --query Account --output text)

Create a VPC.

If you are deploying to a test system, create a VPC using an Amazon EKS AWS CloudFormation template.

aws cloudformation create-stack --stack-name my-eks-custom-networking-vpc \
  --template-url https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-vpc-private-subnets.yaml \
  --parameters ParameterKey=VpcBlock,ParameterValue=192.168.0.0/24 \
  ParameterKey=PrivateSubnet01Block,ParameterValue=192.168.0.64/27 \
  ParameterKey=PrivateSubnet02Block,ParameterValue=192.168.0.96/27 \
  ParameterKey=PublicSubnet01Block,ParameterValue=192.168.0.0/27 \
  ParameterKey=PublicSubnet02Block,ParameterValue=192.168.0.32/27

The AWS CloudFormation stack takes a few minutes to create. To check on the stack’s deployment status, run the following command.

aws cloudformation describe-stacks --stack-name my-eks-custom-networking-vpc --query Stacks\[\].StackStatus  --output text

Don’t continue to the next step until the output of the command is CREATE_COMPLETE.

Define variables with the values of the private subnet IDs created by the template.

subnet_id_1=$(aws cloudformation describe-stack-resources --stack-name my-eks-custom-networking-vpc \
    --query "StackResources[?LogicalResourceId=='PrivateSubnet01'].PhysicalResourceId" --output text)
subnet_id_2=$(aws cloudformation describe-stack-resources --stack-name my-eks-custom-networking-vpc \
    --query "StackResources[?LogicalResourceId=='PrivateSubnet02'].PhysicalResourceId" --output text)

Define variables with the Availability Zones of the subnets retrieved in the previous step.

az_1=$(aws ec2 describe-subnets --subnet-ids $subnet_id_1 --query 'Subnets[*].AvailabilityZone' --output text)
az_2=$(aws ec2 describe-subnets --subnet-ids $subnet_id_2 --query 'Subnets[*].AvailabilityZone' --output text)

Create a cluster IAM role.
1. Run the following command to create an IAM trust policy JSON file.
  cat >eks-cluster-role-trust-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "eks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF
2. Create the Amazon EKS cluster IAM role. If necessary, preface eks-cluster-role-trust-policy.json with the path on your computer that you wrote the file to in the previous step. The command associates the trust policy that you created in the previous step to the role. To create an IAM role, the IAM principal that is creating the role must be assigned the iam:CreateRole action (permission).
  aws iam create-role --role-name myCustomNetworkingAmazonEKSClusterRole --assume-role-policy-document file://"eks-cluster-role-trust-policy.json"
3. Attach the Amazon EKS managed policy named AmazonEKSClusterPolicy to the role. To attach an IAM policy to an IAM principal, the principal that is attaching the policy must be assigned one of the following IAM actions (permissions): iam:AttachUserPolicy or iam:AttachRolePolicy.
  aws iam attach-role-policy --policy-arn region.arniam::aws:policy/AmazonEKSClusterPolicy --role-name myCustomNetworkingAmazonEKSClusterRole

Create an Amazon EKS cluster and configure your device to communicate with it.

Create a cluster.

aws eks create-cluster --name my-custom-networking-cluster \
   --role-arn region.arniam::$account_id:role/myCustomNetworkingAmazonEKSClusterRole \
   --resources-vpc-config subnetIds=$subnet_id_1","$subnet_id_2

The cluster takes several minutes to create. To check on the cluster’s deployment status, run the following command.
```
aws eks describe-cluster --name my-custom-networking-cluster --query cluster.status
```
Don’t continue to the next step until the output of the command is "ACTIVE".

Configure kubectl to communicate with your cluster.

aws eks update-kubeconfig --name my-custom-networking-cluster

Step 2: Configure your VPC

This tutorial requires the VPC created in Step 1: Create a test VPC and cluster. For a production cluster, adjust the steps accordingly for your VPC by replacing all of the example values with your own.

Confirm that your currently-installed Amazon VPC CNI plugin for Kubernetes is the latest version. To determine the latest version for the Amazon EKS add-on type and update your version to it, see updating-an-add-on.title. To determine the latest version for the self-managed add-on type and update your version to it, see managing-vpc-cni.title.
Retrieve the ID of your cluster VPC and store it in a variable for use in later steps. For a production cluster, replace my-custom-networking-cluster with the name of your cluster.
```
vpc_id=$(aws eks describe-cluster --name my-custom-networking-cluster --query "cluster.resourcesVpcConfig.vpcId" --output text)
```

Associate an additional Classless Inter-Domain Routing (CIDR) block with your cluster’s VPC. The CIDR block can’t overlap with any existing associated CIDR blocks.

View the current CIDR blocks associated to your VPC.

aws ec2 describe-vpcs --vpc-ids $vpc_id \
    --query 'Vpcs[*].CidrBlockAssociationSet[*].{CIDRBlock: CidrBlock, State: CidrBlockState.State}' --out table

An example output is as follows.

----------------------------------
|          DescribeVpcs          |
+-----------------+--------------+
|    CIDRBlock    |    State     |
+-----------------+--------------+
|  192.168.0.0/24 |  associated  |
+-----------------+--------------+

Associate an additional CIDR block to your VPC. For more information, see Associate additional IPv4 CIDR blocks with your VPC in the Amazon VPC User Guide.
```
aws ec2 associate-vpc-cidr-block --vpc-id $vpc_id --cidr-block 192.168.1.0/24
```

Confirm that the new block is associated.

aws ec2 describe-vpcs --vpc-ids $vpc_id --query 'Vpcs[*].CidrBlockAssociationSet[*].{CIDRBlock: CidrBlock, State: CidrBlockState.State}' --out table

An example output is as follows.

----------------------------------
|          DescribeVpcs          |
+-----------------+--------------+
|    CIDRBlock    |    State     |
+-----------------+--------------+
|  192.168.0.0/24 |  associated  |
|  192.168.1.0/24 |  associated  |
+-----------------+--------------+

Don’t proceed to the next step until your new CIDR block’s State is associated.

Create as many subnets as you want to use in each Availability Zone that your existing subnets are in. Specify a CIDR block that’s within the CIDR block that you associated with your VPC in a previous step.

Create new subnets. The subnets must be created in a different VPC CIDR block than your existing subnets are in, but in the same Availability Zones as your existing subnets. In this example, one subnet is created in the new CIDR block in each Availability Zone that the current private subnets exist in. The IDs of the subnets created are stored in variables for use in later steps. The Name values match the values assigned to the subnets created using the Amazon EKS VPC template in a previous step. Names aren’t required. You can use different names.

new_subnet_id_1=$(aws ec2 create-subnet --vpc-id $vpc_id --availability-zone $az_1 --cidr-block 192.168.1.0/27 \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=my-eks-custom-networking-vpc-PrivateSubnet01},{Key=kubernetes.io/role/internal-elb,Value=1}]' \
    --query Subnet.SubnetId --output text)
new_subnet_id_2=$(aws ec2 create-subnet --vpc-id $vpc_id --availability-zone $az_2 --cidr-block 192.168.1.32/27 \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=my-eks-custom-networking-vpc-PrivateSubnet02},{Key=kubernetes.io/role/internal-elb,Value=1}]' \
    --query Subnet.SubnetId --output text)

By default, your new subnets are implicitly associated with your VPC’s main route table. This route table allows communication between all the resources that are deployed in the VPC. However, it doesn’t allow communication with resources that have IP addresses that are outside the CIDR blocks that are associated with your VPC. You can associate your own route table to your subnets to change this behavior. For more information, see Subnet route tables in the Amazon VPC User Guide.

View the current subnets in your VPC.

aws ec2 describe-subnets --filters "Name=vpc-id,Values=$vpc_id" \
    --query 'Subnets[*].{SubnetId: SubnetId,AvailabilityZone: AvailabilityZone,CidrBlock: CidrBlock}' \
    --output table

An example output is as follows.

----------------------------------------------------------------------
|                           DescribeSubnets                          |
+------------------+--------------------+----------------------------+
| AvailabilityZone |     CidrBlock      |         SubnetId           |
+------------------+--------------------+----------------------------+
|  us-west-2d      |  192.168.0.0/27    |     subnet-example1        |
|  us-west-2a      |  192.168.0.32/27   |     subnet-example2        |
|  us-west-2a      |  192.168.0.64/27   |     subnet-example3        |
|  us-west-2d      |  192.168.0.96/27   |     subnet-example4        |
|  us-west-2a      |  192.168.1.0/27    |     subnet-example5        |
|  us-west-2d      |  192.168.1.32/27   |     subnet-example6        |
+------------------+--------------------+----------------------------+

You can see the subnets in the 192.168.1.0 CIDR block that you created are in the same Availability Zones as the subnets in the 192.168.0.0 CIDR block.

Step 3: Configure Kubernetes resources

Set the AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG environment variable to true in the aws-node DaemonSet.
```
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
```
Retrieve the ID of your cluster security group and store it in a variable for use in the next step. Amazon EKS automatically creates this security group when you create your cluster.
```
cluster_security_group_id=$(aws eks describe-cluster --name $cluster_name --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text)
```
Create an ENIConfig custom resource for each subnet that you want to deploy Pods in.
1. Create a unique file for each network interface configuration.
  
  + The following commands create separate ENIConfig files for the two subnets that were created in a previous step. The value for name must be unique. The name is the same as the Availability Zone that the subnet is in. The cluster security group is assigned to the ENIConfig.
  
  +

cat >$az_1.yaml <<EOF
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: $az_1
spec:
  securityGroups:
    - $cluster_security_group_id
  subnet: $new_subnet_id_1
EOF

cat >$az_2.yaml <<EOF
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: $az_2
spec:
  securityGroups:
    - $cluster_security_group_id
  subnet: $new_subnet_id_2
EOF

+ For a production cluster, you can make the following changes to the previous commands:

+ * Replace $cluster_security_group_id with the ID of an existing security group that you want to use for each ENIConfig. * We recommend naming your ENIConfigs the same as the Availability Zone that you’ll use the ENIConfig for, whenever possible. You might need to use different names for your ENIConfigs than the names of the Availability Zones for a variety of reasons. For example, if you have more than two subnets in the same Availability Zone and want to use them both with custom networking, then you need multiple ENIConfigs for the same Availability Zone. Since each ENIConfig requires a unique name, you can’t name more than one of your ENIConfigs using the Availability Zone name.

+ If your ENIConfig names aren’t all the same as Availability Zone names, then replace $az_1 and $az_2 with your own names in the previous commands and annotate your nodes with the ENIConfig later in this tutorial.

+ NOTE: If you don’t specify a valid security group for use with a production cluster and you’re using:

version 1.8.0 or later of the Amazon VPC CNI plugin for Kubernetes, then the security groups associated with the node’s primary elastic network interface are used.
a version of the Amazon VPC CNI plugin for Kubernetes that’s earlier than 1.8.0, then the default security group for the VPC is assigned to secondary network interfaces.

IMPORTANT:
AWS_VPC_K8S_CNI_EXTERNALSNAT=false is a default setting in the configuration for the Amazon VPC CNI plugin for Kubernetes. If you’re using the default setting, then traffic that is destined for IP addresses that aren’t within one of the CIDR blocks associated with your VPC use the security groups and subnets of your node’s primary network interface. The subnets and security groups defined in your ENIConfigs that are used to create secondary network interfaces aren’t used for this traffic. For more information about this setting, see external-snat.title.
If you also use security groups for Pods, the security group that’s specified in a SecurityGroupPolicy is used instead of the security group that’s specified in the ENIConfigs. For more information, see security-groups-for-pods.title.
1. Apply each custom resource file that you created to your cluster with the following commands.
  kubectl apply -f $az_1.yaml kubectl apply -f $az_2.yaml
  1. Confirm that your ENIConfigs were created.
    
    kubectl get ENIConfigs
    
    An example output is as follows.
    
    NAME AGE us-west-2a 117s us-west-2d 105s
  2. If you’re enabling custom networking on a production cluster and named your ENIConfigs something other than the Availability Zone that you’re using them for, then skip to the next step to deploy Amazon EC2 nodes.
    
    Enable Kubernetes to automatically apply the ENIConfig for an Availability Zone to any new Amazon EC2 nodes created in your cluster.
2. For the test cluster in this tutorial, skip to the next step.
  
  For a production cluster, check to see if an annotation with the key k8s.amazonaws.com/eniConfig for the ENI_CONFIG_ANNOTATION_DEF environment variable exists in the container spec for the aws-node DaemonSet.
  kubectl describe daemonset aws-node -n kube-system | grep ENI_CONFIG_ANNOTATION_DEF
  If output is returned, the annotation exists. If no output is returned, then the variable is not set. For a production cluster, you can use either this setting or the setting in the following step. If you use this setting, it overrides the setting in the following step. In this tutorial, the setting in the next step is used.
3. Update your aws-node DaemonSet to automatically apply the ENIConfig for an Availability Zone to any new Amazon EC2 nodes created in your cluster.
  kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

Step 4: Deploy Amazon EC2 nodes

Create a node IAM role.

Run the following command to create an IAM trust policy JSON file.

cat >node-role-trust-relationship.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Run the following command to set a variable for your role name. You can replace myCustomNetworkingNodeRole with any name you choose.
```
export node_role_name=myCustomNetworkingNodeRole
```

Create the IAM role and store its returned Amazon Resource Name (ARN) in a variable for use in a later step.

node_role_arn=$(aws iam create-role --role-name $node_role_name --assume-role-policy-document file://"node-role-trust-relationship.json" \
    --query Role.Arn --output text)

Attach three required IAM managed policies to the IAM role.

aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonEKSWorkerNodePolicy \
  --role-name $node_role_name
aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonEC2ContainerRegistryReadOnly \
  --role-name $node_role_name
aws iam attach-role-policy \
    --policy-arn region.arniam::aws:policy/AmazonEKS_CNI_Policy \
    --role-name $node_role_name

For simplicity in this tutorial, the AmazonEKS_CNI_Policy policy is attached to the node IAM role. In a production cluster however, we recommend attaching the policy to a separate IAM role that is used only with the Amazon VPC CNI plugin for Kubernetes. For more information, see cni-iam-role.title.

Create one of the following types of node groups. To determine the instance type that you want to deploy, see choosing-instance-type.title. For this tutorial, complete the Managed, Without a launch template or with a launch template without an AMI ID specified option. If you’re going to use the node group for production workloads, then we recommend that you familiarize yourself with all of the managed node group create-managed-node-group.title and self-managed node group worker.title options before deploying the node group.

Managed – Deploy your node group using one of the following options:
- Without a launch template or with a launch template without an AMI ID specified – Run the following command. For this tutorial, use the example values. For a production node group, replace all example values with your own. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
  aws eks create-nodegroup --cluster-name $cluster_name --nodegroup-name my-nodegroup \ --subnets $subnet_id_1 $subnet_id_2 --instance-types t3.medium --node-role $node_role_arn
- With a launch template with a specified AMI ID::
  1. Determine the Amazon EKS recommended number of maximum Pods `for your nodes. Follow the instructions in Amazon EKS recommended maximum Pods for each Amazon EC2 instance type, adding `--cni-custom-networking-enabled to step 3 in that topic. Note the output for use in the next step.
  2. In your launch template, specify an Amazon EKS optimized AMI ID, or a custom AMI built off the Amazon EKS optimized AMI, then deploy the node group using a launch template and provide the following user data in the launch template. This user data passes arguments into the bootstrap.sh file. For more information about the bootstrap file, see bootstrap.sh on GitHub. You can replace 20 with either the value from the previous step (recommended) or your own value.
    
    /etc/eks/bootstrap.sh my-cluster --use-max-pods false --kubelet-extra-args '--max-pods=20'
    
    If you’ve created a custom AMI that is not built off the Amazon EKS optimized AMI, then you need to custom create the configuration yourself.

Self-managed::

Determine the Amazon EKS recommended number of maximum Pods for your nodes. Follow the instructions in Amazon EKS recommended maximum Pods for each Amazon EC2 instance type, adding --cni-custom-networking-enabled to step 3 in that topic. Note the output for use in the next step.

Deploy the node group using the instructions in Create self-managed Amazon Linux nodes. Specify the following text for the BootstrapArguments parameter. You can replace 20 with either the value from the previous step (recommended) or your own value.

--use-max-pods false --kubelet-extra-args '--max-pods=20'

If you want nodes in a production cluster to support a significantly higher number of Pods, run the script in Amazon EKS recommended maximum Pods for each Amazon EC2 instance type again. Also, add the --cni-prefix-delegation-enabled option to the command. For example, 110 is returned for an m5.large instance type. For instructions on how to enable this capability, see cni-increase-ip-addresses.title. You can use this capability with custom networking.

Node group creation takes several minutes. You can check the status of the creation of a managed node group with the following command.

aws eks describe-nodegroup --cluster-name $cluster_name --nodegroup-name my-nodegroup --query nodegroup.status --output text

Don’t continue to the next step until the output returned is ACTIVE.

For the tutorial, you can skip this step.

For a production cluster, if you didn’t name your ENIConfigs the same as the Availability Zone that you’re using them for, then you must annotate your nodes with the ENIConfig name that should be used with the node. This step isn’t necessary if you only have one subnet in each Availability Zone and you named your ENIConfigs with the same names as your Availability Zones. This is because the Amazon VPC CNI plugin for Kubernetes automatically associates the correct ENIConfig with the node for you when you enabled it to do so in a previous step.
1. Get the list of nodes in your cluster.
  kubectl get nodes
  An example output is as follows.
  NAME STATUS ROLES AGE VERSION ip-192-168-0-126.us-west-2.compute.internal Ready <none> 8m49s v1.22.9-eks-810597c ip-192-168-0-92.us-west-2.compute.internal Ready <none> 8m34s v1.22.9-eks-810597c
2. Determine which Availability Zone each node is in. Run the following command for each node that was returned in the previous step.
  aws ec2 describe-instances --filters Name=network-interface.private-dns-name,Values=ip-192-168-0-126.us-west-2.compute.internal \ --query 'Reservations[].Instances[].{AvailabilityZone: Placement.AvailabilityZone, SubnetId: SubnetId}'
  An example output is as follows.
  [ { "AvailabilityZone": "us-west-2d", "SubnetId": "subnet-Example5" } ]
3. Annotate each node with the ENIConfig that you created for the subnet ID and Availability Zone. You can only annotate a node with one ENIConfig, though multiple nodes can be annotated with the same ENIConfig. Replace the example values with your own.
  kubectl annotate node ip-192-168-0-126.us-west-2.compute.internal k8s.amazonaws.com/eniConfig=EniConfigName1 kubectl annotate node ip-192-168-0-92.us-west-2.compute.internal k8s.amazonaws.com/eniConfig=EniConfigName2
If you had nodes in a production cluster with running Pods before you switched to using the custom networking feature, complete the following tasks:
1. Make sure that you have available nodes that are using the custom networking feature.
2. Cordon and drain the nodes to gracefully shut down the Pods. For more information, see Safely Drain a Node in the Kubernetes documentation.
3. Terminate the nodes. If the nodes are in an existing managed node group, you can delete the node group. Copy the command that follows to your device. Make the following modifications to the command as needed and then run the modified command:
  - Replace my-cluster with the name for your cluster.
  - Replace my-nodegroup with the name for your node group.
    
    aws eks delete-nodegroup --cluster-name my-cluster --nodegroup-name my-nodegroup
Only new nodes that are registered with the k8s.amazonaws.com/eniConfig label use the custom networking feature.

Confirm that Pods are assigned an IP address from a CIDR block that’s associated to one of the subnets that you created in a previous step.

kubectl get pods -A -o wide

An example output is as follows.

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE     IP              NODE                                          NOMINATED NODE   READINESS GATES
kube-system   aws-node-2rkn4             1/1     Running   0          7m19s   192.168.0.92    ip-192-168-0-92.us-west-2.compute.internal    <none>           <none>
kube-system   aws-node-k96wp             1/1     Running   0          7m15s   192.168.0.126   ip-192-168-0-126.us-west-2.compute.internal   <none>           <none>
kube-system   coredns-657694c6f4-smcgr   1/1     Running   0          56m     192.168.1.23    ip-192-168-0-92.us-west-2.compute.internal    <none>           <none>
kube-system   coredns-657694c6f4-stwv9   1/1     Running   0          56m     192.168.1.28    ip-192-168-0-92.us-west-2.compute.internal    <none>           <none>
kube-system   kube-proxy-jgshq           1/1     Running   0          7m19s   192.168.0.92    ip-192-168-0-92.us-west-2.compute.internal    <none>           <none>
kube-system   kube-proxy-wx9vk           1/1     Running   0          7m15s   192.168.0.126   ip-192-168-0-126.us-west-2.compute.internal   <none>           <none>

You can see that the coredns Pods are assigned IP addresses from the 192.168.1.0 CIDR block that you added to your VPC. Without custom networking, they would have been assigned addresses from the 192.168.0.0 CIDR block, because it was the only CIDR block originally associated with the VPC.

If a Pod’s spec contains hostNetwork=true, it’s assigned the primary IP address of the node. It isn’t assigned an address from the subnets that you added. By default, this value is set to false. This value is set to true for the kube-proxy and Amazon VPC CNI plugin for Kubernetes (aws-node) Pods that run on your cluster. This is why the kube-proxy and the plugin’s aws-node Pods aren’t assigned 192.168.1.x addresses in the previous output. For more information about a Pod’s hostNetwork setting, see PodSpec v1 core in the Kubernetes API reference.

Step 5: Delete tutorial resources

After you complete the tutorial, we recommend that you delete the resources that you created. You can then adjust the steps to enable custom networking for a production cluster.

If the node group that you created was just for testing, then delete it.
```
aws eks delete-nodegroup --cluster-name $cluster_name --nodegroup-name my-nodegroup
```
Even after the AWS CLI output says that the cluster is deleted, the delete process might not actually be complete. The delete process takes a few minutes. Confirm that it’s complete by running the following command.
```
aws eks describe-nodegroup --cluster-name $cluster_name --nodegroup-name my-nodegroup --query nodegroup.status --output text
```
Don’t continue until the returned output is similar to the following output.
```
An error occurred (ResourceNotFoundException) when calling the DescribeNodegroup operation: No node group found for name: my-nodegroup.
```

If the node group that you created was just for testing, then delete the node IAM role.

Detach the policies from the role.

aws iam detach-role-policy --role-name myCustomNetworkingNodeRole --policy-arn region.arniam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam detach-role-policy --role-name myCustomNetworkingNodeRole --policy-arn region.arniam::aws:policy/AmazonEC2ContainerRegistryReadOnly
aws iam detach-role-policy --role-name myCustomNetworkingNodeRole --policy-arn region.arniam::aws:policy/AmazonEKS_CNI_Policy

Delete the role.

aws iam delete-role --role-name myCustomNetworkingNodeRole

Delete the cluster.

aws eks delete-cluster --name $cluster_name

Confirm the cluster is deleted with the following command.

aws eks describe-cluster --name $cluster_name --query cluster.status --output text

When output similar to the following is returned, the cluster is successfully deleted.

An error occurred (ResourceNotFoundException) when calling the DescribeCluster operation: No cluster found for name: my-cluster.

Delete the cluster IAM role.

Detach the policies from the role.

aws iam detach-role-policy --role-name myCustomNetworkingAmazonEKSClusterRole --policy-arn region.arniam::aws:policy/AmazonEKSClusterPolicy

Delete the role.

aws iam delete-role --role-name myCustomNetworkingAmazonEKSClusterRole

Delete the subnets that you created in a previous step.

aws ec2 delete-subnet --subnet-id $new_subnet_id_1
aws ec2 delete-subnet --subnet-id $new_subnet_id_2

Delete the VPC that you created.

aws cloudformation delete-stack --stack-name my-eks-custom-networking-vpc

Learn how to enable custom networking for Amazon EKS Pods to deploy them in different subnets or use different security groups than the node’s primary network interface, increasing IP address availability and network isolation.

Applies to: Linux IPv4 Fargate nodes, Linux nodes with Amazon EC2 instances

By default, when the Amazon VPC CNI plugin for Kubernetes creates secondary elastic network interfaces (network interfaces) for your Amazon EC2 node, it creates them in the same subnet as the node’s primary network interface. It also associates the same security groups to the secondary network interface that are associated to the primary network interface. For one or more of the following reasons, you might want the plugin to create secondary network interfaces in a different subnet or want to associate different security groups to the secondary network interfaces, or both:

There’s a limited number of IPv4 addresses that are available in the subnet that the primary network interface is in. This might limit the number of Pods that you can create in the subnet. By using a different subnet for secondary network interfaces, you can increase the number of available IPv4 addresses available for Pods.
For security reasons, your Pods might need to use a different subnet or security groups than the node’s primary network interface.
The nodes are configured in public subnets, and you want to place the Pods in private subnets. The route table associated to a public subnet includes a route to an internet gateway. The route table associated to a private subnet doesn’t include a route to an internet gateway.

Considerations

The following are considerations for using the feature.

With custom networking enabled, no IP addresses assigned to the primary network interface are assigned to Pods. Only IP addresses from secondary network interfaces are assigned to Pods.
If your cluster uses the IPv6 family, you can’t use custom networking.
If you plan to use custom networking only to help alleviate IPv4 address exhaustion, you can create a cluster using the IPv6 family instead. For more information, see cni-ipv6.title.
Even though Pods deployed to subnets specified for secondary network interfaces can use different subnet and security groups than the node’s primary network interface, the subnets and security groups must be in the same VPC as the node.
For Fargate, subnets are controlled through the Fargate profile. For more information, see fargate-profile.title.

Assign more IP addresses to Amazon EKS nodes with prefixes

Increase the available IP addresses for your Amazon EKS node

You can increase the number of IP addresses that nodes can assign to Pods by assigning IP prefixes, rather than assigning individual secondary IP addresses to your nodes.

Complete the following before you start the procedure:

Review the considerations.
You need an existing cluster. To deploy one, see create-cluster.title.
The subnets that your Amazon EKS nodes are in must have sufficient contiguous /28 (for IPv4 clusters) or /80 (for IPv6 clusters) Classless Inter-Domain Routing (CIDR) blocks. You can only have Linux nodes in an IPv6 cluster. Using IP prefixes can fail if IP addresses are scattered throughout the subnet CIDR. We recommend that following:
- Using a subnet CIDR reservation so that even if any IP addresses within the reserved range are still in use, upon their release, the IP addresses aren’t reassigned. This ensures that prefixes are available for allocation without segmentation.
- Use new subnets that are specifically used for running the workloads that IP prefixes are assigned to. Both Windows and Linux workloads can run in the same subnet when assigning IP prefixes.
To assign IP prefixes to your nodes, your nodes must be AWS Nitro-based. Instances that aren’t Nitro-based continue to allocate individual secondary IP addresses, but have a significantly lower number of IP addresses to assign to Pods than Nitro-based instances do.
For clusters with Linux nodes only – If your cluster is configured for the IPv4 family, you must have version 1.9.0 or later of the Amazon VPC CNI plugin for Kubernetes add-on installed. You can check your current version with the following command.
```
kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
```
If your cluster is configured for the IPv6 family, you must have version 1.10.1 of the add-on installed. If your plugin version is earlier than the required versions, you must update it. For more information, see the updating sections of Assign IPs to Pods with the Amazon VPC CNI.

For clusters with Windows nodes only::

Your cluster and its platform version must be at, or later than the versions in the following table. To upgrade your cluster version, see update-cluster.title. If your cluster isn’t at the minimum platform version, then you can’t assign IP prefixes to your nodes until Amazon EKS has updated your platform version.

Kubernetes version Platform version

1.27

eks.3

1.26

eks.4

1.25

eks.5

You can check your current Kubernetes and platform version by replacing my-cluster in the following command with the name of your cluster and then running the modified command: aws eks describe-cluster --name my-cluster --query 'cluster.{"Kubernetes Version": version, "Platform Version": platformVersion}'.

Windows support enabled for your cluster. For more information, see windows-support.title.

Configure your cluster to assign IP address prefixes to nodes. Complete the procedure on the tab that matches your node’s operating system.

Linux

Enable the parameter to assign prefixes to network interfaces for the Amazon VPC CNI DaemonSet. When you deploy a 1.21 or later cluster, version 1.10.1 or later of the Amazon VPC CNI plugin for Kubernetes add-on is deployed with it. If you created the cluster with the IPv6 family, this setting was set to true by default. If you created the cluster with the IPv4 family, this setting was set to false by default.

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Even if your subnet has available IP addresses, if the subnet does not have any contiguous /28 blocks available, you will see the following error in the Amazon VPC CNI plugin for Kubernetes logs.

InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request

This can happen due to fragmentation of existing secondary IP addresses spread out across a subnet. To resolve this error, either create a new subnet and launch Pods there, or use an Amazon EC2 subnet CIDR reservation to reserve space within a subnet for use with prefix assignment. For more information, see Subnet CIDR reservations in the Amazon VPC User Guide. … If you plan to deploy a managed node group without a launch template, or with a launch template that you haven’t specified an AMI ID in, and you’re using a version of the Amazon VPC CNI plugin for Kubernetes at or later than the versions listed in the prerequisites, then skip to the next step. Managed node groups automatically calculates the maximum number of Pods for you.

+ If you’re deploying a self-managed node group or a managed node group with a launch template that you have specified an AMI ID in, then you must determine the Amazon EKS recommend number of maximum Pods for your nodes. Follow the instructions in Amazon EKS recommended maximum Pods for each Amazon EC2 instance type, adding --cni-prefix-delegation-enabled to step 3. Note the output for use in a later step.

+ IMPORTANT: Managed node groups enforces a maximum number on the value of maxPods. For instances with less than 30 vCPUs the maximum number is 110 and for all other instances the maximum number is 250. This maximum number is applied whether prefix delegation is enabled or not. … If you’re using a 1.21 or later cluster configured for IPv6, skip to the next step.

+ Specify the parameters in one of the following options. To determine which option is right for you and what value to provide for it, see WARM_PREFIX_TARGET, WARM_IP_TARGET, and MINIMUM_IP_TARGET on GitHub.

+ You can replace the example values with a value greater than zero.

+ ** WARM_PREFIX_TARGET

kubectl set env ds aws-node -n kube-system WARM_PREFIX_TARGET=1

WARM_IP_TARGET or MINIMUM_IP_TARGET – If either value is set, it overrides any value set for WARM_PREFIX_TARGET.
```
kubectl set env ds aws-node -n kube-system WARM_IP_TARGET=5
```

kubectl set env ds aws-node -n kube-system MINIMUM_IP_TARGET=2

Create one of the following types of node groups with at least one Amazon EC2 Nitro Amazon Linux 2 instance type. For a list of Nitro instance types, see Instances built on the Nitro System in the Amazon EC2 User Guide. This capability is not supported on Windows. For the options that include 110, replace it with either the value from step 3 (recommended), or your own value.

Self-managed – Deploy the node group using the instructions in Create self-managed Amazon Linux nodes. Specify the following text for the BootstrapArguments parameter.
```
--use-max-pods false --kubelet-extra-args '--max-pods=110'
```
If you’re using eksctl to create the node group, you can use the following command.
```
eksctl create nodegroup --cluster my-cluster --managed=false --max-pods-per-node 110
```

Managed – Deploy your node group using one of the following options:

Without a launch template or with a launch template without an AMI ID specified – Complete the procedure in Create a managed node group for your cluster. Managed node groups automatically calculates the Amazon EKS recommended max-pods value for you.

With a launch template with a specified AMI ID – In your launch template, specify an Amazon EKS optimized AMI ID, or a custom AMI built off the Amazon EKS optimized AMI, then deploy the node group using a launch template and provide the following user data in the launch template. This user data passes arguments into the bootstrap.sh file. For more information about the bootstrap file, see bootstrap.sh on GitHub.

/etc/eks/bootstrap.sh my-cluster \
  --use-max-pods false \
  --kubelet-extra-args '--max-pods=110'

If you’re using eksctl to create the node group, you can use the following command.

eksctl create nodegroup --cluster my-cluster --max-pods-per-node 110

If you’ve created a custom AMI that is not built off the Amazon EKS optimized AMI, then you need to custom create the configuration yourself.

If you also want to assign IP addresses to Pods from a different subnet than the instance’s, then you need to enable the capability in this step. For more information, see cni-custom-network.title.

Windows

Enable assignment of IP prefixes.
1. Open the amazon-vpc-cni ConfigMap for editing.
  kubectl edit configmap -n kube-system amazon-vpc-cni -o yaml
2. Add the following line to the data section.
  enable-windows-prefix-delegation: "true"
3. Save the file and close the editor.
4. Confirm that the line was added to the ConfigMap.
  kubectl get configmap -n kube-system amazon-vpc-cni -o "jsonpath={.data.enable-windows-prefix-delegation}"
  If the returned output isn’t true, then there might have been an error. Try completing the step again.
  
  Even if your subnet has available IP addresses, if the subnet does not have any contiguous /28 blocks available, you will see the following error in the node events.

"failed to allocate a private IP/Prefix address: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request"

This can happen due to fragmentation of existing secondary IP addresses spread out across a subnet. To resolve this error, either create a new subnet and launch Pods there, or use an Amazon EC2 subnet CIDR reservation to reserve space within a subnet for use with prefix assignment. For more information, see Subnet CIDR reservations in the Amazon VPC User Guide. … (Optional) Specify additional configuration for controlling the pre-scaling and dynamic scaling behavior for your cluster. For more information, see Configuration options with Prefix Delegation mode on Windows on GitHub.

+ …. Open the amazon-vpc-cni ConfigMap for editing.

kubectl edit configmap -n kube-system amazon-vpc-cni -o yaml

Replace the example values with a value greater than zero and add the entries that you require to the data section of the ConfigMap. If you set a value for either warm-ip-target or minimum-ip-target, the value overrides any value set for warm-prefix-target.
```
  warm-prefix-target: "1"
  warm-ip-target: "5"
  minimum-ip-target: "2"
```
Save the file and close the editor.
1. Create Windows node groups with at least one Amazon EC2 Nitro instance type. For a list of Nitro instance types, see Instances built on the Nitro System in the Amazon EC2 User Guide. By default, the maximum number of Pods that you can deploy to a node is 110. If you want to increase or decrease that number, specify the following in the user data for the bootstrap configuration. Replace max-pods-quantity with your max pods value.
  -KubeletExtraArgs '--max-pods=max-pods-quantity'
  If you’re deploying managed node groups, this configuration needs to be added in the launch template. For more information, see launch-templates.title. For more information about the configuration parameters for Windows bootstrap script, see bootstrap-script-configuration-parameters.title. . Once your nodes are deployed, view the nodes in your cluster.
  
  +
  kubectl get nodes
  + An example output is as follows.
  
  +
  NAME STATUS ROLES AGE VERSION ip-192-168-22-103.region-code.compute.internal Ready <none> 19m v1.XX.X-eks-6b7464 ip-192-168-97-94.region-code.compute.internal Ready <none> 19m v1.XX.X-eks-6b7464
  1. Describe one of the nodes to determine the value of max-pods for the node and the number of available IP addresses. Replace 192.168.30.193 with the IPv4 address in the name of one of your nodes returned in the previous output.
    
    kubectl describe node ip-192-168-30-193.region-code.compute.internal | grep 'pods\|PrivateIPv4Address'
    
    An example output is as follows.
    
    pods: 110 vpc.amazonaws.com/PrivateIPv4Address: 144
    
    In the previous output, 110 is the maximum number of Pods that Kubernetes will deploy to the node, even though 144 IP addresses are available.

Learn how to significantly increase the number of IP addresses that you can assign to Pods by assigning IP prefixes with Amazon EKS, improving scalability and reducing launch delays for large and spiky workloads.

Applies to: Linux and Windows nodes with Amazon EC2 instances

Applies to: Public and private subnets

Each Amazon EC2 instance supports a maximum number of elastic network interfaces and a maximum number of IP addresses that can be assigned to each network interface. Each node requires one IP address for each network interface. All other available IP addresses can be assigned to Pods. Each Pod requires its own IP address. As a result, you might have nodes that have available compute and memory resources, but can’t accommodate additional Pods because the node has run out of IP addresses to assign to Pods.

You can increase the number of IP addresses that nodes can assign to Pods by assigning IP prefixes, rather than assigning individual secondary IP addresses to your nodes. Each prefix includes several IP addresses. If you don’t configure your cluster for IP prefix assignment, your cluster must make more Amazon EC2 application programming interface (API) calls to configure network interfaces and IP addresses necessary for Pod connectivity. As clusters grow to larger sizes, the frequency of these API calls can lead to longer Pod and instance launch times. This results in scaling delays to meet the demand of large and spiky workloads, and adds cost and management overhead because you need to provision additional clusters and VPCs to meet scaling requirements. For more information, see Kubernetes Scalability thresholds on GitHub.

Compatibility with `Amazon VPC CNI plugin for Kubernetes` features

You can use IP prefixes with the following features:

IPv4 Source Network Address Translation - For more information, see external-snat.title.
IPv6 addresses to clusters, Pods, and services - For more information, see cni-ipv6.title.
Restricting traffic using Kubernetes network policies - For more information, see cni-network-policy.title.

The following list provides information about the Amazon VPC CNI plugin settings that apply. For more information about each setting, see amazon-vpc-cni-k8s on GitHub.

WARM_IP_TARGET
MINIMUM_IP_TARGET
WARM_PREFIX_TARGET

Considerations

Consider the following when you use this feature:

Each Amazon EC2 instance type supports a maximum number of Pods. If your managed node group consists of multiple instance types, the smallest number of maximum Pods for an instance in the cluster is applied to all nodes in the cluster.
By default, the maximum number of Pods that you can run on a node is 110, but you can change that number. If you change the number and have an existing managed node group, the next AMI or launch template update of your node group results in new nodes coming up with the changed value.
When transitioning from assigning IP addresses to assigning IP prefixes, we recommend that you create new node groups to increase the number of available IP addresses, rather than doing a rolling replacement of existing nodes. Running Pods on a node that has both IP addresses and prefixes assigned can lead to inconsistency in the advertised IP address capacity, impacting the future workloads on the node. For the recommended way of performing the transition, see Replace all nodes during migration from Secondary IP mode to Prefix Delegation mode or vice versa in the Amazon EKS best practices guide.
The security group scope is at the node-level - For more information, see Security group.
IP prefixes assigned to a network interface support high Pod density per node and have the best launch time.
IP prefixes and IP addresses are associated with standard Amazon EC2 elastic network interfaces. Pods requiring specific security groups are assigned the primary IP address of a branch network interface. You can mix Pods getting IP addresses, or IP addresses from IP prefixes with Pods getting branch network interfaces on the same node.
For clusters with Linux nodes only.
- After you configure the add-on to assign prefixes to network interfaces, you can’t downgrade your Amazon VPC CNI plugin for Kubernetes add-on to a version lower than 1.9.0 (or 1.10.1) without removing all nodes in all node groups in your cluster.
- If you’re also using security groups for Pods, with POD_SECURITY_GROUP_ENFORCING_MODE=standard and AWS_VPC_K8S_CNI_EXTERNALSNAT=false, when your Pods communicate with endpoints outside of your VPC, the node’s security groups are used, rather than any security groups you’ve assigned to your Pods.
  
  If you’re also using security groups for Pods, with POD_SECURITY_GROUP_ENFORCING_MODE=strict, when your Pods communicate with endpoints outside of your VPC, the Pod’s security groups are used.

Assign security groups to individual `Pods`

Configure the `Amazon VPC CNI plugin for Kubernetes` for security groups for Amazon EKS `Pods`

If you use Pods with Amazon EC2 instances, you need to configure the Amazon VPC CNI plugin for Kubernetes for security groups

If you use Fargate Pods only, and don’t have any Amazon EC2 nodes in your cluster, see sg-pods-example-deployment.title.

Check your current Amazon VPC CNI plugin for Kubernetes version with the following command:
```
kubectl describe daemonset aws-node --namespace kube-system | grep amazon-k8s-cni: | cut -d : -f 3
```
An example output is as follows.
```
v1.7.6
```
If your Amazon VPC CNI plugin for Kubernetes version is earlier than 1.7.7, then update the plugin to version 1.7.7 or later. For more information, see managing-vpc-cni.title
Add the AmazonEKSVPCResourceController managed IAM policy to the cluster role that is associated with your Amazon EKS cluster. The policy allows the role to manage network interfaces, their private IP addresses, and their attachment and detachment to and from network instances.
1. Retrieve the name of your cluster IAM role and store it in a variable. Replace my-cluster with the name of your cluster.
  cluster_role=$(aws eks describe-cluster --name my-cluster --query cluster.roleArn --output text | cut -d / -f 2)
2. Attach the policy to the role.
  aws iam attach-role-policy --policy-arn region.arniam::aws:policy/AmazonEKSVPCResourceController --role-name $cluster_role

Enable the Amazon VPC CNI add-on to manage network interfaces for Pods by setting the ENABLE_POD_ENI variable to true in the aws-node DaemonSet. Once this setting is set to true, for each node in the cluster the add-on creates a cninode custom resource. The VPC resource controller creates and attaches one special network interface called a trunk network interface with the description aws-k8s-trunk-eni.

kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true

The trunk network interface is included in the maximum number of network interfaces supported by the instance type. For a list of the maximum number of network interfaces supported by each instance type, see IP addresses per network interface per instance type in the Amazon EC2 User Guide. If your node already has the maximum number of standard network interfaces attached to it then the VPC resource controller will reserve a space. You will have to scale down your running Pods enough for the controller to detach and delete a standard network interface, create the trunk network interface, and attach it to the instance.

You can see which of your nodes have a CNINode custom resource with the following command. If No resources found is returned, then wait several seconds and try again. The previous step requires restarting the Amazon VPC CNI plugin for Kubernetes Pods`, which takes several seconds.

kubectl get cninode -A
     NAME FEATURES
     ip-192-168-64-141.us-west-2.compute.internal [{"name":"SecurityGroupsForPods"}]
     ip-192-168-7-203.us-west-2.compute.internal [{"name":"SecurityGroupsForPods"}]

If you are using VPC CNI versions older than 1.15, node labels were used instead of the CNINode custom resource. You can see which of your nodes have the node label aws-k8s-trunk-eni set to true with the following command. If No resources found is returned, then wait several seconds and try again. The previous step requires restarting the Amazon VPC CNI plugin for Kubernetes Pods, which takes several seconds.

kubectl get nodes -o wide -l vpc.amazonaws.com/has-trunk-attached=true
-

Once the trunk network interface is created, Pods are assigned secondary IP addresses from the trunk or standard network interfaces. The trunk interface is automatically deleted if the node is deleted.

When you deploy a security group for a Pod in a later step, the VPC resource controller creates a special network interface called a branch network interface with a description of aws-k8s-branch-eni and associates the security groups to it. Branch network interfaces are created in addition to the standard and trunk network interfaces attached to the node.

If you are using liveness or readiness probes, then you also need to disable TCP early demux, so that the kubelet can connect to Pods on branch network interfaces using TCP. To disable TCP early demux, run the following command:

kubectl patch daemonset aws-node -n kube-system \
  -p '{"spec": {"template": {"spec": {"initContainers": [{"env":[{"name":"DISABLE_TCP_EARLY_DEMUX","value":"true"}],"name":"aws-vpc-cni-init"}]}}}}'

If you’re using 1.11.0 or later of the Amazon VPC CNI plugin for Kubernetes add-on and set POD_SECURITY_GROUP_ENFORCING_MODE=standard, as described in the next step, then you don’t need to run the previous command.

If your cluster uses NodeLocal DNSCache, or you want to use Calico network policy with your Pods that have their own security groups, or you have Kubernetes services of type NodePort and LoadBalancer using instance targets with an externalTrafficPolicy set to Local for Pods that you want to assign security groups to, then you must be using version 1.11.0 or later of the Amazon VPC CNI plugin for Kubernetes add-on, and you must enable the following setting:
```
kubectl set env daemonset aws-node -n kube-system POD_SECURITY_GROUP_ENFORCING_MODE=standard
```
IMPORTANT: Pod security group rules aren’t applied to traffic between Pods or between Pods and services, such as kubelet or nodeLocalDNS, that are on the same node. Pods using different security groups on the same node can’t communicate because they are configured in different subnets, and routing is disabled between these subnets. Outbound traffic from Pods to addresses outside of the VPC is network address translated to the IP address of the instance’s primary network interface (unless you’ve also set AWS_VPC_K8S_CNI_EXTERNALSNAT=true). For this traffic, the rules in the security groups for the primary network interface are used, rather than the rules in the Pod’s security groups. ** For this setting to apply to existing Pods, you must restart the Pods or the nodes that the Pods are running on.
To see how to use a security group policy for your Pod, see sg-pods-example-deployment.title.

Use a security group policy for an Amazon EKS `Pod`

To use security groups for Pods, you must have an existing security group. The following steps show you how to use the security group policy for a Pod. Unless otherwise noted, complete all steps from the same terminal because variables are used in the following steps that don’t persist across terminals.

If you have a Pod with Amazon EC2 instances, you must configure the plugin before you use this procedure. For more information, see security-groups-pods-deployment.title.

Create a Kubernetes namespace to deploy resources to. You can replace my-namespace with the name of a namespace that you want to use.
```
kubectl create namespace my-namespace
```

Deploy an Amazon EKS SecurityGroupPolicy to your cluster.

Copy the following contents to your device. You can replace podSelector with serviceAccountSelector if you’d rather select Pods based on service account labels. You must specify one selector or the other. An empty podSelector (example: podSelector: {}) selects all Pods in the namespace. You can change my-role to the name of your role. An empty serviceAccountSelector selects all service accounts in the namespace. You can replace my-security-group-policy with a name for your SecurityGroupPolicy and my-namespace with the namespace that you want to create the SecurityGroupPolicy in.

You must replace my_pod_security_group_id with the ID of an existing security group. If you don’t have an existing security group, then you must create one. For more information, see Amazon EC2 security groups for Linux instances in the Amazon EC2 User Guide. You can specify 1-5 security group IDs. If you specify more than one ID, then the combination of all the rules in all the security groups are effective for the selected Pods.

cat >my-security-group-policy.yaml <<EOF
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: my-security-group-policy
  namespace: my-namespace
spec:
  podSelector:
    matchLabels:
      role: my-role
  securityGroups:
    groupIds:
      - my_pod_security_group_id
EOF

The security group or groups that you specify for your Pods must meet the following criteria:

They must exist. If they don’t exist, then, when you deploy a Pod that matches the selector, your Pod remains stuck in the creation process. If you describe the Pod, you’ll see an error message similar to the following one: An error occurred (InvalidSecurityGroupID.NotFound) when calling the CreateNetworkInterface operation: The securityGroup ID 'sg-05b1d815d1EXAMPLE' does not exist.
They must allow inbound communication from the security group applied to your nodes (for kubelet) over any ports that you’ve configured probes for.
They must allow outbound communication over TCP and UDP ports 53 to a security group assigned to the Pods (or nodes that the Pods run on) running CoreDNS. The security group for your CoreDNS Pods must allow inbound TCP and UDP port 53 traffic from the security group that you specify.
They must have necessary inbound and outbound rules to communicate with other Pods that they need to communicate with.
They must have rules that allow the Pods to communicate with the Kubernetes control plane if you’re using the security group with Fargate. The easiest way to do this is to specify the cluster security group as one of the security groups.

Security group policies only apply to newly scheduled Pods. They do not affect running Pods.

Deploy the policy.

kubectl apply -f my-security-group-policy.yaml

Deploy a sample application with a label that matches the my-role value for podSelector that you specified in a previous step.

Copy the following contents to your device. Replace the example values with your own and then run the modified command. If you replace my-role, make sure that it’s the same as the value you specified for the selector in a previous step.

cat >sample-application.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
  labels:
    app: my-app
spec:
  replicas: 4
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
        role: my-role
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: nginx
        image: public.ecr.aws/nginx/nginx:1.23
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: my-namespace
  labels:
    app: my-app
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
EOF

Deploy the application with the following command. When you deploy the application, the Amazon VPC CNI plugin for Kubernetes matches the role label and the security groups that you specified in the previous step are applied to the Pod.
```
kubectl apply -f sample-application.yaml
```

View the Pods deployed with the sample application. For the remainder of this topic, this terminal is referred to as TerminalA.

kubectl get pods -n my-namespace -o wide

An example output is as follows.

NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE                                            NOMINATED NODE   READINESS GATES
my-deployment-5df6f7687b-4fbjm   1/1     Running   0          7m51s   192.168.53.48    ip-192-168-33-28.region-code.compute.internal   <none>           <none>
my-deployment-5df6f7687b-j9fl4   1/1     Running   0          7m51s   192.168.70.145   ip-192-168-92-33.region-code.compute.internal   <none>           <none>
my-deployment-5df6f7687b-rjxcz   1/1     Running   0          7m51s   192.168.73.207   ip-192-168-92-33.region-code.compute.internal   <none>           <none>
my-deployment-5df6f7687b-zmb42   1/1     Running   0          7m51s   192.168.63.27    ip-192-168-33-28.region-code.compute.internal   <none>           <none>

Try these tips if any Pods are stuck.

If any Pods are stuck in the Waiting state, then run kubectl describe pod my-deployment-xxxxxxxxxx-xxxxx -n my-namespace. If you see Insufficient permissions: Unable to create Elastic Network Interface., confirm that you added the IAM policy to the IAM cluster role in a previous step.
If any Pods are stuck in the Pending state, confirm that your node instance type is listed in limits.go and that the product of the maximum number of branch network interfaces supported by the instance type multiplied times the number of nodes in your node group hasn’t already been met. For example, an m5.large instance supports nine branch network interfaces. If your node group has five nodes, then a maximum of 45 branch network interfaces can be created for the node group. The 46th Pod that you attempt to deploy will sit in Pending state until another Pod that has associated security groups is deleted.

If you run kubectl describe pod my-deployment-xxxxxxxxxx-xxxxx -n my-namespace and see a message similar to the following message, then it can be safely ignored. This message might appear when the Amazon VPC CNI plugin for Kubernetes tries to set up host networking and fails while the network interface is being created. The plugin logs this event until the network interface is created.

Failed to create Pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e24268322e55c8185721f52df6493684f6c2c3bf4fd59c9c121fd4cdc894579f" network for Pod "my-deployment-5df6f7687b-4fbjm": networkPlugin
cni failed to set up Pod "my-deployment-5df6f7687b-4fbjm-c89wx_my-namespace" network: add cmd: failed to assign an IP address to container

You can’t exceed the maximum number of Pods that can be run on the instance type. For a list of the maximum number of Pods that you can run on each instance type, see eni-max-pods.txt on GitHub. When you delete a Pod that has associated security groups, or delete the node that the Pod is running on, the VPC resource controller deletes the branch network interface. If you delete a cluster with Pods using Pods for security groups, then the controller doesn’t delete the branch network interfaces, so you’ll need to delete them yourself. For information about how to delete network interfaces, see Delete a network interface in the Amazon EC2 User Guide.

In a separate terminal, shell into one of the Pods. For the remainder of this topic, this terminal is referred to as TerminalB. Replace 5df6f7687b-4fbjm with the ID of one of the Pods returned in your output from the previous step.
```
kubectl exec -it -n my-namespace my-deployment-5df6f7687b-4fbjm -- /bin/bash
```
From the shell in TerminalB, confirm that the sample application works.
```
curl my-app
```
An example output is as follows.
```
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
[...]
```
You received the output because all Pods running the application are associated with the security group that you created. That group contains a rule that allows all traffic between all Pods that the security group is associated to. DNS traffic is allowed outbound from that security group to the cluster security group, which is associated with your nodes. The nodes are running the CoreDNS Pods, which your Pods did a name lookup to.
From TerminalA, remove the security group rules that allow DNS communication to the cluster security group from your security group. If you didn’t add the DNS rules to the cluster security group in a previous step, then replace $my_cluster_security_group_id with the ID of the security group that you created the rules in.
```
aws ec2 revoke-security-group-ingress --group-id $my_cluster_security_group_id --security-group-rule-ids $my_tcp_rule_id
aws ec2 revoke-security-group-ingress --group-id $my_cluster_security_group_id --security-group-rule-ids $my_udp_rule_id
```
From TerminalB, attempt to access the application again.
```
curl my-app
```
An example output is as follows.
```
curl: (6) Could not resolve host: my-app
```
The attempt fails because the Pod is no longer able to access the CoreDNS Pods, which have the cluster security group associated to them. The cluster security group no longer has the security group rules that allow DNS communication from the security group associated to your Pod.

If you attempt to access the application using the IP addresses returned for one of the Pods in a previous step, you still receive a response because all ports are allowed between Pods that have the security group associated to them and a name lookup isn’t required.

Once you’ve finished experimenting, you can remove the sample security group policy, application, and security group that you created. Run the following commands from TerminalA.

kubectl delete namespace my-namespace
aws ec2 revoke-security-group-ingress --group-id $my_pod_security_group_id --security-group-rule-ids $my_inbound_self_rule_id
wait
sleep 45s
aws ec2 delete-security-group --group-id $my_pod_security_group_id

Learn how to configure security groups for Pods on Amazon EKS, integrating Amazon EC2 security groups with Kubernetes Pods to define network traffic rules. Discover the considerations, setup process, and deploy a sample application with assigned security groups.

Applies to: Linux nodes with Amazon EC2 instances

Applies to: Private subnets

Security groups for Pods integrate Amazon EC2 security groups with Kubernetes Pods. You can use Amazon EC2 security groups to define rules that allow inbound and outbound network traffic to and from Pods that you deploy to nodes running on many Amazon EC2 instance types and Fargate. For a detailed explanation of this capability, see the Introducing security groups for Pods blog post.

Compatibility with `Amazon VPC CNI plugin for Kubernetes` features

You can use security groups for Pods with the following features:

IPv4 Source Network Address Translation - For more information, see external-snat.title.
IPv6 addresses to clusters, Pods, and services - For more information, see cni-ipv6.title.
Restricting traffic using Kubernetes network policies - For more information, see cni-network-policy.title.

Considerations

Before deploying security groups for Pods, consider the following limitations and conditions:

Security groups for Pods can’t be used with Windows nodes.
Security groups for Pods can be used with clusters configured for the IPv6 family that contain Amazon EC2 nodes by using version 1.16.0 or later of the Amazon VPC CNI plugin. You can use security groups for Pods with clusters configure IPv6 family that contain only Fargate nodes by using version 1.7.7 or later of the Amazon VPC CNI plugin. For more information, see cni-ipv6.title
Security groups for Pods are supported by most Nitro-based Amazon EC2 instance families, though not by all generations of a family. For example, the m5, c5, r5, m6g, c6g, and r6g instance family and generations are supported. No instance types in the t family are supported. For a complete list of supported instance types, see the limits.go file on GitHub. Your nodes must be one of the listed instance types that have IsTrunkingCompatible: true in that file.
If you’re also using Pod security policies to restrict access to Pod mutation, then the eks:vpc-resource-controller Kubernetes user must be specified in the Kubernetes ClusterRoleBinding for the role that your psp is assigned to. If you’re using the default Amazon EKS psp, role, and ClusterRoleBinding, this is the eks:podsecuritypolicy:authenticated ClusterRoleBinding. For example, you add the user to the subjects: section, as shown in the following example:
```
[...]
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: eks:vpc-resource-controller
  - kind: ServiceAccount
    name: eks-vpc-resource-controller
```
If you’re using custom networking and security groups for Pods together, the security group specified by security groups for Pods is used instead of the security group specified in the ENIConfig.
If you’re using version 1.10.2 or earlier of the Amazon VPC CNI plugin and you include the terminationGracePeriodSeconds setting in your Pod spec, the value for the setting can’t be zero.
If you’re using version 1.10 or earlier of the Amazon VPC CNI plugin, or version 1.11 with POD_SECURITY_GROUP_ENFORCING_MODE=strict, which is the default setting, then Kubernetes services of type NodePort and LoadBalancer using instance targets with an externalTrafficPolicy set to Local aren’t supported with Pods that you assign security groups to. For more information about using a load balancer with instance targets, see network-load-balancing.title.
If you’re using version 1.10 or earlier of the Amazon VPC CNI plugin or version 1.11 with POD_SECURITY_GROUP_ENFORCING_MODE=strict, which is the default setting, source NAT is disabled for outbound traffic from Pods with assigned security groups so that outbound security group rules are applied. To access the internet, Pods with assigned security groups must be launched on nodes that are deployed in a private subnet configured with a NAT gateway or instance. Pods with assigned security groups deployed to public subnets are not able to access the internet.

If you’re using version 1.11 or later of the plugin with POD_SECURITY_GROUP_ENFORCING_MODE=standard, then Pod traffic destined for outside of the VPC is translated to the IP address of the instance’s primary network interface. For this traffic, the rules in the security groups for the primary network interface are used, rather than the rules in the Pod’s security groups.
To use Calico network policy with Pods that have associated security groups, you must use version 1.11.0 or later of the Amazon VPC CNI plugin and set POD_SECURITY_GROUP_ENFORCING_MODE=standard. Otherwise, traffic flow to and from Pods with associated security groups are not subjected to Calico network policy enforcement and are limited to Amazon EC2 security group enforcement only. To update your Amazon VPC CNI version, see managing-vpc-cni.title
Pods running on Amazon EC2 nodes that use security groups in clusters that use NodeLocal DNSCache are only supported with version 1.11.0 or later of the Amazon VPC CNI plugin and with POD_SECURITY_GROUP_ENFORCING_MODE=standard. To update your Amazon VPC CNI plugin version, see managing-vpc-cni.title
Security groups for Pods might lead to higher Pod startup latency for Pods with high churn. This is due to rate limiting in the resource controller.
The EC2 security group scope is at the Pod-level - For more information, see Security group.

If you set POD_SECURITY_GROUP_ENFORCING_MODE=standard and AWS_VPC_K8S_CNI_EXTERNALSNAT=false, traffic destined for endpoints outside the VPC use the node’s security groups, not the Pod’s security groups.

Attach multiple network interfaces to `Pods` with `Multus`

Learn how to use Multus CNI to attach multiple network interfaces to a Pod in Amazon EKS for advanced networking scenarios, while leveraging the Amazon VPC CNI plugin for primary networking.

Multus CNI is a container network interface (CNI) plugin for Amazon EKS that enables attaching multiple network interfaces to a Pod. For more information, see the Multus-CNI documentation on GitHub.

In Amazon EKS, each Pod has one network interface assigned by the Amazon VPC CNI plugin. With Multus, you can create a multi-homed Pod that has multiple interfaces. This is accomplished by Multus acting as a "meta-plugin"; a CNI plugin that can call multiple other CNI plugins. AWS support for Multus comes configured with the Amazon VPC CNI plugin as the default delegate plugin.

Amazon EKS won’t be building and publishing single root I/O virtualization (SR-IOV) and Data Plane Development Kit (DPDK) CNI plugins. However, you can achieve packet acceleration by connecting directly to Amazon EC2 Elastic Network Adapters (ENA) through Multus managed host-device and ipvlan plugins.
Amazon EKS is supporting Multus, which provides a generic process that enables simple chaining of additional CNI plugins. Multus and the process of chaining is supported, but AWS won’t provide support for all compatible CNI plugins that can be chained, or issues that may arise in those CNI plugins that are unrelated to the chaining configuration.
Amazon EKS is providing support and life cycle management for the Multus plugin, but isn’t responsible for any IP address or additional management associated with the additional network interfaces. The IP address and management of the default network interface utilizing the Amazon VPC CNI plugin remains unchanged.
Only the Amazon VPC CNI plugin is officially supported as the default delegate plugin. You need to modify the published Multus installation manifest to reconfigure the default delegate plugin to an alternate CNI if you choose not to use the Amazon VPC CNI plugin for primary networking.
Multus is only supported when using the Amazon VPC CNI as the primary CNI. We do not support the Amazon VPC CNI when used for higher order interfaces, secondary or otherwise.
To prevent the Amazon VPC CNI plugin from trying to manage additional network interfaces assigned to Pods, add the following tag to the network interface:

key

: node.k8s.amazonaws.com/no_manage

value

: true
Multus is compatible with network policies, but the policy has to be enriched to include ports and IP addresses that may be part of additional network interfaces attached to Pods.

For an implementation walk through, see the Multus Setup Guide on GitHub.

Discover how the Amazon VPC CNI plugin for Kubernetes add-on works to assign private IP addresses and create network interfaces for Pods and services in your Amazon EKS cluster.

With Amazon EKS Auto Mode, you don’t need to install or upgrade networking add-ons. Auto Mode includes pod networking and load balancing capabilities.

For more information, see automode.title.

The Amazon VPC CNI plugin for Kubernetes add-on is deployed on each Amazon EC2 node in your Amazon EKS cluster. The add-on creates elastic network interfaces and attaches them to your Amazon EC2 nodes. The add-on also assigns a private IPv4 or IPv6 address from your VPC to each Pod.

A version of the add-on is deployed with each Fargate node in your cluster, but you don’t update it on Fargate nodes. Other compatible CNI plugins are available for use on Amazon EKS clusters, but this is the only CNI plugin supported by Amazon EKS for nodes that run on AWS infrastructure. For more information about the other compatible CNI plugins, see alternate-cni-plugins.title. The VPC CNI isn’t supported for use with hybrid nodes. For more information about your CNI options for hybrid nodes, see hybrid-nodes-cni.title.

The following table lists the latest available version of the Amazon EKS add-on type for each Kubernetes version.

`Amazon VPC CNI` versions

Kubernetes version	Amazon EKS type of VPC CNI version
1.31	v1.19.0-eksbuild.1
1.30	v1.19.0-eksbuild.1
1.29	v1.19.0-eksbuild.1
1.28	v1.19.0-eksbuild.1
1.27	v1.19.0-eksbuild.1
1.26	v1.19.0-eksbuild.1
1.25	v1.19.0-eksbuild.1
1.24	v1.19.0-eksbuild.1
1.23	v1.18.5-eksbuild.1

If you’re self-managing this add-on, the versions in the table might not be the same as the available self-managed versions. For more information about updating the self-managed type of this add-on, see vpc-add-on-self-managed-update.title.

To upgrade to VPC CNI v1.12.0 or later, you must upgrade to VPC CNI v1.7.0 first. We recommend that you update one minor version at a time.

Considerations

The following are considerations for using the feature.

Versions are specified as major-version.minor-version.patch-version-eksbuild.build-number.
Check version compatibility for each feature. Some features of each release of the Amazon VPC CNI plugin for Kubernetes require certain Kubernetes versions. When using different Amazon EKS features, if a specific version of the add-on is required, then it’s noted in the feature documentation. Unless you have a specific reason for running an earlier version, we recommend running the latest version.

11.4.2. Alternate CNI plugins for Amazon EKS clusters

Learn how to use alternate network and security plugins on Amazon EKS to customize networking for your Kubernetes clusters on Amazon EC2 nodes.

The Amazon VPC CNI plugin for Kubernetes is the only CNI plugin supported by Amazon EKS with Amazon EC2 nodes. Amazon EKS supports the core capabilities of Cilium and Calico for Amazon EKS Hybrid Nodes. Amazon EKS runs upstream Kubernetes, so you can install alternate compatible CNI plugins to Amazon EC2 nodes in your cluster. If you have Fargate nodes in your cluster, the Amazon VPC CNI plugin for Kubernetes is already on your Fargate nodes. It’s the only CNI plugin you can use with Fargate nodes. An attempt to install an alternate CNI plugin on Fargate nodes fails.

If you plan to use an alternate CNI plugin on Amazon EC2 nodes, we recommend that you obtain commercial support for the plugin or have the in-house expertise to troubleshoot and contribute fixes to the CNI plugin project.

Amazon EKS maintains relationships with a network of partners that offer support for alternate compatible CNI plugins. For details about the versions, qualifications, and testing performed, see the following partner documentation.

Partner

Product

Documentation

Tigera

Calico

Isovalent

Cilium

Cloud-Native Contrail Networking (CN2)

Juniper

VMware

Antrea

add-ons-aws-ebs-csi-driver.title

Amazon EKS aims to give you a wide selection of options to cover all use cases.

Alternate compatible network policy plugins

Calico is a widely adopted solution for container networking and security. Using Calico on EKS provides a fully compliant network policy enforcement for your EKS clusters. Additionally, you can opt to use Calico’s networking, which conserve IP addresses from your underlying VPC. Calico Cloud enhances the features of Calico Open Source, providing advanced security and observability capabilities.

Traffic flow to and from Pods with associated security groups are not subjected to Calico network policy enforcement and are limited to Amazon VPC security group enforcement only.

If you use Calico network policy enforcement, we recommend that you set the environment variable ANNOTATE_POD_IP to true to avoid a known issue with Kubernetes. To use this feature, you must add patch permission for pods to the aws-node ClusterRole. Note that adding patch permissions to the aws-node DaemonSet increases the security scope for the plugin. For more information, see ANNOTATE_POD_IP in the VPC CNI repo on GitHub.

Considerations for Amazon EKS Auto Mode

Amazon EKS Auto Mode does not support alternate CNI plugins or network policy plugins. For more information, see automode.title.

11.4.3. Route internet traffic with `AWS` Load Balancer Controller

Install `AWS Load Balancer Controller` with `Helm`

Learn how to install the AWS Load Balancer Controller on Amazon EKS using Helm to manage K8s load balancing with AWS Cloud. Discover the prerequisites and steps for creating an IAM role, installing with Helm, and verifying the controller deployment.

With Amazon EKS Auto Mode, you don’t need to install or upgrade networking add-ons. Auto Mode includes pod networking and load balancing capabilities.

For more information, see automode.title.

This topic describes how to install the AWS Load Balancer Controller using Helm, a package manager for Kubernetes, and eksctl. The controller is installed with default options. For more information about the controller, including details on configuring it with annotations, see the AWS Load Balancer Controller Documentation on GitHub.

In the following steps, replace the example values with your own values.

Prerequisites

Before starting this tutorial, you must install and configure the following tools and resources that you need to create and manage an Amazon EKS cluster.

An existing Amazon EKS cluster. To deploy one, see getting-started.title.
An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.
Make sure that your Amazon VPC CNI plugin for Kubernetes, kube-proxy, and CoreDNS add-ons are at the minimum versions listed in Service account tokens.
Familiarity with AWS Elastic Load Balancing. For more information, see the Elastic Load Balancing User Guide.
Familiarity with Kubernetes service and ingress resources.
Helm installed locally.

Step 1: Create IAM Role using `eksctl`

You only need to create an IAM Role for the AWS Load Balancer Controller once per AWS-account. Check if AmazonEKSLoadBalancerControllerRole exists in the IAM Console. If this role exists, skip to Step 2: Install AWS Load Balancer Controller.

Below example is referring to the AWS Load Balancer Controller v2.11.0 release version. For more information about all releases, see the AWS Load Balancer Controller Release Page on GitHub.

Download an IAM policy for the AWS Load Balancer Controller that allows it to make calls to AWS APIs on your behalf.

AWS

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json

AWS GovCloud (US)

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy_us-gov.json

mv iam_policy_us-gov.json iam_policy.json

Create an IAM policy using the policy downloaded in the previous step.

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

If you view the policy in the consolelong, the console shows warnings for the ELB service, but not for the ELB v2 service. This happens because some of the actions in the policy exist for ELB v2, but not for ELB. You can ignore the warnings for ELB.

Replace my-cluster with the name of your cluster, 111122223333 with your account ID, and then run the command. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.

eksctl create iamserviceaccount \
  --cluster=my-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=region.arniam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

Step 2: Install `AWS Load Balancer Controller`

Add the eks-charts Helm chart repository. AWS maintains this repository on GitHub.
```
helm repo add eks https://aws.github.io/eks-charts
```
Update your local repo to make sure that you have the most recent charts.
```
helm repo update eks
```
Install the AWS Load Balancer Controller.

If you’re deploying the controller to Amazon EC2 nodes that have restricted access to the Amazon EC2 instance metadata service (IMDS), or if you’re deploying to Fargate or Amazon EKS Hybrid Nodes, then add the following flags to the helm command that follows:
- --set region=region-code
- --set vpcId=vpc-xxxxxxxx
  
  Replace my-cluster with the name of your cluster. In the following command, aws-load-balancer-controller is the Kubernetes service account that you created in a previous step.
  
  For more information about configuring the helm chart, see values.yaml on GitHub.
  helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system \ --set clusterName=my-cluster \ --set serviceAccount.create=false \ --set serviceAccount.name=aws-load-balancer-controller

The deployed chart doesn’t receive security updates automatically. You need to manually upgrade to a newer chart when it becomes available. When upgrading, change install to upgrade in the previous command.

The helm install command automatically installs the custom resource definitions (CRDs) for the controller. The helm upgrade command does not. If you use helm upgrade, you must manually install the CRDs. Run the following command to install the CRDs:

wget https://raw.githubusercontent.com/aws/eks-charts/master/stable/aws-load-balancer-controller/crds/crds.yaml
kubectl apply -f crds.yaml

Step 3: Verify that the controller is installed

Verify that the controller is installed.
```
kubectl get deployment -n kube-system aws-load-balancer-controller
```
An example output is as follows.
```
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           84s
```
You receive the previous output if you deployed using Helm. If you deployed using the Kubernetes manifest, you only have one replica.
Before using the controller to provision AWS resources, your cluster must meet specific requirements. For more information, see alb-ingress.title and network-load-balancing.title.

Install `AWS Load Balancer Controller` with manifests

Install the AWS Load Balancer Controller add-on for Amazon EKS using Kubernetes manifests to provision Elastic Load Balancing resources. Configure IAM role and install cert-manager before applying controller manifest.

With Amazon EKS Auto Mode, you don’t need to install or upgrade networking add-ons. Auto Mode includes pod networking and load balancing capabilities.

For more information, see automode.title.

This topic describes how to install the controller by downloading and applying Kubernetes manifests. You can view the full documentation for the controller on GitHub.

In the following steps, replace the example values with your own values.

Prerequisites

Before starting this tutorial, you must install and configure the following tools and resources that you need to create and manage an Amazon EKS cluster.

An existing Amazon EKS cluster. To deploy one, see getting-started.title.
An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see enable-iam-roles-for-service-accounts.title.
Make sure that your Amazon VPC CNI plugin for Kubernetes, kube-proxy, and CoreDNS add-ons are at the minimum versions listed in Service account tokens.
Familiarity with AWS Elastic Load Balancing. For more information, see the Elastic Load Balancing User Guide.
Familiarity with Kubernetes service and ingress resources.

Step 1: Configure IAM

You only need to create a role for the AWS Load Balancer Controller one per AWS account. Check if AmazonEKSLoadBalancerControllerRole exists in the IAM Console. If this role exists, skip to Step 2: Install cert-manager.

Below example is referring to the AWS Load Balancer Controller v2.11.0 release version. For more inforamtion about all releases, see the AWS Load Balancer Controller Release Page on GitHub.

Download an IAM policy for the AWS Load Balancer Controller that allows it to make calls to AWS APIs on your behalf.

AWS

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json

AWS GovCloud (US)

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy_us-gov.json

mv iam_policy_us-gov.json iam_policy.json

Create an IAM policy using the policy downloaded in the previous step.

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

eksctl

eksctl create iamserviceaccount \
  --cluster=my-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=region.arniam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

AWS CLI and kubectl

Retrieve your cluster’s OIDC provider ID and store it in a variable.

oidc_id=$(aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

Determine whether an IAM OIDC provider with your cluster’s ID is already in your account. You need OIDC configured for both the cluster and IAM.
```
aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4
```
If output is returned, then you already have an IAM OIDC provider for your cluster. If no output is returned, then you must create an IAM OIDC provider for your cluster. For more information, see enable-iam-roles-for-service-accounts.title.

Copy the following contents to your device. Replace 111122223333 with your account ID. Replace region-code with the AWS Region that your cluster is in. Replace EXAMPLED539D4633E53DE1B71EXAMPLE with the output returned in the previous step. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:. After replacing the text, run the modified command to create the load-balancer-role-trust-policy.json file.

cat >load-balancer-role-trust-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "region.arniam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:aws-load-balancer-controller"
                }
            }
        }
    ]
}
EOF

Create the IAM role.

aws iam create-role \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --assume-role-policy-document file://"load-balancer-role-trust-policy.json"

Attach the required Amazon EKS managed IAM policy to the IAM role. Replace 111122223333 with your account ID.

aws iam attach-role-policy \
  --policy-arn region.arniam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \
  --role-name AmazonEKSLoadBalancerControllerRole

Copy the following contents to your device. Replace 111122223333 with your account ID. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:. After replacing the text, run the modified command to create the aws-load-balancer-controller-service-account.yaml file.

cat >aws-load-balancer-controller-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: aws-load-balancer-controller
  name: aws-load-balancer-controller
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: region.arniam::111122223333:role/AmazonEKSLoadBalancerControllerRole
EOF

Create the Kubernetes service account on your cluster. The Kubernetes service account named aws-load-balancer-controller is annotated with the IAM role that you created named AmazonEKSLoadBalancerControllerRole.
```
kubectl apply -f aws-load-balancer-controller-service-account.yaml
```

Step 2: Install `cert-manager`

Install cert-manager using one of the following methods to inject certificate configuration into the webhooks. For more information, see Getting Started in the cert-manager Documentation.

We recommend using the quay.io container registry to install cert-manager. If your nodes do not have access to the quay.io container registry, Install cert-manager using Amazon ECR (see below).

Quay.io

If your nodes have access to the quay.io container registry, install cert-manager to inject certificate configuration into the webhooks.
```
kubectl apply \
    --validate=false \
    -f https://github.com/jetstack/cert-manager/releases/download/v1.13.5/cert-manager.yaml
```

Amazon ECR

Install cert-manager using one of the following methods to inject certificate configuration into the webhooks. For more information, see Getting Started in the cert-manager Documentation.

Download the manifest.

curl -Lo cert-manager.yaml https://github.com/jetstack/cert-manager/releases/download/v1.13.5/cert-manager.yaml

Pull the following images and push them to a repository that your nodes have access to. For more information on how to pull, tag, and push the images to your own repository, see copy-image-to-repository.title.
```
quay.io/jetstack/cert-manager-cainjector:v1.13.5
quay.io/jetstack/cert-manager-controller:v1.13.5
quay.io/jetstack/cert-manager-webhook:v1.13.5
```
Replace quay.io in the manifest for the three images with your own registry name. The following command assumes that your private repository’s name is the same as the source repository. Replace 111122223333.dkr.ecr.region-code.amazonaws.com with your private registry.
```
sed -i.bak -e 's|quay.io|111122223333.dkr.ecr.region-code.amazonaws.com|' ./cert-manager.yaml
```

Apply the manifest.

kubectl apply \
    --validate=false \
    -f ./cert-manager.yaml

Step 3: Install `AWS Load Balancer Controller`

Download the controller specification. For more information about the controller, see the documentation on GitHub.

curl -Lo v2_11_0_full.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.11.0/v2_11_0_full.yaml

Make the following edits to the file.
1. If you downloaded the v2_11_0_full.yaml file, run the following command to remove the ServiceAccount section in the manifest. If you don’t remove this section, the required annotation that you made to the service account in a previous step is overwritten. Removing this section also preserves the service account that you created in a previous step if you delete the controller.
  sed -i.bak -e '690,698d' ./v2_11_0_full.yaml
  If you downloaded a different file version, then open the file in an editor and remove the following lines.
  apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/name: aws-load-balancer-controller name: aws-load-balancer-controller namespace: kube-system ---
2. Replace your-cluster-name in the Deployment spec section of the file with the name of your cluster by replacing my-cluster with the name of your cluster.
  sed -i.bak -e 's|your-cluster-name|my-cluster|' ./v2_11_0_full.yaml
3. If your nodes don’t have access to the Amazon EKS Amazon ECR image repositories, then you need to pull the following image and push it to a repository that your nodes have access to. For more information on how to pull, tag, and push an image to your own repository, see copy-image-to-repository.title.
  public.ecr.aws/eks/aws-load-balancer-controller:v2.11.0
  Add your registry’s name to the manifest. The following command assumes that your private repository’s name is the same as the source repository and adds your private registry’s name to the file. Replace 111122223333.dkr.ecr.region-code.amazonaws.com with your registry. This line assumes that you named your private repository the same as the source repository. If not, change the eks/aws-load-balancer-controller text after your private registry name to your repository name.
  sed -i.bak -e 's|public.ecr.aws/eks/aws-load-balancer-controller|111122223333.dkr.ecr.region-code.amazonaws.com/eks/aws-load-balancer-controller|' ./v2_11_0_full.yaml
4. (Required only for Fargate or Restricted IMDS)
  
  If you’re deploying the controller to Amazon EC2 nodes that have restricted access to the Amazon EC2 instance metadata service (IMDS), or if you’re deploying to Fargate or Amazon EKS Hybrid Nodes, then add the following parameters under - args:.
  [...] spec: containers: - args: - --cluster-name=your-cluster-name - --ingress-class=alb - --aws-vpc-id=vpc-xxxxxxxx - --aws-region=region-code [...]
Apply the file.
```
kubectl apply -f v2_11_0_full.yaml
```

Download the IngressClass and IngressClassParams manifest to your cluster.

curl -Lo v2_11_0_ingclass.yaml https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/download/v2.11.0/v2_11_0_ingclass.yaml

Apply the manifest to your cluster.
```
kubectl apply -f v2_11_0_ingclass.yaml
```

Step 4: Verify that the controller is installed

Verify that the controller is installed.
```
kubectl get deployment -n kube-system aws-load-balancer-controller
```
An example output is as follows.
```
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           84s
```
You receive the previous output if you deployed using Helm. If you deployed using the Kubernetes manifest, you only have one replica.
Before using the controller to provision AWS resources, your cluster must meet specific requirements. For more information, see alb-ingress.title and network-load-balancing.title.

Migrate apps from deprecated ALB `Ingress Controller`

Learn how to migrate from the deprecated ALB Ingress Controller to the latest AWS Load Balancer Controller release, ensuring smooth transition and uninterrupted load balancing capabilities.

This topic describes how to migrate from deprecated controller versions. More specifically, it describes how to remove deprecated versions of the AWS Load Balancer Controller.

Deprecated versions cannot be upgraded. You must remove them first, and then install a current version.
Deprecated versions include:
- AWS ALB Ingress Controller for Kubernetes ("Ingress Controller"), a predecessor to the AWS Load Balancer Controller.
- Any 0.1.x version of the AWS Load Balancer Controller

Remove the deprecated controller version

You may have installed the deprecated version using Helm or manually with Kubernetes manifests. Complete the procedure using the tool that you originally installed it with.

If you installed the incubator/aws-alb-ingress-controller Helm chart, uninstall it.
```
helm delete aws-alb-ingress-controller -n kube-system
```
If you have version 0.1.x of the eks-charts/aws-load-balancer-controller chart installed, uninstall it. The upgrade from 0.1.x to version 1.0.0 doesn’t work due to incompatibility with the webhook API version.
```
helm delete aws-load-balancer-controller -n kube-system
```
Check to see if the controller is currently installed.
```
kubectl get deployment -n kube-system alb-ingress-controller
```
This is the output if the controller isn’t installed.

+ This is the output if the controller is installed.

+

NAME                   READY UP-TO-DATE AVAILABLE AGE
alb-ingress-controller 1/1   1          1         122d

Enter the following commands to remove the controller.

kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.8/docs/examples/alb-ingress-controller.yaml
kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.8/docs/examples/rbac-role.yaml

Migrate to `AWS Load Balancer Controller`

To migrate from the ALB Ingress Controller for Kubernetes to the AWS Load Balancer Controller, you need to:

Remove the ALB Ingress Controller (see above).
Install the AWS Load Balancer Controller.
Add an additional policy to the IAM Role used by the AWS Load Balancer Controller. This policy permits the LBC to manage resources created by the ALB Ingress Controller for Kubernetes.
Download the IAM policy. This policy permits the AWS Load Balancer Controller to manage resources created by the ALB Ingress Controller for Kubernetes. You can also view the policy.
```
curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy_v1_to_v2_additional.json
```
If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:..
```
sed -i.bak -e 's|region.arn|arn:aws-us-gov:|' iam_policy_v1_to_v2_additional.json
```

Create the IAM policy and note the ARN that is returned.

aws iam create-policy \
  --policy-name AWSLoadBalancerControllerAdditionalIAMPolicy \
  --policy-document file://iam_policy_v1_to_v2_additional.json

Attach the IAM policy to the IAM role used by the AWS Load Balancer Controller. Replace your-role-name with the name of the role, such as AmazonEKSLoadBalancerControllerRole.

If you created the role using eksctl, then to find the role name that was created, open the AWS CloudFormation console and select the eksctl-my-cluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller stack. Select the Resources tab. The role name is in the Physical ID column. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace region.arn with arn:aws-us-gov:.
```
aws iam attach-role-policy \
  --role-name your-role-name \
  --policy-arn region.arniam::111122223333:policy/AWSLoadBalancerControllerAdditionalIAMPolicy
```

Learn how to configure and use the AWS Load Balancer Controller to expose Kubernetes cluster apps to the internet with AWS Elastic Load Balancing for Kubernetes services and ingresses.

The AWS Load Balancer Controller manages AWS Elastic Load Balancers for a Kubernetes cluster. You can use the controller to expose your cluster apps to the internet. The controller provisions AWS load balancers that point to cluster Service or Ingress resources. In other words, the controller creates a single IP address or DNS name that points to multiple pods in your cluster.

Architecture diagram. Illustration of traffic coming from internet users, to Amazon Load Balancer. Amazon Load Balancer distributes traffic to pods in the cluster.

The controller watches for Kubernetes Ingress or Service resources. In response, it creates the appropriate AWS Elastic Load Balancing resources. You can configure the specific behavior of the load balancers by applying annotations to the Kubernetes resources. For example, you can attach AWS security groups to load balancers using annotations.

The controller provisions the following resources:

Kubernetes Ingress: The LBC creates an AWS Application Load Balancer (ALB) when you create a Kubernetes Ingress. Review the annotations you can apply to an Ingress resource.
Kubernetes service of the LoadBalancer type: The LBC creates an AWS Network Load Balancer (NLB)when you create a Kubernetes service of type LoadBalancer. Review the annotations you can apply to a Service resource.

In the past, the Kubernetes network load balancer was used for instance targets, but the LBC was used for IP targets. With the AWS Load Balancer Controller version 2.3.0 or later, you can create NLBs using either target type. For more information about NLB target types, see Target type in the User Guide for Network Load Balancers.

The controller is an open-source project managed on GitHub.

Before deploying the controller, we recommend that you review the prerequisites and considerations in Route application and HTTP traffic with Application Load Balancers and network-load-balancing.title. In those topics, you will deploy a sample app that includes an AWS load balancer.

Install the controller

You can use one of the following procedures to install the AWS Load Balancer Controller:

If you are new to Amazon EKS we recommend that you use Helm for the installation because it simplifies the AWS Load Balancer Controller installation. For more information, see lbc-helm.title.
For advanced configurations, such as clusters with restricted network access to public container registries, use Kubernetes Manifests. For more information, see lbc-manifest.title.

Migrate from deprecated controller versions

If you have deprecated versions of the AWS Load Balancer Controller installed, see lbc-remove.title.
Deprecated versions cannot be upgraded. They must be removed and a current version of the AWS Load Balancer Controller installed.
Deprecated versions include:
- AWS ALB Ingress Controller for Kubernetes ("Ingress Controller"), a predecessor to the AWS Load Balancer Controller.
- Any 0.1.x version of the AWS Load Balancer Controller

Legacy cloud provider

Kubernetes includes a legacy cloud provider for AWS. The legacy cloud provider is capable of provisioning AWS load balancers, similar to the AWS Load Balancer Controller. The legacy cloud provider creates Classic Load Balancers. If you do not install the AWS Load Balancer Controller, Kubernetes will default to using the legacy cloud provider. You should install the AWS Load Balancer Controller and avoid using the legacy cloud provider.

In versions 2.5 and newer, the AWS Load Balancer Controller becomes the default controller for Kubernetes service resources with the type: LoadBalancer and makes an AWS Network Load Balancer (NLB) for each service. It does this by making a mutating webhook for services, which sets the spec.loadBalancerClass field to service.k8s.aws/nlb for new services of type: LoadBalancer. You can turn off this feature and revert to using the legacy Cloud Provider as the default controller, by setting the helm chart value enableServiceMutatorWebhook to false. The cluster won’t provision new Classic Load Balancers for your services unless you turn off this feature. Existing Classic Load Balancers will continue to work.

11.4.4. Manage CoreDNS for DNS in Amazon EKS clusters

Create the `CoreDNS` Amazon EKS add-on

Create the CoreDNS Amazon EKS add-on. You must have a cluster before you create the add-on. For more information, see create-cluster.title.

See which version of the add-on is installed on your cluster.

kubectl describe deployment coredns --namespace kube-system | grep coredns: | cut -d : -f 3

An example output is as follows.

v1.10.1-eksbuild.13

See which type of the add-on is installed on your cluster. Depending on the tool that you created your cluster with, you might not currently have the Amazon EKS add-on type installed on your cluster. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name coredns --query addon.addonVersion --output text
```
If a version number is returned, you have the Amazon EKS type of the add-on installed on your cluster and don’t need to complete the remaining steps in this procedure. If an error is returned, you don’t have the Amazon EKS type of the add-on installed on your cluster. Complete the remaining steps of this procedure to install it.

Save the configuration of your currently installed add-on.

kubectl get deployment coredns -n kube-system -o yaml > aws-k8s-coredns-old.yaml

Create the add-on using the AWS CLI. If you want to use the consolelong or eksctl to create the add-on, see creating-an-add-on.title and specify coredns for the add-on name. Copy the command that follows to your device. Make the following modifications to the command, as needed, and then run the modified command.
- Replace my-cluster with the name of your cluster.
- Replace v1.11.3-eksbuild.1 with the latest version listed in the latest version table for your cluster version.
  aws eks create-addon --cluster-name my-cluster --addon-name coredns --addon-version v1.11.3-eksbuild.1
  If you’ve applied custom settings to your current add-on that conflict with the default settings of the Amazon EKS add-on, creation might fail. If creation fails, you receive an error that can help you resolve the issue. Alternatively, you can add --resolve-conflicts OVERWRITE to the previous command. This allows the add-on to overwrite any existing custom settings. Once you’ve created the add-on, you can update it with your custom settings.
Confirm that the latest version of the add-on for your cluster’s Kubernetes version was added to your cluster. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name coredns --query addon.addonVersion --output text
```
It might take several seconds for add-on creation to complete.

An example output is as follows.
```
v1.11.3-eksbuild.1
```
If you made custom settings to your original add-on, before you created the Amazon EKS add-on, use the configuration that you saved in a previous step to update the Amazon EKS add-on with your custom settings. For instructions to update the add-on, see coredns-add-on-update.title.

Update the `CoreDNS` Amazon EKS add-on

Update the Amazon EKS type of the add-on. If you haven’t added the Amazon EKS add-on to your cluster, either add it or see coredns-add-on-self-managed-update.title.

Before you begin, review the upgrade considerations. For more information, see coredns-upgrade.title.

See which version of the add-on is installed on your cluster. Replace my-cluster with your cluster name.
```
aws eks describe-addon --cluster-name my-cluster --addon-name coredns --query "addon.addonVersion" --output text
```
An example output is as follows.
```
v1.10.1-eksbuild.13
```
If the version returned is the same as the version for your cluster’s Kubernetes version in the latest version table, then you already have the latest version installed on your cluster and don’t need to complete the rest of this procedure. If you receive an error, instead of a version number in your output, then you don’t have the Amazon EKS type of the add-on installed on your cluster. You need to create the add-on before you can update it with this procedure.

Save the configuration of your currently installed add-on.

kubectl get deployment coredns -n kube-system -o yaml > aws-k8s-coredns-old.yaml

Update your add-on using the AWS CLI. If you want to use the consolelong or eksctl to update the add-on, see updating-an-add-on.title. Copy the command that follows to your device. Make the following modifications to the command, as needed, and then run the modified command.
- Replace my-cluster with the name of your cluster.
- Replace v1.11.3-eksbuild.1 with the latest version listed in the latest version table for your cluster version.
- The --resolve-conflictsPRESERVE option preserves existing configuration values for the add-on. If you’ve set custom values for add-on settings, and you don’t use this option, Amazon EKS overwrites your values with its default values. If you use this option, then we recommend testing any field and value changes on a non-production cluster before updating the add-on on your production cluster. If you change this value to OVERWRITE, all settings are changed to Amazon EKS default values. If you’ve set custom values for any settings, they might be overwritten with Amazon EKS default values. If you change this value to none, Amazon EKS doesn’t change the value of any settings, but the update might fail. If the update fails, you receive an error message to help you resolve the conflict.
- If you’re not updating a configuration setting, remove --configuration-values '{"replicaCount":3}' from the command. If you’re updating a configuration setting, replace "replicaCount":3 with the setting that you want to set. In this example, the number of replicas of CoreDNS is set to 3. The value that you specify must be valid for the configuration schema. If you don’t know the configuration schema, run aws eks describe-addon-configuration --addon-name coredns --addon-version v1.11.3-eksbuild.1, replacing v1.11.3-eksbuild.1 with the version number of the add-on that you want to see the configuration for. The schema is returned in the output. If you have any existing custom configuration, want to remove it all, and set the values for all settings back to Amazon EKS defaults, remove "replicaCount":3 from the command, so that you have empty {}. For more information about CoreDNS settings, see Customizing DNS Service in the Kubernetes documentation.
  aws eks update-addon --cluster-name my-cluster --addon-name coredns --addon-version v1.11.3-eksbuild.1 \ --resolve-conflicts PRESERVE --configuration-values '{"replicaCount":3}'
  It might take several seconds for the update to complete.

Confirm that the add-on version was updated. Replace my-cluster with the name of your cluster.

aws eks describe-addon --cluster-name my-cluster --addon-name coredns

It might take several seconds for the update to complete.

An example output is as follows.

{
    "addon": {
        "addonName": "coredns",
        "clusterName": "my-cluster",
        "status": "ACTIVE",
        "addonVersion": "v1.11.3-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "region.arneks:region:111122223333:addon/my-cluster/coredns/d2c34f06-1111-2222-1eb0-24f64ce37fa4",
        "createdAt": "2023-03-01T16:41:32.442000+00:00",
        "modifiedAt": "2023-03-01T18:16:54.332000+00:00",
        "tags": {},
        "configurationValues": "{\"replicaCount\":3}"
    }
}

Update the `CoreDNS` Amazon EKS self-managed add-on

Before you begin, review the upgrade considerations. For more information, see coredns-upgrade.title.

Confirm that you have the self-managed type of the add-on installed on your cluster. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name coredns --query addon.addonVersion --output text
```
If an error message is returned, you have the self-managed type of the add-on installed on your cluster. Complete the remaining steps in this procedure. If a version number is returned, you have the Amazon EKS type of the add-on installed on your cluster. To update the Amazon EKS type of the add-on, use the procedure in Update the CoreDNS Amazon EKS add-on, rather than using this procedure. If you’re not familiar with the differences between the add-on types, see eks-add-ons.title.
See which version of the container image is currently installed on your cluster.
```
kubectl describe deployment coredns -n kube-system | grep Image | cut -d ":" -f 3
```
An example output is as follows.
```
v1.8.7-eksbuild.2
```
If your current CoreDNS version is v1.5.0 or later, but earlier than the version listed in the CoreDNS versions table, then skip this step. If your current version is earlier than 1.5.0, then you need to modify the ConfigMap for CoreDNS to use the forward add-on, rather than the proxy add-on.
1. Open the ConfigMap with the following command.
  kubectl edit configmap coredns -n kube-system
2. Replace proxy in the following line with forward. Save the file and exit the editor.
  proxy . /etc/resolv.conf
If you originally deployed your cluster on Kubernetes 1.17 or earlier, then you may need to remove a discontinued line from your CoreDNS manifest.

You must complete this step before updating to CoreDNS version 1.7.0, but it’s recommended that you complete this step even if you’re updating to an earlier version.
1. Check to see if your CoreDNS manifest has the line.
  kubectl get configmap coredns -n kube-system -o jsonpath='{$.data.Corefile}' | grep upstream
  If no output is returned, your manifest doesn’t have the line and you can skip to the next step to update CoreDNS. If output is returned, then you need to remove the line.
2. Edit the ConfigMap with the following command, removing the line in the file that has the word upstream in it. Do not change anything else in the file. Once the line is removed, save the changes.
  kubectl edit configmap coredns -n kube-system -o yaml

Retrieve your current CoreDNS image version:

kubectl describe deployment coredns -n kube-system | grep Image

An example output is as follows.

602401143452.dkr.ecr.region-code.amazonaws.com/eks/coredns:v1.8.7-eksbuild.2

If you’re updating to CoreDNS 1.8.3 or later, then you need to add the endpointslices permission to the system:coredns Kubernetes clusterrole.
```
kubectl edit clusterrole system:coredns -n kube-system
```
Add the following lines under the existing permissions lines in the rules section of the file.
```
[...]
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
[...]
```
Update the CoreDNS add-on by replacing 602401143452 and region-code with the values from the output returned in a previous step. Replace v1.11.3-eksbuild.1 with the CoreDNS version listed in the latest versions table for your Kubernetes version.
```
kubectl set image deployment.apps/coredns -n kube-system  coredns=602401143452.dkr.ecr.region-code.amazonaws.com/eks/coredns:v1.11.3-eksbuild.1
```
An example output is as follows.
```
deployment.apps/coredns image updated
```
Check the container image version again to confirm that it was updated to the version that you specified in the previous step.
```
kubectl describe deployment coredns -n kube-system | grep Image | cut -d ":" -f 3
```
An example output is as follows.
```
v1.11.3-eksbuild.1
```

Scale `CoreDNS Pods` for high DNS traffic

Learn how the Amazon EKS add-on for CoreDNS autoscales to handle increased load on DNS pods, improving application availability and cluster scalability.

When you launch an Amazon EKS cluster with at least one node, a Deployment of two replicas of the CoreDNS image are deployed by default, regardless of the number of nodes deployed in your cluster. The CoreDNS Pods provide name resolution for all Pods in the cluster. Applications use name resolution to connect to pods and services in the cluster as well as connecting to services outside the cluster. As the number of requests for name resolution (queries) from pods increase, the CoreDNS pods can get overwhelmed and slow down, and reject requests that the pods can’t handle.

To handle the increased load on the CoreDNS pods, consider an autoscaling system for CoreDNS. Amazon EKS can manage the autoscaling of the CoreDNS Deployment in the EKS Add-on version of CoreDNS. This CoreDNS autoscaler continuously monitors the cluster state, including the number of nodes and CPU cores. Based on that information, the controller will dynamically adapt the number of replicas of the CoreDNS deployment in an EKS cluster. This feature works for CoreDNS v1.9 and EKS release version 1.25 and later. For more information about which versions are compatible with CoreDNS Autoscaling, see the following section.

We recommend using this feature in conjunction with other EKS Cluster Autoscaling best practices to improve overall application availability and cluster scalability.

Prerequisites

For Amazon EKS to scale your CoreDNS deployment, there are three prerequisites:

You must be using the EKS Add-on version of CoreDNS.
Your cluster must be running at least the minimum cluster versions and platform versions.
Your cluster must be running at least the minimum version of the EKS Add-on of CoreDNS.

Minimum cluster version

Autoscaling of CoreDNS is done by a new component in the cluster control plane, managed by Amazon EKS. Because of this, you must upgrade your cluster to an EKS release that supports the minimum platform version that has the new component.

A new Amazon EKS cluster. To deploy one, see getting-started.title. The cluster must be Kubernetes version 1.25 or later. The cluster must be running one of the Kubernetes versions and platform versions listed in the following table or a later version. Note that any Kubernetes and platform versions later than those listed are also supported. You can check your current Kubernetes version by replacing my-cluster in the following command with the name of your cluster and then running the modified command:

aws eks describe-cluster
              --name my-cluster --query cluster.version --output
              text

Kubernetes version Platform version

1.29.3

eks.7

1.28.8

eks.13

1.27.12

eks.17

1.26.15

eks.18

1.25.16

eks.19

Every platform version of later Kubernetes versions are also supported, for example Kubernetes version 1.30 from eks.1 and on.

Minimum EKS Add-on version

Kubernetes version 1.29 1.28 1.27 1.26 1.25

v1.11.1-eksbuild.9

v1.10.1-eksbuild.11

v1.9.3-eksbuild.15

Configuring CoreDNS autoscaling in the consolelong

Ensure that your cluster is at or above the minimum cluster version.

Amazon EKS upgrades clusters between platform versions of the same Kubernetes version automatically, and you can’t start this process yourself. Instead, you can upgrade your cluster to the next Kubernetes version, and the cluster will be upgraded to that K8s version and the latest platform version. For example, if you upgrade from 1.25 to 1.26, the cluster will upgrade to 1.26.15 eks.18.

New Kubernetes versions sometimes introduce significant changes. Therefore, we recommend that you test the behavior of your applications by using a separate cluster of the new Kubernetes version before you update your production clusters.

To upgrade a cluster to a new Kubernetes version, follow the procedure in Update existing cluster to new Kubernetes version.
Ensure that you have the EKS Add-on for CoreDNS, not the self-managed CoreDNS Deployment.

Depending on the tool that you created your cluster with, you might not currently have the Amazon EKS add-on type installed on your cluster. To see which type of the add-on is installed on your cluster, you can run the following command. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name coredns --query addon.addonVersion --output text
```
If a version number is returned, you have the Amazon EKS type of the add-on installed on your cluster and you can continue with the next step. If an error is returned, you don’t have the Amazon EKS type of the add-on installed on your cluster. Complete the remaining steps of the procedure Create the CoreDNS Amazon EKS add-on to replace the self-managed version with the Amazon EKS add-on.
Ensure that your EKS Add-on for CoreDNS is at a version the same or higher than the minimum EKS Add-on version.

See which version of the add-on is installed on your cluster. You can check in the consolelong or run the following command:
```
kubectl describe deployment coredns --namespace kube-system | grep coredns: | cut -d : -f 3
```
An example output is as follows.
```
v1.10.1-eksbuild.13
```
Compare this version with the minimum EKS Add-on version in the previous section. If needed, upgrade the EKS Add-on to a higher version by following the procedure Update the CoreDNS Amazon EKS add-on.
Add the autoscaling configuration to the Optional configuration settings of the EKS Add-on.
1. Open the Amazon EKS console.
2. In the left navigation pane, select Clusters, and then select the name of the cluster that you want to configure the add-on for.
3. Choose the Add-ons tab.
4. Select the box in the top right of the CoreDNS add-on box and then choose Edit.
5. On the Configure CoreDNS page:
  1. Select the Version that you’d like to use. We recommend that you keep the same version as the previous step, and update the version and configuration in separate actions.
  2. Expand the Optional configuration settings.
  3. Enter the JSON key "autoscaling": and value of a nested JSON object with a key "enabled": and value true in Configuration values. The resulting text must be a valid JSON object. If this key and value are the only data in the text box, surround the key and value with curly braces { }. The following example shows autoscaling is enabled:
    
    { "autoScaling": { "enabled": true } }
  4. (Optional) You can provide minimum and maximum values that autoscaling can scale the number of CoreDNS pods to.
    
    The following example shows autoscaling is enabled and all of the optional keys have values. We recommend that the minimum number of CoreDNS pods is always greater than 2 to provide resilience for the DNS service in the cluster.
    
    { "autoScaling": { "enabled": true, "minReplicas": 2, "maxReplicas": 10 } }
6. To apply the new configuration by replacing the CoreDNS pods, choose Save changes.
  
  Amazon EKS applies changes to the EKS Add-ons by using a rollout of the Kubernetes Deployment for CoreDNS. You can track the status of the rollout in the Update history of the add-on in the consolelong and with kubectl rollout status deployment/coredns --namespace kube-system.
  
  kubectl rollout has the following commands:
  kubectl rollout history -- View rollout history pause -- Mark the provided resource as paused restart -- Restart a resource resume -- Resume a paused resource status -- Show the status of the rollout undo -- Undo a previous rollout
  If the rollout takes too long, Amazon EKS will undo the rollout, and a message with the type of Addon Update and a status of Failed will be added to the Update history of the add-on. To investigate any issues, start from the history of the rollout, and run kubectl logs on a CoreDNS pod to see the logs of CoreDNS.
If the new entry in the Update history has a status of Successful, then the rollout has completed and the add-on is using the new configuration in all of the CoreDNS pods. As you change the number of nodes and CPU cores of nodes in the cluster, Amazon EKS scales the number of replicas of the CoreDNS deployment.

Configuring CoreDNS autoscaling in the AWS Command Line Interface

Ensure that your cluster is at or above the minimum cluster version.

Amazon EKS upgrades clusters between platform versions of the same Kubernetes version automatically, and you can’t start this process yourself. Instead, you can upgrade your cluster to the next Kubernetes version, and the cluster will be upgraded to that K8s version and the latest platform version. For example, if you upgrade from 1.25 to 1.26, the cluster will upgrade to 1.26.15 eks.18.

New Kubernetes versions sometimes introduce significant changes. Therefore, we recommend that you test the behavior of your applications by using a separate cluster of the new Kubernetes version before you update your production clusters.

To upgrade a cluster to a new Kubernetes version, follow the procedure in Update existing cluster to new Kubernetes version.
Ensure that you have the EKS Add-on for CoreDNS, not the self-managed CoreDNS Deployment.

Depending on the tool that you created your cluster with, you might not currently have the Amazon EKS add-on type installed on your cluster. To see which type of the add-on is installed on your cluster, you can run the following command. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name coredns --query addon.addonVersion --output text
```
If a version number is returned, you have the Amazon EKS type of the add-on installed on your cluster. If an error is returned, you don’t have the Amazon EKS type of the add-on installed on your cluster. Complete the remaining steps of the procedure Create the CoreDNS Amazon EKS add-on to replace the self-managed version with the Amazon EKS add-on.
Ensure that your EKS Add-on for CoreDNS is at a version the same or higher than the minimum EKS Add-on version.

See which version of the add-on is installed on your cluster. You can check in the consolelong or run the following command:
```
kubectl describe deployment coredns --namespace kube-system | grep coredns: | cut -d : -f 3
```
An example output is as follows.
```
v1.10.1-eksbuild.13
```
Compare this version with the minimum EKS Add-on version in the previous section. If needed, upgrade the EKS Add-on to a higher version by following the procedure Update the CoreDNS Amazon EKS add-on.
Add the autoscaling configuration to the Optional configuration settings of the EKS Add-on.

Run the following AWS CLI command. Replace my-cluster with the name of your cluster and the IAM role ARN with the role that you are using.
```
aws eks update-addon --cluster-name my-cluster --addon-name coredns \
    --resolve-conflicts PRESERVE --configuration-values '{"autoScaling":{"enabled":true}}'
```
Amazon EKS applies changes to the EKS Add-ons by using a rollout of the Kubernetes Deployment for CoreDNS. You can track the status of the rollout in the Update history of the add-on in the consolelong and with kubectl rollout status deployment/coredns --namespace kube-system.

kubectl rollout has the following commands:
```
kubectl rollout

history  -- View rollout history
pause    -- Mark the provided resource as paused
restart  -- Restart a resource
resume   -- Resume a paused resource
status   -- Show the status of the rollout
undo     -- Undo a previous rollout
```
If the rollout takes too long, Amazon EKS will undo the rollout, and a message with the type of Addon Update and a status of Failed will be added to the Update history of the add-on. To investigate any issues, start from the history of the rollout, and run kubectl logs on a CoreDNS pod to see the logs of CoreDNS.
(Optional) You can provide minimum and maximum values that autoscaling can scale the number of CoreDNS pods to.

The following example shows autoscaling is enabled and all of the optional keys have values. We recommend that the minimum number of CoreDNS pods is always greater than 2 to provide resilience for the DNS service in the cluster.
```
aws eks update-addon --cluster-name my-cluster --addon-name coredns \
    --resolve-conflicts PRESERVE --configuration-values '{"autoScaling":{"enabled":true,"minReplicas":2,"maxReplicas":10}}'
```
Check the status of the update to the add-on by running the following command:
```
aws eks describe-addon --cluster-name my-cluster --addon-name coredns \
```
If you see this line: "status": "ACTIVE", then the rollout has completed and the add-on is using the new configuration in all of the CoreDNS pods. As you change the number of nodes and CPU cores of nodes in the cluster, Amazon EKS scales the number of replicas of the CoreDNS deployment.

Monitor `Kubernetes` DNS resolution with `CoreDNS` metrics

Learn how to collect CoreDNS metrics in Amazon EKS using Prometheus or CloudWatch Agent, enabling monitoring and observability for your Kubernetes DNS resolution.

CoreDNS as an EKS add-on exposes the metrics from CoreDNS on port 9153 in the Prometheus format in the kube-dns service. You can use Prometheus, the Amazon CloudWatch agent, or any other compatible system to scrape (collect) these metrics.

For an example scrape configuration that is compatible with both Prometheus and the CloudWatch agent, see CloudWatch agent configuration for Prometheus in the Amazon CloudWatch User Guide.

Learn how to manage the CoreDNS Amazon EKS add-on for DNS service discovery in Kubernetes clusters with configuration updates and version upgrades.

With Amazon EKS Auto Mode, you don’t need to install or upgrade networking add-ons. Auto Mode includes pod networking and load balancing capabilities.

For more information, see automode.title.

CoreDNS is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS. When you launch an Amazon EKS cluster with at least one node, two replicas of the CoreDNS image are deployed by default, regardless of the number of nodes deployed in your cluster. The CoreDNS Pods provide name resolution for all Pods in the cluster. The CoreDNS Pods can be deployed to Fargate nodes if your cluster includes a Fargate Profile with a namespace that matches the namespace for the CoreDNS deployment. For more information on Fargate Profiles, see fargate-profile.title. For more information about CoreDNS, see Using CoreDNS for Service Discovery in the Kubernetes documentation.

`CoreDNS` versions

The following table lists the latest version of the Amazon EKS add-on type for each Kubernetes version.

Kubernetes version	`CoreDNS` version
1.31	v1.11.4-eksbuild.2
1.30	v1.11.4-eksbuild.2
1.29	v1.11.4-eksbuild.2
1.28	v1.10.1-eksbuild.17
1.27	v1.10.1-eksbuild.17
1.26	v1.9.3-eksbuild.21
1.25	v1.9.3-eksbuild.21
1.24	v1.9.3-eksbuild.21
1.23	v1.8.7-eksbuild.20

Important `CoreDNS` upgrade considerations

To improve the stability and availability of the CoreDNS Deployment, versions v1.9.3-eksbuild.6 and later and v1.10.1-eksbuild.3 are deployed with a PodDisruptionBudget. If you’ve deployed an existing PodDisruptionBudget, your upgrade to these versions might fail. If the upgrade fails, completing one of the following tasks should resolve the issue:
- When doing the upgrade of the Amazon EKS add-on, choose to override the existing settings as your conflict resolution option. If you’ve made other custom settings to the Deployment, make sure to back up your settings before upgrading so that you can reapply your other custom settings after the upgrade.
- Remove your existing PodDisruptionBudget and try the upgrade again.
In EKS add-on versions v1.9.3-eksbuild.3 and later and v1.10.1-eksbuild.6 and later, the CoreDNS Deployment sets the readinessProbe to use the /ready endpoint. This endpoint is enabled in the Corefile configuration file for CoreDNS.

If you use a custom Corefile, you must add the ready plugin to the config, so that the /ready endpoint is active in CoreDNS for the probe to use.
In EKS add-on versions v1.9.3-eksbuild.7 and later and v1.10.1-eksbuild.4 and later, you can change the PodDisruptionBudget. You can edit the add-on and change these settings in the Optional configuration settings using the fields in the following example. This example shows the default PodDisruptionBudget.
```
{
    "podDisruptionBudget": {
        "enabled": true,
        "maxUnavailable": 1
        }
}
```
You can set maxUnavailable or minAvailable, but you can’t set both in a single PodDisruptionBudget. For more information about PodDisruptionBudgets, see Specifying a PodDisruptionBudget in the Kubernetes documentation.

Note that if you set enabled to false, the PodDisruptionBudget isn’t removed. After you set this field to false, you must delete the PodDisruptionBudget object. Similarly, if you edit the add-on to use an older version of the add-on (downgrade the add-on) after upgrading to a version with a PodDisruptionBudget, the PodDisruptionBudget isn’t removed. To delete the PodDisruptionBudget, you can run the following command:
```
kubectl delete poddisruptionbudget coredns -n kube-system
```
In EKS add-on versions v1.10.1-eksbuild.5 and later, change the default toleration from node-role.kubernetes.io/master:NoSchedule to node-role.kubernetes.io/control-plane:NoSchedule to comply with KEP 2067. For more information about KEP 2067, see KEP-2067: Rename the kubeadm "master" label and taint in the Kubernetes Enhancement Proposals (KEPs) on GitHub.

In EKS add-on versions v1.8.7-eksbuild.8 and later and v1.9.3-eksbuild.9 and later, both tolerations are set to be compatible with every Kubernetes version.
In EKS add-on versions v1.9.3-eksbuild.11 and v1.10.1-eksbuild.7 and later, the CoreDNS Deployment sets a default value for topologySpreadConstraints. The default value ensures that the CoreDNS Pods are spread across the Availability Zones if there are nodes in multiple Availability Zones available. You can set a custom value that will be used instead of the default value. The default value follows:
```
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        k8s-app: kube-dns
```

`CoreDNS` `v1.11` upgrade considerations

In EKS add-on versions v1.11.1-eksbuild.4 and later, the container image is based on a minimal base image maintained by Amazon EKS Distro, which contains minimal packages and doesn’t have shells. For more information, see Amazon EKS Distro. The usage and troubleshooting of the CoreDNS image remains the same.

11.4.5. Manage `kube-proxy` in Amazon EKS clusters

Update the Kubernetes `kube-proxy` self-managed add-on

Prerequisites

An existing Amazon EKS cluster. To deploy one, see getting-started.title.

Considerations

Kube-proxy on an Amazon EKS cluster has the same compatibility and skew policy as Kubernetes. Learn how to Verifying Amazon EKS add-on version compatibility with a cluster.

Confirm that you have the self-managed type of the add-on installed on your cluster. Replace my-cluster with the name of your cluster.
```
aws eks describe-addon --cluster-name my-cluster --addon-name kube-proxy --query addon.addonVersion --output text
```
If an error message is returned, you have the self-managed type of the add-on installed on your cluster. The remaining steps in this topic are for updating the self-managed type of the add-on. If a version number is returned, you have the Amazon EKS type of the add-on installed on your cluster. To update it, use the procedure in Updating an Amazon EKS add-on, rather than using the procedure in this topic. If you’re not familiar with the differences between the add-on types, see eks-add-ons.title.
See which version of the container image is currently installed on your cluster.
```
kubectl describe daemonset kube-proxy -n kube-system | grep Image
```
An example output is as follows.
```
Image:    602401143452.dkr.ecr.region-code.amazonaws.com/eks/kube-proxy:v1.29.1-eksbuild.2
```
In the example output, v1.29.1-eksbuild.2 is the version installed on the cluster.

Update the kube-proxy add-on by replacing 602401143452 and region-code with the values from your output in the previous step. Replace v1.30.6-eksbuild.3 with the kube-proxy version listed in the Latest available self-managed kube-proxy container image version for each Amazon EKS cluster version table.

The manifests for each image type are different and not compatible between the default or minimal image types. You must use the same image type as the previous image, so that the entrypoint and arguments match.

kubectl set image daemonset.apps/kube-proxy -n kube-system kube-proxy=602401143452.dkr.ecr.region-code.amazonaws.com/eks/kube-proxy:v1.30.6-eksbuild.3

An example output is as follows.

daemonset.apps/kube-proxy image updated

Confirm that the new version is now installed on your cluster.

kubectl describe daemonset kube-proxy -n kube-system | grep Image | cut -d ":" -f 3

An example output is as follows.

v1.30.0-eksbuild.3

If you’re using x86 and Arm nodes in the same cluster and your cluster was deployed before August 17, 2020. Then, edit your kube-proxy manifest to include a node selector for multiple hardware architectures with the following command. This is a one-time operation. After you’ve added the selector to your manifest, you don’t need to add it each time you update the add-on. If your cluster was deployed on or after August 17, 2020, then kube-proxy is already multi-architecture capable.
```
kubectl edit -n kube-system daemonset/kube-proxy
```
Add the following node selector to the file in the editor and then save the file. For an example of where to include this text in the editor, see the CNI manifest file on GitHub. This enables Kubernetes to pull the correct hardware image based on the node’s hardware architecture.
```
- key: "kubernetes.io/arch"
  operator: In
  values:
  - amd64
  - arm64
```
If your cluster was originally created with Kubernetes version 1.14 or later, then you can skip this step because kube-proxy already includes this Affinity Rule. If you originally created an Amazon EKS cluster with Kubernetes version 1.13 or earlier and intend to use Fargate nodes in your cluster, then edit your kube-proxy manifest to include a NodeAffinity rule to prevent kube-proxy Pods from scheduling on Fargate nodes. This is a one-time edit. Once you’ve added the Affinity Rule to your manifest, you don’t need to add it each time that you update the add-on. Edit your kube-proxy DaemonSet.
```
kubectl edit -n kube-system daemonset/kube-proxy
```
Add the following Affinity Rule to the DaemonSet`spec` section of the file in the editor and then save the file. For an example of where to include this text in the editor, see the CNI manifest file on GitHub.
```
- key: eks.amazonaws.com/compute-type
  operator: NotIn
  values:
  - fargate
```

Learn how to manage the kube-proxy add-on on your Amazon EKS cluster to manage network rules and enable network communication to your Pods.

With Amazon EKS Auto Mode, you don’t need to install or upgrade networking add-ons. Auto Mode includes pod networking and load balancing capabilities.

For more information, see automode.title.

The kube-proxy add-on is deployed on each Amazon EC2 node in your Amazon EKS cluster. It maintains network rules on your nodes and enables network communication to your Pods. The add-on isn’t deployed to Fargate nodes in your cluster. For more information, see kube-proxy in the Kubernetes documentation.

Install as Amazon EKS Add-on

`kube-proxy` versions

The following table lists the latest version of the Amazon EKS add-on type for each Kubernetes version.

Kubernetes version	`kube-proxy` version
1.31	v1.31.3-eksbuild.2
1.30	v1.30.7-eksbuild.2
1.29	v1.29.11-eksbuild.2
1.28	v1.28.15-eksbuild.4
1.27	v1.27.16-eksbuild.14
1.26	v1.26.15-eksbuild.19
1.25	v1.25.16-eksbuild.22
1.24	v1.24.17-eksbuild.19
1.23	v1.23.17-eksbuild.20

An earlier version of the documentation was incorrect. kube-proxy versions v1.28.5, v1.27.9, and v1.26.12 aren’t available.

If you’re self-managing this add-on, the versions in the table might not be the same as the available self-managed versions.

`kube-proxy` container image migration

There are two types of the kube-proxy container image available for each Amazon EKS cluster version:

Default – This image type is based on a Debian-based Docker image that is maintained by the Kubernetes upstream community.
Minimal – This image type is based on a minimal base image maintained by Amazon EKS Distro, which contains minimal packages and doesn’t have shells. For more information, see Amazon EKS Distro.

The following table lists the latest available self-managed kube-proxy container image version for each Amazon EKS cluster version.

Version

kube-proxy (default type)

kube-proxy (minimal type)

1.31

Only minimal type is available

v1.31.2-minimal-eksbuild.3

1.30

Only minimal type is available

v1.30.6-minimal-eksbuild.3

1.29

Only minimal type is available

v1.29.10-minimal-eksbuild.3

1.28

Only minimal type is available

v1.28.15-minimal-eksbuild.4

1.27

Only minimal type is available

v1.27.16-minimal-eksbuild.14

1.26

Only minimal type is available

v1.26.15-minimal-eksbuild.19

1.25

Only minimal type is available

v1.25.16-minimal-eksbuild.22

1.24

v1.24.10-eksbuild.2

v1.24.17-minimal-eksbuild.19

1.23

v1.23.16-eksbuild.2

v1.23.17-minimal-eksbuild.20

The default image type isn’t available for Kubernetes version 1.25 and later. You must use the minimal image type.
When you update an Amazon EKS add-on type, you specify a valid Amazon EKS add-on version, which might not be a version listed in this table. This is because Amazon EKS add-on versions don’t always match container image versions specified when updating the self-managed type of this add-on. When you update the self-managed type of this add-on, you specify a valid container image version listed in this table.

Learn how to manage networking add-ons for your Amazon EKS cluster, including built-in components like Amazon VPC CNI plugin for Kubernetes, CoreDNS, and kube-proxy, as well as optional AWS add-ons for load balancing and service mesh.

Several networking add-ons are available for your Amazon EKS cluster.

11.4.6. Built-in add-ons

If you create clusters in any way except by using the console, each cluster comes with the self-managed versions of the built-in add-ons. The self-managed versions can’t be managed from the consolelong, AWS Command Line Interface, or SDKs. You manage the configuration and upgrades of self-managed add-ons.

We recommend adding the Amazon EKS type of the add-on to your cluster instead of using the self-managed type of the add-on. If you create clusters in the console, the Amazon EKS type of these add-ons is installed.

Amazon VPC CNI plugin for Kubernetes: This CNI add-on creates elastic network interfaces and attaches them to your Amazon EC2 nodes. The add-on also assigns a private IPv4 or IPv6 address from your VPC to each Pod and service. This add-on is installed, by default, on your cluster. For more information, see managing-vpc-cni.title. If you are using hybrid nodes, the VPC CNI is still installed by default but it is prevented from running on your hybrid nodes with an anti-affinity rule. For more information about your CNI options for hybrid nodes, see hybrid-nodes-cni.title.
CoreDNS: CoreDNS is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS. CoreDNS provides name resolution for all Pods in the cluster. This add-on is installed, by default, on your cluster. For more information, see managing-coredns.title.
kube-proxy: This add-on maintains network rules on your Amazon EC2 nodes and enables network communication to your Pods. This add-on is installed, by default, on your cluster. For more information, see managing-kube-proxy.title.

11.4.7. Optional `AWS` networking add-ons

AWS Load Balancer Controller: When you deploy Kubernetes service objects of type loadbalancer, the controller creates AWS Network Load Balancers . When you create Kubernetes ingress objects, the controller creates AWS Application Load Balancers. We recommend using this controller to provision Network Load Balancers, rather than using the legacy Cloud Provider controller built-in to Kubernetes. For more information, see the AWS Load Balancer Controller documentation.
AWS Gateway API Controller: This controller lets you connect services across multiple Kubernetes clusters using the Kubernetes gateway API. The controller connects Kubernetes services running on Amazon EC2 instances, containers, and serverless functions by using the Amazon VPC Lattice service. For more information, see the AWS Gateway API Controller documentation.

For more information about add-ons, see eks-add-ons.title.

Learn how to configure networking for your Amazon EKS cluster using a VPC, subnets, security groups, and networking add-ons to ensure secure and efficient communication.

Your Amazon EKS cluster is created in a VPC. Pod networking is provided by the Amazon VPC Container Network Interface (CNI) plugin for nodes that run on AWS infrastructure. If you are running nodes on your own infrastructure, see hybrid-nodes-cni.title. This chapter includes the following topics for learning more about networking for your cluster.

[[Topic List]]

12. Learn how to deploy workloads and add-ons to Amazon EKS

Your workloads are deployed in containers, which are deployed in Pods in Kubernetes. A Pod includes one or more containers. Typically, one or more Pods that provide the same service are deployed in a Kubernetes service. Once you’ve deployed multiple Pods that provide the same service, you can:

View information about the workloads running on each of your clusters using the consolelong.
Vertically scale Pods up or down with the Kubernetes Vertical Pod Autoscaler.
Horizontally scale the number of Pods needed to meet demand up or down with the Kubernetes Horizontal Pod Autoscaler.
Create an external (for internet-accessible Pods) or an internal (for private Pods) network load balancer to balance network traffic across Pods. The load balancer routes traffic at Layer 4 of the OSI model.
Create an Application Load Balancer to balance application traffic across Pods. The application load balancer routes traffic at Layer 7 of the OSI model.
If you’re new to Kubernetes, this topic helps you Deploy a sample application.
You can restrict IP addresses that can be assigned to a service with externalIPs.

12.1. Deploy a sample application on Linux

In this topic, you deploy a sample application to your cluster on linux nodes.

12.1.1. Prerequisites

An existing Kubernetes cluster with at least one node. If you don’t have an existing Amazon EKS cluster, you can deploy one using one of the guides in getting-started.title.
Kubectl installed on your computer. For more information, see install-kubectl.title.
Kubectl configured to communicate with your cluster. For more information, see create-kubeconfig.title.
If you plan to deploy your sample workload to Fargate, then you must have an existing Fargate profile that includes the same namespace created in this tutorial, which is eks-sample-app, unless you change the name. If you created a cluster with one of the gudes in getting-started.title, then you’ll have to create a new profile, or add the namespace to your existing profile, because the profile created in the getting started guides doesn’t specify the namespace used in this tutorial. Your VPC must also have at least one private subnet.

Though many variables are changeable in the following steps, we recommend only changing variable values where specified. Once you have a better understanding of Kubernetes Pods, deployments, and services, you can experiment with changing other values.

12.1.2. Create a namespace

A namespace allows you to group resources in Kubernetes. For more information, see Namespaces in the Kubernetes documentation. If you plan to deploy your sample application to Simplify compute management with AWS Fargate, make sure that the value for namespace in your Define which Pods use AWS Fargate when launched is eks-sample-app.

kubectl create namespace eks-sample-app

12.1.3. Create a Kubernetes deployment

Create a Kubernetes deployment. This sample deployment pulls a container image from a public repository and deploys three replicas (individual Pods) of it to your cluster. To learn more, see Deployments in the Kubernetes documentation.

Save the following contents to a file named eks-sample-deployment.yaml. The containers in the sample application don’t use network storage, but you might have applications that need to. For more information, see storage.title.
- The amd64 or arm64 values under the kubernetes.io/arch key mean that the application can be deployed to either hardware architecture (if you have both in your cluster). This is possible because this image is a multi-architecture image, but not all are. You can determine the hardware architecture that the image is supported on by viewing the image details in the repository that you’re pulling it from. When deploying images that don’t support a hardware architecture type, or that you don’t want the image deployed to, remove that type from the manifest. For more information, see Well-Known Labels, Annotations and Taints in the Kubernetes documentation.
- The kubernetes.io/os: linux nodeSelector means that if you had Linux and Windows nodes (for example) in your cluster, the image would only be deployed to Linux nodes. For more information, see Well-Known Labels, Annotations and Taints in the Kubernetes documentation.
  apiVersion: apps/v1 kind: Deployment metadata: name: eks-sample-linux-deployment namespace: eks-sample-app labels: app: eks-sample-linux-app spec: replicas: 3 selector: matchLabels: app: eks-sample-linux-app template: metadata: labels: app: eks-sample-linux-app spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/arch operator: In values: - amd64 - arm64 containers: - name: nginx image: public.ecr.aws/nginx/nginx:1.23 ports: - name: http containerPort: 80 imagePullPolicy: IfNotPresent nodeSelector: kubernetes.io/os: linux
Apply the deployment manifest to your cluster.
```
kubectl apply -f eks-sample-deployment.yaml
```

12.1.4. Create a service

A service allows you to access all replicas through a single IP address or name. For more information, see Service in the Kubernetes documentation. Though not implemented in the sample application, if you have applications that need to interact with other AWS services, we recommend that you create Kubernetes service accounts for your Pods, and associate them to AWS IAM accounts. By specifying service accounts, your Pods have only the minimum permissions that you specify for them to interact with other services. For more information, see iam-roles-for-service-accounts.title.

Save the following contents to a file named eks-sample-service.yaml. Kubernetes assigns the service its own IP address that is accessible only from within the cluster. To access the service from outside of your cluster, deploy the AWS Load Balancer Controller to load balance application or network traffic to the service.
```
apiVersion: v1
kind: Service
metadata:
  name: eks-sample-linux-service
  namespace: eks-sample-app
  labels:
    app: eks-sample-linux-app
spec:
  selector:
    app: eks-sample-linux-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
```
Apply the service manifest to your cluster.
```
kubectl apply -f eks-sample-service.yaml
```

12.1.5. Review resources created

View all resources that exist in the eks-sample-app namespace.

kubectl get all -n eks-sample-app

An example output is as follows.

NAME                                               READY   STATUS    RESTARTS   AGE
pod/eks-sample-linux-deployment-65b7669776-m6qxz   1/1     Running   0          27m
pod/eks-sample-linux-deployment-65b7669776-mmxvd   1/1     Running   0          27m
pod/eks-sample-linux-deployment-65b7669776-qzn22   1/1     Running   0          27m

NAME                               TYPE         CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/eks-sample-linux-service   ClusterIP    10.100.74.8     <none>        80/TCP    32m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/eks-sample-linux-deployment 3/3     3            3           27m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/eks-sample-linux-deployment-776d8f8fd8    3         3         3       27m

In the output, you see the service and deployment that were specified in the sample manifests deployed in previous steps. You also see three Pods. This is because 3 replicas were specified in the sample manifest. For more information about Pods, see Pods in the Kubernetes documentation. Kubernetes automatically creates the replicaset resource, even though it isn’t specified in the sample manifests. For more information about ReplicaSets, see ReplicaSet in the Kubernetes documentation.

Kubernetes maintains the number of replicas that are specified in the manifest. If this were a production deployment and you wanted Kubernetes to horizontally scale the number of replicas or vertically scale the compute resources for the Pods, use the Scale pod deployments with Horizontal Pod Autoscaler and the Adjust pod resources with Vertical Pod Autoscaler to do so.

View the details of the deployed service.

kubectl -n eks-sample-app describe service eks-sample-linux-service

An example output is as follows.

Name:              eks-sample-linux-service
Namespace:         eks-sample-app
Labels:            app=eks-sample-linux-app
Annotations:       <none>
Selector:          app=eks-sample-linux-app
Type:              ClusterIP
IP Families:       <none>
IP:                10.100.74.8
IPs:               10.100.74.8
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         192.168.24.212:80,192.168.50.185:80,192.168.63.93:80
Session Affinity:  None
Events:            <none>

In the previous output, the value for IP: is a unique IP address that can be reached from any node or Pod within the cluster, but it can’t be reached from outside of the cluster. The values for Endpoints are IP addresses assigned from within your VPC to the Pods that are part of the service.

View the details of one of the Pods listed in the output when you viewed the namespace in a previous step. Replace 776d8f8fd8-78w66 with the value returned for one of your Pods.

kubectl -n eks-sample-app describe pod eks-sample-linux-deployment-65b7669776-m6qxz

Abbreviated example output

Name:         eks-sample-linux-deployment-65b7669776-m6qxz
Namespace:    eks-sample-app
Priority:     0
Node:         ip-192-168-45-132.us-west-2.compute.internal/192.168.45.132
[...]
IP:           192.168.63.93
IPs:
  IP:           192.168.63.93
Controlled By:  ReplicaSet/eks-sample-linux-deployment-65b7669776
[...]
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
[...]
Events:
  Type    Reason     Age    From                                                 Message
  ----    ------     ----   ----                                                 -------
  Normal  Scheduled  3m20s  default-scheduler                                    Successfully assigned eks-sample-app/eks-sample-linux-deployment-65b7669776-m6qxz to ip-192-168-45-132.us-west-2.compute.internal
[...]

In the previous output, the value for IP: is a unique IP that’s assigned to the Pod from the CIDR block assigned to the subnet that the node is in. If you prefer to assign Pods IP addresses from different CIDR blocks, you can change the default behavior. For more information, see cni-custom-network.title. You can also see that the Kubernetes scheduler scheduled the Pod on the Node with the IP address 192.168.45.132.

Rather than using the command line, you can view many details about Pods, services, deployments, and other Kubernetes resources in the consolelong. For more information, see view-kubernetes-resources.title.

12.1.6. Run a shell on a Pod

Run a shell on the Pod that you described in the previous step, replacing 65b7669776-m6qxz with the ID of one of your Pods.
```
kubectl exec -it eks-sample-linux-deployment-65b7669776-m6qxz -n eks-sample-app -- /bin/bash
```
From the Pod shell, view the output from the web server that was installed with your deployment in a previous step. You only need to specify the service name. It is resolved to the service’s IP address by CoreDNS, which is deployed with an Amazon EKS cluster, by default.
```
curl eks-sample-linux-service
```
An example output is as follows.
```
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
[...]
```
From the Pod shell, view the DNS server for the Pod.
```
cat /etc/resolv.conf
```
An example output is as follows.
```
nameserver 10.100.0.10
search eks-sample-app.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
options ndots:5
```
In the previous output, 10.100.0.10 is automatically assigned as the nameserver for all Pods deployed to the cluster.
Disconnect from the Pod by typing exit.
Once you’re finished with the sample application, you can remove the sample namespace, service, and deployment with the following command.
```
kubectl delete namespace eks-sample-app
```

12.1.7. Next Steps

After you deploy the sample application, you might want to try some of the following exercises:

Route application and HTTP traffic with Application Load Balancers
Route TCP and UDP traffic with Network Load Balancers

12.2. Deploy a sample application on Windows

In this topic, you deploy a sample application to your cluster on Windows nodes.

12.2.1. Prerequisites

An existing Kubernetes cluster with at least one node. If you don’t have an existing Amazon EKS cluster, you can deploy one using one of the guides in getting-started.title. You must have Windows support enabled for your cluster and at least one Amazon EC2 Windows node.
Kubectl installed on your computer. For more information, see install-kubectl.title.
Kubectl configured to communicate with your cluster. For more information, see create-kubeconfig.title.
If you plan to deploy your sample workload to Fargate, then you must have an existing Fargate profile that includes the same namespace created in this tutorial, which is eks-sample-app, unless you change the name. If you created a cluster with one of the gudes in getting-started.title, then you’ll have to create a new profile, or add the namespace to your existing profile, because the profile created in the getting started guides doesn’t specify the namespace used in this tutorial. Your VPC must also have at least one private subnet.

12.2.2. Create a namespace

kubectl create namespace eks-sample-app

12.2.3. Create a `Kubernetes` deployment

This sample deployment pulls a container image from a public repository and deploys three replicas (individual Pods) of it to your cluster. To learn more, see Deployments in the Kubernetes documentation.

Save the following contents to a file named eks-sample-deployment.yaml. The containers in the sample application don’t use network storage, but you might have applications that need to. For more information, see storage.title.

The kubernetes.io/os: windows nodeSelector means that if you had Windows and Linux nodes (for example) in your cluster, the image would only be deployed to Windows nodes. For more information, see Well-Known Labels, Annotations and Taints in the Kubernetes documentation.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eks-sample-windows-deployment
  namespace: eks-sample-app
  labels:
    app: eks-sample-windows-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: eks-sample-windows-app
  template:
    metadata:
      labels:
        app: eks-sample-windows-app
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
      containers:
      - name: windows-server-iis
        image: mcr.microsoft.com/windows/servercore:ltsc2019
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
        command:
        - powershell.exe
        - -command
        - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; "
      nodeSelector:
        kubernetes.io/os: windows

Apply the deployment manifest to your cluster.
```
kubectl apply -f eks-sample-deployment.yaml
```

12.2.4. Create a service

Save the following contents to a file named eks-sample-service.yaml. Kubernetes assigns the service its own IP address that is accessible only from within the cluster. To access the service from outside of your cluster, deploy the AWS Load Balancer Controller to load balance application or network traffic to the service.
```
apiVersion: v1
kind: Service
metadata:
  name: eks-sample-windows-service
  namespace: eks-sample-app
  labels:
    app: eks-sample-windows-app
spec:
  selector:
    app: eks-sample-windows-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
```
Apply the service manifest to your cluster.
```
kubectl apply -f eks-sample-service.yaml
```

12.2.5. Review resources created

View all resources that exist in the eks-sample-app namespace.

kubectl get all -n eks-sample-app

An example output is as follows.

NAME                                               READY   STATUS    RESTARTS   AGE
pod/eks-sample-windows-deployment-65b7669776-m6qxz   1/1     Running   0          27m
pod/eks-sample-windows-deployment-65b7669776-mmxvd   1/1     Running   0          27m
pod/eks-sample-windows-deployment-65b7669776-qzn22   1/1     Running   0          27m

NAME                               TYPE         CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/eks-sample-windows-service   ClusterIP    10.100.74.8     <none>        80/TCP    32m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/eks-sample-windows-deployment 3/3     3            3           27m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/eks-sample-windows-deployment-776d8f8fd8    3         3         3       27m

View the details of the deployed service.

kubectl -n eks-sample-app describe service eks-sample-windows-service

An example output is as follows.

Name:              eks-sample-windows-service
Namespace:         eks-sample-app
Labels:            app=eks-sample-windows-app
Annotations:       <none>
Selector:          app=eks-sample-windows-app
Type:              ClusterIP
IP Families:       <none>
IP:                10.100.74.8
IPs:               10.100.74.8
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         192.168.24.212:80,192.168.50.185:80,192.168.63.93:80
Session Affinity:  None
Events:            <none>

View the details of one of the Pods listed in the output when you viewed the namespace in a previous step. Replace 776d8f8fd8-78w66 with the value returned for one of your Pods.

kubectl -n eks-sample-app describe pod eks-sample-windows-deployment-65b7669776-m6qxz

Abbreviated example output

Name:         eks-sample-windows-deployment-65b7669776-m6qxz
Namespace:    eks-sample-app
Priority:     0
Node:         ip-192-168-45-132.us-west-2.compute.internal/192.168.45.132
[...]
IP:           192.168.63.93
IPs:
  IP:           192.168.63.93
Controlled By:  ReplicaSet/eks-sample-windows-deployment-65b7669776
[...]
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
[...]
Events:
  Type    Reason     Age    From                                                 Message
  ----    ------     ----   ----                                                 -------
  Normal  Scheduled  3m20s  default-scheduler                                    Successfully assigned eks-sample-app/eks-sample-windows-deployment-65b7669776-m6qxz to ip-192-168-45-132.us-west-2.compute.internal
[...]

12.2.6. Run a shell on a Pod

Run a shell on the Pod that you described in the previous step, replacing 65b7669776-m6qxz with the ID of one of your Pods.
```
kubectl exec -it eks-sample-windows-deployment-65b7669776-m6qxz -n eks-sample-app -- powershell.exe
```

From the Pod shell, view the output from the web server that was installed with your deployment in a previous step. You only need to specify the service name. It is resolved to the service’s IP address by CoreDNS, which is deployed with an Amazon EKS cluster, by default.

Invoke-WebRequest -uri eks-sample-windows-service/default.html -UseBasicParsing

An example output is as follows.

StatusCode        : 200
StatusDescription : OK
Content           : < h t m l > < b o d y > < b r / > < b r / > < m a r q u e e > < H 1 > H e l l o
                      E K S ! ! ! < H 1 > < m a r q u e e > < / b o d y > < h t m l >

From the Pod shell, view the DNS server for the Pod.
```
Get-NetIPConfiguration
```
Abbreviated output
```
InterfaceAlias       : vEthernet
[...]
IPv4Address          : 192.168.63.14
[...]
DNSServer            : 10.100.0.10
```
In the previous output, 10.100.0.10 is automatically assigned as the DNS server for all Pods deployed to the cluster.
Disconnect from the Pod by typing exit.
Once you’re finished with the sample application, you can remove the sample namespace, service, and deployment with the following command.
```
kubectl delete namespace eks-sample-app
```

12.2.7. Next Steps

After you deploy the sample application, you might want to try some of the following exercises:

Route application and HTTP traffic with Application Load Balancers
Route TCP and UDP traffic with Network Load Balancers

12.3. Adjust pod resources with `Vertical Pod Autoscaler`

Discover how the Kubernetes Vertical Pod Autoscaler automatically adjusts CPU and memory reservations for your Pods to optimize resource utilization and right-size applications on Amazon EKS.

The Kubernetes Vertical Pod Autoscaler automatically adjusts the CPU and memory reservations for your Pods to help "right size" your applications. This adjustment can improve cluster resource utilization and free up CPU and memory for other Pods. This topic helps you to deploy the Vertical Pod Autoscaler to your cluster and verify that it is working.

You have an existing Amazon EKS cluster. If you don’t, see getting-started.title.
You have the Kubernetes Metrics Server installed. For more information, see metrics-server.title.
You are using a kubectl client that is configured to communicate with your Amazon EKS cluster.
OpenSSL 1.1.1 or later installed on your device.

12.3.1. Deploy the Vertical Pod Autoscaler

In this section, you deploy the Vertical Pod Autoscaler to your cluster.

Open a terminal window and navigate to a directory where you would like to download the Vertical Pod Autoscaler source code.

Clone the kubernetes/autoscalerGitHub repository.

git clone https://github.com/kubernetes/autoscaler.git

Change to the vertical-pod-autoscaler directory.
```
cd autoscaler/vertical-pod-autoscaler/
```
(Optional) If you have already deployed another version of the Vertical Pod Autoscaler, remove it with the following command.
```
./hack/vpa-down.sh
```
If your nodes don’t have internet access to the registry.k8s.io container registry, then you need to pull the following images and push them to your own private repository. For more information about how to pull the images and push them to your own private repository, see copy-image-to-repository.title.
```
registry.k8s.io/autoscaling/vpa-admission-controller:0.10.0
registry.k8s.io/autoscaling/vpa-recommender:0.10.0
registry.k8s.io/autoscaling/vpa-updater:0.10.0
```
If you’re pushing the images to a private Amazon ECR repository, then replace registry.k8s.io in the manifests with your registry. Replace 111122223333 with your account ID. Replace region-code with the AWS Region that your cluster is in. The following commands assume that you named your repository the same as the repository name in the manifest. If you named your repository something different, then you’ll need to change it too.
```
sed -i.bak -e 's/registry.k8s.io/111122223333.dkr.ecr.region-code.amazonaws.com/' ./deploy/admission-controller-deployment.yaml
sed -i.bak -e 's/registry.k8s.io/111122223333.dkr.ecr.region-code.amazonaws.com/' ./deploy/recommender-deployment.yaml
sed -i.bak -e 's/registry.k8s.io/111122223333.dkr.ecr.region-code.amazonaws.com/' ./deploy/updater-deployment.yaml
```
Deploy the Vertical Pod Autoscaler to your cluster with the following command.
```
./hack/vpa-up.sh
```

Verify that the Vertical Pod Autoscaler Pods have been created successfully.

kubectl get pods -n kube-system

An example output is as follows.

NAME                                        READY   STATUS    RESTARTS   AGE
[...]
metrics-server-8459fc497-kfj8w              1/1     Running   0          83m
vpa-admission-controller-68c748777d-ppspd   1/1     Running   0          7s
vpa-recommender-6fc8c67d85-gljpl            1/1     Running   0          8s
vpa-updater-786b96955c-bgp9d                1/1     Running   0          8s

12.3.2. Test your Vertical Pod Autoscaler installation

In this section, you deploy a sample application to verify that the Vertical Pod Autoscaler is working.

Deploy the hamster.yaml Vertical Pod Autoscaler example with the following command.
```
kubectl apply -f examples/hamster.yaml
```

Get the Pods from the hamster example application.

kubectl get pods -l app=hamster

An example output is as follows.

hamster-c7d89d6db-rglf5   1/1     Running   0          48s
hamster-c7d89d6db-znvz5   1/1     Running   0          48s

Describe one of the Pods to view its cpu and memory reservation. Replace c7d89d6db-rglf5 with one of the IDs returned in your output from the previous step.

kubectl describe pod hamster-c7d89d6db-rglf5

An example output is as follows.

[...]
Containers:
  hamster:
    Container ID:  docker://e76c2413fc720ac395c33b64588c82094fc8e5d590e373d5f818f3978f577e24
    Image:         registry.k8s.io/ubuntu-slim:0.1
    Image ID:      docker-pullable://registry.k8s.io/ubuntu-slim@sha256:b6f8c3885f5880a4f1a7cf717c07242eb4858fdd5a84b5ffe35b1cf680ea17b1
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done
    State:          Running
      Started:      Fri, 27 Sep 2019 10:35:16 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     50Mi
[...]

You can see that the original Pod reserves 100 millicpu of CPU and 50 mebibytes of memory. For this example application, 100 millicpu is less than the Pod needs to run, so it is CPU-constrained. It also reserves much less memory than it needs. The Vertical Pod Autoscaler vpa-recommender deployment analyzes the hamster Pods to see if the CPU and memory requirements are appropriate. If adjustments are needed, the vpa-updater relaunches the Pods with updated values.

Wait for the vpa-updater to launch a new hamster Pods. This should take a minute or two. You can monitor the Pods with the following command.

If you are not sure that a new Pod has launched, compare the Pod names with your previous list. When the new Pod launches, you will see a new Pod name.
```
kubectl get --watch Pods -l app=hamster
```

When a new hamster Pods is started, describe it and view the updated CPU and memory reservations.

kubectl describe pod hamster-c7d89d6db-jxgfv

An example output is as follows.

[...]
Containers:
  hamster:
    Container ID:  docker://2c3e7b6fb7ce0d8c86444334df654af6fb3fc88aad4c5d710eac3b1e7c58f7db
    Image:         registry.k8s.io/ubuntu-slim:0.1
    Image ID:      docker-pullable://registry.k8s.io/ubuntu-slim@sha256:b6f8c3885f5880a4f1a7cf717c07242eb4858fdd5a84b5ffe35b1cf680ea17b1
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done
    State:          Running
      Started:      Fri, 27 Sep 2019 10:37:08 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        587m
      memory:     262144k
[...]

In the previous output, you can see that the cpu reservation increased to 587 millicpu, which is over five times the original value. The memory increased to 262,144 Kilobytes, which is around 250 mebibytes, or five times the original value. This Pod was under-resourced, and the Vertical Pod Autoscaler corrected the estimate with a much more appropriate value.

Describe the hamster-vpa resource to view the new recommendation.

kubectl describe vpa/hamster-vpa

An example output is as follows.

Name:         hamster-vpa
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"hamster-vpa","namespace":"d...
API Version:  autoscaling.k8s.io/v1beta2
Kind:         VerticalPodAutoscaler
Metadata:
  Creation Timestamp:  2019-09-27T18:22:51Z
  Generation:          23
  Resource Version:    14411
  Self Link:           /apis/autoscaling.k8s.io/v1beta2/namespaces/default/verticalpodautoscalers/hamster-vpa
  UID:                 d0d85fb9-e153-11e9-ae53-0205785d75b0
Spec:
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         hamster
Status:
  Conditions:
    Last Transition Time:  2019-09-27T18:23:28Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  hamster
      Lower Bound:
        Cpu:     550m
        Memory:  262144k
      Target:
        Cpu:     587m
        Memory:  262144k
      Uncapped Target:
        Cpu:     587m
        Memory:  262144k
      Upper Bound:
        Cpu:     21147m
        Memory:  387863636
Events:          <none>

When you finish experimenting with the example application, you can delete it with the following command.
```
kubectl delete -f examples/hamster.yaml
```

12.4. Scale pod deployments with `Horizontal Pod Autoscaler`

Learn how to use the Kubernetes Horizontal Pod Autoscaler to automatically scale your Amazon EKS deployments based on CPU utilization for efficient resource management.

The Kubernetes Horizontal Pod Autoscaler automatically scales the number of Pods in a deployment, replication controller, or replica set based on that resource’s CPU utilization. This can help your applications scale out to meet increased demand or scale in when resources are not needed, thus freeing up your nodes for other applications. When you set a target CPU utilization percentage, the Horizontal Pod Autoscaler scales your application in or out to try to meet that target.

The Horizontal Pod Autoscaler is a standard API resource in Kubernetes that simply requires that a metrics source (such as the Kubernetes metrics server) is installed on your Amazon EKS cluster to work. You do not need to deploy or install the Horizontal Pod Autoscaler on your cluster to begin scaling your applications. For more information, see Horizontal Pod Autoscaler in the Kubernetes documentation.

Use this topic to prepare the Horizontal Pod Autoscaler for your Amazon EKS cluster and to verify that it is working with a sample application.

This topic is based on the Horizontal Pod autoscaler walkthrough in the Kubernetes documentation.

You have an existing Amazon EKS cluster. If you don’t, see getting-started.title.
You have the Kubernetes Metrics Server installed. For more information, see metrics-server.title.
You are using a kubectl client that is configured to communicate with your Amazon EKS cluster.

12.4.1. Run a Horizontal Pod Autoscaler test application

In this section, you deploy a sample application to verify that the Horizontal Pod Autoscaler is working.

This example is based on the Horizontal Pod autoscaler walkthrough in the Kubernetes documentation.

Deploy a simple Apache web server application with the following command.
```
kubectl apply -f https://k8s.io/examples/application/php-apache.yaml
```
This Apache web server Pod is given a 500 millicpu CPU limit and it is serving on port 80.
Create a Horizontal Pod Autoscaler resource for the php-apache deployment.
```
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
```
This command creates an autoscaler that targets 50 percent CPU utilization for the deployment, with a minimum of one Pod and a maximum of ten Pods. When the average CPU load is lower than 50 percent, the autoscaler tries to reduce the number of Pods in the deployment, to a minimum of one. When the load is greater than 50 percent, the autoscaler tries to increase the number of Pods in the deployment, up to a maximum of ten. For more information, see How does a HorizontalPodAutoscaler work? in the Kubernetes documentation.
Describe the autoscaler with the following command to view its details.
```
kubectl get hpa
```
An example output is as follows.
```
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          51s
```
As you can see, the current CPU load is 0%, because there’s no load on the server yet. The Pod count is already at its lowest boundary (one), so it cannot scale in.

Create a load for the web server by running a container.

kubectl run -i \
    --tty load-generator \
    --rm --image=busybox \
    --restart=Never \
    -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

To watch the deployment scale out, periodically run the following command in a separate terminal from the terminal that you ran the previous step in.

kubectl get hpa php-apache

An example output is as follows.

NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   250%/50%   1         10        5          4m44s

It may take over a minute for the replica count to increase. As long as actual CPU percentage is higher than the target percentage, then the replica count increases, up to 10. In this case, it’s 250%, so the number of REPLICAS continues to increase.

It may take a few minutes before you see the replica count reach its maximum. If only 6 replicas, for example, are necessary for the CPU load to remain at or under 50%, then the load won’t scale beyond 6 replicas.

Stop the load. In the terminal window you’re generating the load in, stop the load by holding down the Ctrl+C keys. You can watch the replicas scale back to 1 by running the following command again in the terminal that you’re watching the scaling in.

kubectl get hpa

An example output is as follows.

NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          25m

The default timeframe for scaling back down is five minutes, so it will take some time before you see the replica count reach 1 again, even when the current CPU percentage is 0 percent. The timeframe is modifiable. For more information, see Horizontal Pod Autoscaler in the Kubernetes documentation.

When you are done experimenting with your sample application, delete the php-apache resources.

kubectl delete deployment.apps/php-apache service/php-apache horizontalpodautoscaler.autoscaling/php-apache

12.5. Route `TCP` and `UDP` traffic with `Network Load Balancers`

Use the AWS Load Balancer Controller to create network load balancers for Amazon EKS workloads, supporting IP and instance targets with AWS Network Load Balancers.

New: Amazon EKS Auto Mode automates routine tasks for load balancing. For more information, see:

auto-elb-example.title
auto-configure-nlb.title

Network traffic is load balanced at L4 of the OSI model. To load balance application traffic at L7, you deploy a Kubernetes ingress, which provisions an AWS Application Load Balancer. For more information, see alb-ingress.title. To learn more about the differences between the two types of load balancing, see Elastic Load Balancing features on the AWS website.

When you create a Kubernetes Service of type LoadBalancer, the AWS cloud provider load balancer controller creates AWS Classic Load Balancers by default, but can also create AWS Network Load Balancers. This controller is only receiving critical bug fixes in the future. For more information about using the AWS cloud provider load balancer , see AWS cloud provider load balancer controller in the Kubernetes documentation. Its use is not covered in this topic.

We recommend that you use version 2.7.2 or later of the AWS Load Balancer Controller instead of the AWS cloud provider load balancer controller. The AWS Load Balancer Controller creates AWS Network Load Balancers, but doesn’t create AWS Classic Load Balancers. The remainder of this topic is about using the AWS Load Balancer Controller.

An AWS Network Load Balancer can load balance network traffic to Pods deployed to Amazon EC2 IP and instance targets, to AWS Fargate IP targets, or to Amazon EKS Hybrid Nodes as IP targets. For more information, see AWS Load Balancer Controller on GitHub.

12.5.1. Prerequisites

Before you can load balance network traffic using the AWS Load Balancer Controller, you must meet the following requirements.

Have an existing cluster. If you don’t have an existing cluster, see getting-started.title. If you need to update the version of an existing cluster, see update-cluster.title.
Have the AWS Load Balancer Controller deployed on your cluster. For more information, see aws-load-balancer-controller.title. We recommend version 2.7.2 or later.
At least one subnet. If multiple tagged subnets are found in an Availability Zone, the controller chooses the first subnet whose subnet ID comes first lexicographically. The subnet must have at least eight available IP addresses.
If you’re using the AWS Load Balancer Controller version 2.1.1 or earlier, subnets must be tagged as follows. If using version 2.1.2 or later, this tag is optional. You might want to tag a subnet if you have multiple clusters running in the same VPC, or multiple AWS services sharing subnets in a VPC, and want more control over where load balancers are provisioned for each cluster. If you explicitly specify subnet IDs as an annotation on a service object, then Kubernetes and the AWS Load Balancer Controller use those subnets directly to create the load balancer. Subnet tagging isn’t required if you choose to use this method for provisioning load balancers and you can skip the following private and public subnet tagging requirements. Replace my-cluster with your cluster name.
- Key – kubernetes.io/cluster/<my-cluster>
- Value – shared or owned
Your public and private subnets must meet the following requirements, unless you explicitly specify subnet IDs as an annotation on a service or ingress object. If you provision load balancers by explicitly specifying subnet IDs as an annotation on a service or ingress object, then Kubernetes and the AWS Load Balancer Controller use those subnets directly to create the load balancer and the following tags aren’t required.
- Private subnets – Must be tagged in the following format. This is so that Kubernetes and the AWS Load Balancer Controller know that the subnets can be used for internal load balancers. If you use eksctl or an Amazon EKS AWS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they’re created. For more information about the Amazon EKS AWS AWS CloudFormation VPC templates, see creating-a-vpc.title.
  - Key – kubernetes.io/role/internal-elb
  - Value – 1
- Public subnets – Must be tagged in the following format. This is so that Kubernetes knows to use only those subnets for external load balancers instead of choosing a public subnet in each Availability Zone (based on the lexicographical order of the subnet IDs). If you use eksctl or an Amazon EKS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they’re created. For more information about the Amazon EKS AWS CloudFormation VPC templates, see creating-a-vpc.title.
  - Key – kubernetes.io/role/elb
  - Value – 1
If the subnet role tags aren’t explicitly added, the Kubernetes service controller examines the route table of your cluster VPC subnets to determine if the subnet is private or public. We recommend that you don’t rely on this behavior, and instead explicitly add the private or public role tags. The AWS Load Balancer Controller doesn’t examine route tables, and requires the private and public tags to be present for successful auto discovery.

12.5.2. Considerations

The configuration of your load balancer is controlled by annotations that are added to the manifest for your service. Service annotations are different when using the AWS Load Balancer Controller than they are when using the AWS cloud provider load balancer controller. Make sure to review the annotations for the AWS Load Balancer Controller before deploying services.
When using the Amazon VPC CNI plugin for Kubernetes, the AWS Load Balancer Controller can load balance to Amazon EC2 IP or instance targets and Fargate IP targets. When using Alternate compatible CNI plugins, the controller can only load balance to instance targets, unless you are load balancing to Amazon EKS Hybrid Nodes. For hybrid nodes, the controller can load balance IP targets. For more information about Network Load Balancer target types, see Target type in the User Guide for Network Load Balancers
If you want to add tags to the load balancer when or after it’s created, add the following annotation in your service specification. For more information, see AWS Resource Tags in the AWS Load Balancer Controller documentation.
```
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags
```
You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation. Replace the example values with the Allocation IDs of your Elastic IP addresses. The number of Allocation IDs must match the number of subnets that are used for the load balancer. For more information, see the AWS Load Balancer Controller documentation.
```
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xxxxxxxxxxxxxxxxx,eipalloc-yyyyyyyyyyyyyyyyy
```
Amazon EKS adds one inbound rule to the node’s security group for client traffic and one rule for each load balancer subnet in the VPC for health checks for each Network Load Balancer that you create. Deployment of a service of type LoadBalancer can fail if Amazon EKS attempts to create rules that exceed the quota for the maximum number of rules allowed for a security group. For more information, see Security groups in Amazon VPC quotas in the Amazon VPC User Guide. Consider the following options to minimize the chances of exceeding the maximum number of rules for a security group:
- Request an increase in your rules per security group quota. For more information, see Requesting a quota increase in the Service Quotas User Guide.
- Use IP targets, rather than instance targets. With IP targets, you can share rules for the same target ports. You can manually specify load balancer subnets with an annotation. For more information, see Annotations on GitHub.
- Use an ingress, instead of a service of type LoadBalancer, to send traffic to your service. The AWS Application Load Balancer requires fewer rules than Network Load Balancers. You can share an ALB across multiple ingresses. For more information, see alb-ingress.title. You can’t share a Network Load Balancer across multiple services.
- Deploy your clusters to multiple accounts.
If your Pods run on Windows in an Amazon EKS cluster, a single service with a load balancer can support up to 1024 back-end Pods. Each Pod has its own unique IP address.
We recommend only creating new Network Load Balancers with the AWS Load Balancer Controller. Attempting to replace existing Network Load Balancers created with the AWS cloud provider load balancer controller can result in multiple Network Load Balancers that might cause application downtime.

12.5.3. Create a network load balancer

You can create a network load balancer with IP or instance targets.

Create network load balancer — IP Targets

You can use IP targets with Pods deployed to Amazon EC2 nodes, Fargate, or Amazon EKS Hybrid Nodes. Your Kubernetes service must be created as type LoadBalancer. For more information, see Type LoadBalancer in the Kubernetes documentation.

To create a load balancer that uses IP targets, add the following annotations to a service manifest and deploy your service. The external value for aws-load-balancer-type is what causes the AWS Load Balancer Controller, rather than the AWS cloud provider load balancer controller, to create the Network Load Balancer. You can view a sample service manifest with the annotations.

service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

If you’re load balancing to IPv6 Pods, add the following annotation. You can only load balance over IPv6 to IP targets, not instance targets. Without this annotation, load balancing is over IPv4.

service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack

Network Load Balancers are created with the internal aws-load-balancer-scheme, by default. You can launch Network Load Balancers in any subnet in your cluster’s VPC, including subnets that weren’t specified when you created your cluster.

Kubernetes examines the route table for your subnets to identify whether they are public or private. Public subnets have a route directly to the internet using an internet gateway, but private subnets do not.

If you want to create a Network Load Balancer in a public subnet to load balance to Amazon EC2 nodes (Fargate can only be private), specify internet-facing with the following annotation:

service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

The service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip" annotation is still supported for backwards compatibility. However, we recommend using the previous annotations for new load balancers instead of service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip".

Do not edit the annotations after creating your service. If you need to modify it, delete the service object and create it again with the desired value for this annotation.

Create network load balancer — Instance Targets

The AWS cloud provider load balancer controller creates Network Load Balancers with instance targets only. Version 2.2.0 and later of the AWS Load Balancer Controller also creates Network Load Balancers with instance targets. We recommend using it, rather than the AWS cloud provider load balancer controller, to create new Network Load Balancers. You can use Network Load Balancer instance targets with Pods deployed to Amazon EC2 nodes, but not to Fargate. To load balance network traffic across Pods deployed to Fargate, you must use IP targets.

To deploy a Network Load Balancer to a private subnet, your service specification must have the following annotations. You can view a sample service manifest with the annotations. The external value for aws-load-balancer-type is what causes the AWS Load Balancer Controller, rather than the AWS cloud provider load balancer controller, to create the Network Load Balancer.
```
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
```
Network Load Balancers are created with the internal aws-load-balancer-scheme, by default. For internal Network Load Balancers, your Amazon EKS cluster must be configured to use at least one private subnet in your VPC. Kubernetes examines the route table for your subnets to identify whether they are public or private. Public subnets have a route directly to the internet using an internet gateway, but private subnets do not.

If you want to create an Network Load Balancer in a public subnet to load balance to Amazon EC2 nodes, specify internet-facing with the following annotation:
```
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
```
Do not edit the annotations after creating your service. If you need to modify it, delete the service object and create it again with the desired value for this annotation.

12.5.4. (Optional) Deploy a sample application

At least one public or private subnet in your cluster VPC.

Have the AWS Load Balancer Controller deployed on your cluster. For more information, see aws-load-balancer-controller.title. We recommend version 2.7.2 or later.

If you’re deploying to Fargate, make sure you have an available private subnet in your VPC and create a Fargate profile. If you’re not deploying to Fargate, skip this step. You can create the profile by running the following command or in the consolelong using the same values for name and namespace that are in the command. Replace the example values with your own.
```
eksctl create fargateprofile \
    --cluster my-cluster \
    --region region-code \
    --name nlb-sample-app \
    --namespace nlb-sample-app
```

Deploy a sample application.

Create a namespace for the application.
```
kubectl create namespace nlb-sample-app
```

Save the following contents to a file named sample-deployment.yaml` file on your computer.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nlb-sample-app
  namespace: nlb-sample-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: public.ecr.aws/nginx/nginx:1.23
          ports:
            - name: tcp
              containerPort: 80

Apply the manifest to the cluster.
```
kubectl apply -f sample-deployment.yaml
```

Create a service with an internet-facing Network Load Balancer that load balances to IP targets.

Save the following contents to a file named sample-service.yaml` file on your computer. If you’re deploying to Fargate nodes, remove the service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing line.

apiVersion: v1
kind: Service
metadata:
  name: nlb-sample-service
  namespace: nlb-sample-app
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: LoadBalancer
  selector:
    app: nginx

Apply the manifest to the cluster.
```
kubectl apply -f sample-service.yaml
```

Verify that the service was deployed.

kubectl get svc nlb-sample-service -n nlb-sample-app

An example output is as follows.

NAME            TYPE           CLUSTER-IP         EXTERNAL-IP                                                                    PORT(S)        AGE
sample-service  LoadBalancer   10.100.240.137   k8s-nlbsampl-nlbsampl-xxxxxxxxxx-xxxxxxxxxxxxxxxx.elb.region-code.amazonaws.com  80:32400/TCP   16h

The values for 10.100.240.137 and xxxxxxxxxx-xxxxxxxxxxxxxxxx will be different than the example output (they will be unique to your load balancer) and us-west-2 may be different for you, depending on which AWS Region that your cluster is in.

Open the Amazon EC2 consolelong. Select Target Groups (under Load Balancing) in the left navigation pane. In the Name column, select the target group’s name where the value in the Load balancer column matches a portion of the name in the EXTERNAL-IP column of the output in the previous step. For example, you’d select the target group named k8s-default-samplese-xxxxxxxxxx if your output were the same as the previous output. The Target type is IP because that was specified in the sample service manifest.
Select the Target group and then select the Targets tab. Under Registered targets, you should see three IP addresses of the three replicas deployed in a previous step. Wait until the status of all targets is healthy before continuing. It might take several minutes before all targets are healthy. The targets might be in an unhealthy state before changing to a healthy state.
Send traffic to the service replacing xxxxxxxxxx-xxxxxxxxxxxxxxxx and us-west-2 with the values returned in the output for a previous step for EXTERNAL-IP. If you deployed to a private subnet, then you’ll need to view the page from a device within your VPC, such as a bastion host. For more information, see Linux Bastion Hosts on AWS.
```
curl k8s-default-samplese-xxxxxxxxxx-xxxxxxxxxxxxxxxx.elb.region-code.amazonaws.com
```
An example output is as follows.
```
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
[...]
```
When you’re finished with the sample deployment, service, and namespace, remove them.
```
kubectl delete namespace nlb-sample-app
```

12.6. Route application and `HTTP` traffic with `Application Load Balancers`

Learn how to use Application Load Balancing on Amazon EKS to load balance application traffic at L7 with AWS Load Balancer Controller.

New: Amazon EKS Auto Mode automates routine tasks for load balancing. For more information, see:

auto-elb-example.title
auto-configure-alb.title

When you create a Kubernetes ingress, an AWS Application Load Balancer (ALB) is provisioned that load balances application traffic. To learn more, see What is an Application Load Balancer? in the Application Load Balancers User Guide and Ingress in the Kubernetes documentation. ALBs can be used with Pods that are deployed to nodes or to AWS Fargate. You can deploy an ALB to public or private subnets.

Application traffic is balanced at L7 of the OSI model. To load balance network traffic at L4, you deploy a Kubernetes service of the LoadBalancer type. This type provisions an AWS Network Load Balancer. For more information, see network-load-balancing.title. To learn more about the differences between the two types of load balancing, see Elastic Load Balancing features on the AWS website.

12.6.1. Prerequisites

Before you can load balance application traffic to an application, you must meet the following requirements.

Have an existing cluster. If you don’t have an existing cluster, see getting-started.title. If you need to update the version of an existing cluster, see update-cluster.title.
Have the AWS Load Balancer Controller deployed on your cluster. For more information, see aws-load-balancer-controller.title. We recommend version 2.7.2 or later.
At least two subnets in different Availability Zones. The AWS Load Balancer Controller chooses one subnet from each Availability Zone. When multiple tagged subnets are found in an Availability Zone, the controller chooses the subnet whose subnet ID comes first lexicographically. Each subnet must have at least eight available IP addresses.

If you’re using multiple security groups attached to worker node, exactly one security group must be tagged as follows. Replace my-cluster with your cluster name.
- Key – kubernetes.io/cluster/<my-cluster>
- Value – shared or owned
If you’re using the AWS Load Balancer Controller version 2.1.1 or earlier, subnets must be tagged in the format that follows. If you’re using version 2.1.2 or later, tagging is optional. However, we recommend that you tag a subnet if any of the following is the case. You have multiple clusters that are running in the same VPC, or have multiple AWS services that share subnets in a VPC. Or, you want more control over where load balancers are provisioned for each cluster. Replace my-cluster with your cluster name.
- Key – kubernetes.io/cluster/<my-cluster>
- Value – shared or owned
Your public and private subnets must meet the following requirements. This is unless you explicitly specify subnet IDs as an annotation on a service or ingress object. Assume that you provision load balancers by explicitly specifying subnet IDs as an annotation on a service or ingress object. In this situation, Kubernetes and the AWS load balancer controller use those subnets directly to create the load balancer and the following tags aren’t required.
- Private subnets – Must be tagged in the following format. This is so that Kubernetes and the AWS load balancer controller know that the subnets can be used for internal load balancers. If you use eksctl or an Amazon EKS AWS CloudFormation template to create your VPC after March 26, 2020, the subnets are tagged appropriately when created. For more information about the Amazon EKS AWS CloudFormation VPC templates, see creating-a-vpc.title.
  - Key – kubernetes.io/role/internal-elb
  - Value – 1
- Public subnets – Must be tagged in the following format. This is so that Kubernetes knows to use only the subnets that were specified for external load balancers. This way, Kubernetes doesn’t choose a public subnet in each Availability Zone (lexicographically based on their subnet ID). If you use eksctl or an Amazon EKS AWS CloudFormation template to create your VPC after March 26, 2020, the subnets are tagged appropriately when created. For more information about the Amazon EKS AWS CloudFormation VPC templates, see creating-a-vpc.title.
  - Key – kubernetes.io/role/elb
  - Value – 1
If the subnet role tags aren’t explicitly added, the Kubernetes service controller examines the route table of your cluster VPC subnets. This is to determine if the subnet is private or public. We recommend that you don’t rely on this behavior. Rather, explicitly add the private or public role tags. The AWS Load Balancer Controller doesn’t examine route tables. It also requires the private and public tags to be present for successful auto discovery.

The AWS Load Balancer Controller creates ALBs and the necessary supporting AWS resources whenever a Kubernetes ingress resource is created on the cluster with the kubernetes.io/ingress.class: alb annotation. The ingress resource configures the ALB to route HTTP or HTTPS traffic to different Pods within the cluster. To ensure that your ingress objects use the AWS Load Balancer Controller, add the following annotation to your Kubernetes ingress specification. For more information, see Ingress specification on GitHub.

annotations:
    kubernetes.io/ingress.class: alb

If you’re load balancing to IPv6 Pods, add the following annotation to your ingress spec. You can only load balance over IPv6 to IP targets, not instance targets. Without this annotation, load balancing is over IPv4.

alb.ingress.kubernetes.io/ip-address-type: dualstack

The AWS Load Balancer Controller supports the following traffic modes:
- Instance – Registers nodes within your cluster as targets for the ALB. Traffic reaching the ALB is routed to NodePort for your service and then proxied to your Pods. This is the default traffic mode. You can also explicitly specify it with the alb.ingress.kubernetes.io/target-type: instance annotation.
  
  Your Kubernetes service must specify the NodePort or "LoadBalancer" type to use this traffic mode.
- IP – Registers Pods as targets for the ALB. Traffic reaching the ALB is directly routed to Pods for your service. You must specify the alb.ingress.kubernetes.io/target-type: ip annotation to use this traffic mode. The IP target type is required when target Pods are running on Fargate or Amazon EKS Hybrid Nodes.
To tag ALBs created by the controller, add the following annotation to the controller: alb.ingress.kubernetes.io/tags. For a list of all available annotations supported by the AWS Load Balancer Controller, see Ingress annotations on GitHub.
Upgrading or downgrading the ALB controller version can introduce breaking changes for features that rely on it. For more information about the breaking changes that are introduced in each release, see the ALB controller release notes on GitHub.

12.6.2. Reuse ALBs with Ingress Groups

You can share an application load balancer across multiple service resources using IngressGroups.

To join an ingress to a group, add the following annotation to a Kubernetes ingress resource specification.

alb.ingress.kubernetes.io/group.name: my-group

The group name must:

Be 63 or fewer characters in length.
Consist of lower case letters, numbers, -, and .
Start and end with a letter or number.

The controller automatically merges ingress rules for all ingresses in the same ingress group. It supports them with a single ALB. Most annotations that are defined on an ingress only apply to the paths defined by that ingress. By default, ingress resources don’t belong to any ingress group.

Potential security risk

Specify an ingress group for an ingress only when all the Kubernetes users that have RBAC permission to create or modify ingress resources are within the same trust boundary. If you add the annotation with a group name, other Kubernetes users might create or modify their ingresses to belong to the same ingress group. Doing so can cause undesirable behavior, such as overwriting existing rules with higher priority rules.

You can add an order number of your ingress resource.

alb.ingress.kubernetes.io/group.order: '10'

The number can be 1-1000. The lowest number for all ingresses in the same ingress group is evaluated first. All ingresses without this annotation are evaluated with a value of zero. Duplicate rules with a higher number can overwrite rules with a lower number. By default, the rule order between ingresses within the same ingress group is determined lexicographically based namespace and name.

Ensure that each ingress in the same ingress group has a unique priority number. You can’t have duplicate order numbers across ingresses.

12.6.3. (Optional) Deploy a sample application

At least one public or private subnet in your cluster VPC.
Have the AWS Load Balancer Controller deployed on your cluster. For more information, see aws-load-balancer-controller.title. We recommend version 2.7.2 or later.

You can run the sample application on a cluster that has Amazon EC2 nodes, Fargate Pods, or both.

If you’re not deploying to Fargate, skip this step. If you’re deploying to Fargate, create a Fargate profile. You can create the profile by running the following command or in the consolelong using the same values for name and namespace that are in the command. Replace the example values with your own.
```
eksctl create fargateprofile \
    --cluster my-cluster \
    --region region-code \
    --name alb-sample-app \
    --namespace game-2048
```
Deploy the game 2048 as a sample application to verify that the AWS Load Balancer Controller creates an AWS ALB as a result of the ingress object. Complete the steps for the type of subnet you’re deploying to.
1. If you’re deploying to Pods in a cluster that you created with the IPv6 family, skip to the next step.
  - Public::
  kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/examples/2048/2048_full.yaml
  - Private::
    
    Download the manifest.
    
    curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/examples/2048/2048_full.yaml
    
    Edit the file and find the line that says alb.ingress.kubernetes.io/scheme: internet-facing.
    
    Change internet-facing to internal and save the file.
    
    Apply the manifest to your cluster.
    
    kubectl apply -f 2048_full.yaml
2. If you’re deploying to Pods in a cluster that you created with the IPv6 family, complete the following steps.
  1. Download the manifest.
    
    curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/examples/2048/2048_full.yaml
  2. Open the file in an editor and add the following line to the annotations in the ingress spec.
    
    alb.ingress.kubernetes.io/ip-address-type: dualstack
  3. If you’re load balancing to internal Pods, rather than internet facing Pods, change the line that says alb.ingress.kubernetes.io/scheme: internet-facing to alb.ingress.kubernetes.io/scheme: internal
  4. Save the file.
  5. Apply the manifest to your cluster.
    
    kubectl apply -f 2048_full.yaml

After a few minutes, verify that the ingress resource was created with the following command.

kubectl get ingress/ingress-2048 -n game-2048

An example output is as follows.

NAME           CLASS    HOSTS   ADDRESS                                                                   PORTS   AGE
ingress-2048   <none>   *       k8s-game2048-ingress2-xxxxxxxxxx-yyyyyyyyyy.region-code.elb.amazonaws.com   80      2m32s

If you created the load balancer in a private subnet, the value under ADDRESS in the previous output is prefaced with internal-.

If your ingress wasn’t successfully created after several minutes, run the following command to view the AWS Load Balancer Controller logs. These logs might contain error messages that you can use to diagnose issues with your deployment.

kubectl logs -f -n kube-system -l app.kubernetes.io/instance=aws-load-balancer-controller

If you deployed to a public subnet, open a browser and navigate to the ADDRESS URL from the previous command output to see the sample application. If you don’t see anything, refresh your browser and try again. If you deployed to a private subnet, then you’ll need to view the page from a device within your VPC, such as a bastion host. For more information, see Linux Bastion Hosts on AWS.
When you finish experimenting with your sample application, delete it by running one of the the following commands.
- If you applied the manifest, rather than applying a copy that you downloaded, use the following command.
  kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/examples/2048/2048_full.yaml
- If you downloaded and edited the manifest, use the following command.
  kubectl delete -f 2048_full.yaml

12.7. Restrict external IP addresses that can be assigned to services

Kubernetes services can be reached from inside of a cluster through:

A cluster IP address that is assigned automatically by Kubernetes
Any IP address that you specify for the externalIPs property in a service spec. External IP addresses are not managed by Kubernetes and are the responsibility of the cluster administrator. External IP addresses specified with externalIPs are different than the external IP address assigned to a service of type LoadBalancer by a cloud provider.

To learn more about Kubernetes services, see Service in the Kubernetes documentation. You can restrict the IP addresses that can be specified for externalIPs in a service spec.

Deploy cert-manager to manage webhook certificates. For more information, see the cert-manager documentation.
```
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml
```

Verify that the cert-manager Pods are running.

kubectl get pods -n cert-manager

An example output is as follows.

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-58c8844bb8-nlx7q              1/1     Running   0          15s
cert-manager-cainjector-745768f6ff-696h5   1/1     Running   0          15s
cert-manager-webhook-67cc76975b-4v4nk      1/1     Running   0          14s

Review your existing services to ensure that none of them have external IP addresses assigned to them that aren’t contained within the CIDR block you want to limit addresses to.

kubectl get services -A

An example output is as follows.

NAMESPACE                      NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)         AGE
cert-manager                   cert-manager                            ClusterIP      10.100.102.137   <none>          9402/TCP        20m
cert-manager                   cert-manager-webhook                    ClusterIP      10.100.6.136     <none>          443/TCP         20m
default                        kubernetes                              ClusterIP      10.100.0.1       <none>          443/TCP         2d1h
externalip-validation-system   externalip-validation-webhook-service   ClusterIP      10.100.234.179   <none>          443/TCP         16s
kube-system                    kube-dns                                ClusterIP      10.100.0.10      <none>          53/UDP,53/TCP   2d1h
my-namespace                   my-service                              ClusterIP      10.100.128.10    192.168.1.1     80/TCP          149m

If any of the values are IP addresses that are not within the block you want to restrict access to, you’ll need to change the addresses to be within the block, and redeploy the services. For example, the my-service service in the previous output has an external IP address assigned to it that isn’t within the CIDR block example in step 5.

Download the external IP webhook manifest. You can also view the source code for the webhook on GitHub.
```
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/docs/externalip-webhook.yaml
```
Specify CIDR blocks. Open the downloaded file in your editor and remove the \# at the start of the following lines.
```
#args:
#- --allowed-external-ip-cidrs=10.0.0.0/8
```
Replace 10.0.0.0/8 with your own CIDR block. You can specify as many blocks as you like. If specifying mutiple blocks, add a comma between blocks.
If your cluster is not in the us-west-2 AWS Region, then replace us-west-2, 602401143452, and amazonaws.com in the file with the following commands. Before running the commands, replace region-code and 111122223333 with the value for your AWS Region from the list in View Amazon container image registries for Amazon EKS add-ons.
```
sed -i.bak -e 's|602401143452|111122223333|' externalip-webhook.yaml
sed -i.bak -e 's|us-west-2|region-code|' externalip-webhook.yaml
sed -i.bak -e 's|amazonaws.com||' externalip-webhook.yaml
```
Apply the manifest to your cluster.
```
kubectl apply -f externalip-webhook.yaml
```
An attempt to deploy a service to your cluster with an IP address specified for externalIPs that is not contained in the blocks that you specified in the Specify CIDR blocks step will fail.

12.8. Copy a container image from one repository to another repository

This topic describes how to pull a container image from a repository that your nodes don’t have access to and push the image to a repository that your nodes have access to. You can push the image to Amazon ECR or an alternative repository that your nodes have access to.

The Docker engine installed and configured on your computer. For instructions, see Install Docker Engine in the Docker documentation.
Version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.
An interface VPC endpoint for Amazon ECR if you want your nodes to pull container images from or push container images to a private Amazon ECR repository over Amazon’s network. For more information, see Create the VPC endpoints for Amazon ECR in the Amazon Elastic Container Registry User Guide.

Complete the following steps to pull a container image from a repository and push it to your own repository. In the following examples that are provided in this topic, the image for the Amazon VPC CNI plugin for Kubernetes metrics helper is pulled. When you follow these steps, make sure to replace the example values with your own values.

If you don’t already have an Amazon ECR repository or another repository, then create one that your nodes have access to. The following command creates an Amazon ECR private repository. An Amazon ECR private repository name must start with a letter. It can only contain lowercase letters, numbers, hyphens (-), underscores (_), and forward slashes (/). For more information, see Creating a private repository in the Amazon Elastic Container Registry User Guide.

You can replace cni-metrics-helper with whatever you choose. As a best practice, create a separate repository for each image. We recommend this because image tags must be unique within a repository. Replace region-code with an AWS Region supported by Amazon ECR.
```
aws ecr create-repository --region region-code --repository-name cni-metrics-helper
```
Determine the registry, repository, and tag (optional) of the image that your nodes need to pull. This information is in the registry/repository[:tag] format.

Many of the Amazon EKS topics about installing images require that you apply a manifest file or install the image using a Helm chart. However, before you apply a manifest file or install a Helm chart, first view the contents of the manifest or chart’s values.yaml file. That way, you can determine the registry, repository, and tag to pull.

For example, you can find the following line in the manifest file for the Amazon VPC CNI plugin for Kubernetes metrics helper. The registry is 602401143452.dkr.ecr.us-west-2.amazonaws.com, which is an Amazon ECR private registry. The repository is cni-metrics-helper.
```
image: "602401143452.dkr.ecr.us-west-2.amazonaws.com/cni-metrics-helper:v1.12.6"
```
You may see the following variations for an image location:
- Only repository-name:tag. In this case, docker.io is usually the registry, but not specified since Kubernetes prepends it to a repository name by default if no registry is specified.
- repository-name/repository-namespace/repository:tag. A repository namespace is optional, but is sometimes specified by the repository owner for categorizing images. For example, all Amazon EC2 images in the Amazon ECR Public Gallery use the aws-ec2 namespace.
  
  Before installing an image with Helm, view the Helm values.yaml file to determine the image location. For example, the values.yaml file for the Amazon VPC CNI plugin for Kubernetes metrics helper includes the following lines.
  image: region: us-west-2 tag: v1.12.6 account: "602401143452" domain: "amazonaws.com"
Pull the container image specified in the manifest file.
1. If you’re pulling from a public registry, such as the Amazon ECR Public Gallery, you can skip to the next sub-step, because authentication isn’t required. In this example, you authenticate to an Amazon ECR private registry that contains the repository for the CNI metrics helper image. Amazon EKS maintains the image in each registry listed in View Amazon container image registries for Amazon EKS add-ons. You can authenticate to any of the registries by replacing 602401143452 and region-code with the information for a different registry. A separate registry exists for each AWS Region that Amazon EKS is supported in.
  aws ecr get-login-password --region region-code | docker login --username AWS --password-stdin 602401143452.dkr.ecr.region-code.amazonaws.com
2. Pull the image. In this example, you pull from the registry that you authenticated to in the previous sub-step. Replace 602401143452 and region-code with the information that you provided in the previous sub-step.
  docker pull 602401143452.dkr.ecr.region-code.amazonaws.com/cni-metrics-helper:v1.12.6
Tag the image that you pulled with your registry, repository, and tag. The following example assumes that you pulled the image from the manifest file and are going to push it to the Amazon ECR private repository that you created in the first step. Replace 111122223333 with your account ID. Replace region-code with the AWS Region that you created your Amazon ECR private repository in.
```
docker tag cni-metrics-helper:v1.12.6 111122223333.dkr.ecr.region-code.amazonaws.com/cni-metrics-helper:v1.12.6
```
Authenticate to your registry. In this example, you authenticate to the Amazon ECR private registry that you created in the first step. For more information, see Registry authentication in the Amazon Elastic Container Registry User Guide.
```
aws ecr get-login-password --region region-code | docker login --username AWS --password-stdin 111122223333.dkr.ecr.region-code.amazonaws.com
```
Push the image to your repository. In this example, you push the image to the Amazon ECR private repository that you created in the first step. For more information, see Pushing a Docker image in the Amazon Elastic Container Registry User Guide.
```
docker push 111122223333.dkr.ecr.region-code.amazonaws.com/cni-metrics-helper:v1.12.6
```
Update the manifest file that you used to determine the image in a previous step with the registry/repository:tag for the image that you pushed. If you’re installing with a Helm chart, there’s often an option to specify the registry/repository:tag. When installing the chart, specify the registry/repository:tag for the image that you pushed to your repository.

12.9. View Amazon container image registries for Amazon EKS add-ons

When you deploy AWS Amazon EKS add-ons to your cluster, your nodes pull the required container images from the registry specified in the installation mechanism for the add-on, such as an installation manifest or a Helm values.yaml file. The images are pulled from an Amazon EKS Amazon ECR private repository. Amazon EKS replicates the images to a repository in each Amazon EKS supported AWS Region. Your nodes can pull the container image over the internet from any of the following registries. Alternatively, your nodes can pull the image over Amazon’s network if you created an interface VPC endpoint for Amazon ECR (AWS PrivateLink) in your VPC. The registries require authentication with an AWS IAM account. Your nodes authenticate using the Amazon EKS node IAM role, which has the permissions in the AmazonEC2ContainerRegistryReadOnly managed IAM policy associated to it.

AWS Region Registry

af-south-1

877085696533.dkr.ecr.af-south-1.amazonaws.com

ap-east-1

800184023465.dkr.ecr.ap-east-1.amazonaws.com

ap-southeast-3

296578399912.dkr.ecr.ap-southeast-3.amazonaws.com

ap-south-2

900889452093.dkr.ecr.ap-south-2.amazonaws.com

ap-southeast-4

491585149902.dkr.ecr.ap-southeast-4.amazonaws.com

ap-south-1

602401143452.dkr.ecr.ap-south-1.amazonaws.com

ap-northeast-3

602401143452.dkr.ecr.ap-northeast-3.amazonaws.com

ap-northeast-2

602401143452.dkr.ecr.ap-northeast-2.amazonaws.com

ap-southeast-1

602401143452.dkr.ecr.ap-southeast-1.amazonaws.com

ap-southeast-2

602401143452.dkr.ecr.ap-southeast-2.amazonaws.com

ap-southeast-7

121268973566.dkr.ecr.ap-southeast-7.amazonaws.com

ap-northeast-1

602401143452.dkr.ecr.ap-northeast-1.amazonaws.com

cn-north-1

918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn

cn-northwest-1

961992271922.dkr.ecr.cn-northwest-1.amazonaws.com.cn

eu-central-1

602401143452.dkr.ecr.eu-central-1.amazonaws.com

eu-west-1

602401143452.dkr.ecr.eu-west-1.amazonaws.com

eu-west-2

602401143452.dkr.ecr.eu-west-2.amazonaws.com

eu-south-1

590381155156.dkr.ecr.eu-south-1.amazonaws.com

eu-west-3

602401143452.dkr.ecr.eu-west-3.amazonaws.com

eu-south-2

455263428931.dkr.ecr.eu-south-2.amazonaws.com

eu-north-1

602401143452.dkr.ecr.eu-north-1.amazonaws.com

eu-central-2

900612956339.dkr.ecr.eu-central-2.amazonaws.com

il-central-1

066635153087.dkr.ecr.il-central-1.amazonaws.com

mx-central-1

730335286997.dkr.ecr.mx-central-1.amazonaws.com

me-south-1

558608220178.dkr.ecr.me-south-1.amazonaws.com

me-central-1

759879836304.dkr.ecr.me-central-1.amazonaws.com

us-east-1

602401143452.dkr.ecr.us-east-1.amazonaws.com

us-east-2

602401143452.dkr.ecr.us-east-2.amazonaws.com

us-west-1

602401143452.dkr.ecr.us-west-1.amazonaws.com

us-west-2

602401143452.dkr.ecr.us-west-2.amazonaws.com

ca-central-1

602401143452.dkr.ecr.ca-central-1.amazonaws.com

ca-west-1

761377655185.dkr.ecr.ca-west-1.amazonaws.com

sa-east-1

602401143452.dkr.ecr.sa-east-1.amazonaws.com

us-gov-east-1

151742754352.dkr.ecr.us-gov-east-1.amazonaws.com

us-gov-west-1

013241004608.dkr.ecr.us-gov-west-1.amazonaws.com

12.10. Amazon EKS add-ons

Learn how to manage operational software add-ons on Amazon EKS clusters with Amazon EKS add-ons for observability, networking, storage, and security from AWS and third-party vendors.

An add-on is software that provides supporting operational capabilities to Kubernetes applications, but is not specific to the application. This includes software like observability agents or Kubernetes drivers that allow the cluster to interact with underlying AWS resources for networking, compute, and storage. Add-on software is typically built and maintained by the Kubernetes community, cloud providers like AWS, or third-party vendors. Amazon EKS automatically installs self-managed add-ons such as the Amazon VPC CNI plugin for Kubernetes, kube-proxy, and CoreDNS for every cluster. Note that the VPC CNI add-on isn’t compatible with Amazon EKS Hybrid Nodes and doesn’t deploy to hybrid nodes. You can change the default configuration of the add-ons and update them when desired.

Amazon EKS add-ons provide installation and management of a curated set of add-ons for Amazon EKS clusters. All Amazon EKS add-ons include the latest security patches, bug fixes, and are validated by AWS to work with Amazon EKS. Amazon EKS add-ons allow you to consistently ensure that your Amazon EKS clusters are secure and stable and reduce the amount of work that you need to do in order to install, configure, and update add-ons. If a self-managed add-on, such as kube-proxy is already running on your cluster and is available as an Amazon EKS add-on, then you can install the kube-proxy Amazon EKS add-on to start benefiting from the capabilities of Amazon EKS add-ons.

You can update specific Amazon EKS managed configuration fields for Amazon EKS add-ons through the Amazon EKS API. You can also modify configuration fields not managed by Amazon EKS directly within the Kubernetes cluster once the add-on starts. This includes defining specific configuration fields for an add-on where applicable. These changes are not overridden by Amazon EKS once they are made. This is made possible using the Kubernetes server-side apply feature. For more information, see kubernetes-field-management.title.

You can use Amazon EKS add-ons with any Amazon EKS node type. For more information, see eks-compute.title.

You can add, update, or delete Amazon EKS add-ons using the Amazon EKS API, consolelong, AWS CLI, and eksctl. You can also create Amazon EKS add-ons using AWS CloudFormation.

12.10.1. Considerations

Consider the following when you use Amazon EKS add-ons:

To configure add-ons for the cluster your IAM principal must have IAM permissions to work with add-ons. For more information, see the actions with Addon in their name in Actions defined by Amazon Elastic Kubernetes Service.
Amazon EKS add-ons run on the nodes that you provision or configure for your cluster. Node types include Amazon EC2 instances, Fargate, and hybrid nodes.
You can modify fields that aren’t managed by Amazon EKS to customize the installation of an Amazon EKS add-on. For more information, see kubernetes-field-management.title.
If you create a cluster with the consolelong, the Amazon EKS kube-proxy, Amazon VPC CNI plugin for Kubernetes, and CoreDNS Amazon EKS add-ons are automatically added to your cluster. If you use eksctl to create your cluster with a config file, eksctl can also create the cluster with Amazon EKS add-ons. If you create your cluster using eksctl without a config file or with any other tool, the self-managed kube-proxy, Amazon VPC CNI plugin for Kubernetes, and CoreDNS add-ons are installed, rather than the Amazon EKS add-ons. You can either manage them yourself or add the Amazon EKS add-ons manually after cluster creation. Regardless of the method that you use to create your cluster, the VPC CNI add-on doesn’t install on hybrid nodes.
The eks:addon-cluster-admin ClusterRoleBinding binds the cluster-admin ClusterRole to the eks:addon-manager Kubernetes identity. The role has the necessary permissions for the eks:addon-manager identity to create Kubernetes namespaces and install add-ons into namespaces. If the eks:addon-cluster-admin ClusterRoleBinding is removed, the Amazon EKS cluster will continue to function, however Amazon EKS is no longer able to manage any add-ons. All clusters starting with the following platform versions use the new ClusterRoleBinding.
A subset of EKS add-ons from AWS have been validated for compatibility with Amazon EKS Hybrid Nodes. For more information, see the compatibility table on workloads-add-ons-available-eks.title.

Required platform version

Review the table to determine the minimum required platform version to use this feature with your cluster. You can use the listed platform version, or a more recent platform version. For example, if the table lists "eks.14" you can use platform version "eks.15". For more information, see platform-versions.title.

Kubernetes version	EKS platform version
1.25 or newer	All platform versions
1.20	eks.12
1.21	eks.14
1.22	eks.9
1.23	eks.5
1.24	eks.3

12.10.2. Considerations for Amazon EKS Auto Mode

Amazon EKS Auto mode includes capabilities that deliver essential cluster functionality, including:

Pod networking
Service networking
Cluster DNS
Autoscaling
Block storage
Load balancer controller
Pod Identity agent
Node monitoring agent

With Auto mode compute, many commonly used EKS add-ons become redundant, such as:

Amazon VPC CNI
kube-proxy
CoreDNS
Amazon EBS CSI Driver
EKS Pod Identity Agent

However, if your cluster combines Auto mode with other compute options like self-managed EC2 instances, Managed Node Groups, or AWS Fargate, these add-ons remain necessary. AWS has enhanced EKS add-ons with anti-affinity rules that automatically ensure add-on pods are scheduled only on supported compute types. Furthermore, users can now leverage the EKS add-ons DescribeAddonVersions API to verify the supported computeTypes for each add-on and its specific versions. Additionally, with EKS Auto mode, the controllers listed above run on AWS owned infrastructure. So, you many not even see them in your accounts unless you are using EKS auto mode with other types of compute in which case, you will see the controllers you installed on your cluster.

If you are planning to enable EKS Auto Mode on an existing cluster, you may need to upgrade the version of certain addons. For more information, see auto-addons-required.title for EKS Auto Mode.

12.10.3. Support

AWS publishes multiple types of add-ons with different levels of support.

AWS Add-ons: These add-ons are built and fully supported by AWS.
- Use an AWS add-on to work with other AWS services, such as Amazon EFS.
- For more information, see workloads-add-ons-available-eks.title.
AWS Marketplace Add-ons: These add-ons are scanned by AWS and supported by an independent AWS partner.
- Use a marketplace add-on to add valuable and sophisticated features to your cluster, such as monitoring with Splunk.
- For more information, see workloads-add-ons-available-vendors.title.
Community Add-ons: These add-ons are scanned by AWS but supported by the open source community.
- Use a community add-on to reduce the complexity of installing common open source software, such as Kubernetes Metrics Server.
- For more information, see community-addons.title.

The following table details the scope of support for each add-on type:

Category Feature AWS add-ons AWS Marketplace add-ons Community add-ons

Development

Built by AWS

Yes

Development

Validated by AWS

Yes

Development

Validated by AWS Partner

Yes

Maintenance

Scanned by AWS

Yes

Maintenance

Patched by AWS

Yes

Maintenance

Patched by AWS Partner

Yes

Distribution

Published by AWS

Yes

Distribution

Published by AWS Partner

Yes

Support

Basic Install Support by AWS

Yes

Support

Full AWS Support

Yes

Support

Full AWS Partner Support

Yes

AWS Marketplace add-ons can download additional software dependencies from external sources outside of AWS. These external dependencies are not scanned or validated by AWS. Consider your security requirements when deploying AWS Marketplace add-ons that fetch external dependencies.

12.10.4. `AWS` Add-ons

Learn about the availabe Amazon EKS add-ons from AWS.

The following Amazon EKS add-ons are available to create on your cluster. You can view the most current list of available add-ons using eksctl, the consolelong, or the AWS CLI. To see all available add-ons or to install an add-on, see creating-an-add-on.title. If an add-on requires IAM permissions, then you must have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one, or to create one, see enable-iam-roles-for-service-accounts.title. You can an create or delete an add-on after you’ve installed it. For more information, see updating-an-add-on.title or removing-an-add-on.title. For more information about considerations specific to running EKS add-ons with Amazon EKS Hybrid Nodes, see hybrid-nodes-add-ons.title.

You can use any of the following Amazon EKS add-ons.

Description Learn more Compatible compute types

Provide native VPC networking for your cluster

add-ons-vpc-cni.title

EC2

A flexible, extensible DNS server that can serve as the Kubernetes cluster DNS

add-ons-coredns.title

EC2, Fargate, EKS Auto Mode, Amazon EKS Hybrid Nodes

Maintain network rules on each Amazon EC2 node

add-ons-kube-proxy.title

EC2, Amazon EKS Hybrid Nodes

Provide Amazon EBS storage for your cluster

EC2

Provide Amazon EFS storage for your cluster

add-ons-aws-efs-csi-driver.title

EC2, EKS Auto Mode

Provide Amazon S3 storage for your cluster

mountpoint-for-s3-add-on.title

EC2, EKS Auto Mode

Detect additional node health issues

add-ons-eks-node-monitoring-agent.title

EC2

Enable the use of snapshot functionality in compatible CSI drivers, such as the Amazon EBS CSI driver

addons-csi-snapshot-controller.title

EC2, Fargate, EKS Auto Mode, Amazon EKS Hybrid Nodes

SageMaker HyperPod task governance optimizes compute resource allocation and usage across teams in Amazon EKS clusters, addressing inefficiencies in task prioritization and resource sharing.

addons-hyperpod.title

EC2, EKS Auto Mode,

A Kubernetes agent that collects and reports network flow data to Amazon CloudWatch, enabling comprehensive monitoring of TCP connections across cluster nodes.

addons-network-flow.title

EC2, EKS Auto Mode

Secure, production-ready, AWS supported distribution of the OpenTelemetry project

add-ons-adot.title

EC2, Fargate, EKS Auto Mode, Amazon EKS Hybrid Nodes

Security monitoring service that analyzes and processes foundational data sources including AWS CloudTrail management events and Amazon VPC flow logs. Amazon GuardDuty also processes features, such as Kubernetes audit logs and runtime monitoring

add-ons-guard-duty.title

EC2, EKS Auto Mode

Monitoring and observability service provided by AWS. This add-on installs the CloudWatch Agent and enables both CloudWatch Application Signals and CloudWatch Container Insights with enhanced observability for Amazon EKS

amazon-cloudwatch-observability.title

EC2, EKS Auto Mode, Amazon EKS Hybrid Nodes

Ability to manage credentials for your applications, similar to the way that EC2 instance profiles provide credentials to EC2 instances

add-ons-pod-id.title

EC2, Amazon EKS Hybrid Nodes

Amazon VPC CNI plugin for Kubernetes

Learn about the vpc-cni Amazon EKS add-on.

The Amazon VPC CNI plugin for Kubernetes Amazon EKS add-on is a Kubernetes container network interface (CNI) plugin that provides native VPC networking for your cluster. The self-managed or managed type of this add-on is installed on each Amazon EC2 node, by default. For more information, see Kubernetes container network interface (CNI) plugin.

You do not need to install this add-on on Amazon EKS Auto Mode clusters. For more information, see addon-consider-auto.title.

The Amazon EKS add-on name is vpc-cni.

Required IAM permissions

This add-on uses the IAM roles for service accounts capability of Amazon EKS. For more information, see iam-roles-for-service-accounts.title.

If your cluster uses the IPv4 family, the permissions in the AmazonEKS_CNI_Policy are required. If your cluster uses the IPv6 family, you must create an IAM policy with the permissions in IPv6 mode. You can create an IAM role, attach one of the policies to it, and annotate the Kubernetes service account used by the add-on with the following command.

Replace my-cluster with the name of your cluster and AmazonEKSVPCCNIRole with the name for your role. If your cluster uses the IPv6 family, then replace AmazonEKS_CNI_Policy with the name of the policy that you created. This command requires that you have eksctl installed on your device. If you need to use a different tool to create the role, attach the policy to it, and annotate the Kubernetes service account, see associate-service-account-role.title.

eksctl create iamserviceaccount --name aws-node --namespace kube-system --cluster my-cluster --role-name AmazonEKSVPCCNIRole \
    --role-only --attach-policy-arn region.arniam::aws:policy/AmazonEKS_CNI_Policy --approve

Update information

You can only update one minor version at a time. For example, if your current version is 1.28.x-eksbuild.y and you want to update to 1.30.x-eksbuild.y, then you must update your current version to 1.29.x-eksbuild.y and then update it again to 1.30.x-eksbuild.y. For more information about updating the add-on, see vpc-add-on-update.title.

CoreDNS

Learn about the CoreDNS Amazon EKS add-on.

The CoreDNS Amazon EKS add-on is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS. The self-managed or managed type of this add-on was installed, by default, when you created your cluster. When you launch an Amazon EKS cluster with at least one node, two replicas of the CoreDNS image are deployed by default, regardless of the number of nodes deployed in your cluster. The CoreDNS Pods provide name resolution for all Pods in the cluster. You can deploy the CoreDNS Pods to Fargate nodes if your cluster includes a Fargate profile with a namespace that matches the namespace for the CoreDNS deployment. For more information, see fargate-profile.title

You do not need to install this add-on on Amazon EKS Auto Mode clusters. For more information, see addon-consider-auto.title.

The Amazon EKS add-on name is coredns.

Required IAM permissions

This add-on doesn’t require any permissions.

Additional information

To learn more about CoreDNS, see Using CoreDNS for Service Discovery and Customizing DNS Service in the Kubernetes documentation.

`Kube-proxy`

Learn about the Kube-proxy Amazon EKS add-on.

The Kube-proxy Amazon EKS add-on maintains network rules on each Amazon EC2 node. It enables network communication to your Pods. The self-managed or managed type of this add-on is installed on each Amazon EC2 node in your cluster, by default.

You do not need to install this add-on on Amazon EKS Auto Mode clusters. For more information, see addon-consider-auto.title.

The Amazon EKS add-on name is kube-proxy.

Required IAM permissions

This add-on doesn’t require any permissions.

Update information

Before updating your current version, consider the following requirements:

Kube-proxy on an Amazon EKS cluster has the same compatibility and skew policy as Kubernetes.

Additional information

To learn more about kube-proxy, see kube-proxy in the Kubernetes documentation.

Amazon EBS CSI driver

Learn about the Amazon EBS CSI driver Amazon EKS add-on.

The Amazon EBS CSI driver Amazon EKS add-on is a Kubernetes Container Storage Interface (CSI) plugin that provides Amazon EBS storage for your cluster.

You do not need to install this add-on on Amazon EKS Auto Mode clusters. Auto Mode includes a block storage capability. For more information, see sample-storage-workload.title.

The Amazon EKS add-on name is aws-ebs-csi-driver.

Required IAM permissions

This add-on utilizes the IAM roles for service accounts capability of Amazon EKS. For more information, see iam-roles-for-service-accounts.title. The permissions in the AmazonEBSCSIDriverPolicy AWS managed policy are required. You can create an IAM role and attach the managed policy to it with the following command. Replace my-cluster with the name of your cluster and AmazonEKS_EBS_CSI_DriverRole with the name for your role. This command requires that you have eksctl installed on your device. If you need to use a different tool or you need to use a custom KMS key for encryption, see csi-iam-role.title.

eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace kube-system \
    --cluster my-cluster \
    --role-name AmazonEKS_EBS_CSI_DriverRole \
    --role-only \
    --attach-policy-arn region.arniam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
    --approve

Additional information

To learn more about the add-on, see ebs-csi.title.

Amazon EFS CSI driver

Learn about the Amazon EFS CSI driver Amazon EKS add-on.

The Amazon EFS CSI driver Amazon EKS add-on is a Kubernetes Container Storage Interface (CSI) plugin that provides Amazon EFS storage for your cluster.

The Amazon EKS add-on name is aws-efs-csi-driver.

Required IAM permissions

Required IAM permissions – This add-on utilizes the IAM roles for service accounts capability of Amazon EKS. For more information, see iam-roles-for-service-accounts.title. The permissions in the AmazonEFSCSIDriverPolicy AWS managed policy are required. You can create an IAM role and attach the managed policy to it with the following commands. Replace my-cluster with the name of your cluster and AmazonEKS_EFS_CSI_DriverRole with the name for your role. These commands require that you have eksctl installed on your device. If you need to use a different tool, see efs-create-iam-resources.title.

export cluster_name=my-cluster
export role_name=AmazonEKS_EFS_CSI_DriverRole
eksctl create iamserviceaccount \
    --name efs-csi-controller-sa \
    --namespace kube-system \
    --cluster $cluster_name \
    --role-name $role_name \
    --role-only \
    --attach-policy-arn region.arniam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
    --approve
TRUST_POLICY=$(aws iam get-role --role-name $role_name --query 'Role.AssumeRolePolicyDocument' | \
    sed -e 's/efs-csi-controller-sa/efs-csi-*/' -e 's/StringEquals/StringLike/')
aws iam update-assume-role-policy --role-name $role_name --policy-document "$TRUST_POLICY"

Additional information

To learn more about the add-on, see efs-csi.title.

`Mountpoint` for Amazon S3 CSI Driver

Learn about the Mountpoint for Amazon S3 CSI Driver Amazon EKS add-on.

The Mountpoint for Amazon S3 CSI Driver Amazon EKS add-on is a Kubernetes Container Storage Interface (CSI) plugin that provides Amazon S3 storage for your cluster.

The Amazon EKS add-on name is aws-mountpoint-s3-csi-driver.

Required IAM permissions

This add-on uses the IAM roles for service accounts capability of Amazon EKS. For more information, see iam-roles-for-service-accounts.title.

The IAM role that is created will require a policy that gives access to S3. Follow the Mountpoint IAM permissions recommendations when creating the policy. Alternatively, you may use the AWS managed policy AmazonS3FullAccess, but this managed policy grants more permissions than are needed for Mountpoint.

You can create an IAM role and attach your policy to it with the following commands. Replace my-cluster with the name of your cluster, region-code with the correct AWS Region code, AmazonEKS_S3_CSI_DriverRole with the name for your role, and AmazonEKS_S3_CSI_DriverRole_ARN with the role ARN. These commands require that you have eksctl installed on your device. For instructions on using the IAM console or AWS CLI, see s3-create-iam-role.title.

CLUSTER_NAME=my-cluster
REGION=region-code
ROLE_NAME=AmazonEKS_S3_CSI_DriverRole
POLICY_ARN=AmazonEKS_S3_CSI_DriverRole_ARN
eksctl create iamserviceaccount \
    --name s3-csi-driver-sa \
    --namespace kube-system \
    --cluster $CLUSTER_NAME \
    --attach-policy-arn $POLICY_ARN \
    --approve \
    --role-name $ROLE_NAME \
    --region $REGION \
    --role-only

Additional information

To learn more about the add-on, see s3-csi.title.

CSI snapshot controller

Learn about the CSI snapshot controller Amazon EKS add-on.

The Container Storage Interface (CSI) snapshot controller enables the use of snapshot functionality in compatible CSI drivers, such as the Amazon EBS CSI driver.

The Amazon EKS add-on name is snapshot-controller.

Required IAM permissions

This add-on doesn’t require any permissions.

Additional information

To learn more about the add-on, see csi-snapshot-controller.title.

Amazon SageMaker HyperPod task governance

SageMaker HyperPod task governance is a robust management system designed to streamline resource allocation and ensure efficient utilization of compute resources across teams and projects for your Amazon EKS clusters. This provides administrators with the capability to set:

Priority levels for various tasks
Compute allocation for each team
How each team lends and borrows idle compute
If a team preempts their own tasks

HyperPod task governance also provides Amazon EKS cluster Observability, offering real-time visibility into cluster capacity. This includes compute availability and usage, team allocation and utilization, and task run and wait time information, setting you up for informed decision-making and proactive resource management.

The Amazon EKS add-on name is amazon-sagemaker-hyperpod-taskgovernance.

Required IAM permissions

This add-on doesn’t require any permissions.

Additional information

To learn more about the add-on, see SageMaker HyperPod task governance

`AWS` Network Flow Monitor Agent

The Amazon CloudWatch Network Flow Monitor Agent is a Kubernetes application that collects TCP connection statistics from all nodes in a cluster and publishes network flow reports to Amazon CloudWatch Network Flow Monitor Ingestion APIs.

The Amazon EKS add-on name is aws-network-flow-monitoring-agent.

Required IAM permissions

This add-on does require IAM permissions.

You need to attach the CloudWatchNetworkFlowMonitorAgentPublishPolicy managed policy to the add-on.

For more information on the required IAM setup, see IAM Policy on the Amazon CloudWatch Network Flow Monitor Agent GitHub repo.

For more information about the managed policy, see CloudWatchNetworkFlowMonitorAgentPublishPolicy in the Amazon CloudWatch User Guide.

Additional information

To learn more about the add-on, see the Amazon CloudWatch Network Flow Monitor Agent GitHub repo.

Node monitoring agent

The node monitoring agent Amazon EKS add-on can detect additional node health issues. These extra health signals can also be leveraged by the optional node auto repair feature to automatically replace nodes as needed.

You do not need to install this add-on on Amazon EKS Auto Mode clusters. For more information, see addon-consider-auto.title.

The Amazon EKS add-on name is eks-node-monitoring-agent.

Required IAM permissions

This add-on doesn’t require additional permissions.

Additional information

For more information, see node-health.title.

`AWS` Distro for OpenTelemetry

Learn about the AWS Distro for OpenTelemetry Amazon EKS add-on.

The AWS Distro for OpenTelemetry Amazon EKS add-on is a secure, production-ready, AWS supported distribution of the OpenTelemetry project. For more information, see AWS Distro for OpenTelemetry on GitHub.

The Amazon EKS add-on name is adot.

Required IAM permissions

This add-on only requires IAM permissions if you’re using one of the preconfigured custom resources that can be opted into through advanced configuration.

Additional information

For more information, see Getting Started with AWS Distro for OpenTelemetry using EKS Add-Ons in the AWS Distro for OpenTelemetry documentation.

ADOT requires that cert-manager is deployed on the cluster as a prerequisite, otherwise this add-on won’t work if deployed directly using the https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latestcluster_addons property. For more requirements, see Requirements for Getting Started with AWS Distro for OpenTelemetry using EKS Add-Ons in the AWS Distro for OpenTelemetry documentation.

Amazon GuardDuty agent

Learn about the Amazon GuardDuty agent Amazon EKS add-on.

The Amazon GuardDuty agent Amazon EKS add-on is a security monitoring service that analyzes and processes foundational data sources including AWS CloudTrail management events and Amazon VPC flow logs. Amazon GuardDuty also processes features, such as Kubernetes audit logs and runtime monitoring.

The Amazon EKS add-on name is aws-guardduty-agent.

Required IAM permissions

This add-on doesn’t require any permissions.

Additional information

For more information, see Runtime Monitoring for Amazon EKS clusters in Amazon GuardDuty.

To detect potential security threats in your Amazon EKS clusters, enable Amazon GuardDuty runtime monitoring and deploy the GuardDuty security agent to your Amazon EKS clusters.

Amazon CloudWatch Observability agent

Learn about the Amazon CloudWatch Observability agent Amazon EKS add-on.

The Amazon CloudWatch Observability agent Amazon EKS add-on the monitoring and observability service provided by AWS. This add-on installs the CloudWatch Agent and enables both CloudWatch Application Signals and CloudWatch Container Insights with enhanced observability for Amazon EKS. For more information, see Amazon CloudWatch Agent.

The Amazon EKS add-on name is amazon-cloudwatch-observability.

Required IAM permissions

This add-on uses the IAM roles for service accounts capability of Amazon EKS. For more information, see iam-roles-for-service-accounts.title. The permissions in the AWSXrayWriteOnlyAccess and CloudWatchAgentServerPolicy AWS managed policies are required. You can create an IAM role, attach the managed policies to it, and annotate the Kubernetes service account used by the add-on with the following command. Replace my-cluster with the name of your cluster and AmazonEKS_Observability_role with the name for your role. This command requires that you have eksctl installed on your device. If you need to use a different tool to create the role, attach the policy to it, and annotate the Kubernetes service account, see associate-service-account-role.title.

eksctl create iamserviceaccount \
    --name cloudwatch-agent \
    --namespace amazon-cloudwatch \
    --cluster my-cluster \
    --role-name AmazonEKS_Observability_Role \
    --role-only \
    --attach-policy-arn region.arniam::aws:policy/AWSXrayWriteOnlyAccess \
    --attach-policy-arn region.arniam::aws:policy/CloudWatchAgentServerPolicy \
    --approve

Additional information

For more information, see Install the CloudWatch agent.

EKS Pod Identity Agent

Learn about the EKS Pod Identity Agent Amazon EKS add-on.

The Amazon EKS Pod Identity Agent Amazon EKS add-on provides the ability to manage credentials for your applications, similar to the way that EC2 instance profiles provide credentials to EC2 instances.

You do not need to install this add-on on Amazon EKS Auto Mode clusters. Amazon EKS Auto Mode integrates with EKS Pod Identity. For more information, see addon-consider-auto.title.

The Amazon EKS add-on name is eks-pod-identity-agent.

Required IAM permissions

This add-on users permissions from the Amazon EKS node IAM role.

Update information

12.10.5. Community add-ons

You can use AWS APIs to install community add-ons, such as the Kubernetes Metrics Server. You may choose to install community add-ons as Amazon EKS Add-ons to reduce the complexity of maintaining the software on multiple clusters.

For example, you can use the AWS API, CLI, or Management Console to install community add-ons. You can install a community add-on during cluster creation.

You manage community add-ons just like existing Amazon EKS Add-ons. Community add-ons are different from existing add-ons in that they have a unique scope of support.

Community add-ons are built and validated by AWS. Importantly, AWS does not provide full support for community add-ons. AWS supports only lifecycle operations done using AWS APIs, such as installing add-ons or deleting add-ons.

If you require support for a community add-on, utilize the existing project resources. For example, you may create a GitHub issue on the repo for the project.

Determine add-on type

You can use the AWS CLI to determine the type of an Amazon EKS Add-on.

Use the following CLI command to retrieve information about an add-on. You can replace metrics-server with the name of any add-on.

aws eks describe-addon-versions --addon-name metrics-server

Review the CLI output for the owner field.

{
    "addons": [
        {
            "addonName": "metrics-server",
            "type": "observability",
            "owner": "community",
            "addonVersions": [

If the value of owner is community, then the add-on is a community add-on. AWS only provides support for installing, updating, and removing the add-on. If you have questions about the functionality and operation of the add-on itself, use community resources like GitHub issues.

Install or update community add-on

You install or update community add-ons in the same way as other Amazon EKS Add-ons.

creating-an-add-on.title
updating-an-add-on.title
removing-an-add-on.title

Available community add-ons

The following community add-ons are availalbe from Amazon EKS.

`Kubernetes Metrics Server`

The Kubernetes Metrics Server is a scalable and efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines. It collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

Property Value

Add-on name

metrics-server

Namespace

kube-system

Documentation

GitHub Readme

Service account name

None

Managed IAM policy

None

Custom IAM permissions

None

View license attributions for this add-on.

12.10.6. `AWS` Marketplace add-ons

Learn about the Amazon EKS add-ons from independent software vendors.

In addition to the previous list of Amazon EKS add-ons, you can also add a wide selection of operational software Amazon EKS add-ons from independent software vendors. Choose an add-on to learn more about it and its installation requirements.

`Accuknox`

Learn about the Accuknox Amazon EKS add-on.

The add-on name is accuknox_kubearmor and the namespace is kubearmor. Accuknox publishes the add-on.

For information about the add-on, see Getting Started with KubeArmor in the KubeArmor documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Akuity`

Learn about the Akuity Amazon EKS add-on.

The add-on name is akuity_agent and the namespace is akuity. Akuity publishes the add-on.

For information about how the add-on, see Installing the Akuity Agent on Amazon EKS with the Akuity EKS add-on in the Akuity Platform documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Calyptia`

Learn about the Calyptia Amazon EKS add-on.

The add-on name is calyptia_fluent-bit and the namespace is calytia-fluentbit. Calyptia publishes the add-on.

For information about the add-on, see Getting Started with Calyptia Core Agent on the Calyptia documentation website.

Service account name

The service account name is clyptia-fluentbit.

`AWS` managed IAM policy

This add-on uses the AWSMarketplaceMeteringRegisterUsage managed policy. For more information, see AWSMarketplaceMeteringRegisterUsage in the AWS Managed Policy Reference Guide.

Command to create required IAM role

The following command requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one, or to create one, see enable-iam-roles-for-service-accounts.title. Replace my-cluster with the name of your cluster and my-calyptia-role with the name for your role. This command requires that you have eksctl installed on your device. If you need to use a different tool to create the role and annotate the Kubernetes service account, see associate-service-account-role.title.

eksctl create iamserviceaccount --name service-account-name  --namespace calyptia-fluentbit --cluster my-cluster --role-name my-calyptia-role \
    --role-only --attach-policy-arn region.arniam::aws:policy/AWSMarketplaceMeteringRegisterUsage --approve

`Cisco Observability Collector`

Learn about the Cisco Observability Collector Amazon EKS add-on.

The add-on name is cisco_cisco-cloud-observability-collectors and the namespace is appdynamics. Cisco pubishes the add-on.

For information about the add-on, see Use the Cisco Cloud Observability AWS Marketplace Add-Ons in the Cisco AppDynamics documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Cisco Observability Operator`

Learn about the Cisco Observability Operator Amazon EKS add-on.

The add-on name is cisco_cisco-cloud-observability-operators and the namespace is appdynamics. Cisco publishes the add-on.

For information about the add-on, see Use the Cisco Cloud Observability AWS Marketplace Add-Ons in the Cisco AppDynamics documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`CLOUDSOFT`

Learn about the CLOUDSOFT Amazon EKS add-on.

The add-on name is cloudsoft_cloudsoft-amp and the namespace is cloudsoft-amp. CLOUDSOFT publishes the add-on.

For information about the add-on, see Amazon EKS ADDON in the CLOUDSOFT documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Cribl`

Learn about the Cribl Amazon EKS add-on.

The add-on name is cribl_cribledge and the namespace is cribledge. Cribl publishes the add-on.

For information about the add-on, see Installing the Cribl Amazon EKS Add-on for Edge in the Cribl documentation

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Dynatrace`

Learn about the Dynatrace Amazon EKS add-on.

The add-on name is dynatrace_dynatrace-operator and the namespace is dynatrace. Dynatrace publishes the add-on.

For information about the add-on, see Kubernetes monitoring in the dynatrace documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Datree`

Learn about the Datree Amazon EKS add-on.

The add-on name is datree_engine-pro and the namespace is datree. Datree publishes the add-on.

For information about the add-on, see Amazon EKS-intergration in the Datree documentation.

Service account name

The service account name is datree-webhook-server-awsmp.

`AWS` managed IAM policy

The managed policy is AWSLicenseManagerConsumptionPolicy. For more information, see AWSLicenseManagerConsumptionPolicy in the AWS Managed Policy Reference Guide..

Command to create required IAM role

The following command requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one, or to create one, see enable-iam-roles-for-service-accounts.title. Replace my-cluster with the name of your cluster and my-datree-role with the name for your role. This command requires that you have eksctl installed on your device. If you need to use a different tool to create the role and annotate the Kubernetes service account, see associate-service-account-role.title.

eksctl create iamserviceaccount --name datree-webhook-server-awsmp --namespace datree --cluster my-cluster --role-name my-datree-role \
    --role-only --attach-policy-arn region.arniam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy --approve

Custom permissions

Custom permissions aren’t used with this add-on.

`Datadog`

Learn about the Datadog Amazon EKS add-on.

The add-on name is datadog_operator and the namespace is datadog-agent. Datadog publishes the add-on.

For information about the add-on, see Installing the Datadog Agent on Amazon EKS with the Datadog Operator Add-on in the Datadog documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Groundcover`

Learn about the Groundcover Amazon EKS add-on.

The add-on name is groundcover_agent and the namespace is groundcover. groundcover publishes the add-on.

For information about the add-on, see Installing the groundcover Amazon EKS Add-on in the groundcover documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Grafana Labs`

Learn about the Grafana Labs Amazon EKS add-on.

The add-on name is grafana-labs_kubernetes-monitoring and the namespace is monitoring. Grafana Labs publishes the add-on.

For information about the add-on, see Configure Kubernetes Monitoring as an Add-on with Amazon EKS in the Grafana Labs documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Guance`

Publisher – GUANCE
Name – guance_datakit
Namespace – datakit
Service account name – A service account isn’t used with this add-on.
AWS managed IAM policy – A managed policy isn’t used with this add-on.
Custom IAM permissions – Custom permissions aren’t used with this add-on.
Setup and usage instructions – See Using Amazon EKS add-on in the Guance documentation.

`HA Proxy`

Learn about the HA Proxy Amazon EKS add-on.

The name is haproxy-technologies_kubernetes-ingress-ee and the namespace is haproxy-controller. HA Proxy publishes the add-on.

For information about the add-on, see Amazon EKS-intergration in the Datree documentation.

Service account name

The service account name is customer defined.

`AWS` managed IAM policy

The managed policy is AWSLicenseManagerConsumptionPolicy. For more information, see AWSLicenseManagerConsumptionPolicy in the AWS Managed Policy Reference Guide..

Command to create required IAM role

The following command requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one, or to create one, see enable-iam-roles-for-service-accounts.title. Replace my-cluster with the name of your cluster and my-haproxy-role with the name for your role. This command requires that you have eksctl installed on your device. If you need to use a different tool to create the role and annotate the Kubernetes service account, see associate-service-account-role.title.

eksctl create iamserviceaccount --name service-account-name  --namespace haproxy-controller --cluster my-cluster --role-name my-haproxy-role \
    --role-only --attach-policy-arn region.arniam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy --approve

Custom permissions

Custom permissions aren’t used with this add-on.

`Kpow`

Learn about the Kpow Amazon EKS add-on.

The add-on name is factorhouse_kpow and the namespace is factorhouse. Factorhouse publishes the add-on.

For information about the add-on, see AWS Marketplace LM in the Kpow documentation.

Service account name

The service account name is kpow.

`AWS` managed IAM policy

The managed policy is AWSLicenseManagerConsumptionPolicy. For more information, see AWSLicenseManagerConsumptionPolicy in the AWS Managed Policy Reference Guide..

Command to create required IAM role

The following command requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one, or to create one, see enable-iam-roles-for-service-accounts.title. Replace my-cluster with the name of your cluster and my-kpow-role with the name for your role. This command requires that you have eksctl installed on your device. If you need to use a different tool to create the role and annotate the Kubernetes service account, see associate-service-account-role.title.

eksctl create iamserviceaccount --name kpow --namespace factorhouse --cluster my-cluster --role-name my-kpow-role \
    --role-only --attach-policy-arn region.arniam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy --approve

Custom permissions

Custom permissions aren’t used with this add-on.

`Kubecost`

Learn about the Kubecost Amazon EKS add-on.

The add-on name is kubecost_kubecost and the namespace is kubecost. Kubecost publishes the add-on.

For information about the add-on, see AWS Cloud Billing Integration in the Kubecost documentation.

If your cluster is version 1.23 or later, you must have the Store Kubernetes volumes with Amazon EBS installed on your cluster. otherwise you will receive an error.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Kasten`

Learn about the Kasten Amazon EKS add-on.

The add-on name is kasten_k10 and the namespace is kasten-io. Kasten by Veeam publishes the add-on.

For information about the add-on, see Installing K10 on AWS using Amazon EKS Add-on in the Kasten documentation.

If your Amazon EKS cluster is version Kubernetes 1.23 or later, you must have the Amazon EBS CSI driver installed on your cluster with a default StorageClass.

Service account name

The service account name is k10-k10.

`AWS` managed IAM policy

The managed policy is AWSLicenseManagerConsumptionPolicy. For more information, see AWSLicenseManagerConsumptionPolicy in the AWS Managed Policy Reference Guide..

Command to create required IAM role

The following command requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one, or to create one, see enable-iam-roles-for-service-accounts.title. Replace my-cluster with the name of your cluster and my-kasten-role with the name for your role. This command requires that you have eksctl installed on your device. If you need to use a different tool to create the role and annotate the Kubernetes service account, see associate-service-account-role.title.

eksctl create iamserviceaccount --name k10-k10 --namespace kasten-io --cluster my-cluster --role-name my-kasten-role \
    --role-only --attach-policy-arn region.arniam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy --approve

Custom permissions

Custom permissions aren’t used with this add-on.

`Kong`

Learn about the Kong Amazon EKS add-on.

The add-on name is kong_konnect-ri and the namespace is kong. Kong publishes the add-on.

For information about the add-on, see Installing the Kong Gateway EKS Add-on in the Kong documentation.

If your cluster is version 1.23 or later, you must have the Store Kubernetes volumes with Amazon EBS installed on your cluster. otherwise you will receive an error.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`LeakSignal`

Learn about the LeakSignal Amazon EKS add-on.

The add-on name is leaksignal_leakagent and the namespace is leakagent. LeakSignal publishes the add-on.

For information about the add-on, see https://www.leaksignal.com/docs/LeakAgent/Deployment/AWS%20EKS%20Addon/[Install the LeakAgent add-on] in the LeakSignal documentation

If your cluster is version 1.23 or later, you must have the Store Kubernetes volumes with Amazon EBS installed on your cluster. otherwise you will receive an error.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`NetApp`

Learn about the NetApp Amazon EKS add-on.

The add-on name is netapp_trident-operator and the namespace is trident. NetApp publishes the add-on.

For information about the add-on, see Configure the Trident EKS add-on in the NetApp documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`New Relic`

Learn about the New Relic Amazon EKS add-on.

The add-on name is new-relic_kubernetes-operator and the namespace is newrelic. New Relic publishes the add-on.

For information about the add-on, see Installing the New Relic Add-on for EKS in the New Relic documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Rafay`

Learn about the Rafay Amazon EKS add-on.

The add-on name is rafay-systems_rafay-operator and the namespace is rafay-system. Rafay publishes the add-on.

For information about the add-on, see Installing the Rafay Amazon EKS Add-on in the Rafay documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Rad Security`

Publisher – RAD SECURITY
Name – rad-security_rad-security
Namespace – ksoc
Service account name – A service account isn’t used with this add-on.
AWS managed IAM policy – A managed policy isn’t used with this add-on.
Custom IAM permissions – Custom permissions aren’t used with this add-on.
Setup and usage instructions – See Installing Rad Through The AWS Marketplace in the Rad Security documentation.

`SolarWinds`

Publisher – SOLARWINDS
Name – solarwinds_swo-k8s-collector-addon
Namespace – solarwinds
Service account name – A service account isn’t used with this add-on.
AWS managed IAM policy – A managed policy isn’t used with this add-on.
Custom IAM permissions – Custom permissions aren’t used with this add-on.
Setup and usage instructions – See Monitor an Amazon EKS cluster in the SolarWinds documentation.

`Solo`

Learn about the Solo Amazon EKS add-on.

The add-on name is solo-io_istio-distro and the namespace is istio-system. Solo publishes the add-on.

For information about the add-on, see Installing Istio in the Solo.io documentation..

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Snyk`

Publisher – SNYK
Name – snyk_runtime-sensor
Namespace – snyk_runtime-sensor
Service account name – A service account isn’t used with this add-on.
AWS managed IAM policy – A managed policy isn’t used with this add-on.
Custom IAM permissions – Custom permissions aren’t used with this add-on.
Setup and usage instructions – See Snyk runtime sensor in the Snyk user docs.

`Stormforge`

Learn about the Stormforge Amazon EKS add-on.

The add-on name is stormforge_optimize-Live and the namespace is stormforge-system. Stormforge publishes the add-on.

For information about the add-on, see Installing the StormForge Agent in the StormForge documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Splunk`

Learn about the Splunk Amazon EKS add-on.

The add-on name is splunk_splunk-otel-collector-chart and the namespace is splunk-monitoring. Splunk publishes the add-on.

For information about the add-on, see Install the Splunk add-on for Amazon EKS in the Splunk documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Teleport`

Learn about the Teleport Amazon EKS add-on.

The add-on name is teleport_teleport and the namespace is teleport. Teleport publishes the add-on.

For information about the add-on, see How Teleport Works in the Teleport documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Tetrate`

Learn about the Tetrate Amazon EKS add-on.

The add-on name is tetrate-io_istio-distro and the namespace is istio-system. Tetrate Io publishes the add-on.

For information about the add-on, see the Tetrate Istio Distro website.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Upbound Universal Crossplane`

Learn about the Upbound Universal Crossplane Amazon EKS add-on.

The add-on name is upbound_universal-crossplane and the namespace is upbound-system. Upbound publishes the add-on.

For information about the add-on, see Upbound Universal Crossplane (UXP) in the Upbound documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

`Upwind`

Learn about the Upwind Amazon EKS add-on.

The add-on name is upwind and the namespace is upwind. Upwind publishes the add-on.

For information about the add-on, see Upwind documentation.

Service account name

A service account isn’t used with this add-on.

`AWS` managed IAM policy

A managed policy isn’t used with this add-on.

Custom IAM permissions

Custom permissions aren’t used with this add-on.

12.10.7. Create an Amazon EKS add-on

Learn how to create an add-on for your Amazon EKS cluster.

Amazon EKS add-ons are add-on software for Amazon EKS clusters. All Amazon EKS add-ons:

Include the latest security patches and bug fixes.
Are validated by AWS to work with Amazon EKS.
Reduce the amount of work required to manage the add-on software.

You can create an Amazon EKS add-on using eksctl, the consolelong, or the AWS CLI. If the add-on requires an IAM role, see the details for the specific add-on in Amazon EKS add-ons for details about creating the role.

Prerequisites

Complete the following before you create an add-on:

The cluster must exist before you create an add-on for it. For more information, see create-cluster.title.
Check if your add-on requires an IAM role. For more information, see addon-compat.title.
Verify that the Amazon EKS add-on version is compatabile with your cluster. For more information, see addon-compat.title.
Verify that version 0.190.0 or later of the eksctl command line tool installed on your computer or AWS CloudShell. For more information, see Installation on the eksctl website.

Procedure

You can create an Amazon EKS add-on using eksctl, the consolelong, or the AWS CLI. If the add-on requires an IAM role, see the details for the specific add-on in Available Amazon EKS add-ons from AWS for details about creating the role.

Create add-on (eksctl)

View the names of add-ons available for a cluster version. Replace 1.30 with the version of your cluster.

eksctl utils describe-addon-versions --kubernetes-version 1.30 | grep AddonName

An example output is as follows.

"AddonName": "aws-ebs-csi-driver",
                        "AddonName": "coredns",
                        "AddonName": "kube-proxy",
                        "AddonName": "vpc-cni",
                        "AddonName": "adot",
                        "AddonName": "dynatrace_dynatrace-operator",
                        "AddonName": "upbound_universal-crossplane",
                        "AddonName": "teleport_teleport",
                        "AddonName": "factorhouse_kpow",
                        [...]

View the versions available for the add-on that you would like to create. Replace 1.30 with the version of your cluster. Replace name-of-addon with the name of the add-on you want to view the versions for. The name must be one of the names returned in the previous step.
```
eksctl utils describe-addon-versions --kubernetes-version 1.30 --name name-of-addon | grep AddonVersion
```
The following output is an example of what is returned for the add-on named vpc-cni. You can see that the add-on has several available versions.
```
"AddonVersions": [
    "AddonVersion": "v1.12.0-eksbuild.1",
    "AddonVersion": "v1.11.4-eksbuild.1",
    "AddonVersion": "v1.10.4-eksbuild.1",
    "AddonVersion": "v1.9.3-eksbuild.1",
```
1. Determine whether the add-on you want to create is an Amazon EKS or AWS Marketplace add-on. The AWS Marketplace has third party add-ons that require you to complete additional steps to create the add-on.
  eksctl utils describe-addon-versions --kubernetes-version 1.30 --name name-of-addon | grep ProductUrl
  If no output is returned, then the add-on is an Amazon EKS. If output is returned, then the add-on is an AWS Marketplace add-on. The following output is for an add-on named teleport_teleport.
  "ProductUrl": "https://aws.amazon.com/marketplace/pp?sku=3bda70bb-566f-4976-806c-f96faef18b26"
  You can learn more about the add-on in the AWS Marketplace with the returned URL. If the add-on requires a subscription, you can subscribe to the add-on through the AWS Marketplace. If you’re going to create an add-on from the AWS Marketplace, then the IAM principal that you’re using to create the add-on must have permission to create the AWSServiceRoleForAWSLicenseManagerRole service-linked role. For more information about assigning permissions to an IAM entity, see Adding and removing IAM identity permissions in the IAM User Guide.
Create an Amazon EKS add-on. Copy the command and replace the user-data as follows:
- Replace my-cluster with the name of your cluster.
- Replace name-of-addon with the name of the add-on that you want to create.
- If you want a version of the add-on that’s earlier than the latest version, then replace latest with the version number returned in the output of a previous step that you want to use.
- If the add-on uses a service account role, replace 111122223333 with your account ID and replace role-name with the name of the role. For instructions on creating a role for your service account, see the documentation for the add-on that you’re creating. For a list of add-ons, see workloads-add-ons-available-eks.title. Specifying a service account role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.
  
  If the add-on doesn’t use a service account role, delete --service-account-role-arnregion.arniam::111122223333:role/role-name.
- This example command overwrites the configuration of any existing self-managed version of the add-on, if there is one. If you don’t want to overwrite the configuration of an existing self-managed add-on, remove the --force option. If you remove the option, and the Amazon EKS add-on needs to overwrite the configuration of an existing self-managed add-on, then creation of the Amazon EKS add-on fails with an error message to help you resolve the conflict. Before specifying this option, make sure that the Amazon EKS add-on doesn’t manage settings that you need to manage, because those settings are overwritten with this option.
  eksctl create addon --cluster my-cluster --name name-of-addon --version latest \ --service-account-role-arn region.arniam::111122223333:role/role-name --force
  You can see a list of all available options for the command.
  eksctl create addon --help
  For more information about available options see Addons in the eksctl documentation.

Create add-on (`AWS` Console)

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
Choose the name of the cluster that you want to create the add-on for.
Choose the Add-ons tab.
Choose Get more add-ons.
On the Select add-ons page, choose the add-ons that you want to add to your cluster. You can add as many Amazon EKS add-ons and AWS Marketplace add-ons as you require.

For AWS Marketplace add-ons the IAM principal that you’re using to create the add-on must have permissions to read entitlements for the add-on from the AWS LicenseManager. AWS LicenseManager requires AWSServiceRoleForAWSLicenseManagerRole service-linked role (SLR) that allows AWS resources to manage licenses on your behalf. The SLR is a one time requirement, per account, and you will not have to create separate SLR’s for each add-on nor each cluster. For more information about assigning permissions to an IAM principal see Adding and removing IAM identity permissions in the IAM User Guide.

If the AWS Marketplace add-ons that you want to install aren’t listed, you can click the page numbering to view additional page results or search in the search box. In the Filtering options, you can also filter by category, vendor, or pricing model and then choose the add-ons from the search results. Once you’ve selected the add-ons that you want to install, choose Next.
On the Configure selected add-ons settings page, do the following:
1. Choose View subscription options to open the Subscription options form. Review the Pricing details and Legal sections, then choose the Subscribe button to continue.
2. For Version, choose the version that you want to install. We recommend the version marked latest, unless the individual add-on that you’re creating recommends a different version. To determine whether an add-on has a recommended version, see the documentation for the add-on that you’re creating. For a list of add-ons, see workloads-add-ons-available-eks.title.
3. You have two options for configuring roles for add-ons: EKS Pod Identities IAM role and IAM roles for service accounts (IRSA). Follow the appropriate step below for your preferred option. If all of the add-ons that you selected have Requires subscription under Status, choose Next. You can’t configure those add-ons further until you’ve subscribed to them after your cluster is created. For the add-ons that don’t have Requires subscription under Status, do the following:
  1. For Pod Identity IAM role for service account, you can either use an existing EKS Pod Identity IAM role or create one using the Create Recommended Role button. This field will only provide options with the appropriate trust policy. If there’s no role to select, then you don’t have an existing role with a matching trust policy. To configure an EKS Pod Identity IAM role for service accounts of the selected add-on, choose Create recommended role. The role creation wizard opens in a separate window. The wizard will automatically populate the role information as follows. For each add-on where you want to create the EKS Pod Identity IAM role, complete the steps in the IAM wizard as follows.
    
    On the Select trusted entity step, the AWS service option for EKS and the use case for EKS - Pod Identity are preselected, and the appropriate trust policy will be automatically populated for the add-on. For example, the role will be created with the appropriate trust policy containing the pods.eks.amazonaws.com IAM Principal as detailed in pod-id-benefits.title. Choose Next.
    
    On the Add permissions step, the appropriate managed policy for the role policy is preselected for the add-on. For example, for the Amazon VPC CNI add-on, the role will be created with the managed policy ` AmazonEKS_CNI_Policy` as detailed in add-ons-vpc-cni.title. Choose Next.
    
    On the Name, review, and create step, in Role name, the default role name is automatically populated for the add-on. For example, for the Amazon VPC CNI add-on, the role will be created with the name AmazonEKSPodIdentityAmazonVPCCNIRole. In Description, the default description is automatically populated with the appropriate description for the add-on. For example, for the Amazon VPC CNI add-on, the role will be created with the description Allows pods running in Amazon EKS cluster to access AWS resources. In Trust policy, view the populated trust policy for the add-on. Choose Create role.
    
    NOTE: Retaining the default role name enables EKS to pre-select the role for add-ons in new clusters or when adding add-ons to existing clusters. You can still override this name and the role will be available for the add-on across your clusters, but the role will need to be manually selected from the drop down.
  2. For add-ons that do not have Requires subscription under Status and where you want to configure roles using IRSA, see the documentation for the add-on that you’re creating to create an IAM policy and attach it to a role. For a list of add-ons, see workloads-add-ons-available-eks.title. Selecting an IAM role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.
  3. Choose Optional configuration settings.
  4. If the add-on requires configuration, enter it in the Configuration values box. To determine whether the add-on requires configuration information, see the documentation for the add-on that you’re creating. For a list of add-ons, see workloads-add-ons-available-eks.title.
  5. Choose one of the available options for Conflict resolution method. If you choose Override for the Conflict resolution method, one or more of the settings for the existing add-on can be overwritten with the Amazon EKS add-on settings. If you don’t enable this option and there’s a conflict with your existing settings, the operation fails. You can use the resulting error message to troubleshoot the conflict. Before choosing this option, make sure that the Amazon EKS add-on doesn’t manage settings that you need to self-manage.
  6. Choose Next.
On the Review and add page, choose Create. After the add-on installation is complete, you see your installed add-ons.
If any of the add-ons that you installed require a subscription, complete the following steps:
1. Choose the Subscribe button in the lower right corner for the add-on. You’re taken to the page for the add-on in the AWS Marketplace. Read the information about the add-on such as its Product Overview and Pricing Information.
2. Select the Continue to Subscribe button on the top right of the add-on page.
3. Read through the Terms and Conditions. If you agree to them, choose Accept Terms. It may take several minutes to process the subscription. While the subscription is processing, the Return to Amazon EKS Console button is grayed out.
4. Once the subscription has finished processing, the Return to Amazon EKS Console button is no longer grayed out. Choose the button to go back to the Amazon EKS console Add-ons tab for your cluster.
5. For the add-on that you subscribed to, choose Remove and reinstall and then choose Reinstall add-on. Installation of the add-on can take several minutes. When Installation is complete, you can configure the add-on.

Create add-on (`AWS` CLI)

You need version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.

Determine which add-ons are available. You can see all available add-ons, their type, and their publisher. You can also see the URL for add-ons that are available through the AWS Marketplace. Replace 1.30 with the version of your cluster.

aws eks describe-addon-versions --kubernetes-version 1.30 \
    --query 'addons[].{MarketplaceProductUrl: marketplaceInformation.productUrl, Name: addonName, Owner: owner Publisher: publisher, Type: type}' --output table

An example output is as follows.

---------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                 DescribeAddonVersions                                                                 |
+---------------------------------------------------------------+-------------------------------+------------------+--------------+---------------------+
|                     MarketplaceProductUrl                     |             Name              |      Owner       |  Publisher   |        Type         |
+---------------------------------------------------------------+-------------------------------+------------------+--------------+---------------------+
|  None                                                         |  aws-ebs-csi-driver           |  aws             |  eks         |  storage            |
|  None                                                         |  coredns                      |  aws             |  eks         |  networking         |
|  None                                                         |  kube-proxy                   |  aws             |  eks         |  networking         |
|  None                                                         |  vpc-cni                      |  aws             |  eks         |  networking         |
|  None                                                         |  adot                         |  aws             |  eks         |  observability      |
| https://aws.amazon.com/marketplace/pp/prodview-brb73nceicv7u |  dynatrace_dynatrace-operator |  aws-marketplace |  dynatrace   |  monitoring         |
| https://aws.amazon.com/marketplace/pp/prodview-uhc2iwi5xysoc |  upbound_universal-crossplane |  aws-marketplace |  upbound     |  infra-management   |
| https://aws.amazon.com/marketplace/pp/prodview-hd2ydsrgqy4li |  teleport_teleport            |  aws-marketplace |  teleport    |  policy-management  |
| https://aws.amazon.com/marketplace/pp/prodview-vgghgqdsplhvc |  factorhouse_kpow             |  aws-marketplace |  factorhouse |  monitoring         |
|  [...]                                                        |  [...]                        |  [...]           |  [...]       |  [...]              |
+---------------------------------------------------------------+-------------------------------+------------------+--------------+---------------------+

Your output might be different. In this example output, there are three different add-ons available of type networking and five add-ons with a publisher of type eks. The add-ons with aws-marketplace in the Owner column may require a subscription before you can install them. You can visit the URL to learn more about the add-on and to subscribe to it.

You can see which versions are available for each add-on. Replace 1.30 with the version of your cluster and replace vpc-cni with the name of an add-on returned in the previous step.

aws eks describe-addon-versions --kubernetes-version 1.30 --addon-name vpc-cni \
    --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

An example output is as follows.

------------------------------------------
|          DescribeAddonVersions         |
+-----------------+----------------------+
| Defaultversion  |       Version        |
+-----------------+----------------------+
|  False          |  v1.12.0-eksbuild.1  |
|  True           |  v1.11.4-eksbuild.1  |
|  False          |  v1.10.4-eksbuild.1  |
|  False          |  v1.9.3-eksbuild.1   |
+-----------------+----------------------+

The version with True in the Defaultversion column is the version that the add-on is created with, by default.

(Optional) Find the configuration options for your chosen add-on by running the following command:

aws eks describe-addon-configuration --addon-name vpc-cni --addon-version v1.12.0-eksbuild.1

{
    "addonName": "vpc-cni",
    "addonVersion": "v1.12.0-eksbuild.1",
    "configurationSchema": "{\"$ref\":\"#/definitions/VpcCni\",\"$schema\":\"http://json-schema.org/draft-06/schema#\",\"definitions\":{\"Cri\":{\"additionalProperties\":false,\"properties\":{\"hostPath\":{\"$ref\":\"#/definitions/HostPath\"}},\"title\":\"Cri\",\"type\":\"object\"},\"Env\":{\"additionalProperties\":false,\"properties\":{\"ADDITIONAL_ENI_TAGS\":{\"type\":\"string\"},\"AWS_VPC_CNI_NODE_PORT_SUPPORT\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_ENI_MTU\":{\"format\":\"integer\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_EXTERNALSNAT\":{\"format\":\"boolean\",\"type\":\"string\"},\"AWS_VPC_K8S_CNI_LOGLEVEL\":{\"type\":\"string\"},\"AWS_VPC_K8S_CNI_LOG_FILE\":{\"type\":\"string\"},\"AWS_VPC_K8S_CNI_RANDOMIZESNAT\":{\"type\":\"string\"},\"AWS_VPC_K8S_CNI_VETHPREFIX\":{\"type\":\"string\"},\"AWS_VPC_K8S_PLUGIN_LOG_FILE\":{\"type\":\"string\"},\"AWS_VPC_K8S_PLUGIN_LOG_LEVEL\":{\"type\":\"string\"},\"DISABLE_INTROSPECTION\":{\"format\":\"boolean\",\"type\":\"string\"},\"DISABLE_METRICS\":{\"format\":\"boolean\",\"type\":\"string\"},\"DISABLE_NETWORK_RESOURCE_PROVISIONING\":{\"format\":\"boolean\",\"type\":\"string\"},\"ENABLE_POD_ENI\":{\"format\":\"boolean\",\"type\":\"string\"},\"ENABLE_PREFIX_DELEGATION\":{\"format\":\"boolean\",\"type\":\"string\"},\"WARM_ENI_TARGET\":{\"format\":\"integer\",\"type\":\"string\"},\"WARM_PREFIX_TARGET\":{\"format\":\"integer\",\"type\":\"string\"}},\"title\":\"Env\",\"type\":\"object\"},\"HostPath\":{\"additionalProperties\":false,\"properties\":{\"path\":{\"type\":\"string\"}},\"title\":\"HostPath\",\"type\":\"object\"},\"Limits\":{\"additionalProperties\":false,\"properties\":{\"cpu\":{\"type\":\"string\"},\"memory\":{\"type\":\"string\"}},\"title\":\"Limits\",\"type\":\"object\"},\"Resources\":{\"additionalProperties\":false,\"properties\":{\"limits\":{\"$ref\":\"#/definitions/Limits\"},\"requests\":{\"$ref\":\"#/definitions/Limits\"}},\"title\":\"Resources\",\"type\":\"object\"},\"VpcCni\":{\"additionalProperties\":false,\"properties\":{\"cri\":{\"$ref\":\"#/definitions/Cri\"},\"env\":{\"$ref\":\"#/definitions/Env\"},\"resources\":{\"$ref\":\"#/definitions/Resources\"}},\"title\":\"VpcCni\",\"type\":\"object\"}}}"
}

The output is a standard JSON schema.

Here is an example of valid configuration values, in JSON format, that works with the schema above.

{
  "resources": {
    "limits": {
      "cpu": "100m"
    }
  }
}

Here is an example of valid configuration values, in YAML format, that works with the schema above.

  resources:
    limits:
      cpu: 100m

Determine if the add-on requires IAM permissions. If so, you need to (1) determine if you want to use EKS Pod Identities or IAM Roles for Service Accounts (IRSA), (2) determine the ARN of the IAM role to use with the add-on, and (3) determine the name of the Kubernetes service account used by the add-on. For more information, see retreive-iam-info.title.
- Amazon EKS suggests using EKS Pod Identities if the add-on supports it. This requires the Pod Identity Agent is installed on your cluster. For more information about using Pod Identities with Add-ons, see add-ons-iam.title.
- If the add-on or your cluster is not setup for EKS Pod Identities, use IRSA. Confirm IRSA is setup on your cluster.
- Review the Amazon EKS Add-ons documentation to determine if the add-on requires IAM permissions and the name of the associated Kubernetes service account.
  1. Create an Amazon EKS add-on. Copy the command that follows to your device. Make the following modifications to the command as needed and then run the modified command:
- Replace my-cluster with the name of your cluster.
- Replace vpc-cni with an add-on name returned in the output of the previous step that you want to create.
- Replace version-number with the version returned in the output of the previous step that you want to use.
- If the add-on doesn’t require IAM permissions, delete <service-account-configuration>.
- Do one of the following:
  - If the add-on (1) requires IAM permissions, and (2) your cluster uses EKS Pod Identities, replace <service-account-configuration> with the following pod identity association. Replace <service-account-name> with the service account name used by the add-on. Replace <role-arn> with the ARN of an IAM role. The role must have the trust policy required by EKS Pod Identities.
    
    --pod-identity-associations 'serviceAccount=<service-account-name>,roleArn=<role-arn>'
  - If the add-on (1) requires IAM permissions, and (2) your cluster uses IRSA, replace <service-account-configuration> with the following IRSA configuration. Replace 111122223333 with your account ID and role-name with the name of an existing IAM role that you’ve created. For instructions on creating the role, see the documentation for the add-on that you’re creating. For a list of add-ons, see workloads-add-ons-available-eks.title. Specifying a service account role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.
    
    --service-account-role-arn region.arn:iam::111122223333:role/role-name
- These example commands overwrites the --configuration-values option of any existing self-managed version of the add-on, if there is one. Replace this with the desired configuration values, such as a string or a file input. If you don’t want to provide configuration values, then delete the --configuration-values option. If you don’t want the AWS CLI to overwrite the configuration of an existing self-managed add-on, remove the --resolve-conflicts OVERWRITE option. If you remove the option, and the Amazon EKS add-on needs to overwrite the configuration of an existing self-managed add-on, then creation of the Amazon EKS add-on fails with an error message to help you resolve the conflict. Before specifying this option, make sure that the Amazon EKS add-on doesn’t manage settings that you need to manage, because those settings are overwritten with this option.
  aws eks create-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version version-number \ <service-account-configuration> --configuration-values '{"resources":{"limits":{"cpu":"100m"}}}' --resolve-conflicts OVERWRITE
  aws eks create-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version version-number \ <service-account-configuration> --configuration-values 'file://example.yaml' --resolve-conflicts OVERWRITE
  For a full list of available options, see create-addon in the Amazon EKS Command Line Reference. If the add-on that you created has aws-marketplace listed in the Owner column of a previous step, then creation may fail, and you may receive an error message similar to the following error.
  { "addon": { "addonName": "addon-name", "clusterName": "my-cluster", "status": "CREATE_FAILED", "addonVersion": "version", "health": { "issues": [ { "code": "AddonSubscriptionNeeded", "message": "You are currently not subscribed to this add-on. To subscribe, visit the AWS Marketplace console, agree to the seller EULA, select the pricing type if required, then re-install the add-on" } ] } } }
  If you receive an error similar to the error in the previous output, visit the URL in the output of a previous step to subscribe to the add-on. Once subscribed, run the create-addon command again.

12.10.8. Update an Amazon EKS add-on

Learn how to update your Amazon EKS add-on to a new version.

Amazon EKS doesn’t automatically update an add-on when new versions are released or after you update your cluster to a new Kubernetes minor version. To update an add-on for an existing cluster, you must initiate the update. After you initiate the update, Amazon EKS updates the add-on for you. Before updating an add-on, review the current documentation for the add-on. For a list of available add-ons, see workloads-add-ons-available-eks.title. If the add-on requires an IAM role, see the details for the specific add-on in Available Amazon EKS add-ons from AWS for details about creating the role.

Prerequisites

Complete the following before you create an add-on:

Check if your add-on requires an IAM role. For more information, see eks-add-ons.title.
Verify that the Amazon EKS add-on version is compatible with your cluster. For more information, see addon-compat.title.

Procedure

You can update an Amazon EKS add-on using eksctl, the consolelong, or the AWS CLI.

Update add-on (eksctl)

Determine the current add-ons and add-on versions installed on your cluster. Replace my-cluster with the name of your cluster.
```
eksctl get addon --cluster my-cluster
```
An example output is as follows.
```
NAME        VERSION              STATUS  ISSUES  IAMROLE  UPDATE AVAILABLE
coredns     v1.8.7-eksbuild.2    ACTIVE  0
kube-proxy  v1.23.7-eksbuild.1   ACTIVE  0                v1.23.8-eksbuild.2
vpc-cni     v1.10.4-eksbuild.1   ACTIVE  0                v1.12.0-eksbuild.1,v1.11.4-eksbuild.1,v1.11.3-eksbuild.1,v1.11.2-eksbuild.1,v1.11.0-eksbuild.1
```
Your output might look different, depending on which add-ons and versions that you have on your cluster. You can see that in the previous example output, two existing add-ons on the cluster have newer versions available in the UPDATE AVAILABLE column.
Update the add-on.
1. Copy the command that follows to your device. Make the following modifications to the command as needed:
  - Replace my-cluster with the name of your cluster.
  - Replace region-code with the AWS Region that your cluster is in.
  - Replace vpc-cni with the name of an add-on returned in the output of the previous step that you want to update.
  - If you want to update to a version earlier than the latest available version, then replace latest with the version number returned in the output of the previous step that you want to use. Some add-ons have recommended versions. For more information, see the documentation for the add-on that you’re updating. For a list of add-ons, see workloads-add-ons-available-eks.title.* If the add-on uses a Kubernetes service account and IAM role, replace 111122223333 with your account ID and role-name with the name of an existing IAM role that you’ve created. For instructions on creating the role, see the documentation for the add-on that you’re creating. For a list of add-ons, see workloads-add-ons-available-eks.title. Specifying a service account role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.
    
    If the add-on doesn’t use a Kubernetes service account and IAM role, delete the serviceAccountRoleARN: region.arniam::111122223333:role/role-name line.
  - The preserve option preserves existing values for the add-on. If you have set custom values for add-on settings, and you don’t use this option, Amazon EKS overwrites your values with its default values. If you use this option, then we recommend that you test any field and value changes on a non-production cluster before updating the add-on on your production cluster. If you change this value to overwrite, all settings are changed to Amazon EKS default values. If you’ve set custom values for any settings, they might be overwritten with Amazon EKS default values. If you change this value to none, Amazon EKS doesn’t change the value of any settings, but the update might fail. If the update fails, you receive an error message to help you resolve the conflict.
    
    cat >update-addon.yaml <<EOF apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: my-cluster region: region-code addons: - name: vpc-cni version: latest serviceAccountRoleARN: region.arniam::111122223333:role/role-name resolveConflicts: preserve EOF
2. Run the modified command to create the update-addon.yaml file.
3. Apply the config file to your cluster.
  eksctl update addon -f update-addon.yaml
For more information about updating add-ons, see Updating addons in the eksctl documentation.

Update add-on (`AWS` Console)

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
Choose the name of the cluster that you want to update the add-on for.
Choose the Add-ons tab.
Choose the add-on that you want to update.
Choose Edit.

On the Configure name of addon page, do the following:

Choose the Version that you’d like to use. The add-on might have a recommended version. For more information, see the documentation for the add-on that you’re updating. For a list of add-ons, see workloads-add-ons-available-eks.title.

You have two options for configuring roles for add-ons: EKS Pod Identities IAM role and IAM roles for service accounts (IRSA). Follow the appropriate step below for your preferred option. If all of the add-ons that you selected have Requires subscription under Status, choose Next. For the add-ons that don’t have Requires subscription under Status, do the following:

For Pod Identity IAM role for service account, you can either use an existing EKS Pod Identity IAM role or create one using the Create Recommended Role button. This field will only provide options with the appropriate trust policy. If there’s no role to select, then you don’t have an existing role with a matching trust policy. To configure an EKS Pod Identity IAM role for service accounts of the selected add-on, choose Create recommended role. The role creation wizard opens in a separate window. The wizard will automatically populate the role information as follows. For each add-on where you want to create the EKS Pod Identity IAM role, complete the steps in the IAM wizard as follows.

On the Select trusted entity step, the AWS service option for EKS and the use case for EKS - Pod Identity are preselected, and the appropriate trust policy will be automatically populated for the add-on. For example, the role will be created with the appropriate trust policy containing the pods.eks.amazonaws.com IAM Principal as detailed in pod-id-benefits.title. Choose Next.
On the Add permissions step, the appropriate managed policy for the role policy is preselected for the add-on. For example, for the Amazon VPC CNI add-on, the role will be created with the managed policy ` AmazonEKS_CNI_Policy` as detailed in add-ons-vpc-cni.title. Choose Next.

On the Name, review, and create step, in Role name, the default role name is automatically populated for the add-on. For example, for the Amazon VPC CNI add-on, the role will be created with the name AmazonEKSPodIdentityAmazonVPCCNIRole. In Description, the default description is automatically populated with the appropriate description for the add-on. For example, for the Amazon VPC CNI add-on, the role will be created with the description Allows pods running in Amazon EKS cluster to access AWS resources. In Trust policy, view the populated trust policy for the add-on. Choose Create role.

Retaining the default role name enables EKS to pre-select the role for add-ons in new clusters or when adding add-ons to existing clusters. You can still override this name and the role will be available for the add-on across your clusters, but the role will need to be manually selected from the drop down.

For add-ons that do not have Requires subscription under Status and where you want to configure roles using IRSA, see the documentation for the add-on that you’re creating to create an IAM policy and attach it to a role. For a list of add-ons, see workloads-add-ons-available-eks.title. Selecting an IAM role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.

Expand the Optional configuration settings.
In Configuration values, enter any add-on specific configuration information. For more information, see the documentation for the add-on that you’re updating. For a list of add-ons, see workloads-add-ons-available-eks.title… For Conflict resolution method, select one of the options. If you have set custom values for add-on settings, we recommend the Preserve option. If you don’t choose this option, Amazon EKS overwrites your values with its default values. If you use this option, then we recommend that you test any field and value changes on a non-production cluster before updating the add-on on your production cluster.

Choose Save changes.

Update add-on (`AWS` CLI)

You need version 2.12.3 or later or version 1.27.160 or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use aws --version | cut -d / -f2 | cut -d ' ' -f1. Package managers such yum, apt-get, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see Installing and Quick configuration with aws configure in the AWS Command Line Interface User Guide. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide.

See a list of installed add-ons. Replace my-cluster with the name of your cluster.

aws eks list-addons --cluster-name my-cluster

An example output is as follows.

{
    "addons": [
        "coredns",
        "kube-proxy",
        "vpc-cni"
    ]
}

View the current version of the add-on that you want to update. Replace my-cluster with your cluster name and vpc-cni with the name of the add-on that you want to update.
```
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni --query "addon.addonVersion" --output text
```
An example output is as follows.
```
v1.10.4-eksbuild.1
```

Determine which versions of the add-on are available for your cluster’s version. Replace 1.30 with your cluster’s version and vpc-cni with the name of the add-on that you want to update.

aws eks describe-addon-versions --kubernetes-version 1.30 --addon-name vpc-cni \
    --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' --output table

An example output is as follows.

------------------------------------------
|          DescribeAddonVersions         |
+-----------------+----------------------+
| Defaultversion  |       Version        |
+-----------------+----------------------+
|  False          |  v1.12.0-eksbuild.1  |
|  True           |  v1.11.4-eksbuild.1  |
|  False          |  v1.10.4-eksbuild.1  |
|  False          |  v1.9.3-eksbuild.1   |
+-----------------+----------------------+

The version with True in the Defaultversion column is the version that the add-on is created with, by default.

Update your add-on. Copy the command that follows to your device. Make the following modifications to the command, as needed, and then run the modified command. For more information about this command, see update-addon in the Amazon EKS Command Line Reference.
- Replace my-cluster with the name of your cluster.
- Replace vpc-cni with the name of the add-on that you want to update that was returned in the output of a previous step.
- Replace version-number with the version returned in the output of the previous step that you want to update to. Some add-ons have recommended versions. For more information, see the documentation for the add-on that you’re updating. For a list of add-ons, see workloads-add-ons-available-eks.title.* If the add-on uses a Kubernetes service account and IAM role, replace 111122223333 with your account ID and role-name with the name of an existing IAM role that you’ve created. For instructions on creating the role, see the documentation for the add-on that you’re creating. For a list of add-ons, see workloads-add-ons-available-eks.title. Specifying a service account role requires that you have an IAM OpenID Connect (OIDC) provider for your cluster. To determine whether you have one for your cluster, or to create one, see enable-iam-roles-for-service-accounts.title.
  
  If the add-on doesn’t use a Kubernetes service account and IAM role, delete the serviceAccountRoleARN: region.arniam::111122223333:role/role-name line.
- The --resolve-conflicts PRESERVE option preserves existing values for the add-on. If you have set custom values for add-on settings, and you don’t use this option, Amazon EKS overwrites your values with its default values. If you use this option, then we recommend that you test any field and value changes on a non-production cluster before updating the add-on on your production cluster. If you change this value to OVERWRITE, all settings are changed to Amazon EKS default values. If you’ve set custom values for any settings, they might be overwritten with Amazon EKS default values. If you change this value to NONE, Amazon EKS doesn’t change the value of any settings, but the update might fail. If the update fails, you receive an error message to help you resolve the conflict.
- If you want to remove all custom configuration then perform the update using the --configuration-values '{}' option. This sets all custom configuration back to the default values. If you don’t want to change your custom configuration, don’t provide the --configuration-values flag. If you want to adjust a custom configuration then replace {} with the new parameters.
  aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version version-number \ --service-account-role-arn region.arniam::111122223333:role/role-name --configuration-values '{}' --resolve-conflicts PRESERVE
Check the status of the update. Replace my-cluster with the name of your cluster and vpc-cni with the name of the add-on you’re updating.
```
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni
```
An example output is as follows.
```
{
    "addon": {
        "addonName": "vpc-cni",
        "clusterName": "my-cluster",
        "status": "UPDATING",
    }
}
```
The update is complete when the status is ACTIVE.

12.10.9. Verify Amazon EKS add-on version compatibility with a cluster

Learn how to verify the Amazon EKS add-on compatibility with your cluster before you create or update an Amazon EKS add-on.

Before you create an Amazon EKS add-on you need to verify that the Amazon EKS add-on version is compatible with your cluster.

Use the describe-addon-verisions API to list the available versions of EKS add-ons, and which Kubernetes versions each addon version supports.

Verify the AWS CLI is installed and working with aws sts get-caller-identity. If this command doesn’t work, learn how to Get started with the AWS CLI.
Determine the name of the add-on you want to retrieve version compatibility information for, such as amazon-cloudwatch-observability.
Determine the Kubernetes version of your cluster, such as 1.31.

Use the AWS CLI to retrieve the addon versions that are compatible with the Kubernetes version of your cluster.

aws eks describe-addon-versions --addon-name amazon-cloudwatch-observability --kubernetes-version 1.31

An example output is as follows.

{
    "addons": [
        {
            "addonName": "amazon-cloudwatch-observability",
            "type": "observability",
            "addonVersions": [
                {
                    "addonVersion": "vX.X.X-eksbuild.X",
                    "architecture": [
                        "amd64",
                        "arm64"
                    ],
                    "computeTypes": [
                        "ec2",
                        "auto",
                        "hybrid"
                    ],
                    "compatibilities": [
                        {
                            "clusterVersion": "1.31",
                            "platformVersions": [
                                "*"
                            ],
                            "defaultVersion": true
                        }
                    ],
                }
            ]
        }
    ]
}

This output shows that addon version vX.X.X-eksbuild.X is compatible with Kubernetes cluster version 1.31.

Add-on compatibility with compute types

The computeTypes field in the describe-addon-versions output indicates an add-on’s compatibility with EKS Auto Mode Managed Nodes or Hybrid Nodes. Add-ons marked auto work with EKS Auto Mode’s cloud-based, AWS-managed infrastructure, while those marked hybrid can run on on-premises nodes connected to the EKS cloud control plane.

For more information, see addon-consider-auto.title.

12.10.10. Remove an Amazon EKS add-on from a cluster

Learn how to remove an Amazon EKS add-on.

You can remove an Amazon EKS add-on from your cluster using eksctl, the consolelong, or the AWS CLI.

When you remove an Amazon EKS add-on from a cluster:

There is no downtime for the functionality that the add-on provides.
If you are using IAM Roles for Service Accounts (IRSA) and the add-on has an IAM role associated with it, the IAM role isn’t removed.
If you are using Pod Identities, any Pod Identity Associations owned by the add-on are removed. If you specify the --preserve option to the AWS CLI, the associations are preserved.
Amazon EKS stops managing settings for the add-on.
The console stops notifying you when new versions are available.
You can’t update the add-on using any AWS tools or APIs.
You can choose to leave the add-on software on your cluster so that you can self-manage it, or you can remove the add-on software from your cluster. You should only remove the add-on software from your cluster if there are no resources on your cluster are dependent on the functionality that the add-on provides.

Prerequisites

Complete the following before you create an add-on:

An existing Amazon EKS cluster. To deploy one, see getting-started.title.
Check if your add-on requires an IAM role. For more information, see
Version 0.199.0 or later of the eksctl command line tool installed on your device or AWS CloudShell. To install or update eksctl, see Installation in the eksctl documentation..

Procedure

You have two options when removing an Amazon EKS add-on.

Preserve add-on software on your cluster – This option removes Amazon EKS management of any settings. It also removes the ability for Amazon EKS to notify you of updates and automatically update the Amazon EKS add-on after you initiate an update. However, it preserves the add-on software on your cluster. This option makes the add-on a self-managed installation, rather than an Amazon EKS add-on. With this option, there’s no downtime for the add-on.
Remove add-on software entirely from your cluster – We recommend that you remove the Amazon EKS add-on from your cluster only if there are no resources on your cluster that are dependent on it.

You can remove an Amazon EKS add-on using eksctl, the consolelong, or the AWS CLI.

Remove add-on (eksctl)

Determine the current add-ons installed on your cluster. Replace my-cluster with the name of your cluster.

eksctl get addon --cluster my-cluster

An example output is as follows.

NAME        VERSION              STATUS  ISSUES  IAMROLE  UPDATE AVAILABLE
coredns     v1.8.7-eksbuild.2    ACTIVE  0
kube-proxy  v1.23.7-eksbuild.1   ACTIVE  0
vpc-cni     v1.10.4-eksbuild.1   ACTIVE  0
[...]

Your output might look different, depending on which add-ons and versions that you have on your cluster.

Remove the add-on. Replace my-cluster with the name of your cluster and name-of-add-on with the name of the add-on returned in the output of the previous step that you want to remove. If you remove the --preserve option, in addition to Amazon EKS no longer managing the add-on, the add-on software is deleted from your cluster.
```
eksctl delete addon --cluster my-cluster --name name-of-addon --preserve
```
For more information about removing add-ons, see Deleting addons in the eksctl documentation.

Remove add-on (`AWS` Console)

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
Choose the name of the cluster that you want to remove the Amazon EKS add-on for.
Choose the Add-ons tab.
Choose the add-on that you want to remove.
Choose Remove.
In the Remove: name of addon confirmation dialog box, do the following:
1. If you want Amazon EKS to stop managing settings for the add-on, select Preserve on cluster. Do this if you want to retain the add-on software on your cluster. This is so that you can manage all of the settings of the add-on on your own.
2. Enter the add-on name.
3. Choose Remove.

Remove add-on (`AWS` CLI)

You need version 0.199.0 or later of the eksctl command line tool installed on your device or AWS CloudShell. To install or update eksctl, see Installation in the eksctl documentation.

See a list of installed add-ons. Replace my-cluster with the name of your cluster.

aws eks list-addons --cluster-name my-cluster

An example output is as follows.

{
    "addons": [
        "coredns",
        "kube-proxy",
        "vpc-cni",
        "name-of-addon"
    ]
}

Remove the installed add-on. Replace my-cluster with the name of your cluster and name-of-add-on with the name of the add-on that you want to remove. Removing --preserve deletes the add-on software from your cluster.
```
aws eks delete-addon --cluster-name my-cluster --addon-name name-of-addon --preserve
```
The abbreviated example output is as follows.
```
{
    "addon": {
        "addonName": "name-of-add-on",
        "clusterName": "my-cluster",
        "status": "DELETING",
    }
}
```

Check the status of the removal. Replace my-cluster with the name of your cluster and name-of-addon with the name of the add-on that you’re removing.

aws eks describe-addon --cluster-name my-cluster --addon-name name-of-addon

After the add-on is removed, the example output is as follows.

An error occurred (ResourceNotFoundException) when calling the DescribeAddon operation: No addon: name-of-addon found in cluster: my-cluster

12.10.11. IAM roles for Amazon EKS add-ons

Retrieve IAM information about an Amazon EKS add-on

Learn how to determine the role and policy to use for an Amazon EKS add-on.

Before you create an add-on, use the AWS CLI to determine:

If the add-on requires IAM permissions
The suggested IAM policy to use

Procedure

Determine the name of the add-on you want to install, and the Kubernetes version of your cluster. For more information about add-ons, see eks-add-ons.title.

Use the AWS CLI to determine if the add-on requires IAM permissions.

aws eks describe-addon-versions \
--addon-name <addon-name> \
--kubernetes-version <kubernetes-version>

For example:

aws eks describe-addon-versions \
--addon-name aws-ebs-csi-driver \
--kubernetes-version 1.30

Review the following sample output. Note that requiresIamPermissions is true, and the default add-on version. You need to specify the add-on version when retrieving the recommended IAM policy.

{
    "addons": [
        {
            "addonName": "aws-ebs-csi-driver",
            "type": "storage",
            "addonVersions": [
                {
                    "addonVersion": "v1.31.0-eksbuild.1",
                    "architecture": [
                        "amd64",
                        "arm64"
                    ],
                    "compatibilities": [
                        {
                            "clusterVersion": "1.30",
                            "platformVersions": [
                                "*"
                            ],
                            "defaultVersion": true
                        }
                    ],
                    "requiresConfiguration": false,
                    "requiresIamPermissions": true
                },
[...]

If the add-on requires IAM permissions, use the AWS CLI to retrieve a recommended IAM policy.

aws eks describe-addon-configuration \
--query podIdentityConfiguration \
--addon-name <addon-name> \
--addon-version <addon-version>

For example:

aws eks describe-addon-configuration \
--query podIdentityConfiguration \
--addon-name aws-ebs-csi-driver \
--addon-version v1.31.0-eksbuild.1

Review the following output. Note the recommendedManagedPolicies.

[
    {
        "serviceAccount": "ebs-csi-controller-sa",
        "recommendedManagedPolicies": [
            "region.arniam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
        ]
    }
]

Create an IAM role and attach the recommended Managed Policy. Alternatively, review the managed policy and scope down the permissions as appropriate. For more information see pod-id-association-create.title.

Pod Identity Support Reference

The following table indicates if certain Amazon EKS add-ons support EKS Pod Identity.

Add-on Name Pod Identity Support Minimum Version Required

Amazon EBS CSI Driver

Yes

v1.26.0-eksbuild.1

Amazon VPC CNI

Yes

v1.15.5-eksbuild.1

Amazon EFS CSI Driver

Yes

v2.0.5-eksbuild.1

AWS Distro for OpenTelemetry

Yes

v0.94.1-eksbuild.1

Mountpoint for Amazon S3 CSI Driver

N/A

Amazon CloudWatch Observability agent

Yes

v3.1.0-eksbuild.1

This table was last updated on October 28, 2024.

Use Pod Identities to assign an IAM role to an Amazon EKS add-on

Learn how to use a Pod Identity to assign a role for an Amazon EKS add-on.

Certain Amazon EKS add-ons need IAM roles and permissions. Before you add update an Amazon EKS add-on to use a Pod Identity association, verify the role and policy to use. For more information, see retreive-iam-info.title.

Determine:
- cluster-name – The name of the cluster to install the add-on onto.
- addon-name – The name of the add-on to install.
- service-account-name – The name of the Kubernetes Service Account used by the add-on.
- iam-role-arn – The ARN of an IAM role with sufficient permissions for the add-on. The role must have the required trust policy for EKS Pod Identity. For more information see pod-id-association-create.title.

Update the add-on using the AWS CLI. You can also specify Pod Identity associations when creating an add-on, using the same --pod-identity-assocations syntax. Note that when you specify pod identity associations while updating an add-on, all previous pod identity associations are overwritten.

aws eks update-addon --cluster-name <cluster-name> \
--addon-name <addon-name> \
--pod-identity-associations 'serviceAccount=<service-account-name>,roleArn=<role-arn>'

For example:

aws eks update-addon --cluster-name mycluster \
--addon-name aws-ebs-csi-driver \
--pod-identity-associations 'serviceAccount=ebs-csi-controller-sa,roleArn=region.arniam::123456789012:role/StorageDriver'

Validate the Pod Identity association was created:

aws eks list-pod-identity-associations --cluster-name <cluster-name>

If successful, you should see output similar to the following. Note the OwnerARN of the EKS add-on.

{
    "associations": [
        {
            "clusterName": "mycluster",
            "namespace": "kube-system",
            "serviceAccount": "ebs-csi-controller-sa",
            "associationArn": "region.arneks:us-west-2:123456789012:podidentityassociation/mycluster/a-4wvljrezsukshq1bv",
            "associationId": "a-4wvljrezsukshq1bv",
            "ownerArn": "region.arneks:us-west-2:123456789012:addon/mycluster/aws-ebs-csi-driver/9cc7ce8c-2e15-b0a7-f311-426691cd8546"
        }
    ]
}

Remove Pod Identity associations from an Amazon EKS add-on

Learn how to remove a Pod Identity from an Amazon EKS add-on.

Remove the Pod Identity associations from an Amazon EKS add-on.

Determine:
- cluster-name - The name of the EKS cluster to install the add-on onto.
- addon-name - The name of the Amazon EKS add-on to install.

Update the addon to specify an empty array of pod identity associations.

aws eks update-addon --cluster-name <cluster-name> \
--addon-name <addon-name> \
--pod-identity-associations "[]"

Troubleshoot Pod Identities for EKS add-ons

Learn how to troubleshoot Pod Identities for EKS add-ons.

If your add-ons are encountering errors while attempting AWS API, SDK, or CLI operations, confirm the following:

The Pod Identity Agent is installed in your cluster.
- For information about how to install the Pod Identity Agent, see pod-id-agent-setup.title.
The Add-on has a valid Pod Identity association.
- Use the AWS CLI to retrieve the associations for the service account name used by the add-on.
  aws eks list-pod-identity-associations --cluster-name <cluster-name>
The IAM role has the required trust policy for Pod Identities.
- Use the AWS CLI to retrieve the trust policy for an add-on.
  aws iam get-role --role-name <role-name> --query Role.AssumeRolePolicyDocument
The IAM role has the necessary permissions for the add-on.
- Use AWS CloudTrail to review AccessDenied or UnauthorizedOperation events .
The service account name in the pod identity association matches the service account name used by the add-on.
- For information about the available add-ons, see workloads-add-ons-available-eks.title.

Grant an Amazon EKS add-on permission to call AWS APIs. Create a Pod Identity Association for an Amazon EKS add-on.

Certain Amazon EKS add-ons need IAM roles and permissions to call AWS APIs. For example, the Amazon VPC CNI add-on calls certain AWS APIs to configure networking resources in your account. These add-ons need to be granted permission using IAM. More specifically, the service account of the pod running the add-on needs to be associated with an IAM role with a specific IAM policy.

The recommended way to grant AWS permissions to cluster workloads is using the Amazon EKS feature Pod Identities. You can use a Pod Identity Association to map the service account of an add-on to an IAM role. If a pod uses a service account that has an association, Amazon EKS sets environment variables in the containers of the pod. The environment variables configure the AWS SDKs, including the AWS CLI, to use the EKS Pod Identity credentials. For more information, see pod-identities.title

Amazon EKS add-ons can help manage the life cycle of pod identity associations corresponding to the add-on. For example, you can create or update an Amazon EKS add-on and the necessary pod identity association in a single API call. Amazon EKS also provides an API for retrieving suggested IAM policies.

Confirm that Amazon EKS pod identity agent is setup on your cluster.
Determine if the add-on you want to install requires IAM permissions using the describe-addon-versions AWS CLI operation. If the requiresIamPermissions flag is true, then you should use the describe-addon-configurations operation to determine the permissions needed by the addon. The response includes a list of suggested managed IAM policies.
Retrieve the name of the Kubernetes Service Account and the IAM policy using the describe-addon-configuration CLI operation. Evaluate the scope of the suggested policy against your security requirements.
Create an IAM role using the suggested permissions policy, and the trust policy required by Pod Identity. For more information, see pod-id-association-create.title.
Create or update an Amazon EKS add-on using the CLI. Specify at least one pod identity association. A pod identity association is the name of a Kubernetes service account, and the ARN of the IAM role.
- Pod identity associations created using the add-on APIs are owned by the respective add-on. If you delete the add-on, the pod identity association is also deleted. You can prevent this cascading delete by using the preserve option when deleting an addon using the AWS CLI or API. You also can directly update or delete the pod identity association if necessary. Add-ons can’t assume ownership of existing pod identity associations. You must delete the existing association and re-create it using an add-on create or update operation.
- Amazon EKS recommends using pod identity associations to manage IAM permissions for add-ons. The previous method, IAM roles for service accounts (IRSA), is still supported. You can specify both an IRSA serviceAccountRoleArn and a pod identity association for an add-on. If the EKS pod identity agent is installed on the cluster, the serviceAccountRoleArn will be ignored, and EKS will use the provided pod identity association. If Pod Identity is not enabled, the serviceAccountRoleArn will be used.
- If you update the pod identity associations for an existing add-on, Amazon EKS initiates a rolling restart of the add-on pods.

12.10.12. Determine fields you can customize for Amazon EKS add-ons

Learn how to manage Amazon EKS add-on configurations using Kubernetes field management to customize settings without overwriting Amazon EKS managed fields.

Amazon EKS add-ons are installed to your cluster using standard, best practice configurations. For more information about adding an Amazon EKS add-on to your cluster, see eks-add-ons.title.

You may want to customize the configuration of an Amazon EKS add-on to enable advanced features. Amazon EKS uses the Kubernetes server-side apply feature to enable management of an add-on by Amazon EKS without overwriting your configuration for settings that aren’t managed by Amazon EKS. For more information, see Server-Side Apply in the Kubernetes documentation. To achieve this, Amazon EKS manages a minimum set of fields for every add-on that it installs. You can modify all fields that aren’t managed by Amazon EKS, or another Kubernetes control plane process such as kube-controller-manager, without issue.

Modifying a field managed by Amazon EKS prevents Amazon EKS from managing the add-on and may result in your changes being overwritten when an add-on is updated.

Field management syntax

When you view details for a Kubernetes object, both managed and unmanaged fields are returned in the output. Managed fields can be either of the following types:

Fully managed – All keys for the field are managed by Amazon EKS. Modifications to any value causes a conflict.
Partially managed – Some keys for the field are managed by Amazon EKS. Only modifications to the keys explicitly managed by Amazon EKS cause a conflict.

Both types of fields are tagged with manager: eks.

Each key is either a . representing the field itself, which always maps to an empty set, or a string that represents a sub-field or item. The output for field management consists of the following types of declarations:

f:name, where name is the name of a field in a list.
k:keys, where keys is a map of a list item’s fields.
v:value, where value is the exact JSON formatted value of a list item.
i:index, where index is position of an item in the list.

The following portions of output for the CoreDNS add-on illustrate the previous declarations:

Fully managed fields – If a managed field has an f: (field) specified, but no k: (key), then the entire field is managed. Modifications to any values in this field cause a conflict.

In the following output, you can see that the container named coredns is managed by eks. The args, image, and imagePullPolicy sub-fields are also managed by eks. Modifications to any values in these fields cause a conflict.
```
[...]
f:containers:
  k:{"name":"coredns"}:
  .: {}
  f:args: {}
  f:image: {}
  f:imagePullPolicy: {}
[...]
manager: eks
[...]
```
Partially managed fields – If a managed key has a value specified, the declared keys are managed for that field. Modifying the specified keys cause a conflict.

In the following output, you can see that eks manages the config-volume and tmp volumes set with the name key.
```
[...]
f:volumes:
  k:{"name":"config-volume"}:
    .: {}
    f:configMap:
      f:items: {}
      f:name: {}
    f:name: {}
  k:{"name":"tmp"}:
    .: {}
    f:name: {}
[...]
manager: eks
[...]
```
Adding keys to partially managed fields – If only a specific key value is managed, you can safely add additional keys, such as arguments, to a field without causing a conflict. If you add additional keys, make sure that the field isn’t managed first. Adding or modifying any value that is managed causes a conflict.

In the following output, you can see that both the name key and name field are managed. Adding or modifying any container name causes a conflict with this managed key.
```
[...]
f:containers:
  k:{"name":"coredns"}:
[...]
    f:name: {}
[...]
manager: eks
[...]
```

Procedure

You can use kubectl to see which fields are managed by Amazon EKS for any Amazon EKS add-on.

You can modify all fields that aren’t managed by Amazon EKS, or another Kubernetes control plane process such as kube-controller-manager, without issue.

Determine which add-on that you want to examine. To see all of the deployments and DaemonSets deployed to your cluster, see view-kubernetes-resources.title.

View the managed fields for an add-on by running the following command:

kubectl get type/add-on-name -n add-on-namespace -o yaml

For example, you can see the managed fields for the CoreDNS add-on with the following command.

kubectl get deployment/coredns -n kube-system -o yaml

Field management is listed in the following section in the returned output.

[...]
managedFields:
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
[...]

If you don’t see managedFields in the output, add --show-managed-fields to the command and run it again. The version of kubectl that you’re using determines whether managed fields are returned by default.

Next steps

Customize the fields not owned by AWS for you add-on.

12.11. Validate container image signatures during deployment

Learn how to verify signed container images during deployment on Amazon EKS using admission controllers like Gatekeeper with Ratify or Kyverno configured with AWS Signer plugins for validating image signatures.

If you use AWS Signer and want to verify signed container images at the time of deployment, you can use one of the following solutions:

Gatekeeper and Ratify – Use Gatekeeper as the admission controller and Ratify configured with an AWS Signer plugin as a web hook for validating signatures.
Kyverno – A Kubernetes policy engine configured with an AWS Signer plugin for validating signatures.

Before verifying container image signatures, configure the Notation trust store and trust policy, as required by your selected admission controller.

13. Organize and monitor cluster resources

This chapter includes the following topics to help you manage your cluster. You can also view information about your Kubernetes resources with the consolelong.

The Kubernetes Dashboard is a general purpose, web-based UI for Kubernetes clusters. It allows users to manage applications running in the cluster and troubleshoot them, as well as manage the cluster itself. For more information, see The Kubernetes Dashboard GitHub repository.
metrics-server.title – The Kubernetes Metrics Server is an aggregator of resource usage data in your cluster. It isn’t deployed by default in your cluster, but is used by Kubernetes add-ons, such as the Kubernetes Dashboard and horizontal-pod-autoscaler.title. In this topic you learn how to install the Metrics Server.
helm.title – The Helm package manager for Kubernetes helps you install and manage applications on your Kubernetes cluster. This topic helps you install and run the Helm binaries so that you can install and manage charts using the Helm CLI on your local computer.
eks-using-tags.title – To help you manage your Amazon EKS resources, you can assign your own metadata to each resource in the form of tags. This topic describes tags and shows you how to create them.
service-quotas.title – Your AWS account has default quotas, formerly referred to as limits, for each AWS service. Learn about the quotas for Amazon EKS and how to increase them.

13.1. Monitor and optimize Amazon EKS cluster costs

Learn how to monitor and optimize costs for your Amazon EKS clusters using AWS Billing split cost allocation data or Kubecost, a Kubernetes-native cost monitoring tool integrated with AWS.

Cost monitoring is an essential aspect of managing your Kubernetes clusters on Amazon EKS. By gaining visibility into your cluster costs, you can optimize resource utilization, set budgets, and make data-driven decisions about your deployments. Amazon EKS provides two cost monitoring solutions, each with its own unique advantages, to help you track and allocate your costs effectively:

AWS Billing split cost allocation data for Amazon EKS — This native feature integrates seamlessly with the AWS Billing Console, allowing you to analyze and allocate costs using the same familiar interface and workflows you use for other AWS services. With split cost allocation, you can gain insights into your Kubernetes costs directly alongside your other AWS spend, making it easier to optimize costs holistically across your AWS environment. You can also leverage existing AWS Billing features like Cost Categories and Cost Anomaly Detection to further enhance your cost management capabilities. For more information, see Understanding split cost allocation data in the AWS Billing User Guide.

Kubecost — Amazon EKS supports Kubecost, a Kubernetes cost monitoring tool. Kubecost offers a feature-rich, Kubernetes-native approach to cost monitoring, providing granular cost breakdowns by Kubernetes resources, cost optimization recommendations, and out-of-the-box dashboards and reports. Kubecost also retrieves accurate pricing data by integrating with the AWS Cost and Usage Report, ensuring you get a precise view of your Amazon EKS costs. Learn how to Install Kubecost.

13.1.1. View costs by pod in `AWS` billing with split cost allocation

Cost monitoring using AWS split cost allocation data for Amazon EKS

You can use AWS split cost allocation data for Amazon EKS to get granular cost visibility for your Amazon EKS clusters. This enables you to analyze, optimize, and chargeback cost and usage for your Kubernetes applications. You allocate application costs to individual business units and teams based on Amazon EC2 CPU and memory resources consumed by your Kubernetes application. Split cost allocation data for Amazon EKS gives visibility into cost per Pod, and enables you to aggregate the cost data per Pod using namespace, cluster, and other Kubernetes primitives. The following are examples of Kubernetes primitives that you can use to analyze Amazon EKS cost allocation data.

Cluster name
Deployment
Namespace
Node
Workload Name
Workload Type

For more information about using split cost allocation data, see Understanding split cost allocation data in the AWS Billing User Guide.

Set up Cost and Usage Reports

You can turn on Split Cost Allocation Data for EKS in the Cost Management Console, AWS Command Line Interface, or the AWS SDKs.

Use the following for Split Cost Allocation Data:

Opt in to Split Cost Allocation Data. For more information, see Enabling split cost allocation data in the AWS Cost and Usage Report User Guide.
Include the data in a new or existing report.
View the report. You can use the Billing and Cost Management console or view the report files in Amazon Simple Storage Service.

13.1.2. Install Kubecost and access dashboard

Amazon EKS supports Kubecost, which you can use to monitor your costs broken down by Kubernetes resources including Pods, nodes, namespaces, and labels. This topic covers installing Kubecost, and accessing the Kubecost dashboard.

Amazon EKS provides an AWS optimized bundle of Kubecost for cluster cost visibility. You can use your existing AWS support agreements to obtain support. For more information about the available versions of Kubecost, see cost-monitoring-kubecost-bundles.title.

As a Kubernetes platform administrator and finance leader, you can use Kubecost to visualize a breakdown of Amazon EKS charges, allocate costs, and charge back organizational units such as application teams. You can provide your internal teams and business units with transparent and accurate cost data based on their actual AWS bill. Moreover, you can also get customized recommendations for cost optimization based on their infrastructure environment and usage patterns within their clusters.

Kubecost v2 introduces several major new features. Learn more about Kubecost v2.

For more information about Kubecost, see the Kubecost documentation.

Install Kubecost using Amazon EKS Add-ons

Install Kubecost as an Amazon EKS Add-on and benefit from additional features at no additional cost with the Amazon EKS optimized Kubecost bundle. For more information, see kubecost-v2.title.

Amazon EKS Add-ons reduce the complexity of upgrading Kubecost, and managing licenses. EKS Add-ons are integrated with the AWS marketplace.

View Kubecost in the AWS Marketplace console and subscribe.
Determine the name of your cluster, and the region. Verify you are logged into the AWS CLI with sufficient permissions to manage EKS.

Create the Kubecost addon.

aws eks create-addon --addon-name kubecost_kubecost --cluster-name $YOUR_CLUSTER_NAME --region $AWS_REGION

Learn how to remove an EKS Add-on, such as Kubecost.

Install Kubecost using Helm

An existing Amazon EKS cluster. To deploy one, see getting-started.title. The cluster must have Amazon EC2 nodes because you can’t run Kubecost on Fargate nodes.
The kubectl command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.29, you can use kubectl version 1.28, 1.29, or 1.30 with it. To install or upgrade kubectl, see install-kubectl.title.
Helm version 3.9.0 or later configured on your device or AWS CloudShell. To install or update Helm, see helm.title.
If your cluster is version 1.23 or later, you must have the Store Kubernetes volumes with Amazon EBS installed on your cluster.
1. Determine the version of Kubecost to install. You can see the available versions at kubecost/cost-analyzer in the Amazon ECR Public Gallery. For more information about the compatibility of Kubecost versions and Amazon EKS, see the Environment Requirements in the Kubecost documentation.
2. Install Kubecost with the following command. Replace kubecost-version with the value retrieved from ECR, such as 1.108.1.
  helm upgrade -i kubecost oci://public.ecr.aws/kubecost/cost-analyzer --version kubecost-version \ --namespace kubecost --create-namespace \ -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/develop/cost-analyzer/values-eks-cost-monitoring.yaml
  Kubecost releases new versions regularly. You can update your version using helm upgrade. By default, the installation includes a local Prometheus server and kube-state-metrics. You can customize your deployment to use Amazon Managed Service for Prometheus by following the documentation in Integrating with Amazon EKS cost monitoring. For a list of all other settings that you can configure, see the sample configuration file on GitHub.
  
  You can remove Kubecost from your cluster with the following commands.
  helm uninstall kubecost --namespace kubecost kubectl delete ns kubecost

Access Kubecost Dashboard

Make sure the required Pods are running.

kubectl get pods -n kubecost

An example output is as follows.

NAME                                          READY   STATUS    RESTARTS   AGE
kubecost-cost-analyzer-b9788c99f-5vj5b        2/2     Running   0          3h27m
kubecost-kube-state-metrics-99bb8c55b-bn2br   1/1     Running   0          3h27m
kubecost-prometheus-server-7d9967bfc8-9c8p7   2/2     Running   0          3h27m

On your device, enable port-forwarding to expose the Kubecost dashboard.
```
kubectl port-forward --namespace kubecost deployment/kubecost-cost-analyzer 9090
```
Alternatively, you can use the AWS Load Balancer Controller to expose Kubecost and use Amazon Cognito for authentication, authorization, and user management. For more information, see How to use Application Load Balancer and Amazon Cognito to authenticate users for your Kubernetes web apps.
On the same device that you completed the previous step on, open a web browser and enter the following address.
```
http://localhost:9090
```
You see the Kubecost Overview page in your browser. It might take 5–10 minutes for Kubecost to gather metrics. You can see your Amazon EKS spend, including cumulative cluster costs, associated Kubernetes asset costs, and monthly aggregated spend.
To track costs at a cluster level, tag your Amazon EKS resources for billing. For more information, see tag-resources-for-billing.title.
- Cost allocation – View monthly Amazon EKS costs and cumulative costs for each of your namespaces and other dimensions over the past seven days. This is helpful for understanding which parts of your application are contributing to Amazon EKS spend.
- Assets – View the costs of the AWS infrastructure assets that are associated with your Amazon EKS resources.

13.1.3. Learn more about Kubecost

Amazon EKS provides an AWS optimized bundle of Kubecost for cluster cost visibility. Amazon EKS supports Kubecost, which you can use to monitor your costs broken down by Kubernetes resources including Pods, nodes, namespaces, and labels.

This topic covers the available versions of Kubecost, and the differences between the available tiers. EKS supports Kubecost Version 1 and Version 2. Each version is available in different tiers. You can use Amazon EKS optimized Kubecost custom bundle for your EKS clusters at no additional cost. You may be charged for use of associated AWS services, such as Amazon Managed Service for Prometheus. Also, you can use your existing AWS support agreements to obtain support.

What is the difference between the custom bundle of Kubecost and the free version of Kubecost (also known as OpenCost)?

AWS and Kubecost collaborated to offer a customized version of Kubecost. This version includes a subset of commercial features at no additional charge. See the tables below for features that are included with in the custom bundle of Kubecost.

Kubecost v2

What is the difference between Kubecost v1 and v2?

Kubecost 2.0 is a major upgrade from previous versions and includes major new features including a brand new API Backend. Note the Allocation and Assets APIs are fully backwards compatible. Please review the Kubecost documentation to ensure a smooth transition. For the full list of enhancements, please see the Kubecost release notes

Review the Kubecost documentation before upgrading. Upgrading may impact report availability.

Core features comparison:

Feature Kubecost free tier 2.0 Amazon EKS optimized Kubecost bundle 2.0 Kubecost Enterprise 2.0

Cluster cost visibility

Single clusters up to 250 cores

Unified multi-cluster without core limits when integrated with Amazon Managed Service for Prometheus

Unified and unlimited number of clusters across unlimited numbers of environments (i.e. multi-cloud)

Deployment

User hosted

User hosted, Kubecost hosted (dedicated tenant), SaaS

Databases supported

Local Prometheus

Amazon Managed Service for Prometheus or Local Prometheus

Any prometheus flavor and custom databases

Database retention support (raw metrics)

15 days

Unlimited historical data

Kubecost API and UI retention (ETL)

15 days

Unlimited

Hybrid cloud visibility

Amazon EKS and Amazon EKS Anywhere clusters

Multi-cloud and hybrid cloud

Alerts and recurring reports

Only supported on the primary cluster, limited to 250 cores

Efficiency alerts, budget alerts, spend change alerts, and more supported across all clusters

Saved reports

Reports using 15 days of metrics

Reports using unlimited historical data and metrics

Cloud billing integration

Only supported on the primary cluster, limited to 250 cores

Custom pricing support for AWS (including multiple clusters and multiple accounts)

Custom pricing support for any cloud

Savings recommendations

Only supported on the primary cluster, limited to 250 cores

Primary cluster insights, but there is no 250 core limit

Multi-cluster insights

Governance: Audits

Audit historical cost events

Single sign-on (SSO) support

Amazon Cognito supported

Okta, Auth0, PingID, KeyCloak, and anything else custom

Role-based access control (RBAC) with SAML 2.0

Okta, Auth0, PingID, KeyCloak, and anything else custom

Enterprise training and onboarding

Full-service training and FinOps onboarding

Teams

Yes

New Features:

The following features have metric limits:

Kubecost Aggregator
Network Monitoring
Kubecost Actions
Collections
Anomaly detection
Container Request Right-Sizing
Kubecost Forecasting
Autocomplete for filtering and aggregation

Metric limits:

Metric

Kubecost Free Tier 2.0

Amazon EKS Optimized Kubecost Custom Bundle 2.0

Kubecost Enterprise 2.0

Cluster size

Limited to 250 cores

Unlimited

Metric retention

15 days

Unlimited

Multi-cluster support

Not available

Available

Core limits

250 cores per cluster

No core limits

Kubecost v1

Feature Kubecost free tier Amazon EKS optimized Kubecost custom bundle Kubecost Enterprise

Deployment

User hosted

User hosted or Kubecost hosted (SaaS)

Number of clusters supported

Unlimited

Databases supported

Local Prometheus

Local Prometheus or Amazon Managed Service for Prometheus

Prometheus, Amazon Managed Service for Prometheus, Cortex, or Thanos

Database retention support

15 days

Unlimited historical data

Kubecost API retention (ETL)

15 days

Unlimited historical data

Cluster cost visibility

Single clusters

Unified multi-cluster

Hybrid cloud visibility

Amazon EKS and Amazon EKS Anywhere clusters

Multi-cloud and hybrid-cloud support

Alerts and recurring reports

Efficiency alerts, budget alerts, spend change alerts, and more supported

Saved reports

Reports using 15 days data

Reports using unlimited historical data

Cloud billing integration

Required for each individual cluster

Custom pricing support for AWS (including multiple clusters and multiple accounts)

Savings recommendations

Single cluster insights

Multi-cluster insights

Governance: Audits

Audit historical cost events

Single sign-on (SSO) support

Amazon Cognito supported

Okta, Auth0, PingID, KeyCloak

Role-based access control (RBAC) with SAML 2.0

Okta, Auth0, PingID, Keycloak

Enterprise training and onboarding

Full-service training and FinOps onboarding

Frequently asked questions

See the following common questions and answers about using Kubecost with Amazon EKS.

What is the Kubecost API retention (ETL) feature?

The Kubecost ETL feature aggregates and organizes metrics to surface cost visibility at various levels of granularity (such as namespace-level, pod-level, and deployment-level). For the custom Kubecost bundle, customers get data and insights from metrics for the last 15 days.

What is the alerts and recurring reports feature? What alerts and reports does it include?

Kubecost alerts allow teams to receive updates on real-time Kubernetes spend as well as cloud spend. Recurring reports enable teams to receive customized views of historical Kubernetes and cloud spend. Both are configurable using the Kubecost UI or Helm values. They support email, Slack, and Microsoft Teams.

What do saved reports include?

Kubecost saved reports are predefined views of cost and efficiency metrics. They include cost by cluster, namespace, label, and more.

What is cloud billing integration?

Integration with AWS billing APIs allows Kubecost to display out-of-cluster costs (such as Amazon S3). Additionally, it allows Kubecost to reconcile Kubecost’s in-cluster predictions with actual billing data to account for spot usage, savings plans, and enterprise discounts.

What do savings recommendations include?

Kubecost provides insights and automation to help users optimize their Kubernetes infrastructure and spend.

Is there a charge for this functionality?

No. You can use this version of Kubecost at no additional charge. If you want additional Kubecost capabilities that aren’t included in this bundle, you can buy an enterprise license of Kubecost through the AWS Marketplace, or from Kubecost directly.

Is support available?

Yes. You can open a support case with the AWS Support team at Contact AWS.

Do I need a license to use Kubecost features provided by the Amazon EKS integration?

No.

Can I integrate Kubecost with AWS Cost and Usage Report for more accurate reporting?

Yes. You can configure Kubecost to ingest data from AWS Cost and Usage Report to get accurate cost visibility, including discounts, Spot pricing, reserved instance pricing, and others. For more information, see AWS Cloud Billing Integration in the Kubecost documentation.

Does this version support cost management of self-managed Kubernetes clusters on Amazon EC2?

No. This version is only compatible with Amazon EKS clusters.

Can Kubecost track costs for Amazon EKS on AWS Fargate?

Kubecost provides best effort to show cluster cost visibility for Amazon EKS on Fargate, but with lower accuracy than with Amazon EKS on Amazon EC2. This is primarily due to the difference in how you’re billed for your usage. With Amazon EKS on Fargate, you’re billed for consumed resources. With Amazon EKS on Amazon EC2 nodes, you’re billed for provisioned resources. Kubecost calculates the cost of an Amazon EC2 node based on the node specification, which includes CPU, RAM, and ephemeral storage. With Fargate, costs are calculated based on the requested resources for the Fargate Pods.

How can I get updates and new versions of Kubecost?

You can upgrade your Kubecost version using standard Helm upgrade procedures. The latest versions are in the Amazon ECR Public Gallery.

Is the *kubectl-cost CLI supported? How do I install it?*

Yes. Kubectl-cost is an open source tool by Kubecost (Apache 2.0 License) that provides CLI access to Kubernetes cost allocation metrics. To install kubectl-cost, see Installation on GitHub.

Is the Kubecost user interface supported? How do I access it?

Kubecost provides a web dashboard that you can access through kubectl port forwarding, an ingress, or a load balancer. You can also use the AWS Load Balancer Controller to expose Kubecost and use Amazon Cognito for authentication, authorization, and user management. For more information, see How to use Application Load Balancer and Amazon Cognito to authenticate users for your Kubernetes web apps on the AWS blog.

Is Amazon EKS Anywhere supported?

No.

Additional `Kubecost` Features

The following features are available in both Kubecost v1 and v2.
Export cost metrics – Amazon EKS optimized cost monitoring is deployed with Kubecost and Prometheus, which is an open-source monitoring system and time series database. Kubecost reads metric from Prometheus and then performs cost allocation calculations and writes the metrics back to Prometheus. The Kubecost front-end reads metrics from Prometheus and shows them on the Kubecost user interface. The architecture is illustrated in the following diagram.

With Prometheus pre-installed, you can write queries to ingest Kubecost data into your current business intelligence system for further analysis. You can also use it as a data source for your current Grafana dashboard to display Amazon EKS cluster costs that your internal teams are familiar with. To learn more about how to write Prometheus queries, see the https://github.com/opencost/opencost/blob/develop/PROMETHEUS.mdreadme file on GitHub or use the example Grafana JSON models in the Kubecost Github repository as references.
AWS Cost and Usage Report integration – To perform cost allocation calculations for your Amazon EKS cluster, Kubecost retrieves the public pricing information of AWS services and AWS resources from the AWS Price List API. You can also integrate Kubecost with AWS Cost and Usage Report:: to enhance the accuracy of the pricing information specific to your AWS account. This information includes enterprise discount programs, reserved instance usage, savings plans, and spot usage. To learn more about how the AWS Cost and Usage Report integration works, see AWS Cloud Billing Integration in the Kubecost documentation.

13.2. View resource usage with the `Kubernetes` `Metrics Server`

Use the Kubernetes Metrics Server to view resource usage data on your Amazon EKS cluster for autoscaling and monitoring.

The Kubernetes Metrics Server is an aggregator of resource usage data in your cluster, and it isn’t deployed by default in Amazon EKS clusters. For more information, see Kubernetes Metrics Server on GitHub. The Metrics Server is commonly used by other Kubernetes add ons, such as the Scale pod deployments with Horizontal Pod Autoscaler or the Kubernetes Dashboard. For more information, see Resource metrics pipeline in the Kubernetes documentation. This topic explains how to deploy the Kubernetes Metrics Server on your Amazon EKS cluster.

The metrics are meant for point-in-time analysis and aren’t an accurate source for historical analysis. They can’t be used as a monitoring solution or for other non-auto scaling purposes. For information about monitoring tools, see eks-observe.title.

13.2.1. Deploy as community add-on with Amazon EKS Add-ons

New: You can now deploy Metrics Server as a community add-on using the AWS console or Amazon EKS APIs.

Deploy with `AWS` console

Open your EKS cluster in the AWS console
From the "Add-ons" tab, select Get More Add-ons.
From the "Community add-ons" section, select Metrics Server and then Next
EKS determines the appropriate version of the add-on for your cluster. You can change the version using the Version dropdown menu.
Select Next and then Create to install the add-on.

Additional resources

Learn more about community-addons.title.

You install or update community add-ons in the same way as other Amazon EKS Add-ons.

creating-an-add-on.title
updating-an-add-on.title
removing-an-add-on.title

13.2.2. Deploy with manifest

New: You can now deploy Metrics Server as a community add-on using the AWS console or Amazon EKS APIs. These manifest install instructions will be archived.

Deploy the Metrics Server with the following command:
```
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
```
If you are using Fargate, you will need to change this file. In the default configuration, the metrics server uses port 10250. This port is reserved on Fargate. Replace references to port 10250 in components.yaml with another port, such as 10251.

Verify that the metrics-server deployment is running the desired number of Pods with the following command.

kubectl get deployment metrics-server -n kube-system

An example output is as follows.

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           6m

Test the metrics server is working by displaying resource (CPU/memory) usage of nodes.
```
kubectl top nodes
```
If you receive the error message Error from server (Forbidden), you need to update your Kubernetes RBAC configuration. Your Kubernetes RBAC identity needs sufficent permissions to read cluster metrics. Review the minimum required Kubernetes API permissions for reading metrics on GitHub. Learn how to grant AWS IAM Identities such as Roles access to Kubernetes APIs.

13.3. Deploy applications with `Helm` on Amazon EKS

Learn how to install and use Helm, a package manager for Kubernetes, with your Amazon EKS cluster to manage and deploy applications seamlessly.

The Helm package manager for Kubernetes helps you install and manage applications on your Kubernetes cluster. For more information, see the Helm documentation. This topic helps you install and run the Helm binaries so that you can install and manage charts using the Helm CLI on your local system.

Before you can install Helm charts on your Amazon EKS cluster, you must configure kubectl to work for Amazon EKS. If you have not already done this, see create-kubeconfig.title before proceeding. If the following command succeeds for your cluster, you’re properly configured.

kubectl get svc

Run the appropriate command for your client operating system.
- If you’re using macOS with Homebrew, install the binaries with the following command.
  brew install helm
- If you’re using Windows with Chocolatey, install the binaries with the following command.
  choco install kubernetes-helm
- If you’re using Linux, install the binaries with the following commands.
  curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh chmod 700 get_helm.sh ./get_helm.sh
  If you get a message that openssl must first be installed, you can install it with the following command.

sudo yum install openssl

To pick up the new binary in your PATH, Close your current terminal window and open a new one.
See the version of Helm that you installed.
```
helm version | cut -d + -f 1
```
An example output is as follows.
```
v3.9.0
```
At this point, you can run any Helm commands (such as helm install chart-name) to install, modify, delete, or query Helm charts in your cluster. If you’re new to Helm and don’t have a specific chart to install, you can:
- Experiment by installing an example chart. See Install an example chart in the Helm Quickstart guide.
- Create an example chart and push it to Amazon ECR. For more information, see Pushing a Helm chart in the Amazon Elastic Container Registry User Guide.
- Install an Amazon EKS chart from the eks-chartsGitHub repo or from ArtifactHub.

13.4. Organize Amazon EKS resources with tags

Learn how to use tags to categorize and manage your Amazon EKS resources like clusters, managed node groups, and Fargate profiles for billing, cost allocation, and resource identification.

You can use tags to help you manage your Amazon EKS resources. This topic provides an overview of the tags function and shows how you can create tags.

[[Topic List]]

Tags are a type of metadata that’s separate from Kubernetes labels and annotations. For more information about these other metadata types, see the following sections in the Kubernetes documentation:

13.4.1. Tag basics

A tag is a label that you assign to an AWS resource. Each tag consists of a key and an optional value.

With tags, you can categorize your AWS resources. For example, you can categorize resources by purpose, owner, or environment. When you have many resources of the same type, you can use the tags that you assigned to a specific resource to quickly identify that resource. For example, you can define a set of tags for your Amazon EKS clusters to help you track each cluster’s owner and stack level. We recommend that you devise a consistent set of tag keys for each resource type. You can then search and filter the resources based on the tags that you add.

After you add a tag, you can edit tag keys and values or remove tags from a resource at any time. If you delete a resource, any tags for the resource are also deleted.

Tags don’t have any semantic meaning to Amazon EKS and are interpreted strictly as a string of characters. You can set the value of a tag to an empty string. However, you can’t set the value of a tag to null. If you add a tag that has the same key as an existing tag on that resource, the new value overwrites the earlier value.

If you use AWS Identity and Access Management (IAM), you can control which users in your AWS account have permission to manage tags.

13.4.2. Tagging your resources

The following Amazon EKS resources support tags:

clusters
managed node groups
Fargate profiles

You can tag these resources using the following:

If you’re using the Amazon EKS console, you can apply tags to new or existing resources at any time. You can do this by using the Tags tab on the relevant resource page. For more information, see tag-resources-console.title.
If you’re using eksctl, you can apply tags to resources when they’re created using the --tags option.
If you’re using the AWS CLI, the Amazon EKS API, or an AWS SDK, you can apply tags to new resources using the tags parameter on the relevant API action. You can apply tags to existing resources using the TagResource API action. For more information, see TagResource.

When you use some resource-creating actions, you can also specify tags for the resource at the same time that you create it. If tags can’t be applied while the resource is being created, the resource fails to be created. This mechanism ensures that resources that you intend to tag are either created with the tags that you specify or not created at all. If you tag resources when you create them, you don’t need to run custom tagging scripts after you create the resource.

Tags don’t propagate to other resources that are associated with the resource that you create. For example, Fargate profile tags don’t propagate to other resources that are associated with the Fargate profile, such as the Pods that are scheduled with it.

13.4.3. Tag restrictions

The following restrictions apply to tags:

A maximum of 50 tags can be associated with a resource.
Tag keys can’t be repeated for one resource. Each tag key must be unique, and can only have one value.
Keys can be up to 128 characters long in UTF-8.
Values can be up to 256 characters long in UTF-8.
If multiple AWS services and resources use your tagging schema, limit the types of characters you use. Some services might have restrictions on allowed characters. Generally, allowed characters are letters, numbers, spaces, and the following characters: + - = . _ : / @.
Tag keys and values are case sensitive.
Don’t use aws:, AWS:, or any upper or lowercase combination of such as a prefix for either keys or values. These are reserved only for AWS use. You can’t edit or delete tag keys or values with this prefix. Tags with this prefix don’t count against your tags-per-resource limit.

13.4.4. Tagging your resources for billing

When you apply tags to Amazon EKS clusters, you can use them for cost allocation in your Cost & Usage Reports. The metering data in your Cost & Usage Reports shows usage across all of your Amazon EKS clusters. For more information, see AWS cost and usage report in the AWS Billing User Guide.

The AWS generated cost allocation tag, specifically aws:eks:cluster-name, lets you break down Amazon EC2 instance costs by individual Amazon EKS cluster in Cost Explorer. However, this tag doesn’t capture the control plane expenses. The tag is automatically added to Amazon EC2 instances that participate in an Amazon EKS cluster. This behavior happens regardless of whether the instances are provisioned using Amazon EKS managed node groups, Karpenter, or directly with Amazon EC2. This specific tag doesn’t count towards the 50 tags limit. To use the tag, the account owner must activate it in the AWS Billing console or by using the API. When an AWS Organizations management account owner activates the tag, it’s also activated for all organization member accounts.

You can also organize your billing information based on resources that have the same tag key values. For example, you can tag several resources with a specific application name, and then organize your billing information. That way, you can see the total cost of that application across several services. For more information about setting up a cost allocation report with tags, see The Monthly Cost Allocation Report in the AWS Billing User Guide.

If you just enabled reporting, data for the current month is available for viewing after 24 hours.

Cost Explorer is a reporting tool that’s available as part of the AWS Free Tier. You can use Cost Explorer to view charts of your Amazon EKS resources from the last 13 months. You can also forecast how much you’re likely to spend for the next three months. You can see patterns in how much you spend on AWS resources over time. For example, you can use it to identify areas that need further inquiry and see trends that you can use to understand your costs. You also can specify time ranges for the data, and view time data by day or by month.

13.4.5. Working with tags using the console

Using the Amazon EKS console, you can manage the tags that are associated with new or existing clusters and managed node groups.

When you select a resource-specific page in the Amazon EKS console, the page displays a list of those resources. For example, if you select Clusters from the left navigation pane, the console displays a list of Amazon EKS clusters. When you select a resource from one of these lists (for example, a specific cluster) that supports tags, you can view and manage its tags on the Tags tab.

You can also use Tag Editor in the consolelong, which provides a unified way to manage your tags. For more information, see Tagging your AWS resources with Tag Editor in the AWS Tag Editor User Guide.

Adding tags on a resource on creation

You can add tags to Amazon EKS clusters, managed node groups, and Fargate profiles when you create them. For more information, see create-cluster.title.

Adding and deleting tags on a resource

You can add or delete the tags that are associated with your clusters directly from the resource’s page.

Open the Amazon EKS console.
On the navigation bar, select the AWS Region to use.
In the left navigation pane, choose Clusters.
Choose a specific cluster.
Choose the Tags tab, and then choose Manage tags.
On the Manage tags page, add or delete your tags as necessary.
- To add a tag, choose Add tag. Then specify the key and value for each tag.
- To delete a tag, choose Remove tag.
Repeat this process for each tag that you want to add or delete.
Choose Update to finish.

13.4.6. Working with tags using the CLI, API, or `eksctl`

Use the following AWS CLI commands or Amazon EKS API operations to add, update, list, and delete the tags for your resources. You can only use eksctl to add tags while simultaneously creating the new resources with one command.

Task AWS CLI AWS Tools for Windows PowerShell API action

Add or overwrite one or more tags.

tag-resource

Add-EKSResourceTag

TagResource

Delete one or more tags.

untag-resource

Remove-EKSResourceTag

UntagResource

The following examples show how to tag or untag resources using the AWS CLI.

Example 1: Tag an existing cluster

The following command tags an existing cluster.

aws eks tag-resource --resource-arn resource_ARN --tags team=devs

Example 2: Untag an existing cluster

The following command deletes a tag from an existing cluster.

aws eks untag-resource --resource-arn resource_ARN --tag-keys tag_key

Example 3: List tags for a resource

The following command lists the tags that are associated with an existing resource.

aws eks list-tags-for-resource --resource-arn resource_ARN

When you use some resource-creating actions, you can specify tags at the same time that you create the resource. The following actions support specifying a tag when you create a resource.

Task AWS CLI AWS Tools for Windows PowerShell API action eksctl

Create a cluster

create-cluster

New-EKSCluster

CreateCluster

create cluster

Create a managed node group*

create-nodegroup

New-EKSNodegroup

CreateNodegroup

create nodegroup

Create a Fargate profile

create-fargate-profile

New-EKSFargateProfile

CreateFargateProfile.html

create fargateprofile

If you want to also tag the Amazon EC2 instances when you create a managed node group, create the managed node group using a launch template. For more information, see launch-template-tagging.title. If your instances already exist, you can manually tag the instances. For more information, see Tagging your resources in the Amazon EC2 User Guide.

13.5. View and manage Amazon EKS and `Fargate` service quotas

Use Service Quotas to view and manage Amazon EKS and AWS Fargate quotas from the consolelong or AWS CLI.

Amazon EKS has integrated with Service Quotas, an AWS service that you can use to view and manage your quotas from a central location. For more information, see What Is Service Quotas? in the Service Quotas User Guide. With Service Quotas integration, you can quickly look up the value of your Amazon EKS and AWS Fargate service quotas using the consolelong and AWS CLI.

13.5.1. View EKS service quotas in the `AWS` Management Console

Open the Service Quotas console.
In the left navigation pane, choose AWS services.
From the AWS services list, search for and select Amazon Elastic Kubernetes Service (Amazon EKS) or AWS Fargate.

In the Service quotas list, you can see the service quota name, applied value (if it’s available), AWS default quota, and whether the quota value is adjustable.
To view additional information about a service quota, such as the description, choose the quota name.
(Optional) To request a quota increase, select the quota that you want to increase, select Request quota increase, enter or select the required information, and select Request.

To work more with service quotas using the consolelong, see the Service Quotas User Guide. To request a quota increase, see Requesting a Quota Increase in the Service Quotas User Guide.

13.5.2. View EKS service quotas with the `AWS` CLI

Run the following command to view your Amazon EKS quotas.

aws service-quotas list-aws-default-service-quotas \
    --query 'Quotas[*].{Adjustable:Adjustable,Name:QuotaName,Value:Value,Code:QuotaCode}' \
    --service-code eks \
    --output table

Run the following command to view your Fargate quotas.

aws service-quotas list-aws-default-service-quotas \
    --query 'Quotas[*].{Adjustable:Adjustable,Name:QuotaName,Value:Value,Code:QuotaCode}' \
    --service-code fargate \
    --output table

The quota returned is the number of Amazon ECS tasks or Amazon EKS Pods that can run concurrently on Fargate in this account in the current AWS Region.

To work more with service quotas using the AWS CLI, see service-quotas in the AWS CLI Command Reference. To request a quota increase, see the request-service-quota-increase command in the AWS CLI Command Reference.

13.5.3. Amazon EKS service quotas

AWS recommends using the AWS management console to view your current quotas. For more information, see service-quotas-console.title.

To view the default EKS service quotas, see Amazon Elastic Kubernetes Service endpoints and quotas in the AWS General Reference.

These service quotas are listed under Amazon Elastic Kubernetes Service (Amazon EKS) in the Service Quotas console. To request a quota increase for values that are shown as adjustable, see Requesting a quota increase in the Service Quotas User Guide.

The following quotas aren’t available in Service Quotas:

Pod Identity associations per cluster is 1000 in each supported region and this quota isn’t adjustable.
You can use up to 15 CIDRs for Remote Node Networks and 15 CIDRs for Remote Pod Networks per cluster for hybrid nodes. This quota isn’t adjustable.

13.5.4. `AWS` Fargate service quotas

The AWS Fargate service in the Service Quotas console lists several service quotas. You can configure alarms that alert you when your usage approaches a service quota. For more information, see service-quota-alarm.title.

New AWS accounts might have lower initial quotas that can increase over time. Fargate constantly monitors the account usage within each AWS Region, and then automatically increases the quotas based on the usage. You can also request a quota increase for values that are shown as adjustable. For more information, see Requesting a quota increase in the Service Quotas User Guide.

AWS reccomends using the AWS management console to view your current quotas. For more information, see service-quotas-console.title.

To view default AWS Fargate on EKS service quotas, see Fargate service quotas in the AWS General Reference.

Fargate additionally enforces Amazon ECS tasks and Amazon EKS Pods launch rate quotas. For more information, see AWS Fargate throttling quotas in the Amazon ECS guide.

14. Security in Amazon EKS

14.1. Secure Amazon EKS clusters with best practices

Learn how to secure your Amazon EKS clusters by following the best practices from the community.

The Amazon EKS security best practices are in the Best Practices for Security in the Amazon EKS Best Practices Guide.

14.2. Analyze vulnerabilities in Amazon EKS

Learn how to analyze the security configuration and vulnerabilities of your Amazon EKS clusters and resources using tools like the CIS EKS Benchmark, platform versions, vulnerability lists, Amazon Inspector, and Amazon GuardDuty for comprehensive threat detection and protection.

Security is a critical consideration for configuring and maintaining Kubernetes clusters and applications. The following lists resources for you to analyze the security configuration of your EKS clusters, resources for you to check for vulnerabilities, and integrations with AWS services that can do that analysis for you.

14.2.1. The Center for Internet Security (CIS) benchmark for Amazon EKS

The Center for Internet Security (CIS) Kubernetes Benchmark provides guidance for Amazon EKS security configurations. The benchmark:

Is applicable to Amazon EC2 nodes (both managed and self-managed) where you are responsible for security configurations of Kubernetes components.
Provides a standard, community-approved way to ensure that you have configured your Kubernetes cluster and nodes securely when using Amazon EKS.
Consists of four sections; control plane logging configuration, node security configurations, policies, and managed services.
Supports all of the Kubernetes versions currently available in Amazon EKS and can be run using kube-bench, a standard open source tool for checking configuration using the CIS benchmark on Kubernetes clusters.

To learn more, see Introducing The CIS Amazon EKS Benchmark.

14.2.2. Amazon EKS platform versions

Amazon EKS platform versions represent the capabilities of the cluster control plane, including which Kubernetes API server flags are enabled and the current Kubernetes patch version. New clusters are deployed with the latest platform version. For details, see platform-versions.title.

You can update an Amazon EKS cluster to newer Kubernetes versions. As new Kubernetes versions become available in Amazon EKS, we recommend that you proactively update your clusters to use the latest available version. For more information about Kubernetes versions in EKS, see kubernetes-versions.title.

14.2.3. Operating system vulnerability list

AL2023 vulnerability list

Track security or privacy events for Amazon Linux 2023 at the Amazon Linux Security Center or subscribe to the associated RSS feed. Security and privacy events include an overview of the issue affected, packages, and instructions for updating your instances to correct the issue.

Amazon Linux 2 vulnerability list

Track security or privacy events for Amazon Linux 2 at the Amazon Linux Security Center or subscribe to the associated RSS feed. Security and privacy events include an overview of the issue affected, packages, and instructions for updating your instances to correct the issue.

14.2.4. Node detection with Amazon Inspector

You can use Amazon Inspector to check for unintended network accessibility of your nodes and for vulnerabilities on those Amazon EC2 instances.

14.2.5. Cluster and node detection with Amazon GuardDuty

Amazon GuardDuty threat detection service that helps protect your accounts, containers, workloads, and the data within your AWS environment. Among other features, GuardDuty offers the following two features that detect potential threats to your EKS clusters: EKS Protection and Runtime Monitoring.

For more information, see integration-guardduty.title.

14.3. Compliance validation for Amazon EKS clusters

Discover compliance resources and services for Amazon Elastic Kubernetes Service to help secure your AWS workloads, meet regulatory requirements like HIPAA, and validate adherence to security standards like NIST, PCI, and ISO using AWS Config, Security Hub, GuardDuty, and Audit Manager.

To learn whether an AWS service is within the scope of specific compliance programs, see AWS services in Scope by Compliance Program and choose the compliance program that you are interested in. For general information, see AWS Compliance Programs.

You can download third-party audit reports using AWS Artifact. For more information, see Downloading Reports in AWS Artifact.

Your compliance responsibility when using AWS services is determined by the sensitivity of your data, your company’s compliance objectives, and applicable laws and regulations. AWS provides the following resources to help with compliance:

Security and Compliance Quick Start Guides – These deployment guides discuss architectural considerations and provide steps for deploying baseline environments on AWS that are security and compliance focused.
Architecting for HIPAA Security and Compliance on Amazon Web Services – This whitepaper describes how companies can use AWS to create HIPAA-eligible applications.

Not all AWS services are HIPAA eligible. For more information, see the HIPAA Eligible Services Reference.
AWS Compliance Resources – This collection of workbooks and guides might apply to your industry and location.
AWS Customer Compliance Guides – Understand the shared responsibility model through the lens of compliance. The guides summarize the best practices for securing AWS services and map the guidance to security controls across multiple frameworks (including National Institute of Standards and Technology (NIST), Payment Card Industry Security Standards Council (PCI), and International Organization for Standardization (ISO)).
Evaluating Resources with Rules in the AWS Config Developer Guide – The AWS Config service assesses how well your resource configurations comply with internal practices, industry guidelines, and regulations.
AWS Security Hub – This AWS service provides a comprehensive view of your security state within AWS. Security Hub uses security controls to evaluate your AWS resources and to check your compliance against security industry standards and best practices. For a list of supported services and controls, see Security Hub controls reference.
Amazon GuardDuty – This AWS service detects potential threats to your AWS accounts, workloads, containers, and data by monitoring your environment for suspicious and malicious activities. GuardDuty can help you address various compliance requirements, like PCI DSS, by meeting intrusion detection requirements mandated by certain compliance frameworks.
AWS Audit Manager – This AWS service helps you continuously audit your AWS usage to simplify how you manage risk and compliance with regulations and industry standards.

14.4. Security considerations for Amazon Elastic Kubernetes Service

14.4.1. Infrastructure security in Amazon EKS

Access the Amazon EKS using `AWS` PrivateLink

Learn how to securely access Amazon Elastic Kubernetes Service (Amazon EKS) APIs from within your VPC using AWS PrivateLink, avoiding public internet exposure while benefiting from private connectivity, routing optimization, and built-in security controls for enhanced cluster management.

You can use AWS PrivateLink to create a private connection between your VPC and Amazon Elastic Kubernetes Service. You can access Amazon EKS as if it were in your VPC, without the use of an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. Instances in your VPC don’t need public IP addresses to access Amazon EKS.

You establish this private connection by creating an interface endpoint powered by AWS PrivateLink. We create an endpoint network interface in each subnet that you enable for the interface endpoint. These are requester-managed network interfaces that serve as the entry point for traffic destined for Amazon EKS.

For more information, see Access AWS services through AWS PrivateLink in the AWS PrivateLink Guide.

Considerations for Amazon EKS

Before you set up an interface endpoint for Amazon EKS, review Considerations in the AWS PrivateLink Guide.
Amazon EKS supports making calls to all of its API actions through the interface endpoint, but not to the Kubernetes APIs. The Kubernetes API server already supports a private endpoint. The Kubernetes API server private endpoint creates a private endpoint for the Kubernetes API server that you use to communicate with your cluster (using Kubernetes management tools such as kubectl). You can enable private access to the Kubernetes API server so that all communication between your nodes and the API server stays within your VPC. AWS PrivateLink for the Amazon EKS API helps you call the Amazon EKS APIs from your VPC without exposing traffic to the public internet.
You can’t configure Amazon EKS to only be accessed through an interface endpoint.
Standard pricing for AWS PrivateLink applies for interface endpoints for Amazon EKS. You are billed for every hour that an interface endpoint is provisioned in each Availability Zone and for data processed through the interface endpoint. For more information, see AWS PrivateLink pricing.
VPC endpoint policies are not supported for Amazon EKS. By default, full access to Amazon EKS is allowed through the interface endpoint. Alternatively, you can associate a security group with the endpoint network interfaces to control traffic to Amazon EKS through the interface endpoint.
You can use VPC flow logs to capture information about IP traffic going to and from network interfaces, including interface endpoints. You can publish flow log data to Amazon CloudWatch or Amazon S3. For more information, see Logging IP traffic using VPC Flow Logs in the Amazon VPC User Guide.
You can access the Amazon EKS APIs from an on-premises data center by connecting it to a VPC that has an interface endpoint. You can use AWS Direct Connect or AWS Site-to-Site VPN to connect your on-premises sites to a VPC.
You can connect other VPCs to the VPC with an interface endpoint using an AWS Transit Gateway or VPC peering. VPC peering is a networking connection between two VPCs. You can establish a VPC peering connection between your VPCs, or with a VPC in another account. The VPCs can be in different AWS Regions. Traffic between peered VPCs stays on the AWS network. The traffic doesn’t traverse the public internet. A Transit Gateway is a network transit hub that you can use to interconnect VPCs. Traffic between a VPC and a Transit Gateway remains on the AWS global private network. The traffic isn’t exposed to the public internet.
Before August 2024, VPC interface endpoints for Amazon EKS were only accessible over IPv4 using eks.region.amazonaws.com. New VPC interface endpoints that are made after August 2024 use dual-stack of IPv4 and IPv6 IP addresses and both DNS names: eks.region.amazonaws.com and eks.region.api.aws.
AWS PrivateLink support for the EKS API isn’t available in the Asia Pacific (Malaysia) (ap-southeast-5), Asia Pacific (Thailand) (ap-southeast-7), and Mexico (Central) (mx-central-1) AWS Regions. AWS PrivateLink support for eks-auth for EKS Pod Identity is available in the the Asia Pacific (Malaysia) (ap-southeast-5) Region.

Create an interface endpoint for Amazon EKS

You can create an interface endpoint for Amazon EKS using either the Amazon VPC console or the AWS Command Line Interface (AWS CLI). For more information, see Create a VPC endpoint in the AWS PrivateLink Guide.

Create an interface endpoint for Amazon EKS using the following service names:

EKS API

com.amazonaws.region-code.eks

EKS Auth API (EKS Pod Identity)

com.amazonaws.region-code.eks-auth

The private DNS feature is enabled by default when creating an interface endpoint for Amazon EKS and other AWS services. To use the private DNS feature, you must ensure that the following VPC attributes are set to true: enableDnsHostnames and enableDnsSupport. For more information, see View and update DNS attributes for your VPC in the Amazon VPC User Guide. With the private DNS feature enabled for the interface endpoint:

You can make any API request to Amazon EKS using its default Regional DNS name. After August 2024, any new VPC interface endpoint for the Amazon EKS API have two default Regional DNS names and you can choose the dualstack for the IP address type. The first DNS name is eks.region.api.aws which is dual-stack. It resolves to both IPv4 addresses and IPv6 addresses. Before August 2024, Amazon EKS only used eks.region.amazonaws.com which resolved to IPv4 addresses only. If you want to use IPv6 and dual-stack IP addresses with an existing VPC interface endpoint, you can update the endpoint to use the dualstack type of IP address, but it will only have the eks.region.amazonaws.com DNS name. In this configuration, the existing endpoint updates to point that name to both IPv4 and IPv6 IP addresses. For a list of APIs, see Actions in the Amazon EKS API Reference.
You don’t need to make any changes to your applications that call the EKS APIs.

However, To use the dual-stack endpoints with the AWS CLI, see the Dual-stack and FIPS endpoints configuration in the AWS SDKs and Tools Reference Guide.
Any call made to the Amazon EKS default service endpoint is automatically routed through the interface endpoint over the private AWS network.

Learn how Amazon EKS isolates service traffic.

As a managed service, Amazon Elastic Kubernetes Service is protected by AWS global network security. For information about AWS security services and how AWS protects infrastructure, see AWS Cloud Security. To design your AWS environment using the best practices for infrastructure security, see Infrastructure Protection in Security Pillar AWS Well‐Architected Framework.

You use AWS published API calls to access Amazon EKS through the network. Clients must support the following:

Transport Layer Security (TLS). We require TLS 1.2 and recommend TLS 1.3.
Cipher suites with perfect forward secrecy (PFS) such as DHE (Ephemeral Diffie-Hellman) or ECDHE (Elliptic Curve Ephemeral Diffie-Hellman). Most modern systems such as Java 7 and later support these modes.

Additionally, requests must be signed by using an access key ID and a secret access key that is associated with an IAM principal. Or you can use the AWS Security Token Service (AWS STS) to generate temporary security credentials to sign requests.

When you create an Amazon EKS cluster, you specify the VPC subnets for your cluster to use. Amazon EKS requires subnets in at least two Availability Zones. We recommend a VPC with public and private subnets so that Kubernetes can create public load balancers in the public subnets that load balance traffic to Pods running on nodes that are in private subnets.

For more information about VPC considerations, see network-reqs.title.

If you create your VPC and node groups with the AWS CloudFormation templates provided in the Get started with Amazon EKS walkthrough, then your control plane and node security groups are configured with our recommended settings.

For more information about security group considerations, see sec-group-reqs.title.

For more information about modifying cluster endpoint access, see modify-endpoint-access.title.

You can implement Kubernetes network policies with the Amazon VPC CNI or third-party tools such as Project Calico. For more information about using the Amazon VPC CNI for network policies, see cni-network-policy.title. Project Calico is a third party open source project. For more information, see the Project Calico documentation.

14.4.2. Understand resilience in Amazon EKS clusters

Learn how Amazon EKS ensures high availability, data resilience, and fault tolerance for your Kubernetes control plane by leveraging AWS infrastructure across multiple Availability Zones .

The AWS global infrastructure is built around AWS Regions and Availability Zones. AWS Regions provide multiple physically separated and isolated Availability Zones, which are connected with low-latency, high-throughput, and highly redundant networking. With Availability Zones, you can design and operate applications and databases that automatically fail over between Availability Zones without interruption. Availability Zones are more highly available, fault tolerant, and scalable than traditional single or multiple data center infrastructures.

Amazon EKS runs and scales the Kubernetes control plane across multiple AWS Availability Zones to ensure high availability. Amazon EKS automatically scales control plane instances based on load, detects and replaces unhealthy control plane instances, and automatically patches the control plane. After you initiate a version update, Amazon EKS updates your control plane for you, maintaining high availability of the control plane during the update.

This control plane consists of at least two API server instances and three etcd instances that run across three Availability Zones within an AWS Region. Amazon EKS:

Actively monitors the load on control plane instances and automatically scales them to ensure high performance.
Automatically detects and replaces unhealthy control plane instances, restarting them across the Availability Zones within the AWS Region as needed.
Leverages the architecture of AWS Regions in order to maintain high availability. Because of this, Amazon EKS is able to offer an SLA for API server endpoint availability.

For more information about AWS Regions and Availability Zones, see AWS global infrastructure.

Configure Amazon EKS clusters to meet your security and compliance objectives, and learn how to use other AWS services that help you to secure your Amazon EKS clusters.

The following are considerations for security of the cloud, as they affect Amazon EKS.

[[Topic List]]

14.5. Security considerations for `Kubernetes`

14.5.1. Secure workloads with `Kubernetes` certificates

Learn how to request and obtain X.509 certificates from the Certificate Authority (CA) using Certificate Signing Requests (CSRs) in Amazon EKS, including details on migrating from legacy signers, generating CSRs, approving requests, and handling certificate signing considerations before upgrading to Kubernetes 1.24.

The Kubernetes Certificates API automates X.509 credential provisioning. The API features a command line interface for Kubernetes API clients to request and obtain X.509 certificates from a Certificate Authority (CA). You can use the CertificateSigningRequest (CSR) resource to request that a denoted signer sign the certificate. Your requests are either approved or denied before they’re signed. Kubernetes supports both built-in signers and custom signers with well-defined behaviors. This way, clients can predict what happens to their CSRs. To learn more about certificate signing, see signing requests.

One of the built-in signers is kubernetes.io/legacy-unknown. The v1beta1 API of CSR resource honored this legacy-unknown signer. However, the stable v1 API of CSR doesn’t allow the signerName to be set to kubernetes.io/legacy-unknown.

Amazon EKS version 1.21 and earlier allowed the legacy-unknown value as the signerName in v1beta1 CSR API. This API enables the Amazon EKS Certificate Authority (CA) to generate certificates. However, in Kubernetes version 1.22, the v1beta1 CSR API was replaced by the v1 CSR API. This API doesn’t support the signerName of “legacy-unknown.” If you want to use Amazon EKS CA for generating certificates on your clusters, you must use a custom signer. It was introduced in Amazon EKS version 1.22. To use the CSR v1 API version and generate a new certificate, you must migrate any existing manifests and API clients. Existing certificates that were created with the existing v1beta1 API are valid and function until the certificate expires. This includes the following:

Trust distribution: None. There’s no standard trust or distribution for this signer in a Kubernetes cluster.
Permitted subjects: Any
Permitted x509 extensions: Honors subjectAltName and key usage extensions and discards other extensions
Permitted key usages: Must not include usages beyond ["key encipherment", "digital signature", "server auth"]

Client certificate signing is not supported.
Expiration/certificate lifetime: 1 year (default and maximum)
CA bit allowed/disallowed: Not allowed

Example CSR generation with signerName

These steps shows how to generate a serving certificate for DNS name myserver.default.svc using signerName: beta.eks.amazonaws.com/app-serving. Use this as a guide for your own environment.

Run the openssl genrsa -out myserver.key 2048 command to generate an RSA private key.
```
openssl genrsa -out myserver.key 2048
```

Run the following command to generate a certificate request.

openssl req -new -key myserver.key -out myserver.csr -subj "/CN=myserver.default.svc"

Generate a base64 value for the CSR request and store it in a variable for use in a later step.
```
base_64=$(cat myserver.csr | base64 -w 0 | tr -d "
")
```

Run the following command to create a file named mycsr.yaml. In the following example, beta.eks.amazonaws.com/app-serving is the signerName.

cat >mycsr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: myserver
spec:
  request: $base_64
  signerName: beta.eks.amazonaws.com/app-serving
  usages:
    - digital signature
    - key encipherment
    - server auth
EOF

Submit the CSR.
```
kubectl apply -f mycsr.yaml
```
Approve the serving certificate.
```
kubectl certificate approve myserver
```

Verify that the certificate was issued.

kubectl get csr myserver

An example output is as follows.

NAME       AGE     SIGNERNAME                           REQUESTOR          CONDITION
myserver   3m20s   beta.eks.amazonaws.com/app-serving   kubernetes-admin   Approved,Issued

Export the issued certificate.

kubectl get csr myserver -o jsonpath='{.status.certificate}'| base64 -d > myserver.crt

Certificate signing considerations before upgrading your cluster to `Kubernetes` 1.24

In Kubernetes 1.23 and earlier, kubelet serving certificates with unverifiable IP and DNS Subject Alternative Names (SANs) are automatically issued with unverifiable SANs. The SANs are omitted from the provisioned certificate. In 1.24 and later clusters, kubelet serving certificates aren’t issued if a SAN can’t be verified. This prevents the kubectl exec and kubectl logs commands from working.

Before upgrading your cluster to 1.24, determine whether your cluster has certificate signing requests (CSR) that haven’t been approved by completing the following steps:

Run the following command.

kubectl get csr -A

An example output is as follows.

NAME        AGE   SIGNERNAME                      REQUESTOR                                                  REQUESTEDDURATION   CONDITION
csr-7znmf   90m   kubernetes.io/kubelet-serving   system:node:ip-192-168-42-149.region.compute.internal      <none>              Approved
csr-9xx5q   90m   kubernetes.io/kubelet-serving   system:node:ip-192-168-65-38.region.compute.internal      <none>              Approved, Issued

If the returned output shows a CSR with a kubernetes.io/kubelet-serving signer that’s Approved but not Issued for a node, then you need to approve the request.

Manually approve the CSR. Replace csr-7znmf with your own value.
```
kubectl certificate approve csr-7znmf
```

To auto-approve CSRs in the future, we recommend that you write an approving controller that can automatically validate and approve CSRs that contain IP or DNS SANs that Amazon EKS can’t verify.

14.5.2. Understand Amazon EKS created RBAC roles and users

Learn about the Kubernetes roles and users that Amazon EKS creates for cluster components and add-ons. Amazon EKS uses these role-based authorization control (RBAC) identities to operate the cluster.

When you create a Kubernetes cluster, several default Kubernetes identities are created on that cluster for the proper functioning of Kubernetes. Amazon EKS creates Kubernetes identities for each of its default components. The identities provide Kubernetes role-based authorization control (RBAC) for the cluster components. For more information, see Using RBAC Authorization in the Kubernetes documentation.

When you install optional add-ons to your cluster, additional Kubernetes identities might be added to your cluster. For more information about identities not addressed by this topic, see the documentation for the add-on.

You can view the list of Amazon EKS created Kubernetes identities on your cluster using the consolelong or kubectl command line tool. All of the user identities appear in the kube audit logs available to you through Amazon CloudWatch.

consolelong

.Prerequisite The IAM principal that you use must have the permissions described in Required permissions.

Open the Amazon EKS console.
In the Clusters list, choose the cluster that contains the identities that you want to view.
Choose the Resources tab.
Under Resource types, choose Authorization.
Choose, ClusterRoles, ClusterRoleBindings, Roles, or RoleBindings. All resources prefaced with eks are created by Amazon EKS. Additional Amazon EKS created identity resources are:
- The ClusterRole and ClusterRoleBinding named aws-node. The aws-node resources support the Amazon VPC CNI plugin for Kubernetes, which Amazon EKS installs on all clusters.
- A ClusterRole named vpc-resource-controller-role and a ClusterRoleBinding named vpc-resource-controller-rolebinding. These resources support the Amazon VPC resource controller, which Amazon EKS installs on all clusters.

In addition to the resources that you see in the console, the following special user identities exist on your cluster, though they’re not visible in the cluster’s configuration:

eks:cluster-bootstrap – Used for kubectl operations during cluster bootstrap.
eks:support-engineer – Used for cluster management operations.
1. Choose a specific resource to view details about it. By default, you’re shown information in Structured view. In the top-right corner of the details page you can choose Raw view to see all information for the resource.

Kubectl

.Prerequisite The entity that you use (AWS Identity and Access Management (IAM) or OpenID Connect (OIDC)) to list the Kubernetes resources on the cluster must be authenticated by IAM or your OIDC identity provider. The entity must be granted permissions to use the Kubernetes get and list verbs for the Role, ClusterRole, RoleBinding, and ClusterRoleBinding resources on your cluster that you want the entity to work with. For more information about granting IAM entities access to your cluster, see grant-k8s-access.title. For more information about granting entities authenticated by your own OIDC provider access to your cluster, see authenticate-oidc-identity-provider.title. .To view Amazon EKS created identities using kubectl Run the command for the type of resource that you want to see. All returned resources that are prefaced with eks are created by Amazon EKS. In addition to the resources returned in the output from the commands, the following special user identities exist on your cluster, though they’re not visible in the cluster’s configuration:

eks:cluster-bootstrap – Used for kubectl operations during cluster bootstrap.
eks:support-engineer – Used for cluster management operations.

ClusterRoles – ClusterRoles are scoped to your cluster, so any permission granted to a role applies to resources in any Kubernetes namespace on the cluster.

The following command returns all of the Amazon EKS created Kubernetes ClusterRoles on your cluster.
```
kubectl get clusterroles | grep eks
```
In addition to the ClusterRoles returned in the output that are prefaced with, the following ClusterRoles exist.
aws-node – This ClusterRole supports the Amazon VPC CNI plugin for Kubernetes, which Amazon EKS installs on all clusters.
vpc-resource-controller-role – This ClusterRole supports the Amazon VPC resource controller, which Amazon EKS installs on all clusters.

To see the specification for a ClusterRole, replace eks:k8s-metrics in the following command with a ClusterRole returned in the output of the previous command. The following example returns the specification for the eks:k8s-metrics ClusterRole.
```
kubectl describe clusterrole eks:k8s-metrics
```
An example output is as follows.
```
Name:         eks:k8s-metrics
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources         Non-Resource URLs  Resource Names  Verbs
  ---------         -----------------  --------------  -----
                    [/metrics]         []              [get]
  endpoints         []                 []              [list]
  nodes             []                 []              [list]
  pods              []                 []              [list]
  deployments.apps  []                 []              [list]
```
ClusterRoleBindings – ClusterRoleBindings are scoped to your cluster.

The following command returns all of the Amazon EKS created Kubernetes ClusterRoleBindings on your cluster.
```
kubectl get clusterrolebindings | grep eks
```
In addition to the ClusterRoleBindings returned in the output, the following ClusterRoleBindings exist.
aws-node – This ClusterRoleBinding supports the Amazon VPC CNI plugin for Kubernetes, which Amazon EKS installs on all clusters.
vpc-resource-controller-rolebinding – This ClusterRoleBinding supports the Amazon VPC resource controller, which Amazon EKS installs on all clusters.

To see the specification for a ClusterRoleBinding, replace eks:k8s-metrics in the following command with a ClusterRoleBinding returned in the output of the previous command. The following example returns the specification for the eks:k8s-metrics ClusterRoleBinding.

kubectl describe clusterrolebinding eks:k8s-metrics

+ An example output is as follows.

Name:         eks:k8s-metrics
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  eks:k8s-metrics
Subjects:
  Kind  Name             Namespace
  ----  ----             ---------
  User  eks:k8s-metrics

+ Roles – Roles are scoped to a Kubernetes namespace. All Amazon EKS created Roles are scoped to the kube-system namespace.

+ The following command returns all of the Amazon EKS created Kubernetes Roles on your cluster.

kubectl get roles -n kube-system | grep eks

+ To see the specification for a Role, replace eks:k8s-metrics in the following command with the name of a Role returned in the output of the previous command. The following example returns the specification for the eks:k8s-metrics Role.

kubectl describe role eks:k8s-metrics -n kube-system

+ An example output is as follows.

Name:         eks:k8s-metrics
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources         Non-Resource URLs  Resource Names             Verbs
  ---------         -----------------  --------------             -----
  daemonsets.apps   []                 [aws-node]                 [get]
  deployments.apps  []                 [vpc-resource-controller]  [get]

+ RoleBindings – RoleBindings are scoped to a Kubernetes namespace. All Amazon EKS created RoleBindings are scoped to the kube-system namespace.

+ The following command returns all of the Amazon EKS created Kubernetes RoleBindings on your cluster.

kubectl get rolebindings -n kube-system | grep eks

+ To see the specification for a RoleBinding, replace eks:k8s-metrics in the following command with a RoleBinding returned in the output of the previous command. The following example returns the specification for the eks:k8s-metrics RoleBinding.

kubectl describe rolebinding eks:k8s-metrics -n kube-system

+ An example output is as follows.

Name:         eks:k8s-metrics
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  Role
  Name:  eks:k8s-metrics
Subjects:
  Kind  Name             Namespace
  ----  ----             ---------
  User  eks:k8s-metrics

14.5.3. Understand Amazon EKS created `Pod` security policies `(PSP)`

Learn about the Pod Security Policies (PSP) that Amazon EKS creates by default. PSP was deprecated in Kubernetes version 1.21 and removed in Kubernetes 1.25.

The Kubernetes Pod security policy admission controller validates Pod creation and update requests against a set of rules. By default, Amazon EKS clusters ship with a fully permissive security policy with no restrictions. For more information, see Pod Security Policies in the Kubernetes documentation.

The PodSecurityPolicy (PSP) was deprecated in Kubernetes version 1.21 and removed in Kubernetes 1.25. PSPs are being replaced with Pod Security Admission (PSA), a built-in admission controller that implements the security controls outlined in the Pod Security Standards (PSS). PSA and PSS have both reached beta feature states, and are enabled in Amazon EKS by default. To address PSP removal in 1.25, we recommend that you implement PSS in Amazon EKS. For more information, see Implementing Pod Security Standards in Amazon EKS on the AWS blog.

Amazon EKS default `Pod` security policy

Amazon EKS clusters with Kubernetes version 1.13 or higher have a default Pod security policy named eks.privileged. This policy has no restriction on what kind of Pod can be accepted into the system, which is equivalent to running Kubernetes with the PodSecurityPolicy controller disabled.

This policy was created to maintain backwards compatibility with clusters that did not have the PodSecurityPolicy controller enabled. You can create more restrictive policies for your cluster and for individual namespaces and service accounts and then delete the default policy to enable the more restrictive policies.

You can view the default policy with the following command.

kubectl get psp eks.privileged

An example output is as follows.

NAME             PRIV   CAPS   SELINUX    RUNASUSER   FSGROUP    SUPGROUP   READONLYROOTFS   VOLUMES
eks.privileged   true   *      RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            *

For more details, you can describe the policy with the following command.

kubectl describe psp eks.privileged

An example output is as follows.

Name:  eks.privileged

Settings:
  Allow Privileged:                       true
  Allow Privilege Escalation:             0xc0004ce5f8
  Default Add Capabilities:               <none>
  Required Drop Capabilities:             <none>
  Allowed Capabilities:                   *
  Allowed Volume Types:                   *
  Allow Host Network:                     true
  Allow Host Ports:                       0-65535
  Allow Host PID:                         true
  Allow Host IPC:                         true
  Read Only Root Filesystem:              false
  SELinux Context Strategy: RunAsAny
    User:                                 <none>
    Role:                                 <none>
    Type:                                 <none>
    Level:                                <none>
  Run As User Strategy: RunAsAny
    Ranges:                               <none>
  FSGroup Strategy: RunAsAny
    Ranges:                               <none>
  Supplemental Groups Strategy: RunAsAny
    Ranges:                               <none>

You can view the full YAML file for the eks.privileged Pod security policy, its cluster role, and cluster role binding in Install or restore the default Pod security policy.

Delete the default Amazon EKS `Pod` security policy

If you create more restrictive policies for your Pods, then after doing so, you can delete the default Amazon EKS eks.privileged Pod security policy to enable your custom policies.

If you are using version 1.7.0 or later of the CNI plugin and you assign a custom Pod security policy to the aws-node Kubernetes service account used for the aws-node Pods deployed by the Daemonset, then the policy must have NET_ADMIN in its allowedCapabilities section along with hostNetwork: true and privileged: true in the policy’s spec.

Create a file named privileged-podsecuritypolicy.yaml with the contents in the example file in Install or restore the default Pod security policy.
Delete the YAML with the following command. This deletes the default Pod security policy, the ClusterRole, and the ClusterRoleBinding associated with it.
```
kubectl delete -f privileged-podsecuritypolicy.yaml
```

Install or restore the default `Pod` security policy

If you are upgrading from an earlier version of Kubernetes, or have modified or deleted the default Amazon EKS eks.privileged Pod security policy, you can restore it with the following steps.

Create a file called privileged-podsecuritypolicy.yaml with the following contents.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: eks.privileged
  annotations:
    kubernetes.io/description: 'privileged allows full unrestricted access to
      Pod features, as if the PodSecurityPolicy controller was not enabled.'
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  labels:
    kubernetes.io/cluster-service: "true"
    eks.amazonaws.com/component: pod-security-policy
spec:
  privileged: true
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  volumes:
  - '*'
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  hostIPC: true
  hostPID: true
  runAsUser:
    rule: 'RunAsAny'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  fsGroup:
    rule: 'RunAsAny'
  readOnlyRootFilesystem: false

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks:podsecuritypolicy:privileged
  labels:
    kubernetes.io/cluster-service: "true"
    eks.amazonaws.com/component: pod-security-policy
rules:
- apiGroups:
  - policy
  resourceNames:
  - eks.privileged
  resources:
  - podsecuritypolicies
  verbs:
  - use

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks:podsecuritypolicy:authenticated
  annotations:
    kubernetes.io/description: 'Allow all authenticated users to create privileged Pods.'
  labels:
    kubernetes.io/cluster-service: "true"
    eks.amazonaws.com/component: pod-security-policy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eks:podsecuritypolicy:privileged
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated

Apply the YAML with the following command.

kubectl apply -f privileged-podsecuritypolicy.yaml

14.5.4. Migrate from legacy `Pod` security policies (PSP)

Learn about the Pod Security Policy (PSPs) removal in Kubernetes 1.25. Migrate to Pod Security Standards (PSS) or policy-as-code solutions before upgrading Amazon EKS clusters to Kubernetes 1.25 to avoid workload interruptions and maintain pod security controls.

PodSecurityPolicy was deprecated in Kubernetes1.21, and has been removed in Kubernetes 1.25. If you are using PodSecurityPolicy in your cluster, then you must migrate to the built-in Kubernetes Pod Security Standards (PSS) or to a policy-as-code solution before upgrading your cluster to version *1.25 to avoid interruptions to your workloads.* Select any frequently asked question to learn more.

What is a PSP?

PodSecurityPolicy is a built-in admission controller that allows a cluster administrator to control security-sensitive aspects of Pod specification. If a Pod meets the requirements of its PSP, the Pod is admitted to the cluster as usual. If a Pod doesn’t meet the PSP requirements, the Pod is rejected and can’t run.

Is the PSP removal specific to Amazon EKS or is it being removed in upstream Kubernetes?

This is an upstream change in the Kubernetes project, and not a change made in Amazon EKS. PSP was deprecated in Kubernetes 1.21 and removed in Kubernetes 1.25. The Kubernetes community identified serious usability problems with PSP. These included accidentally granting broader permissions than intended and difficulty in inspecting which PSPs apply in a given situation. These issues couldn’t be addressed without making breaking changes. This is the primary reason why the Kubernetes community decided to remove PSP.

How can I check if I’m using PSPs in my Amazon EKS clusters?

To check if you’re using PSPs in your cluster, you can run the following command:

kubectl get psp

To see the Pods that the PSPs in your cluster are impacting, run the following command. This command outputs the Pod name, namespace, and PSPs:

kubectl get pod -A -o jsonpath='{range.items[?(@.metadata.annotations.kubernetes\.io/psp)]}{.metadata.name}{"	"}{.metadata.namespace}{"	"}{.metadata.annotations.kubernetes\.io/psp}{"
"}'

If I’m using PSPs in my Amazon EKS cluster, what can I do?

Before upgrading your cluster to 1.25, you must migrate your PSPs to either one of these alternatives:

Kubernetes PSS.
Policy-as-code solutions from the Kubernetes environment.

In response to the PSP deprecation and the ongoing need to control Pod security from the start, the Kubernetes community created a built-in solution with (PSS) and Pod Security Admission (PSA). The PSA webhook implements the controls that are defined in the PSS.

You can review best practices for migrating PSPs to the built-in PSS in the EKS Best Practices Guide. We also recommend reviewing our blog on Implementing Pod Security Standards in Amazon EKS. Additional references include Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller and Mapping PodSecurityPolicies to Pod Security Standards.

Policy-as-code solutions provide guardrails to guide cluster users and prevents unwanted behaviors through prescribed automated controls. Policy-as-code solutions typically use Kubernetes Dynamic Admission Controllers to intercept the Kubernetes API server request flow using a webhook call. Policy-as-code solutions mutate and validate request payloads based on policies written and stored as code.

There are several open source policy-as-code solutions available for Kubernetes. To review best practices for migrating PSPs to a policy-as-code solution, see the Policy-as-code section of the Pod Security page on GitHub.

I see a PSP called eks.privileged in my cluster. What is it and what can I do about it?

Amazon EKS clusters with Kubernetes version 1.13 or higher have a default PSP that’s named eks.privileged. This policy is created in 1.24 and earlier clusters. It isn’t used in 1.25 and later clusters. Amazon EKS automatically migrates this PSP to a PSS-based enforcement. No action is needed on your part.

Will Amazon EKS make any changes to PSPs present in my existing cluster when I update my cluster to version 1.25?

No. Besides eks.privileged, which is a PSP created by Amazon EKS, no changes are made to other PSPs in your cluster when you upgrade to 1.25.

Will Amazon EKS prevent a cluster update to version 1.25 if I haven’t migrated off of PSP?

No. Amazon EKS won’t prevent a cluster update to version 1.25 if you didn’t migrate off of PSP yet.

What if I forget to migrate my PSPs to PSS/PSA or to a policy-as-code solution before I update my cluster to version 1.25? Can I migrate after updating my cluster?

When a cluster that contains a PSP is upgraded to Kubernetes version 1.25, the API server doesn’t recognize the PSP resource in 1.25. This might result in Pods getting incorrect security scopes. For an exhaustive list of implications, see Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller.

How does this change impact pod security for Windows workloads?

We don’t expect any specific impact to Windows workloads. PodSecurityContext has a field called windowsOptions in the PodSpec v1 API for Windows Pods. This uses PSS in Kubernetes 1.25. For more information and best practices about enforcing PSS for Windows workloads, see the EKS Best Practices Guide and Kubernetes documentation.

14.5.5. Encrypt Kubernetes secrets with `AWS` KMS on existing clusters

Learn how to enable Kubernetes secrets encryption with AWS KMS on an existing Amazon EKS cluster, ensuring secure storage of sensitive data.

If you enable secrets encryption, the Kubernetes secrets are encrypted using the AWS KMS key that you select. The KMS key must meet the following conditions:

Symmetric
Can encrypt and decrypt data
Created in the same AWS Region as the cluster
If the KMS key was created in a different account, the IAM principal must have access to the KMS key.

For more information, see Allowing IAM principals in other accounts to use a KMS key in the AWS Key Management Service Developer Guide.

You can’t disable secrets encryption after enabling it. This action is irreversible.

eksctl

You can enable encryption in two ways:

Add encryption to your cluster with a single command.

To automatically re-encrypt your secrets, run the following command.

eksctl utils enable-secrets-encryption \
    --cluster my-cluster \
    --key-arn region.arnkms:region-code:account:key/key

To opt-out of automatically re-encrypting your secrets, run the following command.

eksctl utils enable-secrets-encryption
    --cluster my-cluster \
    --key-arn region.arnkms:region-code:account:key/key \
    --encrypt-existing-secrets=false

Add encryption to your cluster with a kms-cluster.yaml file.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my-cluster
  region: region-code

secretsEncryption:
  keyARN: region.arnkms:region-code:account:key/key

To have your secrets re-encrypt automatically, run the following command.

eksctl utils enable-secrets-encryption -f kms-cluster.yaml

To opt out of automatically re-encrypting your secrets, run the following command.

eksctl utils enable-secrets-encryption -f kms-cluster.yaml --encrypt-existing-secrets=false

consolelong

Open the Amazon EKS console.
Choose the cluster that you want to add KMS encryption to.
Choose the Overview tab (this is selected by default).
Scroll down to the Secrets encryption section and choose Enable.
Select a key from the dropdown list and choose the Enable button. If no keys are listed, you must create one first. For more information, see Creating keys
Choose the Confirm button to use the chosen key.

AWS CLI

Associate the secrets encryption configuration with your cluster using the following AWS CLI command. Replace the example values with your own.

aws eks associate-encryption-config \
    --cluster-name my-cluster \
    --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"region.arnkms:region-code:account:key/key"}}]'

An example output is as follows.

{
  "update": {
    "id": "3141b835-8103-423a-8e68-12c2521ffa4d",
    "status": "InProgress",
    "type": "AssociateEncryptionConfig",
    "params": [
      {
        "type": "EncryptionConfig",
        "value": "[{\"resources\":[\"secrets\"],\"provider\":{\"keyArn\":\"region.arnkms:region-code:account:key/key\"}}]"
      }
    ],
    "createdAt": 1613754188.734,
    "errors": []
  }
}

You can monitor the status of your encryption update with the following command. Use the specific cluster name and update ID that was returned in the previous output. When a Successful status is displayed, the update is complete.

aws eks describe-update \
    --region region-code \
    --name my-cluster \
    --update-id 3141b835-8103-423a-8e68-12c2521ffa4d

An example output is as follows.

{
  "update": {
    "id": "3141b835-8103-423a-8e68-12c2521ffa4d",
    "status": "Successful",
    "type": "AssociateEncryptionConfig",
    "params": [
      {
        "type": "EncryptionConfig",
        "value": "[{\"resources\":[\"secrets\"],\"provider\":{\"keyArn\":\"region.arnkms:region-code:account:key/key\"}}]"
      }
    ],
    "createdAt": 1613754188.734>,
    "errors": []
  }
}

To verify that encryption is enabled in your cluster, run the describe-cluster command. The response contains an EncryptionConfig string.
```
aws eks describe-cluster --region region-code --name my-cluster
```

After you enabled encryption on your cluster, you must encrypt all existing secrets with the new key:

If you use eksctl, running the following command is necessary only if you opt out of re-encrypting your secrets automatically.

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - kms-encryption-timestamp="time value"

If you enable secrets encryption for an existing cluster and the KMS key that you use is ever deleted, then there’s no way to recover the cluster. If you delete the KMS key, you permanently put the cluster in a degraded state. For more information, see Deleting AWS KMS keys.

By default, the create-key command creates a symmetric encryption KMS key with a key policy that gives the account root admin access on AWS KMS actions and resources. If you want to scope down the permissions, make sure that the kms:DescribeKey and kms:CreateGrant actions are permitted on the policy for the principal that calls the create-cluster API.

For clusters using KMS Envelope Encryption, kms:CreateGrant permissions are required. The condition kms:GrantIsForAWSResource is not supported for the CreateCluster action, and should not be used in KMS policies to control kms:CreateGrant permissions for users performing CreateCluster.

14.5.6. Use `AWS` Secrets Manager secrets with Amazon EKS Pods

To show secrets from Secrets Manager and parameters from Parameter Store as files mounted in Amazon EKS Pods, you can use the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver.

With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on Amazon EKS. You can use IAM roles and policies to limit access to your secrets to specific Kubernetes Pods in a cluster. The ASCP retrieves the Pod identity and exchanges the identity for an IAM role. ASCP assumes the IAM role of the Pod, and then it can retrieve secrets from Secrets Manager that are authorized for that role.

If you use Secrets Manager automatic rotation for your secrets, you can also use the Secrets Store CSI Driver rotation reconciler feature to ensure you are retrieving the latest secret from Secrets Manager.

AWS Fargate (Fargate) node groups are not supported.

For more information, see Using Secrets Manager secrets in Amazon EKS in the AWS Secrets Manager User Guide.

Configure Kubernetes to meet your security and compliance objectives, and learn how to use other AWS services that help you to secure your Kubernetes resources.

The following are considerations for security in the cloud, as they affect Kubernetes in Amazon EKS clusters. For an in-depth review of security controls and practices in Kubernetes, see Cloud Native Security and Kubernetes in the Kubernetes documentation.

[[Topic List]]

14.6. Security considerations for Amazon EKS Auto Mode

This topic describes the security architecture, controls, and best practices for Amazon EKS Auto Mode. As organizations deploy containerized applications at scale, maintaining a strong security posture becomes increasingly complex. EKS Auto Mode implements automated security controls and integrates with AWS security services to help you protect your cluster infrastructure, workloads, and data. Through built-in security features like enforced node lifecycle management and automated patch deployment, EKS Auto Mode helps you maintain security best practices while reducing operational overhead.

Before proceeding with this topic, make sure that you’re familiar with basic EKS Auto Mode concepts and have reviewed the prerequisites for enabling EKS Auto Mode on your clusters. For general information about Amazon EKS security, see security.title.

Amazon EKS Auto Mode builds upon the existing security foundations of Amazon EKS while introducing additional automated security controls for EC2 managed instances.

14.6.1. API security and authentication

Amazon EKS Auto Mode uses AWS platform security mechanisms to secure and authenticate calls to the Amazon EKS API.

Access to the Kubernetes API is secured through EKS access entries, which integrate with AWS IAM identities.
- For more information, see Grant IAM users access to Kubernetes with EKS access entries.
Customers can implement fine-grained access control to the Kubernetes API endpoint through configuration of EKS access entries.

14.6.2. Network security

Amazon EKS Auto Mode supports multiple layers of network security:

VPC integration
- Operates within your Amazon Virtual Private Cloud (VPC)
- Supports custom VPC configurations and subnet layouts
- Enables private networking between cluster components
- For more information, see Managing security responsibilities for Amazon Virtual Private Cloud
Network Policies
- Native support for Kubernetes Network Policies
- Ability to define granular network traffic rules
- For more information, see Limit pod traffic with Kubernetes network policies

14.6.3. EC2 managed instance security

Amazon EKS Auto Mode operates EC2 managed instances with the following security controls:

EC2 security

EC2 managed instances maintain the security features of Amazon EC2.
For more information about EC2 managed instances, see Security in Amazon EC2.

Instance lifecycle management

EC2 managed instances operated by EKS Auto Mode have maximum lifetime of 21 days. Amazon EKS Auto Mode automatically terminates instances exceeding this lifetime. This lifecycle limit helps prevent configuration drift and maintains security posture.

Data protection

Amazon EC2 Instance Storage is encrypted, this is storage directly attached to the instance. For more information, see Data protection in Amazon EC2.
EKS Auto Mode manages the volumes attached to EC2 instances at creation time, including root and data volumes. EKS Auto Mode does not fully manage EBS volumes created using Kubernetes persistent storage features.

Patch management

Amazon EKS Auto Mode automatically applies patches to managed instances.
Patches include:
- Operating system updates
- Security patches
- Amazon EKS Auto Mode components

Customers retain responsibility for securing and updating workloads running on these instances.

Access controls

Direct instance access is restricted:
- SSH access is not available.
- AWS Systems Manager Session Manager (SSM) access is not available.
Management operations are performed through the Amazon EKS API and Kubernetes API.

14.6.4. Automated resource management

Amazon EKS Auto Mode does not fully manage Amazon Elastic Block Store (Amazon EBS) Volumes created using Kubernetes persistent storage features. EKS Auto Mode also does not manage Elastic Load Balancers (ELB). Amazon EKS Auto Mode automates routine tasks for these resources.

Storage security

AWS recommends that you enable encryption for EBS Volumes provisionsed by Kubernetes persistent storage features. For more information, see create-storage-class.title.
Encryption at rest using AWS KMS
You can configure your AWS account to enforce the encryption of the new EBS volumes and snapshot copies that you create. For more information, see Enable Amazon EBS encryption by default in the Amazon EBS User Guide.
For more information, see Security in Amazon EBS.

Load balancer security

Automated configuration of Elastic Load Balancers
SSL/TLS certificate management through AWS Certificate Manager integration
Security group automation for load balancer access control
For more information, see Security in Elastic Load Balancing.

14.6.5. Security best practices

The following section describes security best practices for Amazon EKS Auto Mode.

Regularly review AWS IAM policies and EKS access entries.
Implement least privilege access patterns for workloads.
Monitor cluster activity through AWS CloudTrail and Amazon CloudWatch. For more information, see Log API calls as CloudTrail events and Monitor cluster data with Amazon CloudWatch.
Use AWS Security Hub for security posture assessment.
Implement pod security standards appropriate for your workloads.

14.7. Identity and access management for Amazon EKS

14.7.1. How Amazon EKS works with IAM

Before you use IAM to manage access to Amazon EKS, you should understand what IAM features are available to use with Amazon EKS. To get a high-level view of how Amazon EKS and other AWS services work with IAM, see AWS services that work with IAM in the IAM User Guide.

[[Topic List]]

Amazon EKS identity-based policies

With IAM identity-based policies, you can specify allowed or denied actions and resources as well as the conditions under which actions are allowed or denied. Amazon EKS supports specific actions, resources, and condition keys. To learn about all of the elements that you use in a JSON policy, see IAM JSON policy elements reference in the IAM User Guide.

Actions

Administrators can use AWS JSON policies to specify who has access to what. That is, which principal can perform actions on what resources, and under what conditions.

The Action element of a JSON policy describes the actions that you can use to allow or deny access in a policy. Policy actions usually have the same name as the associated AWS API operation. There are some exceptions, such as permission-only actions that don’t have a matching API operation. There are also some operations that require multiple actions in a policy. These additional actions are called dependent actions.

Include actions in a policy to grant permissions to perform the associated operation.

Policy actions in Amazon EKS use the following prefix before the action: eks:. For example, to grant someone permission to get descriptive information about an Amazon EKS cluster, you include the DescribeCluster action in their policy. Policy statements must include either an Action or NotAction element.

To specify multiple actions in a single statement, separate them with commas as follows:

"Action": ["eks:action1", "eks:action2"]

You can specify multiple actions using wildcards (*). For example, to specify all actions that begin with the word Describe, include the following action:

"Action": "eks:Describe*"

To see a list of Amazon EKS actions, see Actions defined by Amazon Elastic Kubernetes Service in the Service Authorization Reference.

Resources

Administrators can use AWS JSON policies to specify who has access to what. That is, which principal can perform actions on what resources, and under what conditions.

The Resource JSON policy element specifies the object or objects to which the action applies. Statements must include either a Resource or a NotResource element. As a best practice, specify a resource using its Amazon Resource Name (ARN). You can do this for actions that support a specific resource type, known as resource-level permissions.

For actions that don’t support resource-level permissions, such as listing operations, use a wildcard (*) to indicate that the statement applies to all resources.

"Resource": "*"

The Amazon EKS cluster resource has the following ARN.

region.arneks:region-code:account-id:cluster/cluster-name

For more information about the format of ARNs, see Amazon resource names (ARNs) and AWS service namespaces.

For example, to specify the cluster with the name my-cluster in your statement, use the following ARN:

"Resource": "region.arneks:region-code:111122223333:cluster/my-cluster"

To specify all clusters that belong to a specific account and AWS Region, use the wildcard (*):

"Resource": "region.arneks:region-code:111122223333:cluster/*"

Some Amazon EKS actions, such as those for creating resources, can’t be performed on a specific resource. In those cases, you must use the wildcard (*).

"Resource": "*"

To see a list of Amazon EKS resource types and their ARNs, see Resources defined by Amazon Elastic Kubernetes Service in the Service Authorization Reference. To learn with which actions you can specify the ARN of each resource, see Actions defined by Amazon Elastic Kubernetes Service.

Condition keys

Amazon EKS defines its own set of condition keys and also supports using some global condition keys. To see all AWS global condition keys, see AWS Global Condition Context Keys in the IAM User Guide.

You can set condition keys when associating an OpenID Connect provider to your cluster. For more information, see oidc-identity-provider-iam-policy.title.

All Amazon EC2 actions support the aws:RequestedRegion and ec2:Region condition keys. For more information, see Example: Restricting Access to a Specific AWS Region.

For a list of Amazon EKS condition keys, see Conditions defined by Amazon Elastic Kubernetes Service in the Service Authorization Reference. To learn which actions and resources you can use a condition key with, see Actions defined by Amazon Elastic Kubernetes Service.

Examples

To view examples of Amazon EKS identity-based policies, see security-iam-id-based-policy-examples.title.

For more information about working with the ConfigMap, see grant-k8s-access.title.

Amazon EKS resource-based policies

Amazon EKS does not support resource-based policies.

Authorization based on Amazon EKS tags

You can attach tags to Amazon EKS resources or pass tags in a request to Amazon EKS. To control access based on tags, you provide tag information in the condition element of a policy using the aws:ResourceTag/key-name, aws:RequestTag/key-name, or aws:TagKeys condition keys. For more information about tagging Amazon EKS resources, see eks-using-tags.title. For more information about which actions that you can use tags in condition keys with, see Actions defined by Amazon EKS in the Service Authorization Reference.

Amazon EKS IAM roles

An IAM role is an entity within your AWS account that has specific permissions.

Using temporary credentials with Amazon EKS

You can use temporary credentials to sign in with federation, assume an IAM role, or to assume a cross-account role. You obtain temporary security credentials by calling AWS STS API operations such as AssumeRole or GetFederationToken.

Amazon EKS supports using temporary credentials.

Service-linked roles

link:IAM/latest/UserGuide/id_roles.html#iam-term-service-linked-role[Service-linked roles,type="documentation"] allow {aws} services to access resources in other services to complete an action on your behalf. Service-linked roles appear in your IAM account and are owned by the service. An administrator can view but can't edit the permissions for service-linked roles.

Amazon EKS supports service-linked roles. For details about creating or managing Amazon EKS service-linked roles, see using-service-linked-roles.title.

Service roles

This feature allows a service to assume a service role on your behalf. This role allows the service to access resources in other services to complete an action on your behalf. Service roles appear in your IAM account and are owned by the account. This means that an IAM administrator can change the permissions for this role. However, doing so might break the functionality of the service.

Amazon EKS supports service roles. For more information, see cluster-iam-role.title and create-node-role.title.

Choosing an IAM role in Amazon EKS

When you create a cluster resource in Amazon EKS, you must choose a role to allow Amazon EKS to access several other AWS resources on your behalf. If you have previously created a service role, then Amazon EKS provides you with a list of roles to choose from. It’s important to choose a role that has the Amazon EKS managed policies attached to it. For more information, see check-service-role.title and check-worker-node-role.title.

14.7.2. Amazon EKS identity-based policy examples

By default, IAM users and roles don’t have permission to create or modify Amazon EKS resources. They also can’t perform tasks using the consolelong, AWS CLI, or AWS API. An IAM administrator must create IAM policies that grant users and roles permission to perform specific API operations on the specified resources they need. The administrator must then attach those policies to the IAM users or groups that require those permissions.

To learn how to create an IAM identity-based policy using these example JSON policy documents, see Creating policies on the JSON tab in the IAM User Guide.

For more information about working with the ConfigMap, see grant-k8s-access.title.

[[Topic List]]

Policy best practices

Identity-based policies determine whether someone can create, access, or delete Amazon EKS resources in your account. These actions can incur costs for your AWS account. When you create or edit identity-based policies, follow these guidelines and recommendations:

Get started with AWS managed policies and move toward least-privilege permissions – To get started granting permissions to your users and workloads, use the AWS managed policies that grant permissions for many common use cases. They are available in your AWS account. We recommend that you reduce permissions further by defining AWS customer managed policies that are specific to your use cases. For more information, see AWS managed policies or AWS managed policies for job functions in the IAM User Guide.
Apply least-privilege permissions – When you set permissions with IAM policies, grant only the permissions required to perform a task. You do this by defining the actions that can be taken on specific resources under specific conditions, also known as least-privilege permissions. For more information about using IAM to apply permissions, see Policies and permissions in IAM in the IAM User Guide.
Use conditions in IAM policies to further restrict access – You can add a condition to your policies to limit access to actions and resources. For example, you can write a policy condition to specify that all requests must be sent using SSL. You can also use conditions to grant access to service actions if they are used through a specific AWS service, such as AWS CloudFormation. For more information, see IAM JSON policy elements: Condition in the IAM User Guide.
Use IAM Access Analyzer to validate your IAM policies to ensure secure and functional permissions – IAM Access Analyzer validates new and existing policies so that the policies adhere to the IAM policy language (JSON) and IAM best practices. IAM Access Analyzer provides more than 100 policy checks and actionable recommendations to help you author secure and functional policies. For more information, see IAM Access Analyzer policy validation in the IAM User Guide.
Require multi-factor authentication (MFA) – If you have a scenario that requires IAM users or a root user in your AWS account, turn on MFA for additional security. To require MFA when API operations are called, add MFA conditions to your policies. For more information, see Configuring MFA-protected API access in the IAM User Guide.

For more information about best practices in IAM, see Security best practices in IAM in the IAM User Guide.

Using the Amazon EKS console

To access the Amazon EKS console, an IAM principal, must have a minimum set of permissions. These permissions allow the principal to list and view details about the Amazon EKS resources in your AWS account. If you create an identity-based policy that is more restrictive than the minimum required permissions, the console won’t function as intended for principals with that policy attached to them.

To ensure that your IAM principals can still use the Amazon EKS console, create a policy with your own unique name, such as AmazonEKSAdminPolicy. Attach the policy to the principals. For more information, see Adding and removing IAM identity permissions in the IAM User Guide.

The following example policy allows a principal to view information on the Configuration tab in the console. To view information on the Overview and Resources tabs in the consolelong, the principal also needs Kubernetes permissions. For more information, see view-kubernetes-resources-permissions.title.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": "eks.amazonaws.com"
                }
            }
        }
    ]
}

You don’t need to allow minimum console permissions for principals that are making calls only to the AWS CLI or the AWS API. Instead, allow access to only the actions that match the API operation that you’re trying to perform.

Allow IAM users to view their own permissions

This example shows how you might create a policy that allows IAM users to view the inline and managed policies that are attached to their user identity. This policy includes permissions to complete this action on the console or programmatically using the AWS CLI or AWS API.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ViewOwnUserInfo",
            "Effect": "Allow",
            "Action": [
                "iam:GetUserPolicy",
                "iam:ListGroupsForUser",
                "iam:ListAttachedUserPolicies",
                "iam:ListUserPolicies",
                "iam:GetUser"
            ],
            "Resource": ["region.arniam::*:user/${aws:username}"]
        },
        {
            "Sid": "NavigateInConsole",
            "Effect": "Allow",
            "Action": [
                "iam:GetGroupPolicy",
                "iam:GetPolicyVersion",
                "iam:GetPolicy",
                "iam:ListAttachedGroupPolicies",
                "iam:ListGroupPolicies",
                "iam:ListPolicyVersions",
                "iam:ListPolicies",
                "iam:ListUsers"
            ],
            "Resource": "*"
        }
    ]
}

Create a `Kubernetes` cluster on the `AWS` Cloud

This example policy includes the minimum permissions required to create an Amazon EKS cluster named my-cluster in the us-west-2 AWS Region. You can replace the AWS Region with the AWS Region that you want to create a cluster in. If you see a warning that says The actions in your policy do not support resource-level permissions and require you to choose All resources in the consolelong, it can be safely ignored. If your account already has the AWSServiceRoleForAmazonEKS role, you can remove the iam:CreateServiceLinkedRole action from the policy. If you’ve ever created an Amazon EKS cluster in your account then this role already exists, unless you deleted it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "eks:CreateCluster",
            "Resource": "region.arneks:us-west-2:111122223333:cluster/my-cluster"
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "region.arniam::111122223333:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS",
            "Condition": {
                "ForAnyValue:StringEquals": {
                    "iam:AWSServiceName": "eks"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "region.arniam::111122223333:role/cluster-role-name"
        }
    ]
}

Create a local `Kubernetes` cluster on an Outpost

This example policy includes the minimum permissions required to create an Amazon EKS local cluster named my-cluster on an Outpost in the us-west-2 AWS Region. You can replace the AWS Region with the AWS Region that you want to create a cluster in. If you see a warning that says The actions in your policy do not support resource-level permissions and require you to choose All resources in the consolelong, it can be safely ignored. If your account already has the AWSServiceRoleForAmazonEKSLocalOutpost role, you can remove the iam:CreateServiceLinkedRole action from the policy. If you’ve ever created an Amazon EKS local cluster on an Outpost in your account then this role already exists, unless you deleted it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "eks:CreateCluster",
            "Resource": "region.arneks:us-west-2:111122223333:cluster/my-cluster"
        },
        {
            "Action": [
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "iam:GetRole"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "region.arniam::111122223333:role/aws-service-role/outposts.eks-local.amazonaws.com/AWSServiceRoleForAmazonEKSLocalOutpost"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "iam:ListAttachedRolePolicies"
            ]
            "Resource": "region.arniam::111122223333:role/cluster-role-name"
        },
        {
            "Action": [
                "iam:CreateInstanceProfile",
                "iam:TagInstanceProfile",
                "iam:AddRoleToInstanceProfile",
                "iam:GetInstanceProfile",
                "iam:DeleteInstanceProfile",
                "iam:RemoveRoleFromInstanceProfile"
            ],
            "Resource": "region.arniam::*:instance-profile/eks-local-*",
            "Effect": "Allow"
        },
    ]
}

Update a `Kubernetes` cluster

This example policy includes the minimum permission required to update a cluster named my-cluster in the us-west-2 AWS Region.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "eks:UpdateClusterVersion",
            "Resource": "region.arneks:us-west-2:111122223333:cluster/my-cluster"
        }
    ]
}

List or describe all clusters

This example policy includes the minimum permissions required to list and describe all clusters in your account. An IAM principal must be able to list and describe clusters to use the update-kubeconfig AWS CLI command.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster",
                "eks:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}

14.7.3. Using service-linked roles for Amazon EKS

Using roles for Amazon EKS clusters

How to use service-linked roles to give Amazon EKS access to resources in your AWS account.

Amazon Elastic Kubernetes Service uses AWS Identity and Access Management (IAM) service-linked roles. A service-linked role is a unique type of IAM role that is linked directly to Amazon EKS. Service-linked roles are predefined by Amazon EKS and include all the permissions that the service requires to call other AWS services on your behalf.

A service-linked role makes setting up Amazon EKS easier because you don’t have to manually add the necessary permissions. Amazon EKS defines the permissions of its service-linked roles, and unless defined otherwise, only Amazon EKS can assume its roles. The defined permissions include the trust policy and the permissions policy, and that permissions policy cannot be attached to any other IAM entity.

You can delete a service-linked role only after first deleting their related resources. This protects your Amazon EKS resources because you can’t inadvertently remove permission to access the resources.

For information about other services that support service-linked roles, see AWS services that work with IAM and look for the services that have Yes in the Service-linked role column. Choose a Yes with a link to view the service-linked role documentation for that service.

Service-linked role permissions for Amazon EKS

Amazon EKS uses the service-linked role named AWSServiceRoleForAmazonEKS. The role allows Amazon EKS to manage clusters in your account. The attached policies allow the role to manage the following resources: network interfaces, security groups, logs, and VPCs.

The AWSServiceRoleForAmazonEKS service-linked role is distinct from the role required for cluster creation. For more information, see cluster-iam-role.title.

The AWSServiceRoleForAmazonEKS service-linked role trusts the following services to assume the role:

eks.amazonaws.com

The role permissions policy allows Amazon EKS to complete the following actions on the specified resources:

AmazonEKSServiceRolePolicy

You must configure permissions to allow an IAM entity (such as a user, group, or role) to create, edit, or delete a service-linked role. For more information, see Service-linked role permissions in the IAM User Guide.

Creating a service-linked role for Amazon EKS

You don’t need to manually create a service-linked role. When you create a cluster in the consolelong, the AWS CLI, or the AWS API, Amazon EKS creates the service-linked role for you.

If you delete this service-linked role, and then need to create it again, you can use the same process to recreate the role in your account. When you create a cluster, Amazon EKS creates the service-linked role for you again.

Editing a service-linked role for Amazon EKS

Amazon EKS does not allow you to edit the AWSServiceRoleForAmazonEKS service-linked role. After you create a service-linked role, you cannot change the name of the role because various entities might reference the role. However, you can edit the description of the role using IAM. For more information, see Editing a service-linked role in the IAM User Guide.

Deleting a service-linked role for Amazon EKS

If you no longer need to use a feature or service that requires a service-linked role, we recommend that you delete that role. That way you don’t have an unused entity that is not actively monitored or maintained. However, you must clean up your service-linked role before you can manually delete it.

Cleaning up a service-linked role

Before you can use IAM to delete a service-linked role, you must first delete any resources used by the role.

If the Amazon EKS service is using the role when you try to delete the resources, then the deletion might fail. If that happens, wait for a few minutes and try the operation again.

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
If your cluster has any node groups or Fargate profiles, you must delete them before you can delete the cluster. For more information, see delete-managed-node-group.title and delete-fargate-profile.title.
On the Clusters page, choose the cluster that you want to delete and choose Delete.
Type the name of the cluster in the deletion confirmation window, and then choose Delete.
Repeat this procedure for any other clusters in your account. Wait for all of the delete operations to finish.

Manually delete the service-linked role

Use the IAM console, the AWS CLI, or the AWS API to delete the AWSServiceRoleForAmazonEKS service-linked role. For more information, see Deleting a service-linked role in the IAM User Guide.

Supported regions for Amazon EKS service-linked roles

Amazon EKS supports using service-linked roles in all of the regions where the service is available. For more information, see Amazon EKS endpoints and quotas.

Using roles for Amazon EKS node groups

How to use service-linked roles to give Amazon EKS access to resources in your AWS account.

Amazon EKS uses AWS Identity and Access Management (IAM) service-linked roles. A service-linked role is a unique type of IAM role that is linked directly to Amazon EKS. Service-linked roles are predefined by Amazon EKS and include all the permissions that the service requires to call other AWS services on your behalf.

Service-linked role permissions for Amazon EKS

Amazon EKS uses the service-linked role named AWSServiceRoleForAmazonEKSNodegroup. The role allows Amazon EKS to manage node groups in your account. The attached AWSServiceRoleForAmazonEKSNodegroup policy allows the role to manage the following resources: Auto Scaling groups, security groups, launch templates, and IAM instance profiles. For more information, see security-iam-awsmanpol-awsserviceroleforamazoneksnodegroup.title.

The AWSServiceRoleForAmazonEKSNodegroup service-linked role trusts the following services to assume the role:

eks-nodegroup.amazonaws.com

The role permissions policy allows Amazon EKS to complete the following actions on the specified resources:

AWSServiceRoleForAmazonEKSNodegroup

Creating a service-linked role for Amazon EKS

You don’t need to manually create a service-linked role. When you CreateNodegroup in the consolelong, the AWS CLI, or the AWS API, Amazon EKS creates the service-linked role for you.

This service-linked role can appear in your account if you completed an action in another service that uses the features supported by this role. If you were using the Amazon EKS service before January 1, 2017, when it began supporting service-linked roles, then Amazon EKS created the AWSServiceRoleForAmazonEKSNodegroup role in your account. To learn more, see A new role appeared in my IAM account.

Creating a service-linked role in Amazon EKS (AWS API)

You don’t need to manually create a service-linked role. When you create a managed node group in the consolelong, the AWS CLI, or the AWS API, Amazon EKS creates the service-linked role for you.

If you delete this service-linked role, and then need to create it again, you can use the same process to recreate the role in your account. When you create another managed node group, Amazon EKS creates the service-linked role for you again.

Editing a service-linked role for Amazon EKS

Amazon EKS does not allow you to edit the AWSServiceRoleForAmazonEKSNodegroup service-linked role. After you create a service-linked role, you cannot change the name of the role because various entities might reference the role. However, you can edit the description of the role using IAM. For more information, see Editing a service-linked role in the IAM User Guide.

Deleting a service-linked role for Amazon EKS

Cleaning up a service-linked role

Before you can use IAM to delete a service-linked role, you must first delete any resources used by the role.

If the Amazon EKS service is using the role when you try to delete the resources, then the deletion might fail. If that happens, wait for a few minutes and try the operation again.

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
Select the Compute tab.
In the Node groups section, choose the node group to delete.
Type the name of the node group in the deletion confirmation window, and then choose Delete.
Repeat this procedure for any other node groups in the cluster. Wait for all of the delete operations to finish.

Manually delete the service-linked role

Use the IAM console, the AWS CLI, or the AWS API to delete the AWSServiceRoleForAmazonEKSNodegroup service-linked role. For more information, see Deleting a service-linked role in the IAM User Guide.

Supported regions for Amazon EKS service-linked roles

Amazon EKS supports using service-linked roles in all of the regions where the service is available. For more information, see Amazon EKS endpoints and quotas.

Using roles for Amazon EKS Fargate profiles

How to use service-linked roles to give Amazon EKS access to resources in your AWS account.

Service-linked role permissions for Amazon EKS

Amazon EKS uses the service-linked role named AWSServiceRoleForAmazonEKSForFargate. The role allows Amazon EKS Fargate to configure VPC networking required for Fargate Pods. The attached policies allow the role to create and delete elastic network interfaces and describe elastic network Interfaces and resources.

The AWSServiceRoleForAmazonEKSForFargate service-linked role trusts the following services to assume the role:

eks-fargate.amazonaws.com

The role permissions policy allows Amazon EKS to complete the following actions on the specified resources:

AmazonEKSForFargateServiceRolePolicy

Creating a service-linked role for Amazon EKS

You don’t need to manually create a service-linked role. When you create a Fargate profile in the consolelong, the AWS CLI, or the AWS API, Amazon EKS creates the service-linked role for you.

This service-linked role can appear in your account if you completed an action in another service that uses the features supported by this role. If you were using the Amazon EKS service before December 13, 2019, when it began supporting service-linked roles, then Amazon EKS created the AWSServiceRoleForAmazonEKSForFargate role in your account. To learn more, see A New role appeared in my IAM account.

Creating a service-linked role in Amazon EKS (AWS API)

You don’t need to manually create a service-linked role. When you create a Fargate profile in the consolelong, the AWS CLI, or the AWS API, Amazon EKS creates the service-linked role for you.

Editing a service-linked role for Amazon EKS

Amazon EKS does not allow you to edit the AWSServiceRoleForAmazonEKSForFargate service-linked role. After you create a service-linked role, you cannot change the name of the role because various entities might reference the role. However, you can edit the description of the role using IAM. For more information, see Editing a service-linked role in the IAM User Guide.

Deleting a service-linked role for Amazon EKS

Cleaning up a service-linked role

Before you can use IAM to delete a service-linked role, you must first delete any resources used by the role.

If the Amazon EKS service is using the role when you try to delete the resources, then the deletion might fail. If that happens, wait for a few minutes and try the operation again.

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
On the Clusters page, select your cluster.
Select the Compute tab.
If there are any Fargate profiles in the Fargate profiles section, select each one individually, and then choose Delete.
Type the name of the profile in the deletion confirmation window, and then choose Delete.
Repeat this procedure for any other Fargate profiles in the cluster and for any other clusters in your account.

Manually delete the service-linked role

Use the IAM console, the AWS CLI, or the AWS API to delete the AWSServiceRoleForAmazonEKSForFargate service-linked role. For more information, see Deleting a service-linked role in the IAM User Guide.

Supported regions for Amazon EKS service-linked roles

Amazon EKS supports using service-linked roles in all of the regions where the service is available. For more information, see Amazon EKS endpoints and quotas.

Using roles to connect a `Kubernetes` cluster to Amazon EKS

How to use service-linked roles to give Amazon EKS access to resources in your AWS account.

Service-linked role permissions for Amazon EKS

Amazon EKS uses the service-linked role named AWSServiceRoleForAmazonEKSConnector. The role allows Amazon EKS to connect Kubernetes clusters. The attached policies allow the role to manage necessary resources to connect to your registered Kubernetes cluster.

The AWSServiceRoleForAmazonEKSConnector service-linked role trusts the following services to assume the role:

eks-connector.amazonaws.com

The role permissions policy allows Amazon EKS to complete the following actions on the specified resources:

AmazonEKSConnectorServiceRolePolicy

Creating a service-linked role for Amazon EKS

You don’t need to manually create a service-linked role to connect a cluster. When you connect a cluster in the consolelong, the AWS CLI, eksctl, or the AWS API, Amazon EKS creates the service-linked role for you.

If you delete this service-linked role, and then need to create it again, you can use the same process to recreate the role in your account. When you connect a cluster, Amazon EKS creates the service-linked role for you again.

Editing a service-linked role for Amazon EKS

Amazon EKS does not allow you to edit the AWSServiceRoleForAmazonEKSConnector service-linked role. After you create a service-linked role, you cannot change the name of the role because various entities might reference the role. However, you can edit the description of the role using IAM. For more information, see Editing a service-linked role in the IAM User Guide.

Deleting a service-linked role for Amazon EKS

Cleaning up a service-linked role

Before you can use IAM to delete a service-linked role, you must first delete any resources used by the role.

If the Amazon EKS service is using the role when you try to delete the resources, then the deletion might fail. If that happens, wait for a few minutes and try the operation again.

Open the Amazon EKS console.
In the left navigation pane, choose Clusters.
On the Clusters page, select your cluster.
Select the Deregister tab and then select the Ok tab.

Manually delete the service-linked role

Use the IAM console, the AWS CLI, or the AWS API to delete the AWSServiceRoleForAmazonEKSConnector service-linked role. For more information, see Deleting a service-linked role in the IAM User Guide.

Using roles for Amazon EKS local clusters on Outpost

How to use service-linked roles to give Amazon EKS access to resources in your AWS account.

Service-linked role permissions for Amazon EKS

Amazon EKS uses the service-linked role named AWSServiceRoleForAmazonEKSLocalOutpost. The role allows Amazon EKS to manage local clusters in your account. The attached policies allow the role to manage the following resources: network interfaces, security groups, logs, and Amazon EC2 instances.

The AWSServiceRoleForAmazonEKSLocalOutpost service-linked role is distinct from the role required for cluster creation. For more information, see cluster-iam-role.title.

The AWSServiceRoleForAmazonEKSLocalOutpost service-linked role trusts the following services to assume the role:

outposts.eks-local.amazonaws.com

The role permissions policy allows Amazon EKS to complete the following actions on the specified resources:

AmazonEKSServiceRolePolicy

Creating a service-linked role for Amazon EKS

You don’t need to manually create a service-linked role. When you create a cluster in the consolelong, the AWS CLI, or the AWS API, Amazon EKS creates the service-linked role for you.

Editing a service-linked role for Amazon EKS

Amazon EKS does not allow you to edit the AWSServiceRoleForAmazonEKSLocalOutpost service-linked role. After you create a service-linked role, you can’t change the name of the role because various entities might reference the role. However, you can edit the description of the role using IAM. For more information, see Editing a service-linked role in the IAM User Guide.

Deleting a service-linked role for Amazon EKS

Cleaning up a service-linked role

Before you can use IAM to delete a service-linked role, you must first delete any resources used by the role.

If the Amazon EKS service is using the role when you try to delete the resources, then the deletion might fail. If that happens, wait for a few minutes and try the operation again.

Open the Amazon EKS console.
In the left navigation pane, choose Amazon EKS Clusters.
If your cluster has any node groups or Fargate profiles, you must delete them before you can delete the cluster. For more information, see delete-managed-node-group.title and delete-fargate-profile.title.
On the Clusters page, choose the cluster that you want to delete and choose Delete.
Type the name of the cluster in the deletion confirmation window, and then choose Delete.
Repeat this procedure for any other clusters in your account. Wait for all of the delete operations to finish.

Manually delete the service-linked role

Use the IAM console, the AWS CLI, or the AWS API to delete the AWSServiceRoleForAmazonEKSLocalOutpost service-linked role. For more information, see Deleting a service-linked role in the IAM User Guide.

Supported regions for Amazon EKS service-linked roles

Amazon EKS supports using service-linked roles in all of the regions where the service is available. For more information, see Amazon EKS endpoints and quotas.

How to use service-linked roles to give Amazon EKS access to resources in your AWS account.

[[Topic List]]

14.7.4. Amazon EKS `Pod` execution IAM role

The Amazon EKS Pod execution role is required to run Pods on AWS Fargate infrastructure.

When your cluster creates Pods on AWS Fargate infrastructure, the components running on the Fargate infrastructure must make calls to AWS APIs on your behalf. This is so that they can do actions such as pull container images from Amazon ECR or route logs to other AWS services. The Amazon EKS Pod execution role provides the IAM permissions to do this.

When you create a Fargate profile, you must specify a Pod execution role for the Amazon EKS components that run on the Fargate infrastructure using the profile. This role is added to the cluster’s Kubernetes Role based access control (RBAC) for authorization. This allows the kubelet that’s running on the Fargate infrastructure to register with your Amazon EKS cluster so that it can appear in your cluster as a node.

The Fargate profile must have a different IAM role than Amazon EC2 node groups.

The containers running in the Fargate Pod can’t assume the IAM permissions associated with a Pod execution role. To give the containers in your Fargate Pod permissions to access other AWS services, you must use IAM roles for service accounts.

Before you create a Fargate profile, you must create an IAM role with the AmazonEKSFargatePodExecutionRolePolicy.

Check for a correctly configured existing `Pod` execution role

You can use the following procedure to check and see if your account already has a correctly configured Amazon EKS Pod execution role. To avoid a confused deputy security problem, it’s important that the role restricts access based on SourceArn. You can modify the execution role as needed to include support for Fargate profiles on other clusters.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
On the Roles page, search the list of roles for AmazonEKSFargatePodExecutionRole. If the role doesn’t exist, see create-pod-execution-role.title to create the role. If the role does exist, choose the role.
On the AmazonEKSFargatePodExecutionRole page, do the following:
1. Choose Permissions.
2. Ensure that the AmazonEKSFargatePodExecutionRolePolicy Amazon managed policy is attached to the role.
3. Choose Trust relationships.
4. Choose Edit trust policy.
On the Edit trust policy page, verify that the trust relationship contains the following policy and has a line for Fargate profiles on your cluster. If so, choose Cancel.
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Condition": {
         "ArnLike": {
            "aws:SourceArn": "region.arneks:region-code:111122223333:fargateprofile/my-cluster/*"
         }
      },
      "Principal": {
        "Service": "eks-fargate-pods.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```
If the policy matches but doesn’t have a line specifying the Fargate profiles on your cluster, you can add the following line at the top of the ArnLike object. Replace region-code with the AWS Region that your cluster is in, 111122223333 with your account ID, and my-cluster with the name of your cluster.
```
"aws:SourceArn": "region.arneks:region-code:111122223333:fargateprofile/my-cluster/*",
```
If the policy doesn’t match, copy the full previous policy into the form and choose Update policy. Replace region-code with the AWS Region that your cluster is in. If you want to use the same role in all AWS Regions in your account, replace region-code with *. Replace 111122223333 with your account ID and my-cluster with the name of your cluster. If you want to use the same role for all clusters in your account, replace my-cluster with *.

Creating the Amazon EKS `Pod` execution role

If you don’t already have the Amazon EKS Pod execution role for your cluster, you can use the consolelong or the AWS CLI to create it.

consolelong

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.
On the Select trusted entity page, do the following:
1. In the Trusted entity type section, choose AWS service.
2. From the Use cases for other AWS services dropdown list, choose EKS.
3. Choose EKS - Fargate Pod.
4. Choose Next.
On the Add permissions page, choose Next.
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKSFargatePodExecutionRole.
2. Under Add tags (Optional), add metadata to the role by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM resources in the IAM User Guide.
3. Choose Create role.
On the Roles page, search the list of roles for AmazonEKSFargatePodExecutionRole. Choose the role.
On the AmazonEKSFargatePodExecutionRole page, do the following:
1. Choose Trust relationships.
2. Choose Edit trust policy.
On the Edit trust policy page, do the following:
1. Copy and paste the following contents into the Edit trust policy form. Replace region-code with the AWS Region that your cluster is in. If you want to use the same role in all AWS Regions in your account, replace region-code with *. Replace 111122223333 with your account ID and my-cluster with the name of your cluster. If you want to use the same role for all clusters in your account, replace my-cluster with *.
  { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "ArnLike": { "aws:SourceArn": "region.arneks:region-code:111122223333:fargateprofile/my-cluster/*" } }, "Principal": { "Service": "eks-fargate-pods.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
2. Choose Update policy.

AWS CLI

Copy and paste the following contents to a file named pod-execution-role-trust-policy.json. Replace region-code with the AWS Region that your cluster is in. If you want to use the same role in all AWS Regions in your account, replace region-code with *. Replace 111122223333 with your account ID and my-cluster with the name of your cluster. If you want to use the same role for all clusters in your account, replace my-cluster with *.
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Condition": {
         "ArnLike": {
            "aws:SourceArn": "region.arneks:region-code:111122223333:fargateprofile/my-cluster/*"
         }
      },
      "Principal": {
        "Service": "eks-fargate-pods.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

Create a Pod execution IAM role.

aws iam create-role \
  --role-name AmazonEKSFargatePodExecutionRole \
  --assume-role-policy-document file://"pod-execution-role-trust-policy.json"

Attach the required Amazon EKS managed IAM policy to the role.

aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy \
  --role-name AmazonEKSFargatePodExecutionRole

14.7.5. Amazon EKS connector IAM role

You can connect Kubernetes clusters to view them in your consolelong. To connect to a Kubernetes cluster, create an IAM role.

Check for an existing EKS connector role

You can use the following procedure to check and see if your account already has the Amazon EKS connector role.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
Search the list of roles for AmazonEKSConnectorAgentRole. If a role that includes AmazonEKSConnectorAgentRole doesn’t exist, then see create-connector-role.title to create the role. If a role that includes AmazonEKSConnectorAgentRole does exist, then select the role to view the attached policies.
Choose Permissions.
Ensure that the AmazonEKSConnectorAgentPolicy managed policy is attached to the role. If the policy is attached, your Amazon EKS connector role is properly configured.
Choose Trust relationships, and then choose Edit trust policy.

Verify that the trust relationship contains the following policy. If the trust relationship matches the following policy, choose Cancel. If the trust relationship doesn’t match, copy the policy into the Edit trust policy window and choose Update policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ssm.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Creating the Amazon EKS connector agent role

You can use the consolelong or AWS CloudFormation to create the connector agent role.

AWS CLI

Create a file named eks-connector-agent-trust-policy.json that contains the following JSON to use for the IAM role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ssm.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Create a file named eks-connector-agent-policy.json that contains the following JSON to use for the IAM role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SsmControlChannel",
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateControlChannel"
            ],
            "Resource": "region.arneks:*:*:cluster/*"
        },
        {
            "Sid": "ssmDataplaneOperations",
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenDataChannel",
                "ssmmessages:OpenControlChannel"
            ],
            "Resource": "*"
        }
    ]
}

Create the Amazon EKS Connector agent role using the trust policy and policy you created in the previous list items.

aws iam create-role \
     --role-name AmazonEKSConnectorAgentRole \
     --assume-role-policy-document file://eks-connector-agent-trust-policy.json

Attach the policy to your Amazon EKS Connector agent role.

aws iam put-role-policy \
     --role-name AmazonEKSConnectorAgentRole \
     --policy-name AmazonEKSConnectorAgentPolicy \
     --policy-document file://eks-connector-agent-policy.json

AWS CloudFormation

Save the following AWS CloudFormation template to a text file on your local system.

This template also creates the service-linked role that would otherwise be created when the registerCluster API is called. See using-service-linked-roles-eks-connector.title for details.

---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Provisions necessary resources needed to register clusters in EKS'
Parameters: {}
Resources:
  EKSConnectorSLR:
    Type: AWS::IAM::ServiceLinkedRole
    Properties:
      AWSServiceName: eks-connector.amazonaws.com

  EKSConnectorAgentRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: [ 'sts:AssumeRole' ]
            Principal:
              Service: 'ssm.amazonaws.com'

  EKSConnectorAgentPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: EKSConnectorAgentPolicy
      Roles:
        - {Ref: 'EKSConnectorAgentRole'}
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: 'Allow'
            Action: [ 'ssmmessages:CreateControlChannel' ]
            Resource:
            - Fn::Sub: 'arn:${AWS::Partition}:eks:*:*:cluster/*'
          - Effect: 'Allow'
            Action: [ 'ssmmessages:CreateDataChannel', 'ssmmessages:OpenDataChannel', 'ssmmessages:OpenControlChannel' ]
            Resource: "*"
Outputs:
  EKSConnectorAgentRoleArn:
    Description: The agent role that EKS connector uses to communicate with AWS services.
    Value: !GetAtt EKSConnectorAgentRole.Arn

Open the AWS CloudFormation console.
Choose Create stack with new resources (standard).
For Specify template, select Upload a template file, and then choose Choose file.
Choose the file you created earlier, and then choose Next.
For Stack name, enter a name for your role, such as eksConnectorAgentRole, and then choose Next.
On the Configure stack options page, choose Next.
On the Review page, review your information, acknowledge that the stack might create IAM resources, and then choose Create stack.

14.7.6. `AWS` managed policies for Amazon Elastic Kubernetes Service

Learn about AWS managed policies for Amazon EKS and recent changes to those policies.

An AWS managed policy is a standalone policy that is created and administered by AWS. AWS managed policies are designed to provide permissions for many common use cases so that you can start assigning permissions to users, groups, and roles.

Keep in mind that AWS managed policies might not grant least-privilege permissions for your specific use cases because they’re available for all AWS customers to use. We recommend that you reduce permissions further by defining customer managed policies that are specific to your use cases.

You cannot change the permissions defined in AWS managed policies. If AWS updates the permissions defined in an AWS managed policy, the update affects all principal identities (users, groups, and roles) that the policy is attached to. AWS is most likely to update an AWS managed policy when a new AWS service is launched or new API operations become available for existing services.

For more information, see AWS managed policies in the IAM User Guide.

`AWS` managed policy: AmazonEKS_CNI_Policy

You can attach the AmazonEKS_CNI_Policy to your IAM entities. Before you create an Amazon EC2 node group, this policy must be attached to either the node IAM role, or to an IAM role that’s used specifically by the Amazon VPC CNI plugin for Kubernetes. This is so that it can perform actions on your behalf. We recommend that you attach the policy to a role that’s used only by the plugin. For more information, see managing-vpc-cni.title and cni-iam-role.title.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

ec2:*NetworkInterface and ec2:*PrivateIpAddresses – Allows the Amazon VPC CNI plugin to perform actions such as provisioning Elastic Network Interfaces and IP addresses for Pods to provide networking for applications that run in Amazon EKS.
ec2 read actions – Allows the Amazon VPC CNI plugin to perform actions such as describe instances and subnets to see the amount of free IP addresses in your Amazon VPC subnets. The VPC CNI can use the free IP addresses in each subnet to pick the subnets with the most free IP addresses to use when creating an elastic network interface.

To view the latest version of the JSON policy document, see AmazonEKS_CNI_Policy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSClusterPolicy

You can attach AmazonEKSClusterPolicy to your IAM entities. Before creating a cluster, you must have a cluster IAM role with this policy attached. Kubernetes clusters that are managed by Amazon EKS make calls to other AWS services on your behalf. They do this to manage the resources that you use with the service.

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

autoscaling – Read and update the configuration of an Auto Scaling group. These permissions aren’t used by Amazon EKS but remain in the policy for backwards compatibility.
ec2 – Work with volumes and network resources that are associated to Amazon EC2 nodes. This is required so that the Kubernetes control plane can join instances to a cluster and dynamically provision and manage Amazon EBS volumes that are requested by Kubernetes persistent volumes.
elasticloadbalancing – Work with Elastic Load Balancers and add nodes to them as targets. This is required so that the Kubernetes control plane can dynamically provision Elastic Load Balancers requested by Kubernetes services.
iam – Create a service-linked role. This is required so that the Kubernetes control plane can dynamically provision Elastic Load Balancers that are requested by Kubernetes services.
kms – Read a key from AWS KMS. This is required for the Kubernetes control plane to support secrets encryption of Kubernetes secrets stored in etcd.

To view the latest version of the JSON policy document, see AmazonEKSClusterPolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSFargatePodExecutionRolePolicy

You can attach AmazonEKSFargatePodExecutionRolePolicy to your IAM entities. Before you can create a Fargate profile, you must create a Fargate Pod execution role and attach this policy to it. For more information, see fargate-sg-pod-execution-role.title and fargate-profile.title.

This policy grants the role the permissions that provide access to other AWS service resources that are required to run Amazon EKS Pods on Fargate.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

ecr – Allows Pods that are running on Fargate to pull container images that are stored in Amazon ECR.

To view the latest version of the JSON policy document, see AmazonEKSFargatePodExecutionRolePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSForFargateServiceRolePolicy

You can’t attach AmazonEKSForFargateServiceRolePolicy to your IAM entities. This policy is attached to a service-linked role that allows Amazon EKS to perform actions on your behalf. For more information, see AWSServiceRoleforAmazonEKSForFargate.

This policy grants necessary permissions to Amazon EKS to run Fargate tasks. The policy is only used if you have Fargate nodes.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks.

ec2 – Create and delete Elastic Network Interfaces and describe Elastic Network Interfaces and resources. This is required so that the Amazon EKS Fargate service can configure the VPC networking that’s required for Fargate Pods.

To view the latest version of the JSON policy document, see AmazonEKSForFargateServiceRolePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSComputePolicy

You can attach AmazonEKSComputePolicy to your IAM entities. You may attach this policy to your cluster IAM role to expand the resources EKS can manage in your account.

This policy grants the permissions required for Amazon EKS to create and manage EC2 instances for the EKS cluster, as well as the necessary IAM permissions to configure EC2.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

ec2 Permissions:
- ec2:CreateFleet and ec2:RunInstances - Allows creating EC2 instances and using specific EC2 resources (images, security groups, subnets) for EKS cluster nodes.
- ec2:CreateLaunchTemplate - Allows creating EC2 launch templates for EKS cluster nodes.
- The policy also includes conditions to restrict the use of these EC2 permissions to resources tagged with the EKS cluster name and other relevant tags.
- ec2:CreateTags - Allows adding tags to EC2 resources created by the CreateFleet, RunInstances, and CreateLaunchTemplate actions.
iam Permissions:
- iam:AddRoleToInstanceProfile - Allows adding an IAM role to the EKS compute instance profile.
- iam:PassRole - Allows passing the necessary IAM roles to the EC2 service.

To view the latest version of the JSON policy document, see AmazonEKSComputePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSNetworkingPolicy

You can attach AmazonEKSNetworkingPolicy to your IAM entities. You may attach this policy to your cluster IAM role to expand the resources EKS can manage in your account.

This policy is designed to grant the necessary permissions for Amazon EKS to create and manage network interfaces for the EKS cluster, allowing the control plane and worker nodes to communicate and function properly.

Permissions details

This policy grants the following permissions to allow Amazon EKS to manage network interfaces for the cluster:

ec2 Network Interface Permissions:
- ec2:CreateNetworkInterface - Allows creating EC2 network interfaces.
- The policy includes conditions to restrict the use of this permission to network interfaces tagged with the EKS cluster name and the Kubernetes CNI node name.
- ec2:CreateTags - Allows adding tags to the network interfaces created by the CreateNetworkInterface action.
ec2 Network Interface Management Permissions:
- ec2:AttachNetworkInterface, ec2:DetachNetworkInterface - Allows attaching and detaching network interfaces to EC2 instances.
- ec2:UnassignPrivateIpAddresses, ec2:UnassignIpv6Addresses, ec2:AssignPrivateIpAddresses, ec2:AssignIpv6Addresses - Allows managing the IP address assignments of the network interfaces.
- These permissions are restricted to network interfaces tagged with the EKS cluster name.

To view the latest version of the JSON policy document, see AmazonEKSNetworkingPolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSBlockStoragePolicy

You can attach AmazonEKSBlockStoragePolicy to your IAM entities. You may attach this policy to your cluster IAM role to expand the resources EKS can manage in your account.

This policy grants the necessary permissions for Amazon EKS to create, manage, and maintain EC2 volumes and snapshots for the EKS cluster, enabling the control plane and worker nodes to provision and use persistent storage as required by Kubernetes workloads.

Permissions details

This IAM policy grants the following permissions to allow Amazon EKS to manage EC2 volumes and snapshots:

ec2 Volume Management Permissions:
- ec2:AttachVolume, ec2:DetachVolume, ec2:ModifyVolume, ec2:EnableFastSnapshotRestores - Allows attaching, detaching, modifying, and enabling fast snapshot restores for EC2 volumes.
- These permissions are restricted to volumes tagged with the EKS cluster name.
- ec2:CreateTags - Allows adding tags to the EC2 volumes and snapshots created by the CreateVolume and CreateSnapshot actions.
ec2 Volume Creation Permissions:
- ec2:CreateVolume - Allows creating new EC2 volumes.
- The policy includes conditions to restrict the use of this permission to volumes tagged with the EKS cluster name and other relevant tags.
- ec2:CreateSnapshot - Allows creating new EC2 volume snapshots.
- The policy includes conditions to restrict the use of this permission to snapshots tagged with the EKS cluster name and other relevant tags.

To view the latest version of the JSON policy document, see AmazonEKSBlockStoragePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSLoadBalancingPolicy

You can attach AmazonEKSLoadBalancingPolicy to your IAM entities. You may attach this policy to your cluster IAM role to expand the resources EKS can manage in your account.

This IAM policy grants the necessary permissions for Amazon EKS to work with various AWS services to manage Elastic Load Balancers (ELBs) and related resources.

Permissions details

The key permissions granted by this policy are:

elasticloadbalancing: Allows creating, modifying, and managing Elastic Load Balancers and Target Groups. This includes permissions to create, update, and delete load balancers, target groups, listeners, and rules.
ec2: Allows creating and managing security groups, which are required for the Kubernetes control plane to join instances to a cluster and manage Amazon EBS volumes. Also allows describing and listing EC2 resources such as instances, VPCs, Subnets, Security Groups, and other networking resources.
iam: Allows creating a service-linked role for Elastic Load Balancing, which is required for the Kubernetes control plane to dynamically provision ELBs.
kms: Allows reading a key from AWS KMS, which is required for the Kubernetes control plane to support encryption of Kubernetes secrets stored in etcd.
wafv2 and shield: Allows associating and disassociating Web ACLs and creating/deleting AWS Shield protections for the Elastic Load Balancers.
cognito-idp, acm, and elasticloadbalancing: Grants permissions to describe user pool clients, list and describe certificates, and describe target groups, which are required for the Kubernetes control plane to manage the Elastic Load Balancers.

The policy also includes several condition checks to ensure that the permissions are scoped to the specific EKS cluster being managed, using the eks:eks-cluster-name tag.

To view the latest version of the JSON policy document, see AmazonEKSLoadBalancingPolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSServicePolicy

You can attach AmazonEKSServicePolicy to your IAM entities. Clusters that were created before April 16, 2020, required you to create an IAM role and attach this policy to it. Clusters that were created on or after April 16, 2020, don’t require you to create a role and don’t require you to assign this policy. When you create a cluster using an IAM principal that has the iam:CreateServiceLinkedRole permission, the AWSServiceRoleforAmazonEKS service-linked role is automatically created for you. The service-linked role has the managed policy: AmazonEKSServiceRolePolicy attached to it.

This policy allows Amazon EKS to create and manage the necessary resources to operate Amazon EKS clusters.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks.

eks – Update the Kubernetes version of your cluster after you initiate an update. This permission isn’t used by Amazon EKS but remains in the policy for backwards compatibility.
ec2 – Work with Elastic Network Interfaces and other network resources and tags. This is required by Amazon EKS to configure networking that facilitates communication between nodes and the Kubernetes control plane. Read information about security groups. Update tags on security groups.
route53 – Associate a VPC with a hosted zone. This is required by Amazon EKS to enable private endpoint networking for your Kubernetes cluster API server.
logs – Log events. This is required so that Amazon EKS can ship Kubernetes control plane logs to CloudWatch.
iam – Create a service-linked role. This is required so that Amazon EKS can create the service-linked-role-permissions-eks.title service-linked role on your behalf.

To view the latest version of the JSON policy document, see AmazonEKSServicePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSServiceRolePolicy

You can’t attach AmazonEKSServiceRolePolicy to your IAM entities. This policy is attached to a service-linked role that allows Amazon EKS to perform actions on your behalf. For more information, see service-linked-role-permissions-eks.title. When you create a cluster using an IAM principal that has the iam:CreateServiceLinkedRole permission, the AWSServiceRoleforAmazonEKS service-linked role is automatically created for you and this policy is attached to it.

This policy allows the service-linked role to call AWS services on your behalf.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks.

ec2 – Create and describe Elastic Network Interfaces and Amazon EC2 instances, the cluster security group, and VPC that are required to create a cluster. For more information, see sec-group-reqs.title. Read information about security groups. Update tags on security groups.
iam – List all of the managed policies that attached to an IAM role. This is required so that Amazon EKS can list and validate all managed policies and permissions required to create a cluster.
Associate a VPC with a hosted zone – This is required by Amazon EKS to enable private endpoint networking for your Kubernetes cluster API server.
Log event – This is required so that Amazon EKS can ship Kubernetes control plane logs to CloudWatch.
Put metric – This is required so that Amazon EKS can ship Kubernetes control plane logs to CloudWatch.
eks - Manage cluster access entries and policies, allowing fine-grained control over who can access EKS resources and what actions they can perform. This includes associating standard access policies for compute, networking, load balancing, and storage operations.
elasticloadbalancing - Create, manage, and delete load balancers and their components (listeners, target groups, certificates) that are associated with EKS clusters. View load balancer attributes and health status.
events - Create and manage EventBridge rules for monitoring EC2 and AWS Health events related to EKS clusters, enabling automated responses to infrastructure changes and health alerts.
iam - Manage EC2 instance profiles with the "eks" prefix, including creation, deletion, and role association, which is necessary for EKS node management.
pricing & shield - Access AWS pricing information and Shield protection status, enabling cost management and advanced security features for EKS resources.
Resource cleanup - Safely delete EKS-tagged resources including volumes, snapshots, launch templates, and network interfaces during cluster cleanup operations.

To view the latest version of the JSON policy document, see AmazonEKSServiceRolePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSVPCResourceController

You can attach the AmazonEKSVPCResourceController policy to your IAM identities. If you’re using security groups for Pods, you must attach this policy to your Amazon EKS cluster IAM role to perform actions on your behalf.

This policy grants the cluster role permissions to manage Elastic Network Interfaces and IP addresses for nodes.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

ec2 – Manage Elastic Network Interfaces and IP addresses to support Pod security groups and Windows nodes.

To view the latest version of the JSON policy document, see AmazonEKSVPCResourceController in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSWorkerNodePolicy

You can attach the AmazonEKSWorkerNodePolicy to your IAM entities. You must attach this policy to a node IAM role that you specify when you create Amazon EC2 nodes that allow Amazon EKS to perform actions on your behalf. If you create a node group using eksctl, it creates the node IAM role and attaches this policy to the role automatically.

This policy grants Amazon EKS Amazon EC2 nodes permissions to connect to Amazon EKS clusters.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

ec2 – Read instance volume and network information. This is required so that Kubernetes nodes can describe information about Amazon EC2 resources that are required for the node to join the Amazon EKS cluster.
eks – Optionally describe the cluster as part of node bootstrapping.
eks-auth:AssumeRoleForPodIdentity – Allow retrieving credentials for EKS workloads on the node. This is required for EKS Pod Identity to function properly.

To view the latest version of the JSON policy document, see AmazonEKSWorkerNodePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSWorkerNodeMinimalPolicy

You can attach the AmazonEKSWorkerNodeMinimalPolicy to your IAM entities. You may attach this policy to a node IAM role that you specify when you create Amazon EC2 nodes that allow Amazon EKS to perform actions on your behalf.

This policy grants Amazon EKS Amazon EC2 nodes permissions to connect to Amazon EKS clusters. This policy has fewer permissions compared to AmazonEKSWorkerNodePolicy.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

eks-auth:AssumeRoleForPodIdentity - Allow retrieving credentials for EKS workloads on the node. This is required for EKS Pod Identity to function properly.

To view the latest version of the JSON policy document, see AmazonEKSWorkerNodePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AWSServiceRoleForAmazonEKSNodegroup

You can’t attach AWSServiceRoleForAmazonEKSNodegroup to your IAM entities. This policy is attached to a service-linked role that allows Amazon EKS to perform actions on your behalf. For more information, see service-linked-role-permissions-eks-nodegroups.title.

This policy grants the AWSServiceRoleForAmazonEKSNodegroup role permissions that allow it to create and manage Amazon EC2 node groups in your account.

Permissions details

This policy includes the following permissions that allow Amazon EKS to complete the following tasks:

ec2 – Work with security groups, tags, capacity reservations, and launch templates. This is required for Amazon EKS managed node groups to enable remote access configuration and to describe capacity reservations that can be used in managed node groups. Additionally, Amazon EKS managed node groups create a launch template on your behalf. This is to configure the Amazon EC2 Auto Scaling group that backs each managed node group.
iam – Create a service-linked role and pass a role. This is required by Amazon EKS managed node groups to manage instance profiles for the role being passed when creating a managed node group. This instance profile is used by Amazon EC2 instances launched as part of a managed node group. Amazon EKS needs to create service-linked roles for other services such as Amazon EC2 Auto Scaling groups. These permissions are used in the creation of a managed node group.
autoscaling – Work with security Auto Scaling groups. This is required by Amazon EKS managed node groups to manage the Amazon EC2 Auto Scaling group that backs each managed node group. It’s also used to support functionality such as evicting Pods when nodes are terminated or recycled during node group updates.

To view the latest version of the JSON policy document, see AWSServiceRoleForAmazonEKSNodegroup in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEBSCSIDriverPolicy

The AmazonEBSCSIDriverPolicy policy allows the Amazon EBS Container Storage Interface (CSI) driver to create, modify, attach, detach, and delete volumes on your behalf. This includes modifying tags on existing volumes and enabling Fast Snapshot Restore (FSR) on EBS volumes. It also grants the EBS CSI driver permissions to create, restore, and delete snapshots, and to list your instances, volumes, and snapshots.

To view the latest version of the JSON policy document, see AmazonEBSCSIDriverServiceRolePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEFSCSIDriverPolicy

The AmazonEFSCSIDriverPolicy policy allows the Amazon EFS Container Storage Interface (CSI) to create and delete access points on your behalf. It also grants the Amazon EFS CSI driver permissions to list your access points file systems, mount targets, and Amazon EC2 availability zones.

To view the latest version of the JSON policy document, see AmazonEFSCSIDriverServiceRolePolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSLocalOutpostClusterPolicy

You can attach this policy to IAM entities. Before creating a local cluster, you must attach this policy to your cluster role. Kubernetes clusters that are managed by Amazon EKS make calls to other AWS services on your behalf. They do this to manage the resources that you use with the service.

The AmazonEKSLocalOutpostClusterPolicy includes the following permissions:

ec2 read actions – Allows control plane instances to describe Availability Zone, route table, instance, and network interface properties. Required permissions for Amazon EC2 instances to successfully join the cluster as control plane instances.
ssm – Allows Amazon EC2 Systems Manager connection to the control plane instance, which is used by Amazon EKS to communicate and manage the local cluster in your account.
logs – Allows instances to push logs to Amazon CloudWatch.
secretsmanager – Allows instances to get and delete bootstrap data for the control plane instances securely from AWS Secrets Manager.
ecr – Allows Pods and containers that are running on the control plane instances to pull container images that are stored in Amazon Elastic Container Registry.

To view the latest version of the JSON policy document, see AmazonEKSLocalOutpostClusterPolicy in the AWS Managed Policy Reference Guide.

`AWS` managed policy: AmazonEKSLocalOutpostServiceRolePolicy

You can’t attach this policy to your IAM entities. When you create a cluster using an IAM principal that has the iam:CreateServiceLinkedRole permission, Amazon EKS automatically creates the AWSServiceRoleforAmazonEKSLocalOutpost service-linked role for you and attaches this policy to it. This policy allows the service-linked role to call AWS services on your behalf for local clusters.

The AmazonEKSLocalOutpostServiceRolePolicy includes the following permissions:

ec2 – Allows Amazon EKS to work with security, network, and other resources to successfully launch and manage control plane instances in your account.
ssm – Allows Amazon EC2 Systems Manager connection to the control plane instances, which is used by Amazon EKS to communicate and manage the local cluster in your account.
iam – Allows Amazon EKS to manage the instance profile associated with the control plane instances.
secretsmanager - Allows Amazon EKS to put bootstrap data for the control plane instances into AWS Secrets Manager so it can be securely referenced during instance bootstrapping.
outposts – Allows Amazon EKS to get Outpost information from your account to successfully launch a local cluster in an Outpost.

To view the latest version of the JSON policy document, see AmazonEKSLocalOutpostServiceRolePolicy in the AWS Managed Policy Reference Guide.

Amazon EKS updates to `AWS` managed policies

View details about updates to AWS managed policies for Amazon EKS since this service began tracking these changes. For automatic alerts about changes to this page, subscribe to the RSS feed on the Amazon EKS Document history page.

Change Description Date

Added permissions to AmazonEBSCSIDriverPolicy.

Added a new statement authorizing the EBS CSI Driver to restore all snapshots. This was previously allowed by the existing policy but a new explicit statement is required due to a change in the handling of IAM for CreateVolume.

Added the ability for the EBS CSI Driver to modify tags on existing volumes. The EBS CSI Driver can modify tags of existing volumes via a parameters in Kubernetes `VolumeAttributesClass`es.

Added the ability for the EBS CSI Driver to enable Fast Snapshot Restore (FSR) on EBS volumes. The EBS CSI Driver can enable FSR on new volumes via parameters in Kubernetes `StorageClass`es.

January 13, 2025

Added permissions to security-iam-awsmanpol-AmazonEKSLoadBalancingPolicy.title.

Updated AmazonEKSLoadBalancingPolicy to allow listing and describing networking and IP address resources.

December 26, 2024

Added permissions to security-iam-awsmanpol-awsserviceroleforamazoneksnodegroup.title.

Updated AWSServiceRoleForAmazonEKSNodegroup for compatibility with China regions.

November 22, 2024

Added permissions to security-iam-awsmanpol-amazonekslocaloutpostclusterpolicy.title

Added ec2:DescribeAvailabilityZones permission to AmazonEKSLocalOutpostClusterPolicy so the AWS Cloud Controller Manager on the cluster control plane can identify the Availability Zone that each node is in.

November 21, 2024

Added permissions to security-iam-awsmanpol-awsserviceroleforamazoneksnodegroup.title.

Updated AWSServiceRoleForAmazonEKSNodegroup policy to allow ec2:RebootInstances for instances created by Amazon EKS managed node groups. Restricted the ec2:CreateTags permissions for Amazon EC2 resources.

November 20, 2024

Added permissions to security-iam-awsmanpol-amazoneksservicerolepolicy.title.

EKS updated AWS managed policy AmazonEKSServiceRolePolicy. Added permissions for EKS access policies, load balancer management, and automated cluster resource cleanup.

November 16, 2024

Introduced security-iam-awsmanpol-AmazonEKSComputePolicy.title.

EKS updated AWS managed policy AmazonEKSComputePolicy. Updated resource permissions for the iam:AddRoleToInstanceProfile action.

November 7, 2024

Introduced security-iam-awsmanpol-AmazonEKSComputePolicy.title.

AWS introduced the AmazonEKSComputePolicy.

November 1, 2024

Added permissions to AmazonEKSClusterPolicy

Added ec2:DescribeInstanceTopology permission to allow Amazon EKS to attach topology information to the node as labels.

November 1, 2024

Introduced security-iam-awsmanpol-AmazonEKSBlockStoragePolicy.title.

AWS introduced the AmazonEKSBlockStoragePolicy.

October 30, 2024

Introduced security-iam-awsmanpol-AmazonEKSLoadBalancingPolicy.title.

AWS introduced the AmazonEKSLoadBalancingPolicy.

October 30, 2024

Added permissions to AmazonEKSServiceRolePolicy.

Added cloudwatch:PutMetricData permissions to allow Amazon EKS to publish metrics to Amazon CloudWatch.

October 29, 2024

Introduced security-iam-awsmanpol-AmazonEKSNetworkingPolicy.title.

AWS introduced the AmazonEKSNetworkingPolicy.

October 28, 2024

Added permissions to AmazonEKSServicePolicy and AmazonEKSServiceRolePolicy

Added ec2:GetSecurityGroupsForVpc and associated tag permissions to allow EKS to read security group information and update related tags.

October 10, 2024

Introduced AmazonEKSWorkerNodeMinimalPolicy.

AWS introduced the AmazonEKSWorkerNodeMinimalPolicy.

October 3, 2024

Added permissions to AWSServiceRoleForAmazonEKSNodegroup.

Added autoscaling:ResumeProcesses and autoscaling:SuspendProcesses permissions to allow Amazon EKS to suspend and resume AZRebalance in Amazon EKS-managed Auto Scaling groups.

August 21, 2024

Added permissions to AWSServiceRoleForAmazonEKSNodegroup.

Added ec2:DescribeCapacityReservations permission to allow Amazon EKS to describe capacity reservation in user’s account. Added autoscaling:PutScheduledUpdateGroupAction permission to enable setting scheduled scaling on CAPACITY_BLOCK node groups.

June 27, 2024

AmazonEKS_CNI_Policy – Update to an existing policy

Amazon EKS added new ec2:DescribeSubnets permissions to allow the Amazon VPC CNI plugin for Kubernetes to see the amount of free IP addresses in your Amazon VPC subnets. The VPC CNI can use the free IP addresses in each subnet to pick the subnets with the most free IP addresses to use when creating an elastic network interface.

March 4, 2024

AmazonEKSWorkerNodePolicy – Update to an existing policy

Amazon EKS added new permissions to allow EKS Pod Identities. The Amazon EKS Pod Identity Agent uses the node role.

November 26, 2023

Introduced AmazonEFSCSIDriverPolicy.

AWS introduced the AmazonEFSCSIDriverPolicy.

July 26, 2023

Added permissions to AmazonEKSClusterPolicy.

Added ec2:DescribeAvailabilityZones permission to allow Amazon EKS to get the AZ details during subnet auto-discovery while creating load balancers.

February 7, 2023

Updated policy conditions in AmazonEBSCSIDriverPolicy.

Removed invalid policy conditions with wildcard characters in the StringLike key field. Also added a new condition ec2:ResourceTag/kubernetes.io/created-for/pvc/name: "*" to ec2:DeleteVolume, which allows the EBS CSI driver to delete volumes created by the in-tree plugin.

November 17, 2022

Added permissions to AmazonEKSLocalOutpostServiceRolePolicy.

Added ec2:DescribeVPCAttribute, ec2:GetConsoleOutput and ec2:DescribeSecret to allow better prerequisite validation and managed lifecycle control. Also added ec2:DescribePlacementGroups and "region.arnec2:*:*:placement-group/*" to ec2:RunInstances to support placement control of the control plane Amazon EC2 instances on Outposts.

October 24, 2022

Update Amazon Elastic Container Registry permissions in AmazonEKSLocalOutpostClusterPolicy.

Moved action ecr:GetDownloadUrlForLayer from all resource sections to a scoped section. Added resource region.arnecr:*:*:repository/eks/. Removed resource region.arnecr::*:repository/eks/eks-certificates-controller-public. This resource is covered by the added region.arnecr:*:*:repository/eks/* resource.

October 20, 2022

Added permissions to AmazonEKSLocalOutpostClusterPolicy.

Added the region.arnecr:*:*:repository/kubelet-config-updater Amazon Elastic Container Registry repository so the cluster control plane instances can update some kubelet arguments.

August 31, 2022

Introduced AmazonEKSLocalOutpostClusterPolicy.

AWS introduced the AmazonEKSLocalOutpostClusterPolicy.

August 24, 2022

Introduced AmazonEKSLocalOutpostServiceRolePolicy.

AWS introduced the AmazonEKSLocalOutpostServiceRolePolicy.

August 23, 2022

Introduced AmazonEBSCSIDriverPolicy.

AWS introduced the AmazonEBSCSIDriverPolicy.

April 4, 2022

Added permissions to AmazonEKSWorkerNodePolicy.

Added ec2:DescribeInstanceTypes to enable Amazon EKS-optimized AMIs that can auto discover instance level properties.

March 21, 2022

Added permissions to AWSServiceRoleForAmazonEKSNodegroup.

Added autoscaling:EnableMetricsCollection permission to allow Amazon EKS to enable metrics collection.

December 13, 2021

Added permissions to AmazonEKSClusterPolicy.

Added ec2:DescribeAccountAttributes, ec2:DescribeAddresses, and ec2:DescribeInternetGateways permissions to allow Amazon EKS to create a service-linked role for a Network Load Balancer.

June 17, 2021

Amazon EKS started tracking changes.

Amazon EKS started tracking changes for its AWS managed policies.

June 17, 2021

14.7.7. Troubleshooting IAM

This topic covers some common errors that you may see while using Amazon EKS with IAM and how to work around them.

AccessDeniedException

If you receive an AccessDeniedException when calling an AWS API operation, then the IAM principal credentials that you’re using don’t have the required permissions to make that call.

An error occurred (AccessDeniedException) when calling the DescribeCluster operation:
User: region.arniam::111122223333:user/user_name is not authorized to perform:
eks:DescribeCluster on resource: region.arneks:region:111122223333:cluster/my-cluster

In the previous example message, the user does not have permissions to call the Amazon EKS DescribeCluster API operation. To provide Amazon EKS admin permissions to an IAM principal, see security-iam-id-based-policy-examples.title.

For more general information about IAM, see Controlling access using policies in the IAM User Guide.

Can’t see Nodes on the Compute tab or anything on the Resources tab and you receive an error in the `consolelong`

You may see a console error message that says Your current user or role does not have access to Kubernetes objects on this EKS cluster. Make sure that the IAM principal user that you’re using the consolelong with has the necessary permissions. For more information, see view-kubernetes-resources-permissions.title.

aws-auth `ConfigMap` does not grant access to the cluster

The AWS IAM Authenticator doesn’t permit a path in the role ARN used in the ConfigMap. Therefore, before you specify rolearn, remove the path. For example, change region.arniam::111122223333:role/team/developers/eks-admin to region.arniam::111122223333:role/eks-admin.

I am not authorized to perform iam:PassRole

If you receive an error that you’re not authorized to perform the iam:PassRole action, your policies must be updated to allow you to pass a role to Amazon EKS.

Some AWS services allow you to pass an existing role to that service instead of creating a new service role or service-linked role. To do this, you must have permissions to pass the role to the service.

The following example error occurs when an IAM user named marymajor tries to use the console to perform an action in Amazon EKS. However, the action requires the service to have permissions that are granted by a service role. Mary does not have permissions to pass the role to the service.

User: {arn-aws}iam::123456789012:user/marymajor is not authorized to perform: iam:PassRole

In this case, Mary’s policies must be updated to allow her to perform the iam:PassRole action.

If you need help, contact your AWS administrator. Your administrator is the person who provided you with your sign-in credentials.

I want to allow people outside of my `AWS` account to access my Amazon EKS resources

You can create a role that users in other accounts or people outside of your organization can use to access your resources. You can specify who is trusted to assume the role. For services that support resource-based policies or access control lists (ACLs), you can use those policies to grant people access to your resources.

To learn more, consult the following:

To learn whether Amazon EKS supports these features, see security-iam-service-with-iam.title.
To learn how to provide access to your resources across AWS accounts that you own, see Providing access to an IAM user in another AWS account that you own in the IAM User Guide.
To learn how to provide access to your resources to third-party AWS accounts, see Providing access to AWS accounts owned by third parties in the IAM User Guide.
To learn how to provide access through identity federation, see Providing access to externally authenticated users (identity federation) in the IAM User Guide.
To learn the difference between using roles and resource-based policies for cross-account access, see Cross account resource access in IAM in the IAM User Guide.

Pod containers receive the following error: `An error occurred (SignatureDoesNotMatch) when calling the GetCallerIdentity operation: Credential should be scoped to a valid region`

Your containers receive this error if your application is explicitly making requests to the AWS STS global endpoint (https://sts.amazonaws) and your Kubernetes service account is configured to use a regional endpoint. You can resolve the issue with one of the following options:

Update your application code to remove explicit calls to the AWS STS global endpoint.
Update your application code to make explicit calls to regional endpoints such as https://sts.us-west-2.amazonaws.com. Your application should have redundancy built in to pick a different AWS Region in the event of a failure of the service in the AWS Region. For more information, see Managing AWS STS in an AWS Region in the IAM User Guide.
Configure your service accounts to use the global endpoint. All versions earlier than 1.22 used the global endpoint by default, but version 1.22 and later clusters use the regional endpoint by default. For more information, see configure-sts-endpoint.title.

14.7.8. Amazon EKS cluster IAM role

Learn how to create and configure the required AWS Identity and Access Management role for Amazon EKS clusters to manage nodes and load balancers using managed or custom IAM policies.

An Amazon EKS cluster IAM role is required for each cluster. Kubernetes clusters managed by Amazon EKS use this role to manage nodes and the legacy Cloud Provider uses this role to create load balancers with Elastic Load Balancing for services.

Before you can create Amazon EKS clusters, you must create an IAM role with either of the following IAM policies:

AmazonEKSClusterPolicy

A custom IAM policy. The minimal permissions that follow allows the Kubernetes cluster to manage nodes, but doesn’t allow the legacy Cloud Provider to create load balancers with Elastic Load Balancing. Your custom IAM policy must have at least the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags"
      ],
      "Resource": "region.arnec2:*:*:instance/*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:TagKeys": "kubernetes.io/cluster/*"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeVpcs",
        "ec2:DescribeDhcpOptions",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstanceTopology",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}

Prior to October 3, 2023, AmazonEKSClusterPolicy was required on the IAM role for each cluster.

Prior to April 16, 2020, AmazonEKSServicePolicy and AmazonEKSClusterPolicy was required and the suggested name for the role was eksServiceRole. With the AWSServiceRoleForAmazonEKS service-linked role, the AmazonEKSServicePolicy policy is no longer required for clusters created on or after April 16, 2020.

Check for an existing cluster role

You can use the following procedure to check and see if your account already has the Amazon EKS cluster role.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
Search the list of roles for eksClusterRole. If a role that includes eksClusterRole doesn’t exist, then see create-service-role.title to create the role. If a role that includes eksClusterRole does exist, then select the role to view the attached policies.
Choose Permissions.
Ensure that the AmazonEKSClusterPolicy managed policy is attached to the role. If the policy is attached, your Amazon EKS cluster role is properly configured.
Choose Trust relationships, and then choose Edit trust policy.
Verify that the trust relationship contains the following policy. If the trust relationship matches the following policy, choose Cancel. If the trust relationship doesn’t match, copy the policy into the Edit trust policy window and choose Update policy.
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

Creating the Amazon EKS cluster role

You can use the consolelong or the AWS CLI to create the cluster role.

consolelong

Open the IAM console at https://console.aws.amazon.com/iam/.
Choose Roles, then Create role.
Under Trusted entity type, select AWS service.
From the Use cases for other AWS services dropdown list, choose EKS.
Choose EKS - Cluster for your use case, and then choose Next.
On the Add permissions tab, choose Next.
For Role name, enter a unique name for your role, such as eksClusterRole.
For Description, enter descriptive text such as Amazon EKS - Cluster role.
Choose Create role.

AWS CLI

Copy the following contents to a file named cluster-trust-policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create the role. You can replace eksClusterRole with any name that you choose.

aws iam create-role \
  --role-name eksClusterRole \
  --assume-role-policy-document file://"cluster-trust-policy.json"

Attach the required IAM policy to the role.

aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonEKSClusterPolicy \
  --role-name eksClusterRole

14.7.9. Amazon EKS node IAM role

The Amazon EKS node kubelet daemon makes calls to AWS APIs on your behalf. Nodes receive permissions for these API calls through an IAM instance profile and associated policies. Before you can launch nodes and register them into a cluster, you must create an IAM role for those nodes to use when they are launched. This requirement applies to nodes launched with the Amazon EKS optimized AMI provided by Amazon, or with any other node AMIs that you intend to use. Additionally, this requirement applies to both managed node groups and self-managed nodes.

You can’t use the same role that is used to create any clusters.

Before you create nodes, you must create an IAM role with the following permissions:

Permissions for the kubelet to describe Amazon EC2 resources in the VPC, such as provided by the AmazonEKSWorkerNodePolicy policy. This policy also provides the permissions for the Amazon EKS Pod Identity Agent.
Permissions for the kubelet to use container images from Amazon Elastic Container Registry (Amazon ECR), such as provided by the AmazonEC2ContainerRegistryPullOnly policy. The permissions to use container images from Amazon Elastic Container Registry (Amazon ECR) are required because the built-in add-ons for networking run pods that use container images from Amazon ECR.
(Optional) Permissions for the Amazon EKS Pod Identity Agent to use the eks-auth:AssumeRoleForPodIdentity action to retrieve credentials for pods. If you don’t use the AmazonEKSWorkerNodePolicy, then you must provide this permission in addition to the EC2 permissions to use EKS Pod Identity.`
(Optional) If you don’t use IRSA or EKS Pod Identity to give permissions to the VPC CNI pods, then you must provide permissions for the VPC CNI on the instance role. You can use either the ` AmazonEKS_CNI_Policy` managed policy (if you created your cluster with the IPv4` family) or an IPv6 policy that you create (if you created your cluster with the IPv6 family). Rather than attaching the policy to this role however, we recommend that you attach the policy to a separate role used specifically for the Amazon VPC CNI add-on. For more information about creating a separate role for the Amazon VPC CNI add-on, see cni-iam-role.title.

The Amazon EC2 node groups must have a different IAM role than the Fargate profile. For more information, see pod-execution-role.title.

Check for an existing node role

You can use the following procedure to check and see if your account already has the Amazon EKS node role.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
Search the list of roles for eksNodeRole, AmazonEKSNodeRole, or NodeInstanceRole. If a role with one of those names doesn’t exist, then see create-worker-node-role.title to create the role. If a role that contains eksNodeRole, AmazonEKSNodeRole, or NodeInstanceRole does exist, then select the role to view the attached policies.
Choose Permissions.

Ensure that the AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryPullOnly managed policies are attached to the role or a custom policy is attached with the minimal permissions.

If the AmazonEKS_CNI_Policy policy is attached to the role, we recommend removing it and attaching it to an IAM role that is mapped to the aws-node Kubernetes service account instead. For more information, see cni-iam-role.title.

Choose Trust relationships, and then choose Edit trust policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com"
                ]
            }
        }
    ]
}

Creating the Amazon EKS node IAM role

You can create the node IAM role with the consolelong or the AWS CLI.

consolelong

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.
On the Select trusted entity page, do the following:
1. In the Trusted entity type section, choose AWS service.
2. Under Use case, choose EC2.
3. Choose Next.
On the Add permissions page, attach a custom policy or do the following:
1. In the Filter policies box, enter AmazonEKSWorkerNodePolicy.
2. Select the check box to the left of AmazonEKSWorkerNodePolicy in the search results.
3. Choose Clear filters.
4. In the Filter policies box, enter AmazonEC2ContainerRegistryPullOnly.
5. Select the check box to the left of AmazonEC2ContainerRegistryPullOnly in the search results.
  
  Either the AmazonEKS_CNI_Policy managed policy, or an IPv6 policy that you create must also be attached to either this role or to a different role that’s mapped to the aws-node Kubernetes service account. We recommend assigning the policy to the role associated to the Kubernetes service account instead of assigning it to this role. For more information, see cni-iam-role.title.
6. Choose Next.
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKSNodeRole.
2. For Description, replace the current text with descriptive text such as Amazon EKS - Node role.
3. Under Add tags (Optional), add metadata to the role by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM resources in the IAM User Guide.
4. Choose Create role.

AWS CLI

Run the following command to create the node-role-trust-relationship.json file.

cat >node-role-trust-relationship.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com"
                ]
            }
        }
    ]
}
EOF

Create the IAM role.

aws iam create-role \
  --role-name AmazonEKSNodeRole \
  --assume-role-policy-document file://"node-role-trust-relationship.json"

Attach two required IAM managed policies to the IAM role.

aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonEKSWorkerNodePolicy \
  --role-name AmazonEKSNodeRole
aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonEC2ContainerRegistryPullOnly \
  --role-name AmazonEKSNodeRole

Attach one of the following IAM policies to the IAM role depending on which IP family you created your cluster with. The policy must be attached to this role or to a role associated to the Kubernetes aws-node service account that’s used for the Amazon VPC CNI plugin for Kubernetes. We recommend assigning the policy to the role associated to the Kubernetes service account. To assign the policy to the role associated to the Kubernetes service account, see cni-iam-role.title.

IPv4

aws iam attach-role-policy \
  --policy-arn region.arniam::aws:policy/AmazonEKS_CNI_Policy \
  --role-name AmazonEKSNodeRole

IPv6

Copy the following text and save it to a file named vpc-cni-ipv6-policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AssignIpv6Addresses",
                "ec2:DescribeInstances",
                "ec2:DescribeTags",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInstanceTypes"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": [
                "region.arnec2:*:*:network-interface/*"
            ]
        }
    ]
}

Create the IAM policy.

aws iam create-policy --policy-name AmazonEKS_CNI_IPv6_Policy --policy-document file://vpc-cni-ipv6-policy.json

Attach the IAM policy to the IAM role. Replace 111122223333 with your account ID.

aws iam attach-role-policy \
  --policy-arn region.arniam::111122223333:policy/AmazonEKS_CNI_IPv6_Policy \
  --role-name AmazonEKSNodeRole

14.7.10. Amazon EKS Auto Mode cluster IAM role

Learn how to create and configure the required AWS Identity and Access Management role for Amazon EKS Auto Mode clusters to automate routine tasks for storage, networking, and compute autoscaling.

An Amazon EKS cluster IAM role is required for each cluster. Kubernetes clusters managed by Amazon EKS use this role to automate routine tasks for storage, networking, and compute autoscaling.

Before you can create Amazon EKS clusters, you must create an IAM role with the policies required for EKS Auto Mode. You can either attach the suggested AWS IAM managed policies, or create custom polices with equivalent permissions.

AmazonEKSComputePolicy
AmazonEKSBlockStoragePolicy
AmazonEKSLoadBalancingPolicy
AmazonEKSNetworkingPolicy
AmazonEKSClusterPolicy

Check for an existing cluster role

You can use the following procedure to check and see if your account already has the Amazon EKS cluster role.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
Search the list of roles for AmazonEKSAutoClusterRole. If a role that includes AmazonEKSAutoClusterRole doesn’t exist, then see the instructions in the next section to create the role. If a role that includes AmazonEKSAutoClusterRole does exist, then select the role to view the attached policies.
Choose Permissions.
Ensure that the AmazonEKSClusterPolicy managed policy is attached to the role. If the policy is attached, your Amazon EKS cluster role is properly configured.
Choose Trust relationships, and then choose Edit trust policy.
Verify that the trust relationship contains the following policy. If the trust relationship matches the following policy, choose Cancel. If the trust relationship doesn’t match, copy the policy into the Edit trust policy window and choose Update policy.
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}
```

AWS does not require the name AmazonEKSAutoClusterRole for this role.

Creating the Amazon EKS cluster role

You can use the consolelong or the AWS CLI to create the cluster role.

`consolelong`

Open the IAM console at https://console.aws.amazon.com/iam/.
Choose Roles, then Create role.
Under Trusted entity type, select AWS service.
From the Use cases for other AWS services dropdown list, choose EKS.
Choose EKS - Cluster for your use case, and then choose Next.
On the Add permissions tab, select the policies and then choose Next.
For Role name, enter a unique name for your role, such as AmazonEKSAutoClusterRole.
For Description, enter descriptive text such as Amazon EKS - Cluster role.
Choose Create role.

`AWS` CLI

Copy the following contents to a file named cluster-trust-policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

Create the role. You can replace AmazonEKSAutoClusterRole with any name that you choose.

aws iam create-role \
  --role-name AmazonEKSAutoClusterRole \
  --assume-role-policy-document file://"cluster-trust-policy.json"

Attach the required IAM policies to the role:

AmazonEKSClusterPolicy:

aws iam attach-role-policy \
    --role-name AmazonEKSAutoClusterRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy

AmazonEKSComputePolicy:

aws iam attach-role-policy \
    --role-name AmazonEKSAutoClusterRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSComputePolicy

AmazonEKSBlockStoragePolicy:

aws iam attach-role-policy \
    --role-name AmazonEKSAutoClusterRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy

AmazonEKSLoadBalancingPolicy:

aws iam attach-role-policy \
    --role-name AmazonEKSAutoClusterRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy

AmazonEKSNetworkingPolicy:

aws iam attach-role-policy \
    --role-name AmazonEKSAutoClusterRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy

14.7.11. Amazon EKS Auto Mode node IAM role

You can’t use the same role that is used to create any clusters.

Before you create nodes, you must create an IAM role with the following policies, or equivalent permissions:

Check for an existing node role

You can use the following procedure to check and see if your account already has the Amazon EKS node role.

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
Search the list of roles for AmazonEKSAutoNodeRole. If a role with one of those names doesn’t exist, then see instructions in the next section to create the role. If a role that contains AmazonEKSAutoNodeRole does exist, then select the role to view the attached policies.
Choose Permissions.
Ensure that the required policies above are attached, or equivalent custom policies.
Choose Trust relationships, and then choose Edit trust policy.
Verify that the trust relationship contains the following policy. If the trust relationship matches the following policy, choose Cancel. If the trust relationship doesn’t match, copy the policy into the Edit trust policy window and choose Update policy.
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

Creating the Amazon EKS node IAM role

You can create the node IAM role with the consolelong or the AWS CLI.

`consolelong`

Open the IAM console at https://console.aws.amazon.com/iam/.
In the left navigation pane, choose Roles.
On the Roles page, choose Create role.
On the Select trusted entity page, do the following:
1. In the Trusted entity type section, choose AWS service.
2. Under Use case, choose EC2.
3. Choose Next.
On the Add permissions page, attach the following policies:
- AmazonEKSWorkerNodeMinimalPolicy
- AmazonEC2ContainerRegistryPullOnly
On the Name, review, and create page, do the following:
1. For Role name, enter a unique name for your role, such as AmazonEKSAutoNodeRole.
2. For Description, replace the current text with descriptive text such as Amazon EKS - Node role.
3. Under Add tags (Optional), add metadata to the role by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM resources in the IAM User Guide.
4. Choose Create role.

`AWS` CLI

Create the Node IAM Role

Use the node-trust-policy.json file from the previous step to define which entities can assume the role. Run the following command to create the Node IAM Role:

aws iam create-role \
    --role-name AmazonEKSAutoNodeRole \
    --assume-role-policy-document file://node-trust-policy.json

Note the Role ARN

After creating the role, retrieve and save the ARN of the Node IAM Role. You will need this ARN in subsequent steps. Use the following command to get the ARN:

aws iam get-role --role-name AmazonEKSAutoNodeRole --query "Role.Arn" --output text

Attach Required Policies

Attach the following AWS managed policies to the Node IAM Role to provide the necessary permissions:

To attach AmazonEKSWorkerNodeMinimalPolicy:

aws iam attach-role-policy \
    --role-name AmazonEKSAutoNodeRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy

To attach AmazonEC2ContainerRegistryPullOnly:

aws iam attach-role-policy \
    --role-name AmazonEKSAutoNodeRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly

How to authenticate requests and manage access your Amazon EKS resources.

AWS Identity and Access Management (IAM) is an AWS service that helps an administrator securely control access to AWS resources. IAM administrators control who can be authenticated (signed in) and authorized (have permissions) to use Amazon EKS resources. IAM is an AWS service that you can use with no additional charge.

14.7.12. Audience

How you use AWS Identity and Access Management (IAM) differs, depending on the work that you do in Amazon EKS.

Service user – If you use the Amazon EKS service to do your job, then your administrator provides you with the credentials and permissions that you need. As you use more Amazon EKS features to do your work, you might need additional permissions. Understanding how access is managed can help you request the right permissions from your administrator. If you cannot access a feature in Amazon EKS, see security-iam-troubleshoot.title.

Service administrator – If you’re in charge of Amazon EKS resources at your company, you probably have full access to Amazon EKS. It’s your job to determine which Amazon EKS features and resources your service users should access. You must then submit requests to your IAM administrator to change the permissions of your service users. Review the information on this page to understand the basic concepts of IAM. To learn more about how your company can use IAM with Amazon EKS, see security-iam-service-with-iam.title.

IAM administrator – If you’re an IAM administrator, you might want to learn details about how you can write policies to manage access to Amazon EKS. To view example Amazon EKS identity-based policies that you can use in IAM, see security-iam-id-based-policy-examples.title.

14.7.13. Authenticating with identities

Authentication is how you sign in to AWS using your identity credentials. You must be authenticated (signed in to AWS) as the AWS account root user, as an IAM user, or by assuming an IAM role.

You can sign in to AWS as a federated identity by using credentials provided through an identity source. AWS IAM Identity Center (IAM Identity Center) users, your company’s single sign-on authentication, and your Google or Facebook credentials are examples of federated identities. When you sign in as a federated identity, your administrator previously set up identity federation using IAM roles. When you access AWS by using federation, you are indirectly assuming a role.

Depending on the type of user you are, you can sign in to the consolelong or the AWS access portal. For more information about signing in to AWS, see How to sign in to your AWS account in the AWS Sign-In User Guide.

If you access AWS programmatically, AWS provides a software development kit (SDK) and a command line interface (CLI) to cryptographically sign your requests by using your credentials. If you don’t use AWS tools, you must sign requests yourself. For more information about using the recommended method to sign requests yourself, see Signing AWS API requests in the IAM User Guide.

Regardless of the authentication method that you use, you might be required to provide additional security information. For example, AWS recommends that you use multi-factor authentication (MFA) to increase the security of your account. To learn more, see Multi-factor authentication in the AWS IAM Identity Center User Guide and Using multi-factor authentication (MFA) in AWS in the IAM User Guide.

`AWS` account root user

When you create an AWS account, you begin with one sign-in identity that has complete access to all AWS services and resources in the account. This identity is called the AWS account root user and is accessed by signing in with the email address and password that you used to create the account. We strongly recommend that you don’t use the root user for your everyday tasks. Safeguard your root user credentials and use them to perform the tasks that only the root user can perform. For the complete list of tasks that require you to sign in as the root user, see Tasks that require root user credentials in the IAM User Guide.

IAM users and groups

An _ IAM user_ is an identity within your AWS account that has specific permissions for a single person or application. Where possible, we recommend relying on temporary credentials instead of creating IAM users who have long-term credentials such as passwords and access keys. However, if you have specific use cases that require long-term credentials with IAM users, we recommend that you rotate access keys. For more information, see Rotate access keys regularly for use cases that require long-term credentials in the IAM User Guide.

An IAM group is an identity that specifies a collection of IAM users. You can’t sign in as a group. You can use groups to specify permissions for multiple users at a time. Groups make permissions easier to manage for large sets of users. For example, you could have a group named IAMAdmins and give that group permissions to administer IAM resources.

Users are different from roles. A user is uniquely associated with one person or application, but a role is intended to be assumable by anyone who needs it. Users have permanent long-term credentials, but roles provide temporary credentials. To learn more, see When to create an IAM user (instead of a role) in the IAM User Guide.

IAM roles

An _ IAM role_ is an identity within your AWS account that has specific permissions. It is similar to an IAM user, but is not associated with a specific person. You can temporarily assume an IAM role in the consolelong by switching roles. You can assume a role by calling an AWS CLI or AWS API operation or by using a custom URL. For more information about methods for using roles, see Using IAM roles in the IAM User Guide.

IAM roles with temporary credentials are useful in the following situations:

Federated user access – To assign permissions to a federated identity, you create a role and define permissions for the role. When a federated identity authenticates, the identity is associated with the role and is granted the permissions that are defined by the role. For information about roles for federation, see Creating a role for a third-party Identity Provider in the IAM User Guide. If you use IAM Identity Center, you configure a permission set. To control what your identities can access after they authenticate, IAM Identity Center correlates the permission set to a role in IAM. For information about permissions sets, see Permission sets in the AWS IAM Identity Center User Guide.
Temporary IAM user permissions – An IAM user or role can assume an IAM role to temporarily take on different permissions for a specific task.
Cross-account access – You can use an IAM role to allow someone (a trusted principal) in a different account to access resources in your account. Roles are the primary way to grant cross-account access. However, with some AWS services, you can attach a policy directly to a resource (instead of using a role as a proxy). To learn the difference between roles and resource-based policies for cross-account access, see Cross account resource access in IAM in the IAM User Guide.
Cross-service access – Some AWS services use features in other AWS services. For example, when you make a call in a service, it’s common for that service to run applications in Amazon EC2 or store objects in Amazon S3. A service might do this using the calling principal’s permissions, using a service role, or using a service-linked role.
- Forward access sessions (FAS) – When you use an IAM user or role to perform actions in AWS, you are considered a principal. When you use some services, you might perform an action that then initiates another action in a different service. FAS uses the permissions of the principal calling an AWS service, combined with the requesting AWS service to make requests to downstream services. FAS requests are only made when a service receives a request that requires interactions with other AWS services or resources to complete. In this case, you must have permissions to perform both actions. For policy details when making FAS requests, see Forward access sessions.
- Service role – A service role is an IAM role that a service assumes to perform actions on your behalf. An IAM administrator can create, modify, and delete a service role from within IAM. For more information, see Creating a role to delegate permissions to an AWS service in the IAM User Guide.
- Service-linked role – A service-linked role is a type of service role that is linked to an AWS service. The service can assume the role to perform an action on your behalf. Service-linked roles appear in your AWS account and are owned by the service. An IAM administrator can view, but not edit the permissions for service-linked roles.
Applications running on Amazon EC2 – You can use an IAM role to manage temporary credentials for applications that are running on an EC2 instance and making AWS CLI or AWS API requests. This is preferable to storing access keys within the EC2 instance. To assign an AWS role to an EC2 instance and make it available to all of its applications, you create an instance profile that is attached to the instance. An instance profile contains the role and enables programs that are running on the EC2 instance to get temporary credentials. For more information, see Using an IAM role to grant permissions to applications running on Amazon EC2 instances in the IAM User Guide.

To learn whether to use IAM roles or IAM users, see When to create an IAM role (instead of a user) in the IAM User Guide.

14.7.14. Managing access using policies

You control access in AWS by creating policies and attaching them to AWS identities or resources. A policy is an object in AWS that, when associated with an identity or resource, defines their permissions. AWS evaluates these policies when a principal (user, root user, or role session) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents. For more information about the structure and contents of JSON policy documents, see Overview of JSON policies in the IAM User Guide.

Administrators can use AWS JSON policies to specify who has access to what. That is, which principal can perform actions on what resources, and under what conditions.

By default, users and roles have no permissions. To grant users permission to perform actions on the resources that they need, an IAM administrator can create IAM policies. The administrator can then add the IAM policies to roles, and users can assume the roles.

IAM policies define permissions for an action regardless of the method that you use to perform the operation. For example, suppose that you have a policy that allows the iam:GetRole action. A user with that policy can get role information from the consolelong, the AWS CLI, or the AWS API.

Identity-based policies

Identity-based policies are JSON permissions policy documents that you can attach to an identity, such as an IAM user, group of users, or role. These policies control what actions users and roles can perform, on which resources, and under what conditions. To learn how to create an identity-based policy, see Creating IAM policies in the IAM User Guide.

Identity-based policies can be further categorized as inline policies or managed policies. Inline policies are embedded directly into a single user, group, or role. Managed policies are standalone policies that you can attach to multiple users, groups, and roles in your AWS account. Managed policies include AWS managed policies and customer managed policies. To learn how to choose between a managed policy or an inline policy, see Choosing between managed policies and inline policies in the IAM User Guide.

Resource-based policies

Resource-based policies are JSON policy documents that you attach to a resource. Examples of resource-based policies are IAM role trust policies and Amazon S3 bucket policies. In services that support resource-based policies, service administrators can use them to control access to a specific resource. For the resource where the policy is attached, the policy defines what actions a specified principal can perform on that resource and under what conditions. You must specify a principal in a resource-based policy. Principals can include accounts, users, roles, federated users, or AWS services.

Resource-based policies are inline policies that are located in that service. You can’t use AWS managed policies from IAM in a resource-based policy.

Access control lists (ACLs)

Access control lists (ACLs) control which principals (account members, users, or roles) have permissions to access a resource. ACLs are similar to resource-based policies, although they do not use the JSON policy document format.

Amazon S3, AWS WAF, and Amazon VPC are examples of services that support ACLs. To learn more about ACLs, see Access control list (ACL) overview in the Amazon Simple Storage Service Developer Guide.

Other policy types

AWS supports additional, less-common policy types. These policy types can set the maximum permissions granted to you by the more common policy types.

Permissions boundaries – A permissions boundary is an advanced feature in which you set the maximum permissions that an identity-based policy can grant to an IAM entity (IAM user or role). You can set a permissions boundary for an entity. The resulting permissions are the intersection of an entity’s identity-based policies and its permissions boundaries. Resource-based policies that specify the user or role in the Principal field are not limited by the permissions boundary. An explicit deny in any of these policies overrides the allow. For more information about permissions boundaries, see Permissions boundaries for IAM entities in the IAM User Guide.
Service control policies (SCPs) – SCPs are JSON policies that specify the maximum permissions for an organization or organizational unit (OU) in AWS Organizations. AWS Organizations is a service for grouping and centrally managing multiple AWS accounts that your business owns. If you enable all features in an organization, then you can apply service control policies (SCPs) to any or all of your accounts. The SCP limits permissions for entities in member accounts, including each AWS account root user. For more information about Organizations and SCPs, see Service control policies in the AWS Organizations User Guide.
Session policies – Session policies are advanced policies that you pass as a parameter when you programmatically create a temporary session for a role or federated user. The resulting session’s permissions are the intersection of the user or role’s identity-based policies and the session policies. Permissions can also come from a resource-based policy. An explicit deny in any of these policies overrides the allow. For more information, see Session policies in the IAM User Guide.

Multiple policy types

When multiple types of policies apply to a request, the resulting permissions are more complicated to understand. To learn how AWS determines whether to allow a request when multiple policy types are involved, see Policy evaluation logic in the IAM User Guide.

Configure Amazon EKS to meet your security and compliance objectives, and learn how to use other AWS services that help you to secure your Amazon EKS resources.

Cloud security at AWS is the highest priority. As an AWS customer, you benefit from a data center and network architecture that is built to meet the requirements of the most security-sensitive organizations.

Security is a shared responsibility between AWS and you. The shared responsibility model describes this as security of the cloud and security in the cloud:

Security of the cloud – AWS is responsible for protecting the infrastructure that runs AWS services in the AWS Cloud. For Amazon EKS, AWS is responsible for the Kubernetes control plane, which includes the control plane nodes and etcd database. Third-party auditors regularly test and verify the effectiveness of our security as part of the AWS compliance programs. To learn about the compliance programs that apply to Amazon EKS, see AWS Services in Scope by Compliance Program.
Security in the cloud – Your responsibility includes the following areas.
- The security configuration of the data plane, including the configuration of the security groups that allow traffic to pass from the Amazon EKS control plane into the customer VPC
- The configuration of the nodes and the containers themselves
- The node’s operating system (including updates and security patches)
- Other associated application software:
  - Setting up and managing network controls, such as firewall rules
  - Managing platform-level identity and access management, either with or in addition to IAM
- The sensitivity of your data, your company’s requirements, and applicable laws and regulations

This documentation helps you understand how to apply the shared responsibility model when using Amazon EKS. The following topics show you how to configure Amazon EKS to meet your security and compliance objectives. You also learn how to use other AWS services that help you to monitor and secure your Amazon EKS resources.

Linux containers are made up of control groups (cgroups) and namespaces that help limit what a container can access, but all containers share the same Linux kernel as the host Amazon EC2 instance. Running a container as the root user (UID 0) or granting a container access to host resources or namespaces such as the host network or host PID namespace are strongly discouraged, because doing so reduces the effectiveness of the isolation that containers provide.

[[Topic List]]

15. Monitor your cluster performance and view logs

You can observe your data in Amazon EKS using many available monitoring or logging tools.

You can observe your data in Amazon EKS using many available monitoring or logging tools. Your Amazon EKS log data can be streamed to AWS services or to partner tools for data analysis. There are many services available in the consolelong that provide data for troubleshooting your Amazon EKS issues. You can also use an AWS-supported open-source solution for monitoring Amazon EKS infrastructure.

After selecting Clusters in the left navigation pane of the Amazon EKS console, you can view cluster health and details by choosing your cluster’s name and choosing the Observability tab. To view details about any existing Kubernetes resources that are deployed to your cluster, see view-kubernetes-resources.title.

Monitoring is an important part of maintaining the reliability, availability, and performance of Amazon EKS and your AWS solutions. We recommend that you collect monitoring data from all of the parts of your AWS solution. That way, you can more easily debug a multi-point failure if one occurs. Before you start monitoring Amazon EKS, make sure that your monitoring plan addresses the following questions.

What are your goals? Do you need real-time notifications if your clusters scale dramatically?
What resources need to be observed?
How frequently do you need to observe these resources? Does your company want to respond quickly to risks?
What tools do you intend to use? If you already run AWS Fargate as part of your launch, then you can use the built-in log router.
Who do you intend to perform the monitoring tasks?
Whom do you want notifications to be sent to when something goes wrong?

15.1. Monitoring and logging on Amazon EKS

Amazon EKS provides built-in tools for monitoring and logging. For supported versions, the observability dashboard gives visibility into the performance of your cluster. It helps you to quickly detect, troubleshoot, and remediate issues. In addition to monitoring features, it includes lists based on the control plane audit logs. The Kubernetes control plane exposes a number of metrics that that can also be scraped outside of the console.

Control plane logging records all API calls to your clusters, audit information capturing what users performed what actions to your clusters, and role-based information. For more information, see Logging and monitoring on Amazon EKS in the AWS Prescriptive Guidance.

When you check the Amazon EKS authenticator logs in Amazon CloudWatch, the entries are displayed that contain text similar to the following example text.

level=info msg="mapping IAM role" groups="[]" role="region.arniam::111122223333:role/XXXXXXXXXXXXXXXXXX-NodeManagerRole-XXXXXXXX" username="eks:node-manager"

Entries that contain this text are expected. The username is an Amazon EKS internal service role that performs specific operations for managed node groups and Fargate.

For low-level, customizable logging, then Kubernetes logging is available.

Amazon EKS is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Amazon EKS. CloudTrail captures all API calls for Amazon EKS as events. The calls captured include calls from the Amazon EKS console and code calls to the Amazon EKS API operations. For more information, see logging-using-cloudtrail.title.

The Kubernetes API server exposes a number of metrics that are useful for monitoring and analysis. For more information, see prometheus.title.

To configure Fluent Bit for custom Amazon CloudWatch logs, see Setting up Fluent Bit in the Amazon CloudWatch User Guide.

15.2. Amazon EKS monitoring and logging tools

Amazon Web Services provides various tools that you can use to monitor Amazon EKS. You can configure some tools to set up automatic monitoring, but some require manual calls. We recommend that you automate monitoring tasks as much as your environment and existing toolset allows.

The following table describes various monitoring tool options.

Areas Tool Description Setup

Control plane

Observability dashboard

For supported versions, the observability dashboard gives visibility into the performance of your cluster. It helps you to quickly detect, troubleshoot, and remediate issues.

Applications / control plane

Prometheus

Prometheus can be used to monitor metrics and alerts for applications and the control plane.

CloudWatch Container Insights

Applications

CloudWatch Container Insights collects, aggregates, and summarizes metrics and logs from your containerized applications and microservices.

AWS Distro for OpenTelemetry (ADOT)

Applications

ADOT can collect and sends correlated metrics, trace data, and metadata to AWS monitoring services or partners. It can be set up through CloudWatch Container Insights.

Applications

Amazon DevOps Guru

Amazon DevOps Guru detects node-level operational performance and availability.

Applications

AWS X-Ray

AWS X-Ray receives trace data about your application. This trace data includes ingoing and outgoing requests and metadata about the requests. For Amazon EKS, the implementation requires the OpenTelemetry add-on.

Applications

Amazon CloudWatch

CloudWatch provides some basic Amazon EKS metrics for free on supported versions. You can expand this functionality with the CloudWatch Observability Operator to handle collecting metrics, logs, and trace data.

The following table describes various logging tool options.

Areas Tool Description Setup

Control plane

Observability dashboard

For supported versions, the observability dashboard shows lists based on the control plane audit logs. It also includes links to control plane logs in Amazon CloudWatch.

Amazon CloudWatch Container Insights

Applications

Amazon CloudWatch Container Insights collects, aggregates, and summarizes metrics and logs from your containerized applications and microservices.

Control plane

Amazon CloudWatch Logs

You can send audit and diagnostic logs directly from the Amazon EKS control plane to CloudWatch Logs in your account.

Control plane

AWS CloudTrail

It logs API calls by a user, role, or service.

Multiple areas for AWS Fargate instances

AWS Fargate log router

For AWS Fargate instances, the log router streams logs to AWS services or partner tools. It uses AWS for Fluent Bit. Logs can be streamed to other AWS services or partner tools.