Save EKS IP Resources with Pod SNAT

Background

Since the company-assigned VPC is an Intranet segment, used for interfacing with the local end, from the organization’s perspective, the VPC is not truly ‘Private’, therefore the IP quantity is still limited. Following the discovery that EKS Pods occupy ENIs, in order to solve this issue, I added another set of Private IP associated with the VPC, then performed a layer of translation via a Private NAT Gateway. This enables all EKS Pods to communicate with the local network segment through the same Intranet IP, achieving the goal of saving IP resources. This article will document the implementation details and related explanations.

1. Add another Private CIDR, associate with VPC

Directly assign a Private IP segment, here I added a 100.64.0.0/16 segment.
The setting method is very simple, it can be added directly from the AWS Console, but since the team has introduced IaC, it was created with Terraform:

resource "aws_vpc_ipv4_cidr_block_association" "secondary_cidr" {
  vpc_id     = var.vpc_id
  cidr_block = "100.64.0.0/16"
}

2. Add Subnet and assign Private IP segment

Since my VPC is actually an Intranet IP segment, the quantity of IPs in this segment is limited, so a true Private Subnet must be created.

The creation method will not be detailed here, but if it is a Multi-AZ architecture, remember to create a Subnet in each AZ.

2. Add Private NAT Gateway, update Route Table

Create a Private NAT Gateway in the Intranet segment, and configure the Route Table to translate the IP coming from the Private Subnet into an Intranet IP

resource "aws_nat_gateway" "example" {
  connectivity_type = "private"
  subnet_id         = aws_subnet.example.id
}

Special attention is needed here, the default route 0.0.0.0/0 of the Private Subnet must go towards the Private NAT GW

3. Customize EKS Network Settings

According to the official documentation[1], firstly in the aws-node DaemonSet, set the AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG environment variable to true.

1

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

Then, for the Private Subnet added in step 2, add ENIConfig:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

cat >$az_1.yaml <<EOF
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata: 
  name: $az_1
spec: 
  securityGroups: 
    - $cluster_security_group_id
  subnet: $new_subnet_id_1
EOF

Next is the most crucial step, setting SNAT connections for EKS Pod[2].
This way all Pods can connect out via the Private NAT Gateway IP.

1

kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true

Conclusion

This article discussed how to effectively manage limited private IP resources by adding a set of private IP segments, associating them with the VPC, and creating a true Private Subnet to ensure optimal utilization of IP resources. Then, by translating via the Private NAT Gateway, all EKS Pods can communicate with the local network segment through the same internal IP address, thereby saving IP resources. Through these measures, we achieved the goal of optimizing IP resource usage while ensuring smooth internal network connections.

References

[1] https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html

[2] https://docs.aws.amazon.com/eks/latest/userguide/external-snat.html