Troubleshoot EC2 Configuration in a Private Subnet with AWS

If your organization has Amazon EC2 (elastic compute cloud) instances that are not internet-facing, it’s a best practice to run them within a private subnet behind a NAT gateway. This way, they can still download software updates, access other Amazon resources, and implement security patches.

When setting up this configuration, you might run into issues that keep your private subnet from connecting to the internet.

If that happens, here are the manual steps to troubleshooting your EC2 configuration.

Blink Automation: Troubleshoot Your EC2 Configuration in a Private Subnet

Blink + AWS

Try This Automation

Troubleshooting EC2 Configuration in a Private Subnet

1. Confirm That Your Destination Is Available

The issue may not be on your end. If you have access to a computer on a different network, open up the command line and use ping followed by the web address you're trying to reach to confirm that your destination is available.

2. Confirm That the NAT Gateway Is Available

The next potential point of failure is the NAT gateway. From the AWS console, type VPC in the search bar. This will take you to the VPC Service. Click on NAT gateways and look under "status". If your NAT gateway is listed as “available”, skip step 3 and move to step 4. Otherwise, check out step 3.

3. Resolve NAT Gateway Failed State

If your NAT gateway is in a failed state, click the "Details" tab associated with your NAT gateway and find the "State" message. This should display one of the following messages:

3a. "Subnet has insufficient free addresses to create this NAT gateway"

To view the available IPs, go to the subnets page in the VPC console. Click on the details pain for your subnet and look at the available IPs. If there are no available IPs, try terminating instances or deleting network interfaces.

3b. "Network has no Internet gateway attached"

Create and attach an Internet gateway to your VPC. In the navigation page of the VPC console, click Internet gateways, then click create Internet gateway.

Once your gateway is created, you must attach it to your VPC. Select the gateway, select actions, and then select attach to VPC. The menu will list all available VPCs. Select yours and click Attach Internet gateway.

3c. "Elastic IP address could not be associated with this NAT gateway"

Check the Elastic IP address for typos and confirm that it is in the same AWS region as the NAT gateway.

3d. "Elastic IP address is already associated"

Navigate to the Elastic IPs page in the Amazon VPC console. Check if another EC2 instance or network interface is associated with the Elastic IP you tried using for the NAT gateway. If so, reassign or deactivate that instance or interface.

3e. "Network interface created and used internally by this NAT gateway is in an invalid state. Please try again."

This error can't be addressed. Create a new NAT gateway.

4. Confirm Public Subnet Settings

If your NAT gateway is available, but you still can’t connect, it may not be set up on a public subnet. In other words, the route table of that subnet may not include a default route targeting an Internet gateway (IGW) or virtual private gateway.

The NAT gateway display page includes a link labeled subnet. Click that link to check your subnet settings.

On the subnet display page, click on the public subnet hosting the NAT gateway and navigate to the route table tab. Confirm that the destination for the subnet is listed as 0.0.0.0/0 and that you've connected the route table to IGW (displayed as 'igw-' followed by a string of characters).

5. Confirm Private Subnet Settings

You can also confirm your private subnet's settings from the same subnets page listed above. Select your private subnet from the list of subnets, and navigate to the route table tab. The destination should again read 0.0.0.0/0. Your private subnet should be targeting your NAT gateway by name rather than an Internet gateway.

6. Check VPC Security Settings

Make sure that you've allowed access to the Internet on the security group and network access control lists (ACL) associated with your VPC and that you don't have any outbound or inbound rules that would prohibit access to the Internet.

To do this, move from the VPC dashboard to the EC2 dashboard via the 'services' dropdown menu. Click security groups on the left-hand side. Locate the security groups relevant to your instance. Click on the outbound rules tab and make sure that there are open ports through which you can access the Internet. The destination of any ports you would use to access the Internet should be listed as 0.0.0.0/0. If your instance is the target of a load balancer located in your public subnet, then needed ports should be allowed for sg-xxxx where sg-xxxx is the security group attached to the load balancer.

7. Ping from Your Private Subnet

Open the command prompt on a computer connected to your private subnet and use ping, followed by your desired destination to see if you're connected. This test can also be performed using the web browser.

Troubleshooting with Blink:

You might have solved your problem quickly or ended up down a research rabbit hole.

With Blink, you can manage your troubleshooting in one place with automations like this one:

This automation is available in the Blink library. When it runs, it does the following steps:

1. Detects if it is a public instance from a public VPC.
2. Checks security groups and access control list rules.
3. For private instances, gets the route tables.

This simple automation is easy to customize. Run it on a schedule, trigger it by an event, or send the final report via email, Slack, or Teams.

There are over 5K automations in the Blink library to choose from, or you can build your own to match your unique needs.

Get started with Blink today and see how easy automation can be.