How to Troubleshoot Kubernetes Service Deployment

When a Kubernetes deployment malfunctions, you need to identify the scope of the problem before you can begin fixing it. As with any troubleshooting method, you can find the root cause of the problem by starting small and working your way up. For Kubernetes errors, the issue can be either at the Pod, Service, or Ingress level. 

First, start small and confirm that your Pods are running and ready. Once you’ve done this, you know to next look at your Service configuration.

In this guide, we’ll assume that you’ve already confirmed that the issue is not at the Pod level and we’ll walk you through how to troubleshoot your Kubernetes Service.

Blink Automation: Troubleshoot Your Kubernetes Service

Blink + Kubernetes

Try This Automation

Service Troubleshooting Steps

Services work to route traffic to designated Pods according to their labels. There are several ways that a Service might fail to link to the proper Pod. Although some fixes for these issues are as simple as ensuring that the Service in question exists, others are a bit more involved. 

Step 1: Checking that the Service Exists

The first step in troubleshooting a Service misconfiguration is to make sure the one you're calling for is present. As fundamental as it sounds, it’s an often-overlooked error that the Service attempting to be accessed doesn't actually exist.

You can check this with this command:

kubectl get svc hostnames

If the Service doesn't exist, you'll receive this message:

No resources found.
Error from server (NotFound): services "hostnames" not found

To resolve this problem, create a new Service by using something similar to this sample command (note that the specifics of your code — like the port and target port labels — are likely to vary):

kubectl expose deployment hostnames --port=80 --target-port=9376
service/hostnames exposed

To confirm that the Service now exists, you can read it back with this command:

kubectl get svc hostnames

Then, a display similar to this one should appear:

NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
hostnames   ClusterIP   10.0.1.175   <none>        80/TCP    5s

Once you've checked to make sure that all Services exist, take the time to make sure that each one is correctly defined and matches the port it's assigned to. This is a basic step, but it could prevent you from having to troubleshoot any further.

Step 2: Checking the Service Endpoint

Once a Service is shown to exist and match its ports, you should make sure that the Service is selecting the Pods attached to it. Start by reviewing the activity level of the Pods with this command:

kubectl get pods -l app=hostnames

The argument "-l app=hostnames" is called the selector label, and the Kubernetes system contains a control loop that keeps track of each one for every service in play. It saves the results into an "endpoints" object, which can be displayed with this command:

kubectl get endpoints hostnames

The output might look something like this:

NAME        ENDPOINTS
hostnames   10.244.0.5:9376,10.244.0.6:9376,10.244.0.7:9376

Of course, specific data will vary, but some information on the Service endpoints should appear. If the "ENDPOINTS" column simply reads "None," then the "spec.selector" service field may not be selecting the Pods according to the "metadata.labels" that were specified in the YAML. This is likely due to a typo or similar error.

Step 3: Testing the Service

As you troubleshoot your Service, it will be necessary to test it along the way. This is useful when endpoints and running Pods can be found in the queries above. The command used to test the Service is:

kubectl port-forward service/<service-name> 3000:80

In this instance, "<service-name>" represents the name of the service, "3000" represents the port you're looking to open, and "80" corresponds to the port that the Service exposes.

If you are still running into issues after troubleshooting your Service, the issue might be at the Ingress level. You can read this guide on Ingress Troubleshooting to learn more.

Troubleshooting Kubernetes Services with Blink

Every time you run into an error, you can work through these manual checks but it might be time-consuming to remember the specifics of each one. There’s an easier way to manage your Kubernetes troubleshooting.

With Blink, you can use automations like this one to get the information you need to resolve errors faster:

This automation is available in the Blink library. When it runs, it does the following steps:

1. Gets Service details
2. Gets pods matching Service selector.
3. Gets Service endpoint IP.
4. Gets Ingress Service details.

This simple automation is easy to customize. Run it on a schedule or send the report via email, Slack, or Teams.

There are over 5K automations in the Blink library to choose from, or you can build your own to match your unique needs.

Get started with Blink today and see how easy automation can be.