How to Delete Secrets in Kubernetes Using kubectl

What Are Kubernetes Secrets and Why They Matter

Kubernetes secrets are special objects for storing sensitive data such as API keys, passwords, or tokens in your cluster. Secrets provide a flexible way of separating this information from Pod specifications or Workload configurations, keeping critical data accessible only to the applications that need it.

This separation of concern is crucial for reducing the risk of accidental exposure of sensitive data and unauthorized access within the cluster. Some common uses of Kubernetes secrets include setting up environment variables, providing authentication credentials to pods, and pulling container images from private registries. 

Types of Kubernetes Secrets

Kubernetes offers various secrets for handling different types of sensitive data. These are the most essential types:

‍

Check out the Kubernetes documentation on secret management to learn more about the different types of Secrets available and how to use them correctly in your applications.

Blink Automation: Find and Delete Orphaned Secrets with Slack Approval

Kubernetes + Slack

Try This Automation

Deleting Secrets with kubectl: Step by Step

Since Kubernetes secrets usually contain sensitive data, you should not let unused secrets lie around your cluster for no reason. Thankfully, the kubectl command-line tool makes it very easy to delete Kubernetes secrets.

You should consider a few things before deleting a secret:

• Verify the Pods, Deployments, or Services using the secret. Only delete a secret if it's no longer required.
• Double-check the namespace to avoid deleting the wrong secret, especially when using multiple namespaces for deployments.
• You can backup a secret before deleting it:

kubectl get secret [secret-name] -o yaml > backup-secret.yaml

Delete a Single Secret

Once you've identified the secret you want to delete, you can delete it using the following kubectl command.

kubectl delete secret [secret-name]

Replace [secret-name] with the name of your actual secret. This command will delete the specified secret from your Kubernetes cluster's default namespace.

Delete Secrets by Namespace

You can also use kubectl to delete Kubernetes secrets from a namespace. Use the command below to delete a secret from a specific namespace.

kubectl delete secret --namespace [namespace-name] [secret-name]

Replace the placeholders with the actual names of your secret and its namespace.

Finding and Removing Orphaned Secrets in Kubernetes

Your Kubernetes secrets become orphaned when any Pod or Workload no longer uses them. Since Kubernetes Pods are frequently created and destroyed, orphaned secrets can accumulate over time.

Secrets left out this way cause security risks and increase the use of unutilized resources. That's why you should regularly audit and remove orphaned secrets in Kubernetes.

Identify Orphaned Secrets

The first step in deleting orphaned secrets is identifying them. This phase involves several steps:

Step 1: Find All Secrets

First, you need to retrieve a list of all Kubernetes cluster secrets. Use the below kubectl command to do this:

kubectl get secrets -all-namespaces -o json

This command will return the list of secrets across all cluster namespaces and store them as a JSON object. However, this output will not display the resources that use these secrets. So, you can't differentiate between active secrets and orphaned secrets at this point.

Step 2: Compare with a List of Actively Referenced Secrets

To separate the active secrets from the orphaned ones, you have to reference all secrets and see which are referenced by any Kubernetes resources. You need to check for secret references in several places.

• Secrets from Environment Variables: Use the following commands to fetch all secrets referenced in environment variables across all Pods and store them in envSecrets and envSecrets2.

envSecrets=$(kubectl get pods -o jsonpath='{.items[*].spec.containers[*].env[*].valueFrom.secretKeyRef.name}' | xargs -n1)

envSecrets2=$(kubectl get pods -o jsonpath='{.items[*].spec.containers[*].envFrom[*].secretRef.name}' | xargs -n1)

• Secrets from Volume Mounts: Use the following command to gather all Kubernetes secrets mounted as volume mounts within some pods.

volumeSecrets=$(kubectl get pods -o jsonpath='{.items[*].spec.volumes[*].secret.secretName}' | xargs -n1)

• Secrets for Private Registries: The following kubectl command retrieves all the secrets needed to pull images from private registries.

pullSecrets=$(kubectl get pods -o jsonpath='{.items[*].spec.imagePullSecrets[*].name}' | xargs -n1)

• TLS Secrets: Use the below command to get all the TLS secrets for your clusters.

tlsSecrets=$(kubectl get ingress -o jsonpath='{.items[*].spec.tls[*].secretName}' | xargs -n1)

Once you have all the references for the active secrets, you can compare them against the list of all secrets and identify which ones are currently orphaned. You can do this using the following command.

diff \
<(echo "$envSecrets\n$envSecrets2\n$volumeSecrets\n$pullSecrets\n$tlsSecrets" | sort | uniq) \
<(kubectl get secrets -o jsonpath='{.items[*].metadata.name}' | xargs -n1 | sort | uniq)

This diff command compares the combined list of all active secrets with the list of all cluster secrets and identifies any unreferenced secrets that are no longer required.

Delete Orphaned Secrets

Once you've identified the orphaned secrets, you can delete them individually using:

kubectl delete secret [secret-name] --namespace [namespace-name]

You can automate this process with scripts or custom controllers for a more streamlined secret management experience.

After you delete all the orphaned secrets, you'll have removed unneeded resources from your cluster, freeing up storage space and making it more secure. If you remove orphaned resources regularly, you'll ensure that your team maintains optimal Kubernetes resource management.

What Happens When You Delete a Secret in Kubernetes?

When you delete a Kubernetes secret, it can immediately affect the resources that rely on it. For example, pods and services using that secret may fail and be restarted by the kubelet. Understanding these impacts is essential to avoid unexpected issues within your applications.

How Pods and Services React to Secret Deletion

Kubernetes resources that depend on a deleted secret will respond differently based on how they reference it.

• Secrets as Environment Variables: If a pod uses a secret as an environment variable, it'll retain the secret's original values in memory. However, if Kubernetes restarts the Pod after you delete the secret, it will fail to initialize correctly, as the environment variable reference will be invalid. You can recreate or update the deleted secret and restart the affected Pods to fix this issue.

• Secrets Mounted as Volumes: Deleting secrets mounted as volumes will remove their filesystem reference and can crash any application or workload that tries to reference them. To resolve this issue, you can recreate and remount the secret or restart the Pod after updating its configuration with a new secret.

• ImagePullSecrets: If you delete an ImagePullSecret required for pulling container images, new Pod deployments or replicas referencing the secret will fail. Existing Pods will remain unaffected, as they don't need to re-pull the image after initialization. The fix to this problem is to recreate the ImagePullSecret and reapply it to the affected Pods or restart them after updating them with new credentials.

By understanding these common issues caused by active secret deletion and their fixes, you can minimize downtime and prevent issues caused by missing secrets in your Kubernetes environment.

Best Practices for Secret Management

Effective secret management is essential for maintaining a secure and stable Kubernetes environment. Here are some best practices for streamlining your secret management workflow.

‍

This table provides a quick overview of essential practices for secure Kubernetes secret management. The sections below further explain each practice and its implementation.

Regular Audits and Rotation

Audits are crucial to finding unused or orphaned secrets and removing them before they become security risks. You should also rotate your Kubernetes secrets regularly to minimize the impact of potential breaches.

Implement Role-Based Access Control (RBAC)

Implementing Role-Based Access Control (RBAC) policies helps limit who can access, create, or delete secrets in your clusters. RBAC helps enforce the principle of least privilege, ensuring that only authorized users and applications can access sensitive data like authentication tokens and passwords.

Enable Encryption for Secrets at Rest

Kubernetes stores its secrets unencrypted in etcd by default. You can enable etcd encryption for secrets at rest to increase their security. Encrypting secrets reduces the likelihood of unauthorized access to sensitive data, even if a malicious user gains access to the etcd storage.

Use External Secret Management Solutions

Integrating external secret management tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault can considerably improve your Kubernetes cluster's security. These solutions offer advanced features such as automatic rotation, audit logging, and centralized secret management across multiple clusters.

Avoid Using Secrets in Environment Variables

Avoid storing secrets as environment variables since they can be easily exposed or leaked if not handled correctly. Instead, mount them using file system volumes. Using file mounts reduces the risk of accidental exposure and provides a more secure method for applications to access sensitive data.

volumeMounts:
  - name: secret-volume
    mountPath: "/etc/my-secret"
volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

Following these best practices can strengthen the security of your Kubernetes cluster, minimize the risk of unauthorized access to secrets, and improve the overall security of your Kubernetes clusters.

Automatically Identify Orphaned Secrets with Blink

Finding and removing orphaned secrets takes a few steps, and you might need to remember to make it a part of your routine.

With Blink, you can schedule automated checks like this one so you can maintain your Kubernetes clusters in just a couple clicks:

This automation is available in the Blink library. When it runs, it does the following steps:

1. Find unused Secrets on the cluster.
2. Find unused ConfigMaps on the cluster.
3. Find unused Services on the cluster.
4. Send a report to a Slack channel.

This simple automation is easy to customize. Run it on a schedule, add approval steps to remove resources, or send the report via email or Teams instead.

There are over 5K automation in the Blink library to choose from, or you can build your own to match your unique needs.

Get started with Blink today and see how easy automation can be.

Conclusion

Deleting orphaned secrets is an excellent way to reduce sensitive data exposure. However, you must implement a streamlined security management workflow consisting of regular audits, proper access control, secret encryption, and proactive automation to keep your data safe and your cluster running smoothly. Tools like Blink make it easy to spot and clear out unused secrets so your clusters stay lean and protected. 

By implementing a few best practices and incorporating the right tools, you’ll have a more robust, resilient Kubernetes environment that can handle even the most challenging workloads.