- Security TWENTY
- Women in Security
A common query I receive from Kubernetes users is how best to confirm their deployment is configured safely and running securely. Although the best answer is usually to hire someone to pen test it for you, this is also the most expensive option, and the budget conscious amongst you might be looking for something a little more cost-effective, writes Liz Rice, VP Open Source Engineering, at the cloud security company Aqua Security.
Cheaper and quicker options to identify common configuration issues do exist though. Free DIY open source pen testing tools such as our own open source creation, kube-hunter, are a potential alternate route.
That said, I want to stress that kube-hunter is not a replacement for comprehensive pen testing. Penetration testers use the same tools that hackers do to search for weaknesses in your deployment. They will run various types of tests to locate exploitable weaknesses on all possible entry points, adjusting and adapting their approach to go above and beyond common attack scenarios, and mimic the actions of a live attacker.
However, if you do want to focus on evaluating your own environment, kube-hunter enables you to search for security weaknesses in your Kubernetes clusters. In essence, it scans the ports typically used by Kubernetes APIs. If they’re open it makes requests over the various REST API interfaces to see if they could be abused.
One infamous example of how Kubernetes can be exploited was Tesla back in 2018. Someone at the company had left the Kubernetes dashboard open to the internet with cluster administrator privileges. An attacker discovered this access point and subsequently used the cluster to run crypto mining software at the company’s expense.
Since that high profile incident there has been a lot of work dedicated towards making Kubernetes more secure out of the box. For instance, now the dashboard isn’t installed by default, and there are a number of effective security defaults built into the kubeadm installer. That said, Kubernetes is a double edged sword, and the flexibility it provides also brings complexity, and even the most experienced users run the risk of deploying a system configured in a way that creates vulnerabilities.
Over the remainder of this article I’ll provide an overview of what you should be looking out for when pen testing your Kubernetes deployment, and provide best practices for ensuring no vulnerable ports are left open.
You should always bear in mind the differences between HTTP and HTTPS. Regular, insecure HTTP requests are open to interception by anyone. If you’re using HTTPS, the connection begins life as HTTP before being bumped up to encrypted and secure HTTPS, which prevents you from running both HTTP and HTTPS on the same port. Although HTTPS is of course much more secure than HTTP, you’ll need a certificate to run the server end of an HTTPS connection. Some Kubernetes components, in order to save developers the extra work of certificate configuration, offer the option of using an insecure port. In practice you don’t want to be running an insecure port for obvious reasons, and you want to be sure that your cluster hasn’t inadvertently left any insecure ports open! This is something that kube-hunter will look for and report back on.
If you’re using HTTPS, the client may choose to identify itself over the request, and then the server can use that identity to figure out whether the client is authorised to make the request. For today’s Kubernetes installations, that authorisation is typically configured using role-based access control (RBAC).
When you make a request to the Kubernetes API without supplying any identity, it automatically gets run under the system:anonymous service account which, like any other service account, can have permissions associated with it through RBAC. This of course is a potential security risk, as it means by default some capabilities can be made available even to anonymous users.
Although it’s possible to entirely disable anonymous requests, this isn’t necessarily a worthwhile prospect in practice, as it requires all clients to identify themselves even if they only want to conduct innocuous health checks or service discovery. A worthwhile middle ground could be allowing anonymous requests, but have very restrictive RBAC policies on them. This is how the current defaults are configured on an out-of-the-box installation from the kubeadm tool.
Threat hunting with kube-hunter
As the name suggests, kube-hunter automatically searches for and flags open Kubernetes ports, so you don’t have to do this manually. Start by running kube-hunter on any machine outside of your cluster, and when prompted, provide the domain name or IP address of the cluster. Alternatively, run kube-hunter on a machine within the cluster and select the option to probe all the local network interfaces.
Yet another option is to run kube-hunter inside a pod in your cluster so as to simulate an attack through a compromised application running in that pod. Pods run under a service account, and by default the code running in the pod – in this case, kube-hunter – has access to a token which identifies that service account. Kube-hunter uses the token to explore the API access that this service account token permits. The results of this are dependent on the RBAC settings for that pod’s service account. Ideally then you want to run this in a pod under every possible service account. This provides a view of what an attacker might be able to access if they were able to break into any one of the pods in your deployment.
Passive and active hunters
Kube-hunter will only run “passive hunters” by default. These are a series of tests that probe for potential access points (open ports etc.) within your cluster and then perform read-only tests that won’t change any state. “Active hunting”, on the other hand, can demonstrate that there exists a possible exploit, but could potentially result in state-changing operations on the cluster. For instance, whilst a passive hunter might try to obtain a list of pods; an active hunter might create a pod with a randomised name, and then (if successful) delete it again. As such, this option should be used with some element of care, preferably solely on non-production servers.
As mentioned, kube-hunter is an evolving project that can help you get a good start on analysing vulnerabilities within your Kubernetes deployment. As the project expands, and more hunters are added (you can find a list of the tests by running kube-hunter with the –list parameter), it will provide even more functionality thanks to the efforts of the community at large. The project is open source, and we welcome ideas for new hunters that test for additional potential weaknesses. I hope this article has helped you grasp the basics, and happy hunting!