Crashloop.sh

Kubernetes and Tailscale: What Are Your Options?

Youcef GuichiSep 11, 2024

Today we will show you:

How to Lock down your Kubernetes API server so it is not waving at strangers on the internet.
How to keep your internal services truly internal and secure .
How can Tailscale act as an identity provider.
How to Limit users access with Tailscale.

Ready to make your Kubernetes experience smoother and more secure?

Let’s dive in!

Secure Your Cluster API Server from Public Access

Did you know that 1.6 million Kubernetes clusters are publicly accessible? You can check the numbers here

Tailscale offers a way to limit access to the Kubernetes API server to specific people or services.

But first you would need to get rid of that public ip that is pointing to your cluster api server.

if you are using AKS for example, you will need to create a private cluster from the start. AKS doesn not allow switching from a public to a private cluster after creation, so don not set up Tailscale thinking you can just flip a switch to disable public access later, there isn't one. If you're using a different provider, make sure to check their documentation concerning switching from a public to a private cluster.

Alright let's say now your api server is private, we can deploy a tailscale operator, which will act as a reverse proxy server, all the traffic coming from tailscale users, will go through the tailscale kubernetes operator, the later will forward traffic to default kubernetes api service.

To setup th tailscale operator check their official page they have a very clear instructions. tailscale kubernetes operator

Expose internal services internally

Let's say you have a grafana instance deployed, and you would like to give access to the developers team, so they can debug their application. How are they gonna do that?

I can assume that they already have access to the cluster, and then they will port-forward the grafana service locally.
Or there is an ingress entry with a public endpoint that points to grafana service.
Maybe some other way

The first option sounds neither practical nor a smooth operation, and you have to onboard developers on how to port-forward to there laptops, and also assist them whenever the port-forwad failed.

Come to the second option, great! grafana is accessible directly under a public endpoint, but why would we expose grafana publically, while we only need it internally?

Tailscale offer a simple way to expose the service to your tailnet users, via the following annotation:

apiVersion: v1
kind: Service
metadata:
  name: grafana
  annotations:
    tailscale.com/expose: true

That's it! Go to your tailscale account, browse machines and you will see grafana with a dns name that looks like grafana.tailxxx.ts.net.

DNS, Https and TLS Certificates

Tailscale’s Magic DNS feature simplifies managing DNS and TLS certificates for your local services. Instead of manually handling DNS entries or setting up complex certificate management processes, Magic DNS automatically assigns domain names to your local services based on their Tailscale identity. This greatly simplifies the process of accessing local services securely, even across different environments.

Magic DNS also integrates with Let’s Encrypt to automatically issue TLS certificates, ensuring that all traffic between your services is encrypted. This approach reduces the operational overhead of managing DNS and certificates while enhancing security for internal communications.

Tailscale as an Identity Provider

Tailscale can be used as an identity provider for your Kubernetes clusters, and it operates on two modes when using the tailscale kubernetes operator.

Auth mode

In auth mode, requests from the tailnet proxied over to the Kubernetes API server are additionally impersonated using the sender's tailnet identity. Kubernetes RBAC can then be used to configure granular API server permissions for individual tailnet identities or groups.

Meaning that the users when they perform kubectl operations, the requests goes through the tailcae operator (which act as the api server proxy), the latter will impersonate your identity and forward the request to the kubernetes api claiming that it is you who did the request.

and then to make sure that the user doesn't do more than he should, we can rbac your tailscale identity to be able to do only what you should be able to do.

For more details you can check users impersonation on kubernetes docs here

Example on how to set the auth mode to true on the tailscale helm chart:

values:
  apiServerProxyConfig:
    mode: "true"

Noauth mode

In noauth mode, requests from the tailnet will be proxied over to the Kubernetes API server but not authenticated. This mechanism can be combined with another authentication/authorization mechanism, such as an authenticating proxy provided by an external IDP or a cloud provider.

If you already have an IDP, like let's say microsoft entra id, then you can still leverage it, and tailscale in this case will just proxy traffic to your api server.

ACL Your Users

Tailscale allows you to define ACLs for users accessing your Kubernetes clusters, ensuring that only authorised users can access specific resources or perform specific actions.

You can easily configure ACLs in Tailscale's admin console to control which users have access to certain clusters, services, or resources.

For example group beta can only access the staging-cluster

{
    "acls": [
    {
        "action": "accept",
        "src": [ "group:beta" ], // These sources (devices or users)
        "dst": [ "tag@staging-cluster" ], // can access these destination devices on their defined ports
    }
  ]
}

Conclusion

Integrating Tailscale with Kubernetes offers numerous benefits, from securing your API server to simplifying service exposure between clusters and improving authentication. By leveraging features like Magic DNS, TLS certificates, and access control, you can create a more secure, manageable, and efficient Kubernetes environment.

./Crashloop.sh