Here’s a simple way to expose an Ingress controller running in a private network to the public. My use case is for adhoc standup of Kubernetes clusters via K3d on my computer while testing automation (eg, self managed ArgoCD) so that I can validate cert-manager & external-dns.

I purposefully kept the tools required to a minimum - this is proof of concept stage more than anything. The concepts aren’t new, but so far I haven’t see this exact solution shared. Not that there isn’t prior art in the space: see InletsPRO or ngrok. Both are paid tools - which is a good thing, pay people for good tools!

My use-case can be met with simpler tooling though:

  • 1 VPS w/ public IP (DigitalOcean is a great option at $5/mo)
  • An SSH client sidecar attached to an Ingress controller Pod

That’s it! I don’t have this polished up, but it is working on my home cluster right now.

Generate SSH Key

Use key-based authentication. Just do it.

❯ ssh-keygen -t ed25519 -f ./digitalocean.clusteraccess -C "remote port forward DO to private ingress"

Don’t create a passphrase. Copy the digitalocean.clusteraccess.pub file generated and use it when creating the VPS.

Configure VPS

Again, this isn’t polished - ideally I’d have a simple command here using doctl to create the perfect droplet with the appropriate sshd_config. Suck it up, we’re using the cloud console:

Login or create a DigitalOcean account - if it’s a new account you can get $5 credit by getting an invite from someone (drop me a line on Twitter). You don’t need anything crazy, just a basic droplet in a reasonably close zone. Under Authentication choose New SSH Key. Paste in the .pub key you copied in the previous step.

Once your droplet is created, grab the public IP address and login:

❯ ssh [email protected] -i ./digitalocean.clusteraccess

We need to modify the sshd config to enable GatewayPorts. This will permit listening on the public interface so our SSH port forwarding will have the desired result. Edit /etc/ssh/sshd_config using your favorite editor, vim. Uncomment or add the GatewayPorts entry and set it to yes. Then reload ssh: service ssh restart.

Validate it works

Run an http server on your system. I already had mkdocs running on localhost:8000, but you could as easily do the following:

❯ echo "<b>hello world</b>" > index.html

❯ python3 -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
127.0.0.1 - - [10/Jul/2021 08:59:45] "GET / HTTP/1.1" 200 -

Use whatever port you like, I overrode the default 8000 to 8080 because… mkdocs was running.

Now, test your SSH forwarding:

❯ ssh [email protected] -i ./digitalocean.clusteraccess -R 80:'*':8000

You should now be able to curl 161.35.225.155:

❯ curl 161.35.225.155
<b>hello world</b>

Patch Ingress Controller

Here’s the really fun part, and it’s going to depend heavily on how you deployed your Ingress controller. If you like, you could easily just edit the Deployment object directly. I use helmfile so I opted to use the built-in Kustomize support to patch it in.

I created a ConfigMap in my ingress controller namespace:

⚠️ this is just for the proof of concept, this should really go into a secret!

apiVersion: v1
kind: ConfigMap
metadata:
  name: digitalocean-externalip-key
data: 
  ssh_key: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    **********************************************************************
    **********************************************************************
    *************************itsasecretyo*********************************
    **********************************************************************
    **********************************************************************
    *****=
    -----END OPENSSH PRIVATE KEY-----

And then patched my Ingress controller deployment (note, this is a helmfile specific example):

- name: ingress-nginx-external
  namespace: nginx-system
  chart: nginx-stable/nginx-ingress
  labels:
    repo: nginx-stable
    chart: nginx-ingress
    component: ingress
    domain: external
  version: ~0.9.3
  values:
    - controller:
        defaultTLS.secret: nginx-system/default-{{ .Environment.Values | get "external_domain" "unknown" }}-tls
        wildcardTLS.secret: nginx-system/default-{{ .Environment.Values | get "external_domain" "unknown" }}-tls
        ingressClass: nginx-external
        service:
          annotations:
            metallb.universe.tf/address-pool: k8s-services
  strategicMergePatches:
    - apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ingress-nginx-external-nginx-ingress
        namespace: nginx-system
      spec:
        template:
          spec:
            containers:
            - name: ssh-sidecar
              image: gfleury/ssh-client
              command: ["ssh"]
              args:
                - "-i"
                - "/etc/sshkeys/ssh_key"
                - "-o"
                - "UserKnownHostsFile=/dev/null"
                - "-o"
                - "StrictHostKeyChecking=no"
                - "-N"
                - "-R"
                - "80:0.0.0.0:80"
                - "-R"
                - "443:0.0.0.0:443"
                - "-o ExitOnForwardFailure=yes"
                - "[email protected]"
              volumeMounts:
              - name: sshkey-volume
                mountPath: /etc/sshkeys
            volumes:
              - name: sshkey-volume
                configMap:
                  name: digitalocean-externalip-key
                  defaultMode: 256

You’ll note the extra options in the SSH command. First, I want both 80 and 443 available. Since I’m not running interactively, I’d prefer to skip the host key checking. Finally, if the port forwarding fails I want to retry. So the SSH command will exit causing the container to be restarted. Nifty huh? You can verify by tailing the logs of the ssh-sidecar container, or connecting to :80 on the public IP. You should get an nginx 404.

Create an Ingress

Ok, almost done! Now to create an Ingress. I like using httpbin as a test application.

Run httpbin & create a Kubernetes service:

❯ kubectl -n default run httpbin --image=kennethreitz/httpbin:latest --port=80 --expose=true

And apply this ingress:

⚠️ This assumes you have cert-manager already setup, otherwise drop the cert-manager & tls items

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
  name: httpbin
  namespace: default
spec:
  ingressClassName: nginx-external
  rules:
  - host: httpbin.somedomain.tld
    http:
      paths:
      - backend:
          service:
            name: httpbin
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - httpbin.somedomain.tld
    secretName: httpbin-tls

I created the appropriate DNS record on my domain, but if you’re not using cert-manager you could as easily just add a host entry.

Let’s test:

curl -X GET "https://httpbin.somedomain.tld/base64/ZHVkZSwgdGhpcyBpcyBhYnNvbHV0ZWx5IGFtYXppbmcu" -H  "accept: text/html"

Next Steps & Acknowledgements

This could definitely use polishing up. I added a firewall to my DigitalOcean droplet to allow port 80/443 from anywhere but only accept 22 connections from my home IP. The whole VPS configuration is a simple Terraform or doctl automation. The SSH key should go in a secret, and this solution only works with a single Ingress controller instance at a time. But for a quick hack this works nicely!

I modified my ssh-sidecar config from the one found here. Thanks for the great example gfleury!