Mosquitto MQTT broker in Kubernetes
Mosquitto MQTT broker in Kubernetes
Venturing into the world of deploying an MQTT broker in Kubernetes can lead you to some intriguing challenges, especially when compared to the more traditional tasks of serving webpages or APIs
- Unlike HTTP-based services, MQTT is served over TCP. Consequently, some common ingress solutions, such as ingress-nginx, do not natively support serving TCP traffic. We'll explore how we could bypass this limitation.
- Ensuring secure communication between the MQTT broker and clients is crucial. However, manually creating and managing SSL certificates can be a tedious process. We'll demonstrate an automated method for managing SSL certificates, making your MQTT communication secure and worry-free
First things first, let's generate the certificate we want to use. The "de facto standard" for this task is cert-manager, and we'll use a cert-manager-generated certificate in the same way we do with other ingress-generated certificates. This ensures the certificate stays up-to-date.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: mosquitto annotations: kubernetes.io/ingress.class: "nginx" cert-manager.io/cluster-issuer: "letsencrypt" spec: tls: - hosts: - mqtt.example.com secretName: mosquitto-certs rules: - host: mqtt.example.com http: paths: - path: / pathType: Prefix backend: service: name: mosquitto-mqtts port: number: 8883
As long as your DNS points to the Ingress controller's load balancer for the specified host making it possible for cert-manager to validate your host, you should have a secret containing the certificate in your cluster.
Now, it's time to deploy an MQTT broker. Although we often prefer cloud-native brokers like EMQX, we've opted for Mosquitto in this example due to its simplicity.
We'll refer to the generated secret when specifying the certificate location in the config.
apiVersion: apps/v1 kind: Deployment metadata: name: mosquitto spec: replicas: 1 selector: matchLabels: app: mosquitto template: metadata: labels: app: mosquitto spec: containers: - name: mosquitto image: eclipse-mosquitto ports: - containerPort: 8883 - containerPort: 9001 volumeMounts: - mountPath: /mosquitto/config/mosquitto.conf subPath: mosquitto.conf name: config - mountPath: /mosquitto/certs/ name: certs volumes: - name: config configMap: name: mosquitto-config - name: certs secret: secretName: mosquitto-certs apiVersion: v1 kind: ConfigMap metadata: name: mosquitto-config data: mosquitto.conf: | # DO NOT USE IN PRODUCTION allow_anonymous true # MQTT with TLS (MQTTS) listener 8883 protocol mqtt # Fetch the generated certificates cafile /etc/ssl/certs/ca-certificates.crt keyfile /mosquitto/certs/tls.key certfile /mosquitto/certs/tls.crt apiVersion: v1 kind: Service metadata: name: mosquitto-mqtts spec: type: ClusterIP selector: app: mosquitto ports: - port: 8883
Keep in mind that the example above allows anonymous access to the broker. In a production environment, you'll need to secure your endpoint with an ACL, password list, or similar method and disable anonymous access.
After deploying the components mentioned above, you should have a working MQTT broker with a certificate securing it. However, it's not yet accessible from outside since TCP traffic isn't routed by ingress-nginx.
To overcome this, we can leverage a lesser-known feature in ingress-nginx that allows routing specific ports to TCP services within the cluster. Note that we're not using any host headers here, so any request to any host pointing to the ingress service will be routed on the specified port.
To do this we will use a configmap called tcp-services to keep track of routed ports.
apiVersion: v1 kind: ConfigMap metadata: name: tcp-services namespace: ingress-nginx data: 8883: "mosquitto/mosquitto:8883"
The last part is to tell ingess-nginx to read the configmap and allow port 8883 into to load balancer.
Add this arg to the ingress controller
containers: - args: - /nginx-ingress-controller - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key # Add this - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
Add port 8883 to the ingress load balancer ports.
ports: - name: mqtts port: 8883 protocol: TCP targetPort: 8883 - appProtocol: http name: http port: 80 protocol: TCP targetPort: http - appProtocol: https name: https port: 443 protocol: TCP targetPort: https
Assuming you have a service named "mosquitto-mqtts" in the "mosquitto" namespace, this should route the port to the broker. Now, you're all set with a fully functioning Mosquitto MQTT broker with encrypted communication in Kubernetes. Enjoy!