↓ Skip to main content
locals.tf

K8sizing homeassistant & Zigbee2MQTT

·1492 words·8 mins
Max Severance
Author
Max Severance
Architecting fabric convergence at hyperscale.
Table of Contents

From Containerd to Kubernetes: Migrating Home Assistant, Mosquitto, and Zigbee2MQTT
#

Introduction
#

In this comprehensive guide, I’ll walk you through the process of migrating standalone containers running in containerd (managed via nerdctl) to Kubernetes. This session covers discovering non-Kubernetesized containers, creating Kubernetes manifests that preserve volumes and configurations, troubleshooting deployment issues, and successfully running IoT services in a Kubernetes cluster.

Whether you’re running Home Assistant, an MQTT broker, or Zigbee home automation services, this step-by-step tutorial will help you make the leap from standalone containers to container orchestration.

TLDR
#

  • Discovered 5 non-Kubernetes containers running via nerdctl: thincast, xrdp, homeassistant, mosquitto, and zigbee2mqtt
  • Created Kubernetes manifests preserving all existing volumes and configurations using hostPath volumes
  • Encountered and fixed a Kubernetes manifest error (invalid devices field in Deployment)
  • Resolved port conflicts by stopping old containers before starting Kubernetes pods
  • Successfully migrated all three services to Kubernetes with full functionality verified

Discovering Non-Kubernetesized Containers
#

The first step was to identify what was running in containerd that wasn’t managed by Kubernetes. We used nerdctl to list all containers and then checked the Kubernetes namespace to identify which containers were NOT part of the k8s.io namespace.

Checking Containerd Containers
# 

# First attempt (without sudo) - failed due to rootless containerd not running
nerdctl ps -a

Output:

time="2026-02-21T00:53:23+02:00" level=fatal msg="rootless containerd not running? (hint: use `containerd-rootless-setuptool.sh install` to start rootless containerd): stat /run/user/1000/containerd-rootless: no such file or directory"

Running with Sudo
# 

sudo nerdctl ps -a

Output:

CONTAINER ID    IMAGE                                           COMMAND                   CREATED         STATUS     PORTS                                               NAMES
8ef5a4c9deda    registry.amzk8s.io/thincast-ltf:1.0.0           "/confine.sh /usr/sb…"    4 weeks ago     Up         10.222.1.22:443->443/tcp, 10.222.1.22:80->80/tcp    thincast
201ef8466c93    locals.tf/xrdp:latest                           "/home/user/start-kd…"    7 weeks ago     Created    0.0.0.0:3389->3389/tcp                              xrdp
a0b74a61e429    ghcr.io/home-assistant/home-assistant:stable    "/init"                   2 months ago    Up                                                             homeassistant
12ed5c83f8cf    docker.io/library/eclipse-mosquitto:latest      "/docker-entrypoint.…"    2 months ago    Up         10.222.1.22:1883->1833/tcp                          mosquitto
0a6a196d7bb1    ghcr.io/koenkk/zigbee2mqtt:latest               "docker-entrypoint.s…"    2 months ago    Up         10.222.1.22:8080->8080/tcp                          zigbee2mqtt

Checking Kubernetes Namespace
# 

sudo nerdctl ps -a --namespace k8s.io

This showed many Kubernetes-managed containers in the k8s.io namespace. The non-Kubernetes containers were in the default namespace:

Container Image Status Ports
thincast registry.amzk8s.io/thincast-ltf:1.0.0 Up 443, 80
xrdp locals.tf/xrdp:latest Created 3389
homeassistant ghcr.io/home-assistant/home-assistant:stable Up -
mosquitto docker.io/library/eclipse-mosquitto:latest Up 1883
zigbee2mqtt ghcr.io/koenkk/zigbee2mqtt:latest Up 8080

Inspecting Volume Configurations
#

Before creating Kubernetes manifests, we needed to understand the volume configuration of each container. We used nerdctl inspect to extract mount information.

Home Assistant Volumes
# 

sudo nerdctl inspect a0b74a61e429 --format '{{json .Mounts}}'

Output:

[
  {"Type":"bind","Source":"/data/hass","Destination":"/config","Mode":"","RW":true,"Propagation":""},
  {"Type":"bind","Source":"/run/dbus","Destination":"/run/dbus","Mode":"ro","RW":false,"Propagation":""}
]

Mosquitto Volumes
# 

sudo nerdctl inspect 12ed5c83f8cf --format '{{json .Mounts}}'

Output:

[
  {"Type":"bind","Source":"/data/mosquitto","Destination":"/mosquitto/config","Mode":"","RW":true,"Propagation":""},
  {"Type":"bind","Source":"/data/mosquitto/data","Destination":"/mosquitto/data","Mode":"","RW":true,"Propagation":""},
  {"Type":"bind","Source":"/data/mosquitto/logs","Destination":"/mosquitto/logs","Mode":"","RW":true,"Propagation":""},
  {"Type":"volume","Name":"8ef721084e53f8798f71e33009cf26e024000bda98da2a6c92c1ff99d83fc742","Source":"/var/lib/nerdctl/1935db59/volumes/default/8ef721084e53f8798f71e33009cf26e024000bda98da2a6c92c1ff99d83fc742/_data","Destination":"/mosquitto/log","Driver":"local","Mode":"","RW":true,"Propagation":""}
]

Zigbee2MQTT Volumes
# 

sudo nerdctl inspect 0a6a196d7bb1 --format '{{json .Mounts}}'

Output:

[
  {"Type":"bind","Source":"/data/zigbee2mqtt/data","Destination":"/app/data","Mode":"","RW":true,"Propagation":""},
  {"Type":"bind","Source":"/data/zigbee2mqtt/logs","Destination":"/app/logs","Mode":"","RW":true,"Propagation":""},
  {"Type":"bind","Source":"/dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e2207336378aef11bd64bea3ef8776e9-if00-port0","Destination":"/dev/ttyUSB0","Mode":"","RW":true,"Propagation":""}
]

The zigbee2mqtt container requires a USB Zigbee dongle (Sonoff Zigbee 3.0 USB Dongle Plus) to be passed through.

Creating Kubernetes Manifests
#

We created a development directory and wrote Kubernetes manifests for each service. The key design decisions were:

  • hostNetwork: true - Preserves the existing IP address (10.222.1.22) and behavior
  • hostPath volumes - Preserves the existing /data/* paths on the host
  • privileged: true - Required for homeassistant and zigbee2mqtt for hardware access
  • New namespace: home-iot - Isolated namespace for IoT services

Directory Setup
# 

mkdir -p ~/dev/hassk8s

Home Assistant Manifest
# 

apiVersion: v1
kind: Service
metadata:
  name: homeassistant
  namespace: home-iot
spec:
  selector:
    app: homeassistant
  ports:
    - port: 8123
      targetPort: 8123
      name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: homeassistant
  namespace: home-iot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: homeassistant
  template:
    metadata:
      labels:
        app: homeassistant
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: homeassistant
          image: ghcr.io/home-assistant/home-assistant:stable
          volumeMounts:
            - name: config
              mountPath: /config
            - name: dbus
              mountPath: /run/dbus
              readOnly: true
          securityContext:
            privileged: true
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "500m"
      volumes:
        - name: config
          hostPath:
            path: /data/hass
        - name: dbus
          hostPath:
            path: /run/dbus

Mosquitto Manifest
# 

apiVersion: v1
kind: Service
metadata:
  name: mosquitto
  namespace: home-iot
spec:
  selector:
    app: mosquitto
  ports:
    - port: 1883
      targetPort: 1883
      name: mqtt
    - port: 9001
      targetPort: 9001
      name: mqtt-websockets
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto
  namespace: home-iot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: mosquitto
          image: docker.io/library/eclipse-mosquitto:latest
          volumeMounts:
            - name: config
              mountPath: /mosquitto/config
            - name: data
              mountPath: /mosquitto/data
            - name: logs
              mountPath: /mosquitto/logs
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "200m"
      volumes:
        - name: config
          hostPath:
            path: /data/mosquitto
        - name: data
          hostPath:
            path: /data/mosquitto/data
        - name: logs
          hostPath:
            path: /data/mosquitto/logs

Zigbee2MQTT Manifest
# 

apiVersion: v1
kind: Service
metadata:
  name: zigbee2mqtt
  namespace: home-iot
spec:
  selector:
    app: zigbee2mqtt
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zigbee2mqtt
  namespace: home-iot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zigbee2mqtt
  template:
    metadata:
      labels:
        app: zigbee2mqtt
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: zigbee2mqtt
          image: ghcr.io/koenkk/zigbee2mqtt:latest
          volumeMounts:
            - name: data
              mountPath: /app/data
            - name: logs
              mountPath: /app/logs
            - name: serial
              mountPath: /dev/ttyUSB0
          securityContext:
            privileged: true
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          env:
            - name: TZ
              value: "Europe/Helsinki"
      volumes:
        - name: data
          hostPath:
            path: /data/zigbee2mqtt/data
        - name: logs
          hostPath:
            path: /data/zigbee2mqtt/logs
        - name: serial
          hostPath:
            path: /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e2207336378aef11bd64bea3ef8776e9-if00-port0

Namespace and Kustomization
# 

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: home-iot
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: home-iot

resources:
  - namespace.yaml
  - homeassistant.yaml
  - mosquitto.yaml
  - zigbee2mqtt.yaml

Initial Deployment and Troubleshooting
#

First Deployment Attempt
#

We started with a limited test deployment (mosquitto and zigbee2mqtt only) to minimize risk:

kubectl apply -k ~/dev/hassk8s/

Output:

namespace/home-iot created
service/homeassistant created
service/mosquitto created
service/zigbee2mqtt created
deployment.apps/homeassistant created
deployment.apps/mosquitto created
Error from server (BadRequest): error when creating "/dev/hassk8s/": Deployment in version "v1" cannot be handled as a Deployment: strict decoding error: unknown field "spec.template.spec.devices"

Fixing the Zigbee2MQTT Manifest
#

The devices field is not valid for a standard Kubernetes Deployment. Instead, we need to use a hostPath volume to expose the USB device. We corrected the manifest:

Before (invalid):

spec:
  containers:
    - name: zigbee2mqtt
      # ... container config
  devices:
    - hostPath: /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e2207336378aef11bd64bea3ef8776e9-if00-port0
      containerPath: /dev/ttyUSB0

After (corrected):

spec:
  containers:
    - name: zigbee2mqtt
      volumeMounts:
        - name: serial
          mountPath: /dev/ttyUSB0
      # ... container config
  volumes:
    - name: serial
      hostPath:
        path: /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_e2207336378aef11bd64bea3ef8776e9-if00-port0

Port Conflict Issues
#

After the manifests were applied, pods entered CrashLoopBackOff or Error states. Checking the logs revealed the root cause:

tail -30 /data/mosquitto/logs/mosquitto.log

Output:

1771801044: Opening ipv4 listen socket on port 1883.
1771801044: Error: Address in use
1771801044: mosquitto version 2.1.2 terminating

The old containerd containers were still running and holding the ports! We needed to stop them before the Kubernetes pods could start.

Stopping Old Containers
# 

sudo nerdctl stop mosquitto
sudo nerdctl stop zigbee2mqtt
sudo nerdctl stop homeassistant

After stopping the old containers, we had to delete and recreate the pods since they were in CrashLoopBackOff:

kubectl delete pod -n home-iot mosquitto-f94c5fb76-qxwp2 zigbee2mqtt-5c99bc6889-ms5q6

Final Pod Status
#

After resolving all issues:

kubectl get pods -n home-iot

Output:

NAME                             READY   STATUS    RESTARTS        AGE
homeassistant-66c57fc64b-4gzlh   1/1     Running   5 (2m22s ago)   5m46s
mosquitto-f94c5fb76-4g2kw        1/1     Running   0               7s
zigbee2mqtt-5c99bc6889-m29h7     1/1     Running   0               7s

Verifying Functionality
#

Testing Home Assistant
# 

curl -s http://10.222.1.22:8123/ | head -5

Home Assistant was responding with its web interface. The logs showed it was connecting to a Roborock device:

2026-02-22 23:04:01.852 INFO (MainThread) [roborock.devices.rpc.v1_channel] [******fRN5E] Connected to device

Testing Zigbee2MQTT
# 

curl -s http://10.222.1.22:8080/ | head -3

The Zigbee2MQTT frontend was accessible. The logs showed successful initialization:

[2026-02-23 01:02:03] info: 	z2m: Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20250321,"transportrev":2},"type":"ZStack3x0"}'
[2026-02-23 01:02:03] info: 	z2m: Connecting to MQTT server at mqtt://10.222.1.22:1883
[2026-02-23 01:02:04] info: 	z2m: Connected to MQTT server
[2026-02-23 01:02:04] info: 	z2m: Zigbee2MQTT started!

Testing Mosquitto
# 

nc -zv 10.222.1.22 1883 2>&1

Output:

Connection to 10.222.1.22 1883 port [tcp/*] succeeded!

Verifying MQTT Message Flow
#

We subscribed to the zigbee2mqtt topic to verify messages were flowing:

kubectl exec -n home-iot deploy/mosquitto -- mosquitto_sub -t 'zigbee2mqtt/#' -C 3 -i test-sub

This confirmed MQTT messages were being published from Zigbee2MQTT to Mosquitto, including Home Assistant auto-discovery messages.

Cleaning Up Old Containers
#

After verification, we removed the old containerd containers:

sudo nerdctl rm homeassistant mosquitto zigbee2mqtt

Summary
#

Successfully migrated three critical IoT services from standalone containerd containers to Kubernetes:

Service Endpoint Status
Home Assistant http://10.222.1.22:8123 ✓ Running
Mosquitto 10.222.1.22:1883 ✓ Running
Zigbee2MQTT http://10.222.1.22:8080 ✓ Running

Key Takeaways
#

  1. Use hostPath volumes carefully: When migrating from standalone containers, hostPath volumes preserve your existing data paths but require the same paths to exist on Kubernetes nodes.

  2. The devices field doesn’t work in Deployments: To pass through hardware devices like USB dongles, use hostPath volumes with volumeMounts instead.

  3. Stop old containers before starting Kubernetes pods: Port conflicts will cause pods to crash if the old containers are still holding the ports.

  4. hostNetwork mode has trade-offs: Using hostNetwork: true preserves the original IP address but bypasses some Kubernetes networking features. For production, consider using ClusterIP services with ingress.

  5. Verify before cleanup: Always verify that Kubernetes pods are working correctly before removing the original containers.

The manifests are now available in ~/dev/hassk8s/ and can be managed with kubectl apply -k ~/dev/hassk8s/. This provides the foundation for future GitOps deployments and easier management of your home IoT infrastructure.

← Blog Automation Setup with OpenCode Docker with xrdp & lxqt →