↓ Skip to main content
locals.tf

XRDP in k8s

·2473 words·12 mins Draft
Max Severance
Author
Max Severance
Architecting fabric convergence at hyperscale.
Table of Contents

Running xrdp with GPU Acceleration in Kubernetes: A Complete Troubleshooting Guide
#

Introduction
#

In this blog post, I’ll walk you through the complete journey of deploying xrdp (Remote Desktop Protocol server) in a Kubernetes environment with GPU acceleration. What seemed like a straightforward deployment turned into a multi-hour troubleshooting session covering storage classes, volume mounting issues, container security contexts, and resource allocation.

Whether you’re trying to run a graphical desktop environment in a containerized environment or just need remote desktop access to a Linux machine with GPU passthrough, this guide will help you avoid the pitfalls I encountered.

TLDR
#

  • Storage Class Solution: Use hostPath volumes instead of PVCs when the local-path provisioner won’t bind
  • Volume Mount Gotcha: Never mount a hostPath volume directly over /home/user - it overwrites container image files
  • Container Security: Set runAsUser: 1000 to match the USER directive in your Dockerfile
  • CPU Throttling Fix: Start with generous CPU requests (2+ cores) to avoid cgroup throttling
  • Pod Spam Issue: If you see dozens of “NodePorts” pods, force delete stuck replicasets

Table of Contents
#

  1. Initial Setup: Storage Class Issues
  2. The Stuck PVC Problem
  3. Duplicate Resources in Kustomize
  4. Switching to hostPath Volumes
  5. The “start-kde.sh Not Found” Error
  6. Understanding the Volume Mount Issue
  7. Fixing the Container Security Context
  8. CPU Throttling and Performance Tuning
  9. The Mysterious Pod Spam Issue
  10. Final Working Configuration
  11. Summary

1. Initial Setup: Storage Class Issues
#

The session began with attempting to set up persistent storage for the xrdp container. The initial configuration used a manual storage class that didn’t exist in the cluster.

Checking Available Storage Classes
#

First, I checked what storage classes were available in the Kubernetes cluster:

sudo kubectl get storageclass

Output:

NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEATTACHMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  117d

The only available storage class was local-path, which is the default storage class provided by Rancher’s local-path-provisioner.

Updating the PersistentVolume Definition
#

I updated the PV (PersistentVolume) and PVC (PersistentVolumeClaim) to use local-path instead of manual:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: xrdp-home-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-path
  hostPath:
    path: /var/lib/rancher/k3s/storage/xrdp-home
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: xrdp-home
  namespace: xrdp
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: local-path

After updating the file, I reapplied the configuration:

sudo kubectl delete pvc xrdp-home -n xrdp 2>/dev/null
sudo kubectl apply -k dev/xrdpdocker/k8s/

However, this led to a new problem - the PVC got stuck in a “Terminating” state.

2. The Stuck PVC Problem
#

The PVC was stuck in a Terminating state, preventing any new PVC from being created:

sudo kubectl get pvc -n xrdp

Output:

NAME        STATUS        VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
xrdp-home   Terminating                                      manual         <unset>                 3m50s

Force Deleting the Stuck PVC
#

To resolve this, I first deleted the deployment to release any claims on the PVC:

sudo kubectl delete deployment xrdp -n xrdp

Then checked the PVC status again:

sudo kubectl get pvc -n xrdp

Output:

No resources found in xrdp namespace.

The PVC was now deleted. However, there was another issue lurking - duplicate resource definitions.

3. Duplicate Resources in Kustomize
#

When I tried to reapply the Kustomize configuration, I encountered an error:

sudo kubectl apply -k dev/xrdpdocker/k8s/

Error:

error: accumulating resources: accumulation err='merging resources from '10-xrdp.yaml': may not add resource with an already registered id: PersistentVolumeClaim.v1.[noGrp]/xrdp-home.xrdp': must build at directory: 'dev/xrdpdocker/k8s/10-xrdp.yaml': file is not directory

Finding the Duplicate
#

The error revealed there was a duplicate PVC definition. I checked the 10-xrdp.yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: xrdp-home
  namespace: xrdp
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: manual
---
apiVersion: v1
kind: Service
# ... rest of the file

Removing the Duplicate
#

I edited the file to remove the duplicate PVC definition:

apiVersion: v1
kind: Service
metadata:
  name: xrdp
  namespace: xrdp
spec:
  selector:
    app: xrdp
  ports:
    - name: rdp
      port: 3389
      targetPort: 3389
  clusterIP: None

After this fix, the deployment went through:

sudo kubectl apply -k dev/xrdpdocker/k8s/

Output:

namespace/xrdp unchanged
service/xrdp unchanged
persistentvolume/xrdp-home-pv created
persistentvolumeclaim/xrdp-home created
deployment.apps/xrdp created

However, the pod was stuck in ContainerCreating state, and the events showed the PVC was still having issues.

4. Switching to hostPath Volumes
#

The local-path provisioner wasn’t properly binding the PVC. After troubleshooting, I decided to take a simpler approach - using hostPath volumes directly in the deployment instead of dealing with PVCs.

The local-path-provisioner Configuration
#

I checked the local-path-provisioner configuration:

sudo kubectl get configmap -n kube-system local-path-config -o yaml

The provisioner was configured to use /data/k3s/storage as the storage path. Rather than continue fighting with the dynamic provisioning, I switched to a simpler hostPath-based approach.

Simplified Deployment with hostPath
#

I rewrote the deployment to use hostPath volumes directly:

apiVersion: v1
kind: Service
metadata:
  name: xrdp
  namespace: xrdp
spec:
  selector:
    app: xrdp
  ports:
    - name: rdp
      port: 3389
      targetPort: 3389
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xrdp
  namespace: xrdp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xrdp
  template:
    metadata:
      labels:
        app: xrdp
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: xrdp
          image: registry.amzk8s.io/xrdp:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3389
              hostPort: 3389
              name: rdp
          env:
            - name: XDG_RUNTIME_DIR
              value: /tmp/
            - name: XDG_DATA_DIRS
              value: /usr/share/icons/Adwaita:/usr/share/icons:/usr/share/desktop-icons:/usr/share
          volumeMounts:
            - name: home
              mountPath: /home/user
            - name: data
              mountPath: /data
            - name: dri-card0
              mountPath: /dev/dri/card0
            - name: dri-render
              mountPath: /dev/dri/renderD128
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 2000m
              memory: 4Gi
          livenessProbe:
            tcpSocket:
              port: 3389
            initialDelaySeconds: 30
            periodSeconds: 10
      volumes:
        - name: home
          hostPath:
            path: /data/xrdp-home
            type: DirectoryOrCreate
        - name: data
          hostPath:
            path: /data
            type: Directory
        - name: dri-card0
          hostPath:
            path: /dev/dri/card0
            type: CharDevice
        - name: dri-render
          hostPath:
            path: /dev/dri/renderD128
            type: CharDevice
      tolerations:
        - key: "kubernetes.io/arch"
          operator: "Equal"
          value: "amd64"
          effect: "NoSchedule"
      nodeName: changwang

I also updated the kustomization.yaml to only include the necessary files:

resources:
  - 00-namespace.yaml
  - 10-xrdp.yaml

Applied the changes:

sudo kubectl apply -k dev/xrdpdocker/k8s/

The pod started, but then crashed with a mysterious error about a missing file.

5. The “start-kde.sh Not Found” Error
#

The pod went into CrashLoopBackOff status with this error in the events:

Warning  Failed   6s (x4 over 45s)  kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "/home/user/start-kde.sh": stat /home/user/start-kde.sh: no such file or directory

Verifying the File Exists in the Image
#

I was confused because the file should exist in the container image. Let me verify:

sudo nerdctl run --rm registry.amzk8s.io/xrdp:latest ls -la /home/user/

Output:

total 36
drwxr-xr-x 1 root root 4096 Mar  8 20:53 ..
-rw-r--r-- 1 user data  220 Feb 13 10:43 .bash_logout
-rw-r--r-- 1 user data 3526 Feb 13 10:43 .bashrc
-rw-r--r-- 1 user data  5290 Feb 24 17:46 .face
lrwxrwxrwx 1 user data    5 Feb 24 17:46 .face.icon -> .face
-rw-r--r-- 1 user data  807 Feb 13 10:43 .profile
drwxr-xr-x 1 user data 4096 Feb 13 10:40 Desktop
-rwxr-xr-x 1 user data 1464 Mar  8 20:47 start-kde.sh

The file definitely exists in the image! So why isn’t it being found?

Testing the Image Locally
#

I tested running the image locally with nerdctl:

sudo nerdctl run -d --name testxrdp --network=host registry.amzk8s.io/xrdp:latest
sleep 5
sudo nerdctl logs testxrdp

Output:

LANGUAGE=
LC_CTYPE="C.UTF-8"
# ... (environment variables)
[2026-03-08T21:11:32.871+0000] [INFO ] starting xrdp with pid 41
[2026-03-08T21:11:32.872+0000] [INFO ] address [0.0.0.0] port [3389] mode 1
[2026-03-08T21:11:32.872+0000] [INFO ] listening to port 3389 on 0.0.0.0
[2026-03-08T21:11:32.872+0000] [INFO ] xrdp_listen_pp done
[2026-03-08T21:11:32.892+0000] [INFO ] starting xrdp-sesman with pid 44
[2026-03-08T21:11:32.898+0000] [INFO ] Sesman now listening on /run/xrdp/sockdir/sesman.socket

The image runs fine locally! So the issue was specific to how Kubernetes was starting the container.

6. Understanding the Volume Mount Issue
#

The key insight came from examining the volume mounts in the deployment. I had:

volumeMounts:
  - name: home
    mountPath: /home/user
volumes:
  - name: home
    hostPath:
      path: /data/xrdp-home
    type: DirectoryOrCreate

The problem? The hostPath volume was being mounted directly over /home/user, which overwrote everything in that directory from the container image, including the start-kde.sh script!

The hostPath volume at /data/xrdp-home was empty, so when it was mounted at /home/user, it replaced all the files that were in the container image at that location.

The Fix: Mount to a Different Path
#

I changed the mount path to /userdata instead of /home/user:

volumeMounts:
  - name: home
    mountPath: /userdata
  - name: data
    mountPath: /data
  - name: dri-card0
    mountPath: /dev/dri/card0
  - name: dri-render
    mountPath: /dev/dri/renderD128

This way, the container’s /home/user directory (with all its files including start-kde.sh) remains intact, while the persistent data is stored at /userdata.

7. Fixing the Container Security Context
#

Another issue was that the container might be running as root by default in Kubernetes, which could cause issues with the USER directive in the Dockerfile. Let me check what the Dockerfile specifies:

USER user

WORKDIR /home/user

CMD ["/home/user/start-kde.sh"]

The Dockerfile switches to the user (UID 1000). To ensure Kubernetes respects this, I added security context settings:

securityContext:
  privileged: true
  runAsUser: 1000
  runAsGroup: 1004

Applying the Fix
# 

sudo kubectl delete deployment xrdp -n xrdp
sudo kubectl apply -k dev/xrdpdocker/k8s/

Now the pod started successfully:

sudo kubectl get pods -n xrdp

Output:

NAME                   READY   STATUS    RESTARTS   AGE
xrdp-66bccbd8b4-6f277   1/1     Running   0          23s

I verified xrdp was listening on port 3389:

sudo ss -tlnp | grep 3389

Output:

LISTEN 0      2                  *:3389             *:*    users:(("xrdp",pid=2801815,fd=12))

8. CPU Throttling and Performance Tuning
#

The user connected and reported that the session was quite slow. The likely culprit was CPU throttling due to cgroup restrictions from the CPU resource requests.

Initial Resource Configuration
#

The initial configuration had:

resources:
  requests:
    cpu: 500m    # 0.5 CPU cores
    memory: 1Gi
  limits:
    cpu: 2000m   # 2 CPU cores
    memory: 4Gi

First Increase: 1 CPU Request, 4 CPU Limit
#

I increased the resources:

resources:
  requests:
    cpu: "1"     # 1 CPU core
    memory: 2Gi
  limits:
    cpu: "4"     # 4 CPU cores
    memory: 4Gi

Applied the change:

sudo kubectl apply -k dev/xrdpdocker/k8s/

Second Increase: 2 CPU Request, 8 CPU Limit
#

The user still reported slowness, so I increased further:

resources:
  requests:
    cpu: "2"     # 2 CPU cores
    memory: 2Gi
  limits:
    cpu: "8"     # 8 CPU cores
    memory: 8Gi

Applied with a rollout restart:

sudo kubectl rollout restart deployment/xrdp -n xrdp

9. The Mysterious Pod Spam Issue
#

Throughout the troubleshooting, there was a persistent issue where Kubernetes would create dozens (sometimes in a100+) of pods “NodePorts” status. This was highly unusual since the deployment specified only 1 replica.

Symptoms
# 

sudo kubectl get pods -n xrdp

Output:

NAME                    READY   STATUS        RESTARTS   AGE
xrdp-664c4f7b64-85p24   0/1     NodePorts     0          43s
xrdp-664c4f7b64-bdrc6   0/1     NodePorts     0          42s
xrdp-664c4f7b64-c5tgf   0/1     NodePorts     0          39s
# ... 100+ more pods

The events showed errors like:

Warning   NodePorts   pod/xrdp-664c4f7b64-xxx   Predicate NodePorts failed: node(s) didn't have free ports for the requested pod ports

Root Cause
#

This appeared to be a conflict between the old ReplicaSet and the new one during updates, combined with the hostNetwork: true setting and hostPort: 3389. The old pods weren’t being properly cleaned up.

The Fix
#

I had to perform a complete cleanup:

# Delete the deployment
sudo kubectl delete deployment xrdp -n xrdp

# Delete all replicasets
sudo kubectl delete rs --all -n xrdp

# Force delete ALL pods
sudo kubectl delete pods --all -n xrdp --force --grace-period=0

Then reapply the deployment fresh:

sudo kubectl apply -k dev/xrdpdocker/k8s/

This resulted in a clean deployment with just one pod:

sudo kubectl get pods -n xrdp

Output:

NAME                    READY   STATUS    RESTARTS   AGE
xrdp-857cb9ff98-msm5h   1/1     Running   0          24s

10. Final Working Configuration
#

After all the troubleshooting, here’s the final working Kubernetes deployment configuration:

k8s/10-xrdp.yaml
# 

apiVersion: v1
kind: Service
metadata:
  name: xrdp
  namespace: xrdp
spec:
  selector:
    app: xrdp
  ports:
    - name: rdp
      port: 3389
      targetPort: 3389
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xrdp
  namespace: xrdp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xrdp
  template:
    metadata:
      labels:
        app: xrdp
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: xrdp
          image: registry.amzk8s.io/xrdp:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3389
              hostPort: 3389
              name: rdp
          env:
            - name: XDG_RUNTIME_DIR
              value: /tmp/
            - name: XDG_DATA_DIRS
              value: /usr/share/icons/Adwaita:/usr/share/icons:/usr/share/desktop-icons:/usr/share
          volumeMounts:
            - name: home
              mountPath: /userdata
            - name: data
              mountPath: /data
            - name: dri-card0
              mountPath: /dev/dri/card0
            - name: dri-render
              mountPath: /dev/dri/renderD128
          securityContext:
            privileged: true
            runAsUser: 1000
            runAsGroup: 1004
          resources:
            requests:
              cpu: "2"
              memory: 2Gi
            limits:
              cpu: "8"
              memory: 8Gi
          livenessProbe:
            tcpSocket:
              port: 3389
            initialDelaySeconds: 30
            periodSeconds: 10
      volumes:
        - name: home
          hostPath:
            path: /data/xrdp-home
            type: DirectoryOrCreate
        - name: data
          hostPath:
            path: /data
            type: Directory
        - name: dri-card0
          hostPath:
            path: /dev/dri/card0
            type: CharDevice
        - name: dri-render
          hostPath:
            path: /dev/dri/renderD128
            type: CharDevice
      tolerations:
        - key: "kubernetes.io/arch"
          operator: "Equal"
          value: "amd64"
          effect: "NoSchedule"
      nodeName: changwang

k8s/kustomization.yaml
# 

resources:
  - 00-namespace.yaml
  - 10-xrdp.yaml

k8s/00-namespace.yaml
# 

apiVersion: v1
kind: Namespace
metadata:
  name: xrdp

Key Configuration Points
#

Setting Value Purpose
hostNetwork true Use host’s network namespace directly
hostPort 3389 Expose RDP directly on host port
runAsUser 1000 Match the container’s user
runAsGroup 1004 Match the container’s group
privileged true Allow GPU and device access
CPU Request 2 Guarantee minimum CPU
CPU Limit 8 Allow bursting for GUI apps
Volume Mount /userdata Don’t overwrite /home/user

11. Summary
#

This troubleshooting session covered several important Kubernetes and container concepts:

What We Learned
#

  1. Storage Classes: Not all storage classes work the same. The local-path-provisioner in K3s uses WaitForFirstConsumer binding mode, which can cause confusion. For simple use cases, direct hostPath volumes may be easier.

  2. Volume Mount Overwriting: When you mount a hostPath volume to a directory in a container, it completely overwrites whatever was there in the image. Always mount to a different path if you need the image’s files.

  3. Security Context: The USER directive in Dockerfiles only takes effect if Kubernetes respects it. Use runAsUser and runAsGroup in the security context to ensure proper user execution.

  4. CPU Resource Throttling: Desktop environments are CPU-intensive. When cgroups throttle CPU, the desktop becomes unresponsive. Start with generous CPU allocations and adjust based on actual usage.

  5. Stuck Resources: Kubernetes resources can get stuck in terminating states. Force deletion with --force --grace-period=0 is sometimes necessary, followed by a clean redeployment.

  6. hostNetwork Mode: Using hostNetwork: true with hostPort is powerful for exposing services directly, but can cause issues with port conflicts during rolling updates.

Final Result
#

The xrdp deployment is now running successfully with:

  • GPU acceleration via /dev/dri device passthrough
  • Sound support through the container’s PulseAudio/PipeWire configuration
  • 2 CPU cores guaranteed, up to 8 available
  • Persistent storage at /userdata
  • Direct access on host port 3389

The commit was pushed with the message: “Add k8s deployment with hostNetwork, GPU, and CPU resources”

Related Files #

  • Dockerfile - Defines the xrdp container with KDE, PulseAudio, and PipeWire
  • start-kde.sh - Startup script that launches KDE and xrdp
  • k8s/10-xrdp.yaml - Kubernetes Deployment and Service
  • k8s/kustomization.yaml - Kustomize configuration
  • k8s/00-namespace.yaml - Namespace definition

If you’re trying to run a similar setup, I hope this guide saves you the hours I spent debugging these issues!

← From Virtual to Virtualized: Enabling GPU Hardware Acceleration in containerd/xrdp Containers XRdp sound support →