MIG vs MPS in Kubernetes: Re-Carving a GPU Is Not a Pod Restart

"Can we just move transcription to MPS on one GPU and see if it's faster?" It sounded like a ten-minute job: change a config map and restart a pod. It ended up as a de-MIG'd GPU, a hand-written systemd unit, a node that rebooted itself, and an evening where every GPU pod on the node failed with the same CUDA error. This is what I learned.

TL;DR

• MIG splits a GPU in hardware: separate SMs, memory, and fault domain. Kubernetes counts the slices exactly. It's rigid, though: at most 7 slices on an A100/H100.
• MPS shares a GPU in software: many processes in one GPU context, with shared memory and a shared fault domain. It's denser (up to 48 clients), and in our setup Kubernetes can't see it at all.
• Under the GPU Operator, MIG and MPS are mutually exclusive per node. One MIG GPU is enough to block device-plugin MPS on the whole node.
• Changing MIG layout is not a pod restart. mig-manager doesn't drain your workloads, rebuilds every MIG device in the config, can reboot the host, and on failure can leave the node with zero GPUs.
• If you run a host MPS daemon, never use the default pipe path. /tmp/nvidia-mps gets bind-mounted into every GPU container on the node.

MIG vs MPS in One Table

Our setups:

• A DGX A100 dev server: 8× A100-80GB in mig.strategy=mixed, mostly 1g.10gb × 7 per GPU, plus one GPU with 3g.40gb + 4 × 1g.10gb for vLLM. It's a single-node cluster, so the GPU host is also the only control-plane node. Keep that in mind.
• H100 NVL production nodes: one node is MIG'd as a single 7g.94gb for vLLM, and the worker nodes run non-MIG with host-managed MPS.

The Wall: You Can't Mix Them on One Node

The GPU Operator's device plugin can run MPS. But nvidia.com/mps.capable only becomes true when every GPU on the node has MIG disabled. The MPS control daemon DaemonSet is gated on that label and stays at 0/0.

If you enable sharing.mps in the device-plugin config while any GPU is in MIG mode, the plugin refuses to start and the node advertises zero GPUs. If your node ever suddenly shows nvidia.com/gpu: 0, check for this.

I wanted one service on MPS and everything else, including vLLM, on MIG. Converting the whole node wasn't an option.

The Workaround: De-MIG One GPU, Run MPS on the Host, Pin by UUID

1. Take one GPU out of MIG in the mig-parted config:

mig-configs:
  custom-config:
    - devices: [0,1,3,5,6,7]
      mig-enabled: true
      mig-devices: {"1g.10gb": 7}
    - devices: [2]
      mig-enabled: true
      mig-devices: {"3g.40gb": 1, "1g.10gb": 4}
    - devices: [4]
      mig-enabled: false

2. Run an MPS daemon on that GPU, on a private pipe directory, as a systemd unit so it survives reboots:

nvidia-smi -i 4 -c EXCLUSIVE_PROCESS
export CUDA_VISIBLE_DEVICES=<GPU-UUID>
export CUDA_MPS_PIPE_DIRECTORY=/var/run/nvidia-mps-gpu4
export CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps-gpu4
nvidia-cuda-mps-control -d

3. Point the service at the GPU by UUID and skip the device plugin:

spec:
  template:
    spec:
      hostIPC: true                       # mandatory — see below
      containers:
        - name: transcription
          resources:
            limits: {nvidia.com/mig-1g.10gb: null}
          env:
            - {name: NVIDIA_VISIBLE_DEVICES, value: <GPU-UUID>}
            - {name: CUDA_MPS_PIPE_DIRECTORY, value: /var/run/nvidia-mps-gpu4}
          volumeMounts: [{name: mps-pipe, mountPath: /var/run/nvidia-mps-gpu4}]
      volumes: [{name: mps-pipe, hostPath: {path: /var/run/nvidia-mps-gpu4}}]

Use the UUID, not the index. The UUID stays the same across MIG changes and reboots, and the index doesn't.

hostIPC: true is required. Without it you get Error 205: mapping of buffer object failed at cudaGetDeviceCount, and our service quietly fell back to CPU. The only sign was one log line: Inference device: CPU — CUDA not available.

The trade-off: the scheduler doesn't know this GPU is in use. It still advertises nvidia.com/gpu: 1, and any other pod that lands on it fails CUDA init because of EXCLUSIVE_PROCESS.

The worst gotcha: the default MPS pipe affects every pod on the node

My first attempt used the default /tmp/nvidia-mps. libnvidia-container automatically bind-mounts host /tmp/nvidia-mps into every GPU container whenever that path exists. Every CUDA process on the node, including pods on MIG slices that had nothing to do with the test, became an MPS client of my daemon and failed with the same Error 205.

It looked random, because only containers created after the socket appeared were affected. The diarization pods crash-looped, embedding servers hung at "Starting model backend", and vLLM failed during init. It took an hour and a half. The giveaway was that mount | grep mps inside a failing container showed the bind mount.

Always use a private pipe directory.

Why a MIG Change Is Not a Pod Restart

mig-manager doesn't drain your workloads

It pauses the operator's own components and leaves your pods alone. If any GPU it touches has a live CUDA context:

error destroying Compute instance for profile '(0, 0)': In use by another client

And "any GPU it touches" means every MIG device in the config, not only the one you changed. A context on a GPU I hadn't changed blocked the whole apply.

So drain by hand: scale every GPU deployment to 0 and pause KEDA (autoscaling.keda.sh/paused-replicas: "0" on each ScaledObject), or KEDA will scale them straight back up. Our drain was 14 ScaledObject pauses and 17 deployments scaled to zero. Save the replica counts before you start.

Changing MIG mode needs a GPU reset, and draining isn't enough

Changing slice counts inside an already-MIG'd GPU only needs the drain. Turning MIG on or off needs a GPU reset, and on an NVSwitch DGX, nvidia-fabricmanager and nvidia-persistenced keep all 8 GPUs open. mig-manager doesn't stop them. The cleanest approach is to update the config map first, then reboot. mig-manager applies the config at boot, before fabricmanager attaches.

Turn off the reboot fallback before you start

With WITH_REBOOT=true, a failed apply makes mig-manager reboot the host. On our single-node cluster that's also the control plane, so the Kubernetes API was gone for about 3 minutes. I hadn't planned on that.

kubectl patch clusterpolicy cluster-policy --type=json -p \
  '[{"op":"replace","path":"/spec/migManager/env","value":[{"name":"WITH_REBOOT","value":"false"}]}]'

With it off, a failed apply stops at nvidia.com/mig.config.state=failed and waits for you. Have BMC or out-of-band console access ready anyway. A DGX can sit at the AMI BIOS screen for 5–10 minutes during POST, and that's normal.

Also: editing the config map does nothing by itself. Only changing the nvidia.com/mig.config node label (or restarting the mig-manager pod) triggers a re-carve. Check the result with nvidia-smi -L and the node's allocatable counts, not the label.

The mig-manager vs nvsm-core Race

After applying, mig-manager restarts the host GPU services it stopped, with a 2-minute timeout. On a DGX, nvsm-core.service has an ExecStartPost loop that waits for an MQTT probe to return 200, and that takes about 2 minutes 3 seconds. It misses the deadline by 3 seconds, every time:

failed to restart host GPU clients: timeout starting service nvsm-core.service

The label says failed, even though the logs also say "MIG configuration applied successfully".

Fix: wait until systemctl is-active nvsm-core reports active, then restart the mig-manager pod. It sees the config is already applied and reports success.

There was a second, separate cause of the same failed label: a broken host alias, /etc/systemd/system/dcgm.service, symlinked to nvidia-dcgm.service, which fails to load with "File exists". Nothing used it, since dcgm-exporter embeds DCGM. Removing the symlink and running daemon-reload fixed it.

A Failed Apply Leaves You With Zero GPUs

This is the one that pages you. Before applying, mig-manager pauses operator components by setting node labels:

nvidia.com/gpu.deploy.device-plugin=paused-for-mig-change
nvidia.com/gpu.deploy.dcgm-exporter=paused-for-mig-change
...

The DaemonSets select =true, so their pods go away. A successful apply sets the labels back. A failed apply leaves them as they are.

Symptoms:

• Only nvidia-mig-manager is running in the gpu-operator namespace.
• The device-plugin, dcgm-exporter, GFD, and validator DaemonSets show 0 pods.
• The node has nvidia.com/gpu: 0 and every nvidia.com/mig-*: 0.
• Every GPU pod is Pending.

Fix:

kubectl label node <node> \
  nvidia.com/gpu.deploy.device-plugin=true \
  nvidia.com/gpu.deploy.gpu-feature-discovery=true \
  nvidia.com/gpu.deploy.dcgm-exporter=true \
  nvidia.com/gpu.deploy.dcgm=true \
  nvidia.com/gpu.deploy.operator-validator=true \
  nvidia.com/gpu.deploy.container-toolkit=true \
  nvidia.com/gpu.deploy.nvsm=true --overwrite

Devices come back within 30–60 seconds.

Unless the failure happened mid-apply. mig-parted deletes the existing instances first, then fails at the reset step. You end up with MIG mode enabled and zero instances, so fixing the labels still shows 0 devices. Recovery: with the GPUs still drained, change the config map back to match the current MIG mode so no reset is needed, and restart mig-manager to recreate the instances.

One more way to get failed: Longhorn's admission webhook intercepts every Node update, and its failurePolicy is Fail. If longhorn-manager is flapping, mig-manager's label update is rejected (no endpoints available for service longhorn-admission-webhook) and you're stuck with 0 GPUs. When you change MIG, check the webhooks on Node updates too.

So, Was MPS Better?

I don't have a clean benchmark to share. The fair comparison kept getting interrupted by the problems above. One thing I did find: at one point the MIG "control group" image was logging Inference device: CPU (int8) with no error. That comparison would have been GPU vs CPU. Check the device line in the logs before you trust any benchmark.

What I did measure was density:

• 7 MPS replicas on one A100 used 15.2 GB of 80 GB.
• Each replica uses ~2.2 GB idle and ~4 GB at peak, so about 18–20 fit by memory.
• MIG stops at 7 per GPU, no matter how little memory each pod needs.

For a small model that's mostly waiting on a queue, that difference matters. For anything where one tenant crashing must not affect others, MIG is still the right choice.

One operational note: helm upgrade reverts the live patch. After a reinstall, the service quietly went back to MIG slices. The durable fix is to put NVIDIA_VISIBLE_DEVICES, hostIPC, and the hostPath mount in the chart values. We did that on the production H100 nodes.

Lessons

• Choose based on what you need to isolate. MIG gives you isolation and exact scheduling. MPS gives you density and no accounting. Neither is simply "faster".
• MIG and MPS can't share a node under the operator. If you need both, either split by node or run MPS on the host and work around the scheduler, knowing what you give up.
• Treat a MIG change like maintenance. Drain by hand, pause the autoscalers, turn off WITH_REBOOT, have BMC ready, and know if the node is also your control plane.
• Verify on the node, not with the label. nvidia-smi -L and allocatable counts are the truth. mig.config.state can be wrong in both directions.
• Know the gpu.deploy.* labels. If a node suddenly shows 0 GPUs after a MIG change, look at those labels first.
• Never use the default MPS pipe path on a shared node.
• Check the inference device in the logs before benchmarking. A silent fallback to CPU can make any comparison meaningless.

A note on how this was written: I used AI to help with phrasing and structure. The tone, the opinions, the war stories, the code, and everything else here are mine.