Homelab enthusiasts and small IT teams often run everything in virtual machines on a hypervisor such as Proxmox. When they add a GPU for local LLMs, the question arises: should inference run in a VM with the GPU passed through, in a container on the host, or directly on bare metal? Each choice works, but they differ in complexity and in how problems show up.
Why people choose a VM
A VM isolates the AI stack from everything else on the server. You can snapshot it before upgrades, restore it if something breaks, and give it its own network rules. If the same machine also hosts file shares or home automation, isolation feels reassuring.
What passthrough requires
GPU passthrough gives the VM direct access to the physical card. To make it work you typically need a motherboard and CPU that support the relevant virtualization features, correct firmware settings, and a host configuration that releases the GPU so the VM can claim it. Some consumer boards group devices in ways that complicate passthrough. Expect an evening of reading documentation and testing the first time.
Once configured, performance inside the VM is usually close to bare metal for inference, because the heavy work happens on the GPU. The main overhead is administrative, not computational.
Common pitfalls
- Driver mismatch: the VM needs its own GPU driver, and the host must not grab the card first.
- Memory allocation: give the VM enough system RAM for model loading and any CPU offloading. Models load into RAM before moving to VRAM in many runtimes.
- Storage speed: put model files on fast storage accessible to the VM. A slow virtual disk turns model switching into a long wait.
- Reset problems: some cards do not reset cleanly when the VM restarts, requiring a full host reboot.
Containers as an alternative
Running inference in a container on the host, with GPU access through the container runtime, avoids passthrough complexity. You lose some isolation but gain simplicity. The Windows WSL2 and Docker guide shows a similar container approach on Windows, and many of the same ideas apply on Linux hosts.
When bare metal is best
If the machine is dedicated to AI, bare metal is the simplest and easiest to troubleshoot. Use snapshots of configuration files and a written setup log instead of VM snapshots.
A practical recommendation
Start on bare metal or with containers unless you already run a hypervisor and know it well. If you do choose passthrough, document every step, including firmware settings, so you can repeat it after hardware changes. For broader thinking about serving models from a home or office server, the local LLM API server guide is a useful companion.
Virtualization is a tool for managing complexity, not for adding it. Use it when it simplifies your life, and skip it when it does not.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.