Jellyfin + NVENC Hardware Acceleration Setup (Quadro P400)

This guide outlines how to enable NVIDIA GPU hardware acceleration for transcoding in Jellyfin, using a Quadro P400 GPU on a Linux-based media server (Debian 12). It covers NVENC setup, resolving playback errors, and performance tuning.

🧱 Hardware and OS

  • GPU: NVIDIA Quadro P400
  • Host OS: Debian 12 (Bookworm)
  • Jellyfin: Installed via APT repository
  • Container: LXC on Proxmox (privileged)

1️⃣ NVIDIA Driver Installation

Add non-free components

Edit /etc/apt/sources.list and ensure it includes:

# Debian Bookworm
deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware

Install NVIDIA drivers

sudo apt update && sudo apt install -y libnvcuvid1 libnvidia-encode1
sudo apt install -y nvidia-driver firmware-misc-nonfree
sudo reboot

⚠️ Note: Disable Secure Boot in your BIOS/UEFI settings — otherwise the NVIDIA driver may not load, and the VM may fail to start.

Verify:

nvidia-smi

2️⃣ Check FFmpeg NVENC Support

sudo -u jellyfin ffmpeg -hide_banner -encoders | grep nvenc

You should see:

  • h264_nvenc
  • hevc_nvenc

If av1_nvenc appears but fails — it’s not supported on P400. Ignore it or disable AV1 encoding.

3️⃣ Add Jellyfin to Video Group

sudo usermod -aG video jellyfin
sudo reboot

4️⃣ Fix /etc/jellyfin/encoding.xml

Ensure:

<HardwareAccelerationType>nvenc</HardwareAccelerationType>
<EnableHardwareEncoding>true</EnableHardwareEncoding>
<AllowAv1Encoding>false</AllowAv1Encoding> <!-- Disable this for P400 -->

And confirm:

<HardwareDecodingCodecs>
  <string>h264</string>
  <string>hevc</string>
  <string>mpeg2video</string>
  <string>mpeg4</string>
</HardwareDecodingCodecs>

5️⃣ Set Permissions on Media Directory

Ensure qbittorrent and jellyfin can read/write to /mnt/media:

sudo chown -R jellyfin:jellyfin /mnt/media
sudo chmod -R 775 /mnt/media

6️⃣ Test Playback (1080p H264/HEVC)

Enable in Jellyfin:

  • ✔️ Hardware decoding: H264, HEVC, etc.
  • ✔️ Enable Enhanced NVDEC Decoder
  • ✔️ Enable Hardware Encoding (disable AV1 encoding if playback fails)

7️⃣ Troubleshooting

❌ Provided device doesn't support required NVENC features

  • You’re trying to use AV1 on a GPU that doesn’t support it (P400).
  • Fix: Set <AllowAv1Encoding>false</AllowAv1Encoding>

❌ Transcode fails to start

  • Check logs: 
    less /var/log/jellyfin/FFmpeg.Transcode-*.log
  • Manually test transcoding: 
    sudo -u jellyfin ffmpeg -i input.mkv -c:v h264_nvenc -f null -

🔍 Confirm NVENC is used

Run during playback:

watch -n 1 nvidia-smi

Look for ffmpeg under processes using GPU.

✅ Final Result

Once setup is correct:

  • Jellyfin will use NVENC for transcoding
  • CPU load will stay low
  • Logs will show lines like: Using NVENC encoder: h264_nvenc