Focus Determines Reality
Cheap, fast, redundant storage for Proxmox.
Published:

Table of Contents

Introduction and constraints

I just want to document and contribute a little bit of computer science to people using Proxmox. In March 2025, I built a relatively cheap storage array for the disobey.net Proxmox server that got installed into Emerald Onion's Amsterdam datacenter. I had some unique constraints:

  1. 2U custom built Ryzen 9 5900X server w/ 128GB RAM
  2. 1x PCIe 4.0 x16 slot with x4/x4/x4/x4 bifurcation
  3. No storage bays of any kind, only 1x onboard M.2 NVMe slot

What I built

  1. 1x Lian Li PW-PCIV-4-90X PCIe 4.0 riser cable (104 EUR)
  2. 1x ASUS Hyper M.2 x16 Gen 4 PCIe expansion card (75 EUR)
  3. 4x Kingston KC3000 4TB M.2 NVMe SSD (278 EUR each)

With this setup, I installed Proxmox 8.x using RAIDZ1 on this PCIe disk array. The total size of this array came out to be 16.37 TB, or 14.9 TiB; and after RAIDZ1 parity, 10.47 TiB usable.

Advantages

  1. Only cost me 1291 EUR (hardware donation to Emerald Onion)
  2. Provides ZFS resiliency and native snapshot capability
  3. One disk can fail and still be fully operational

Disadvantages

  1. If a disk fails, I have to take the whole system offline to do a replacement.

Why the Kingston KC3000 4TB?

I wanted some enterprise specs without the enterprise cost.

  1. Controller: Phison E18
  2. NAND: 3D TLC
  3. Total Bytes Written (TBW): 4096GB – 3.2PBW
  4. Mean Time Between Failures (MTBF): 2,000,000 hours
  5. Warranty: 5 years

Real world speed test

In order to move a Proxmox VM (in this example, 107), 1TB in size, to an off-site non-clustered host, I made an on-disk, compressed backup. This shows near real-time I/O that I thought would be a useful data point for someone.

Notice, at the end: transferred 1.00 TiB in 738 seconds (1.4 GiB/s)

root@proxmox:~# mkdir -p /var/lib/vz/dump

root@proxmox:~# df -h /var/lib/vz/dump 
Filesystem        Size  Used Avail Use% Mounted on
rpool/var-lib-vz  5.7T  7.7G  5.7T   1% /var/lib/vz

root@proxmox:~# vzdump 107 --mode stop --compress zstd --dumpdir /var/lib/vz/dump
INFO: starting new backup job: vzdump 107 --dumpdir /var/lib/vz/dump --mode stop --compress zstd
INFO: Starting Backup of VM 107 (qemu)
INFO: Backup started at 2025-11-08 22:27:24
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: Immich
INFO: include disk 'scsi0' 'local-zfs:vm-107-disk-0' 1T
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-107-2025_11_08-22_27_24.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task 'bb1f85b8-ddea-425c-b21a-805dbdd6d012'
INFO:   0% (2.8 GiB of 1.0 TiB) in 3s, read: 961.8 MiB/s, write: 408.8 MiB/s
INFO:   1% (10.8 GiB of 1.0 TiB) in 20s, read: 477.9 MiB/s, write: 467.0 MiB/s
INFO:   2% (21.0 GiB of 1.0 TiB) in 38s, read: 582.0 MiB/s, write: 572.2 MiB/s
INFO:   3% (31.3 GiB of 1.0 TiB) in 56s, read: 588.9 MiB/s, write: 580.2 MiB/s
INFO:   4% (41.3 GiB of 1.0 TiB) in 1m 16s, read: 512.0 MiB/s, write: 500.9 MiB/s
INFO:   5% (51.4 GiB of 1.0 TiB) in 1m 36s, read: 514.8 MiB/s, write: 504.7 MiB/s
INFO:   6% (61.8 GiB of 1.0 TiB) in 1m 52s, read: 669.0 MiB/s, write: 658.9 MiB/s
INFO:   7% (72.3 GiB of 1.0 TiB) in 2m 9s, read: 629.9 MiB/s, write: 618.6 MiB/s
INFO:   8% (82.2 GiB of 1.0 TiB) in 2m 27s, read: 564.9 MiB/s, write: 553.2 MiB/s
INFO:   9% (92.3 GiB of 1.0 TiB) in 2m 47s, read: 514.0 MiB/s, write: 505.1 MiB/s
INFO:  10% (102.8 GiB of 1.0 TiB) in 3m 11s, read: 449.5 MiB/s, write: 441.7 MiB/s
INFO:  11% (113.0 GiB of 1.0 TiB) in 3m 31s, read: 519.8 MiB/s, write: 509.6 MiB/s
INFO:  12% (123.3 GiB of 1.0 TiB) in 3m 48s, read: 624.6 MiB/s, write: 614.5 MiB/s
INFO:  13% (133.3 GiB of 1.0 TiB) in 4m 7s, read: 538.6 MiB/s, write: 528.6 MiB/s
INFO:  14% (143.8 GiB of 1.0 TiB) in 4m 26s, read: 564.7 MiB/s, write: 555.5 MiB/s
INFO:  15% (153.8 GiB of 1.0 TiB) in 4m 46s, read: 512.5 MiB/s, write: 502.8 MiB/s
INFO:  16% (163.9 GiB of 1.0 TiB) in 5m 6s, read: 515.9 MiB/s, write: 505.4 MiB/s
INFO:  17% (174.2 GiB of 1.0 TiB) in 5m 26s, read: 528.3 MiB/s, write: 516.7 MiB/s
INFO:  18% (184.3 GiB of 1.0 TiB) in 5m 45s, read: 546.8 MiB/s, write: 538.5 MiB/s
INFO:  19% (194.8 GiB of 1.0 TiB) in 6m 6s, read: 511.6 MiB/s, write: 500.2 MiB/s
INFO:  20% (205.4 GiB of 1.0 TiB) in 6m 20s, read: 768.8 MiB/s, write: 461.0 MiB/s
INFO:  21% (217.5 GiB of 1.0 TiB) in 6m 31s, read: 1.1 GiB/s, write: 425.8 MiB/s
INFO:  22% (225.7 GiB of 1.0 TiB) in 6m 34s, read: 2.8 GiB/s, write: 5.3 KiB/s
INFO:  23% (235.8 GiB of 1.0 TiB) in 6m 45s, read: 937.2 MiB/s, write: 507.9 MiB/s
INFO:  24% (246.0 GiB of 1.0 TiB) in 7m 11s, read: 401.5 MiB/s, write: 389.7 MiB/s
INFO:  25% (256.2 GiB of 1.0 TiB) in 7m 34s, read: 452.7 MiB/s, write: 432.3 MiB/s
INFO:  26% (266.4 GiB of 1.0 TiB) in 7m 50s, read: 655.0 MiB/s, write: 305.9 MiB/s
INFO:  27% (278.5 GiB of 1.0 TiB) in 7m 57s, read: 1.7 GiB/s, write: 149.2 MiB/s
INFO:  28% (287.5 GiB of 1.0 TiB) in 8m, read: 3.0 GiB/s, write: 5.3 KiB/s
INFO:  29% (299.3 GiB of 1.0 TiB) in 8m 4s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  30% (308.1 GiB of 1.0 TiB) in 8m 7s, read: 3.0 GiB/s, write: 178.7 KiB/s
INFO:  31% (319.4 GiB of 1.0 TiB) in 8m 11s, read: 2.8 GiB/s, write: 5.0 KiB/s
INFO:  32% (328.4 GiB of 1.0 TiB) in 8m 14s, read: 3.0 GiB/s, write: 6.7 KiB/s
INFO:  33% (340.4 GiB of 1.0 TiB) in 8m 18s, read: 3.0 GiB/s, write: 6.0 KiB/s
INFO:  34% (349.3 GiB of 1.0 TiB) in 8m 21s, read: 3.0 GiB/s, write: 5.3 KiB/s
INFO:  35% (361.0 GiB of 1.0 TiB) in 8m 25s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  36% (369.6 GiB of 1.0 TiB) in 8m 28s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  37% (381.2 GiB of 1.0 TiB) in 8m 32s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  38% (390.0 GiB of 1.0 TiB) in 8m 35s, read: 2.9 GiB/s, write: 6.7 KiB/s
INFO:  39% (401.6 GiB of 1.0 TiB) in 8m 39s, read: 2.9 GiB/s, write: 134.0 KiB/s
INFO:  40% (410.2 GiB of 1.0 TiB) in 8m 42s, read: 2.9 GiB/s, write: 6.7 KiB/s
INFO:  41% (421.7 GiB of 1.0 TiB) in 8m 46s, read: 2.9 GiB/s, write: 5.0 KiB/s
INFO:  42% (430.3 GiB of 1.0 TiB) in 8m 49s, read: 2.9 GiB/s, write: 6.7 KiB/s
INFO:  43% (441.7 GiB of 1.0 TiB) in 8m 53s, read: 2.9 GiB/s, write: 5.0 KiB/s
INFO:  44% (450.7 GiB of 1.0 TiB) in 8m 56s, read: 3.0 GiB/s, write: 6.7 KiB/s
INFO:  45% (462.3 GiB of 1.0 TiB) in 9m, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  46% (471.1 GiB of 1.0 TiB) in 9m 3s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  47% (482.5 GiB of 1.0 TiB) in 9m 7s, read: 2.8 GiB/s, write: 6.0 KiB/s
INFO:  48% (494.0 GiB of 1.0 TiB) in 9m 11s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  49% (502.7 GiB of 1.0 TiB) in 9m 14s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  50% (514.0 GiB of 1.0 TiB) in 9m 18s, read: 2.8 GiB/s, write: 6.0 KiB/s
INFO:  51% (522.6 GiB of 1.0 TiB) in 9m 21s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  52% (534.2 GiB of 1.0 TiB) in 9m 25s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  53% (545.5 GiB of 1.0 TiB) in 9m 29s, read: 2.8 GiB/s, write: 5.0 KiB/s
INFO:  54% (554.2 GiB of 1.0 TiB) in 9m 32s, read: 2.9 GiB/s, write: 6.7 KiB/s
INFO:  55% (565.9 GiB of 1.0 TiB) in 9m 36s, read: 2.9 GiB/s, write: 5.0 KiB/s
INFO:  56% (574.2 GiB of 1.0 TiB) in 9m 39s, read: 2.7 GiB/s, write: 6.7 KiB/s
INFO:  57% (585.8 GiB of 1.0 TiB) in 9m 43s, read: 2.9 GiB/s, write: 5.0 KiB/s
INFO:  58% (594.0 GiB of 1.0 TiB) in 9m 46s, read: 2.7 GiB/s, write: 5.3 KiB/s
INFO:  59% (605.3 GiB of 1.0 TiB) in 9m 50s, read: 2.8 GiB/s, write: 6.0 KiB/s
INFO:  60% (616.9 GiB of 1.0 TiB) in 9m 54s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  61% (625.5 GiB of 1.0 TiB) in 9m 57s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  62% (637.0 GiB of 1.0 TiB) in 10m 1s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  63% (645.6 GiB of 1.0 TiB) in 10m 4s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  64% (656.8 GiB of 1.0 TiB) in 10m 8s, read: 2.8 GiB/s, write: 6.0 KiB/s
INFO:  65% (665.7 GiB of 1.0 TiB) in 10m 11s, read: 3.0 GiB/s, write: 5.3 KiB/s
INFO:  66% (677.5 GiB of 1.0 TiB) in 10m 15s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  67% (686.2 GiB of 1.0 TiB) in 10m 18s, read: 2.9 GiB/s, write: 6.7 KiB/s
INFO:  68% (698.0 GiB of 1.0 TiB) in 10m 22s, read: 3.0 GiB/s, write: 6.0 KiB/s
INFO:  69% (709.5 GiB of 1.0 TiB) in 10m 26s, read: 2.9 GiB/s, write: 5.0 KiB/s
INFO:  70% (717.1 GiB of 1.0 TiB) in 10m 29s, read: 2.5 GiB/s, write: 5.3 KiB/s
INFO:  71% (727.4 GiB of 1.0 TiB) in 10m 33s, read: 2.6 GiB/s, write: 5.0 KiB/s
INFO:  72% (738.0 GiB of 1.0 TiB) in 10m 37s, read: 2.6 GiB/s, write: 5.0 KiB/s
INFO:  73% (749.6 GiB of 1.0 TiB) in 10m 41s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  74% (758.2 GiB of 1.0 TiB) in 10m 44s, read: 2.9 GiB/s, write: 6.7 KiB/s
INFO:  75% (770.1 GiB of 1.0 TiB) in 10m 48s, read: 3.0 GiB/s, write: 6.0 KiB/s
INFO:  76% (781.1 GiB of 1.0 TiB) in 10m 52s, read: 2.8 GiB/s, write: 5.0 KiB/s
INFO:  77% (789.7 GiB of 1.0 TiB) in 10m 55s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  78% (801.2 GiB of 1.0 TiB) in 10m 59s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  79% (810.1 GiB of 1.0 TiB) in 11m 2s, read: 3.0 GiB/s, write: 6.7 KiB/s
INFO:  80% (821.9 GiB of 1.0 TiB) in 11m 6s, read: 2.9 GiB/s, write: 5.0 KiB/s
INFO:  81% (830.7 GiB of 1.0 TiB) in 11m 9s, read: 2.9 GiB/s, write: 178.7 KiB/s
INFO:  82% (842.4 GiB of 1.0 TiB) in 11m 13s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  83% (851.3 GiB of 1.0 TiB) in 11m 16s, read: 3.0 GiB/s, write: 5.3 KiB/s
INFO:  84% (862.9 GiB of 1.0 TiB) in 11m 20s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  85% (872.1 GiB of 1.0 TiB) in 11m 23s, read: 3.1 GiB/s, write: 6.7 KiB/s
INFO:  86% (880.7 GiB of 1.0 TiB) in 11m 26s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  87% (892.5 GiB of 1.0 TiB) in 11m 30s, read: 3.0 GiB/s, write: 6.0 KiB/s
INFO:  88% (904.2 GiB of 1.0 TiB) in 11m 34s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  89% (912.9 GiB of 1.0 TiB) in 11m 38s, read: 2.2 GiB/s, write: 4.0 KiB/s
INFO:  90% (924.6 GiB of 1.0 TiB) in 11m 42s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  91% (932.4 GiB of 1.0 TiB) in 11m 45s, read: 2.6 GiB/s, write: 5.3 KiB/s
INFO:  92% (943.4 GiB of 1.0 TiB) in 11m 49s, read: 2.8 GiB/s, write: 5.0 KiB/s
INFO:  93% (954.7 GiB of 1.0 TiB) in 11m 53s, read: 2.8 GiB/s, write: 6.0 KiB/s
INFO:  94% (963.3 GiB of 1.0 TiB) in 11m 56s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  95% (974.9 GiB of 1.0 TiB) in 12m, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  96% (983.7 GiB of 1.0 TiB) in 12m 3s, read: 2.9 GiB/s, write: 5.3 KiB/s
INFO:  97% (995.3 GiB of 1.0 TiB) in 12m 7s, read: 2.9 GiB/s, write: 6.0 KiB/s
INFO:  98% (1004.0 GiB of 1.0 TiB) in 12m 10s, read: 2.9 GiB/s, write: 6.7 KiB/s
INFO:  99% (1015.6 GiB of 1.0 TiB) in 12m 14s, read: 2.9 GiB/s, write: 5.0 KiB/s
INFO: 100% (1.0 TiB of 1.0 TiB) in 12m 18s, read: 2.1 GiB/s, write: 7.0 KiB/s
INFO: backup is sparse: 792.60 GiB (77%) total zero data
INFO: transferred 1.00 TiB in 738 seconds (1.4 GiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 212.09GB
trying to acquire lock...
 OK
INFO: Finished Backup of VM 107 (00:12:20)
INFO: Backup finished at 2025-11-08 22:39:44
INFO: Backup job finished successfully
INFO: notified via target `mail-to-root`

root@proxmox:~# df -h /var/lib/vz/dump 
Filesystem        Size  Used Avail Use% Mounted on
rpool/var-lib-vz  5.7T  220G  5.5T   4% /var/lib/vz

More data

root@proxmox:~# zpool list -o name,size,alloc,free,cap,health
NAME    SIZE  ALLOC   FREE    CAP    HEALTH
rpool  14.9T  7.28T  7.61T    48%    ONLINE

root@proxmox:~# zpool status -v
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 01:19:02 with 0 errors on Sun Oct 12 01:43:03 2025
config:

        NAME                                                 STATE     READ WRITE CKSUM
        rpool                                                ONLINE       0     0     0
          raidz1-0                                           ONLINE       0     0     0
            nvme-eui.00000000000000000026b76870365735-part3  ONLINE       0     0     0
            nvme-eui.00000000000000000026b76870352675-part3  ONLINE       0     0     0
            nvme-eui.00000000000000000026b76870355b15-part3  ONLINE       0     0     0
            nvme-eui.00000000000000000026b768703527c5-part3  ONLINE       0     0     0