General ZFS Notes
Just some random notes on how to setup ZFS.
ZFS on Linux
When setting up a zpool you specify disks in a way that remains consistent across reboots and systems. Reading https://wiki.archlinux.org/index.php/Persistent_block_device_naming#by-id_and_by-path it mentions that the wwn-XXXX names should be the fully persistent.
ZFS Permissions:
1
2
3
zfs mount mypool/myzfs
chown myuser:mygroup /mypool/myzfs
ZFS volumes
View volume list:
1
zfs list
ZFS volume devices:
These devices are in /dev/zvol/
Pool can be multi-layer (e.g. rpool/data).
With a setup like the following (done via proxmox host but should apply to other ZFS systems):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
root@pve001:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 663G 236G 104K /rpool
rpool/ROOT 567G 236G 96K /rpool/ROOT
rpool/ROOT/pve-1 567G 236G 567G /
rpool/data 66.0G 236G 96K /rpool/data
rpool/data/subvol-102-disk-0 404M 7.61G 404M /rpool/data/subvol-102-disk-0
rpool/data/vm-100-disk-0 104K 236G 104K -
rpool/data/vm-100-disk-1 18.8G 236G 18.8G -
rpool/data/vm-100-disk-2 64K 236G 64K -
rpool/data/vm-101-disk-0 3.82G 236G 3.82G -
rpool/data/vm-103-disk-0 3.99G 236G 3.99G -
rpool/data/vm-104-disk-0 25.4G 236G 25.4G -
rpool/data/vm-105-disk-0 4.38G 236G 4.38G -
rpool/data/vm-106-disk-0 9.27G 236G 9.27G -
rpool/var-lib-vz 29.3G 236G 29.3G /var/lib/vz
And a VM config like the following:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
root@pve001:~# qm config 101
balloon: 0
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
description:
ide2: none,media=cdrom
machine: q35
memory: 4096
meta: creation-qemu=9.0.2,ctime=1725865589
name: ansible001
net0: virtio=BC:24:11:23:36:AE,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-zfs:vm-101-disk-0,iothread=1,size=80G
scsihw: virtio-scsi-single
smbios1: uuid=5adf1be4-29ba-4fdb-86fb-e470e7042498
sockets: 1
vga: qxl
vmgenid: 84875a30-c554-423d-96d1-7602dcf93b92
If you want to view partition layout via fdisk:
1
2
3
4
5
6
7
8
9
10
11
root@pve001:~# fdisk -l /dev/zvol/rpool/data/vm-101-disk-0
Disk /dev/zvol/rpool/data/vm-101-disk-0: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 16384 bytes
Disklabel type: gpt
Disk identifier: E4685CB0-C04D-43E5-B7C5-45919E117E79
Device Start End Sectors Size Type
/dev/zvol/rpool/data/vm-101-disk-0p1 2048 4095 2048 1M BIOS boot
/dev/zvol/rpool/data/vm-101-disk-0p2 4096 167770111 167766016 80G Linux filesystem
View devices names:
1
2
3
4
root@pve001:~# ls -lh /dev/zvol/rpool/data/vm-101-disk-0*
lrwxrwxrwx 1 root root 12 Oct 10 15:54 /dev/zvol/rpool/data/vm-101-disk-0 -> ../../../zd0
lrwxrwxrwx 1 root root 14 Oct 10 15:54 /dev/zvol/rpool/data/vm-101-disk-0-part1 -> ../../../zd0p1
lrwxrwxrwx 1 root root 14 Oct 10 15:54 /dev/zvol/rpool/data/vm-101-disk-0-part2 -> ../../../zd0p2
Mount a VM disk (WARNING: DO NOT DO THIS AGAINST A RUNNING VM)
1
2
3
4
5
6
root@pve001:~# mkdir /mnt/vm-disk
root@pve001:~# mount /dev/zvol/rpool/data/vm-101-disk-0-part2 /mnt/vm-disk/
root@pve001:~# ls /mnt/vm-disk/
bin bin.usr-is-merged boot cdrom dev etc home lib lib64 lib.usr-is-merged lost+found media mnt opt proc root run sbin sbin.usr-is-merged snap srv swap.img sys tmp usr var
root@pve001:~# umount /mnt/vm-disk
You can also resize partitions on those disks (the current resize disk only increases the disk itself, you can just add a new partition as well).