CentOS 7 and ZFS

zfs-linux
 
 
 
 
 

I have used ZFS before on FreeBSD but never on any Linux variant.
So it became time to play around with ZFS on Linux, but as you can read below, there still are some mayor issues.

As I just downloaded the latest CentOS 7 Minimal CD I used CentOS as my first try.
After the minimal install and updating all packages on my little VM we were ready to go.

ZFS is not part of CentOS so you will need to enable extra repository’s to install ZFS.

First we enable EPEL and add the zfs repo:

1
2

sudo yum install epel-release
sudo yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm

Then install ZFS:

1

sudo install kernel-devel zfs

This is where the first issue came up, ZFS did not seem to have installed correctly giving me not much more than an:

1

Failed to load ZFS module stack.

It seemed that ZFS was not build correctly because of the kernel update that was installed when I updated the system for the first time. After numerous attempts to fix ZFS I decided to just remove all the ZFS packages and old kernel and start over.

An tip online on gmane also mentioned to remove all modules, so sure lets go:

1
2
3

find /lib/modules/$(uname -r)/extra -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f
 
find /lib/modules/$(uname -r)/weak-updates/ -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f

Then I removed my old kernel, headers and all ZFS packages and finally an reboot:

1
2
3
4

sudo yum erase kernel-3.10.0-229.el7.x86_64
sudo yum erase kernel-headers kernel-devel kernel-tools
sudo yum erase zfs zfs-dkms libzfs2 spl spl-dkms dkmsc
reboot

After the reboot it was time to install everything again:

1
2
3

sudo yum localinstall --nogpgcheck https://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
sudo yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm
sudo yum install kernel-devel zfs

And finally ZFS worked like it should, no more error messages.

I have 4 small 5GB drives added to my VM to test with,
sdb, sdc, sdd and sde

So lets create an pool called data01 in raidz, as the drives are blank and unformatted you will need to force the creation of an new GPT.

1

zpool create data01 raidz sdb sdc sdd sde -f

See if all went well:

1
2
3
4
5
6
7
8
9
10
11
12
13

zpool status
  pool: data01
 state: ONLINE
  scan: none requested
config:
 
	NAME        STATE     READ WRITE CKSUM
	data01      ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    sdb     ONLINE       0     0     0
	    sdc     ONLINE       0     0     0
	    sdd     ONLINE       0     0     0
	  sde       ONLINE       0     0     0

That looks fine, now lets see what zdb says:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

zdb
data01:
    version: 5000
    name: 'data01'
    state: 0
    txg: 141
    pool_guid: 11371218379924896498
    errata: 0
    hostname: 'gluster01'
    vdev_children: 2
    vdev_tree:
        type: 'root'
        id: 0
        guid: 11371218379924896498
        create_txg: 4
        children[0]:
            type: 'raidz'
            id: 0
            guid: 832996839702454674
            nparity: 1
            metaslab_array: 34
            metaslab_shift: 27
            ashift: 9
            asize: 16060514304
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 2105630605786225987
                path: '/dev/sdb1'
                whole_disk: 1
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 6145009721102371365
                path: '/dev/sdc1'
                whole_disk: 1
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 14635352678309772165
                path: '/dev/sdd1'
                whole_disk: 1
                create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 1544288362510023523
            path: '/dev/sde1'
            whole_disk: 1
            metaslab_array: 112
            metaslab_shift: 25
            ashift: 9
            asize: 5353504768
            is_log: 0
            create_txg: 134
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

That all looks fine so we should have an 15GB /data01 partition:

1
2
3
4
5
6
7
8
9

df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  8.5G  1.1G  7.5G  13% /
devtmpfs                 481M     0  481M   0% /dev
tmpfs                    490M     0  490M   0% /dev/shm
tmpfs                    490M  6.7M  484M   2% /run
tmpfs                    490M     0  490M   0% /sys/fs/cgroup
/dev/sda1                497M  119M  378M  24% /boot
data01                    15G     0   15G   0% /data01

Perfect, that all works fine, lets see if duplication works:

I will create a 1GB file just filled with zero’s to see if this works.

1
2
3
4
5
6
7
8
9
10
11

dd if=/dev/zero of=/data01/zerofile1 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 4.45455 s, 241 MB/s
 
ls -lh /data01/
total 1.0G
-rw-r--r--. 1 root root 1.0G Jul 28 22:42 zerofile1
 
df -h | grep data01
data01                    15G  1.0G   14G   7% /data01

Okay we have a single file on /data01 that uses 1GB of space, lets copy the file a few times and see what happens, if everything works like it should the usage of the drive should not increase:

1
2
3
4
5
6
7
8
9
10
11
12

i=2; while [ $i -lt 7 ]; do cp /data01/zerofile1 /data01/zerofile$i; i=$[i+1]; done
ls -lh /data01/
total 1.0G
-rw-r--r--. 1 root root 1.0G Jul 28 22:42 zerofile1
-rw-r--r--. 1 root root 1.0G Jul 28 22:53 zerofile2
-rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile3
-rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile4
-rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile5
-rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile6
 
df -h | grep data01
data01                    15G  1.0G   14G   7% /data01

Yep we have 6 of the same files that are each 1GB in size but still 1GB of total space used, looks good thus far, lets clean up and the drive should be empty again:

1
2
3
4
5

rm -rf zerofile*
ls -lh /data01/
total 0
df -h | grep data01
data01                    15G  1.0G   14G   7% /data01

Thats weird, I removed the files but still the drive has 1GB of space in use.
But if you unmount the drive and mount the drive again the disk space is free again:

1
2
3
4
5

cd /
umount /data01/
zfs mount data01
df -h | grep data
data01                    15G     0   15G   0% /data01

This seems to be an bug, creating an sub drive with xattr=sa and modify the drop_caches seems to ‘fix’ the issue

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

zfs create -o xattr=sa data01/fs
echo 3 > /proc/sys/vm/drop_caches
 
dd if=/dev/zero of=/data01/fs/testfile bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.98644 s, 541 MB/s
 
# df -h | grep data
data01/fs                 15G  1.0G   14G   7% /data01/fs
 
# rm /data01/fs/testfile
rm: remove regular file ‘/data01/fs/testfile’? y
 
#  df -h | grep data
data01/fs                 15G  128K   15G   1% /data01/fs

So it seems ZFS on CentOS 7 still has some issues, next up in test is Debian.


Copyright 2021. All rights reserved.

Posted July 28, 2015 by TroubleNow in category "linux