Changing the capacity of each VDEV in a ZPOOL without losing data and no downtime with ZFS

-

Before we start, let's define some acronyms.

FreeBSD: Free Berkeley Software Distribution
ZFS: Zettabyte File System
ZPOOL: ZFS POOL
DEV: DEVice
VDEV: Virtual DEVice
GEOM: disk GEOMetry
BHYVE: BSD HYperVisor
ZVOL: ZFS VOLume

***

I have been tinkering with FreeBSD, ZFS, GEOM, BHYVE, and other interesting technologies.

***

Let's experiment on how to change the capacity of each VDEV of a ZPOOL without losing data and no downtime with ZFS.

To do so, you need a set of disks that are bigger.
Example, if the ZPOOL has 2 mirror VDEVs with 2 10GB disks each (total: 4 10GB disks), and you want to upgrade to 20 GB disks, than you need 4 20GB disks for the upgrade.


Figure 1: Live upgrade of disks in a ZPOOL

ZPOOL tank                           ZPOOL tank
    VDEV mirror-0                        VDEV mirror-0
        DEV /dev/vtbd1                    DEV /dev/vtbd9
        DEV /dev/vtbd2    --->         DEV /dev/vtbd10
    VDEV mirror-1                       VDEV mirror-1
        DEV /dev/vtbd3                    DEV /dev/vtbd11
        DEV /dev/vtbd4                    DEV /dev/vtbd12
    VDEV mirror-2                       VDEV mirror-2
        DEV /dev/vtbd5                    DEV /dev/vtbd13
        DEV /dev/vtbd6                    DEV /dev/vtbd14
    VDEV mirror-3                       VDEV mirror-3
        DEV /dev/vtbd7                    DEV /dev/vtbd15
        DEV /dev/vtbd8                    DEV /dev/vtbd16


***

Create a ZPOOL

root@nova:~ # zpool create tank \
    mirror vtbd1 vtbd2 \
    mirror vtbd3 vtbd4 \
    mirror vtbd5 vtbd6 \
    mirror vtbd7 vtbd8

 root@nova:~ # zpool status
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            vtbd1   ONLINE       0     0     0
            vtbd2   ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            vtbd3   ONLINE       0     0     0
            vtbd4   ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            vtbd5   ONLINE       0     0     0
            vtbd6   ONLINE       0     0     0
          mirror-3  ONLINE       0     0     0
            vtbd7   ONLINE       0     0     0
            vtbd8   ONLINE       0     0     0

errors: No known data errors
***

Add data to it.

root@nova:~ # zfs create -o mountpoint=/tank/dataset-x tank/dataset-x

 root@nova:~ # rsync -av /boot /tank/dataset-x/


***

Now, we see that the ZPOOL tank is using 155M, with data on each VDEV.


root@nova:~ # df -h /tank/dataset-x/
Filesystem        Size    Used   Avail Capacity  Mounted on
tank/dataset-x     37G    153M     37G     0%    /tank/dataset-x

root@nova:~ # zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
tank             154M  36.7G   176K  /tank
tank/dataset-x   153M  36.7G   153M  /tank/dataset-x
 
root@nova:~ # zpool list -v
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank          38G   155M  37.8G        -         -     0%     0%  1.00x  ONLINE  -
  mirror    9.50G  41.0M  9.46G        -         -     0%     0%
    vtbd1       -      -      -        -         -      -      -
    vtbd2       -      -      -        -         -      -      -
  mirror    9.50G  41.9M  9.46G        -         -     0%     0%
    vtbd3       -      -      -        -         -      -      -
    vtbd4       -      -      -        -         -      -      -
  mirror    9.50G  36.4M  9.46G        -         -     0%     0%
    vtbd5       -      -      -        -         -      -      -
    vtbd6       -      -      -        -         -      -      -
  mirror    9.50G  35.4M  9.47G        -         -     0%     0%
    vtbd7       -      -      -        -         -      -      -
    vtbd8       -      -      -        -         -      -      -
***

Let's upgrade the 10-GB disks to 20-GB disks to double the capacity.
We will do it in-place, without putting the service offline, and without creating a new temporary pool.

***

Replace the disks of mirror-0.

root@nova:~ # zpool status
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            vtbd1   ONLINE       0     0     0
            vtbd2   ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            vtbd3   ONLINE       0     0     0
            vtbd4   ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            vtbd5   ONLINE       0     0     0
            vtbd6   ONLINE       0     0     0
          mirror-3  ONLINE       0     0     0
            vtbd7   ONLINE       0     0     0
            vtbd8   ONLINE       0     0     0

root@nova:~ # zpool remove tank mirror-0
 
root@nova:~ # zpool status
  pool: tank
 state: ONLINE
  scan: none requested
remove: Removal of vdev 0 copied 41.0M in 0h0m, completed on Thu Jun 11 20:46:12 2020
    2.06K memory used for removed device mappings
config:

        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          mirror-1    ONLINE       0     0     0
            vtbd3     ONLINE       0     0     0
            vtbd4     ONLINE       0     0     0
          mirror-2    ONLINE       0     0     0
            vtbd5     ONLINE       0     0     0
            vtbd6     ONLINE       0     0     0
          mirror-3    ONLINE       0     0     0
            vtbd7     ONLINE       0     0     0
            vtbd8     ONLINE       0     0     0

errors: No known data errors

 ***

We can see that ZFS is smart and that it copied the data that was on mirror-0 to the other ONLINE VDEVs of the ZPOOL tank.


 root@nova:~ # zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank          28.5G   156M  28.3G        -         -     0%     0%  1.00x  ONLINE  -
  mirror      9.50G  57.2M  9.44G        -         -     0%     0%
    vtbd3         -      -      -        -         -      -      -
    vtbd4         -      -      -        -         -      -      -
  mirror      9.50G  50.2M  9.45G        -         -     0%     0%
    vtbd5         -      -      -        -         -      -      -
    vtbd6         -      -      -        -         -      -      -
  mirror      9.50G  48.7M  9.45G        -         -     0%     0%
    vtbd7         -      -      -        -         -      -      -
    vtbd8         -      -      -        -         -      -      -

***

Now, add the new 20-GB disks for mirror-0.



root@nova:~ # zpool add tank mirror vtbd9 vtbd10

 root@nova:~ # zpool status
  pool: tank
 state: ONLINE
  scan: none requested
remove: Removal of vdev 0 copied 41.0M in 0h0m, completed on Thu Jun 11 20:46:12 2020
    2.06K memory used for removed device mappings
config:

        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          mirror-1    ONLINE       0     0     0
            vtbd3     ONLINE       0     0     0
            vtbd4     ONLINE       0     0     0
          mirror-2    ONLINE       0     0     0
            vtbd5     ONLINE       0     0     0
            vtbd6     ONLINE       0     0     0
          mirror-3    ONLINE       0     0     0
            vtbd7     ONLINE       0     0     0
            vtbd8     ONLINE       0     0     0
          mirror-4    ONLINE       0     0     0
            vtbd9     ONLINE       0     0     0
            vtbd10    ONLINE       0     0     0

errors: No known data errors

root@nova:~ # zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank            48G   157M  47.8G        -         -     0%     0%  1.00x  ONLINE  -
  mirror      9.50G  57.2M  9.44G        -         -     0%     0%
    vtbd3         -      -      -        -         -      -      -
    vtbd4         -      -      -        -         -      -      -
  mirror      9.50G  50.3M  9.45G        -         -     0%     0%
    vtbd5         -      -      -        -         -      -      -
    vtbd6         -      -      -        -         -      -      -
  mirror      9.50G  48.5M  9.45G        -         -     0%     0%
    vtbd7         -      -      -        -         -      -      -
    vtbd8         -      -      -        -         -      -      -
  mirror      19.5G   552K  19.5G        -         -     0%     0%
    vtbd9         -      -      -        -         -      -      -
    vtbd10        -      -      -        -         -      -      -


***

Do the same thing for the other 10-GB disks.

root@nova:~ # zpool remove tank mirror-1
root@nova:~ # zpool add tank mirror vtbd11 vtbd12
root@nova:~ # zpool remove tank mirror-2
root@nova:~ # zpool add tank mirror vtbd13 vtbd14
root@nova:~ # zpool remove tank mirror-3
root@nova:~ # zpool add tank mirror vtbd15 vtbd16

***

The only problem is that the data is not balanced across the VDEVs of the ZPOOL tank. But hey, at least, we did it in place.

root@nova:~ # zpool status
  pool: tank
 state: ONLINE
  scan: none requested
remove: Removal of vdev 3 copied 88.8M in 0h0m, completed on Thu Jun 11 20:55:49 2020
    11.7K memory used for removed device mappings
config:

        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          mirror-4    ONLINE       0     0     0
            vtbd9     ONLINE       0     0     0
            vtbd10    ONLINE       0     0     0
          mirror-5    ONLINE       0     0     0
            vtbd11    ONLINE       0     0     0
            vtbd12    ONLINE       0     0     0
          mirror-6    ONLINE       0     0     0
            vtbd13    ONLINE       0     0     0
            vtbd14    ONLINE       0     0     0
          mirror-7    ONLINE       0     0     0
            vtbd15    ONLINE       0     0     0
            vtbd16    ONLINE       0     0     0

errors: No known data errors
 
root@nova:~ # zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank            78G   157M  77.8G        -         -     0%     0%  1.00x  ONLINE  -
  mirror      19.5G  69.9M  19.4G        -         -     0%     0%
    vtbd9         -      -      -        -         -      -      -
    vtbd10        -      -      -        -         -      -      -
  mirror      19.5G  56.2M  19.4G        -         -     0%     0%
    vtbd11        -      -      -        -         -      -      -
    vtbd12        -      -      -        -         -      -      -
  mirror      19.5G    31M  19.5G        -         -     0%     0%
    vtbd13        -      -      -        -         -      -      -
    vtbd14        -      -      -        -         -      -      -
  mirror      19.5G   128K  19.5G        -         -     0%     0%
    vtbd15        -      -      -        -         -      -      -
    vtbd16        -      -      -        -         -      -      -

***

Now, verify the integrity of the data. The only file that changed is  /boot/zpool.cache, which means that we should not have used the files in /boot in the first place...

root@nova:~ # diff -r -u /boot /tank/dataset-x/boot
Binary files /boot/zfs/zpool.cache and /tank/dataset-x/boot/zfs/zpool.cache differ

***

Let's rebalance the ZPOOL, the ZFS way. We simply need to create a snapshot, and send it in replicate mode, and then receive it.

 root@nova:~ # zfs snapshot -r tank@2020-06-11-2111-0001
 
root@nova:~ # zfs list -t snapshot
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
tank@2020-06-11-2111-0001                0      -   176K  -
tank/dataset-x@2020-06-11-2111-0001      0      -   153M  -

 root@nova:~ # zfs send -R tank@2020-06-11-2111-0001 | zfs recv tank/rebalanced

root@nova:~ # zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank            78G   314M  77.7G        -         -     0%     0%  1.00x  ONLINE  -
  mirror      19.5G   108M  19.4G        -         -     0%     0%
    vtbd9         -      -      -        -         -      -      -
    vtbd10        -      -      -        -         -      -      -
  mirror      19.5G  96.1M  19.4G        -         -     0%     0%
    vtbd11        -      -      -        -         -      -      -
    vtbd12        -      -      -        -         -      -      -
  mirror      19.5G  69.3M  19.4G        -         -     0%     0%
    vtbd13        -      -      -        -         -      -      -
    vtbd14        -      -      -        -         -      -      -
  mirror      19.5G  41.0M  19.5G        -         -     0%     0%
    vtbd15        -      -      -        -         -      -      -
    vtbd16        -      -      -        -         -      -      -
 
root@nova:~ # zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
tank                        310M  75.3G   176K  /tank
tank/dataset-x              153M  75.3G   153M  /tank/dataset-x
tank/rebalanced             153M  75.3G   176K  /tank/rebalanced
tank/rebalanced/dataset-x   153M  75.3G   153M  /tank/dataset-x

The only thing left to do is to destroy the old dataset, and to rename the rebalanced one.

root@nova:~ # zfs destroy -r tank/dataset-x

 root@nova:~ # zfs rename tank/rebalanced/dataset-x tank/dataset-x
 
root@nova:~ # zfs destroy -r tank/rebalanced

root@nova:~ # zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
tank             156M  75.4G   176K  /tank
tank/dataset-x   153M  75.4G   153M  /tank/dataset-x

***

Now, let's check if the pool is balanced. It looks good.

root@nova:~ # zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank            78G   157M  77.8G        -         -     0%     0%  1.00x  ONLINE  -
  mirror      19.5G  38.6M  19.5G        -         -     0%     0%
    vtbd9         -      -      -        -         -      -      -
    vtbd10        -      -      -        -         -      -      -
  mirror      19.5G  40.3M  19.5G        -         -     0%     0%
    vtbd11        -      -      -        -         -      -      -
    vtbd12        -      -      -        -         -      -      -
  mirror      19.5G  38.7M  19.5G        -         -     0%     0%
    vtbd13        -      -      -        -         -      -      -
    vtbd14        -      -      -        -         -      -      -
  mirror      19.5G  39.6M  19.5G        -         -     0%     0%
    vtbd15        -      -      -        -         -      -      -
    vtbd16        -      -      -        -         -      -      -

***


Finally, verify the integrity of the files again.

Strangely, we maybe have hit a ZFS bug ! Oh no.

root@nova:~ # ls /tank/dataset-x/ | wc -l
0

But have no fear, ZFS garantees integrity.

 root@nova:~ # zfs mount tank/dataset-x

 root@nova:~ # diff -r -u /boot /tank/dataset-x/boot
Binary files /boot/zfs/zpool.cache and /tank/dataset-x/boot/zfs/zpool.cache differ

 Everything seems in order.



***

We now have more capacity.

root@nova:~ # df -h /tank/dataset-x/
Filesystem        Size    Used   Avail Capacity  Mounted on
tank/dataset-x     76G    153M     75G     0%    /tank/dataset-x


***

While preparing this post, I found a bug.

See #247188
Bug 247188 - "camcontrol devlist" does not show zvol-backed virtio devices
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247188

Comments

Post a Comment

Popular posts from this blog

Learning to solve the example 1 of puzzle 3aa6fb7a in the ARC prize

-
Image
Hi dear reader, Puzzle rules The example 1 of the puzzle 3aa6fb7a of the ARC prize looks like this: A playout starts with an blank board of width 7 and of height 7. There are therefore 49 cells. Each cell can be blank or can contain one of the 10 possible colors. The colors are identified from 0 to 9.  I added a rule. This rule is that each cell has to be changed exactly once.  The game ends when all cells have been changed. The player wins if all the cell colors match the cell colors in the expected output. Some numbers: Each cell has 11 possible values: 10 colors + blank state There are 49! ( 6.0828186e+62) distinct possible orders of cell changes in a playout.  There are 49!*10 ( 6.0828186e+63) 49 * 10 = 490 possible actions for the first action.  There are 11^49 ( 1.0671896e+51) possible board states The number of possible pairs (s, a) is even larger and upper-bounded by 11^49 * 49 * 10 = 5.2292289e+53 . Given a board state s and a proposed action a, an actio...
Read more

The Thorium actor engine is operational now, we can start to work on actor applications for metagenomics

-
I have been very busy during the last months. In particular, I completed my doctorate on April 10th, 2014 and we moved from Canada to the United States on April 15th, 2014. I started a new occupation on April 21st, 2014 at Argonne National Laboratory (a U.S. Department of Energy laboratory). But the biggest change, perhaps, was not one listed in the enumeration above. The biggest change was to stop working on Ray. Ray is built on top of RayPlatform, which in turn uses MPI for the parallelism and distribution. But this approach is not an easy way of devising applications because message passing alone is a very leaky, not self-contained, abstraction. Ray usually works fine, but it has some bugs . The problem with leaky abstractions is that they lack simplicity and are way too complex to scale out. For example, it is hard to add new code to an existing code base without breaking anything. This is the case because MPI only offers a fixed number of ranks. Sure, the MPI standard has s...
Read more

The source code of SOAPdenovo2 sits in the shadows

-
Update 2012-12-28: SOAPdenovo 2.04-r223 source code was posted online on 2012-12-28. Some minor concerns: 1. The tarball suffers from the bundled library problem (libbam.a and libbammac.a are shipped precompiled). This is incompatible with the GPLv3 license. 2. The Makefile is not portable with its hard-coded paths to compilers (/opt/blc/gcc-4.5.0/bin/gcc should just be gcc). The SOAPdenovo2 article was published today under the terms of the Creative Commons Attribution License. Here are the Availability and requirements for SOAPdenovo2 as reported in the article: Availability and requirements Project name: SOAPdenovo2 Project home page: http://soapdenovo2.sourceforge.net/ Operating system(s): e.g. Platform independent Programming language: C, C++ Other requirements: GCC version ≥ 4.5.0 License: GNU General Public License version 3.0 (GPLv3) Any restrictions to use by non-academics: none Contact: bgi-soap@googlegroups.com I fired up my browser and in a ...
Read more