For the last couple of weeks, I’ve been playing around with a FreeBSD distribution that’s quite interesting, FreeNAS. FreeNAS basically turns any old box you have lying around into a NAS, though you should definitely not run it on any old box. The only problem was, I got a bit excited when setting up my system, so I configured a RAID-Z vdev for my zpool. Technically, this was not an issue at the time when I set up the pool, since I was using old leftover 160GB and 250GB drives. But after a while I replaced the 160GB drives with two 1 TB drives I had laying around. And a couple of days later, I ordered 2 4TB drives to replace the 250GB ones. And then RAID-Z started to be a problem. You see, RAID-Z suffers from the same problems as RAID5, aka. that if you set up an array with large SATA drives (1 TB or larger), and one drive fails, you’re almost guaranteed a bad day (if you’re not running enterprise grade drives, more on this later in a separate post). So before I went and installed the new drives I migrated my whole pool to a backup drive and destroyed the old vdev and pool. Here’s a short guide on how I, relatively successfully did it.

The process is quite straight forward, copy the data off the pool using ZFS so we preserve all the file permissions and everything, destroy and recreate the vDev and pool, and move everything back as it were. I used two USB drives, one 2 TB drive for the data, and one 16 GB thumb drive for the system dataset.

Before you begin!

I do want to note that this isn’t the “correct” way of doing things. The single USB drive that I use for alternative storage while I rebuild the pool isn’t safe for storing data, since it’s isn’t protected properly against drive failure. And the use of a USB drive with ZFS makes this even more risky, as ZFS doesn’t write to disk synchronously, aka. even though you have finished copying something to the drives, it might not actually be written to disc yet. If your drive becomes disconnected with writes pending, this might lead to data loss or pool corruption.

The correct way would be to set up a second FreeNAS and copy everything to it while you work on your pool. The new FreeNAS would need to have equal or greater protection than the source pool in place.

That being said, this way works just fine as long as you have a second copy sitting around somewhere. If you’re replacing all the discs in the pool, you might also be able to just detach the current pool without destroying it and keep the old drives as a backup.

1. Saving the System Configuration

Just in case anything goes completely wrong, it’s good to take a backup of the system configuration. That way you could reinstall FreeNAS and get it up and running without too much difficulty.

Select System from the ribbon, then the General tab, and click Save Config at the bottom. Your web browser downloads the configuration to a file. That’s it!

2. Moving the System Dataset

Update: I’ve been informed that you can actually use the system thumb drive to temporarily store your system dataset as well if it’s large enough. The procedure is the same, just select freenas-boot instead!

Since FreeNAS is normally installed on a thumb drive, which a limited amount of write cycles, it by default stores it’s system files on the zpool where all your data is stored. Since we’re going to destroy that pool in a later step, we need to move it somewhere. Moving it to the backup drive is not an option since we’re going to be overwriting the whole ZFS pool on that drive. The easiest way I found is to create a new zpool on an empty USB drive. Any drive will do, the system dataset is usually quite small (~100MB).

First, plug a USB thumb drive of decent quailty (I used a Kingston Datatraveller) into a free USB port. Then select Storage -> Volume Manager and create a new volume/pool. The name doesn’t matter, I named mine sysbackup

After the new pool is created, select System -> System Dataset and change System dataset pool: to your new pool (in my case sysbackup). Leave syslog checked and Reporting Database unchecked and click ok! After a while, your System Dataset is on the new pool.

3. Creating the Backup Pool

Now, plug in your large USB drive and repeat the steps for creating a new pool as you did with sysbackup. I named the new pool backup.

4. Backup the Data Pool

Next, we need to create a snapshot of the root dataset on the pool you’re going to backup. Go to Storage and select your root dataset (it’s one level down from the volume in the tree, and has the same name as the volume). Click Create Snapshot at the bottom, check Recursive snapshot and give it a simple name, I chose migrate.

Alternativly, you could do it through the CLI as well, using the following command.

zfs snapshot -r tank@migrate

Then open an SSH session to your FreeNAS box as root.

Type zfs send -R tank@migrate | zfs receive -F backup

“tank” is the name of the volume I’m backing up. -F allows you to overwrite the existing pool that FreeNAS created when you created the backup pool.

For 550 GB, this command took me 6 hours to run. It doesn’t give you any info on the progress of the command, but you can check how much data has been written to the USB drive through the normal reports in the FreeNAS Web GUI. Keep in mind that I did this over USB 2.0, so I was limited by the speed of the USB connection.

Also note that you need to keep the SSH session open until the command completes, my computer rebooted while I ran it the first time, so I took no chances and just reran it.

5. Destroy and Remake the Data Pool

Now that you have all your data safe and sound on the backup pool (you can and should verify this before continuing), we need to detach the old pool. Select Storage, select your old pool (in my case tank) and click detach at the bottom. Select Destroy, don’t delete the shares and click ok. Note that this will completely delete all you data on that pool with no chance of recovery! I recommend that you, in addition of the backup pool you just created, make sure that you have another copy of your data laying around somewhere, so that in case anything goes wrong you could recover the data the “long way round”.

Also note that destroying the pool is only necessary if you plan on reusing all or some of the old drives. In my case I wanted to add 2 new 4 TB drives, but keep 2 other drives from the old pool.

Now, make the changes necessary, in my case pulling the drives, installing the 4 TB drives, and recreating the pool tank as a RAID-Z2 vdev. Make sure you name your volume the same when you recreate it.

6. Move the Data Back

We now need to get the data back from the USB drive, and preferably as quickly as possible, since running ZFS on a USB drive is far from recommended.

What we’re going to do is to run the copy command in reverse.

zfs send -R backup@migrate | zfs receive -F tank

Again, the command took me 5 hours and remember to leave the SSH session alive.

7. Export and Import the New Pool

In the GUI, now select the new tank, and select Detach. Don’t select destroy this time! What we want is for FreeNAS to release tank so we can import it again and let FreeNAS “re-inventory” the pool.

Now in the GUI under Storage, select Import Volume and import tank again!

8. Restore the system dataset

After your data is back in the protected pool, we need to move the system dataset back to the main pool. The steps are exactly the same as in step 2, just select tank instead.

Update: If you used your system drive instead of a separate drive, you can skip the detachment steps below.

After the system dataset is moved back, detach the sysbackup pool, no need to destroy it, and remove the USB flash drive.

Now everything should be fine and dandy. If everything isn’t the you’ll need to restore your system configuration. The process is the same as in step 1, you just need to select Upload Config instead. The FreeNAS reboots two times, and everything should be back to normal.

Caveats

One thing that didn’t go as plan when I did this was my Plugins/Jails. For some reason, they all stopped working correctly, and I ended up reinstalling them all. Granted, this isn’t much work since you jails shouldn’t contain any actual data, but it’s something to keep in mind.

Update: The problem I had with the plugins was due to a snafu on my part. I added a new step 7, which should fix this.

Also note that this process is only really necessary if you can’t have both the new pool and old pool hooked up to the FreeNAS at the same time, for example when you want to change the RAID-Z setup on the current drives. If you can hook up both pools simultaneously, then you can get away with just copying the data from the old pool to the new pool once and never have to delete the old pool. Here’s some instructions on how to do exactly that.  These also happen to be the instructions that I based my own manoeuvre on. Big thanks to depasseg over at the FreeNAS Forums for letting me bounce ideas about this project around with him!

Categories: Tech

Tags: freebsd, freenas, it, nas, Tech, zfs