Step-by-step upgrade from DRBD 0.7 to DRBD 8

I’ve been asked by a number of people on how to do an upgrade from DRBD version 0.7 to DRBD 8. This upgrade does necessitate some minimal service down time, but it’s really not rocket science. And no, it does not force you to sync all of your data all over again.

Here’s my quick write-up.

An initial word of caution

Do not, I repeat do not attempt your upgrade unless you have at least read this blog entry to the finish.

Getting ready

First, you need to make sure that you have both your DRBD 8 userland binaries and kernel module ready to install. For our support customers, this means that you simply download two RPMs (or .debs) from our support web site. Make sure you have the right packages; you want those that match your system architecture and (for the kernel module) also your running kernel. Also, make sure you get DRBD 8.2.3 or later; there were some bugs in pre-8.2.3 versions that ran a slight chance of messing up 0.7 metadata conversions.

To be on the perfectly safe side, it may be wise to also have the RPMs of your latest 0.7 version at the ready. Just in case you wish to abort the upgrade in the middle and revert.

Second, and here is your down time: you need to shut down your Heartbeat services on both nodes, unmount any file systems that reside on DRBDs, and make all DRBD devices Secondary. This is necessary for the meta data conversion that follows. You want to be sure your DRBD devices are in sync, block by block.

The sequence of commands you issue is this:

• /etc/init.d/heartbeat stop
• mount (this should show no DRBDs being mounted anymore)
• drbdadm secondary all (just in case)
• cat /proc/drbd (this should show all your DRBDs as being Secondary)
• drbdadm down all
• rmmod drbd

Out with the old…

At this point, DRBD is completely disabled, you should not see a /proc/drbd file anymore.

You can now uninstall your old (DRBD 0.7) packages. The DRBD 8 userland binaries will conflict with the 0.7 packages, so you won’t be able to complete the next step without previously uninstalling those.

… in with the new.

Now issue rpm -ivh, dpkg -i, or whatever else is appropriate for your distribution, giving the names of the two DRBD packages you’re installing.

Your distribution may also provide DRBD 8 packages in an installation archive; in order to install from there, you will need to issue apt-get install package, yum install package, rug install package or similar. Just make sure that your distro archive indeed carries 8.2.3 or above.

After you have installed your new packages, for every configured resource, issue this command:

• drbdadm create-md resourcename

At this point, drbdadm will detect that DRBD 0.7 metadata is already in place for the resource, and will ask you whether you want to upgrade that metadata to DRBD 8 format.

If, for any reason, drbdadm fails to detect the existing DRBD 0.7 metadata on that device, and asks whether you want to create a new set of metadata, do not continue. Say “no”. If you’re a support customer, open a support incident. If you are a community user, post a problem report to the mailing list. You have not changed anything about your DRBD configuration at this point. You may safely uninstall your DRBD 8 packages, reinstall the DRBD 0.7 ones, and restart DRBD and Heartbeat to maintain full service redundancy until your issue is resolved.

If DRBD’s 0.7 metadata has been detected correctly, say “yes” to convert to version 8 format. This conversion step must be completed on both nodes. From that point forward, your DRBD metadata is no longer compatible with DRBD 0.7. But that’s probably of no importance, since you’re upgrading your second node simultaneously, anyway.

Then, you can continue:

• modprobe drbd
• cat /proc/drbd

At this point, /proc/drbd will not yet show any configured devices, but you should already see the DRBD 8 version information. Now, after you have completed all of the above steps on both nodes, do:

• drbdadm up all

And again:

• cat /proc/drbd

Finishing up

Now, all of your devices should be listed as cs:Connected, st:Secondary/Secondary, and ds:UpToDate/UpToDate.

When everything is hunky dory, go ahead and fire up Heartbeat again. Your trusted cluster manager should now make DRBDs Primary as necessary, mount filesystems, and start your cluster resources.

If you know your stuff and are well prepared, the entire upgrade procedure should take no more than 10 minutes or so.

One more thing: if you have been using the group keyword in the syncer section(s) of your drbd.conf, be advised that it is no longer supported in version 8. It has been supplanted by the after keyword which enables you to sync resources after one another. More often than not, however, you’ll want to sync all your resources in parallel, so in most cases you won’t be needing the after option.