DRBD 8.2.4 released; boasts online device verification, CPU affinity optimization

DRBD 8.2.4 is out. Ahem. Turned out that in some configurations, online device verification in 8.2.3 (which we’re so proud of!) wasn’t working. It is now.

And yes, we’re just a wee bit embarrassed by this. But at least to our own credit, we found out ourselves. And very quickly.🙂

Anyway, here is the summary of our brand new features.

Tunable CPU affinity

This is the minor one of the new features. It allows you to pin DRBD’s kernel threads to a particular CPU or CPU core.

Being lightweight and lean as it is, DRBD does not gain much performance from parallelization. However, it does gain a lot (in terms of latency benefits) from minimizing CPU cache misses. This comes in particularly handy in database applications, even more so in those with small transaction sizes. Indeed the potential for this performance enhancement was discovered while running mysqlslap benchmarks against a MySQL database on a multi-core system.

For those who have previously worked with taskset, setting a CPU affinity mask for a process or thread is probably a familiar concept. In DRBD, you now use the cpu-mask option to set the processor affinity (more precisely, its decimal representation):

resource foo {
  syncer {
    cpu-mask 1;
    # ...
  }
}

Please be advised that although this option is syntactically required to go into the syncer section, it does not only apply to background device resynchronization. The CPU affinity mask is set for all DRBD threads associated with the resource (worker, sender, and receiver).

On-line device verification

This is the major new feature in 8.2.4. It allows you to do a block-by-block verification of device contents versus the peer block device, while keeping DRBD online and usable the whole time.

This is helpful when you suspect that something (or someone) has modified DRBD’s backing device in a manner that bypassed DRBD. For example, you might have suffered a disk issue while DRBD was unloaded, or you might have accidentally overwritten data on the backing device. Previously, all you could do in this case was issue drbdadm invalidate resource to force a full resync. This could be a very lengthy operation, particularly if your DRBD held hundreds of gigabytes of data.

Now, you can just do a full run of checking every block on the corresponding DRBD device. Of course, this is also an operation that might take several hours — but now your device remains online and redundant the whole time.

To enable online verification for a device, you need to add verify-alg hash-alg to the syncer section of your resource in drbd.conf, like so:

resource foo {
  syncer {
    verify-alg crc32c;
    # ...
  }
}

hash-alg can be any of the message digest algorithms provided by the kernel crypto API; recent kernel configurations typically support at least md5, sha1, and/or crc32c.To actually invoke online verification, do:

drbdadm verify resource

You can then monitor the verification process via /proc/drbd, as usual. Out-of-sync blocks are not synchronized automagically. To synchronize any out-of-sync blocks the verification process has detected, issue the following commands on your current Primary:

drbdadm disconnect resource
drbdadm connect resource

Your Primary will then become SyncSource and resynchronize those blocks that were marked out of sync by the verification process.

About the version number

DRBD 8.2.4 is the direct successor release post DRBD 8.2.1. drbd-8.2.2 is a tag that exists in our public git repository, but one critical bug was discovered by QA after that version was tagged, precluding a DRBD 8.2.2 release. 8.2.3 did get released, but it had the online verify screwup described above. So, 8.2.4 is the new current release number.

Future DRBD releases will have release candidates tagged separately from actual releases. For obvious reasons.🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: