A completely non-technical post for a change.

Last weekend, seven employees of LINBIT’s European division participated in the World Games of Mountain Biking in Saalbach-Hinterglemm, Austria. We took part as Marathon race participants and co-sponsors of Biking for Butterfly Children, a charity dedicated to the fight against epidermolysis bullosa (EB).

Currently, no cure for any of the over 30 subtypes of EB exists — dermatologists and care givers can, however, greatly improve patients’
quality of life. Still, EB can be an excruciatingly painful, disfiguring, and debilitating disease that affects one in 20,000 live births and as such, makes the condition an orphan disease. The health care industry has little incentive to research the condition (as there is little money to be made off of it), and those who dedicate their careers to EB patient care and research rely on charitable donations for funding. Biking for Butterfly Children acts as a reliable fund raiser rounding up much-needed donations in the course of amateur cycling events (such as the World Games).

During this year’s World Games, B4BC raised a total of about 7,000 euros in donations — a respectable sum for an all-amateur event, but a lot more money is needed to improve the quality of life of EB patients, and potentially discover a cure to the disease. If you consider joining the fight against EB, please contact your local DebRA chapter.

Every once in a while, people ask us something along the lines of “why do I need DRBD? Can’t I accomplish what it does by other means?” You mean build high availability clusters with block-level synchronous replication? Well, sure you can. But all of the available alternatives have serious drawbacks.

• Use a SAN. Well, SANs are great in terms of management, even though they’re sometimes prohibitively expensive. But your regular SAN box does not offer physically distributed redundancy at the data level. In order words, if your SAN box goes down, all your beautiful high availability infrastructure turns to shreds. And even if you naïvely believe (and you shouldn’t) that a storage box can never crash, just think about air conditioning going down in just that part of your data center where your storage shelf is at. Repair time of several hours means down time of several hours, even though your servers in a different cabinet may be up and running. You’re dealing with a single point of failure. Not high availability in my book.
• Use a SAN with native replication. This means using two separate storage boxes with synchronous block-level replication between them. This eliminates the above-mentioned SPOF and is available from just about any SAN vendor (using proprietary implementations under various product names). The downside is that it costs you serious bucks, and I am not referring to just the additional piece of hardware. Those firmware licenses can hit six figures. Plus, switchover times (changing the direction of replication) can be extremely long, up to 4 minutes in some cases. And there is little to no support for replication management from open source cluster management software.
• Use a SAN with host-based mirroring. This means that you have two separate SAN boxes, hosts import LUNs from both, and mirror those pairs using software RAID (such as md). Eliminates the SPOF and saves you dollars on firmware licensing. Downsides: you still need a SAN (and the associated infrastructure — fibre channel, for example, isn’t exactly cheap), and as such your clusters are still not shared-nothing. And the integration with open source cluster management is also lacking.
• Use host-based mirroring between a local device, and a network block device. In this case, you have one disk that is local, and another that is exported from a remote host using NBD or iSCSI. Those two disks are then mirrored with software RAID. Now this one is really terrible in terms of management. Role reversal always requires some custom glue, no support from cluster manages is available whatsoever, and split brain detection is poor or non existant. So if you really want to go down that alley then do — but please don’t call it a high availability cluster.

Compare this to DRBD: no need for a SAN, so you can use it to operate a fully shared-nothing cluster. No firmware licensing cost. No need for expensive infrastructure as everything can replicate over regular IP networks. Role switch in a matter of seconds. Tight integration with both Pacemaker and Red Hat Cluster Suite.  Excellent split brain detection to make sure you don’t wreck your data accidentally. So if you are considering alternatives then that’s perfectly fine, but our soaring usage numbers are there for a reason.

With all of the discussions over the past years about MySQL Replication vs. DRBD (where the “vs.” part is in fact grossly misled of course — they are two technologies that complement each other quite well), here’s one with a slightly different angle: does it make sense to roll your own cluster manager around MySQL Replication, or is it smarter to plug into an existing, proven cluster architecture?

You’ll expect my own view to be fairly well defined, and it is. But make up your own mind!

Unfortunately, we’ve had a small issue with the drbd-user mailing list with some posts from new members not properly coming through. If you’ve been affected by this, our sincere apologies. Full functionality of the mailing list has now been restored.

Here is a question recently seen on drbd-user:

I cannot get the timeout parameter in [drbd.conf] to work (I set it up as in all the examples I saw). I set it low (say 1 second), kill the remote box IO [re]commences after 10 seconds (as the other parameters state).

Anything I’m doing wrong?

Well, sort of. The timeout parameter specifies the timeout DRBD uses for the blocking sockets it transmits data over. So if you issue I/O on the DRBD device while Connected, and a corresponding replication packet does not complete within timeout, then DRBD concludes that the peer has gone away and transitions to WFConnection, effectively switching into disconnected mode.

If however the connection is lost while the DRBD device is idle (not handling any write I/O), then there are no packets to replicate, and none to wait for. By itself, this would mean that while idle, DRBD would be unable to detect that its peer has gone away. Clearly, this would not be desirable.

Here’s where DRBD’s in-protocol “pings” come into play. Don’t confuse this with real ICMP echo requests. A DRBD “ping” is simply a no-op message inside the DRBD replication layer, as in the peers shouting at each other, “hello, I’m still here.” DRBD sends these “pings” in a configurable interval, specified by the ping-int configuration option and defaulting to 10 seconds. DRBD “pings” time out within the time specified as ping-timeout, which by default is 0.5 seconds.

So: while I/O is being issued on a device, it’s timeout that governs disconnection. While it is idle, however, disconnection is initiated by a “ping” packet (which is issued every 10 seconds, unless otherwise configured with ping-int) not being received within half a second (unless otherwise configured with ping-timeout).

Finally, for the sake of completeness, I should add that there is also a connect-int option, which is the interval DRBD uses for re-connecting to the peer in case of a connection failure. timeout must be lower than both connect-int and ping-int, otherwise it will be ignored. That is the issue that the user I quoted ran into.

All of this is, of course, explained in more detail in the drbd.conf man page.

LINBIT will be a proud sponsor at this year’s LinuxCon in Portland, OR, held on September 21-23. We will also participate in the Linux Plumbers Conference, co-located with LinuxCon.

Phil is to speak at LinuxCon about DRBD 8.3 and beyond. At LPC, we will primarily focus on building highly available storage systems with DRBD, Pacemaker, and the STGT and LIO iSCSI targets.

So, storage and data center folks on the West Coast, come and meet us! MySQLers and Drizzlers from the Pacific Northwest (mordred, that means you!), drop us a line to see how we can link up! Novell folks, let’s sit down and chat about what we can do to help improve SLE 11 HAE! Red Hat/Fedora people, let’s talk about how we can improve DRBD integration in your distributions! Virtualization folks, let’s talk about seamless DRBD integration with libvirt, like it already exists in Xen!

Please drop us a line at linuxcon at linbit dot com or lpc at linbit dot com if you are coming to either event, so we can arrange a get-together. Thanks!

LINBIT just announced a collaboration with Logicworks.

Logicworks, based in New York City, is a long-term enterprise user of DRBD based storage replication. If you’re into baseball and follow MLBlogs, DRBD is what it runs on and Logicworks is who runs it. Their Managed Storage Services offering has leveraged DRBD for a long time. The same is of course true for their Private Cloud service.

This is really cool stuff.

One of the things that repeatedly seem to puzzle users about the DRBD is the question of whether to use internal or external metadata. Remember, DRBD sets aside a small area on a local disk (on every cluster node) where it keeps the Activity Log, the quick-sync bitmap, data generation UUIDs, and a few other bits and pieces for local housekeeping.

The specific aspect that is to be discussed here is the Activity Log. Without going into too much detail, let’s be satisfied with the factoid that DRBD “occasionally” (it’s a little more involved in reality) writes to the AL, and has to wait for that write to complete before it can handle user data again. This wait is the crucial point. It’s usually on the order of just a few milliseconds, but on busy systems this can add up to where it throttles throughput just a little.

Now, what makes I/O fast or slow (on rotational hard drives, solid state is a different matter)? That’s right, it’s disk seeks. So when we use internal meta data, so the theory goes, the read-write head has to do something in the data area, then move to the AL and do something there, then move back to the data area, and so forth. Which, intuitively, can be speeded up if you put user data and meta data on different spindles. Different “logical” disks won’t do, it has to be on a separate spindle, so read-write heads can move in parallel. Again, this is as the naïve theory goes. Use external meta data, devise a clever scheme on how to spread your meta data apart from your user data, and you’ll be fine. And you can call yourself a great wizard in storage subsystem tuning. Well, not quite, unfortunately.

The problem is, you’ve made a crucial mistake in performance tuning. You are completely ignoring the effects of a battery-backed write cache. If, as we always recommend, you use a reasonable useful storage controller, which comes with a decent write cache and a battery backup unit, then the whole issue is moot. Because then you are no longer waiting for actual disk seeks to complete. What you think you are writing into disk sectors actually goes into a piece of controller RAM, and completes pretty much instantaneously. It’s the controller’s job to get this stuff onto stable storage later, and guarantee that it does so even in the face of a power failure. That’s what the BBU is for. But the whole idea of avoiding disk seeks for meta data writes is pretty much irrelevant now.

Which means you can scrap your grand user data/meta data distribution scheme and focus on important issues.

Bottom line: if using external metadata actually improves your performance versus internal metadata, you have underlying performance problems to fix. And you should fix those rather than patch them up at the DRBD level.

Of course I should add that we offer performance tuning and architectural consultancy for the complete high availability stack, including hardware tuning recommendations. So if you are uncomfortable with these tuning issues or could use some help, drop us a line. We’ll be happy to support you.

The latest addition to the Planetarium is www.planet-ha.org, a feed aggregator covering blogs and news feeds about all things HA. Active as of today, Planet HA is intended as a central location where you can find the latest open source high availability additions to the blogosphere.

An aggregated RSS feed is available from www.planet-ha.org/rss20.xml.

If you blog about HA and would like to get your feed added to Planet HA, here’s how:

• If your blog primarily covers HA (like my own), then simply submit your feed URL by email to planet dash ha at linbit dot com. We’ll take anything that Planet can grok; RSS and Atom feeds are usually perfectly fine.
• If you occasionally blog about HA, then please create a tag or category (if you haven’t already done so), and submit a filtered feed.
• The site is (for the time being) English-only, so if you blog in multiple languages it would be tremendously helpful if you could create an English-only tag/category and submit a filtered feed for that.

LINBIT is sponsoring CPU cycles and bandwidth for this site. But we absolutely don’t mind if you happen to never blog about DRBD. At all. Just so that’s clear. If you blog about HA, your feed is more than welcome.

Brilliant article from Charlie on whether SANs make sense. Excellent food for thought. Do take a look.