… here’s our first RC of Heartbeat 3.0.2!

High availability, that’s us. Bottom to top.

DRBD has entered a new phase. After being developed out of tree for 9 years, and after an extended review and streamlining phase since March, Phil submitted DRBD to be merged into 2.6.32 release of the Linux mainline kernel. The submission was accepted by block layer maintainer Jens Axboe, who merged DRBD in September, then deferred to the 2.6.33 merge window, and this morning Linus pulled DRBD into his tree.

That makes DRBD an integral part of Linux, starting with the 2.6.33 release expected in a few weeks’ time.

We have something to celebrate.

Andrew, whom I helped convince to switch to DocBook for writing Pacemaker documentation, pointed me to Publican, a DocBook publishing framework developed by tech writers at Red Hat.

Publican packages have been available in Debian squeeze for a while, but as of today are not included in Ubuntu. Thus, I built packages for karmic, which you may fetch by adding the following line to your /etc/apt/sources.list file:

deb http://people.linbit.com/~florian/ubuntu/publican karmic main

Then fetch my GnuPG key so you can verify the Release file’s integrity:

apt-key adv --keyserver pgp.mit.edu --recv-keys 7349B897BC759CF1

Now you can install with

aptitude update
aptitude install publican

… and then hack away.

Ubuntu folks (Andres, that means you!), can we have this in the upstream Ubuntu distros please?

Here is an announcement we made earlier today on the linux-ha and linux-ha-dev mailing lists.

This is to announce that LINBIT, with the kind permission from the Linux-HA project board, will act as the “steward” of the Heartbeat cluster messaging layer code base, from this point forward. This is a summary of our motivation and plans related to that role.

What does this entail?

• LINBIT will assume responsibility for bug fixes for the Heartbeat code base, currently hosted at http://hg.linux-ha.org/dev/.
• LINBIT will bundle up the 3.0 beta codebase, make a 3.0 final release (currently this is planned for the month of January 2010), and subsequently make bugfix releases as deemed necessary.
• LINBIT will further collaborate with the Pacemaker project to keep the existing dual-stack capability in Pacemaker.
• LINBIT will continue making the public Mercurial repository available at the present location (any eventual relocation, if desired by the Board, would be publicly announced with ample advance notice).
• LINBIT will administer the public mailing lists (linux-ha and linux-ha-dev) on the servers currently hosting them (again, any eventual relocation, would be publicly announced with ample advance notice).
• LINBIT intends to offer improved documentation for the Heartbeat messaging layer. This is meant to consolidate the content currently found on the linux-ha.org wiki site.
• LINBIT intends to offer support services for the Heartbeat/Pacemaker cluster stack (i.e. the Pacemaker cluster resource manager running on top of the Heartbeat cluster communication layer).
• LINBIT will continue to respect he Board as the final authority on matters affecting the project as a whole.

What does this not entail?

• LINBIT has no intention to add significant features to the Heartbeat code base, or extend its functionality significantly.
• LINBIT has no intention to apply changes to the licensing, development model, or collaboration model for the Linux-HA code base.
• LINBIT has no intention to establish the Heartbeat code base as a long-term alternative or competition to the OpenAIS/Corosync cluster messaging layer. However, we do believe that it is a valid alternative for the short to mid term, and for some configurations where OpenAIS/Corosync is currently suffering from some growing pains.
• LINBIT has no intention to support or advocate continued use of Heartbeat in v1 (haresources) configurations. We will continue to recommend to switch to the Pacemaker cluster stack, now that two (technically and commercially) supported cluster messaging layers are available.

At this time, the primary contact in charge of Heartbeat development matters at LINBIT is Lars Ellenberg, the person in charge of documentation is myself. The best means of relaying comments and asking questions continues to be the public mailing list.

We hope that this is a useful service to the Heartbeat user community. I want to reiterate that we have no intention whatsoever to change the current, proven, community centric approach to how the Heartbeat code base is managed. We continue to welcome, and depend on, community suggestions, feedback, and collaboration. Heartbeat is a community project and will remain so.

If you have any questions about our intentions and plans, please post them on the mailing list, or peruse the comment fields below.

A completely non-technical post for a change.

Last weekend, seven employees of LINBIT’s European division participated in the World Games of Mountain Biking in Saalbach-Hinterglemm, Austria. We took part as Marathon race participants and co-sponsors of Biking for Butterfly Children, a charity dedicated to the fight against epidermolysis bullosa (EB).

Currently, no cure for any of the over 30 subtypes of EB exists — dermatologists and care givers can, however, greatly improve patients’
quality of life. Still, EB can be an excruciatingly painful, disfiguring, and debilitating disease that affects one in 20,000 live births and as such, makes the condition an orphan disease. The health care industry has little incentive to research the condition (as there is little money to be made off of it), and those who dedicate their careers to EB patient care and research rely on charitable donations for funding. Biking for Butterfly Children acts as a reliable fund raiser rounding up much-needed donations in the course of amateur cycling events (such as the World Games).

During this year’s World Games, B4BC raised a total of about 7,000 euros in donations — a respectable sum for an all-amateur event, but a lot more money is needed to improve the quality of life of EB patients, and potentially discover a cure to the disease. If you consider joining the fight against EB, please contact your local DebRA chapter.

Every once in a while, people ask us something along the lines of “why do I need DRBD? Can’t I accomplish what it does by other means?” You mean build high availability clusters with block-level synchronous replication? Well, sure you can. But all of the available alternatives have serious drawbacks.

• Use a SAN. Well, SANs are great in terms of management, even though they’re sometimes prohibitively expensive. But your regular SAN box does not offer physically distributed redundancy at the data level. In order words, if your SAN box goes down, all your beautiful high availability infrastructure turns to shreds. And even if you naïvely believe (and you shouldn’t) that a storage box can never crash, just think about air conditioning going down in just that part of your data center where your storage shelf is at. Repair time of several hours means down time of several hours, even though your servers in a different cabinet may be up and running. You’re dealing with a single point of failure. Not high availability in my book.
• Use a SAN with native replication. This means using two separate storage boxes with synchronous block-level replication between them. This eliminates the above-mentioned SPOF and is available from just about any SAN vendor (using proprietary implementations under various product names). The downside is that it costs you serious bucks, and I am not referring to just the additional piece of hardware. Those firmware licenses can hit six figures. Plus, switchover times (changing the direction of replication) can be extremely long, up to 4 minutes in some cases. And there is little to no support for replication management from open source cluster management software.
• Use a SAN with host-based mirroring. This means that you have two separate SAN boxes, hosts import LUNs from both, and mirror those pairs using software RAID (such as md). Eliminates the SPOF and saves you dollars on firmware licensing. Downsides: you still need a SAN (and the associated infrastructure — fibre channel, for example, isn’t exactly cheap), and as such your clusters are still not shared-nothing. And the integration with open source cluster management is also lacking.
• Use host-based mirroring between a local device, and a network block device. In this case, you have one disk that is local, and another that is exported from a remote host using NBD or iSCSI. Those two disks are then mirrored with software RAID. Now this one is really terrible in terms of management. Role reversal always requires some custom glue, no support from cluster manages is available whatsoever, and split brain detection is poor or non existant. So if you really want to go down that alley then do — but please don’t call it a high availability cluster.

Compare this to DRBD: no need for a SAN, so you can use it to operate a fully shared-nothing cluster. No firmware licensing cost. No need for expensive infrastructure as everything can replicate over regular IP networks. Role switch in a matter of seconds. Tight integration with both Pacemaker and Red Hat Cluster Suite.  Excellent split brain detection to make sure you don’t wreck your data accidentally. So if you are considering alternatives then that’s perfectly fine, but our soaring usage numbers are there for a reason.

With all of the discussions over the past years about MySQL Replication vs. DRBD (where the “vs.” part is in fact grossly misled of course — they are two technologies that complement each other quite well), here’s one with a slightly different angle: does it make sense to roll your own cluster manager around MySQL Replication, or is it smarter to plug into an existing, proven cluster architecture?

You’ll expect my own view to be fairly well defined, and it is. But make up your own mind!

Unfortunately, we’ve had a small issue with the drbd-user mailing list with some posts from new members not properly coming through. If you’ve been affected by this, our sincere apologies. Full functionality of the mailing list has now been restored.

Here is a question recently seen on drbd-user:

I cannot get the timeout parameter in [drbd.conf] to work (I set it up as in all the examples I saw). I set it low (say 1 second), kill the remote box IO [re]commences after 10 seconds (as the other parameters state).

Anything I’m doing wrong?

Well, sort of. The timeout parameter specifies the timeout DRBD uses for the blocking sockets it transmits data over. So if you issue I/O on the DRBD device while Connected, and a corresponding replication packet does not complete within timeout, then DRBD concludes that the peer has gone away and transitions to WFConnection, effectively switching into disconnected mode.

If however the connection is lost while the DRBD device is idle (not handling any write I/O), then there are no packets to replicate, and none to wait for. By itself, this would mean that while idle, DRBD would be unable to detect that its peer has gone away. Clearly, this would not be desirable.

Here’s where DRBD’s in-protocol “pings” come into play. Don’t confuse this with real ICMP echo requests. A DRBD “ping” is simply a no-op message inside the DRBD replication layer, as in the peers shouting at each other, “hello, I’m still here.” DRBD sends these “pings” in a configurable interval, specified by the ping-int configuration option and defaulting to 10 seconds. DRBD “pings” time out within the time specified as ping-timeout, which by default is 0.5 seconds.

So: while I/O is being issued on a device, it’s timeout that governs disconnection. While it is idle, however, disconnection is initiated by a “ping” packet (which is issued every 10 seconds, unless otherwise configured with ping-int) not being received within half a second (unless otherwise configured with ping-timeout).

Finally, for the sake of completeness, I should add that there is also a connect-int option, which is the interval DRBD uses for re-connecting to the peer in case of a connection failure. timeout must be lower than both connect-int and ping-int, otherwise it will be ignored. That is the issue that the user I quoted ran into.

All of this is, of course, explained in more detail in the drbd.conf man page.

LINBIT will be a proud sponsor at this year’s LinuxCon in Portland, OR, held on September 21-23. We will also participate in the Linux Plumbers Conference, co-located with LinuxCon.

Phil is to speak at LinuxCon about DRBD 8.3 and beyond. At LPC, we will primarily focus on building highly available storage systems with DRBD, Pacemaker, and the STGT and LIO iSCSI targets.

So, storage and data center folks on the West Coast, come and meet us! MySQLers and Drizzlers from the Pacific Northwest (mordred, that means you!), drop us a line to see how we can link up! Novell folks, let’s sit down and chat about what we can do to help improve SLE 11 HAE! Red Hat/Fedora people, let’s talk about how we can improve DRBD integration in your distributions! Virtualization folks, let’s talk about seamless DRBD integration with libvirt, like it already exists in Xen!

Please drop us a line at linuxcon at linbit dot com or lpc at linbit dot com if you are coming to either event, so we can arrange a get-together. Thanks!