After last year’s talk in Brisbane, where I greatly enjoyed co-presenting with Tim Serong, I have the privilege of returning to Australia for this year’s linux.conf.au in Ballarat, Victoria.

This time I have a brief talk opening up the High Availability and Distributed Storage miniconf on Monday, January 16, and a tutorial entitled High Availability Sprint in the morning on Thursday, January 19. Tim Serong is again joining me for the tutorial, and Pacemaker author Andrew Beekhof will be chiming in too. See you all in Ballarat!

Bear with me while I rant, please. Note: I love the concept of European integration, and I think despite all its shortcomings the EU is a great thing. But this is just an example of plain and simple WTF, and a very evident SPOF that is potentially affecting thousands of businesses in Europe at this time.

Background: Within the EU, we have the concept of intra-community supply, where business-to-business transactions between companies registered in different member states are (I’m oversimplifying) exempt of Value-Added Tax (VAT). In order to verify that the customer who receives the invoice is in fact a business, rather than a consumer, the burden is on the supplier to verify the customer’s VAT ID — a unique identifier for tax purposes.

To this end, the European commission provides a web site for manual VAT ID verification by humans. That site also hosts a SOAP-based web service for automated VAT verification by business-to-business commerce sites. Note: when this verification is unavailable, it means a supplier can either stop issuing invoices, or resort to manual verification via the human-readable interface.

And that SOAP-based service has been non-functional for almost 24 hours now. It’s not like the WSDL definition is unavailable, although it has produced 404s occasionally. It’s just that any POST requests to that service yield a 500 with the following beautiful stack trace:

org.apache.axis2.AxisFault: The service cannot be found for the endpoint reference (EPR) http://taxudp5.cc.cec.eu.int:7174/taxation_customs/vies/services/checkVatService
	at org.apache.axis2.engine.DispatchPhase.checkPostConditions(DispatchPhase.java:65)
	at org.apache.axis2.engine.Phase.invoke(Phase.java:333)
	at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:264)
	at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:163)
	at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)
	at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:133)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
	at cec.taxud.fiscalis.vies.viesweb.web.servlet.Axis2FilterServlet.service(Axis2FilterServlet.java:49)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1072)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cec.taxud.fiscalis.vies.viesweb.api.CheckVatServiceHTTPFilter.doFilter(CheckVatServiceHTTPFilter.java:172)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cec.taxud.fiscalis.vies.viesweb.util.EncodingFilter.doFilter(EncodingFilter.java:26)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6987)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3892)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2766)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

Looks like someone either totally forgot about making that backend service highly available, or messed up their load balancer, or is monitoring their endpoints incorrectly. Not fun.

But fear not, there is actually an email address posted on the EC’s web site where you can notify people of the problem. Alas, no response there at all in over 12 hours. No autoreply, no “thanks, we’re working on it”, nothing. Maybe it’s because of this priceless statement (found here):

Please note that the European Commission’s offices will be closed from 23 December 2011 to 2 January 2012 inclusive. Your question will therefore not be handled before 3 January 2012.
Thank you for your understanding.

(Emphasis in original). Fun, eh?

Martin has updated the Pacemaker packages in the Debian squeeze-backports repository to release 1.1.6; the latest Pacemaker release available from upstream. If you’re running the Pacemaker 1.1 series on Debian, you’re advised to update.

So I’ll reiterate, at their request: LINBIT is not affiliated with, invested in, or otherwise connected to hastexo in any way. We’re definitely not a spin-off. We continue to be part of the same open-source community, but there is no corporate connection whatsoever.

I hope that makes it clearer. By the way, their name is a registered trademark.

DRBD core developer Lars Ellenberg has made yet another attempt at explaining why this won’t work in a post to the drbd-user mailing list last night. Here’s an excerpt:

“Dual-primary” iSCSI targets for multipath: does not work. iSCSI is a stateful protocol, there is more to it that than just reads and writes. To run multipath (or multi-connections per session) against distinct targets on separate nodes you’d need to have cluster aware iSCSI targets which coordinate with each other in some fashion. To my knowledge, this does not exist (not for Linux, anyways).

(Emphasis in original, I merely reformatted for HTML.)

He goes on to explain that these distinct targets just

happen to live on top of data that, due to replication, happens to be the same, most of the time, unless the replication link was lost for whatever reason; in which case you absolutely want to make sure that at least one box reboots hard before it even thinks about completing or even submitting an other IO request…

Please, folks, listen to Lars. This is all very much in line with the “DRBD doesn’t do magic” note we’ve had in the User’s Guide for several years now:

DRBD is, by definition and as mandated by the Linux kernel architecture, agnostic of the layers above it. Thus, it is impossible for DRBD to miraculously add features to upper layers that these do not possess.

So please, if you’re seeing Concurrent local write detected or the tell-all DRBD is not a random data generator! message in your logs, don’t come complaining. And even if you don’t see them yet, you will, eventually.

When you think you must run on dual-Primary DRBD, then run a cluster aware service on top of it. Such as cLVM, or OCFS2, or GFS2, or even live migration capable KVM under Pacemaker management with fencing, but not iSCSI. Unless you write a cluster aware iSCSI target.

Until then, run iSCSI on single-Primary DRBD under Pacemaker management. Configure your iSCSI initiator properly so it does not throw I/O errors on failover. If you don’t know how, ask someone who does. Find us on IRC, or on the mailing lists; we’re happy to help. Or give us a call.

We’ve just announced our new training format, hastexo High Availability Expert. If you’re an experienced high-availability engineer and you want to learn how to become a true top-notch clustering expert, this is for you.

Brought to you by the same people who devised the entry-level High Availability Engineer class that continues to be offered by Linbit, this three-day course adds significantly advanced coverage of the Pacemaker based Linux High Availability stack.

Among the issues we cover are

• Advanced Pacemaker concepts: active/active clustering, dynamic resource load balancing, system health monitoring, and advanced fencing.
• Cluster filesystems inside and out: OCFS2 and GFS2 with and without dual-Primary DRBD. Performance and reliability considerations for clustered filesystems, and alternative solution approaches.
• Multi-node replicated storage: Using Gluster and Ceph for scaled-out storage clusters and enterprise clouds.

And of course, you get to pick our brains. We work with customers on complex high availability issues every day. We’ve seen the common traps and pitfalls you want to avoid. And we’re happy to share our experience with you.

For our first class of this new format, we’re returning to one of Europe’s finest cities, Berlin. And we are accepting signups immediately. Due to the highly advanced nature of this class, we are limiting the attendance to just 8 seats. So hurry — seats won’t last long!

The cost for this course is 30 hastexo Professional Service Credits (€ 1,275 if booked before Dec 31). The course description contains a detailed agenda and FAQ. If you have questions about the format, please don’t hesitate to drop us a line!

This is something that many users tripped over previously, so I’m happy to finally see it available in the published Guide.

There are several other additions and fixes currently pending, so I’m hoping they will publish those soon, too.

This is a somewhat overdue maintenance update highlighting that the upstream resource agents repo has

• switched to the Git version control system (from Mercurial);
• moved to GitHub.

It also contains a few other minor changes, but nothing spectacular. The recent commit history is here. Comments and feedback are always welcome.

The Pacemaker based Linux cluster stack is gaining a freshly supported service: the Asterisk open-source PBX. hastexo‘s Martin Loschwitz has contributed a resource agent for the popular telephony stack.

I’ve merged and committed the ocf:heartbeat:asterisk resource agent to the upstream repository this morning, and it’s going to ship with the next release of the resource-agents package. If you already run a Pacemaker cluster, and your own version of resource-agents is reasonably recent, you can use the agent already: just fetch it from git, and drop it into the /usr/lib/ocf/resource.d/heartbeat directory on your machines.

The RA supports Asterisk in realtime mode including monitoring via astcanary, and you can also use Pacemaker’s resource auto-recovery features by specifying a SIP URI to be monitored: if the SIP URI does not respond to an OPTIONS request, then the resource is considered failed, and Pacemaker recovers it either on the same node, or on another (depending on your cluster configuration).

The basic Pacemaker configuration for a highly-available Asterisk PBX is quite simple:

primitive p_asterisk ocf:heartbeat:asterisk

A more realistic real-world example, though, is an Asterisk with a virtual IP address and a DRBD-backed /var/lib/asterisk filesystem, and automated resource monitoring:

primitive p_drbd_asterisk ocf:linbit:drbd \
  params drbd_resource="asterisk"
ms ms_drbd_asterisk p_drbd_asterisk \
  meta notify=true clone-max=2
primitive p_fs_asterisk ocf:heartbeat:Filesystem \
  params device="/dev/drbd0" directory="/var/lib/asterisk"
primitive p_ip_asterisk ocf:heartbeat:IPaddr2 \
params ip="192.168.42.42" cidr_netmask="24" \
  op monitor interval="10s"
primitive p_asterisk ocf:heartbeat:asterisk \
  params realtime="true" monitor_sipuri="sip:42@192.168.42.42" \
  op monitor interval="10s"
group g_asterisk p_fs_asterisk p_ip_asterisk p_asterisk
colocation c_asterisk_on_drbd_master inf: g_asterisk ms_drbd_asterisk:Master
order o_drbd_promote_before_asterisk_start inf: ms_drbd_asterisk:promote g_asterisk:start

Besides this standard HA configuration, you can of course also include a highly-available MySQL database, or an iSCSI backed filesystem (instead of DRBD), multiple Asterisk instances, etc.

Special thanks goes to first-time OCF resource agent reviewer Russell Bryant (of Asterisk fame) and all other reviewers on the linux-ha-dev and linux-ha mailing lists.

hastexo will of course offer installation support, consulting and training for Pacemaker-based Asterisk PBX high availability as soon as the release happens.

• In my tutorial MySQL High Availability Sprint: Launch the Pacemaker, I gave a fast walk-through of Pacemaker high availability for MySQL. Tutorial presented at Percona Live UK 2011 in London, England.
• In Fencing and Maintaining Sanity in High Availability Clusters, Madison Kelly and I explored the importance of fencing in HA clusters, highlighted important considerations for fencing, and shared technical details and best practices. Talk presented at Linuxcon Europe 2011 in Prague, Czech Republic.

All slides are available entirely free of charge for logged-in users on our web site. To log in, you don’t even need to register — just use your Google Profile, or Google Apps account, or your WordPress account, or anything else that uses OpenID, and you’ll be good to go.

Comments on our slides are, of course, always highly appreciated.