Looks like the European Commission could use some high availability

Bear with me while I rant, please. Note: I love the concept of European integration, and I think despite all its shortcomings the EU is a great thing. But this is just an example of plain and simple WTF, and a very evident SPOF that is potentially affecting thousands of businesses in Europe at this time.

Background: Within the EU, we have the concept of intra-community supply, where business-to-business transactions between companies registered in different member states are (I’m oversimplifying) exempt of Value-Added Tax (VAT). In order to verify that the customer who receives the invoice is in fact a business, rather than a consumer, the burden is on the supplier to verify the customer’s VAT ID — a unique identifier for tax purposes.

To this end, the European commission provides a web site for manual VAT ID verification by humans. That site also hosts a SOAP-based web service for automated VAT verification by business-to-business commerce sites. Note: when this verification is unavailable, it means a supplier can either stop issuing invoices, or resort to manual verification via the human-readable interface.

And that SOAP-based service has been non-functional for almost 24 hours now. It’s not like the WSDL definition is unavailable, although it has produced 404s occasionally. It’s just that any POST requests to that service yield a 500 with the following beautiful stack trace:

org.apache.axis2.AxisFault: The service cannot be found for the endpoint reference (EPR) http://taxudp5.cc.cec.eu.int:7174/taxation_customs/vies/services/checkVatService
	at org.apache.axis2.engine.DispatchPhase.checkPostConditions(DispatchPhase.java:65)
	at org.apache.axis2.engine.Phase.invoke(Phase.java:333)
	at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:264)
	at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:163)
	at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)
	at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:133)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
	at cec.taxud.fiscalis.vies.viesweb.web.servlet.Axis2FilterServlet.service(Axis2FilterServlet.java:49)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1072)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cec.taxud.fiscalis.vies.viesweb.api.CheckVatServiceHTTPFilter.doFilter(CheckVatServiceHTTPFilter.java:172)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cec.taxud.fiscalis.vies.viesweb.util.EncodingFilter.doFilter(EncodingFilter.java:26)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6987)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3892)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2766)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

Looks like someone either totally forgot about making that backend service highly available, or messed up their load balancer, or is monitoring their endpoints incorrectly. Not fun.

But fear not, there is actually an email address posted on the EC’s web site where you can notify people of the problem. Alas, no response there at all in over 12 hours. No autoreply, no “thanks, we’re working on it”, nothing. Maybe it’s because of this priceless statement (found here):

Please note that the European Commission’s offices will be closed from 23 December 2011 to 2 January 2012 inclusive. Your question will therefore not be handled before 3 January 2012.
Thank you for your understanding.

(Emphasis in original). Fun, eh?

Dual-primary DRBD, iSCSI, and multipath: Don’t Do That!

Excuse that deliberately Google-optimized blunt and inelegant title, folks, but this is getting old. If you run dual-Primary DRBD, and then export an iSCSI target from both nodes, and then you want to to do dm-multipath or somesuch for what you think constitutes failover, don’t do that. There. Bold and italics. Really and truly, don’t.

For the n-th time, ReiserFS is not a cluster file system

Neither is ext3. Nor ext4. Nor btrfs. And thus, none of these will work on dual-Primary DRBD. Nor active-active shared storage. Nor any synchronously replicated active-active SAN. And we’re telling you very clearly.

So if you choose to ignore all warnings and put ReiserFS on dual-Primary DRBD, and mount it from two nodes, you’ve just signed up for wrecking your data. And when that happens, don’t come whining. And don’t blame DRBD or any other of the technologies you may be choosing to employ while ignoring the documentation.