Excuse that deliberately Google-optimized blunt and inelegant title, folks, but this is getting old. If you run dual-Primary DRBD, and then export an iSCSI target from both nodes, and then you want to to do dm-multipath or somesuch for what you think constitutes failover, don’t do that. There. Bold and italics. Really and truly, don’t.
“Dual-primary” iSCSI targets for multipath: does not work. iSCSI is a stateful protocol, there is more to it that than just reads and writes. To run multipath (or multi-connections per session) against distinct targets on separate nodes you’d need to have cluster aware iSCSI targets which coordinate with each other in some fashion. To my knowledge, this does not exist (not for Linux, anyways).
(Emphasis in original, I merely reformatted for HTML.)
He goes on to explain that these distinct targets just
happen to live on top of data that, due to replication, happens to be the same, most of the time, unless the replication link was lost for whatever reason; in which case you absolutely want to make sure that at least one box reboots hard before it even thinks about completing or even submitting an other IO request…
Please, folks, listen to Lars. This is all very much in line with the “DRBD doesn’t do magic” note we’ve had in the User’s Guide for several years now:
DRBD is, by definition and as mandated by the Linux kernel architecture, agnostic of the layers above it. Thus, it is impossible for DRBD to miraculously add features to upper layers that these do not possess.
So please, if you’re seeing
Concurrent local write detected or the tell-all
DRBD is not a random data generator! message in your logs, don’t come complaining. And even if you don’t see them yet, you will, eventually.
When you think you must run on dual-Primary DRBD, then run a cluster aware service on top of it. Such as cLVM, or OCFS2, or GFS2, or even live migration capable KVM under Pacemaker management with fencing, but not iSCSI. Unless you write a cluster aware iSCSI target.
Until then, run iSCSI on single-Primary DRBD under Pacemaker management. Configure your iSCSI initiator properly so it does not throw I/O errors on failover. If you don’t know how, ask someone who does. Find us on IRC, or on the mailing lists; we’re happy to help. Or give us a call.