Fundraising tech/Message queues

Message Queue
Queues are used to decouple the payments frontend from the CiviCRM server. This is important for several reasonsː it allows us to continue accepting donations even if the backend servers are down, it keeps our private database more secure, and it enforces write-only communication from the payments cluster.

The main data flow is over the donations queue. Completed payment transactions are encoded as JSON and sent over the wire, to be consumed by the queue2civicrm Drupal module and recorded in the CiviCRM database.

Another important queue is the limbo queue, which is used both as a key-value store and as a FIFO queue. Before we pass control to any hosted page or iframe, we record the donor's personal information we've collected to the limbo queue, indexed by the gateway and transaction ID. We store the information in this temporary fashion so that a) it does not leave the payments cluster, and b) so we aren't storing any data about people who aren't donors, which is mandated by our privacy policies. When (and if) control is returned to the payments server, the PHP session is used to build the key and search for a corresponding limbo message. We delete the message, and merge this information into the completed donation message sent to the regular queue.

However, if control is never returned, then limbo queue messages will sit around for some time. After about 20 minutes, they become eligible for "orphan slaying", which is currently only performed for GlobalCollect credit card transactions. We attempt to complete settlement on these orders, and if successful, the completed message is sent to the donations queue. If unsuccessful, the personal information should be purged.

At Wikimedia, we are currently using the ActiveMQ (http://activemq.apache.org/) message broker as the queue backend for everything but the limbo queue. Messages go over the wire using the aging STOMP protocol. The limbo queue on the other hand is stored in Redis on the payments-cluster.

Queues
Queue names may be configured differently on development or test machines. The names given here are the production defaults.


 * donations: Primary queue for incoming donations. Written to by DonationInterface, and by IPN listeners.
 * job-requests: SmashPig jobs to be processed
 * limbo: donation methods which use an iframe (GC, Adyen) will leave a message on the limbo queue before transferring UI flow to the processor. When the processor returns control, we delete the limbo message.
 * pending: temporary storage used by some IPN listeners. Messages are supposed to be popped off after processing is finished.  They will only be left on the queue if an unexpected fatal error kills the listener process.
 * pending_paypal: another pending queue (please deprecate)
 * pending_paypal_recurring: another pending queue (please deprecate)
 * refund-notifications: incoming IPN notifications that the processor has completed a refund
 * unsubscribe: The FundraisingEmailUnsubscribe module allows donors to opt-out of further mailings and sends requests over a queue.

Consumers

 * queue2civi: The main consumer is the q2c job, which reads from the donations queue and stores in our CRM database.
 * refund: Q2C module to record refund notifications, marking the affected contribution
 * recurring: Q2C module
 * unsubscribe: (mostly unused) Decouples the mailing list unsubscribe UI from the
 * orphan slayer: A script that attempts to "rectify" the orphaned message by settling at the gateway.