Fundraising tech/Message queues

This page gives an overview of the message queues used to decouple fundraising subsystems. For a description of the message format, see "Normalized donation messages". See also the article on WMF-specific configuration.

Message Queue
Queues are used to decouple the payments frontend from the CiviCRM server. This is important for several reasonsː it allows us to continue accepting donations even if the backend servers are down, it keeps our private database more secure, and it enforces write-only communication from the payments cluster.

The main data flow is over the donations queue. Completed payment transactions are encoded as JSON and sent over the wire, to be consumed by the queue2civicrm Drupal module and recorded in the CiviCRM database.

Another important queue is the limbo queue, which is used both as a key-value store and as a FIFO queue. Before we pass control to any hosted page or iframe, we record the donor's personal information we've collected to the limbo queue, indexed by the gateway and transaction ID. We store the information in this temporary fashion so that a) it does not leave the payments cluster, and b) so we aren't storing any data about people who aren't donors, which is mandated by our privacy policies. When (and if) control is returned to the payments server, the PHP session is used to build the key and search for a corresponding limbo message. We delete the message, and merge this information into the completed donation message sent to the regular queue.

However, if control is never returned, then limbo queue messages will sit around for some time. After about 20 minutes, they become eligible for "orphan slaying", which is currently only performed for GlobalCollect credit card transactions. We attempt to complete settlement on these orders, and if successful, the completed message is sent to the donations queue. If unsuccessful, the personal information should be purged.

At Wikimedia, we are currently using the ActiveMQ (http://activemq.apache.org/) message broker as the queue backend for everything but the limbo queue. Messages go over the wire using the aging STOMP protocol. The limbo queue on the other hand is stored in Redis on the payments-cluster.

Queues
Queue names may be configured differently on development or test machines. The names given here are the production defaults.


 * donations: Primary queue for incoming donations. Written to by DonationInterface, and by IPN listeners.
 * globalcollect-cc-limbo: See the "limbo" queue below. GlobalCollect credit card limbo messages are segregated into their own bucket, to make it easier for the orphan slayer to pull only those messages.
 * job-requests: SmashPig jobs to be processed. TODO: explain.
 * limbo: donation methods which use an iframe (GC, Adyen) will leave a message on the limbo queue before transferring UI flow to the processor. When the processor returns control, we delete the limbo message.
 * pending: Temporary storage used by some IPN listeners and frontends. Messages will either be popped off the queue after a fixed time, to complete settlement steps; or upon incoming notification of a status change; or will expire in a short amount of time.
 * pending_paypal: another pending queue (please deprecate)
 * pending_paypal_recurring: another pending queue (please deprecate)
 * refund-notifications: incoming IPN notifications that the processor has completed a refund
 * unsubscribe: The FundraisingEmailUnsubscribe module allows donors to opt out of bulk mailings and sends these requests over a queue, to be consumed by the CRM.

Consumers

 * queue2civi: The main consumer is the q2c job, which reads from the donations queue and stores in our CRM database.
 * refund: Q2C module to record refund notifications, marking the affected contribution
 * recurring: Q2C module
 * unsubscribe: (mostly unused) Decouples the mailing list unsubscribe UI from the
 * orphan slayer: A script that attempts to "rectify" the orphaned message by settling at the gateway.