Fundraising tech/Message queues

This page gives an overview of the message queues used to decouple fundraising subsystems. For a description of the message formats, see "Normalized donation messages". See also the Wikitech article on WMF-specific configuration.

Message Queue
Queues are used to decouple the payments frontend from the CiviCRM server. This is important for several reasonsː it allows us to continue accepting donations even if the backend servers are down, it keeps our private database more secure, and it enforces write-only communication from the payments cluster.

The main data flow is over the donations queue. Completed payment transactions are encoded as JSON and sent over the wire, to be consumed by the queue2civicrm Drupal module and recorded in the CiviCRM database.

Another important queue is the limbo queue, which is used both as a key-value store and as a FIFO queue. Before we redirect the donor to a payment processor hosted page or iframe, we record the donor's personal information and push it to the limbo queue, where it's indexed by gateway name and transaction ID. We store this information in a temporary fashion rather than in a database out of concern for storing data about people who aren't donors. When (and if) control is returned to the payments server, the PHP session is used to build the key and search for a corresponding limbo message. We delete the message, and merge this information into the completed donation message sent to the regular queue.

However, if control is never returned, then limbo queue messages will sit around for some time in case the data will become useful again. After about 20 minutes, they become eligible for orphan rectification, currently only applied to GlobalCollect credit card transactions. We attempt to complete settlement on these orders, and if successful, the completed message including limbo-provided details is sent to the donations queue. If unsuccessful, the personal information should be purged.

At Wikimedia, we are currently using the ActiveMQ (http://activemq.apache.org/) message broker as the queue backend for everything but some limbo and inflight queues. Messages go over the wire using the aging STOMP protocol.

Overhaul 2016
See the main article: Fundraising Queue Overhaul.