Fundraising tech/Message queues

This page gives an overview of the message queues used to decouple fundraising subsystems. For a description of the message formats, see "Normalized donation messages". See also the Wikitech article on WMF-specific configuration.

Message Queue
Queues are used to decouple the payments frontend from the CiviCRM server. This is important for several reasonsː it allows us to continue accepting donations even if the backend servers are down, it keeps our private database more secure, and it enforces write-only communication from the payments cluster.

The main data flow is over the donations queue. Completed payment transactions are encoded as JSON and sent over the wire, to be consumed by the queue2civicrm Drupal module and recorded in the CiviCRM database.

Another important queue is the pending queue, which is used both as a key-value store and as a FIFO queue. Before we redirect the donor to a payment processor hosted page or iframe, we record the donor's personal information and push it to the pending queue, where it's consumed to a table indexed by gateway name and transaction ID. We store this information in a temporary fashion rather out of concern for storing data about people who aren't donors. When the donation is entered into the civi database, we search for a corresponding pending message. We delete the message, and merge this information into the completed donation message sent to the regular queue.

However, if control is never returned, then pending db messages will sit around for some time in case the data will become useful again. After about 20 minutes, they become eligible for orphan rectification, currently only applied to Ingenico credit card transactions and PayPal Express Checkout Payments. We attempt to complete settlement on these orders, and if successful, the completed message including pending-provided details is sent to the donations queue. If unsuccessful, the personal information should be purged. All pending information is purged after at most 30 days.

At Wikimedia, we are currently using lists in Redis (https://redis.io/) as the queue backend.

Overhaul 2016
See the main article: Fundraising Queue Overhaul.