Fundraising tech/Pending queue consumers

This page exists to document where we read messages from the 'pending' queues, prior to consolidation into a better key-value store.

queue2civicrm_import_completion_message
This runs inside the queue2civicrm completed donation queue import job. When a message comes in with a 'completion-message-id', the job calls this function to retrieve a message from the pending queue with missing donor details.

This is used for importing messages from Amazon and AstroPay.

Amazon message ID format is 'amazon-' + Amazon's CaptureReferenceId. The CaptureReferenceId is also stored as the gateway_txn_id for Amazon donations.

AstroPay message ID format is 'astropay-' + our order ID, which is the contribution tracking ID, a period, and the API sequence number. The order ID is included in AstroPay IPN messages as x_invoice and in audit files under the Invoice column. We don't have access to the AstroPay-side transaction ID when we enqueue the pending message at the payments frontend.

Adyen IPN message processing jobs
ProcessCaptureRequestJob reads from the pending queue, updates the message, and writes back. RecordCaptureJob reads from the pending queue, deletes that message, and adds a message to the completed donation queue.

Message ID format is 'adyen-' + our order ID, which is the ct_id + '.' + sequence number as with AstroPay

Expired message removal
This job removes all messages over a certain age from queues.

Reads by date

Paypal IPN
Written to and read from by standalone IPN listener, within the same process. Correlation ID is randomly generated by the IPN listener. Looks like it's supposed to hold messages in case of failure, but nothing picks up messages except for the expiry job.

DI frontend
Writes to 'limbo' when sending donor to processor, deletes when they come back. Could be used to accept successful donations after sessions expire, but isn't. Also writes to 'globalcollect-cc-limbo' when sending donor to Ingenico. All IDs are gateway name + '-' + order_id.

Orphan rectification job
Reads from 'globalcollect-cc-limbo' by date. Using the pending queue for this would mean selecting by date and by gateway name.

Conclusions
Some queue usage can simply be removed: nothing useful happens with Paypal IPN messages or with any limbo queue messages besides globalcollect-cc-limbo

The pending queue will need to make messages available by several different selectors: date, gateway name, gateway txn id, and order ID. Adding gateway account would be prudent in case we need multiple accounts per gateway in the future.

Indexes required

 * idx_pending_date (date) : Expired message removal
 * idx_pending_date_gateway (date, gateway) : Orphan rectification, potentially also expired message removal if we wanted gateway-specific expiration dates)
 * idx_pending_order_id_gateway (order_id, gateway) : Adyen & AstroPay IPN message processing
 * idx_pending_gateway_txn_id_gateway (gateway_txn_id, gateway) : Amazon message processing