Fundraising tech/Contribution tracking



Contribution tracking, as of early 2020, is the last piece of our payments infrastructure that ties the payments-wiki frontend to the database server in the backend. We want to be able to take the backend db server down for maintenance without needing to shut down the payments frontend. To do this, we will stop creating drupal.contribution_tracking rows directly from the wikis and move to a queue-based insert system.

The sequence generator has an interface and is provided by a configurable class, all defined in SmashPig under the Core\SequenceGenerators namespace. Implementations for Redis and PDO backends are provided. We will use the Redis one in production and the PDO one with SQLite for tests. There is a convenience method Factory::getSequenceGenerator($sequenceName) that instantiates the generator defined in SmashPig's main.yaml under key sequence-generators/$seqenceName. Currently only one is defined, named contribution-tracking.

In DonationInterface, there is a new global $wgDonationInterfaceEnableContributionTrackingQueue which needs to be set to true to enable pulling new IDs from the sequence generator and sending c_t message to the queue. In the CRM codebase, instead of an enabling variable we have a patch that fully switches c_t table writes over to using the c_t sequence generator and queue.

The ContributionTrackingQueueConsumer class consumes messages from the c_t queue and inserts or updates messages in the contribution_tracking table. We should perhaps add a check on updates to ensure that only the expected type of update (adding a contribution ID) is happening. If messages are coming in with ct_ids that are already in the table and have more attributes than just a contribution ID, that could mean the sequence generator is malfunctioning.

Source for diagrams
The diagrams were created with sequencediagram.org

title Current Contribution Tracking (one-time card)

participant DB participant CRM participant Queue participant Payments

group DonationData::normalize Payments->DB:Insert row with non-normalized data DB->Payments:Return autoincremented ID for use in creating order ID Payments->DB:Update row with normalized data end group GatewayAdapter::postProcessDonation Payments->Queue:Donation message with ct_id end group DonationQueueConsumer Queue->CRM:Pop donation message group wmf_civicrm_contribution_message_import CRM->DB:Update c_t row with contribution_id end end

title Queue Contribution Tracking (one-time card)

participant DB participant CRM participant Queue participant SequenceGenerator participant Payments

group DonationData::normalize Payments->SequenceGenerator:get next ID Payments->Queue:C_T message with normalized data end group ContributionTrackingQueueConsumer Queue->CRM:Pop C_T message CRM->DB:Write new row end group GatewayAdapter::postProcessDonation Payments->Queue:Donation message with ct_id end group DonationQueueConsumer Queue->CRM:Pop donation message group wmf_civicrm_contribution_message_import CRM->Queue:C_T message with contribution ID end end group ContributionTrackingQueueConsumer Queue->CRM:Pop C_T message with contrib ID CRM->DB:Update row end