Fundraising tech/Contribution tracking



Contribution tracking, as of early 2020, is the last piece of our payments infrastructure that ties the payments-wiki frontend to the database server in the backend. We want to be able to take the backend db server down for maintenance without needing to shut down the payments frontend. To do this, we will stop creating drupal.contribution_tracking rows directly from the wikis and move to a queue-based insert system.

Sequence Generator
The first step is to stop autoincrementing ids in the database so that updates can happen asynchronously from the front end. To that end, we are using a sequence generator to create non-duplicate unique identifiers.

The sequence generator is provided by a configurable class that implements an interface, all defined in SmashPig under the Core\SequenceGenerators namespace.


 * Implementations for Redis and PDO backends are provided.
 * We will use the Redis one in production
 * We use the PDO one with SQLite for tests.
 * There is a convenience method Factory::getSequenceGenerator($sequenceName) that instantiates the generator defined in SmashPig's main.yaml under key sequence-generators/$seqenceName. Currently only one is defined, named contribution-tracking.

The transition
In DonationInterface, the new global $wgDonationInterfaceEnableContributionTrackingQueue needs to be set to true to enable pulling new IDs from the sequence generator and sending c_t message to the queue. In the CRM codebase, instead of an enabling variable we will deploy a patch that fully switches c_t table writes over to using the c_t sequence generator and queue.

CRM Implementation
The ContributionTrackingQueueConsumer class consumes messages from the c_t queue and inserts or updates messages in the contribution_tracking table. If messages are coming in with ct_ids that are already in the table and have more attributes than just a contribution ID, that could mean the sequence generator is malfunctioning.

Source for diagrams
The diagrams were created with sequencediagram.org

title Current Contribution Tracking (one-time card)

participant DB participant CRM participant Queue participant Payments

group DonationData::normalize Payments->DB:Insert row with non-normalized data DB->Payments:Return autoincremented ID for use in creating order ID Payments->DB:Update row with normalized data end group GatewayAdapter::postProcessDonation Payments->Queue:Donation message with ct_id end group DonationQueueConsumer Queue->CRM:Pop donation message group wmf_civicrm_contribution_message_import CRM->DB:Update c_t row with contribution_id end end

title Queue Contribution Tracking (one-time card)

participant DB participant CRM participant Queue participant SequenceGenerator participant Payments

group DonationData::normalize Payments->SequenceGenerator:get next ID Payments->Queue:C_T message with normalized data end group ContributionTrackingQueueConsumer Queue->CRM:Pop C_T message CRM->DB:Write new row end group GatewayAdapter::postProcessDonation Payments->Queue:Donation message with ct_id end group DonationQueueConsumer Queue->CRM:Pop donation message group wmf_civicrm_contribution_message_import CRM->Queue:C_T message with contribution ID end end group ContributionTrackingQueueConsumer Queue->CRM:Pop C_T message with contrib ID CRM->DB:Update row end