Fundraising tech/Queue message formats

Note: This documentation is being copied and updated bit by bit from wikitech.mediawiki.org but is not yet complete.

This page documents the in-flight format for messages sent and received by fundraising components.

Overview
Incoming donations are sent between subsystems as a single packet of data, in a flat dictionary. We use JSON over the wire, and PHP arrays internally. Subscription creations and modifications, refunds, and unsubscribe requests are also sent using similar techniques, and are documented here as well.

We use message queues to decouple the CRM database from public payments servers, and to not block performance-sensitive tasks.

Legacy format notice: This is the current schema for our messages. Please do not make edits here unless they reflect the existing queue consumer code. Message formats have developed over time and suffer from inconsistency and accumulated cruft, there is a message roadmap with plans to fix this.

Common fields
Some metadata is added to queue messages regardless of producer and consumer.


 * source_name : freeform name of the application which generated this message


 * source_type : category of application.
 * payments : came in direct from donatewiki or a banner (payments.wikimedia.org)
 * listener : real time messages coming in via a listener service created by the payment provider
 * audit : created by the nightly reconciliation job
 * direct : message was generated within our CRM system itself, for example a hand-entered donation or refund.


 * source_host
 * machine name where initial intake interfacing occurred.


 * source_run_id
 * process identifier of the generating code


 * source_version
 * revision level of the originating code


 * source_enqueued_time
 * unix timestamp encoding when the message was first added to a queue. This header is not updated during requeueing or other operations.

Queue: donations
This is the most common type of donation message. They land in the "donations" queue, and are processed by the DonationsQueueConsumer in the queue2civicrm module.


 * gateway
 * Gateway identifier string, e.g. "paypal" or "ingenico".


 * gateway_txn_id
 * Transaction ID string used by the gateway.


 * invoice_id
 * An ID string generated on our side. Usually the contribution_trakcing_id plus a sequence number, separated by a '.' or a '-'


 * contribution_tracking_id :


 * completion_message_id
 * If a message does not contain enough information to import a donation into Civi, this property gives the ID of a message in the pending queue with the rest of the data. Currently used for AstroPay and Amazon IPN messages.


 * log_id
 * Like completion_message_id, but for looking up missing information in payment cluster logs. Used for AstroPay and Amazon audit file messages.


 * date
 * Time donation was received, in unix timestamp seconds since epoch.


 * currency
 * (required) Original currency of transaction. Do not use the deprecated original_currency or original_gross fields, these are too confusing.  We'll introduce "settled_currency", etc., when it becomes necessary to track FOREX across processor accounts.


 * gross
 * (required) Total amount of transaction, in original currency.


 * fee
 * (optional) Fees charged by the payment processor, in original currency. If unspecified, this will be assumed zero, or calculated from gross - net if available.


 * net
 * (optional) Amount after subtracting fees.


 * email
 * Donor's email. If unspecified, we may substitute with "nobody@wikimedia.org" for validation purposes.  This default is stripped out again before storing to the database.


 * first_name


 * middle_name


 * last_name


 * organization_name
 * (optional) If given in place of first/last_name, an Organization contact will be created rather than an Individual.


 * street_address


 * supplemental_address_1


 * city


 * state_province


 * postal_code


 * country
 * Billing or mailing address country&mdash;not necessarily the same as the contribution_tracking country of web origin.


 * payment_method
 * Primary payment method, e.g. "cc". TODO: enumerate.


 * payment_submethod
 * Payment method details, e.g. "visa". TODO


 * gateway_status
 * Raw gateway status code, if available.


 * gift_source : Maps to 'Campaign' custom field
 * restrictions : Maps to 'Fund' custom field'
 * import_batch_number
 * Conceptually broken, this should be renamed. It's a number internal to AZ Lockbox.


 * check_number :


 * contact_type
 * (optional) "Organization" if you want to create an Org record in Civi. This is normally implicit, according to which name fields were filled out. Defaults to "Individual".


 * anonymous
 * (legacy, if sent would just update contribution_tracking)


 * optout
 * (legacy, if sent would just update contribution_tracking)


 * opt_in
 * (optional) "0" or "1", saved to the Opt In custom field in the Communications group. When exporting to Silverpop, a "0" in opt_in will land the donor on the unsubscribe list. When not present or null, indicates that the donor has not been shown the opt_in choice.


 * contact_source
 * Maps to contact_source field in civicrm_contact, defaults to "online donation".


 * notes
 * Text blob will be stored as a CiviCRM note, associated with the contact. Can't find anything that sends this.


 * language
 * Our best guess at the donor's preferred contact language.


 * recurring
 * "0" for one-time donations, "1" for recurring donations


 * recurring_payment_token
 * when sent with a recurring donation, this is stored in the civicrm_payment_token field to be used to charge future installments


 * utm_campaign
 * Mapped into the direct_mail_appeal custom field


 * utm_medium
 * used to update or create contribution tracking


 * utm_source
 * used to update or create contribution tracking