Fundraising tech/Components

Jump to navigation Jump to search

Long-term goals for our components[edit]

  • We have started to push CiviCRM customizations upstream or into CiviCRM extensions.
  • CentralNotice and A/B testing should be librarized and ideally supported by dedicated teams.
  • Banner impressions parsing is being rewritten from scratch.
  • Payments audit scripts should be ported to PHP.
  • DonationInterface payments API integration should be librarized and decoupled from MediaWiki.
  • DonationInterface UI is being ported to a unified Mustache form.

Drupal and CiviCRM[edit]

Software Description Source Documentation
Audit modules (drupal) Compares automatically-downloaded nightly audit files against the CRM database, and logs discrepancies. wikimedia/fundraising/crm
CiviCRM Customer relationship management (CRM) layer for managing donor information. wikimedia/fundraising/crm/civicrm Installation | Upgrade | Testing
Communications Generalized mailing job management and templating. Wraps the sendmail library (currently PHPMailer). wikimedia/fundraising/crm
Contribution Tracking (Redundant) applies the contribution_tracking table schema. This should be done using the MediaWiki extension instead. wikimedia/fundraising/crm Wikitech:Fundraising
Drupal CMS framework our stuff runs on, manages users and permissions, provides a hook-based API. wikimedia/fundraising/crm/drupal Installation | Upgrade
Drush Commandline frontend for Drupal wikimedia/fundraising/drush
Exchange Rates This module handles converting foreign currencies into US Dollars. It consists of two major components. One function handles doing the conversions based on exchange rates stored in the drupal.exchange_rates table. This function is called from the queue2civicrm module when it is processing donations from the queue. Another function, which is activated by a cron job, looks up the exchange rates on the web and updates the exchange_rates table. The exchange rates are currently pulled from, with used as a back-up source. wikimedia/fundraising/crm
Fredge queue consumer wikimedia/fundraising/crm
Metrics Reporter Exposes per-gateway throughput counts and processing time statistics as seen on wikimedia/fundraising/crm
Large Donation Send an email to the Major Gifts team when any online donation exceeds a configured dollar amount threshold. wikimedia/fundraising/crm
Offline to CiviCRM Import check and chargeback spreadsheets into the CRM. wikimedia/fundraising/crm Importing donation files
Donations queue consumer Takes donations added to the queue by IPN listeners or the front-end payments servers and inserts them into CiviCRM wikimedia/fundraising/crm Fundraising tech/Message queues
Recurring queue consumer Same as the donations queue consumer, but for recurring donations. wikimedia/fundraising/crm Fundraising tech/Recurring
Recurring Ingenico Ingenico does not actively recur subscriptions. This module is run daily, and will query the database for recurring donations which should be charged that day. wikimedia/fundraising/crm Fundraising tech/Recurring
Thank You This module sends a thank you message to online donors. There's a job polling for new donations with a null no_thank_you and an empty thankyou_date field. The body of the thank you message is taken from the modules/thank_you/templates directory, and an interface to configure or test mailings can be found in Administration » Configuration » Thank you settings. wikimedia/fundraising/crm Generate new Thank You email templates


Software Description Source Documentation
MediaWiki for donations This is the version we use to accept donations. mediawiki/core in Gerrit (fundraising/* branches)] | Upgrade
Extension:CentralNotice Display and manage banners CentralNotice in Gerrit Extension:CentralNotice
Extension:ContributionTracking Unified, lifecycle tracking for donations. ContributionTracking in Gerrit Extension:ContributionTracking
Extension:DonationInterface This generates forms and handles payment processing. DonationInterface in Gerrit Extension:DonationInterface
Extension:LandingCheck This handles landing pages for banners. LandingCheck in Gerrit Extension:LandingCheck


Software Description Source Documentation
Audit modules (python) Audit modules built on a Python stack. Sadly, deprecated. wikimedia/fundraising/tools
Stats parser Written in Django for some reason, has 3 tasks
  • Loads banner impressions from the sampled beacon files, parsing out banner, campaign, project, country, language, and banner display result (show/hide, hide reason), and rounding off the timestamp to the nearest minute
  • Load landing page impressions from the unsampled donatewiki and foundationwiki logs, selecting certain pages and parsing out the utm_* variables, project, language, and country
  • Count unique email clicks. For landing page impressions with a contact_id,insert utm_source, utm_ampaign, contact_id, and link_id into a table with a unique constraint on those columns, using 'ON DUPLICATE KEY' to discard clicks after the initial one for a given donor and email.

For all the tasks, it filters out certain IPs, UAs, and response codes.

Mustache Recommended templating language for all new components. TODO: MediaWiki+zordius flavored docs
Twig Legacy templating language, used in the CRM Thank You module and some minor mailing stuff.
Minor tools Currently houses: auditing, banner screenshotter, statistical queries (see also "br"), bulk mail wikimedia/fundraising/tools Fundraising_tech/tools
SmashPig Real-time listener framework and nascent payment processing library wikimedia/fundraising/SmashPig
PHPMailer Native PHP SMTP client used by some CRM modules.
Maxmind GeoIP Library we use to resolve IP addresses to a place.
Maxmind Minfraud Fraud solution we use for front-line defense.

Server applications[edit]

Software Description Source Documentation
Apache httpd 2.2 The httpd web server Configuring httpd
MariaDB RDBMS backend for CRM, MediaWiki, and tools. TODO
Redis Data store for transient or high-availability information
rsyslog All persistent logging must be done through syslog.
process-control Schedules fundraising cron jobs and collects output. wikimedia/fundraising/process-control

See also[edit]

A quick pre-campaign checklist.

A selection of essential systems and interactions.