Fundraising tech/CiviCRM

We use Open Source CRM CiviCRM to manage our donor data.

Extensions
We use both CiviCRM extensions and custom drupal modules to manage our workflows and donor data

CiviCRM extensions (checked in to drupal/sites/default/civicrm/extensions/ in the CRM repo):
 * civi-data-translate Allows more extensive localization of CiviCRM messages than is possible in Civi core - this is being phased out in favour of the functionality having been moved into core
 * Extended Mailing Stats
 * Email Amender Corrects common errors in domain names in email addresses
 * Deduper Combines multiple contact records for the same person
 * Matching Gifts Retrieves lists of employer matching gift policies with a third party provider. Also exports the list of employers and IDs so that we can use it in front-end forms
 * Monolog Use monolog streams to output logging to e.g. syslog rather than the files written by Civi core.
 * Export Permission Allows us to disable exporting search results for certain user roles
 * CiviToken Provides extra tokens for anything Civi renders from templates
 * Extended Report More report templates
 * Angular Profiles - this was required for Contact layout editor - it is likely we no longer need it
 * Contact Layout Editor GUI for editing contact summary screen
 * Shoreditch theme based on Bootstrap
 * Tutorial
 * Contact Editor Allows changing contact types
 * Data Checks Detects and batch-fixes contact data inconsistencies such as no primary location.
 * Forgetme UI for complying with privacy data deletion requests
 * Wikimedia Geocoder Privacy respecting but imprecise address geocoder that uses a local zip code database
 * Omnimail Integrates with bulk email providers (Acoustic is most fully implemented). Attaches CiviCRM activities to contacts based on email sends from bulk mail providers
 * Relationship Block Shows relationships in contact summary
 * RIP Sets the "No Bulk E-mail (User opt out)" flag on deceased people
 * SmashPig Uses the SmashPig library to make nightly recurring charges for monthly donors
 * Wikimedia System Tools debugging helpers
 * The The Normalizes use of articles in organization names
 * TargetSmart Helper for adding contact data from TargetSmart
 * Unsubscribe Email Dedicated screen to facilitate opting contacts out by email address
 * WMF Fraud reports Helps the payments team review suspicious transactions when deciding whether to capture, reject, or refund.
 * Replay-On-Write Allows using CiviCRM with a Primary+Replica database setup
 * Activity Assignee Settings Limit who can be assigned activities
 * WMF CiviCRM Catchall for our site-specific code which has migrated out of Drupal modules
 * WMF Thank You Uses our twig templates to automatically thank donors for donations. Fast and multilingual, but uses a different template language from the rest of Civi and has no editing UI. Should eventually evolve to use templates from civi-data-translate

Drupal modules (Checked in at drupal/sites/all/modules/ )

See outdated list at https://wikitech.wikimedia.org/wiki/Fundraising#Drupal_modules

We use the civix utility to generate and update Civi extension boilerplate.

Email sending - Acoustic interaction
We send bulk email via Acoustic based on nightly uploads of our contact dataset.

The exports work by uploading 4 separate files to Acoustic which are imported into Acoustic using mappings that Katie H looks after. The files are

1) DatabaseUpdate-*.csv - this file holds detailed information about contacts who have recently been updated. The update is cumulative - ie. we add/update contacts but don't delete them. We use the field civicrm_contact.modified_date to determine which contacts to include in the update.

2) Unsubscribes-*.csv

3) Optout-*.csv

4) MatchingGifts-*.csv

. The nightly uploads include a calculation of which mails are opted_out based on if any of the following fields indicate opt_out:

civicrm_contact.is_opt_out

do_not_email

do_not_solicit

email.on_hold

Contacts can become opted out by the following ways


 * Filling in a donation form with an opt-in option and not selecting it - the contact record is set to opt_in = 0 on import & on merge the latest is kept
 * An action we import from Silverpop causing them to put on hold - see Fundraising tech/CiviCRM
 * An action we import from Silverpop causing them to be unsubscribed - see Fundraising tech/CiviCRM
 * Donor services manually unsubscribing them based on donor feedback via the Unsubscribe form (our url is civicrm/a/#/email/unsubscribe)
 * Major gifts marking them do_not_solicit
 * The donor accessing our unsubscribe form & their desire not to be subscribed being processed through our queue.

Data transfer between CiviCRM and Acoustic (aka Watson Campaign Automation Silverpop, IBM )
Mailings are sent out from the external tool Watson Campaign Automation - formerly Silverpop.

We retrieve data back into CiviCRM from WCA in the following forms

TODO: document technical details of how this is synced (Omnimail library and CiviCRM extension, which jobs do what)

Mailing data
WCA only stores data for 450 days. We retrieve this data for longer storage.

We store the following data in the civicrm_mailing table: And we store additional fields in the table civicrm_mailing_stats - these fields are generally aggregate information as calculated by WCA

This table is provided by the extension Extended Mailing Stats. - it also adds the table civicrm_mailing_stats_performance which we don't use. There is double up between this & the civicrm_mailing table on some fields but that is because the schema was

determined by an external extension writer with slightly different needs.

Remind me later contacts
Contacts who are in Silverpop but who do not have a contact_id from CiviCRM are considered to be 'remind me later contacts' - who signed up via silverpop. We retrieve these contacts and add them to the group in CiviCRM called 'Silverpop imports' (group id is 310 - url is https://civicrm.wikimedia.org/civicrm/group/search?reset=1&force=1&context=smog&gid=310&component_mode=1 ). We retrieve the following information about them

- language

- source

- created date

- country

- email

Recipient data
We retrieve information about mailing actions (sending, opening etc) for each contact. This information is stored in the civicrm_mailing_provider_data table with the following fields

Matching Gifts
We are building a CiviCRM extension to pull data about corporate matching gift policies from SSBInfo and store it as Organization records with custom fields.


 * Matching gifts provider ID
 * Matching gifts provider info URL (for SSB: https://javamatch.matchinggifts.com/search/companyprofile/wikimedia_iframe/ from json: online_resources.id)
 * Minimum (USD) gift matched
 * Name as specified by matching gifts provider
 * Guide URL
 * Online form URL
 * Matching gifts info last updated date?
 * List of subsidiaries in a text blob (JSON? delimited list? what delimiter?)
 * Flag to suppress companies from the frontend dropdown if they come in on the search but turn out to NOT actually match our donations.