Fundraising tech

This document should explain or link to everything needed for fundraising tech development.

About Fundraising Tech
Fundraising Tech is responsible for the security, stability, and development of the Wikimedia Foundation’s online donation systems. Millions of relatively small donations make up the majority of the Wikimedia Foundation’s operating budget every year. The donation systems created and maintained by Fundraising Tech were built specifically to make the small donor model a reality, across as many localities as possible to further ensure the continued independence of our mission.

We do not write banners or run tests, we support the people and software that run them.

International payment processing

 * The majority of donations run through integrations with 6 payment processors. We are also integrated with several other processors for accessibility, location and fundraising event uses. These integrations enable us to support online fundraising campaigns in approximately 30 countries each year.

Retention of donor/donation data

 * We maintain an internal donor database and CRM (CiviCRM). This data is crucial in measuring the impact of our fundraising campaigns, developing our global strategy, and maintaining relationships with past donors.

Tools to create, manage, and deliver online fundraising campaigns

 * We are the primary developers and maintainers of the CentralNotice system, which delivers banner notices to the wikipedias and the sister projects. Historically, banners on the wikis have been the primary method by which we entice users to donate.
 * Note: We partner with the Advancement department, Major Gifts, Banner and Email teams, operations and Donor services. Fundraising tech provides stable platforms and tools to Advancement, and they create specific fundraising campaigns and other donor-facing messaging and content.

Fundraising Tech
Technical work supporting the Wikimedia Foundation's fundraising efforts.
 * Fundraising Tech Director: Erika Bjune
 * Fundraising Tech Engineering Manager: Maggie Epps
 * Fundraising Tech Lead: Elliott Eggleston
 * Fundraising Tech Product Manager: David Strine
 * Phabricator task board: #fundraising-backlog

Goals
Fiscal Year 2018-2019


 * Goals by quarter

Roadmap
fr-tech's detailed Roadmap

Rhythm and code freeze
We have a special yearly window of not deploying major changes to some critical systems starting in the fall, in preparation for the Big English drive. Game on again in mid-January, assuming we haven't emitted a puff of smoke in early December.

This code freeze gives our development an annual cadence, with each season seeing similar types and intensities of work, year-over-year. Any long-running FR-tech project should take this into account.

Documents

 * Glossary: Fundraising tech/Glossary
 * Regular meetings and their agendas Fundraising tech/FR-tech Meetings
 * Definition of Done: Fundraising tech/Definition of Done
 * Onboarding new team members: Fundraising tech/Onboarding
 * Team training matrix: Fundraising tech/Training
 * Roles/Responsibilities: Fundraising tech/Roles and Responsibilities
 * User requirements:
 * Specifications:
 * Software design document: Fundraising tech/Components
 * Essential systems: Fundraising tech/Essential systems
 * New integration manual: Fundraising tech/New integration manual
 * Test plan:
 * Documentation plan:
 * User interface design docs:
 * Schedule:
 * Task management: Phabricator "Fundraising Backlog" tag. More information.
 * Release management plan: Fundraising tech/Deployment
 * Communications plan:
 * Quarterly review documentation: April 2015
 * Payment processor documentation
 * Adyen
 * Amazon
 * AstroPay
 * Ingenico and see filesrv (
 * Developer manual: WebCollect_technical_guide_2013_Q4.pdf
 * Workflows (needs updating): smb://filesrv1/fundraising/Tech/Ingenico/Old/Tech%20specs%20and%20guides%20from%20PaymentProcessing )
 * Note: In the codebase, Ingenico can refer to the older GlobalCollect--Ingenico's former name--integration and the new integration with their updated API
 * PayPal / PayPal Express Checkout

Installing the software stack
Much of our toolchain is provided by MediaWiki-Vagrant, please follow setup instructions at the main MediaWiki-Vagrant page and when you are ready to select a vagrant role, use 'fundraising' and then vagrant provision (must be done after you have successfully executed vagrant up):

vagrant enable-role fundraising vagrant provision

Also note that the fundraising role assumes that the vagrant repo is in /vagrant and that drush is in /usr/local/bin/drush.

Assuming your box builds without error, visit http://payments.wiki.local.wmftest.net:8080/ and the wiki's main page will have links to help you get started. Please note that the wmftests.net subdomain is actually a wildcard DNS that resolves to 127.0.0.1, so if you are running vagrant on a different host than you're browsing from, or if you're offline, you'll need to add hosts entries yourself.

This role also installs CiviCRM, at http://crm.local.wmftest.net:8080/. The admin login can be found in the puppet settings.

To use CentralNotice, navigate to http://dev.wiki.local.wmftest.net:8080/.

If you wish to forsake Vagrant and try to do it all the hard way, see Fundraising Tech/Donation Pipeline Setup

Running PHPUnit tests on vagrant (mediawiki-fr)
The fundraising role checks out a separate branch of mediawiki core into /vagrant/mediawiki-fr. From the vagrant directory, do:

vagrant ssh cd /vagrant/mediawiki-fr/ git submodule foreach git checkout master composer install export MW_INSTALL_PATH=/vagrant/mediawiki-fr php tests/phpunit/phpunit.php --wiki paymentswiki --group DonationInterface

Running PHPUnit tests on vagrant (CRM)
From within the vagrant directory: vagrant ssh cd /vagrant/srv/org.wikimedia.civicrm/ ./vendor/bin/phpunit

Testing queue operations under vagrant
Please see Fundraising tech/Queue testing.

Communications
You can find our engineering team on the #wikimedia-fundraising irc channel. Include "fr-tech" in your message to make sure we see it.

Fundraising Tech Systems Dashboard
The Fundraising Tech dashboard is located here https://grafana.wikimedia.org/dashboard/db/fundraising-overview Prometheus is used to record metric data which is then displayed in Grafana.

Current items being monitored include:


 * Server load - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=1&fullscreen
 * Nginx requests - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=6&fullscreen
 * Syslog message - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=16&fullscreen
 * Syslog messages lost to rate-limit - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=17&fullscreen
 * MySQL replication lag - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=2&fullscreen
 * MySQL read binlog position - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=3&fullscreen
 * MySQL read binlog position - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=4&fullscreen
 * MySQL queries - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=19&fullscreen
 * Donations queued to civicrm - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=15&fullscreen
 * Donations queue - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=9&fullscreen
 * Jobs queue - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=10&fullscreen
 * Other queues - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=11&fullscreen
 * Eqiad payments memcache current items - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=5&fullscreen
 * Payments Nginx responses - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=8&fullscreen
 * Nginx avg request time - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=7&fullscreen
 * Nginx avg bytes sent - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=13&fullscreen
 * Mail queue - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=14&fullscreen
 * Civi mail - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=18&fullscreen
 * Process-control jobs - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=20&fullscreen
 * Queue delay - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=21&fullscreen
 * Donation to Civi Processing Rate - https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&panelId=22&fullscreen