Fundraising tech/notes/Draft:Documentation overhaul

From mediawiki.org
Jump to navigation Jump to search

Draft possible outline[edit]

Below is a proposal for topics and subtopics for Fundraising Tech documentation. Topics and subtopics have codes, which are linked to brainstormed topics and the review of existing documentation, below.

In terms of on-wiki layout, subtopics below could be included in the same page as a general topic, or could have separate pages, depending on their length and complexity.

This outline should be seen as a work in progress. Some topics have been filled out in more detail than others, and it seems certain that some will be expanded or re-organized as we go.

The general logic here is:

  • Main starting point, with general overviews and links to more specific overviews and technical details [X]
  • Explanations of all the different systems involved in donation and donor processes, with technical details that are intrinsic to each system/area [A], [B], [C], [D]
  • How it all fits together: data, data flows and processes that involve multiple components [E]
  • People processes and dev tools [F]

Detailed outline:

  • Introductory overviews and links [X]
    • Introduction: General explanation of WMF donation systems [X1]
    • Donor-facing endpoints: Payments, Donate Wiki, Thank-you Wiki, E-mail preference center, Wikis on main cluster (sidebar links/CentralNotice), Privatebin [A] (same as below)
    • Internal-user-facing endpoints: Civicrm, Superset [B] (same as below)
    • Externally managed services/providers: Acoustic, PSPs, Minfraud, Maxmind geolocation, currency rates provider, Google Docs spreadsheets, Zendesk [C] (same as below)
    • Cluster layout, codebases, repositories, deploy and upgrade processes, internal services [D] (same as below)
    • Data: flows throughout the stack, database schemas, identifiers numbers, details of PSP interactions, import/export of data, UML sequence and interaction diagrams [E] (same as below)
    • Team processes, roles, contact links, code freeze, code review, campaign cadences, bug triage, dev tools, onboarding [F] (same as below)
  • Donor-facing endpoints [A]
    • Payments/Donation Interface [A1]
      • Overview of DI functioning on Payments (note: details of data flows and PSP integrations go under [E])
      • (If still pertinent: soon-to-be-removed DI bits used as a library)
      • Doc or link to doc of globals/config variables
      • YAML-based config
      • Form translations
      • Form variants
      • Unit tests
      • Separation of concerns with SmashPig
    • Donate wiki [A2]
    • Foundation wiki, Foundation site [A3]
    • E-mail preference center [A4]
    • Main cluster wikis, portal [A5]
    • Privatebin [A6]
    • Public donation statistics (frdata.wikimedia.org) [A7]
    • E-mail sends, how e-mail translations work [A8]
    • Wikipedia mobile apps [A9]
  • Internal-user-facing endpoints [B]
    • Civicrm [B1] (note: data processes, such as database schema, sample queries, import/export details and queue message formats go under [E])
      • Overview of uses
      • Extensions, upstream interactions
      • Config
      • Reporting
      • Unit tests
      • Separation of concerns with SmashPig
    • Superset [B2]
  • Externally managed services/providers: Acoustic, PSPs, Minfraud, Maxmind geolocation, currency rates provider, Google Docs spreadsheets, Zendesk, external platforms (Facebook, Google Search) where we run ads soliciting donations [C] (technical details of the integration of these services with our systems and data flows would go in [A], [B], [D] or [E], depending; this section would contain overviews of these services, as well as contractual aspects, processes not on the Fundraising cluster, and links to technical details in other sections)
  • Cluster layout, codebases, repositories, deploy and upgrade processes, internal services [D]
    • Internal services [D1]
      • Redis
      • Process-control
        • How to find existing jobs
        • Where logs go
      • Database servers
      • Database backups
      • Log management
      • IPN listeners
      • Failmail alerts (explanation of how failmails work; tips for debugging based on failmail content go under [E9])
      • Icinga, nagios, cluster monitoring
      • Grafana
      • Stats pipelines (jobs, files, logs, tools; stats data flow, database schema and queries go under [E10])
      • SMTP for sending mail
    • Codebases (DI, Civi extensions, SmashPig), frameworks checked into version control, (Civi, Mediawiki, Drupal, others), upstream relationships (details in DonationInterface and Civicrm sections), git repositories [D2]
    • Clusters, datacenters, servers, deploy processes, dependency management on production, production puppet, ssh access, server permissions, code upgrade procedures (such as for Mediawiki, CiviCRM, or Drupal), staging environments/testing code on the cluster, running one-off jobs and scripts [D3]
  • Data and multi-component processes: data flows throughout the stack, database schemas, identifier numbers, details of PSP interactions, import/export of data, UML sequence and interaction diagrams [E]
    • PSP Integrations [E1]
      • Currently active integrations
        • Details of how they work
        • Links to payment consoles
      • List of deprecated/removed integrations (mainly for the sake of knowing which code is not important/archiving old doc)
    • Database schemas, typical queries, triggers, meanings of identifiers and other fields [E2]
    • Queues [E3]
      • Existing queues
      • Types of messages
      • Code that produces and consumes
      • Sequence generator
    • Silverpop export [E4]
    • Recurring data flows [E5]
    • Audit jobs [E6]
    • Pipeline and data flow from banners/e-mail links/landing pages through Donate Wiki, Payments [E7]
    • Fraud filtering/review of possibly fraudulent charges [E8]
    • General troubleshooting tips and suggested queries related to PSP processes, queues, logs, failmails and databases [E9]
    • Data flow for stats generation [E10]
    • Overview of steps for adding a new gateway [E11]
  • Team processes, task management, dev tools [F]
    • Team processes, roles, Phabricator, bug triage, code review, sprints, agile, task estimation, definitions of done, code freeze cycles, contact links, onboarding [F1]
    • Advancement processes: Campaigns cycle, procedures for testing new gateways/features, A/B tests overview, donor services, Asana, Zendesk [F2]
    • Developer tools: fundraising-dev, Gerrit, Github, Translate Wiki, local setup for ssh cluster access, CI, running tests locally [F3]
    • Emergency procedures [F4]

Brainstorming: topics to include[edit]

  • FR-Tech source repositories and codebases, what they're used for, where they're deployed, how they're used, links to related documentation pages , links to upstream repos (if applicable) [D2]
  • List of public and non-public sites involved in FR systems, and which codebases are installed on each of them, and which servers host them. [A] [B]


  • Composer: How we run it locally , on production and in CI, and why, and troubleshooting tips [D3]
  • Process-control [D1]
  • Deployment: Steps for all service in the FR stack and config changes [D3]
  • CI: how we use it, where the config files are [F3]
  • Outline of end-to-end donation flows, including code paths [E]
    • Also new, more formal diagram(s) with overview of flows
  • Logs [D1]
  • Versions and config options used for PHP, Python, other basic tools used on production [D]


  • Civicrm: Procedures for updating repos from upstream (civicrm and buildkit) [B1]
  • Civicrm: Triggers: What they're for, how to update, how they work for adding fields, how to test, how they're deployed on production and locally [E2]
  • Acoustic/silverpop export: how it works, how to add fields and backfill, which database export runs (staging?) on and why. [E4]
  • Recurring: general flow, how retries are handled [E5]
  • Civi extensions/Drupal modules used, maintained, vs. added just as dependencies, and their relationship to upstream [B1]


  • Integrations: Adyen Checkout: Explanation of flow for different payment options (cc, cc with 3DS, rtbt), including pointers to codebases and methods responsible for each step, and UML interaction diagrams [E1]
    • It would be nice to have at least just an up to date basic overview of all the integrations with the same topics
    • Or the opposite and have our flow pieces and then describe each of the payment processors
  • DonationInterface: Overview of main classes and methods and their functions in the system [A1]
  • LandingCheck extension: where it's installed, how FR systems use it, link to documentation from other FR-Tech documentation. [A2]
  • Same idea as above with the other ones we have but don't touch often eg cldr, FundraisingEmailUnsubscribe [A1]
  • DonationInterface: Config: Short summary of global config variables and per-processor config mechanism on wiki, with link to README for details; complete doc of all config variables in README, explanation of config in yml files in config folders in source code, explanation of which types of config should go in which locations. [A1]
  • Pipeline for selection of country, payment processor, method, recurring, and other variants, from banners or links in e-mails to Payments site. [E7]
  • Permissions, use, content layout of wikis: Foundation wiki, Thank you wiki. [A2] [A3]
  • Meaning and usage of different types of ids and codes throughout the pipeline and across payment processors. [E2]



  • Phabricator and task creation, triage process [F1]
  • Sprint, agile, scrum and other development methodology (how Phabricator boards are used for that) [F1]


  • Fundraising campaigns cycle, code freeze, yearly cadence [F2]
  • Tools used outside FR-Tech: Asana, Zendesk [F2]

Review of existing documentation[edit]

wikitech.wikimedia.org[edit]

See:

Note: This table omits pages in Fundraising categories or subcategories that are already in the Obsolote namespace.

Wikitech page Description Useful topic Outdated or obsolete Topic code(s)
Fundraising Old, outdated starting point for FR-Tech documentation, with a wide variety of topics Y (but should be broken up) Y X1, F4, F1, E1, E3, B1, A1, D1, A7, D3
Fundraising.wikimedia.org Explanation of old, decomissionned site at fundraising.wikimedia.org N Y -
Fundraising/Bug Triaging Fundraising bug methodology Y Y F
Fundraising/CiviCRM Civicrm site setup and maintenance Y Y B1, D3, D1, D2
Fundraising/CiviCRM Reporting How to run and customize Civicrm reports Y Y B1
Fundraising/ContributionTracking Old ContributionTracking extension N Y -
Fundraising/DonationForm Proposal for generic donation form logic N Y -
Fundraising/Normalized donation messages Format of donation queue messages Y Y E3
Fundraising/Queue messages (Redirect to above) N/A N/A -
Fundraising/Queue wrangling Instructions for recreating queue messages from logs Y Y E1, E9
Fundraising/RFC/Abstract CRM import Proposal for abstract Civi import extension N Y -
Fundraising/RFC/Direct Mail Appeal campaign management Proposal for a system for direct mail appeals for Major Gifts N Y -
Fundraising/Syslog Settings Suggested local dev setup for syslog Y Y D1
Fundraising/Translation (redirects to Fundraising tech/Translation deployments on mediawiki.org) N/A N/A -
Fundraising/analytics (empty) Y N/A -
Fundraising/crm (redirects to a list of pages under Fundraising/tech/crm) Y N/A -
Fundraising/tech Incomplete overview of FR-Tech projects Y Y X
Fundraising/tech/Alerts Mostly empty, with just a "Self-recurring checks" heading and a snippet of an IRC conversation pasted in Y Y D1
Fundraising/tech/Apache httpd Suggestions for local dev setup for apache config N Y -
Fundraising/tech/Banner impressions Outdated details and tips on banner impressions pipeline to pgehres Y Y D1
Fundraising/tech/Coinbase Working document with explanation and questions about Coinbase (for bitcoin donations) Y Y E1
Fundraising/tech/Currency conversion sample XML snippet with currency conversion rates, without any other context N Y -
Fundraising/tech/Installing Drupal and CiviCRM Instructions for manual installation of Civicrm N Y -
Fundraising/tech/Spikes Old notes on FR-Tech spikes in progress N Y -
Fundraising/tech/Spikes/Fundraiser 2012 412 Notes for implementing an IP velocity filter N Y -
Fundraising/tech/Training (redirects to Fundraising tech/Training on mediawiki.org) N/A N/A -
Fundraising/tech/Transformer Old draft specification for a field transformer system. N Y -
Fundraising/tech/access Instructions for requesting production shell access on the Fundraising cluster Y Y F3
Fundraising/tech/crm (redirects to a list of pages under Fundraising/tech/crm) Y N/A
Fundraising/tech/crm/Add a New Gateway Old, very general notes on how to create a new payment gateway. Y Y E11
Fundraising/tech/crm/Quirks Old tips for local dev setup N Y -
Fundraising/tech/crm/Running tests Old notes on running Civi tests locally Y Y F3
Fundraising/tech/crm/Upgrade CiviCRM Instructions for upgrading Civicrm Y Y D3
Fundraising/tech/crm/Upgrade Drupal Instructions for upgrading Drupal for Civicrm Y Y D3
Fundraising/tech/deployments Links to obsolete deployment instructions Y Y D3
Fundraising/tech/deployments/minfraud upgrade to 1.3 Old notes on minfraud uprade Y Y D3
Fundraising/tech/deployments/recurring globalcollect Old notes for a GlobalCollect deploy N Y D3
Fundraising/tech/git Git and Gerrit instructions N Y F3
Fundraising/tech/payments1004 Instructions for using payments1004 to test code on the production cluster Y ? D3
Fundraising/tech/ssh config (points to new doc on Collab) Y Y -
Fundraising Analytics Old notes on FR analytics setup Y Y E2
Fundraising Analytics/Impression Stats Old notes on banner impressions data setup Y Y D1
Fundraising Auditing Old explanation of audit code Y Y E6, D3
Fundraising Logging Old notes on banner impressions data processing Y Y D1
Fundraising Monitoring Old notes and wishlist about monitoring services on the Fundraising cluster Y Y D1
Fundraising Tracking Old notes on data collection for Fundraising analytics Y Y E1, E2, E10
Fundraising Tracking/TrackingJS Old code for for data collection, without context N Y -
ActiveMQ Old doc on ActiveMQ N Y -
CentralNotice/Optimizing banner loading Old ideas for how to make banners load better N Y -
CiviCRM upgrade 4.2.x Old checklist and notes for Civi upgrade N Y -
CiviCRM upgrade 4.6 Old checklist and notes for Civi upgrade N Y -
civicrm-dev.wikimedia.org Old doc on civicrm-dev server N Y -
civicrm.wikimedia.org Old doc on civicrm install Y Y B1, D1, D3
db1008 Old doc on db1008 server N Y -
db1025 Old doc on db1024 server N Y -
donate.wikimedia.org Old doc on donate.wikimedia.org Y Y A2
donate.wikipedia.org Old doc on donate.wikipedia.org Y Y A2
erzurumi Old doc on ActiveMQ host N Y -
frdata.wikimedia.org Old doc on frdata server Y Y A7
Geolocation Old doc on geolocation in Varnish N Y -
hume Old doc on decommissioned hume server N Y -
Incident_documentation/20111127-Reddit Report on incident from 2011 N ? -
loudon Old doc stub on decommissioned loudon server N Y -
Payments deployment Stub with link to Office Wiki Y Y -
payments.wikimedia.org Stub with link to Office Wiki Y Y -
Paypal Old doc on Paypal Web Checkout integration Y Y E1
Sender Policy Framework Notes on e-mail spam identification Y Y D1
silicon Old doc on silicon server N Y -
Civicrm tables Old, incomplete doc on Civi schema Y Y E2
CentralNotice Old, icomplete doc on WMF's CentralNotice install Y Y A5, D3
Image: DonationPipeline_201302.png Informal, creative overview of donation flow Y Y E
Image: Landing_page_to_activemq.jpg Old, informal overview of queue processing Y Y E, E3
Image: Activemq_to_civicrm.jpg Old, informal overview of queue processing Y Y E, E3
Image: Fundraising_Tracking.jpg Old, informal overview of queue processing Y Y E, E3
Image: Fundraising_Tracking_Tags.jpg Old, informal overview of flow from banner to landing page to payment form Y Y E7

mediawiki.org[edit]

See:

Wikitech page Description Useful topic Outdated or obsolete
Fundraising tech General overview of FR-Tech systems and team Y Y
Fundraising tech/Adyen Overview of old Adyen integration and notes on testing Apple Pay Y Y
Fundraising tech/Amazon Pay Notes on Amazon Pay integration Y ?
Fundraising tech/Chat Namespace for FR-Tech on-wiki discussion N Y
Fundraising tech/Chat/ActiveMQ Outdate notes on replacing ActiveMQ N Y
Fundraising tech/Chat/GatewayFormChooser Notes on GatewayFormChooser Y Y
Fundraising tech/Chat/Ingenico Connect Notes on migration to Ingenico Connect Y Y
Fundraising tech/Chat/Scap3 Possible Scap3 migration notes N Y
Fundraising tech/Chores Ongoing chores for FR-Tech engineers Y N
Fundraising tech/CiviCRM Many details on CiviCRM, extensions, database fields and e-mail export Y ?
Fundraising tech/CiviProxy Docker Setting up CiviProxy on Docker Y Y
Fundraising tech/Code quality Links to (not often used?) automatic code quality assessment tool Y n/a
Fundraising tech/Components Overviews of some components used in FR-Tech stack Y ?
Fundraising tech/Continuous Integration Placeholder for explanation of CI used by FR-Tech Y n/a
Fundraising tech/Contribution tracking Notes and diagrams for contribution tracking upgrade Y Y
Fundraising tech/Dashboard Link to Fundraising/Engineering/Monitoring on Collab. ? n/a
Fundraising tech/Database cheat sheet Status/magic number reference for contribution and contribution_recur in CiviCrm Y ?
Fundraising tech/Database schema Overviews and some details of DB schemas for pgehres, drupal, civicrm and fredge databases Y ?
Fundraising tech/Deadlines Past and upcoming long-term deadlines for EOL of infrastructure code used in our stack Y ?
Fundraising tech/Definition of Done General criteria for declaring a task "done". Y N
Fundraising tech/Deployment Deployment instructions for code on FR cluster Y ?
Fundraising tech/End of year receipt Explanation of End-of-year e-mails for recurring donors, and how to send Y ?
Fundraising tech/Essential systems Checklist of most important stack components to check ahead of campaigns Y Y
Fundraising tech/Estimation Cheat Sheet (not understood) ? ?
Fundraising tech/Exchange rates Incomplete explanation of drupal exchange rates module Y ?
Fundraising tech/FR-tech Meetings Explanation of regular FR-tech meetings Y N
Fundraising tech/Failmail zoo Description of various common failmails and what to do when they arrive Y N
Fundraising tech/Form variants Explanation of how variant=??? works in DonationInterface Y N
Fundraising tech/Free Software Citizenship Brief overview of oals and practices related to making our code useful to others outside the WMF Y ?
Fundraising tech/Glossary Some terms used with the FR-Tech stack ? N
Fundraising tech/Hive query notes Placeholder for doing Hive queries N n/a
Fundraising tech/IPN listener IPN listener overview and brief explanation of old Adyen listeners Y Y
Fundraising tech/Importing donation files Brief explanation of importing spreadsheets to CiviCRM using offline2civicrm Y ?
Fundraising tech/Ingenico Connect Partial explanation of Ingenico Connect integration. Y N
Fundraising tech/Installation Overview of and links to some unrelated stuff N Y
Fundraising tech/Language variants Rough notes for possible work on i18n in various parts of the FR-Tech stack N Y
Fundraising tech/Message queues Overview and details of queues in use in FR-Tech stack Y ?
Fundraising tech/Message queues/Overhaul Description of 2016 project to move from ActiveMQ to configurable queue backend (Redis) N Y
Fundraising tech/Monthly convert Brief explanation of month convert config for Payments Y ?
Fundraising tech/New integration manual Suggested processes for creating new integrations Y ?
Fundraising tech/Onboarding Notes on accounts and privileges to set up for new FR-Tech team members Y ?
Fundraising tech/Outcome 1 FY 2018-19 Old notes on goals for FY2018-19 N Y
Fundraising tech/Payment methods Old notes for possible Payments refactor ? ?
Fundraising tech/Paymentswiki upgrade Instructions for upgrading Mediawiki version used on Payments Y ?
Fundraising tech/Pending queue consumers Supporting page for 2016 queue migration project N Y
Fundraising tech/Pre-campaign checklist Checklist for things to verify ahead of a big campaign, especially the year-end fundraiser Y Y
Fundraising tech/Process-control Overview of process-control and instructions for configuring jobs Y N
Fundraising tech/Queue message formats Details of properties in queue messages (related: Fundraising_tech/Message_queues) Y ?
Fundraising tech/Queue testing Instructions on testing queues locally (vagrant-specific) Y Y
Fundraising tech/Recurring donations Incomplete explanation of how recurring works across different processors Y Y
Fundraising tech/Refactor Notes for possible refactoring of various parts of the FR-Tech stack Y ?
Fundraising tech/Roadmap Old roadmap for FY2018-19 N Y
Fundraising tech/Roadmap/Campaign fallback Sub-national targeting Draft RFP for contractor work on CentralNotice N Y
Fundraising tech/Roadmap/Live Banner Preview Draft RFP for contractor work on CentralNotice N Y
Fundraising tech/Roadmap/Outcome 2 FY 2018-19 Old notes for FY2018-19 outcomes N Y
Fundraising tech/Roadmap/Outcome 3 FY 2018-19 Old notes for FY2018-19 outcomes N Y
Fundraising tech/Roadmap/Program 7: Payment processor investigation and long-term strategy Placeholder for old long-term strategy notes N Y
Fundraising tech/Roadmap/Program 8: Donor retention Placeholder for old notes N Y
Fundraising tech/Roles and Responsibilities Notes on roles and responsibilities in FR-Tech Y Y
Fundraising tech/SmashPig Overview and notes on possible directions and issues for SmashPig Y Y
Fundraising tech/Test links Links for testing different processors on production Y Y
Fundraising tech/Testing Notes for locally testing various processors and bits of the stack Y Y
Fundraising tech/Training Table of what parts of the stack different team members are familiar with Y Y
Fundraising tech/Transaction IDs Meaning and usage of IDs in different parts of the FR-Tech stack Y ?
Fundraising tech/Translation deployments Explanation of how translations are deployed on Payments and Donate wiki Y Y
Fundraising tech/Update Thank You Emails Explanation of how thank-you e-mails are updated Y ?
Fundraising tech/WR1 auditor (redirects to Fundraising tech/audit parser) N Y
Fundraising tech/audit parser Documents old Ingenico audit parser format. File format and standalone drush job are both gone. Also describes PayPal audit parsing. Y Y
Fundraising tech/donation pipeline setup/settings Old recommended settings to manually copy for local setup N Y
Fundraising tech/notes/Draft:Documentation overhaul This page of notes N N
Fundraising tech/queries Example query for how to check for multiple donations from one donor Y ?
Fundraising tech/status Unused stub for tracking something in 2014 N Y
Fundraising tech/tools Explanations of two things in the tools repo: e-mail export, landing page impression counter Y Y
Fundraising tech/vagrant fundraising vagrant is dead, long live docker! N Y
Extension:FundraiserLandingPage Explanation and general documentation of FundraiserLandingPage Y ?
Extension:CentralNotice General explanation and installation instructions for CentralNotice Y Y
Extension:LandingCheck Explanation of LandingCheck Y ?
Git/Conversion/Fundraising Old notes on Fundraising's move from svn to git N Y
User:MarkTraceur/FR-tech setup notes Old notes on setting up DI locally for testing N Y
Extension:Popups/Fundraising test 1 Summary of test of popups and fundraising campaign interactions N Y
Requests for comment/CentralNotice backend improvements Old proposal for changing CentralNotice use of MessageCache N Y
Requests for comment/CentralNotice Caching Overhaul - Frontend Proxy Old proposal for changing how CentralNotice works N Y
Image: Mediawiki.extensions.donationinteface.jpg Code layout of DonationInterface N Y
Image: Wikimedia.fundraising-misc.queue handler.jpg Flow chart of GlobalCollect integration Y Y

Resources[edit]

Documentation

Documentation/Style_guide

Writing Documentation as a Community