Fundraising tech/notes/Draft:Documentation overhaul

What's new
Here's an updated proposal for topics and subtopcis for FR-Tech documentation. (The previous version is retained below.) The main changes are:


 * Two new top-level section have been added: Libraries and internal systems (under which we'd put the main page about SmashPig) and Dev tools (previously lumped together with Team processes.)
 * Subtopics within each top-level section have been re-ordered.
 * More attention has been paid to possible short titles for topics and subtopics, so these may hopefully be used as prefixes for wiki page names.

Topic codes in the second draft use lower-case letters to distinguish them from the codes for the first draft.

General considerations
Each top-level topic (External endpoints, Internal endpoints, etc.) would have its own summary page, with links to pages for subtopics as needed. (In some cases, however, the subtopics might not need there own page, and in other cases, the subtopics might need to be broken down into pages with subsubtopics.)

In the outline below, details of some large subtopics haven't been filled out (for example, Fundraising/Internal endpoints/CiviCRM and Fundraising/External endpoints/Payments).

When appropriate, we might standardize subsubtopic organization. (For example, Roadmap or Running tests locally subsubtopics might be appropriate in several places.)

Topics that can only be documented on a private wiki should be located using a similar structure on that wiki, if possible.

First draft
Below is a proposal for topics and subtopics for Fundraising Tech documentation. Topics and subtopics have codes, which are linked to brainstormed topics and the review of existing documentation, below.

In terms of on-wiki layout, subtopics below could be included in the same page as a general topic, or could have separate pages, depending on their length and complexity.

This outline should be seen as a work in progress. Some topics have been filled out in more detail than others, and it seems certain that some will be expanded or re-organized as we go.

The general logic here is:


 * Main starting point, with general overviews and links to more specific overviews and technical details [X]
 * Explanations of all the different systems involved in donation and donor processes, with technical details that are intrinsic to each system/area [A], [B], [C], [D]
 * How it all fits together: data, data flows and processes that involve multiple components [E]
 * People processes and dev tools [F]

Detailed outline:
 * Introductory overviews and links [X]
 * Introduction: General explanation of WMF donation systems [X1]
 * Donor-facing endpoints: Payments, Donate  Wiki,  Thank-you Wiki, E-mail preference center, Wikis on main cluster (sidebar links/CentralNotice), Privatebin  [A] (same as below)
 * Internal-user-facing endpoints: Civicrm, Superset [B] (same as below)
 * Externally managed services/providers: Acoustic, PSPs, Minfraud, Maxmind geolocation, currency rates provider, Google Docs spreadsheets, Zendesk [C] (same as below)
 * Cluster layout, codebases, repositories, deploy and upgrade processes, internal services [D] (same as below)
 * Data: flows throughout the stack, database schemas, identifiers numbers, details of PSP interactions, import/export of data, UML sequence and interaction diagrams [E] (same as below)
 * Team processes, roles, contact links, code freeze, code review, campaign cadences, bug triage, dev tools, onboarding [F] (same as below)
 * External-facing endpoints [A]
 * Payments/Donation Interface [A1]
 * Overview of DI functioning on Payments (note: details of data flows and PSP integrations go under [E])
 * (If still pertinent: soon-to-be-removed DI bits used as a library)
 * Doc or link to doc of globals/config variables
 * YAML-based config
 * Form translations
 * Form variants
 * Unit tests
 * Separation of concerns with SmashPig
 * Donate wiki [A2]
 * Foundation wiki, Foundation site [A3]
 * E-mail preference center [A4]
 * Main cluster wikis, portal [A5]
 * Privatebin [A6]
 * Public donation statistics (frdata.wikimedia.org) [A7]
 * E-mail sends, how e-mail translations work [A8]
 * Wikipedia mobile apps [A9]
 * Smashpig [A10]
 * Internal-user-facing endpoints [B]
 * Civicrm [B1] (note: data processes, such as database schema, sample queries, import/export details and queue message formats go under [E])
 * Overview of uses
 * Extensions, upstream interactions
 * Config
 * Reporting
 * Unit tests
 * Separation of concerns with SmashPig
 * Superset [B2]
 * Externally managed services/providers: Acoustic, PSPs, Minfraud, Maxmind geolocation, currency rates provider, Google Docs spreadsheets, Zendesk, external platforms (Facebook, Google Search) where we run ads soliciting donations [C] (technical details of the integration of these services with our systems and data flows would go in [A], [B], [D] or [E], depending; this section would contain overviews of these services, as well as contractual aspects, processes not on the Fundraising cluster, and links to technical details in other sections)
 * Cluster layout, codebases, repositories, deploy and upgrade processes, internal services [D]
 * Internal services [D1]
 * Redis
 * Process-control
 * How to find existing jobs
 * Where logs go
 * Database servers
 * Database backups
 * Log management
 * IPN listeners
 * Failmail alerts (explanation of how failmails work; tips for debugging based on failmail content go under [E9])
 * Icinga, nagios, cluster monitoring
 * Grafana
 * Stats pipelines (jobs, files, logs, tools; stats data flow, database schema and queries go under [E10])
 * SMTP for sending mail
 * Codebases (DI, Civi extensions, SmashPig), frameworks checked into version control, (Civi, Mediawiki, Drupal, others), upstream relationships (details in DonationInterface and Civicrm sections), git repositories [D2]
 * Clusters, datacenters, servers, deploy processes, dependency management on production, production puppet, ssh access, server permissions, code upgrade procedures (such as for Mediawiki, CiviCRM, or Drupal), staging environments/testing code on the cluster, running one-off jobs and scripts [D3]
 * Data and multi-component processes: data flows throughout the stack, database schemas, identifier numbers, details of PSP interactions, import/export of data, UML sequence and interaction diagrams [E]
 * PSP Integrations [E1]
 * Currently active integrations
 * Details of how they work
 * Links to payment consoles
 * List of deprecated/removed integrations (mainly for the sake of knowing which code is not important/archiving old doc)
 * Database schemas, typical queries, triggers, meanings of identifiers and other fields [E2]
 * Queues [E3]
 * Existing queues
 * Types of messages
 * Code that produces and consumes
 * Sequence generator
 * Silverpop export [E4]
 * Recurring data flows [E5]
 * Audit jobs [E6]
 * Pipeline and data flow from banners/e-mail links/landing pages through Donate Wiki, Payments [E7]
 * Fraud filtering/review of possibly fraudulent charges [E8]
 * General troubleshooting tips and suggested queries related to PSP processes, queues, logs, failmails and databases [E9]
 * Data flow for stats generation [E10]
 * Overview of steps for adding a new gateway [E11]
 * Team processes, task management, dev tools [F]
 * Team processes, roles, Phabricator, bug triage, code review, sprints, agile, task estimation, definitions of done, code freeze cycles, contact links, onboarding [F1]
 * Advancement processes: Campaigns cycle, procedures for testing new gateways/features, A/B tests overview, donor services, Asana, Zendesk [F2]
 * Developer tools:  fundraising-dev, Gerrit, Github, Translate Wiki, local setup for ssh cluster access, CI, running tests locally [F3]
 * Emergency procedures [F4]

Brainstorming: topics to include

 * FR-Tech source repositories and codebases, what they're used for, where they're deployed, how they're used, links to related documentation pages, links to upstream repos (if applicable) [D2]
 * List of public and non-public sites involved in FR systems, and which codebases are installed on each of them, and which servers host them. [A] [B]


 * Composer: How we run it locally, on production and in CI, and why, and troubleshooting tips [D3]
 * Process-control [D1]
 * Fundraising tech/Process-control June 2017
 * A list of the jobs, short description, and how often they run but on one page would be nice
 * Deployment: Steps for all service in the FR stack and config changes [D3]
 * Fundraising tech/Deployment August 2021
 * CI: how we use it, where the config files are [F3]
 * Outline of end-to-end donation flows, including code paths [E]
 * Also new, more formal diagram(s) with overview of flows
 * Logs [D1]
 * Versions and config options used for PHP, Python, other basic tools used on production [D]


 * Civicrm: Procedures for updating repos from upstream (civicrm and buildkit) [B1]
 * Civicrm: Triggers: What they're for, how to update, how they work for adding fields, how to test, how they're deployed on production and locally [E2]
 * Acoustic/silverpop export: how it works, how to add fields and backfill, which database export runs (staging?) on and why. [E4]
 * Fundraising_tech/tools has some documentation that needs updating to reflect the new incremental process
 * Fundraising tech/CiviCRM some documentation for the reverse dataflow (jobs that pull data from Acoustic to Civi) is here (written December 2019)
 * Recurring: general flow, how retries are handled [E5]
 * Fundraising tech/Recurring_donations January 2020
 * Civi extensions/Drupal modules used, maintained, vs. added just as dependencies, and their relationship to upstream [B1]


 * Integrations: Adyen Checkout: Explanation of flow for different payment options (cc, cc with 3DS, rtbt), including pointers to codebases and methods responsible for each step, and UML interaction diagrams [E1]
 * It would be nice to have at least just an up to date basic overview of all the integrations with the same topics
 * Or the opposite and have our flow pieces and then describe each of the payment processors
 * DonationInterface: Overview of main classes and methods and their functions in the system [A1]
 * Extension:DonationInterface November 2017
 * LandingCheck extension: where it's installed, how FR systems use it, link to documentation from other FR-Tech documentation. [A2]
 * Same idea as above with the other ones we have but don't touch often eg cldr, FundraisingEmailUnsubscribe [A1]
 * DonationInterface: Config: Short summary of global config variables and per-processor config mechanism on wiki, with link to README for details; complete doc of all config variables in README, explanation of config in yml files in config folders in source code, explanation of which types of config should go in which locations. [A1]
 * Pipeline for selection of country, payment processor, method, recurring, and other variants, from banners or links in e-mails to Payments site. [E7]
 * Permissions, use, content layout of wikis: Foundation wiki, Thank you wiki. [A2] [A3]
 * Meaning and usage of different types of ids and codes throughout the pipeline and across payment processors. [E2]


 * IPN listeners: what uses them and how they work [D1] [E1]
 * Fundraising tech/IPN_listener January 2020
 * Redis and queue setup [D1]


 * Phabricator and task creation, triage process [F1]
 * Sprint, agile, scrum and other development methodology (how Phabricator boards are used for that) [F1]


 * Fundraising campaigns cycle, code freeze, yearly cadence [F2]
 * Tools used outside FR-Tech: Asana, Zendesk [F2]

wikitech.wikimedia.org
See:
 * Fundraising prefix on Wikitech
 * Category:Fundraising on Wikitech
 * Category:Fundraising - Needs Updated on Wikitech
 * Category:Fundraising Analytics on Wikitech
 * Category:Fundraising Archive on Wikitech

Note: This table omits pages in Fundraising categories or subcategories that are already in the Obsolote namespace.

mediawiki.org
See:


 * Fundraising tech prefix on Mediawiki (included below)
 * Category:Fundraising on Mediawiki (included below, except for Extension:CentralNotice subpages)
 * Extension:CentralNotice prefix on Mediawiki (TODO)

Collab
See:


 * Fundraising/Engineering prefix on Collab
 * Category:Fundraising Engineering on Collab

Meta
See:


 * Category:Fundraising Engineering on Meta
 * CentralNotice help page on Meta
 * Help:CentralNotice prefix on Meta
 * Central notice administrators page on Meta

Office
See:


 * Category:Fundraising Engineering on Office

Resources
Documentation

Documentation/Style_guide