Fundraising tech/notes/Draft:Documentation overhaul

Brainstorming: topics to include

 * FR-Tech source repositories and codebases, what they're used for, where they're deployed, how they're used, links to related documentation pages, links to upstream repos (if applicable)
 * List of public and non-public sites involved in FR systems, and which codebases are installed on each of them, and which servers host them.


 * Composer: How we run it locally, on production and in CI, and why, and troubleshooting tips
 * Process-control
 * Fundraising tech/Process-control June 2017
 * A list of the jobs, short description, and how often they run but on one page would be nice
 * Deployment: Steps for all service in the FR stack and config changes
 * Fundraising tech/Deployment August 2021
 * CI: how we use it, where the config files are
 * Outline of end-to-end donation flows, including code paths
 * Also new, more formal diagram(s) with overview of flows
 * Logs
 * Versions and config options used for PHP, Python, other basic tools used on production


 * Civicrm: Procedures for updating repos from upstream (civicrm and buildkit)
 * Civicrm: Triggers: What they're for, how to update, how they work for adding fields, how to test, how they're deployed on production and locally
 * Acoustic/silverpop export: how it works, how to add fields and backfill, which database export runs (staging?) on and why.
 * Fundraising_tech/tools has some documentation that needs updating to reflect the new incremental process
 * Fundraising tech/CiviCRM some documentation for the reverse dataflow (jobs that pull data from Acoustic to Civi) is here (written December 2019)
 * Recurring: general flow, how retries are handled
 * Fundraising tech/Recurring_donations January 2020
 * Civi extensions/Drupal modules used, maintained, vs. added just as dependencies, and their relationship to upstream


 * Integrations: Adyen Checkout: Explanation of flow for different payment options (cc, cc with 3DS, rtbt), including pointers to codebases and methods responsible for each step, and UML interaction diagrams
 * It would be nice to have at least just an up to date basic overview of all the integrations with the same topics
 * Or the opposite and have our flow pieces and then describe each of the payment processors
 * DonationInterface: Overview of main classes and methods and their functions in the system
 * Extension:DonationInterface November 2017
 * LandingCheck extension: where it's installed, how FR systems use it, link to documentation from other FR-Tech documentation.
 * Same idea as above with the other ones we have but don't touch often eg cldr, FundraisingEmailUnsubscribe
 * DonationInterface: Config: Short summary of global config variables and per-processor config mechanism on wiki, with link to README for details; complete doc of all config variables in README, explanation of config in yml files in config folders in source code, explanation of which types of config should go in which locations.
 * Pipeline for selection of country, payment processor, method, recurring, and other variants, from banners or links in e-mails to Payments site.
 * Permissions, use, content layout of wikis: Foundation wiki, Thank you wiki.
 * Meaning and usage of different types of ids and codes throughout the pipeline and across payment processors.


 * IPN listeners: what uses them and how they work
 * Fundraising tech/IPN_listener January 2020
 * Redis and queue setup


 * Phabricator and task creation, triage process
 * Sprint, agile, scrum and other development methodology (how Phabricator boards are used for that)


 * Fundraising campaigns cycle, code freeze, yearly cadence
 * Tools used outside FR-Tech: Asana, Zendesk

Draft possible outline

 * Overviews
 * Introduction: General explanation of WMF donation systems
 * User-facing endpoints:  Payments,  Civi,  Donate  Wiki,  Thank-you Wiki, Wikis on main cluster (sidebar links/CentralNotice), Superset
 * Internal services:  Redis,  process-control,  DBs,  log  management, deployment systems, IPN listeners, Failmail alerts, icinga, stats pipelines
 * Externally managed services: Acoustic, PSPs, Minfraud, Maxmind geolocation
 * Codebases (DI, Civi extensions, SmashPig), frameworks (Mediawiki, Drupal), upstream relationships (details in DonationInterface and Civicrm sections)
 * Git repositories
 * Clusters, datacenters and servers
 * UML sequence and interaction diagrams
 * Processes, process management and dev tools (overview with link to details, see below)
 * Donation Interface
 * How it works on Payments
 * Soon-to-be-removed bits used as a library
 * Doc or link to doc of globals/config variables
 * Dependency management
 * Config setup
 * Civicrm
 * Extensions, upstream interactions
 * Databases
 * Silverpop export
 * Recurring
 * Dependency management
 * Config
 * Wikis that are wikis (sort of)
 * Donate wiki
 * Foundation wiki
 * PSP Integrations
 * Currently active integrations
 * Details of how they work
 * Links to payment consoles
 * List of deprecated/removed integrations (mainly for the sake of knowing which code is not important/archiving old doc)
 * Queues
 * Existing queues
 * Types of messages
 * Code that produces and consumes
 * Sequence generator
 * Process control
 * How to find existing jobs
 * Where logs go
 * Deployment
 * Tools
 * Processes
 * Statistics processing and Superset
 * Processes, process  management, dev tools
 * Development: Phabricator, code  review, sprints, agile, task estimation, definitions of done, code freeze cycles
 * Advancement: Campaigns cycle, Asana, Zendesk
 * Developer tools:  fundraising-dev,  Gerrit,  Translate Wiki, ssh cluster access, CI, grafana