Analytics/Archive/Editor Engagement Vital Signs/Backfilling

Backfilling
Some benchmarks of how log did it take to backfill data for Rolling active editor EEVS metric.

We use as baseline our master branch on 2014-06-12 versus changes on this patchset: https://gerrit.wikimedia.org/r/#/c/150475/

Labs db infrstructure ( labsdb1002 dewiki, commons, etc) was upgraded to maria db about the last week of July. All data is on an SSD now.

Config for celery was: BROKER_URL                         : redis://localhost:6379/0 CELERY_RESULT_BACKEND              : redis://localhost:6379/0 CELERY_TASK_RESULT_EXPIRES         : 2592000 CELERY_DISABLE_RATE_LIMITS         : True CELERY_STORE_ERRORS_EVEN_IF_IGNORED : True CELERYD_CONCURRENCY                : 10 CELERYD_TASK_TIME_LIMIT            : 3630 CELERYD_TASK_SOFT_TIME_LIMIT       : 3600 DEBUG                              : False LOG_LEVEL                          : INFO MAX_PARALLEL_PER_RUN               : 10 MAX_INSTANCES_PER_RECURRENT_REPORT : 365 CELERY_BEAT_DATAFILE               : /var/run/wikimetrics/celerybeat_scheduled_tasks CELERY_BEAT_PIDFILE                : /var/run/wikimetrics/celerybeat.pid CELERYBEAT_SCHEDULE                : 'update-daily-recurring-reports': 'task'     : 'wikimetrics.schedules.daily.recurring_reports' # The schedule can be set to 'daily' for a crontab-like daily recurrence 'schedule' : debug

rowiki

 * Backfilling of 3 months of data takes about 3 minutes
 * Backfilling of 1 year of data takes about 10 minutes

eswiki

 * Backfilling of 3 months of data took 8 minutes.
 * Backfilling of 5 months of data took 10 minutes
 * Backfilling of 1 year of data took 30 minutes

rowiki

 * Backfilling of 3 months of data takes about 3 minutes
 * Backfilling of 1 year of data takes about 6 minutes

frwiki

 * Backfilling 3 months: 7 mins