ORES/Metrics

From mediawiki.org
Warning Warning: The ORES infrastructure is being deprecated by the Machine Learning team, please check wikitech:ORES for more info.

We collect some metrics in the Python backend, and forward them to Grafana via a statsd endpoint. See wikitech:ORES/Metrics for the statsd endpoints used in Wikimedia installations of ORES.

precache_request - Incremented once for each "precache" score that is requested. Note that this is not the same scale as scores_request.

revision_scored - Unused.

scores_request - Incremented once for each non-"precache" score that is requested. (TODO: Give a more descriptive name, like "external_score_request")

score_processed - Incremented after each worker scores a revision (for each model that is scored). This is not called when a score is cached or found to be in-progress from a recent request.

score_processor_overloaded - Sent when a request is rejected due to the Celery queue exceeding the configured maximum queue size. We also return a "503 server overloaded" to the client.

score_cache_hit - Incremented for each revision score looked up in the Redis cache. (Note that the cache lookup is circumvented for requests that include "?features")

score_cache_miss - Incremented for each score cache lookup that fails. (Note that the cache lookup is circumvented for requests that include "?features")

score_errored - Incremented each time a score cannot be calculated because of a nonexistent revision ID, timeout, or other errors.

score_timed_out - Incremented when score processing takes longer than the scoring system's configured timeout (15 seconds as of Oct 2016).

datasources_extracted - Unused. TODO: ORES PR #176

precache_scores - Unused.

precache_score - Incremented within the precached utility when a score is successfully precached. (not used in ChangePropagation)

precache_scoring_error - Incremented within the precached utility when a score errors. (not used in ChangePropagation)

TODO:

  • One more pass to tighten up how each event stream maps to a score e.g. per model.