Moderator Tools/Automoderator/Testando

Automoderator

Measurement plan

Testing

Diagrama demonstrando o processo de decisão do software Automoderator.

A Moderator Tools equipe está construindo Automoderator - uma ferramenta que pode reverter automaticamente edições ruins com base em um modelo de aprendizado de máquina, desempenhando uma função semelhante a bots comunitários de combate ao vandalismo, como ClueBot NG, SeroBOT, Dexbot, Salebot, $2 e $3. Para ajudar as comunidades a testar e avaliar a precisão do Automoderator, estamos disponibilizando uma planilha de teste com dados sobre edições anteriores e se o Automoderator as teria revertido ou não.

As decisões do Automoderator resultam de uma combinação de pontuação do modelo de aprendizado de máquina e configurações internas. Embora o modelo melhore com o tempo por meio de retreinamento, também estamos buscando aprimorar sua precisão ao definir algumas regras internas adicionais. Por exemplo, observamos que o Automoderator ocasionalmente identifica erroneamente usuários revertendo suas próprias edições como vandalismo. Para melhorar, estamos buscando exemplos semelhantes e agradecemos sua assistência em identificá-los.

Observe que este teste não reflete necessariamente a forma final do Automoderator - usaremos os resultados deste teste para aprimorá-lo!

Como testar o Automoderator

Captura de tela da planilha, com exemplos de respostas preenchidos.

• Se você tiver uma conta no Google:
  1. Use the Google Sheet link below and make a copy of it
     1. You can do this by clicking File > Make a Copy ... after opening the link.
  2. Após o carregamento da cópia, clique em Compartilhar no canto superior, em seguida, conceda qualquer acesso por avardhanawikimedia.org (deixando 'Notificar' marcado), para que possamos agregar suas respostas e coletar dados sobre a precisão do Automoderator.
     1. Alternatively, you can change 'General access' to 'Anyone with the link' and share a link with us directly or on-wiki.
• Alternatively, use the .ods file link to download the file to your computer.
  • Após adicionar suas decisões, por favor, envie a planilha de volta para nós em avardhanawikimedia.org, para que possamos agregar suas respostas e coletar dados sobre a precisão do Automoderator.

Após acessar a planilha...

1. Siga as instruções na planilha para selecionar um conjunto de dados aleatório, revisar 30 edições e, em seguida, descobrir quais decisões o Automoderator tomaria para cada edição.
   1. Feel free to explore the full data in the 'Edit data & scores' tab.
   2. If you want to review another dataset please make a new copy of the sheet to avoid conflicting data.
2. Participe da discussão na página de discussão.

Alternativamente, você pode simplesmente navegar nas abas individuais do projeto e começar a investigar os dados diretamente.

---

Google Sheet .ods file (download)

*We welcome translations of this sheet - if you would like to submit a translation please make a copy, translate the strings on the 'String translations' tab, and send it back to us at swaltonwikimedia.org.

If you want us to add data from another Wikipedia please let us know and we would be happy to do so.

About Automoderator

Automoderator’s model is trained exclusively on Wikipedia’s main namespace pages, limiting its dataset to edits made to Wikipedia articles. Further details can be found below:

Internal configuration

In the current version of the spreadsheet, in addition to considering the model score, Automoderator does not take actions on:

• Edits made by administrators
• Edits made by bots
• Edits which are self-reverts
• New page creations

The datasets contain edits which meet these criteria, but Automoderator should never say it will revert them. This behaviour and the list above will be updated as testing progresses if we add new exclusions or configurations.

Caution levels

In this test Automoderator has five 'caution' levels, defining the revert likelihood threshold above which Automoderator will revert an edit.

• At high caution, Automoderator will need to be very confident to revert an edit. This means it will revert fewer edits overall, but do so with a higher accuracy.

• At low caution, Automoderator will be less strict about its confidence level. It will revert more edits, but be less accurate.

The caution levels in this test have been set by the Moderator Tools team based on our observations of the models accuracy and coverage. To illustrate the number of reverts expected at different caution levels see below:

	Daily edits	Daily edit reverts	Average daily reverts by Automoderator
			Very cautious >0.99	Cautious >0.985	Somewhat cautious >0.98	Low caution >0.975	Not cautious >0.97
Wikipédia em inglês	140,000	14,600	152	350	680	1077	1509
Wikipédia em francês	23,200	1,400	24	40	66	98	136
Wikipédia em alemão	23,000	1,670	14	25	43	65	89
Wikipédia em espanhol	18,500	3,100	57	118	215	327	445
Wikipédia em russo	16,500	2,000	34	57	88	128	175
Wikipédia em japonês	14,500	1,000	27	37	48	61	79
Wikipédia em chinês	13,600	890	9	16	25	37	53
Wikipédia em italiano	13,400	1,600	40	61	99	151	211
Wikipédia em polonês	5,900	530	10	16	25	35	45
Wikipédia em português	5,700	440	2	7	14	21	30
Wikipédia em hebraico	5,400	710	16	22	30	38	48
Wikipédia em persa	5,200	900	13	26	44	67	92
Wikipédia em coreano	4,300	430	12	17	23	30	39
Wikipédia em indonésio	3,900	340	7	11	18	29	42
Wikipédia em turco	3,800	510	4	7	12	17	24
Wikipédia em árabe	3,600	670	8	12	18	24	31
Wikipédia em checo	2,800	250	5	8	11	15	20
Wikipédia em romeno	1,300	110	2	2	4	6	9
Wikipédia em croata	500	50	1	2	2	3	4
...	...	...	...	...	...	...	...
All Wikipedia projects			538	984	1683	2533	3483

This data can be viewed for other Wikimedia projects here.

Score an individual edit

Importing this user script will give you a 'Get revert risk score' button in your Tools menu.

We have created a simple user script to retrieve a Revert Risk score for an individual edit. Simply import User:JSherman (WMF)/revertrisk.js into your commons.js with mw.loader.load( 'https://en.wikipedia.org/wiki/User:JSherman_(WMF)/revertrisk.js?action=raw&ctype=text/javascript' );.

You should then find a 'Get revert risk score' in the Tools menu in your sidebar. Note that this will only display the model score, and does not take into account Automoderator's internal configurations as detailed above. See the table above for the scores above which we are investigating Automoderator's false positive rate.

Initial results

Quantitative

22 testing spreadsheets were shared back with us, totalling more than 600 reviewed edits from 6 Wikimedia projects. We have aggregated the data to analyse how accurate Automoderator would be at different caution levels:

Not cautious (0.97)	Low caution (0.975)	Somewhat cautious (0.98)	Cautious (0.985)	Very cautious (0.99)
75%	82%	93%	95%	100%

In our measurement plan we said that we wanted the most permissive option Automoderator could be set at to have an accuracy of 90%. The ‘Not cautious’ and ‘Low caution’ levels are clearly below this, which isn’t surprising as we didn’t have clear data from which to select these initial thresholds. We will be removing the ‘Not cautious’ threshold, as a 25% error rate is clearly too low for any communities. We will retain ‘Low caution’ for now, and monitor how its accuracy changes as model and Automoderator improvements occur leading up to deployment. We want to err on the side of Automoderator not removing bad edits, so this is a priority for us to continue reviewing.

When we have real world accuracy data from Automoderator's pilot deployment we can investigate this further and consider changing the available thresholds further.

Qualitative

On the testing talk page and elsewhere we also received qualitative thoughts from patrollers.

Overall feedback about Automoderator’s accuracy was positive, with editors feeling comfortable at various thresholds, including some on the lower end of the scale.

Some editors raised concerns about the volume of edits Automoderator would actually revert being relatively low. This is something that we’ll continue to discuss with communities. From our analysis (T341857#9054727) we found that Automoderator would be operating at a somewhat similar capacity to existing anti-vandalism bots developed by volunteers, but we’ll continue to investigate ways to increase Automoderator’s coverage while minimising false positives.

Next steps

Based on the results above, we feel confident in the model’s accuracy and plan to continue our work on Automoderator. We will now start technical work on the software, while exploring designs for the user interface. We expect that the next update we share will contain configuration wireframes for feedback.

In the meantime please feel free to continue testing Automoderator via the process above - more data and insights will continue to have a positive impact on this project.