Developer Wishlist/2017/Tools (Phabricator, Gerrit, etc.)

Problem
In contrast to Gerrit patches with "Bug: TXXXXXX" in the commit message, there is no "patch-for-review" tag added for Differential patches on.

Who would benefit
The tag is helpful to maintain better overview on the state of tasks for developers and product managers in workboards, like for example #ui-standardization-kanban Currently you have to manually add the tag, see T147612: Variable naming in WikimediaUI Base as comparison.

Proposed solution
Add #patch-for-review tag when triggered trough valid commit message at.

Re-evaluate how we implement phabricator's search engine
IMPORTANT: Experimental elasticsearch support has been enabled, to try it out please do the following. Enter something in the searchbar then you will get a url like https://phabricator.wikimedia.org/search/query/.trvn2LNszXn/#R please replace #R with ?elastic=1 which will give you elasticsearch results, example elasticsearch url https://phabricator.wikimedia.org/search/query/.trvn2LNszXn/?elastic=1

When we originally deployed Phabricator, we started out using the #ElasticSearch back-end for searching phabricator Tasks / Objects. Unfortunately, after launching with #elasticsearch there were quite a few user complaints about unexpected search behavior and we eventually made the (IMO, somewhat hasty) decision to switch to using the MySQL Full-text search backend.

Now that we reached the scalability limit of MyISAM fulltext search, we've been sort of forced to switch to Innodb Fulltext search (see T146673: Contention on search phabricator database creating full phabricator outages )... The fulltext search feature in innodb is not as mature as MyISAM fulltext. The ranking algorithm is very simplistic and I expect results to be lower quality than before.

ElasticSearch should be far superior to mysql for implementing phabricator search. Lets try again to solve the problems with phabricator's elasticsearch integration rather than settling for a much worse search experience in Innodb.

A bit of history
There is quite a bit of history behind Phabricator's search implementation. Some of the more relevant tasks I was able to dig up are linked below:

Original ElasticSearch task
T95: Use ElasticSearch backend for Maniphest to get stemming feature

Stuff about switching to MySQL
T75854: Fix provided search results in Wikimedia Phabricator T86805: Lots of unrelated results when searching for specific string

Problems after switching to Mysql
T89274: Mysql search issues flagged by Phabricator setup T146789: Phab Advanced Search no longer showing typical results

Small enhancements to current system of dumps
I know dumps are being re-written but it's taking several years so in the mean time, it would be great to have small enhancements (low hanging fruits) to the current system:
 * Make it faster and nimble:
 * Remove Yahoo! summaries: For example it's 38 GBs for Wikidata and it slows down the whole process of generating dumps for all wikis. The Yahoo! itself is about to die and I don't know why we keep doing this.
 * Keep one compression method and abandon the other. Why we build two dumps one for 7z and one for bz2?
 * Notifications: I would love to receive an email once the monthly dump of Wikidata is done so I can run some checks and fix it on-wiki.
 * UI. It's so 1990s. Please have something a little bit prettier.

Consolidate the many tech events calendars in Phabricator's calendar
We have three problems related to calendars that need to be resolved, which are related but perhaps may be fixed separately, even at different points in time:


 * Introducing calendar data. Now we are doing this in 1001 places. Ideally it would be just one place. The assumption of this task is that the one place would be Phabricator's calendar (today a prototype), but other candidates should be considered.
 * Pulling calendar data in other supports, mainly wiki pages. While we could start linking from wiki pages to Phabricator calendar queries, the desired scenario would be a way for editors to embed calendars based on queries in wiki pages. This probably implies a Calendar API to query data (which is missing) and a MediaWiki extension allowing to embed those queried calendars in wiki pages (which we don't have). Maybe this is just too much, and at least the tech-inclined Wikimedians would be fine clicking a Phabricator link.
 * Integrating calendar data with other systems (mainly Google Calendar). "Future work" upstream, no plans today.

As we can see, deciding on Phabricator Calendar as the one place to introduce data would not provide a quick path to use that data out of Phabricator. However, what would be the alternatives? Other than defaulting to Google Calendar (which I think would be wrong as a matter of Wikimedia principles), I don't see a shortcut but others may have better ideas.

Phabricator Calendar in a nutshell
After playing a bit with https://phabricator.wikimedia.org/calendar/, I think the most practical option is to focus on having that calendar up to date, and then link to it from the right places in mediawiki.org.


 * Entering events is simpler. The usual you can expect in a calendar application. There is no need to archive past events, since this is done automatically.
 * Users can RSVP, getting notifications about the event and letting others know that they are attending.
 * The calendars shown are the result of queries allowing for multiple parameters, including month view vs list, show only upcoming events, etc. These queries have static URLs.
 * Events can be assigned to projects, which means that
 * We could have projects for i.e. Tech Talks, Technical Office Hours, etc, people could subscribe to these projects and receive notifications when a new event is filed.
 * By adding an event to existing related projects (i.e. #Web-APIs-hub, #Google-Summer-of-Code-2015), users following that project would receive notifications as well.
 * Queries can be fine tuned to show a specific set of events.

If we agree on this, the implementation would be relatively simple, since (at least in Wikimedia tech) a few people create most of the events. Redirecting from the current calendar pages to their equivalent Phabricator calendar queries should be enough to tell current users about these calendars about the new way to create new events.

Basic review of existing MediaWiki extensions
I went through https://www.mediawiki.org/wiki/Extension:Calendar (a linkhub for the ~12 different extensions or methods of integrating calendars with MediaWiki). There was no extension surviving a basic check against our needs. In fact, most of them are more than abandoned, and in general they focus on presentation, not on easy input, even less on semantic data (except the SemanticMediaWiki one, perhaps). SimpleCalendar looks like the only candidate if we would ever want to start from this point.

The reason for a lack of a proper MediaWiki Calendar extension may well be that MediaWiki was not designed to input calendar data, subscribe to events, etc.

After this short investigation, I still think that it is better to start from something other than MediaWiki, and the main candidate keeps being Phabricator. Then work separately in a way to import Phabricator Calendar data and present it in MediaWiki pages.

Background / previous description
https://www.mediawiki.org/wiki/Project:Calendar has a lot of room for improvement. Creating tasks is not trivial, and although the calendar can be watched and the entries appear in the mediawiki.org homepage, it doesn't have any of the features expected nowadays in an online calendar, i.e. integration with the calendar apps people use to organize their lives.

Also, for some low hanging fruit, a reporter explains in an email:

Enable Gerrit reviewers-by-blame plugin
Per discussion in T91190: When uploading a new patch, reviewers should be added automatically, it would be nice to enable reviewers-by-blame which helps inexperienced developers find the right person for code review. (I'm not sure what the roadmap is for Gerrit -> Phabricator migration - please decline if it's too close for Gerrit improvements to be worth the effort. Closest Phabricator equivalent I have found is phabricator:T1743, still under development.)

When a project has no workboards enabled, many clicks are required to open the task list of that project
I used to be able to click a tag, click "open tasks" on the sidebar and see a list of open reports for that project. ( T89865: Phabricator project page should not default to workboard made that slower but still feasible, at least for people with good enough CPUs and networking.) Nowadays, I have no idea how to reach the same result in less than 10 clicks or so: can someone point out a way? Thanks.

Upstream task (with potential Phabricator extension code): T10308: Please undo some changes

[MediaWiki-commits] Reverts are not notified by gerrit
New changesets which are reverts, and the respective comment on the reverted change, are not announced on mailing list; the merge of the revert is. Example: http://blog.gmane.org/gmane.org.wikimedia.mediawiki.cvs/day=20130415/page=7 (59135).


 * This finds nothing: http://markmail.org/search/?q=%22This%20patchset%20was%20reverted%20in%20change%22%20list%3Aorg.wikimedia
 * 508 mentions of reverts: http://markmail.org/search/?q=%22Change+subject%3A+Revert%22+list%3Aorg.wikimedia.lists.mediawiki-cvs#query:%22Change%20subject%3A%20Revert%22%20list%3Aorg.wikimedia.lists.mediawiki-cvs
 * 394 of which seem to be merges: http://markmail.org/search/?q=%22Change+subject%3A+Revert%22+%22has+submitted+this+change+and+it+was+merged%22+list%3Aorg.wikimedia.lists.mediawiki-cvs

Phabricator silently overwrites changes (no mid-air collision/conflict detection)
Upstream task for Phriction and Maniphest: https://secure.phabricator.com/T4768

It looks like Phabricator may be unintentionally removing CCs when a comment is submitted shortly after a person subscribes? This is pretty worrying.

Examples: T77611: Investigate Erik's ideas about why enwiki is not equivalent to other wikis, T78116: 503 Varnish error when adding template to a page on Commons., T78607: Update from 1.19 to 1.23 fails: Error: 1054 Unknown column 'page_content_model' in 'field list'., T85196: Checkuser results should update usernames after a user is renamed

Issues with subscribers additions: T96464: Upon edit, a task description which mentions a Phab user (re)adds that Phab user to CC/Subscribers field, T913: Adding subscribers to a Phabricator task should be more lightweight.

Announce all creations, deletions and renaming of gerrit repos (for e.g. translatewiki.net workflow)
Creation, deletion and renaming of Gerrit repositories regularly has a very negative impact on the translatewiki.net workflow.

Recently the repo DonationEmailUnsubscribe appears to have been deleted. The repo FundraisingEmailUnsubscribe appears to have been moved from wikimedia to mediawiki, without updating the permissions on the repo, and without updating .gitreview.

Deleting repos causes errors in mass update scripts. They will exit with inexplicable errors, which then means that you have to start chasing ghosts in https://gerrit.wikimedia.org/r/#/admin/projects/

Phabricator is unfriendly to assistive technology
Filed upstream: https://secure.phabricator.com/T4843

Nothing ever shows on cursor hover, and none of the popular images (avatars, Phabricator logo, edit menu buttons) include the "alt" attribute.

Actually, the pictures all seem to be added through CSS (as part of background for div, span, and similar tags), which is very bad for screen readers and similar software (IIRC).

Free-form tagging in gerrit
As with CodeReview, to organize work. See https://www.mediawiki.org/wiki/Gerrit/Tagging

-- See Also: http://code.google.com/p/gerrit/issues/detail?id=287

Phabricator should suggest possible duplicates when creating a new task
When you create a new bug report in Bugzilla, you get a list of possible duplicates. I find it useful. It might be even more useful for new users filing what is going to be probably an obvious duplicate.

Upstream ticket: https://secure.phabricator.com/T4828

Cannot disable "Notify" for token award in phabricator
Upstream report: https://secure.phabricator.com/T7468

I am missing a way to disable the "Notify" when a task was getting a token.

In the Email Preferences under https://phabricator.wikimedia.org/settings/panel/emailpreferences/ I have not found a way to disable this. I did not need tokens and therefore no notify if a subscribed task get a token.

Phabricator email notifications should bundle events as the web interface does
As a user with limited email digesting capabilities*, I want phabricator to make some basic digesting on my stead, so that I'm not overwhelmed; and to have a consistent behaviour across web interface and emails, so that I'm not confused.

I. Observed: the web interface bundles/digests/collates tightly connected events by a single user under a single header/timestamp, e.g. https://phabricator.wikimedia.org/T85304#943638. The emails instead show separately the actions which were made separately. Not only the additional emails are annoying, but whether some actions were made together or in multiple steps is totally irrelevant for me as a subscriber, to the point they're often impossible to verify in the web interface (because not displayed). II. Expected: a single email should be sent for all the events which the web interface bundles together. A delay in sending the email notifications may be added, if needed for this purpose.

(*) Note: I don't fall in this category so I don't have a COI in filing this task.

Duplicate tasks are not listed in or near the description of the target task
As a user, I want to have a list of tasks currently marked as merged to the one I'm reading, so that I know what are the tasks currently marked as merged to the one I'm reading.

The description of T857: Imported bugs from bugzilla should be assigned the same number as their bugzilla ID (i.e. Bug 1 -> Task 1; Bug 2007 -> Task 2007) has no mention of the fact T625: Bug 1 be bug 1 was merged into it. The fact should be noted:
 * in or around the description, with a list of all merged tasks (as with blocking tasks);
 * in the history of actions/comments [this is currently satisfied, but hard to find].

Convert Bugzilla's "Bug NNNNN" links to "TNNNNN" links in Phabricator
In comments, [Bb]ug #? should be parsed as T: that is, "bug 323" would be rendered as T2323; or converted to "bug 323 ". Ideally, the full bugzilla syntax would be parsed/converted, see https://bzr.mozilla.org/bugzilla/4.4/view/head:/Bugzilla/Template.pm#L234

Potentially incomplete proposal:

Use case:

@Chasemp: For sentences imported from Bugzilla tickets in comments like

'''* Bug 12345 has been marked as a duplicate of this bug. *'''

(yes, those are three stars at the start of the line turned into a bullet point, meh, another cosmetic thingy)

I know that some folks like to look at duplicates because they might provide more information about a bug and clicking instead of manual copy and paste fiddling sounds easier...

Allow to search tasks about MediaWiki core and core only (create MediaWiki umbrella project?)
As a user looking for a report about a MediaWiki core bug, I want to quickly look for it without bothering about non-core reports or specific components/projects, so that I find what I'm looking for before I give up out of desperation.

Currently, I must type N MediaWiki core projects manually in the "Projects" field. (Cf. https://secure.phabricator.com/T3670#65849, first bullet.)