Flow/Community process workflow interviews (June 2015)

Flow is an alternative to the current talk page format under development by the Wikimedia Foundation, which aims to provide a cleaner talk page experience/interface for readers and editors on Wikipedia and Wikimedia sites. The Collaboration team is beginning to design a Workflows feature for Flow -- a key feature for the more experienced Wikipedia contributors. We need to check in with a group of experienced editors early in the design process, to make sure we’re going in the right direction.



Summary of findings



 * while workflows are idiosyncratic, many follow a general Proposal -> Discussion -> Resolution pattern
 * specialized tools (bots, gadgets, templates, userscripts) have been developed to support the needs of individual processes
 * key challenges for participants include starting a discussion, notifying relevant people and projects, and tracking discussion progress
 * key challenges for process leaders (closers, moderators, admins) include decision-making and backlog triage

Summary of recommendations

 * Provide configurable notifications for each conversation pattern to help users track discussion progress.
 * Provide fine-grained configuration options for the structure of each conversation pattern, including standardized sub-headings, !vote tags, character limits, announcement templates, and closing summaries.
 * Provide features that allow users to monitor multiple discussions concurrently.
 * Provide features that help closers make decisions more efficiently.
 * Provide features that help closers triage open discussions to reduce backlogs.
 * Pilot test initial Workflow products with less complex community processes that involve lots of discussion and have fewer specialized rules and roles.
 * Avoid designing features that encourage people to !vote rather than have a conversation.
 * Avoid artificially limiting who can perform certain actions within a discussion--keep everything open by default.
 * Avoid breaking existing gadget- and bot- mediated discussion archiving mechanisms.
 * Don't assume that processes with similar names/functions on different Wikimedia projects work the same way.

Goals of this research
The goal of this round of research, to be completed in June 2015, is to hold interviews with prospective users of Flow’s proposed Workflows feature.

The purpose of this research is twofold: (a) to learn how Wikipedia editors participate in community processes, what problems they experience, and what would help them in their work; (b) to find out whether our current ideas for a Workflows feature would help them reach their goals.

Specific goals

 * Understand the mental model of Wikipedia-related workflows (how do they normally participate).
 * Road-test the underlying logic with a knowledgeable user:
 * Is this definition of workflows easy to understand?
 * Is it able to capture the kind of workflows you know or can imagine on Wikipedia?
 * Is the way to execute workflows easy to understand, and would this be a more efficient process than the current system?


 * Identify opportunities to improve our concepts and thinking.
 * Get new perspectives and suggestions.

Key terms

 * Community process: Any discussion that starts with a proposal, has some specific rules/structure, and ends with a decision.
 * Workflow: The steps involved in any formal, multi-step community process.
 * Conversation pattern: A Flow board that is configured to support a particular kind of workflow.

Methods
We conducted interviews with Wiki[m|p]edians who participate regularly in one or more community process (such as Articles for Deletion, Administrator’s noticeboard, Good Article review, Dispute Resolution). The goal is to get as much "coverage" as possible with a relatively small set of participants (5-10), so we sampled for diversity.

In the first round, we interviewed seven participants about 14 processes across English Wikipedia, Meta, Italian Wikisource, and Wikimedia Commons. The interviewees are very active contributors on Wikimedia projects (more than 1,000 edits).

We used a semi-structured interview protocol to learn about the basic mechanics of the process, its purpose and relative importance, the context in which in occurs, who participates, etc. After the participant has described the process thoroughly, we introduced them to some early-stage prototypes and wireframes for the proposed Flow workflows interface that showed how Flow might be configured by editors to support a specific process, and how the subsequent discussion could work.

Interview protocol
This is a semi-structured interviewsemi-structured interview protocol. Not all questions will be asked in every interview, and the investigator may ask additional probe questions if interesting things come up.

When possible, the investigator will ask the participant to share links to pages that they talk about or diffs that point to particular actions or events which are referred to over the course of the interview. The investigator will also encourage the participant to share their screen as they describe their workflows.

Part 1

 * First, I'd like to get a little information about who you are and what you do on Wikipedia
 * What kind of work do you do on Wikipedia?
 * Now I'd like to learn about the community processes you participate in most frequently
 * which community processes have you participated in most during the past year?
 * how do these processes relate to other work you do?


 * Now, let's talk about a process that you've been heavily involved with recently.
 * tell me a little bit about the process and its goals
 * how many people participate?
 * In a normal week, how many of these discussions are in progress?
 * where does discussion take place?
 * how would someone who had never participated in this process learn that it exists? How would they learn how to participate?
 * Tell me a little bit about how it is organized
 * under what circumstances is this process initiated? who initiates it?
 * how long does it take before resolution?
 * do people have specific roles in this process? tell me a little bit about how people in different roles participate in the process.
 * Now I'd like to learn a little more about how you participate in this process. Thinking about the 'work' you do...
 * what role do you generally play in the process?
 * How do you decide whether to participate in a new discussion?
 * How do you keep up to date with what's going on?
 * Now I'd like to hear about what you think about the process in general.
 * Tell me about a recent fun or satisfying experience you had while participating in this process
 * Tell me about a situation where you were frustrated while working participating in this process
 * How successful is this process? How do you measure the success of a particular discussion? Of the process as a whole?
 * How important is this process to Wikipedia? Why?
 * Are there specialized tools, like userscripts, gadgets, or templates, that are used in this process?
 * Is there any tool that you use that you wish worked differently or better? How would it help you?
 * Is there any tool that your project doesn’t have that you wish it did? How would it help the project?
 * Are there improvements that would be especially useful for newcomers? For experienced editors?

Part 2
Walk through the Conversation Pattern prototype and ask participants how it might work for the processes that they participate in.

Basically, from here, we want to have a conversation with the user. We’ll have several concepts and mockups, and possibly a clickable prototype.

We want to show each artifact that we have, and at different points, ask these questions:


 * Does this piece make sense? Can you imagine how you might use it?
 * Is there anything that jumps out at you that looks exciting?
 * Is there anything that just feels wrong, and you don’t think it’ll work?
 * Can you think of a workflow (or a type of workflow) that wouldn’t work with this model?
 * Do you have suggestions?

The Flow team could make (for example) the Articles for deletion workflow in software. There’s a place to nominate the article, and it automatically fills out the user and the article name, prompts for a reason, puts a template on the article and then lists it on the AfD list. That would be relatively easy to build. But -- there are a thousand different workflows, some of them change over time, and there are different versions on different languages. So what we have to build is the set of tools that helps experienced editors to create their own workflows, for other people to use.
 * How to explain the prototypes

Prototypes
(developed before research began)
 * Conceptual prototype (AfD)
 * interactive prototype
 * demo video

(developed during phase 1)
 * Workflow configuration mock-ups
 * Generic workflow builder
 * Generic workflow builder (alternate)
 * Workflow builder - FAC
 * Workflow builder - FAC (alternate)
 * Workflow builder - RfC
 * Workflow builder - AfD
 * Workflow builder - AfD (alternate)

(developed during phase 1)
 * Workflow discussion mock-ups
 * File:Workflow_Pattern_in_action.png
 * File:AfD_in_action.jpg
 * File:RfC_in_action.jpg

Initiating a proposal
Notifying people and projects. Notifying people that a discussion has been initiated is important. People who might need to be notified of a new discussion (potential stakeholders) include: Some proposal creation workflows include tools for posting a notification (on a userpage, article or project page, or noticeboard queue). But many community processes initial notification is largely done manually by the proposer. As a result, many people who would otherwise have relevant information to contribute to a discussion aren't aware these discussion are happening. People who are responsible for guiding or deciding a proposal (such as AfC delegates and dispute resolution mediators) rely on their watchlists, Special: feeds, page histories, and (in a few cases) alert bots to track new proposals that come in.
 * people who work on an article (FAC, FAR, AfD)
 * subject matter/domain experts (FAR, FAC, AfD)
 * closers/delegates/mediators/admins (FAR, RfC, Dispute Resolution, Deletion requests, ANEW)
 * people working on a draft (AfC)
 * image uploaders (Deletion requests)

In some community processes, notifications are optional and are either a courtesy or a proactive strategy for recruiting participants to join the discussion. But other processes (ANEW, FAR) require notification of particular interested parties, but don't provide supporting tools for notification.

Creating the discussion thread. Proposing a new discussion can be a challenge even for experienced editors. Some processes (AfC, DRN, Deletion requests) provide structured workflows with good instructions that end with the creation of a properly-formatted discussion thread. Others (FAC, BAG, Requests for permissions, RfC) provide instructions on their portals which detail the steps for initiating a new proposal. In these cases, creating the thread usually involves copying and pasting a wiki template into the designated section of a page (either a page on the portal itself, or a content-namespace talkpage), or creating a new page with an inputbox and a preloaded template, and then filling in required parameters. It's easy to mess up.

Discussing a proposal
Gaining critical mass and maintaining momentum. Proposals often languish, or fail to get off the ground, due to lack of participation. Many processes (AfC, RfC, PR) have large backlogs, and it can be difficult to get enough people to participate to move the discussion forward, or to get a closer to resolve the discussion.

Tracking discussion activity. Editors use watchlists to monitor the progress of discussions. But as many editors have thousands of pages on their watchlists, it is easy to miss new comments on a discussion you're following without manually checking the discussion thread regularly. In portals that host all discussions on a single page (DRN, SRP, etc), you can check the edit comments on the page history to look for new comments.

Working in stages. Many processes operate in time-bound stages: a discussion cannot be closed until a specified time-interval has passed, or a discussion allows a particular time interval for reviews to be submitted and addressed, after which it is open to support/oppose voting. Getting an individual to return to the discussion (for example, to "sign off" on your changes after you've responded to their review) can be accomplished by manually @notifying them. Putting out an "open call" to get a whole group of people (such as all the discussion participants up to that point) to come back and vote is not currently supported.

Resolving a proposal
Finding a closer. In many community processes, closers are a scarce resource. It's not always clear who is allowed to close a discussion, or who is available to do so. In the case of FAC and FAR, only designated coordinators can make a determination of consensus and close a discussion. In DEL and ANRFC any admin can close the discussion. In some processes (DRN), a closer assigns themselves at a particular point in the discussion. In other processes (AfC) the criteria for becoming a closer are less stringent, or open to anyone who is willing to volunteer themselves and agree to the local rules (PR).

Making a final decision. For closers, coming to a decision, and justifying that decision, can be time-consuming. However this is a very important part of the process. Closers of discussions that involve long time scales, many participants, and/or lots of back and forth reported that their decision-making process was quite involved. They usually combed carefully through the various arguments, paying more attention to the strength of the arguments than the number or volume of the voices in support/opposition of the proposal. Since supporting material (policy citations, links to precedents, and other evidence) are buried in long threads and walls of text, this takes time. Participants also reported on the importance of crafting a succinct but informative rationale for their decision.

Documenting the results. The results of previous discussions can be used to inform subsequent decisions around related topics. On English Wikipedia, WikiProject Deletion Sorting performs the task of retroactively sorting closed deletion discussions by topic area, as a resource for editors looking for precedents to support the deletion or retention of other articles in that topic area. Other processes, such as RFC, maintain an archive of previous discussions organized by month.

In Featured Article Review, the result of the discussion (keep, delist) and a link to the discussion are posted by a bot on the article talkpage (example).

Common patterns
Several common patterns were observed across the community processes we investigated. Other features varied widely between processes.

Use of supporting tools
Many processes provide supporting tools such as bots, userscripts, gadgets, and templates to streamline parts of the workflow. Supporting tools help speed up repetitive actions to allow closers to manage a higher volume of proposals, facilitate the creation new proposals according to a specific template, deliver notifications, and archive completed discussions.

Examples
 * DRN proposal workflow
 * Commons QuickDelete gadget
 * AfD bot
 * AfC Helper script
 * Twinkle

Triage dashboards
Several processes also provide dashboards or indexes of open discussions, sorted by recency, topic, and/or status, in order to help closers decide which proposals to work on. The dashboards are either updated manually, or managed through transclusions of special pages (Special:RecentChangesLinked), extensions (DynamicPageList, and bots (Dispute resolution ClerkBot).

Examples
 * Peer review backlog
 * Good article nominations board
 * Bot request for approval status table
 * Dispute resolution case template

Variables

 * Location of the discussion:


 * Meso-structure of discussions:


 * Scale of the discussion:


 * Rules and roles:

Recommendations for deployment
Pilot tests - start with smaller, simpl(er), less essential process. Get support of essential process participants. Ask them to experiment with configuring and using the CP in a sandbox wiki, provide clear mechanisms for feedback Arrange a time-limite pilot with clear feedback mechanisms and the ability to restore the legacy workflow if desired, at the pilot's conclusion. Survey/interview all participants (core and perpheral) at the conclusion