Topic on Talk:Page Previews/Flow

There is a rollout plan in progress(?!) but the only community consensus on it was quite opposed to default-on deployment.

Perhaps this was unclear because the close was written as "No Consensus" (to enable) rather than "Consensus against". This is not unusual when the effect is expected to be essentially the same. Often language like "no consensus (for a proposal)" is used to more openly invite a new discussion, if anyone feels it may lead to a different result. (Either because changes have been made, or because circumstances have changed, or because someone wants to bring new information or arguments to the table, or becomes someone simply thinks that views on the subject may have shifted over time.)

The WMF may be able to (cautiously) go forwards with a project in a genuine non-consensus situation, where the results were close and other factors might support it. However in this case the closer cited a fuzzy&incomplete vote ratio of about 3-to-1 against, and the closer noted that factors other than raw vote count don't lean in the support direction. My generous/optimistic interpretation would put the result at about 2-to-1 opposed. This is important, the RFC topic was "Proposal: Enable Hovercards by default" and there was "no consensus" to take that action. However imagine we rewrote the RFC question as "Proposal: Disable Hovercards". Imagine everyone responded the same way they did the first time. That same 2-to-1 result would pretty well constitute a consensus to disable. Therefore the "No consensus to enable" is effectively a "Consensus against enabling". It was just phrased to be more friendly if you want to try raising the topic again. (Or maybe the closer always uses "no consensus" language to summarize any failed proposal.)

I didn't participate in that RFC, but I'd like to offer my experience with Hovercards. It sounds like a great idea, and I actively tried to help the project along. I hate how often the WMF&community get into conflict and I really wanted to help a project succeed. I enabled Hovercards and played with it, and it seemed great and useful. "Neat, I can get a preview whenever I want". Then I stopped playing with it and when back to normal. Over the next few days it became clear that I was never actually using Hovercards. In fact it became increasingly annoying to have unwanted popups randomly blocking my ability to read the current page. The longer I left Hovercards enabled, the more frustrating it became to have my view repeatedly and randomly blocked by popups. Hovercards are a great idea in theory - you get preview when you want it. In practice, for me at least, ~100% of the popups turned out to be unwanted and frustrating.

Hovercards are a love-it or hate-it feature. I know your survey got pretty positive numbers, but I have to wonder if some of that positive feedback was like mine... that Hovercards only seem great on first impression when you're actively playing with it. Even if a majority of people do like Hovercards, even if you try to make them easy to disable, I think too many people will get pissed off. This is especially critical for logged out users. They'll get really pissed off repeatedly trying to shut off Hovercards every week or every month or whatever. If there is another RFC then I'll have to regretfully oppose any enabled-by-default.

 

Hey Alsee, when you say community are you referring specifically to the English Wikipedia? I'm assuming so in my response.

Beta Features/Hovercards#Rollout Plan shows there's no check mark for the English Wikipedia. We have not approached that community since the RfC to address the concerns raised. However, a quick look at that RfC I can see that most of the more egregious issues are addressed with our plans.

1. Editors are opt-in. They won't get the feature enabled by default.
2. Non-logged in folks can opt-out. There's a preference they can set without logging in to disable the feature.
3. The big bugs have been addressed (and many smaller ones). Especially Page Previews getting stuck - the most frustrating of all if you take a look at the talk page here :)
4. There were concerns around user experience. The research and feedback received during A/B tests address if folks understand the feature and how to disable/enable.

This also doesn't mention the qualitative and quantitative research the team's done since the English Wikipedia RfC to investigate if Page Previews negatively impact things like page views, performance, and fundraising. Spoiler alert! Nothing strongly negative was discovered. :)

As you described in great detail, the RfC did not obtain consensus. Like we've done for the 4 communities we have already approached, we state our intent for the feature, reference our rational and research, provide the community time to respond with questions and concerns, and then hopefully move forward if there no outstanding issues. The same will apply with the English Wikipedia.

Really, now that I'm looking over the RfC a little more I think the latter part of Jon Katz's descriptions of the next steps in the RfC are pretty much what I've described above!

I hope that alleviates some of your concerns. Thank you for taking the time to investigate and greet us here with the information and feedback you've provided. I appreciate it.

Oh, and another thing we haven't mentioned yet. The new refactored version of Page Previews will arrive (as the current opt-in Beta Feature) next Thursday (23 February) if things go to plan. We should have a message out to the wikitech-l maling list tomorrow. Having the updated code in the Beta feature allows communities use the feature and provide feedback as we approach them.

I hope you (and others reading) will give the feature another try when the updated code arrives and provide feedback on performance, bugs, and what not.

@CKoerner (WMF) thanx. And yep, I was referring to the (linked) English discussion on it.

Like we've done for the 4 communities we have already approached, we state our intent for the feature, reference our rational and research, provide the community time to respond with questions and concerns, and then hopefully move forward if there no outstanding issues.

Three :) and I see you're going back for #4.

However I'd like to point out a common pattern. Stuff gets pushed out to small wikis because they don't/can't effectively participate. English Wiki shares a common language with the WMF. English is the oldest, largest, most established, and most engaged Wiki. Some of the the other large wikis have a well developed central community and sometimes engage, but it often takes a pretty big issue for them to climb over the language barrier to effectively participate. This is amplified when WMF postings tend to come across as "announcements". On EnWiki there was a "Proposal to enable Hovercards". It was posted as an RFC for a meaningful examination of consensus. I used machine translation to look at the other four wiki postings. You posted the A/B results, you stated that you want to enable Hovercards, you invited feedback, and you announced that you intend to deploy it unless you see serious problems. That got almost no participation. It was not treated as a significant discussion, much less a consensus process. For example on Russian Wiki it looks like two people posted clearly supportive comments, one person explicitly tried to cast an oppose , and I think another badly-translated response is a second oppose !vote. There was negligible participation (2 vs 2), and no attempt to form any kind of consensus because it was just an announcement. People aren't likely to turn on a beta feature and seriously investigate the issue just to post a throw-away comment after an announcement.

Let's put it this way. Let's say other major wikis do seriously examine Hovercards, with consensus in favor. That would suggest that maybe EnWiki got it wrong for some reason, and consistency across wikis is a reasonable factor to weigh. That would make a strong case for reconsideration of the EnWiki position. Now let's flip it the other way. English is almost half of our global community, and let's say a few other other major wikis agree with the EnWiki consensus. I think that would make a seriously strong case that the WMF shouldn't push a product out to the smaller wikis that can't/don't effectively participate. If wikis representing a majority of the global community consider it a problem then it's safe to conclude that it would be a problem for small wikis too. And again, consistency itself is a factor to weigh. Tiny wikis shouldn't have weird random configuration differences without a really good reason. A lot of people visit more than one wiki. It's undesirable for things to randomly appear or disappear.

It's a bad idea to do these bottom-up deployments, especially when there's a known problem. Micro wikis never reject anything, but that means they never accept anything either. The last thing you want is to deploy something to 200+ micro wikis that never speak up, then reach to the top few wikis and discover that the whole global community sees a problem with it.

I strongly urge RFCs on the other top wikis before the "deploy everywhere" step.

@CKoerner (WMF), I'm correcting myself. I mentioned a Russian discussion above, but I now see there were two discussions. The earlier discussion, which I missed, was posted as a question and it did get a meaningful discussion. (It was small by EnWiki standards, but reasonable in proportion to community size.) It looks like it was mostly positive.

I was particularly concerned when it looked like nowhere had really asked for Hovercards. I have no complaints about deployment to Russia. That reduces, but does not eliminate, my concern for the interests of other wikis. It looks like we basically have a small positive result at RuWiki, and a large negative result at EnWiki. Is there anything additional to indicate that silent wikis want it? (I'm trying to objectively evaluate the good-faith expectation, if they actively voiced their interests.)

One small correction.

1. Catalan
2. Greek
3. Russian
4. Italian

4, but who's counting. :)

I want to make sure I understand your suggestion. Are you suggesting to start deployments of features with the largest, most active, most vocal communities? Is that correct?

The near default method of deployment for software by the Foundation has been to start with smaller - and smaller interested communities - then build up to the bigger projects. It has many advantages - eager communities are more likely to adopt new things, working with smaller communities and discovering bugs - that don't mess with bigger projects! That sort of thing. With bigger projects - more people, more eyes - the alarm is often louder when things do break! :)

I see what you mean that these larger communities are more active - in articles, edits, page views, etc. I'd be curious to see numbers of percentages between say page views, active contributors and edits to the Project: namespace. That might be one way to amplify your claim, if I understand it correctly, that larger wikis would catch problems quicker/better than smaller projects.

Rereading your comment, this sounds a little like you're advocating for the smaller wiki's because "English Wikipedia knows better". That seems a little dangerous. I don't want to speak for communities I'm not familiar with, but I'm not sure they'd be comfortable - formally or not - with English Wikipedia leading decision making on what features their communities should consider. But then again, maybe they'd like a leg up? Something to consider.

>Tiny wikis shouldn't have weird random configuration differences without a really good reason

Of course. If you have an issue with the methodologies used and don't believe the work of the team was done with good reason, please explain them so we can improve.

It's not called out in our roadmap, but we are sending a notice to the Village Pump of communities in Phase 1. I have a draft of a short version and we're considering reusing the same announcement for the communities we have reached out to as well.

>It's undesirable for things to randomly appear or disappear.

Agreed, which is why we are approaching each community early, to the best of our ability, notifying them in an appropriate (most appropriate?) venue and giving them time to discuss.

>I strongly urge RFCs on the other top wikis before the "deploy everywhere" step.

You are more familiar with the RfC process than I. :)

Re-reading my reply I hope I'm not coming across as disparaging. I think the conundrum is that an RFC at the end of the development of a product is not a suitable replacement for early engagement with communities on changes. However, and in all honesty, this makes it sticky to talk about now as we're so far down the path for this feature. That's not to say I'm not enjoying talking about it or that it's not worth it, quite the opposite! I'm also/obviously talking to others in the team as well.

It's late where I'm at. Happy to continue the discussion with you later.

I am delighted with your reply. The subject of WMF-community engagement is sometimes awkward.

Early input on a project is definitely critical, especially prior to build phase. It has been challenging to build that early engagement process. Trying to engage a diffuse global community is hard. I fear that arguments for inclusivity have, at times, led to an all-or-nothing trap. The valid goal of inclusivity should not undermine easiest or "reasonable effort" engagement. I hate to single out EnWiki, but it shares the WMF's language and its size and experience make it the easiest and most effective point of engagement. EnWiki alone represents forty-odd percent of the global community. EnWiki plus two other big wikis covers a majority of the community. A top-down approach can get you far, for whatever outreach is manageable.

I'm not sure (small wikis would) be comfortable - formally or not - with English Wikipedia leading decision making on what features their communities should consider.

To clarify, all communities can consider and request anything. A main concern was how silent wikis are handled.

When smaller wikis can't or don't speak up, the rest of the community is their best representative. This is basically how we already manage wikis. There are a vast number of casual editors with no knowledge or interest in community management, and lots of major editors with no interest in doing that work. The most dedicated, most experienced, and most interested community members do the work of community-wide management. Everyone is welcome to show up, and the work is done by those who show up. I think our public-service mission and our culture of consensus makes the core community pretty good at trying to serve those who didn't show up.

I very much want a better voice for other communities. I'm sure smaller wikis feel powerless and voiceless. An announcement that invites an RFC is likely to be far more effective. People volunteer to do work when they know it's meaningful. A concrete question is key. In the pre-build phase it would be great to see something like "Is this something you think you would want". Along with that you can ask for feedback on features etc. That feedback will generally be very fuzzy unless there is a focused discussion on it.

Are you suggesting to start deployments of features with the largest, most active, most vocal communities? Is that correct?

Deployment is complicated and has competing factors. It absolutely does makes sense to start small, to get bugs fixed, to get initial feedback, and make sure things don't explode :) What I am saying is that top wikis evaluate it before global-hundreds-of-wikis deployment. Bottom-up deployments are risky. It's the "path of least resistance", but that path also leads directly to the worst case scenario. The WMF wouldn't built a product unless you believe it is beneficial, so the WMF is sometimes eager to deploy that benefit to silent wikis. Everything seems to be going great, then you get to the last few wikis and suddenly a majority of the global community objects. At that point the options get bad. Trying to "finalize" the project by pushing out the last few deployments can result in a crisis. Rolling back hundreds of deployments is ugly, especially if a rollback itself is somehow disruptive. Partial deployment may or may not be viable, depending on the product. But if the wikis who do speak up are objecting to deployment, the good-faith assumption should be that the silent wikis would also object. That can make the silent deployments look bad.

If a product is expected to be optional per wiki, I'd suggest testing small, then validate whether any engaged wikis actually find it desirable before pushing it to silent wikis.

If a product is expected to be "everywhere", I'd suggest testing small then (maybe) consider jumping to the top. If a universal deployment is going to blow up, EnWiki will give you that answer. Chuckle. It's a tough call. You want the product as bugfixed and improved as possible before getting there, but it's not good for virtually half of the community to be the last ones to say it needs a major change or that there's some fundamental problem. It mostly comes back to effective early engagement to

I see what you mean that these larger communities are more active - in articles, edits, page views, etc. I'd be curious to see numbers of percentages between say page views, active contributors and edits to the Project: namespace. That might be one way to amplify your claim, if I understand it correctly, that larger wikis would catch problems quicker/better than smaller projects.

And it's not merely a question of catching an issue. People need to believe they can do something meaningful to address it. The information needs to bubble up to somewhere useful. For small wikis, the WMF is impossibly far away. They don't feel they can do anything meaningful about the software, they don't have people focusing on it. This will grow, if people see they can contribute valuable work in the area.

Probably the best and simplest thing is to just to look at the community's equivalent of Village Pump. That's where generic wiki-wide issues are handled. That should show the organisational and human resources available for arbitrary engagement. That's where people show up looking to handle generic community work, where you'd find people who would opt-in to a beta feature specifically as community-service work testing it.

Phew, it's been a crazy few weeks. I had a moment to pop in and catch up.

>A top-down approach can get you far, for whatever outreach is manageable.

I think this may be the first time anyone has mentioned top-down in the context of Wikimedia. :)

>What I am saying is that top wikis evaluate it before global-hundreds-of-wikis deployment.

I think I understand what you're saying - English Wikipedia can appear to be "low-hanging fruit" when it comes to getting feedback, iterating, etc. However, we're a global movement and while that makes it hard we have to try (as much as we can) to treat each community equally. Showing favoritism, even unintentional, is something we try to avoid.

Fun fact I leaned since we last chatted. Did you know that the foundation use to do deployments on the English Wikipedia first? Apparently other communities didn't like it (and neither did our development teams). It even was mentioned as a process idea a few years back.

I am under the impression that this is exactly what the Beta Features process is for. Do you agree? We can create a way for users to opt-in to early features, test and provide feedback, and then deploy once stable and issues have been addressed. Side note: Page Previews has been a Beta Feature for quite some time. Which over 60,000 editors have enabled (more than the NavPopups gadget!) on English Wikipedia).

>And it's not merely a question of catching an issue. People need to believe they can do something meaningful to address it. The information needs to bubble up to somewhere useful. For small wikis, the WMF is impossibly far away. They don't feel they can do anything meaningful about the software, they don't have people focusing on it. This will grow, if people see they can contribute valuable work in the area.

How so? I can't speak for other WMF staff (or the WMF as a whole!), but in my work I make sure communication - to any community of any size - has a clear actionable point for feedback. A talk page, replying in-thread, pinging me on my talk page - something. Those don't seem "impossibly far away". In fact, looking at the IP addresses and home wikis of the very Talk page we are discussing on shows that folks can find us from all corners of the movement. :)

I don't know much about your editing habits, do you experience this frustration on a small wiki? Relevant discussions would be helpful here, even if I have to use translation software, to get the gist.

Another conundrum, and one that lends credence to your concerns, on smaller wiki's it can be very easy for the Village Pump (if they have one) to be nothing but posts by the WMF. We are one, communities are many. That's still a sticky point we're still thinking on.

I'm trying to help share some understanding on why we do things the way we do, listen to your concerns, and try to come to a shared understanding. I'm not sure if any of this helps, but I'm happy to continue the conversation - when I have time.

Back to the salt mines. :)

>>What I am saying is that top wikis evaluate it before global-hundreds-of-wikis deployment.

>I think I understand what you're saying - English Wikipedia can appear to be "low-hanging fruit" when it comes to getting feedback, iterating, etc.

I'm talking about more than just bug reports and iterating. The WMF has good intentions and great developers, but any company would go bankrupt if it tried to build banking software without talking to bankers first. If the WMF builds the wrong thing, it's bad to deploy it to hundreds of silent wikis and then discover the wrong thing was built. Everyone wants it gone or fundamentally redesigned.

Too many products have been running into problems, and too often the WMF side finds it difficult to understand why. Inviting bug reports isn't enough. The community are out there on the front end running the site on a day-to-day basis, we're immersed in the human aspects. Sometimes what we want may seem strange, but there are generally good reasons for it. The best developer team in the world can't build banking software without seriously engaging bankers about the design.

Ideally there should be effective community input before something is built. And independent of that, there needs to be effective community examination before mass-deployment to silent wikis.

The 2017 New Wikitext Editor was designed and built with zero community input. There is 90+% consensus at EnWiki to block deployment. The problem isn't "bugs". The problems are due to design. I don't know if it is fixable, and if it is fixable it would take a serious overhaul. We would be in a bad position if the 2017 New Wikitext Editor were mass-deployed to silent wikis, prior to effective community examination finding it undeployable.

Small wikis can have trouble just getting a routine bug submitted, and even harder time getting effective action on that bug. A micro wiki can't realistically engage on the design of a New Editor, and it can't provide serious evaluation it after it's built. And even if they could, you'd drown under the chaos of 200 languages. Everyone loses if the "equal/inclusive" goal has the effect of blocking or diminishing effective engagement. It shouldn't prevent large wikis from helping to ensure projects succeed.

>I am under the impression that this is exactly what the Beta Features process is for. Do you agree?

Beta Features is a valuable tool, but it definitely does not serve the same function. Imagine you make a Beta Feature that puts dancing kittens on every page. Some people may love it, and you may get bug reports about kittens that freeze up, but it doesn't answer "Should we deploy this as default for everyone".

>in my work I make sure communication - to any community of any size - has a clear actionable point for feedback. A talk page, replying in-thread, pinging me on my talk page - something. Those don't seem "impossibly far away". In fact, looking at the IP addresses and home wikis of the very Talk page we are discussing on shows that folks can find us from all corners of the movement. :) I don't know much about your editing habits, do you experience this frustration on a small wiki?

We experience the "million miles away" effect even on EnWiki. I can only begin to imagine what it's like for small wikis.

EnWiki is lucky that we have well known and active liaisons, but even then it often feels like tossing a message-in-a-bottle into a million mile ocean. Almost no one leaves their home wiki. There are a very few community members, such as myself, who essentially specialize as liaisons in the opposite direction. I've spent years familiarizing myself with mediawiki wiki and meta and phabricator, and tracking what goes on over here. It's so costly that it often leaves me no time to do regular editing and community work.

For example: I don't read Polish, but I happened to Google Translate their Village Pump regarding a WMF-community matter. I happened to stumble across an unrelated unanimous consensus over on Polish wiki, objecting to a bug in the SingleEditTab deployment. (The bug was fixed for EnWiki, but nowhere else.) And how was it being handled? It was just going to disappear into the page-archives... because the WMF is a billion miles away, no one from the WMF caught it, and no one at the wiki could bring it to you.

I had to bring the Polish request to phabricator myself. Small wikis don't have specialists like me, who bring community concerns to Phab/meta/mediawiki and then track what happens with it. It's especially hard when the candidate pool for such specialists is limited to people who know English. And simply bringing it to Phabricator isn't enough. I have spent three weeks tending to this item, and I still can't get a response whether the SingleEditTab bug will be fixed on Polish wiki... and whether it will (hopefully) be fixed everywhere.

The 2017 New Wikitext Editor is another example. In the WMF-universe this project was "publicly posted". However the number of community members who visit WMF wikis is close to zero. And even for the few of us who do venture over here, the information was impossible to find unless we blindly stumbled across it. Two or three of us did find it. We tried to ask for information on the project. No information was available. We had to wait for the initial version to be completed.

Oops. Another product designed and mostly-completed as an internal project with zero community input. No community input on a project to replace our most critical tool. For all practical purposes, it was designed and built on a different planet.

As soon as the prototype was released, several of us desperately tried to tell the team that the design was wrong. We desperately tried to raise the biggest clearest red flags we could. We stated that there were problems, we stated that there would be a community consensus against it, we said that it couldn't be deployed. We tried for months. Eventually the only option left was to abandon the WMF channels and just plain deal with it as a community matter. Six weeks ago an RFC was opened. It has 90+% consensus to block deployment due to these design issues. The WMF still hasn't engaged the issue in any constructive manner. We're living on two different planets.

The WMF generally does well at listening to bug reports. The WMF eagerly wants to hear enhancement requests. But the WMF still has difficulty allowing any input on design phase, or on whether something should even be built in the first place. And once the product is built, the WMF has a hard time seriously considering anything that would require fundamental changes to what was already decided and built. And all of this is from a EnWiki perspective. For smaller wikis, it's just plain hopeless. They're too small to be taken seriously. The battle to be heard is just to hard. They don't have the population to have people focusing on these issues. The language barrier means almost no one can even try.

We started talking about the consultation and deployment of Page Previews but got into the old trope of how the WMF does product development work in general. :) I can't speak to the projects you describe as I was not part of their development (nor am I currently) and we're surely taking up more space on this talk page than necessary. Perhaps there's a better venue to continue the discussion?

Sorry for wandering so far from the original topic, chuckle. If you had a specific reply to something, you can always jump to my talk page. I'm always interested in anything that could potentially lead to smoother relations between the WMF and community.

Hey Alsee, I see your message here, but it's late on a Friday where I'm at. Monday is a holiday, and then I'm traveling to San Francisco for some annual plan...planning the rest of the week. I wanted to let you know I hear ya, I'll think on this, and get back to you when I have a moment.

I'll ping the closer, Dank, in case they would like to agree or disagree with my comments regarding their close. My "optimistic 2-to-1 opposed" was based on the initial 7 responses that you didn't include in your tally, as well as assuming that bugfixes or other improvements converted mixed-votes into supports.

Alsee, I was following a random link to meta today and found out you pinged me 3 days ago here ... Generally, I only get pings left on en.wp. Reading quickly, it looks like the hovercard feature will not be enabled by default for registered en.wp users, which (for them) is in line with those RfC results. For unregistered users ... I'm not sure what I think, let me ask around. As soon as I have something, I'll ping you on en.wp.

Dan

Thank you for making this post here first. From my point of view you should create another discussion (the other one is almost a year old now) and there pay attention to two things:

• Make sure that those who use or have used the hovercards feature participate. For this e.g. make a post about the discussion here or show a banner somewhere to all users who make use of this feature.
• What arguments are made in opposition to the default-rollout? Address these issues by either changes to the software or at least a reply.

Once the 2nd discussion is closed examine what the main issues are, how significant they are and what could be done about them and how many oppose the rollout due to these issues.

If there's for whatever reason rough consensus against or no rough consensus for a potential way of conduct would be enabling it by default for just a group of Wikipedia readers and getting feedback from them. Then, with this feedback-data, create another discussion.