Wikimedia Engineering/Product Strategy Offsite

Meeting Notes 

Howie: Thank you everybody for taking the time to be here. I wanted to kick things off by talking a little bit about why we are doing this. This past year was a really big year for us, please take a look at the chart on page 2, the chart basically shows the number for features that we’ve collectively, as a team, have released this past year and I think that everybody should feel really good about the work that we’ve done - it’s really awesome.

At the same time, I don’t think that we’re operating as a team, people generally know what their teams are working on and why they are working on them - we don’t have a shared goal across all of our teams. I’ve read feedback that people don’t really know what the big picture is, so part of what we wanted to do over the next 2 days is to create that big picture together.

In terms of the agenda, we’re going to spend the first hour reviewing some material, this is to give us some base of knowledge on our projects. We’re also going to have each individual team share what they have in mind for next year, keep in mind that these are not firm plans for next year... In the afternoon we are going to have conversations around interdependencies... Tomorrow morning, we’ll talk about the next 2-3 years out, so ask ourselves, what types of things need to be in place, how are trends that are affecting the internet impact us and how do we as a team and a movement respond...

...Part of what I think we don’t do well is describing our work internally (to the foundation) and externally to the community, so we’re going to go through an exercise in describing our work. Any questions?

Erik M.: I think you said the important stuff already, Howie, is that we now have a lot more things in play than ever before. I wanted to ask here in the room who was there during the vector roll-out either as a volunteer, or as a staff member? So that’s like a third or less of the room here. Part of what’s happened in the foundation is that we now have vector-sized projects in play in multiple departments at the same time. Vector was a big deal in 2009/2010 and it actually meant that all of mediawiki core and the other parts of the development community needed to be aware of what was happening and we did do an okay job with that and over time managed to embrace it as part of the mediawiki technology ecosystem, but now we have a lot of things in play and our capacity to grapple with that is pretty poor. Particularly, as Howie mentioned, if you think about Flow, Flow is about discussions and discussions are an extremely important part of collaboration, collaboration in our context means working on articles, so we really have to figure out what discussing an article should look like in 2013/14 instead of necessarily just copying the model of the talk page or even copying models of other sites. So we have to examine things in context, in the context of these many vector sized projects that we have in play....

---Team Introductions---

Howie: ...when we look across our projects, there are four key disciplines involved with our work - there’s product, engineering, design and data. When I look at our projects, I can think of us doing any one of our projects without those four disciplines, so that’s why we have the people that we have here. I do want to emphasize in terms of how we work that these four disciplines are peers... we’re a collaborative organization and the spirit of collaboration needs to be present in how we work... Any questions on this notion?

Terry: ...we do a lot of stuff like Howie mentioned and there’s a lot of us... what I’d like to come out of our conversations is that if we could come-up with principles/habits/frames/patterns - whatever, to describe the things we do and the reasons we do them, so that even if I was about a project that was in Mobile, I could answer the same answer Tomasz would answer because I know we’re operating from the same playbook as each other... whenever we do things, I think we’re really good at focusing on small things and we’re very good at talking about the vision, but we’re not good at bringing those things together... I think this would be an amazing outcome.

Howie: Alright, let’s start it then. Let’s go to page 3... some of you are pretty new and I’m not sure if you have seen this information, so I’m going to walk through this pretty quickly and it’s broken out into forces that shape our existing community, drilling down to our first edit, talking about the edit-reversion system, and finally the career path of Wikipedia. As I go through this, I’m probably going to skip through a lot of slides, but I wanted everyone to this information in one place so you can hopefully take a look at it in your spare time.

So let’s go down to page 4, I hope everybody has seen this graph, this is the graph of active editors on en wikipedia, and what I did here is I’ve broken this up into 3 phases: start-up phase 01-05, hyper growth phase 05-07, maturity phase 07-present... this is kind of a retrospective... If we go to the the next page, let’s drill down into this hyper growth phase because I think a lot of what’s happened in our community, has been a result of what happened between 2005’ish and 2007.

So what we saw during that time period was an increase interest in Wikipedia... a lot editors were entering the system and what ended-up happening, was to deal with the onslaught of new editors, the community put down a lot of processes to basically ensure that the content would be of high quality.... during this period, there was a lot of focus on increasing the quality of the encyclopedia. Another thing that happened during this period was we started to see the introduction of bots and automated tools, and again, part of this was to deal with the onslaught of new users and a focus on quality. So Twinkle was introduced in January 2007, Huggle was introduced in January 2008, so as you can see, during this hyper growth period, there was a lot of things that happened in the Wikimedia community and a lot of these things actually exist today. This is kind of the legacy of this time period.

And if you go to the next page, this is the retention rate of users and how that retention rate dropped during this period......

Jared: Do we have any understanding of how this looks against other online communities? ...like these same patterns, same numbers?

Brandon: The wiki-verse, others tend to have the exact or very similar growth graphs. The one exception is actually wikiHow and that’s just because they started turning it around, so they actually have an uptick.

Steven: So did they see a similar dip happen?

Brandon: Yes, they did see a similar dip happen, but they actually were, interestingly, in a position where they could start experimenting with gamification with their software in ways that we and Wikia cannot, or have not...

Ori: ...looking at edits made by new editors gives you a sense of their quality along one dimension of quality, but there could be other factors like general computer savviness and things like that may influence retention that isn’t necessary reflected in the substance of the edit itself.

Steven: There’s also standards...it’s really obvious just from looking at policy and how experienced editors react to new editors that a large degree has to do with increasing standards for quality....

James: ...the first response you’d likely get as a new user is, “Hey, I see you made this change, this is not allowed, this is not the policy, read the policy first please.” Even if it was nice, it was still bad...

Howie: ...last year at Wikimania, there was a session on wiki culture...where there was kind of resounding agreement in the room that of those people that had been editing Wikipedia for a long time that if they made their first edits today, they probably would have not gone back. There’s a recognition that the system is not as encouraging for experimentation and as encouraging place as it used to be.

Jared: When you talk about the quality of a new users edit...?

Brandon: ...when we were stuck in ACTRIAL and we were having a serious problem, the ACTRIAL was this thing that happened two years ago where the community came together and they decided that in order to create an article, you had to have auto confirm status which was a high bar for a new user, and the people who were claiming were saying that all the new articles that were created by people that don’t have auto confirm status are really low quality, and by that they mean no references, or one sentence stubs...

Maryana: So by high quality, you just mean that it wasn’t vandalism, it wasn’t a test edit, it wasn’t malicious, it was fairly legitimate...

Jared: Quick question, can someone define the exact meaning of this retention rate graph?

Howie: Yeah, it’s users that have joined Wikipedia. So let’s take a look at January 2004, so that’s a cohort of users that created an account in January of 2004 who had at least 10 accumlative edits, the subset of which ended up making at least 1 edit in January of the following year.

Let’s move on to the next page which gives a list of bots and automated tools...some of these are actually helpful...most of these I think are bots that have helped the community deal with the onslaught of new work, but may have had unintended consequences when it comes to how the community treats newcomers. For instance, when you think about Twinkle, it’s a very helpful automated tool, but the way the interface is, it actually lends itself to some pretty harsh interactions which we’ll actually walk through.

James: I just wanted to jump in here “unintended consequences” -- possibly assumed too much good faith. [point here is that some of the tools had the intention of, for example, pushing users away

Erik M.: To be fair, we are really dealing with an increase in vandalism...there is inexperience like if you’re in the process of being a new page patroller or a recent changes page patrol, you do see a lot of spam, vanity biographies, conflict of interest edits that puts you into this (frustrated) mentality...

Brandon: Huggle keeps a kill tally as you go, it literally gamifies destroying pages

James: And users

Howie: The next page shows the percentage of messages to new users split up between actual human messages and bots...and what you can see that the rise of automated messages increases dramatically between 2006 and 2008...I think it’s something like 80%’ish of all new user messages are delivered by either a bot or some kind of automated tool...

Siebrand: I think this is exactly the same graph we saw last year; is there an update?

Howie: No, it hasn’t been updated

Erik M.: A lot of this data came from the Summer of Research - a one time fundraising effort that Zack organized....

Siebrand: ...it would be interesting to see what the current situation is...

Brandon: ...Mako released a paper yesterday that says that our gender gap is completely different - closer to 33%

Howie: So page 9 summarizes these forces...(policy proliferation, rise of bots/automated tools, change in mentality from fun/creation to protection)...Let’s move on to the first edit...look at some of the work the E3 team has done...(review of page 10, 11, 12 and 13)

Terry: What’s the other 14%?

Maryana: Probably other tools, I mean there are a bunch of other surrounding tools...

Howie: (page 13)...if a newbie gets reverted, they typically go back and find that the article is missing...so that pointed to the absence of notifications...revert is off by default for new users, is that right?

Fabrice: Yes, we took it off because it was a slap in the face of the new user...

Dario: ….

James: I think that not telling them that they’re going to be reverted is more of the slap in the face...

Howie: ….users don’t necessarily know that talk pages exist...that points to the need for a discussion system....and even if they do find their talk page and see that there are messages, there’s a lot of ambiguity on who these messages are from - are these from bots or users...this points to the need for profiles that users can understand...so this type of analysis led us to the next page (page 14)

Erik M.: Do you want to speak briefly about affiliations? ….

Howie: Yes, so affiliations is the last piece.... Wikipedia doesn’t do a good job of giving you other things to do...we know that when users move through the life cycle, they tend to edit based on a topic area...if you are actually interested in say wine, there’s no way that the system reaches out to you and draws you in. What ends-up happening today is somehow you have to find a wiki project, so the notion around affiliations is that users have an interest rout whether they express it explicitly or whether we can infer it...the notion of affiliations is matching that interest rout with work that needs to be done on the wiki. (review of page 15 and 16)

Tomasz: Are majority tools usually English based?

Howie: Yes

Tomasz: So purely English, no other languages use these tools?

Maryana: Yes, they do actually...German, Portuguese...

Siebrand: It takes significant community effort to actually make these tools work.

James: There’s also a tool you can’t use on en wiki because of a preference change we agreed to in 2005 called …

Howie: (review page 16 and 17)

Matt: What does the playbook actually do?

Howie: It starts the whole thing. I actually have a computer with Huggle loaded in the back (of the room) if you guys want to take a look at it...the next page is the interface and the resulting message... So you can see that as a new user, if you got these messages, you’ll be like, “What’s going on?”

Brandon: In the user tests that we ran about this, we actually used the nice template and it 50/50 whether or not a users can detect if a human left it or a robot.

Kaldari: Isn’t this a little bit outdated though because on en wiki we have a page creation tool now which about half or less of the page curators are using now, we actually built that with a lot of this in mind to try to improve it, where we put a lot more options for positive feedback like wiki love, and then we also tailored the warning messages to be more personalized and less verbose...to hopefully improve the experience for people who were creating the articles...

James: …

Kaldari: Yes, it’s only for article creation and it’s only on en wiki

Fabrice: Also the number has declined greatly, it used to be at 56% and it’s down to 15%....

Brandon: Do you know why?

Fabrice: No, no yet.

Howie: The next slide gives you another piece of analysis...what we see over the past couple of years is that the percentage of templates that involve criticism has been growing quite a bit and the ones that involve praise is rather sad. The next page gives you reversion rates... The next page shows that many editors end-up leaving because of conflicts with existing editors...

James: It’s also the number one troll complaint on news articles...

Tomasz: What’s the date on this?

Howie: 2010

The next page if about finding the right kind of contributor and this is just more food for thought...

Brandon: ...this speaks to a philosophical question I know we had for several years about the distinction between whether wikipedians could be created or if they are found...and this reads to me that we find them, we can’t make them...

Howie: My position is that if we really believe that Wikipedians are made not born and we think the interface is good enough to attract the ones that are born, then what are we doing here?

Brandon: What my view here is that Wikipedian is kind of a mindset...and I think our software is a barrier to have most of those people willing to join...

Steven:...I think the things Wikipedians are saying that motivate them we could map that …

Howie: (review of page 22)

Kenan: Despite the fact that the software is the same?

Howie: Yes, despite the fact that the software is the same and that was Joel’s main point....

Fabrice: ….for the past decade we’ve basically assumed that anyone can edit is the right thing to do and I’m getting growing concerns that the basic principles may need to be revisited. A lot of the negativity that we get is in a large part do to the fact that anyone could edit that our editors are so focused on dealing with vandals that perhaps having some small barriers to entry, very minimal ones, ... could potentially be beneficial.

Howie: We’ll talk about that more tomorrow morning. We’re running 5 minutes over, so I just want to point people to a couples of things (review of page 46, then to page 24) ...when we take a look at users involved in the meta spaces, a lot of them are class of 2007 and before...(review of page 28)

--- Break---

Howie: ...the goal of this is for each of the areas to describe what they had in mind for next year and kind of why, so that we can hear as a group what everybody is working on. … The spirit of this this is really for understanding.

James: So, VE goals for 2013/14, so we are now the default editor on one wiki right now out of 900... We are not the best editor we can be; we do not cover all content, we do not enable all users to make preview edits without having to worry about wiki text and these are things we want to fix. We’ve got the here as continue to develop VE features; fixing bugs and stability; improving scalability and performance. … We’ve kind of broken up the year in kind of a bunch of these we’d like to do - a lot performance improvements and bug fixing, so that things that are meant to work should definitely work even if you’re using an iPad... And things that aren’t meant to work should consistently not work in the same way depending on whether the article is in French, Hebrew, English and things like that. Performance is a big pain point for our users and thus for us right now. VE is necessarily quite heavyweight and necessarily more heavyweight, and it’s getting rid of the unnecessary weight both in terms of processing, speed, for loading but also for execution....

So for the first half of this year, we are focusing on scalability and performance. …throw-in a much better set of workflows around templates and references…table editing….file uploading….meta data editing … explore concept on VE as a platform versus a mode…

Trevor: …what we produce is like this minimum viable product…we just got something out there because we knew this progressive enhancement model which we’re using where a lot of the content isn’t editable was going to be superior to building up this huge tower and releasing it…tons of features have very little re

James: Erik said a couple of times that what we built is a visual explosion of wiki text editing concept. it’s not so much that now it’s data graphics, but we haven’t rethought the workflow...

Erik M.: …as we start thinking about these user interfaces, we probably want to think about just adding a category on the page - you don’t want to go into an edit….

Trevor: …we have begun abstracting certain things, like when you reuse a reference, we automatically name both the new reference and the other one - we’ve completely abstracting that...with categories, we preserve where categories are located on a page, but we put them all together in one piece…people have a mental model of the way the markup works...we have to transition them.view of way interactions happen and we’ve started engaging with designers to fix a lot of the stuff...a lot of the processes of doing various tasks are very inefficient and we’re working on improving those…we just tried to make sure we could make it work before we fussed about how easy it is...

Erik M.: …regrets that we don’t have C. Scott or Gabriel here because that’s the other big component of VE …we’re changing the way wiki text is translated into html and back, but we are also potentially changing the way content is stored in the first place... so might ultimately move to html 5 as the native storage form...that’s a really big deal...and we change from wiki text to html 5 as the model for templates, that really changes everything...we need to think about what does it actually mean if we move to html 5…

James: ...so why wiki text sucks is not just because people have to learn it, but we are restricted by how it works...and there are things in VE that we could not build based on wiki text...

Howie: Can you give us an example?

James: For example, inserting an image pulls-up gallery images …. In terms of what we learned about our users, we’ve learned or rather witnessed that how users are attached to the tools that have brought them to where they are right now and they are unlikely to change graphically unless we make it compellingly obvious as to why they should do that... new users are flocking to use VE...

Brandon: Have we published these user tests anywhere?

James: I think so, it was done 2 months ago

Trevor:  …we’ve actually gotten a lot of advance users to spend a significant amount of time failing on our product and giving us good bugs and stuff like that, so that’s been kind of an upside.

Kaldari?: Do we know anything about the impact of total number of edits yet?

Trevor: ... we don’t really have great data yet, but hopefully will have that soon.

Ori: Not hopefully, it won’t happen unless there’s a commitment to it.

James: (talks about visual diffing, edit complex)

randon: Is cut and paste on here?

James: That should be next week …

Howie: Let’s move on to Mobile…

Kenan: At a high level, what we’re looking to do is to make it easier to contribute through Mobile. In the past couple of years from a readership standpoint, we’ve gotten mobile to a good point and now we’re looking to open it up to make it easier for people to contribute, last that that was through uploads, uploads of photos now we’re looking at it more broadly ...editing text, making contributions through gnoming...goal for team is to get more contributions…Mobile presents a different opportunity for us to get users through a different workflow… James and I have been talking a little bit about what it means for VE to go to Mobile…feature parody in terms of Echo, talk pages...

Maryana: …when and if we do figure out the VE/Parsoid piece, we’ll actually be able to start identifying certain pieces of wiki text that we’re interested in, so highlighting things for people to do on a touch interface rather than having to go into the editor and do mark-up stuff…add links, add category...also open-up photo upload stream...identifying sections....figure out what kinds of small easy things we can get users to do in a really structured way… People are really excited to try our stuff out…we can get a bunch of new people in, but we’ll have to be really careful and give them really structured workflows to work off of and not just tap everything...

Tomasz: In apps we’ve really been focusing on our power users. Last year we looked at photo uploads, multi-file uploads, refining the upload process. This year we’re going to be doing more, but what we’ve noticed that that while we can acquire a lot of new users, retaining them is hard, so we’ve started looking at campaigns…we want to build-out and we’re currently exploring what the costs are for a generic infrastructure...where we build it for the community as a whole so that campaigns can be build year round...focused contributions within designated areas...meet our 6000 contributor mark across web and apps. For the second half of this year, we’re having open discussions about what it means to start retooling some of the native Wikipedian apps...how they’ll tie into VE...there’s some technical hurdles will have to solve...

James: Are we currently based on PhoneGap for the apps?

Tomasz: Were based on PhoneGap for all apps except for the commons app. The commons app in our native application and it was our first big dive to see if this makes life easier for us and the answer is yes it does.

James: Unless you want to integrate a bunch of javascript in VE.

Ori: Have you guys seen graphs of Android share of new computing platforms?

Tomasz: ...2.2...2.3 is plummeting as well...

Ori: ...computing platforms that people have is pushing out everything else...

Tomasz: ...exciting to see 4.X take a much larger part of the market...

Vibha: Why are we focusing on photo upload campaigns to meet the 6000 contributor mark? …

Howie: We can dig deeper this afternoon. … These campaigns will require a lot of support… we need to think about what an MVP could be…

Siebrand: ...In last year's discussions, we spoke about how the design for our site for mobile and our desktop design could or should converge.

Trevor: I think we’ve pretty much decided that we’re gonna do that...

Siebrand: But I don’t see it stated explicitly anywhere on the major projects for either of these two teams.

Howie: I think that’s a part of this exercise...

Jared: ...I think that just kind of doing my own audit of all the historic, very successful projects...they’ve formed organically, they’ve solved the needs of a particular set of users and stakeholders of the project, but because of that, they aren’t holistic in any way both visually and with interaction patterns. So I think for us (design team) the goal this year is that we need to renew that audit and figure out where things differ, figure out the best of each of these and codify those to the visual design and interaction patterns that we can then distribute to everyone...which will hopefully cause future projects to use that tool kit of patterns and visual design assets...to look consistent...we gonna accomplish that by starting to socialize inneractive change with users, people are very averse to that...beta experiments framework...

James: ...all our current skins will have to be burnt to the ground and start again...if you are going to make the big rewrite work, we should probably work with you...

Jared: There’s definitely going to be some slight of hand change at some point...

Maryana: E2, really just one feature: Flow. The goal is to modernize user talk...we gonna start with really basic user to user messaging… Second half of the year…concentrate on user to user piece...wiki projects collaboration piece….letting users follow projects and things they are interested in and continue to receive information in things that they are interested in without relying on crazy copy and pasting templates...

Vibha: What’s the validation? …

Maryana: ...I don’t think we’ve talked about what kind of metrics we’d use to measure the success of something like Flow, but a big part of the first release is figuring out what do we actually want to achieve with this product...

Howie: We have a good model for evaluating linear flows...but we don’t have a good model for things that fall outside of that, so Echo and Flow are examples of that...

Steven: What E3 does is that we built features that grow that active contributor base... the goal for the team this year is to gain 2400 additional active editors to Wikipedia in any language…categories of experiments…increasing conversion rates...internationalizing...in terms of retention, that’s an area we haven’t spent a huge amount of time on...reverts...reactivation...we have to build a robust experimental tool chain to go along with our features that we build, that includes event logging, navigational timing...

Jared: ...E3 does a really good job of instrumenting things for analytics and then actually using that data in a very quick way, and I know that’s a lot of overhead, but do you think that their could be some kind of sharing of that learning and how other teams could more effectively use data as quickly as you have? Or do you think it’s something that’s particular to what you’re doing?

Steven: No, I don’t think it’s something that’s special to E3, it’s a way that we work and I think there are aspects of it that any team could use...good example is user metrics api user metrics...

Ori: …there’s a lot of mistakes that we’ve made that I think that we can spare people the cost of making the again...I think we’re doing well, but I don’t think we’re there yet in terms of being data driven...there’s a while to go still.

Erik M.: There’s also the fact that data is still an underrepresented discipline in the organization today compared to Product, etc...The way I think Toby thinks about it is that he’s trying to grow this team of analysts with Dario who will be essentially consulting to the highest priority product teams in the organization and accompanying the development process throughout...

Fabrice: ...Kaldari has been kind enough to step-in and represent an engineering perspective as far as multi-media is concerned...muti-media is actually spread-out across a range of different areas starting with improving the viewing experience to enabling more contributions to helping curation of new content and most importantly publishing that content on the articles where most of our readers can come to, so we’re looking to make some incremental progress in all of those areas...readers expect richer media experience...For the first half of the year, we plan to focus on viewing and contributions...file notifications...improving upload pipeline...encourage people to add file categories...batch uploads...In the second half of the year, we’d like to turn our attention to the important topic of feedback and or curation...we hope to able to work with the VE team to expand the insert media tool...maybe introduce the concept of tags...

Siebrand: Are these in order of priority....

Fabrice: It’s not so much priority, it’s more that you got to pick a few things first...this quarter, we can have the most bang for the buck by improving the viewing experience, it’s not a difficult problem to solve, curation is difficult and so are search and meta data, which is why we plan to tackle these in 2014 ...

Siebrand: So two big time boxes and you’ll see how far you can get?

Fabrice: Exactly. I should also say that we have a lot of technical debt...for a long time there was no multi media team, so we did hack after hack after hack...

Siebrand: LE – We do a lot of the plumbing that allows our software to be available in many languages, there are also parts of making software available in other languages that are not part of our core activities, and then it comes to something that Steven mentioned earlier as in localizing the workflow - we ensure that it can be done, but we are not actually doing it, that something that either the features team has to facilitate, or the community has to do by itself. …we have to ensure that the core content creation and delivery platform of Wikimedia remains able to provide a localized user experience. What are we going to do this fiscal year…we want to develop a tool for translating main namespace content. We want to use the latest technologies for that, things like VE and that’s also a risk here because we had actually planned to start developing this feature this week, but given some of the very severe internationalization related issues in VE at the moment, it may delay for a few months. We want to provide onscreen keyboards…things like key mappings, users have to be able to find out how to use them…we provide help links….a gamification of the translation process…we assume that adding these features might encourage people to translate more… improve security risk…in scope the features we have in media wiki, we want to untie them…

Ori: Just want to express appreciation for the huge amount of support you guys provide to other teams...

Siebrand: You’re welcome. And to all product and development managers, please don’t hesitate to ask our opinion in advance, even before code is being written because it may help you do less rework.

Breakout Session (interdependencies in upcoming year)

Performance & impact on user experience; Consistent use of dataVE on FlowVE on MobileGeodataCross-device Design and DevelopmentInter-team processUnified i18n planningFirst-time UX