Wikimedia Engineering/Product Strategy Offsite

Meeting Notes 

Howie: Thank you everybody for taking the time to be here. I wanted to kick things off by talking a little bit about why we are doing this. This past year was a really big year for us, please take a look at the chart on page 2, the chart basically shows the number for features that we’ve collectively, as a team, have released this past year and I think that everybody should feel really good about the work that we’ve done - it’s really awesome.

At the same time, I don’t think that we’re operating as a team, people generally know what their teams are working on and why they are working on them - we don’t have a shared goal across all of our teams. I’ve read feedback that people don’t really know what the big picture is, so part of what we wanted to do over the next 2 days is to create that big picture together.

In terms of the agenda, we’re going to spend the first hour reviewing some material, this is to give us some base of knowledge on our projects. We’re also going to have each individual team share what they have in mind for next year, keep in mind that these are not firm plans for next year... In the afternoon we are going to have conversations around interdependencies... Tomorrow morning, we’ll talk about the next 2-3 years out, so ask ourselves, what types of things need to be in place, how are trends that are affecting the internet impact us and how do we as a team and a movement respond...

...Part of what I think we don’t do well is describing our work internally (to the foundation) and externally to the community, so we’re going to go through an exercise in describing our work. Any questions?

Erik Moeller: I think you said the important stuff already, Howie, is that we now have a lot more things in play than ever before. I wanted to ask here in the room who was there during the vector roll-out either as a volunteer, or as a staff member? So that’s like a third or less of the room here. Part of what’s happened in the foundation is that we now have vector-sized projects in play in multiple departments at the same time. Vector was a big deal in 2009/2010 and it actually meant that all of mediawiki core and the other parts of the development community needed to be aware of what was happening and we did do an okay job with that and over time managed to embrace it as part of the mediawiki technology ecosystem, but now we have a lot of things in play and our capacity to grapple with that is pretty poor. Particularly, as Howie mentioned, if you think about Flow, Flow is about discussions and discussions are an extremely important part of collaboration, collaboration in our context means working on articles, so we really have to figure out what discussing an article should look like in 2013/14 instead of necessarily just copying the model of the talk page or even copying models of other sites. So we have to examine things in context, in the context of these many vector sized projects that we have in play....

---Team Introductions---

Howie: ...when we look across our projects, there are four key disciplines involved with our work - there’s product, engineering, design and data. When I look at our projects, I can think of us doing any one of our projects without those four disciplines, so that’s why we have the people that we have here. I do want to emphasize in terms of how we work that these four disciplines are peers... we’re a collaborative organization and the spirit of collaboration needs to be present in how we work... Any questions on this notion?

Terry: ...we do a lot of stuff like Howie mentioned and there’s a lot of us... what I’d like to come out of our conversations is that if we could come-up with principles/habits/frames/patterns - whatever, to describe the things we do and the reasons we do them, so that even if I was about a project that was in Mobile, I could answer the same answer Tomasz would answer because I know we’re operating from the same playbook as each other... whenever we do things, I think we’re really good at focusing on small things and we’re very good at talking about the vision, but we’re not good at bringing those things together... I think this would be an amazing outcome.

Howie: Alright, let’s start it then. Let’s go to page 3... some of you are pretty new and I’m not sure if you have seen this information, so I’m going to walk through this pretty quickly and it’s broken out into forces that shape our existing community, drilling down to our first edit, talking about the edit-reversion system, and finally the career path of Wikipedia. As I go through this, I’m probably going to skip through a lot of slides, but I wanted everyone to this information in one place so you can hopefully take a look at it in your spare time.

So let’s go down to page 4, I hope everybody has seen this graph, this is the graph of active editors on en wikipedia, and what I did here is I’ve broken this up into 3 phases: start-up phase 01-05, hyper growth phase 05-07, maturity phase 07-present... this is kind of a retrospective... If we go to the the next page, let’s drill down into this hyper growth phase because I think a lot of what’s happened in our community, has been a result of what happened between 2005’ish and 2007.

So what we saw during that time period was an increase interest in Wikipedia... a lot editors were entering the system and what ended-up happening, was to deal with the onslaught of new editors, the community put down a lot of processes to basically ensure that the content would be of high quality.... during this period, there was a lot of focus on increasing the quality of the encyclopedia. Another thing that happened during this period was we started to see the introduction of bots and automated tools, and again, part of this was to deal with the onslaught of new users and a focus on quality. So Twinkle was introduced in January 2007, Huggle was introduced in January 2008, so as you can see, during this hyper growth period, there was a lot of things that happened in the Wikimedia community and a lot of these things actually exist today. This is kind of the legacy of this time period.

And if you go to the next page, this is the retention rate of users and how that retention rate dropped during this period......

Jared: Do we have any understanding of how this looks against other online communities? ...like these same patterns, same numbers?

Brandon: The wiki-verse, others tend to have the exact or very similar growth graphs. The one exception is actually wikiHow and that’s just because they started turning it around, so they actually have an uptick.

Steven: So did they see a similar dip happen?

Brandon: Yes, they did see a similar dip happen, but they actually were, interestingly, in a position where they could start experimenting with gamification with their software in ways that we and Wikia cannot, or have not...

Ori: ...looking at edits made by new editors gives you a sense of their quality along one dimension of quality, but there could be other factors like general computer savviness and things like that may influence retention that isn’t necessary reflected in the substance of the edit itself.

Steven: There’s also standards...it’s really obvious just from looking at policy and how experienced editors react to new editors that a large degree has to do with increasing standards for quality....

James: ...the first response you’d likely get as a new user is, “Hey, I see you made this change, this is not allowed, this is not the policy, read the policy first please.” Even if it was nice, it was still bad...

Howie: ...last year at Wikimania, there was a session on wiki culture...where there was kind of resounding agreement in the room that of those people that had been editing Wikipedia for a long time that if they made their first edits today, they probably would have not gone back. There’s a recognition that the system is not as encouraging for experimentation and as encouraging place as it used to be.

Jared: When you talk about the quality of a new users edit...?

Brandon: ...when we were stuck in ACTRIAL and we were having a serious problem, the ACTRIAL was this thing that happened two years ago where the community came together and they decided that in order to create an article, you had to have auto confirm status which was a high bar for a new user, and the people who were claiming were saying that all the new articles that were created by people that don’t have auto confirm status are really low quality, and by that they mean no references, or one sentence stubs...

Maryana: So by high quality, you just mean that it wasn’t vandalism, it wasn’t a test edit, it wasn’t malicious, it was fairly legitimate...

Jared: Quick question, can someone define the exact meaning of this retention rate graph?

Howie: Yeah, it’s users that have joined Wikipedia. So let’s take a look at January 2004, so that’s a cohort of users that created an account in January of 2004 who had at least 10 accumlative edits, the subset of which ended up making at least 1 edit in January of the following year.

Let’s move on to the next page which gives a list of bots and automated tools...some of these are actually helpful...most of these I think are bots that have helped the community deal with the onslaught of new work, but may have had unintended consequences when it comes to how the community treats newcomers. For instance, when you think about Twinkle, it’s a very helpful automated tool, but the way the interface is, it actually lends itself to some pretty harsh interactions which we’ll actually walk through.

James: I just wanted to jump in here “unintended consequences” -- possibly assumed too much good faith. [point here is that some of the tools had the intention of, for example, pushing users away

Erik: To be fair, we are really dealing with an increase in vandalism...there is inexperience like if you’re in the process of being a new page patroller or a recent changes page patrol, you do see a lot of spam, vanity biographies, conflict of interest edits that puts you into this (frustrated) mentality...

Brandon: Huggle keeps a kill tally as you go, it literally gamifies destroying pages

James: And users

Howie: The next page shows the percentage of messages to new users split up between actual human messages and bots...and what you can see that the rise of automated messages increases dramatically between 2006 and 2008...I think it’s something like 80%’ish of all new user messages are delivered by either a bot or some kind of automated tool...

Siebrand: I think this is exactly the same graph we saw last year; is there an update?

Howie: No, it hasn’t been updated

Erik: A lot of this data came from the Summer of Research - a one time fundraising effort that Zack organized....

Siebrand: ...it would be interesting to see what the current situation is...

Brandon: ...Mako released a paper yesterday that says that our gender gap is completely different - closer to 33%

Howie: So page 9 summarizes these forces...(policy proliferation, rise of bots/automated tools, change in mentality from fun/creation to protection)...Let’s move on to the first edit...look at some of the work the E3 team has done...(review of page 10, 11, 12 and 13)

Terry: What’s the other 14%?

Maryana: Probably other tools, I mean there are a bunch of other surrounding tools...

Howie: (page 13)...if a newbie gets reverted, they typically go back and find that the article is missing...so that pointed to the absence of notifications...revert is off by default for new users, is that right?

Fabrice: Yes, we took it off because it was a slap in the face of the new user...

Dario: ….

James: I think that not telling them that they’re going to be reverted is more of the slap in the face...

Howie: ….users don’t necessarily know that talk pages exist...that points to the need for a discussion system....and even if they do find their talk page and see that there are messages, there’s a lot of ambiguity on who these messages are from - are these from bots or users...this points to the need for profiles that users can understand...so this type of analysis led us to the next page (page 14)

Erik: Do you want to speak briefly about affiliations? ….

Howie: Yes, so affiliations is the last piece.... Wikipedia doesn’t do a good job of giving you other things to do...we know that when users move through the life cycle, they tend to edit based on a topic area...if you are actually interested in say wine, there’s no way that the system reaches out to you and draws you in. What ends-up happening today is somehow you have to find a wiki project, so the notion around affiliations is that users have an interest rout whether they express it explicitly or whether we can infer it...the notion of affiliations is matching that interest rout with work that needs to be done on the wiki. (review of page 15 and 16)

Tomasz: Are majority tools usually English based?

Howie: Yes

Tomasz: So purely English, no other languages use these tools?

Maryana: Yes, they do actually...German, Portuguese...

Siebrand: It takes significant community effort to actually make these tools work.

James: There’s also a tool you can’t use on en wiki because of a preference change we agreed to in 2005 called …

Howie: (review page 16 and 17)

Matt: What does the playbook actually do?

Howie: It starts the whole thing. I actually have a computer with Huggle loaded in the back (of the room) if you guys want to take a look at it...the next page is the interface and the resulting message... So you can see that as a new user, if you got these messages, you’ll be like, “What’s going on?”

Brandon: In the user tests that we ran about this, we actually used the nice template and it 50/50 whether or not a users can detect if a human left it or a robot.

Kaldari: Isn’t this a little bit outdated though because on en wiki we have a page creation tool now which about half or less of the page curators are using now, we actually built that with a lot of this in mind to try to improve it, where we put a lot more options for positive feedback like wiki love, and then we also tailored the warning messages to be more personalized and less verbose...to hopefully improve the experience for people who were creating the articles...

James: …

Kaldari: Yes, it’s only for article creation and it’s only on en wiki

Fabrice: Also the number has declined greatly, it used to be at 56% and it’s down to 15%....

Brandon: Do you know why?

Fabrice: No, no yet.

Howie: The next slide gives you another piece of analysis...what we see over the past couple of years is that the percentage of templates that involve criticism has been growing quite a bit and the ones that involve praise is rather sad. The next page gives you reversion rates... The next page shows that many editors end-up leaving because of conflicts with existing editors...

James: It’s also the number one troll complaint on news articles...

Tomasz: What’s the date on this?

Howie: 2010

The next page if about finding the right kind of contributor and this is just more food for thought...

Brandon: ...this speaks to a philosophical question I know we had for several years about the distinction between whether wikipedians could be created or if they are found...and this reads to me that we find them, we can’t make them...

Howie: My position is that if we really believe that Wikipedians are made not born and we think the interface is good enough to attract the ones that are born, then what are we doing here?

Brandon: What my view here is that Wikipedian is kind of a mindset...and I think our software is a barrier to have most of those people willing to join...

Steven:...I think the things Wikipedians are saying that motivate them we could map that …

Howie: (review of page 22)

Kenan: Despite the fact that the software is the same?

Howie: Yes, despite the fact that the software is the same and that was Joel’s main point....

Fabrice: ….for the past decade we’ve basically assumed that anyone can edit is the right thing to do and I’m getting growing concerns that the basic principles may need to be revisited. A lot of the negativity that we get is in a large part do to the fact that anyone could edit that our editors are so focused on dealing with vandals that perhaps having some small barriers to entry, very minimal ones, ... could potentially be beneficial.

Howie: We’ll talk about that more tomorrow morning. We’re running 5 minutes over, so I just want to point people to a couples of things (review of page 46, then to page 24) ...when we take a look at users involved in the meta spaces, a lot of them are class of 2007 and before...(review of page 28)

--- Break---