Thread:Talk:Flow Portal/Community consent?/reply (4)

Of course it's possible for normal, non-programmer people to test unfinished software, it's even the recommended approach to achieve usability. It's never seen at Wikipedia because WP software roll-outs rarely follow best practices, but it works really well for big companies like Google (who gets usability fairly well). The way it's done at other places is with an iterative process for gathering feedback:
 * 1) You develop surveys and interviews with users from your target audience, or perform a similar field study to ask for ways that a tools like yours would be used. In a place like Wikipedia, where users form a huge world-wide community, you publish a well-publicized survey and analyze what respondents have to say about the tool's purpose.
 * 2) You select a really small set of users (three to five is enough) and present them with the first, non-functional interface (preferably offline, in the same room). Without telling them anything of how to use the tool (this part is important), you ask them whether they can make sense of the proposed interface, and how they would use it to achieve whatever goal is in their minds.
 * 3) You annotate all the proposed ways the users intended to use the application, and where they didn't understand something. You fix anything that didn't make sense, and try to change the design to accommodate as many new proposals as possible, before showing the tool again to the next five users. Iterate steps 2 and 3 until no more major problems are found.
 * 4) You implement a prototype with the design that emerged from the initial stage. You randomly select an average-sized and diverse group of people and repeat steps 2 and 3 with the functional prototype. You register any proposal of new ways to use the tool, and annotate the places where the initial software architecture wouldn't accommodate those ways of use.
 * 5) You throw away the prototype (that step is also important), and design a new architecture that is adequate for all the new usages that were originally unexpected and for which the prototype architecture was a bad fit. At this point you have a software design that suits the needs for a moderately large set of users and that can be expanded with small functional increments.
 * 6) Now comes the step that Google does particularly well. You announce worldwide the roll out of a new beta service or interface, explaining that it will be made available in the near future at incremental stages. At each stage you increase the number of users exposed to the new tool, either by providing an invite-only system or by randomly providing opt-ins to larger and larger groups each time. Any user that doesn't want to use the beta software should be allowed to easily revert back to the old system until the end of the beta.
 * 7) During the beta phase you add polish, polish and more polish, fix abolutely all workflow-stopper bugs, and include those new functions that are frequently requested by users during the beta stages.
 * 8) Finally, you should be confident that your tool is polished, almost bug-free and that it accommodates the needs of the majority of your user base (as compiled from user feedback during the whole process); only the most obscure use cases should be not supported by the tool at this point. This, and no earlier, is when you can do a final roll-out for all users. If you did a proper work, there should be no backlash, because the tool really provides an improved workflow, and nobody complains when they're given a better tool (provided that it is really better for them, not just better for the developer). If there's widespread backlash anyway, it means that you failed somewhere at the previous steps; you then keep track of that failure, so that the next time you don't repeat the same mistakes.

It's a lot of work, and it requires that developers are willing to re-design the whole product according to users specifications; if you instead start from an almost-finished design and architecture, and only try to accommodate the few features that make sense for developers, the end result does not satisfy user needs. As far as I can tell the VE design completely missed steps 1 and 5, and was coerced by the community to implement step 6 (the initial roll-out was nowhere like this best practice, but it's somewhat there now). This means that there's hope for a final release that won't be rejected in full, but that the current editor is not based on a complete understanding of all user needs, and that it stood in the prototype stage - so the current architecture will not be able to accommodate all the expectations from the community without a major redesign.

...and no true ability to judge whether it meets their needs. WhatamIdoing, I find that comment is incredibly dismissive of non-programmer people, for someone who claims to represent the interests of newcomers. Nobody better than the final user can judge whether a software meets their needs; if you assume what should be the needs of users, and later claim that users don't understand those assumed needs, you'd be doing everything backwards.