Contributing to WordPress, Part III: Usability Testing
One of the reasons WordPress 2.7 was such a success is the amount of usability testing that took place during the development cycle. Starting with testing 2.5 and the Crazyhorse prototype and following with the 2.7 beta, the testing program looked at almost every feature and function in the application. That kind of thing? Takes a lot of time. 
For readers who aren’t familiar with the process behind usability testing, here’s an overview. First, determine the scope of your test and create a test protocol/script. Determine the audience segments to be included in the test group(s), and begin recruiting. Recruiting may mean hiring an agency to find participants, but for testing WordPress, it makes more sense to recruit from within this community, so that means making a screening survey, reading all the responses, segmenting the respondents into categories and contacting people until you’ve filled your desired quotas (for whatever segments you’re seeking, such as newbie, experienced user, developer, CMS user, photoblogger, mobile user, etc. ). Then come the test sessions.
Depending on what is being tested, these last anywhere from half an hour to an hour and half apiece. Sessions are generally recorded using screen capture and web cam, with a video camera for backup. The moderator(s) generally take notes during sessions and/or (depending on what software is being used for the session capture) set markers in the video to indicate task completion, comments of interest, etc. In some cases, auxiliary test methods such as eye-tracking may be included. When the sessions are complete, the results are analyzed. All the notes and videos are reviewed, patterns are identified, and ultimately a report is written and the feedback informs the next round of revisions.
Some people think it shouldn’t take much time to do all this. I’ve lost count of the number of people who cite an old article by Jakob Nielsen that says you don’t need to test with more than 5 users because usability issues become clear right away. While I’ve found that to be generally true, when your user base is as diverse in experience level, usage, platform configuration, language (right to left languages have a pretty different experience) and demography as the WordPress community is, 5 users really isn’t enough to get a clear picture. We try to test with at least a dozen people each round, but then we are limited to a geographic region (test in NY, test in SF, or test wherever we can schedule enough people back to back to make it worthwhile), while WordPress users are located all over the globe.
To address this, we’re introducing a set of new contribution opportunities to expand our usability testing program. As with development and graphic design, we’re going to create an infrastructure to allow community participation in usability testing on a regular basis and in a much broader capacity than existed before, when it was limited to announcements that we needed participants in x city on y date. We’ll be looking for volunteer moderators as well as participants, hopefully from all over the world.
Moderators. Observational usability testing isn’t rocket science, but neither is it a simple task to reduce bias. Because of this, at first we’ll choose only moderators who have professional experience conducting usability tests. People who conduct testing for design agencies, software companies, usability consulting firms and the like will be our first round draft picks. In the future, when we have a group of regular volunteers and have ironed out any kinks in the process, we’ll ideally match up experienced testers with aspiring ones, using a mentorship model to increase the number of people who can contribute in this fashion.
Participants. If you use WordPress, chances are you could participant in a usability test at some point. In some cases we’re looking for particular behaviors (people who upload large video files, people who blog from their iPhone, people who manage more than 5 blogs, etc.), while other times the behaviors we’re looking for are much more common (do you have widgets in your sidebar, have you changed themes in the last 6 months, is there more than one author on your blog, etc.).
So how will these opportunities come into play, and how will it make WordPress better?
We’ll start with the moderators, and try to get volunteers with a decent geographic spread. Then, we’ll start signing up potential test participants in those areas (though we’ll also allow at-large registrations, since traveling testing will still be happening). We’re working on a registration process for potential participants. You’ll enter your basic info (location, contact info) and answer some questions about your WordPress usage to be entered in the database, and when there’s a testing opportunity coming up that’s appropriate for you, a local moderator will get in touch to see if you’re interested. Further down the road we may experiment with remote testing and other methods, but for now, this approach will broaden the geographic scope of our testing.
All moderators will follow the same test protocols and script, and their results/reports/video will be reviewed and collated, with a composite report (including the protocol/script that was used) published to the community. This will provide designers and developers with broader feedback during the dev cycle, and will allow the wider community to both understand and participate in the testing program.
If you’re interested in being a moderator during this initial phase (meaning you do it professionally), send me an email and introduce yourself. If you’re interested in signing up as a potential test participant, watch this space. We’ll post a link to the registration survey once it’s ready.