Wikimedia Research/Usability testing/Browser extension

The browser extension allows designers and researchers at the Wikimedia Foundation to do usability testing with logged-out people, using a fully featured prototype (i.e. within Wikipedia itself), over an extended period of time. The browser extension can also be used for testing with logged-in people. Participants (i.e. people who are testing the feature/workflow) install the browser extension on their preferred browser, and then anytime they visit Wikipedia (or another WMF site) they will get a modified experience. They use the modified experience for a given period of time (e.g. one month), and either periodically, or at the end of the period, submit feedback about the workflow/feature. Feedback can be submitted either within the site (this requires special configuration of the browser extension), or through external means such as a Talk page, a Google form, or simply by emailing the host of the test.

Why is a browser extension a useful testing tool?

At the Wikimedia Foundation usability testing is typically done by having people use some kind of prototype (or mockup) of a feature/workflow. Three important considerations when doing usability testing are: 1) how fully featured the prototype needs to be, 2) who you want to test with, and 3) how much time you need participants to spend with the prototype. Sometimes you can get the necessary feedback by showing a simple prototype to people one time and getting their initial reactions. Other times, perhaps if you are testing a more complex feature/workflow, you need a fully featured prototype, and you want people to try it out multiple times.

1) Minimally vs. fully featured

An example of a minimally featured prototype is a series of JPEGs with clickable hot spots that allow a participant to click from one to the next. An example of a fully featured prototype would be a prototype of a new feature/workflow embedded within the actual website/product. Testing with a fully featured prototype is more difficult because it means you have to spend the time building out the feature/workflow and integrating it into the actual website/product, however you can potentially get better feedback because the test is more realistic (see Hawthorn effect). The three options we currently have available to us at the Wikimedia Foundation for testing fully featured prototypes are: A/B tests, gadgets, and the browser extension.

2) Who you want to test with

The browser extension is currently the only option for testing fully featured prototyped with logged-out people. A/B testing and gadget testing are only available to logged-in people. Other methods, such as usertesting.com, allow us to test with logged-out people, however they do not allow for fully featured prototypes.

3) How much time you need participants to spend with the prototype

Because the browser extension provides a prototype of a new feature/workflow embedded within the actual website/product, there is no real limit to how much time you can have a participants spend with the prototype. Anytime they visit Wikipedia they will be exposed to the prototype. There is no additional effort or awareness needed on their part to engage with it.

In summary, the browser extension allows us to test a fully featured prototype, with logged-out people, for indefinite periods of time.

How to do usability testing with the browser extension

(coming soon)