Abstract/Background edit

The new Recent Changes filters have been out as a beta feature on various language wikis. While Collaboration and Design have previously examined the usability of the new Recent Changes filters, we’d like to know from users directly how the experience of using the filters over a period of time has been for them.

Since the initial rounds of generative and usability testing, the team has surfaced usability concerns and tweaks that have been applied to a new prototype, termed ‘integrated’ filters, along with a set of filters made specifically for the functionality on watchlist. The goal is to evaluate the usability of these newest iterations in conjunction with the beta satisfaction testing.

Research Questions edit

Beta satisfaction

What do users think of the filters? Have their opinions changed over time?
Have they put the filters to use?
- If so, how? What are some of these uses?
Are the filters supporting their existing activities and workflows (better, well)?
Are they creating new ‘existing activities/workflows’ as a result of having the additional capabilities?
How do they feel about the beta Recent Changes vs current Recent Changes?
More specifically, how do the new set of filters compare with the current set?
What does the page need to contain for it to reach peak effectiveness for the users?
Do users have particular feelings about the “related links” at the top of the page?

Integrated filters

How are discoverability of the filters and general navigation for users? Overall, has there been improvement?
What reactions do users have to the live updates button and pagination options?
Do users know about / recall / use the resources listed in the related links section?

Watchlist actions usability

What actions do users typically take on their watchlists?
What filters would be useful? How do they feel about the watchlist filter set on the prototype? What filters are still missing?
- Is the purpose of the watchlist-related filters provided in the Watchlist page understood, discoverable, and useful for watchlist users.
- Do users create the same filter sets in the Watchlist they would for Recent Changes?
- Are users able to identify which pages require their attention (are not visited) at a glance and why the distinction is made? Is it clear for users how to mark some/all of them as visited?

Methodology edit

Participant recruiting
- 5 participants
Youtube Live / Google Hangout on Air moderated sessions
Test environments
- Users can interact with the beta page on their own accounts, but will be requested to use the english version or the english version in conjunction with their language version of the beta.
- updated prototype
- prototype's watchlist mirror

Participants edit

Heavy users of Recent Changes page and the beta.

Protocol and Researcher Notes edit

Keep in mind that you’ll be working with prototypes, and not everything will function as expected.

Beta Satisfaction edit

You recently activated a beta feature adding filters to the Recent Changes page. Tell/show me a little about what you’ve been doing on this page.
Filters
- Have you used any of these? If so, can you show me how?
  - What do you think of them?
  - Which of these filters are you using most often and why?
  - Can you show me some other examples of filters you’ve used?
  - Which of these filters do you not intend to use and why?
- Are there any elements or filters that you don’t understand or don’t understand how to use?
- Look at the links at the top of the page -- under ‘This page” “Utilities” and “about us.” Have you ever used these? What do you think of these links and where they are located?
- What other page features have you been using (or not)?
You’ve been using the beta for a while; can you compare your experience on the beta and the current RC page? (visual, layout, elements, experience)
Do your current actions on RC page match what you used to do on RC page?
Can you tell me about the differences in the filtering capabilities and other page features on the two pages? What do you think of them? Which do you prefer?
Think about when you first started using the beta. Do you think your opinions about the filters/features change over time, or not?
Was there anything that surprised or confused you at first?
Based on your experience reviewing Recent Changes, can you tell me what would your ideal RC page look like? Which elements from either current/beta pages would you keep? Is there anything still missing from both that you’d like to see implemented?
Do you see any good reasons for why some people would want to stick with the current version? I.e., should this remain in beta or should they be released to everyone
Imagine you logged in, and your RC page by default looks like the beta version. How would you feel about that?
What other pages on wiki do you think could utilize these filter functionalities?
Imagine that you came in and this toolset was on your Watchlist. How would you feel?
Can you imagine using any of these filters on Watchlist? If not, how would you feel about having them available?

Integrated filters usability edit

Test filter menus discoverability:

On this page, you’re hoping to find problematic vandal edits to user and user talk pages made by the user Cronopio. How would you show only these edits?
How would you narrow these results down further to edits made using a mobile device?
Now, you’re only looking for good edits made by newer editors so you can thank them for their contributions. How would you show these edits?

Pagination

Most reviewers work on edits as they occur, but sometimes edits are missed. How would you review older edits?
Let’s say, instead of searching for old edits, there is another way to show old edits first. How do you feel about having that capability? Could you sort the results so that the oldest results are shown first?
As you’re working through the list, you want the set of results that are slightly more recent. How would you show these?
The controls don’t work as this is a prototype. How would you expect pagination to work when sorting like this?

Hidden related links panel

We talked earlier about those links that appear near the top of the RC page. Looking at this page, where do you see anywhere where those links might be?
- If never used: What do you think ‘other review tools’ are? What would happen if you clicked it? Take a quick look, and tell me what you think of what you see.

Live update

Now, on this page, let’s imagine you’ve organized an editathon for scientist biographies, and you want to be able to monitor edits as they are being made. How would you do this? Let’s say you also want to specify viewing just the problematic changes that are being made. How would you show these?
What do you think the live updates button does?
How do you feel about the way the new results are being shown when the live updates are on?
Would you use this feature on Recent Changes page? When, and why?
What other tools or controls might improve this feature for you?

Filters on Watchlist edit

Can you tell me what you usually do on the watchlist page? What actions do you typically take? If you’re willing to show us on your personal watchlist, please do, if not, just describing is fine as well.
What filters, if any, do you currently use on your watchlist?
Let’s go to the prototype link, and take a minute to look around. Can you tell me about these results (which pages have changes you haven’t seen yet, and which do not)? What do you think the filled/unfilled bullet points denote?
Let’s say you went on a vacation for a week, and now you just want to focus on the pages that have new activity. Can you show us a list of those pages only?
Review some of these pages. Do they notice any changes?
Let’s say there are too many changes to review that happened during the vacation, and you’d only like to review new edits to these pages starting from today. How would you modify your watchlist so that it will only show you new updates to the page starting now?
What do you think of the filters you saw in the watchlist activity section?
You’re pretty familiar with the other available filters because of your experience with the Recent Changes beta, in addition to the watchlist filters you just saw. What specific combinations of results might you look for in your watchlist? If applicable, can you show me some of these combinations of filters you might use?
If applicable, how do these filter sets compare to the ones you make on Recent Changes page?
How about the default filter sets on RC vs watchlist? Overall, would you want to have the same filter combinations you have on RC to exist on watchlist page as well, or would you want separate filter combinations on watchlist?
Are there any filters you would find useful to include on the watchlist page that aren’t yet included here?