Winter/User tests

This page details various user test strategies and results that have been run against Winter and describes take-aways.

Abstract

Full user testing of Winter will require multiple test harnesses because:

• Some questions will spoil the tester for other questions
• Some questions require comparison between Vector and Winterized Vector
• Some questions have multiple routes to the same answer

User Tests

Harness One: Winter

1. Can the user identify the search box? (Test search box identification)
2. Can the user find the article discussion? (Test page-context action ribbon)
3. Can the user find out how to edit the full article? (test page-context edit button)
   1. Second test with edit button as mw-ui-constructive at outset rather than on hover.
4. Can the user discover how to find their contributions? (Test collapsed user menu)

Set Up

• User is "auto-logged in" as User:Accedie under the hood, so that there is dummy data for contributions and the user page.
• Page is pre-loaded with "Winter".

Introduction

Background: You're helping us test out a prototype of some software changes on Wikipedia, the world's largest free encyclopedia. Don't worry about messing anything up – this is just a prototype, not the real site! Mindset: You have made some edits to the encyclopedia in the past, but today you're researching various seasons on Wikipedia.

This test is best run using Google Chrome but will work well in Mozilla Firefox, too.

Tasks

1. Navigate to the starting URL (http://unicorn.wmflabs.org/t/847). You will be asked to enter a user name; pick any name you like (maybe even use your own name). Don't worry, no data will be saved.
2. Once you've "logged in", take a quick moment to look at this page and talk through what you're seeing.
3. Use the search box to look up "Autumn" and go to that page (If you can't find this after 2-3 minutes, follow _this link_).
4. All articles (and pages) on Wikipedia have an associated discussion. Can you find the discussion about "Autumn"? (If you're stuck after 2-3 minutes, it's okay - just move on to the next task).
5. If you haven't found the discussion, make sure you've scrolled to the top of the page and look at some links directly underneath the title of the page ("Autumn"). Click on the "Discussion" link and talk through what you think you see.
6. Can you find your way back to the "Autumn" page from the discussion? If you can't, just go to the next step.
7. If you were unable to find your way back to the "Autumn" page, scroll back to the top of the page and click the "Article" link. Once you're there, go on to the next step.
8. Now that you're back on the main text of the article, let's say you discover a small typo in the first paragraph of the page "Autumn". Can you find out how to make the change? If you can't figure out how to make the change, go ahead and move on to the next step.
9. If you weren't able to discover how to edit the page, scroll back to the top of the page and click on the "Edit" button in the top-right area of the article. This should bring you to the edit screen.
10. Take a few moments to describe what you see and then click "cancel" to return to the main article.
11. Let's say you're an experienced editor who has made several hundred contributions and edits to the encyclopedia. Can you find your list of contributions? Take a few moments to look for them but if you can't find them after a minute or two just move on to the next step.
12. Take a few moments to scroll around and describe your general impressions of this interface. After that, we're done.

Questions

1. How does this interface match with your expectations of using Wikipedia, what parts were easy to use and behaved as expected?
2. In what ways did this interface not match your expectations of using Wikipedia, what parts were hard to use, or weren't quite clear to you?
3. Are there tasks that you felt you were not successful in completing?
4. Can we show this video to our editor community?

Test Videos

Five videos were made using Test Harness One. Videos are annotated with highlights, but major take-aways from each are included below.

1. misterbeauds
   • Immediately notices the static header and thinks this is a good idea.
   • Discovers page context icons in the static header easily (though we were not testing this).
   • Discovers existence of Messages and Talk and thinks this is a a new feature for Wikipedia.
   • Quickly discovers page-context action ribbon. The fact that there are discussions about articles is completely new to him.
   • Instantly finds search box and expresses bafflement that anyone could not find it.
   • Figures out talk pages, but was expecting something more forum-like (speaks to Flow).
   • Instantly understands article-context navigation. "Easy-peasy".
   • No problems with edit or cancel.
   • Easily finds "Contributions" in the user menu.
   • Expressed that he really likes the search bar and the "new format". "It just feels more modern."
2. kayaker
   • User actually reads the introductory text on the login screen and opens the MediaWiki page. Oops. Time to remove that text.
   • User knew that tabs existed, but is pleased that they have moved.
   • Instantly finds search. "I like how it's big! It's nice!"
   • Finds the edit button and the discussion link easily.
   • User has looked at talk pages before but only "kind of" understands them. Has confusion about "posts" (sections).
   • User finds the edit button easily enough, and also explores other options in the edit button drop down.
     • She says, "I don't really want to edit visually, do I?" which speaks to a possible verbiage/branding problem with VE.
   • User cannot easily find the "cancel" button in the editor.
   • User correctly finds "Contributions" through the user menu, but encounters a different bug. She ends up back to contributions by going through her user page.
   • Is slightly confused by the way the buttons in the header change, but expresses a bit of joy shortly thereafter.
   • Definintely likes the bigger font.
3. blairsmama
   • Easily finds Search.
   • Initially thinks that "Discussion" may be in the table of contents, but then quickly notices its location at the top of the screen.
   • Does not understand talk page templates.
   • User thinks that "Main page" link in the sidebar goes back to the "main page" for the article at first, but then quickly corrects that mistake.
   • Easily finds out how to edit the page.
   • No problems finding the editor's "Cancel" button.
   • User has IE, and runs afoul of broken menu bug (see below). Ends up finding "Contributions" through the in-page context action ribbon from the user page.
4. Emilymae
   • Easily finds Search and Discussion.
   • Does not understand talk page templates.
   • Easily finds edit button.
   • No problems locating the editor's "Cancel" button.
   • First looks for "Contributions" under the Table of Contents icon but then quickly finds them in the user menu.
5. Srap33
   • Easily finds Search and Discussion.
   • "It's pretty basic. Easy to read, easy to access information."
   • User understands purpose of talk pages but not necessarily how to use them.
   • "Edit was easy to find".
   • No problems locating the editor's "Cancel" button.
   • User has IE, and runs afoul of broken menu bug (see below). Ends up finding "Contributions" through the in-page context action ribbon.

Takeaways

• No tester had difficulty recognizing the search bar.
• All testers quickly grasped the in-page context action ribbon.
• Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible.
• The "Cancel" button from the editor may need work.
• All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.

Other Notes

• The end of the test needs to be rewritten. Rather than discuss Winter itself, all participants discussed the faked-up "Contributions" view.
• Several minor display bugs exist in Safari and Internet Explorer
• The user drop down menu does not appear to work in Internet Explorer, which made for some confusion. The testers did not notice this, however, and completed the tasks anyway using other methods.

Harness Two: Winter

1. After scrolling, can users identify the search box? (test adjusted search box identification)
2. After scrolling, can users find the discussion? (test in-header discussion icon)
3. After scrolling, can users find their own contributions? (test secondarily collapsed user menu)

• It is quite likely that many of these will result in the user scrolling back to the top of the page.

Set Up

• User is "auto-logged in" as User:Accedie under the hood, so that there is dummy data for contributions and the user page.
• Page is pre-loaded with "Winter".

Introduction

Background: You're helping us test out a prototype of some software changes on Wikipedia, the world's largest free encyclopedia. Don't worry about messing anything up – this is just a prototype, not the real site! Mindset: You have made some edits to the encyclopedia in the past, but today you're researching various seasons on Wikipedia.

This test is best run using Google Chrome but will work well in Mozilla Firefox, too.

Notes

1. This version of Harness Two has a confusing situation where people would find themselves on their own talk page rather than an article talk page and get lost. This is corrected in a later version of the test.
2. Users were also a bit confused by not seeing the user name they provided in the contributions page. This was corrected in a later version of the test.
3. There was internal concern about the article talk icons not matching from within the action ribbon and the header. This is corrected in a later version of the test.

Tasks

1. Please move the Usertesting instruction panel (this panel) to the lower left of your screen so that it does not cover the upper area of your browser.
2. Navigate to the starting URL (http://unicorn.wmflabs.org/t/848). You will be asked to enter a user name; pick any name you like (maybe even use your own name). Don't worry, no data will be saved.
3. Once you've "logged in", take a quick moment to look at this page and talk through what you're seeing.
4. Navigate to the section of the page called "Festivals". Did you have any difficulty doing so?
5. Use the search box to look up "Autumn" and go to that page (If you can't find this after 2-3 minutes, follow _this link_). Navigate to the section entitled "Melancholy association".
6. All articles (and pages) on Wikipedia have an associated discussion. Can you find the discussion about "Autumn"? (If you're stuck after 2-3 minutes, it's okay - just move on to the next task).
7. If you haven't found the discussion, look in the top right area of the screen and find the icon that looks like two speech bubbles. Click on this icon and talk through what you think you see.
8. Can you find your way back to the "Autumn" page from the discussion? If you can't, scroll back to the top of the page and click the "Article" link, just below the article title.
9. Now that you're back on the "Autumn" article, go to the section about "Tourism".
10. Let's say you're an experienced editor who has made several hundred contributions and edits to the encyclopedia. Can you find your list of contributions? Take a few moments to look for them but if you can't find them after a minute or two just move on to the next step.
11. Search for and go to the article "Summer". Take a few moments to scroll around and describe your general impressions of this interface. After that, we're done.

Questions

1. How does this interface match with your expectations of using Wikipedia, what parts were easy to use and behaved as expected?
2. In what ways did this interface not match your expectations of using Wikipedia, what parts were hard to use, or weren't quite clear to you?
3. Are there tasks that you felt you were not successful in completing?
4. Can we show this video to our editor community?

Test Videos

Five videos were made using Test Harness Two. Videos are annotated with highlights, but major take-aways from each are included below.

1. asher
   • Instantly finds search after scrolling.
   • Did not know Wikipedia had discussions.
   • Confusion with double icons for talk page.
   • User assocatied icons in upper-right with "my stuff"; did not think that any page-context actions would appear there.
   • User had definite icon confusion.
   • Liked the in-header version of the Table of Contents
   • User found contributions but only by going to their user page first and then finding the Contributions in the in-page context ribbon.
   • Finds the edit button easily, once he's looking for it.
2. limitlessappz - User did not give permission to share their test video.
   • User likes how the search box scrolls with the page; has no difficulty with it.
   • Immediately discovers Echo.
   • User is blind to the existence of the table of contents and does not use it until the very end.
   • User finds way to article talk, but through a link in a page template, not through navigation.
   • User finds Contributions but only by going to user page first and then using the context action menu.
3. bablador
   • User scrolls to the top of the page to use search box. Does not notice the header being sticky.
   • User does not know that there are discussions.
   • User goes to discussion page using the context action ribbon, not the header icons.
   • Takes VERY little time to find Contributions link in the user menu pull down, but scrolls to the top of the page first.
4. eatglitter
   • User is super happy that Wikipedia has added a table of contents.
   • Scrolls to the top to use the search box but finds it easily.
   • Goes to discussion through the action ribbon.
   • User correctly identifies the personal tools section.
   • User finds contributions but does so by going to their talk page and then clicking "contributions" in the page context ribbon.
5. reneeineastbay
   • User easily finds search box. Does not scroll up. "Just look for the little magnifying glass."
   • User scrolls to the top of the page to use the action ribbon to get to the discussion page.
   • While on the user page, user does not see the contributions link inside the context ribbon.
   • Does not recognize pull-down menu icon in personal tools.
   • Never finds contributions.

Takeaways

• Search is very easy to find.
• Users are almost hard-wired to return to the top of the page to look for actions.
• No users correctly discovered the contextual header icons.
• Everyone is baffled by talk page templates.

Harness Two: Electric Boogaloo

1. After scrolling, can users identify the search box? (test adjusted search box identification)
2. After scrolling, can users find the table of contents? (test in-header table of contents icon)
3. After scrolling, can users find their own contributions? (test secondarily collapsed user menu)

• It is quite likely that many of these will result in the user scrolling back to the top of the page.

Notes

• This is a slightly modified version of Harness Two to account for questions that users tended to get lost in.
• Several modifications were made to the prototype to eliminate areas of confusion. Icons between the context ribbon and the header were synchronized. Opening dialog text was modified to be less "inside baseball."

Set Up

• User is "auto-logged in" as User:Accedie under the hood, so that there is dummy data for contributions and the user page.
• Page is pre-loaded with "Winter".

Introduction

Background: You're helping us test out a prototype of some software changes on Wikipedia, the world's largest free encyclopedia. Don't worry about messing anything up – this is just a prototype, not the real site! Mindset: You have made some edits to the encyclopedia in the past, but today you're researching various seasons on Wikipedia.

This test is best run using Google Chrome but will work well in Mozilla Firefox, too.

Tasks

1. Please move the Usertesting instruction panel (this panel) to the lower left of your screen so that it does not cover the upper area of your browser.
2. Navigate to the starting URL (http://unicorn.wmflabs.org/t/849). You will be asked to enter a user name; pick any name you like (maybe even use your own name). Don't worry, no data will be saved.
3. Once you've "logged in", take a quick moment to look at this page and talk through what you're seeing.
4. Navigate to the section of the page called "Festivals". Did you have any difficulty doing so?
5. Use the search box to look up "Autumn" and go to that page (If you can't find this after 2-3 minutes, follow this link). Navigate to the section entitled "Melancholy association".
6. All articles (and pages) on Wikipedia have an associated discussion. Can you find the discussion about "Autumn"? (If you're stuck after 2-3 minutes, it's okay - just move on to the next task).
7. If you haven't found the discussion, scroll back to the top of the page and click the "Discussion" link, just below the article title. Talk through what you think you see.
8. Can you find your way back to the "Autumn" page from the discussion? If you can't find it after a few minutes, it's okay - just move on to the next task.
9. If you haven't found your way back to the article, scroll back to the top of the page and click the "Article" link, just below the title.
10. Now that you're back on the "Autumn" article, go to the section about "Tourism".
11. Let's say you're an experienced editor who has made several hundred contributions and edits to the encyclopedia. Can you find your list of contributions? Take a few moments to look for them but if you can't find them after a minute or two just move on to the next step.
12. Search for and go to the article "Summer". Take a few moments to scroll around and describe your general impressions of this interface. After that, we're done.

Questions

1. How does this interface match with your expectations of using Wikipedia, what parts were easy to use and behaved as expected?
2. In what ways did this interface not match your expectations of using Wikipedia, what parts were hard to use, or weren't quite clear to you?
3. Are there tasks that you felt you were not successful in completing?
4. Can we show this video to our editor community?

Test Videos

bryanfm

1. • This tester is extremely literal minded. Only clicks on things when directed to.
   • User quickly discovers their personal bar (and identifies Echo)
   • User correctly finds search but scrolls to the top of the page first.
   • Finds the discussion link in the action ribbon. It takes a while, though.
   • Completely, utterly baffled by the article talk page.
   • No problems finding the "back to article" link now that he knows what the context ribbon is. Does not use header icons.
   • User scrolls to the top to access his personal toolbar. He does NOT associate the person icon in the static header to him.
   • User likes to zoom in a bit because he has "less than perfect eyesight"
   • Did not discover the in-header TOC until the very end.
2. LuckyMe
   • User scrolls to the top to get to the search box but has no trouble finding it there.
   • User scrolls to the top of the screen and quickly locates the Discussion link in the article context ribbon.
   • User correctly identifies talk page topics, but isn't sure she is allowed to edit them. Cannot identify the authors of comments (doesn't recognize signatures).
   • User is absolutely not associating the person icon with "her stuff". Eventually she finds it, but only after her username is next to it.
3. ddsurfer
   • Instantly finds search box. Does not scroll back up.
   • Finds the discussion link in the context action ribbon, but it takes a little time.
   • Navigates back to the article using the context ribbon.
   • She discovers the top bar icons, but she's looking for her contributions now, and doesn't think to open the user menu.
   • When looking for her contributions, the user clicks on the article history icon in the header. This is the first time I've seen anyone use those icons, and it's done incorrectly.
   • Does NOT find contributions and gives up looking. . .BUT she finds them at the very end (once the user name is visible again).
4. HyruleHero57
   • User *immediately* recognizes that he has a "personal tools" area though he doesn't associate Wikipedia with having accounts.
   • User is fairly familiar with Wikipedia's interface. Finds the search box change and calls it out.
   • Finds the in-page context action ribbon fairly easily.
   • User scrolls to the top to use the search box.
   • Wonders a bit about how hard the search is to see (even though he found it instantly). Expresses delight at the auto-complete feature, but we've had that forever.
   • Scrolls to the top to use the context action ribbon to find the Discussion.
   • Does NOT see talk pages as "discussions". "It's just people writing stuff." ABSOLUTELY expects the talk page to behave like a modern discussion system.
   • Re: Contributions: "I assume it's going to be in this top-right corner." Scrolls to the top of the page.
   • Goes to the user page and ignores the pull down menu. Finds Contributions in the context ribbon.
5. sav7
   • This tester is the only tester in both versions of this test to correctly find and use the header action icons. He also appears more technically savvy than the others.
     • However, he does suffer from icon confusion. It's possible he thinks the icons are shuffled.
   • Immediately discovers the personal tools section and points out that he can see his username there.
   • Finds search without scrolling to the top.
   • Navigates to article talk by using the header icon. Is the only one to do so.
   • User understands talk pages but thinks they are confusing.
   • Navigates back to the article using the context ribbon.
   • Correctly finds the contributions page by using the personal tools pull down. Does not scroll to the top of the page (again, only user to do so).
   • Very happy with the personal tools bar and header icons.

Takeaways

• Most testers do NOT associate the personal tools with "my stuff" unless the username is visible. If the username is there, they can find it easily.
• With one exception, users do not notice the header icons changing in a meaningful way.
• At this point, with 10 testers and a 10% success rate, I'd say that the benefits of putting page icons in the header are outweighed by the negatives of losing the username for discoverability.

Harness Three: Winter: No Header Icons

1. After scrolling, can users identify the search box? (test adjusted search box identification)
2. After scrolling, can users find the Discussion? (test location of context ribbon)
3. After scrolling, can users find their own contributions? (test secondarily collapsed user menu)

Notes

This test is functionally identical to Harness Two: Electric Boogaloo (same set up, background, questions, and tasks). However, in this version of the prototype, the following changes were made:

• No page-context actions are loaded into the fixed header.
• The user tools section does not shrink/collapse.

Test Videos

1. ragelle72
   • User instantly recognizes the personal tools area.
   • Easily finds search box; does not scroll to the top.
   • Goes to talk page through a link in an internal template. Does not understand what talk pages are.
   • When she discovers the action ribbon, indicates that she didn't see it at first (suggested a different color). However, uses it easily from that point on and says she likes it.
   • Quickly finds contributions in the user tools menu, but scrolls to the top of the page.
   • "It's very easy to navigate through and figure it out." She *loves* the personal tools area.
2. ThatDuckGuy
   • This is a highlight test. The user is very chatty. It is a 100% success.
   • Immediately notices the personal tools area and notes that he is logged in (discoverability of username).
   • Immediately finds the context action ribbon.
   • Notices that the page scrolled for him.
   • Immediately goes to the sticky search box; does not scroll.
   • Immediately scrolls to the top to find the discussion link.
   • Instantaneously goes to the user tools pulldown menu to find his contributions.
   • Likes the interface, thinks it is clean and modern. Loves the scrolling behavior.
   • Likes the context action ribbon. Thinks its easy to switch back and forth between pages.
   • Likes the search at the top, and likes that it is sticky. Likes the auto-complete.
3. kittypurry
   • Tester's set up has a lot of microphone buzz, so be warned.
   • Instantly grasps their personal tools section by pointing out their user name.
   • Scrolls to the top to find search. Likes the size of the search bar.
   • User expects a link to the discussion in the Table of Contents. As with many others, also scrolls to the bottom of the page (expecting blog/news story like comments area).
   • Finds the discussion link in the action ribbon but it takes a little while.
   • Quickly goes to and uses the context action ribbon to return to the article.
   • Easily finds contributions in the user pull down menu. Has scrolled to the top, though. Does not understand what the contributions are.
4. alonb1977
   • Instantly notes that he is logged in by recognizing that his username is visible in the personal tools section. Examines each icon.
   • Notices the context action ribbon. Correctly grasps what the watch link does and associates it with his watchlist.
   • Instantly goes to the search box. Does not scroll up.
   • Immediately scrolls to the top to use the context action ribbon to go to the talk page. Has NEVER seen the discussions before.
   • Immediately and unconsciously uses the action ribbon to return to the article. He continues to use it to navigate for the rest of the session without hesitation.
   • Instantly goes to his user pull down menu to find contributions. No hesitation.
   • He has some good words for the help documentation.
5. wldrlb
   • Notes that she is "logged in" by the presence of her name in the user tools section.
   • Scrolls to the top for the search box. Expects it in the upper right; finds it easily enough but thinks it's weird. However, states later "Now that I know the search is here, I like it."
   • Scrolls to the top of the page and easily finds the Discussion link in the action ribbon.
   • Immediately goes to the "Article" link in the context ribbon to return to "Autumn"
   • Quickly and easily goes to her pull down menu in the user tools area to find her contributions.

Takeaways

• All testers completed all tasks.
• Most users quickly pointed out that they knew they were "logged in" because of their username appearing in the corner and noted that they have a set of personal or profile tools there. More than one tester referred to this area as "their profile". No users had difficulty finding their contributions because of this.
• With one exception, all users found the context action ribbon quickly. The one who did not later was observed using it with proficiency.
• Many users unconsciously navigated with the context action ribbon afterwards.
• The search box remains mostly discoverable. Several testers unconsciously used it in the sticky format without scrolling to the top.
• No one used the table of contents in the header.

Harness Four: Winter: Only Header Actions

1. After scrolling, can users identify the search box? (test adjusted search box identification)
2. After scrolling, can users find the discussion? (test in-header discussion icon)
3. After scrolling, can users find their own contributions? (test secondarily collapsed user menu)

Notes

For this test, the page context action ribbon (the links under the title) was removed. The only pointers to discussion/history/etc. are within the sticky header icons. These icons are always present. Further, the personal tools menu always remains at it's fully uncollapsed state.

The tasks were modified to avoid ever calling out the existence of the icons in the menu (since once a user was sent there, they would know to go there, spoiling the discoverability of the test).

This test was an expected "train wreck" but needed to be done for completeness' sake.

The test set up, questions, and introduction were the same.

Tasks

1. Please move the Usertesting instruction panel (this panel) to the lower left of your screen so that it does not cover the upper area of your browser.
2. Navigate to the starting URL (http://unicorn.wmflabs.org/t/851). You will be asked to enter a user name; pick any name you like (maybe even use your own name). Don't worry, no data will be saved.
3. Once you've "logged in", take a quick moment to look at this page and talk through what you're seeing.
4. Use the search box to look up "Autumn" and go to that page (If you can't find this after 2-3 minutes, follow this link).
5. All articles (and pages) on Wikipedia have an associated discussion. Can you find the discussion about "Autumn"? (If you're stuck after 2-3 minutes, it's okay - just move on to the next task).
6. If your found the discussion, great! Go ahead and go back to the "Autumn" article by clicking the browser back button or using _this link_.
7. Now that you're back on the main text of the article, let's say you discover a small typo in the first paragraph of the page "Autumn". Can you find out how to make the change? # If you can't figure out how to make the change, go ahead and move on to the next step.
8. If you weren't able to find out how to edit the article, that's okay. Go ahead and move on to the next step. If you were able to do so, take a few moments to describe what you see and then click "cancel" to return to the main article.
9. Let's say you're an experienced editor who has made several hundred contributions and edits to the encyclopedia. Can you find your list of contributions? Take a few moments to look for them but if you can't find them after a minute or two just move on to the next step.
10. Take a few moments to scroll around and describe your general impressions of this interface. After that, we're done.

Test Videos

1. andreanicole
   • Quickly finds the search box. Likes it.
   • Takes a very long time to find the header icons. However, she does not distinguish between the article talk icon and her user talk icon, and ends up on her user talk page.
   • User cannot find the article discussion and gives up. However, she later finds the link during a different task. Expresses that it took way too long to find and that having two similar icons is confusing.
   • Does not find the full page edit control (in the header); only for a section.
   • She is unable to complete the contributions test because, contrary to instructions, she is doing the test with Internet Explorer. IE has a bug where the pull down menus are not working correctly.
   • She suffered a great deal from icon confusion and let us all know about it.
2. pedagogue
   • Tester discovers page action icons while exploring the interface. Takes a moment to find the search box.
   • Quickly finds the discussion link now that he's been here, but points out the only reason he found it is because he was looking for the search bar.
   • Quickly finds the edit icon now that he knows where to look.
   • Has difficulty finding the user drop down menu; ends up on his user page while looking for his contributions. Does not find them.
   • Has icon confusion and cannot distinguish between his watchlist and the "watch this page" action.
   • Eventually finds contributions in the pull down.
3. mflnamore
   • Instantly finds the search box.
   • Tries to find "discussion" using CTRL-F. Eventually finds the talk page but only through a link in a page template.
   • Takes a while to find the edit icon in the header. Relies heavily on the tooltips.
   • Goes to his user page to find contributions. Does not actually succeed; assumes that text on the user page is his contributions.
4. jefflou
   • User cannot find the search box and gives up, using the link. The link actually opens a new instance of the site in a differnt browser (Opera).
   • Looks for the discussion at the bottom of the page before eventually finding the icon in the header.
   • Correctly finds the edit icon, now that he's been exposed to the header icons.
   • No trouble with the cancel button.
   • Interestingly, he finds the article actions pull down, but does not associate the same symbol (v) next to his own username with a pull down at first.
   • He finds the contributions link but runs into an Opera bug.
   • He wants labels on the icons.
5. blazed75
   • This tester is a fast talker.
   • Instantly finds the search box and notes that it has changed. Really likes the changes.
   • Cannot find the discussion and moves on.
   • Ends up in the editor, but from a section edit link. Likes the editor over all.
   • Correctly grasps that their contributions will be associated with their account. Did not know that you can create an account.

Takeaways

• As usual, the search box is fairly easy to find and is somewhat of a hit.
• Most users encountered signficant icon confusion. In some cases, the confusion was enough to prevent them from accomplishing a task. In other cases, the confusion will obviously be problematic as cental editing themes are confused.
• The discoverability of the user tools section, when the username is present, is very high and remains so.
• The discoverability of the page action icons is almost non-existent. Several users expressed a desire for action labels. Many users failed to accomplish tasks that required these icons, or assumed they had achieved success when they did not. It is my belief that at least two testers assumed that the page action icons were just parts of their personal tools, and never looked there for page actions.
• No one has ever used the in-header Table of Contents.

Harness: Vector

1. Can the user identify the search box?
2. Can the user find the discussion button?
3. Can the user find the edit button?

Harness Five and Five point One: Winter

1. Can users find the history of the page?​​ (testing whether users can identify the 'last updated' action button as the history)
2. Can users​ identify the ToC icon on top left as a way to access the ToC/additional elements and actions for the article? (testing intuitiveness/visibility of ToC icon as an alternative to scrolling)
3. Do users find search intuitive/easy to find?​ (testing ability to find another wiki page)
4. Can users find the talk/discussions page? (testing whether users can identify another button in the actions bar)

Notes

• ​Latest version of Winter
• 'Table of contents' icon added to header of page next to wikipedia logo
• 'Actions' tabs/buttons bar added under​ title of wiki page
• ​'Personal tools' icons only on the top right corner​

Set Up

• User will log in with dummy account in winter-working​ prototype

Introduction

Today​, you'll be helping us test some potential software changes we may make to Wikipedia. You may be an experienced editor or a casual reader, and today you will be researching seasons on Wikipedia. The version of Wikipedia you'll be working with today is a prototype, not the actual site, so no need to worry about ​messing anything up as you move through the tasks below.​ Please make sure you're running this prototype on Google Chrome. Remember, we're testing our interface, not you, so if you can't complete a task or find it difficult, it's *our* fault, not yours.

Tasks

1. Please move the Usertesting instruction panel (this panel) to the lower left of your screen so that it does not cover the upper area of your browser.
2. Please navigate to this link (you're probably here already). You'll be prompted to enter a username; feel free to use a nickname or your own name.
3. A​fter you've logged in, please take a moment to briefly describe your initial reactions ​ ​to the main elements you see on the page and tell us what purposes you think they might serve. In 2-3 minutes, please move on to the next task.
4. Can you find a way to discover who edited this page last? If you don't find it after 1-2 minutes, don't worry about it, and move on to the next task.
5. Let's make sure we're back on the Winter article. Please click this link back to the "Winter" article page and move on to the next task.
6. Wikipedia allows users to discuss issues related to the article. Can you find and go to the Winter discussion page? If you can't find it in a minute, feel free to move to the next task.
7. If you figured it out, fantastic! If you didn't find the discussion/talk page, go to the top of the page and click on the "Discussions" tab right under the article title. Take a minute or two to describe what you're seeing and thinking about this page and then move on to the next task.
8. Can you find your way back to the "Winter" article? If so, go ahead. If not, click this link.
9. Now that you're back on the article page, navigate to the "Festivals" section of the article. (If you can't find this after a minute, follow this link).
10. Please search for the "Autumn" page and go to that page. Haven't found the page after 1-2 minutes? No worries, skip to the next task​.
11. If you searched for and found yourself on the "Autumn" article, great! Otherwise, click this link. Scroll down the page until you see the photo titled "Halloween pumpkins".
12. Without scrolling, can you find a way to jump to the section of this article titled "Harvest Association"? If you're stuck after a minute or two, just carry on to the next task.
13. You're almost done! Using search, look up the "Summer" article page. Talk us through your general impressions and your experience finding and navigating the information on the site. If you'd like, use our short list of questions below to guide your responses. After this, you're all finished!

Questions

1. How does this interface compare to what you expect from Wikipedia?​ Do you feel confident that you could find the information that you are looking for?
2. What features were intuitive and easy to use?
3. What did you find difficult to use, confusing, or generally not what you expected or envisioned?
4. Do we have your permission to share this video with our volunteer community?

Test Videos

Five videos were made using Test Harness Five and Five point Zero. Videos are annotated with highlights, but major take-aways from each are included below.

1. papabear
   • Thinks 'discussions' are a new feature
   • Confused about global activity sidebar - wikipedia generally or article-specific?
   • Page feels 'more interactive', 'more personalized and social' than current
   • Initial confusion about where page history can be found
   • Though unnecessary, scrolled up once to engage search
   • Didn't notice ToC button initially, but found it immediately upon the specific task prompt
   • Commented on ToC gutter, wants wrap text
2. superth1 did not want video shared
   • Likes page immediately. Wants bigger font size for page title
   • Comments throughout that page looks more modern. Points out the personal tools
   • Notices majority of new features during initial scan (minus the ToC icon). Notes the language select option and comments that landing on a page on current site in a duff language can be confusing to navigate
   • Edit button hover options redirects user correctly to history last updated button
   • Does not figure out how to 'add a comment' to discussions page. Even clicks to edit and says, 'don't want to do that!'. notes that you can edit existing but not add. interesting
   • No luck finding ToC button. Not sure if issue was content-oriented or because of the 'box' syndrome around the page. However, did move cursor around the page extensively, like for searching how to edit the discussions page earlier
   • Wants ToC moved to the right (I'm assuming because of the white space it creates)
3. ssoohoo
   • Notes the personal tools and in-page ToC section (as if it were not noticed previously on current version)
   • Thinks a gallery of images in the ToC gutter would be a good use of space
   • Finds history immediately and highlights Jared's username link
   • Finds discussions tab immediately - wants a more structured discussions page
   • Does not use search for 'Autumn' page - uses in page navigation
   • Does not find ToC button, searches in page instead, not looking for a new navigation tool
   • Uses search for last task, initially expresses confusion, changes mind
4. cbr600ps
   • User has issues with finding history page. Brandon notes that last modified timestamp on bottom of the page needs to be fixed. Clicks on discussion, noting that it's right next to the history page. Then clicks history
   • Finds discussions page quickly, but has already been here
   • Confused about discussions page. goes to his own talk page. realizes discussions page is the discussions/talk page
   • Does not use in-page ToC section to go to festivals - scrolls through page. Also searches in-page for 'Autumn', and ultimately uses the seasons outline portal. Knows search exists, but doesn't use
   • Does not find ToC button
   • Likes that the translation/languages button/drop-down exists
   • UK user
5. jlapidus
   • Comments on search bar and how big it is immediately
   • Expresses some confusion around the personal tools section, pokes around the watchlist
   • Finds and understands history page immediately
   • Instantly finds discussions page. Expects more of a forum or blog comment layout, confused at how it does look
   • Likes ToC section. Uses search instantly, no scrolling
   • Finds ToC header button after a moment, likes the pop-up/pull-down menu, really likes the button
   • Comments that it captures the existing site well but adds good features

Takeaways

• Users take notice of the personal tools area.
• Users really want something in the ToC gutter area if it exists.
• Users may have issues figuring out that the task for the ToC button is regarding a new navigation tool - perhaps the target section name in the task needs to be changed to another without 'association' in the name, to avoid users looking into the content vs searching for nav tool as desired/goal.
• Search box might not be as apparent as it could be. Search sometimes bypassed in favor of scrolling.
• Users tend to like the font.
• Users like the language translation button/drop-down.
• Users sometimes comment on talk pages not being organized enough, or have different expectations of what it should look like.

Other Notes

• User does not see questions at the end while working on tasks - keep this in mind for future write-ups. The last task was edited for tests 6A-D.

Harness Six: Winter (1034 No ToC and 1035 No Sidebar)

1. Do users understand the concept of a watchlist (in the context of Wikipedia/general star and un-star metaphor)?
2. Can users​ add articles to and then locate their watchlists?
3. Does the test process around watchlists improve their understanding of what they are on Wikipedia?
4. Can users find their own contributions?
5. Can users find their own profile? (menu and a target)
6. Can users find the article history/when it was last updated?

1034: (No in-page ToC)

1. Do users instinctively know that there should be a ToC?
2. Can users jump to sections of the page without scrolling, if the in-page ToC is removed?

1035: (No Sidebar)

1. Can users navigate to the Wikipedia main page/homepage or another page located in the sidebar when it is hidden by default?
2. Will users click on the logo or the sidebar 'main page' link once it appears?

Notes

• ​Latest version of Winter
• ​Watchlist functionalities added
• In-page ToC will be removed for 1034, present for 1035
• Sidebar will be present for 1034, hidden for 1035
• ToC button/dropdown will be reconfigured (icons removed and replaced with verticals; tabbed/outline content instead of buttons across the top)
• Username menu also expands on hover versus click

Set Up

• ​User will log in with dummy account in winter-working​ prototype

Introduction

Today​, you'll be helping us test some potential software changes we may make to Wikipedia. You may be an experienced editor or a casual reader. The version of Wikipedia you'll be working with today is a prototype, not the actual site, so no need to worry about ​messing anything up as you move through the tasks below.​ Please make sure you're running this prototype on Google Chrome. Remember, we're testing our interface, not you, so if you can't complete a task or find it difficult, it's *our* fault, not yours.

Tasks

1034: No in-page Table of Contents

1. Please move the Usertesting instruction panel (this panel) to the lower left of your screen so that it does not cover the upper area of your browser.
2. Please navigate to this link (you're probably here already). You'll be prompted to enter a username; feel free to use a nickname or your own name.
3. Wikipedia allows you to add pages to a watchlist. Take 1-2 minutes to describe what you think the 'watchlist' is, and what value it can provide on Wikipedia.
4. Can you find a way to add this Winter page to your watchlist? If you don't figure out how to do this in 1-2 minutes, don't worry about it, and move on to the next task.
5. If you found a way to add Winter to your watchlist, great! If not, that's fine. Can you now discover a way to locate your watchlist? If you can't find it after a minute, feel free to move on.
6. From your experience over the last couple tasks, can you please take 1-2 minutes to describe more specifically what you think a watchlist is and what purpose it serves on Wikipedia?
7. Let's make sure we're back on the Winter article. Please click this link back to the "Winter" article page and move on to the next task.
8. On Wikipedia's many article pages, like this Winter page, there is a way to navigate back to the Wikipedia main page, or the main page of English Wikipedia in this case. From here, do you see a way to navigate to this main page? If you can't find it after 1-2 minutes, don't worry about it and move on to the next task.
9. If you found the main page, great job! If not, that's okay. Regardless, let's jump back to the Winter page. Please click this link back to the "Winter" article page and move on to the next task.
10. Similarly to the main page, there is a way to access a portal of Wikipedia called 'Featured Content' from all Wikipedia article pages. In this portal, you can find a variety of articles that are held in the highest regard in terms of robust article content, collaborative editing and excellent sourcing. Do you see a way to navigate to the 'Featured Content' page from here? If you can't find it after 1-2 minutes, don't worry about it and move on to the next task.
11. Let's make sure you're back on the Winter page again. Please click this link back to the "Winter" article page and move on to the next task.
12. Can you find a way to navigate to the section titled 'Exceptionally cold winters' on this Winter page? If you can't find it after a minute or so, just jump to the next task. Did you have any difficulty doing so?
13. If you found it, awesome! Please stay where you're at and don't scroll or click around the page until the next task. If you didn't find it, please scroll down the Winter page to the 'Exceptionally cold winters' section and move on to the next task.
14. Please make sure that the 'Exceptionally cold winters' section is still at the top of your page. Now, without scrolling, can you find a way to jump directly to another section of the page entitled 'Festivals'? Haven't found a way after 1-2 minutes? Don't worry, just move on to the next task.
15. If you're on the 'Festivals' section, great job! If not, that's fine too. Now, let's say that you're an editor who has made many contributions and edits to Wikipedia. Can you find your list of contributions? Take a minute or so to look for it - if you can't find it, just move on to the next task.
16. If you found the list, awesome. If not, we're totally okay with that too. Click here to reset back to the top of the "Winter" article page and move on to the next task.
17. Can you find a way to discover who edited this page last? If you don't find it after 1-2 minutes, don't worry about it, and move on to the next task.
18. Almost done! Last thing to try: can you navigate to your own profile/user page? And whether or not you find it, you're done!

1035: No Sidebar

1. Please move the Usertesting instruction panel (this panel) to the lower left of your screen so that it does not cover the upper area of your browser.
2. Please navigate to this link (you're probably here already). You'll be prompted to enter a username; feel free to use a nickname or your own name.
3. Wikipedia allows you to add pages to a watchlist. Take 1-2 minutes to describe what you think the 'watchlist' is, and what value it can provide on Wikipedia.
4. Can you find a way to add this Winter page to your watchlist? If you don't figure out how to do this in 1-2 minutes, don't worry about it, and move on to the next task.
5. If you found a way to add Winter to your watchlist, great! If not, that's fine. Can you now discover a way to locate your watchlist? If you can't find it after a minute, feel free to move on.
6. From your experience over the last couple tasks, can you please take 1-2 minutes to describe more specifically what you think a watchlist is and what purpose it serves on Wikipedia?
7. Let's make sure we're back on the Winter article. Please click this link back to the "Winter" article page and move on to the next task.
8. On Wikipedia's many article pages, like this Winter page, there is a way to navigate back to the Wikipedia main page, or the main page of English Wikipedia in this case. From here, do you see a way to navigate to this main page? If you can't find it after 1-2 minutes, don't worry about it and move on to the next task.
9. If you found the main page, great job! If not, that's okay. Regardless, let's jump back to the Winter page. Please click this link back to the "Winter" article page and move on to the next task.
10. Similarly to the main page, there is a way to access a portal of Wikipedia called 'Featured Content' from all Wikipedia article pages. In this portal, you can find a variety of articles that are held in the highest regard in terms of robust article content, collaborative editing and excellent sourcing. Do you see a way to navigate to the 'Featured Content' page from here? If you can't find it after 1-2 minutes, don't worry about it and move on to the next task.
11. Let's make sure you're back on the Winter page again. Please click this link back to the "Winter" article page and move on to the next task.
12. Can you find a way to navigate to the section titled 'Exceptionally cold winters' on this Winter page? If you can't find it after a minute or so, just jump to the next task. Did you have any difficulty doing so?
13. If you found it, awesome! Please stay where you're at and don't scroll or click around the page until the next task. If not, go ahead and click the 'Exceptionally cold winters' section link in the in-page Table of Contents near the top of the Winter page and move on to the next task.
14. Please make sure that the 'Exceptionally cold winters' section is still at the top of your page. Now, without scrolling, can you find a way to jump directly to another section of the page entitled 'Festivals'? Haven't found a way after 1-2 minutes? Don't worry, just move on to the next task.
15. If you're on the 'Festivals' section, great job! If not, that's fine too. Now, let's say that you're an editor who has made many contributions and edits to Wikipedia. Can you find your list of contributions? Take a minute or so to look for it - if you can't find it, just move on to the next task.
16. If you found the list, awesome. If not, we're totally okay with that too. Click here to reset back to the top of the "Winter" article page and move on to the next task.
17. Can you find a way to discover who edited this page last? If you don't find it after 1-2 minutes, don't worry about it, and move on to the next task.
18. Almost done! Last thing to try: can you navigate to your own profile/user page? And whether or not you find it, you're done!

Questions

1. How does this interface compare to what you expect from Wikipedia?​ Do you feel confident that you could find the information and features that you are looking for?
2. What features were intuitive and easy to use? What did you find difficult to use, confusing, or generally not what you expected or envisioned?
3. Did you feel there were any important or essential elements the page or the site as a whole were missing?
4. Do we have your permission to share this video with our volunteer community?

Test Videos

Five videos each were made using Test Harness Six, versions 1034 and 1035. Videos are annotated with highlights, but major take-aways from each are included below.

1034: No in-page Table of Contents

1. Condawg
   • immediate positive reaction to the prototype look
   • comments that he likes the seasons panel on the right as opposed to where the ToC usually is (doesn't seem to realize it isn't the ToC)
   • Not sure if watchlist is a current or new feature, but guesses at the functionality accurately
   • Checks under edit menu, extras arrow dropdown under the actions bar, personal items (finds watchlist and checks notifications), after finally finding the other star. Remarks that it is quick and easy visual cue that might take some time for new users, but straightforward after learning the function of the star
   • Uses main page link to access main page. No problem with either this or featured content, as links are in sidebar
   • Notices ToC missing after realizing the outline box aren't contents of the page - 'strange thing to get rid of'
   • Refuses to scroll to get to the section, 'it (ToC) must be somewhere'. Goes to action bar's extra actions, but then finds ToC menu button pretty quickly
   • Confused why ToC isn't readily available, but is hidden under an extra click. Didn't 'feel right', even though not difficult to find
   • No issue with finding my contributions page and page history. Clicks user page link in the hover menu.
2. Kangolf
   • Guesses at watchlist functionality well - checks edit menu, personal items (gets to watchlist first) then goes back to the page and clicks the add star. 'it's like a favorite button'
   • Uses main page link, no problem with this or featured content. Calls sidebar a 'navigation section'
   • Thinks the seasons panel is a ToC initially. Can't find festivals on the section on the page, so goes immediately to the ToC menu button - 'aha!'
   • Finds my contributions in personal items after checking the edit hover menu first. Clicks edit menu again to find the last edit, then goes to update __ days ago section and finds history there
   • Clicks/tests user icon, user name and user name link in hover menu - all go to user page
3. chriswild1122
   • Guesses at watchlist functionality well. Gets to the watchlist via personal items first, sees that that's not the way to add, then clicks the corresponding add star quickly. star icons are typical for this 'sort of thing'
   • Clicks wikipedia logo for main page, remarking that it's instinctual. Not sure if he even saw the text link
   • Goes to personal items, ToC menu, then finally finds featured content on the sidebar. Says it wasn't as obvious as the other tasks so far.
   • Looks for ECW section in seasons panel, 'instinct is to look' here. Goes to a separate page. Returns, tries to find in search box. Finally scrolls down and finds it. Remarked at how difficult this was, even though he'd already clicked on the ToC menu previously
   • Clicks on ToC immediately for festivals section, 'wasn't as intuitive as i would've liked'
   • Goes to ToC menu for my contributions, goes to history page, goes to personal items menu but has some issues clicking, some confusion here
   • Scrolls up to the top of the page immediately and identifies the history button
   • Uses user page text link in personal items hove menu
4. jmscout
   • Guesses at watchlist functionality well. Gets to the watchlist via personal items first, sees that that's not the way to add, checks out the other watchlist icon shortcut, then clicks the corresponding add star quickly. 'that was confusing'. 'don't remember adding talk winter'
   • Hovers briefly over ToC menu, then talks through how she assumes she could click the wiki logo to get to the main page. then notices the text link and clicks there
   • Scrolls down page to find ECW section, does not mention seasons panel at all. later, takes issue with not being able to 'navigate' directly, comments that she'd expect the panel to have ToC content, but that it is about seasons instead. Tries sidebar, everything else, essentially
   • Clicks ToC menu immediately after reading next prompt. 'was not aware of this tool' 'nice to have, but wouldn't know what this was in general'
   • Tries ToC menu, then personal items hover menu for my contributions. Confusion - says it's a list of 'everyone who has contributed'
   • Goes to ToC menu for history
   • Clicks on user page text link in hover menu. Seems to want to try and click the username part and icon in the top bar, but doesn't ultimately
5. pacelliangela
   • Guesses at watchlist functionality well, clicks on the correct star immediately. 'other sites use a similar system'. Gets to the watchlist immediately via other star in personal items area. May think that the last edited date is the page added date, as she remarks that 'i've been watching the page for some time'
   • Venturing to click wiki logo, but sees main page text link and clicks that instead, while remarking that 'logo typically will take you to the homepage on other sites'. no problem with featured content
   • Notices lack of in-page ToC. 'had to scroll' to get down to the specific section, muses about if she did the task incorrectly. Finds ToC menu button 'ah, found it now'. 'glad i figured it out. but to be honest, i don't like this'. notes that it's not just because it's not familiar. didn't seem obvious. thinks the current in-page toc is more user-friendly
   • Gets to the my contributions page right away, gets to history (updatead __ days ago) right away
   • Clicks user page text link in hover menu, also clicks on user icon to test if it goes to the same page

1035: No Sidebar

1. gilagirl66
   • Guesses at watchlist functionality well, checks personal items area first, some confusion about whether the page was added. clicks to empty watchlist, then goes back on winter page and clicks the correct star
   • Sidebar links staying active after hovering over wiki icon is a known issue - circumstances around the links staying up and disappearing unclear
   • Some hesitation about whether we 'wanted' her to click the icon or the main page link; notes that on most sites, 'you can click the icon'. ultimately clicks text link
   • Finds featured content link after some hesitant scrolling and buggy links not showing up after hovering over the wiki logo/icon. Not a strong indication that she found the main page link purposefully on previous task
   • Uses in-page ToC and clicks to ECW section. Starts searching for festivals in search bar - realizes that it maybe searching all of wikipedia. Not sure if she figured this out intuitively, though
   • Hovers over ToC button to the point where 'view contents' messaging appears, but doesn't click. Would assume that some/most of the discovery for this user is accidental, not intuitive. After trying contents on sidebar, goes back to ToC button and clicks. remarks that 'that was not as intuitive'
   • Doesn't end up going to festivals section. Had some issues getting out of the ToC menu. Accidentally? navigates to userpage trying to get the ToC menu off the page. Gets to contributions page via personal items hover menu
   • Long sequence of actions to find history: finds updated last button, but activates drop down instead upon clicking a little too much to the right. clicks unsupported action, tries discussions page, tries edit hover menu, scrolls to last modified at the bottom of page, remembers watchlist tells you when it was last edited. however, goes back up and tries the updated last button again (works this time)
   • Clicks username button to go to userpage. Doesn't realize she's there? Waits for the hover menu again, clicks the text link, then finally comments that she's on the userpage, saying it was intuitive
2. asher
   • Guesses at watchlist functionality well, checks all the watchlist-looking icons and the watchlist option in personal items hover menu, clicks the correct star
   • Notices that talk page is also added. Checks the other ways to get to watchlist as well
   • Some confusion about the fact that some of these pages are taking him to the actual asher's page ' that was not me'
   • Inadvertently hovers over wiki logo upon moving into the sidebar tests. Encounters the appearing/disappearing randomly sidebar links 'why is that disappearing, i don't know'
   • Assumes main page is accessible via logo as well, but clicks text link and comments that the logo 'is now not the home link'; not sure why that is
   • Notices ToC menu button is a new feature - doesn't seem to realize it is page-specific quite yet, but does upon trying the next task. 'a little weird that it is next to the search bar - feels like it should be a little lower down so it's part of the article frame'
   • Looks in ToC menu briefly for contributions, but then goes to personal items and finds it there
   • Finds updated 20 days ago button right away, mentions that it can also be accessed in ToC menu
   • Finds userpage immediately
3. Innovention
   • Guesses at watchlist functionality well, tries edit, clicks view watchlist icon even though it says view, not add. Realizes it wasn't added, assumes the way to add is back on the page somewhere but doesn't try it
   • Assumes wiki logo would work, but clicks main page text link when it pops up. No problem with featured content. Likes this new menu
   • Finds ECW section in-page ToC. Then checks search bar, and the ToC menu button to find festivals section
   • Finds contributions page immediately. Same for updated 20 days ago button 'pretty easy to see right off the bat'. However, ends up off track because of the arrow dropdown... eventually circles back to the updated button and clicks it
   • Seems to want to click just the username in top right corner, but sees user page text link in dropdown and clicks that
4. rabbit_winner
   • Guesses at watchlist functionality well, clicks the correct star to add page right away 'this is kind of like a favorite', 'straightforward, pretty intuitive'
   • Gets to watchlist in dropdown menu, talks about going to 'my account' (personal items section)
   • Goes immediately to logo and says i would click this, but ends up clicking the text link after seeing it. Finds featured content easily also; 'using wiki logo as a navigation bar, almost'. 'i'd want to look there'
   • Gets to ECW section immediately. Mouses over page and over ToC button menu for festivals, muses some, then finally tries ToC button and navigates there. 'don't think I would've' seen or known, says she would've just scrolled up and used the in-page again
   • Looks for contributions at bottom of page, ToC menu, history page. Reads prompt again, muses that she'd search the history page for her own contributions, but that she'd also try my 'profile'... then finds it in the personal items dropdown
   • Remembers history in ToC menu, but also sees updated 20 days ago button
   • Clicks user page text link in hover menu
5. IAMCARL
   • Guesses at watchlist functionality well. Gets to the watchlist first, and mouses over other icons, then to the correct add page star
   • Says he'd usually click on the logo, but sees main page link and clicks that instead. No problem with featured content.
   • Uses in-page ToC for ECW 'love the way it scrolls down' (the navigation 'animation')
   • Checks out the personal items menu, then finds the ToC menu button for the festivals task
   • Finds contributions in personal items menu drop down
   • No problem with finding the history page, no problem getting to the userpage in the personal items menu

Takeaways

Bugs/Suggestions

• Some users have issues distinguishing between the 'last updated section' and the arrow. Would be useful to put 'additional/extra actions' or 'more options' or some other descriptor next to the arrow to avoid this confusion.
• Issue with icons in the personal items hover menu not clickable. Confused some users when nothing happened upon clicking.
• Hover message on the add/remove star - any way to change that to add page when it's not filled in and remove page when it is filled in?
• Content cuts off in the main and featured content pages. Specific to testing version/known bug, or an issue?
• Sidebar links staying active after hovering over wiki icon is a known issue - circumstances around the links staying up and disappearing unclear at the moment.
• Unsure about having the search bar show the article name after scrolling down. I'd suggest 'enter search terms' or 'search wikipedia' language more, to avoid confusion about whether this is searching within the current article or searching the wider site.
• Any way to cut off the user testing pages entirely from the 'real' wikipedia? User asher had confusion because he accessed the real asher's userpage via some methods
• Suggestion from user that the ToC button should be lower down so it is 'part of the article frame', not wikipedia as a whole.

Other

• Users seem to have a good overall grasp of what a watchlist is, with some differences here and there.
• Many users find the watchlist first. Not sure if they already realize the star icon connection or if it influences in that direction when they then add or try to add the page.
• Generally no issue going to sidebar for either set of users for the main page and featured content prompts.
• Some users still scroll up to access personal items or just items usually at the top of the page (even though it moves with scrolling now). Perhaps just an ingrained thing.
• Sometimes there is a tendency to over-use/assume ToC menu is the go-to for all following tasks.
• Most people seem to feel that clicking the logo would be intuitive for the main page, but end up clicking the text link that appears.