Wikimedia Research/Design Research/2015 workshop

Design Research Workshop

November 20, 2015

LinksEdit

  • Slides: LINK TO COMMONS
  • Youtube steam

CommentsEdit

  • The referenced user testing for UploadWizard led to the Multimedia team writing an in-editor upload tool for VisualEditor (WikiEditor on the way...)
  • Another important consideration about sample size in usability testing is that if you have resources to put 15 people in front of your solution is it better to do 3 rounds of testing with 5 users each and improving your solution after each one rather than doing a big round with 15 users that allows to catch all issues but having no room to check if the solution to those was effective.
  • The exercise of completing the spreadsheet was really good! It's not only useful to plan and understand current workflows, but also, to better understand how to collaborate cross-teams (María).
  • Personas! Would be great if you can give us a guide to how to make our own personas if we don't see them in the list you have created.
  • http://design-research-methods.wmflabs.org/ is beautifully designed, and seems to contain useful information, but I doubt many people will remember the URL two weeks from now, and the content doesn't seem searchable or editable. Also, Labs doesn't seem a good place for publishing text in the long term (we'll still be interested in this stuff two years from now, right?). How about putting a copy on Meta-wiki or mediawiki.org?

Questions for design researchEdit

1. From Halfak: Is there documentation for persona research on MediaWiki/Meta? Where can I find it?

Answer: TODO add links to related resources to Meta/MW

2. what is understood by "the Global South"? And do you think one persona will represent "the Global South"?

Answer: not one persona for Global South (first part of question not really answered...)

3. what is the Ghana research? Can we access that?

Answer: analysis is ongoing, results will be published on wiki (TODO)

Building a productEdit

Understand Concept generation Concept evaluation Develop Review Release Maintain
Description Wear your researcher hat.

Discover personas, needs, challenges, and opportunities.

Generate many concepts and imagine possible futures. Evaluate concepts against what was learned in "understand", business needs, technical resources other constraints. Build, test and iterate on concept. Review with stakeholders to decide whether to release. Release user-friendly product that supports foundation's mission and meets our users' needs. Make sure that our product continues working for our users, and measure impact.
Design researchers Generative research

Get to know users and their context Define problem & opportunities Analyze and synthesize (see patterns, gain insights)

Develop lots of concepts Evaluate concepts against what we learned from users

Evaluate concepts with users Combine similar concepts if possible Define success and how to measure it Match concepts with foundation's mission Consult / share with stakeholders

Build prototypes

Refine and build MVP Usability testing / qualitative evaluation A/B testing or quantitative evaluation Iteration if needed

Consult with stakeholders regarding readiness of product

If product is not ready, go back to “Develop” scoped amount of time to reach the decision for release

Publish the product

Communicate about the release

Measurement of product

Maintain product (bug fixes, etc.) Observe need for iteration or next feature set. Potentially retire something based on product performance and more user feedback Triage bugs

Designers - Putting research hat on, listening to users and understanding audience. -

- Look at community channels, feedback channels and figure out things to fix. - Guide questions being asked

- Sketch possible solutions to explore many different directions.

-

- Make higher-fidelity representations (mockups/animations/prototypes) to communicate the design ideas internally and externally.

- Get feedback from other designers

- Define scenarios and create prototypes to make ideas testable.

-

- Check if product spec, interaction spec and visual design spec is done correctly

- Get notified about patchsets without screenshots and trying to figure out what is going on.

- Help evaluate how the current solution works (in collab. with Research)

- Work with communication team to craft presentation to public

- Respond to feeback and identify opportunities to improve the designs.

- Continue to check for bugs

Product Managers Identify problems to investigate, and user segments that we want to understand better. Prioritize goals to work on. Brainstorm possible options to solve the problem, connect with possible stakeholders. Communicate concepts and plans to community team, developers, get feedback. Create and communicate framework for how we evaluate potential solutions, keep process moving, decide which solution to build Define spec for development. Evaluate tradeoffs between options. Keep the team focused on the most important pieces. Plan iterations so that every release adds value for the user. Coordinate analysis of impact of solution, communicate results Announce feature, gather feedback. Communicate learning with team for further development. Prioritize bug fixes & feature updates into technical workflow.
Community Liaisons Get clear understanding of the problem the product intends to solve. Define users that the product is being created for. Contact wide spread of communities (all areas of communities, ask for audiences. Ask the question of "what problem does this solve?", and "whom is this for?" "What do you need to be able to do, as a user" Help users generate their own user stories. Create working groups of users to discuss needs (roubndtables). Conduct IRC office Hours and orther public meetings Help product owners compare the new idea with similar existing ones, or previous trials that existed on wikis. Define MVP from audience perspective. Are there blockers? Are there power user tools we need to consider? Get the community updated on planning, data, etc.+

help incorporate community in product development, as far as the process allows. Coordinate users for testing/feedback loops. Discuss data and impacts, review concerns.

Gather feedback and correlate accross different areas of product teams. Cast wide net for feedback, remind users of goals of product. Ensure community (readers or editors) 's feedback from previous step is incorporated. What is go-no go? Discuss timing of release with different communities. Discuss data and how it correlates to decisions. Ensure a smooth roll out, that doesn't disturb any set of users, or block any other workflow. Continue to work with the communities on features to add, bugs to fix, performance to enhance, etc + accomodate the needs to replicate the product in different languages, if it was originally done only in one wiki or in one platform.
Program Capacity & Learning Who are the teams we are trying to serve? For example, we work a lot with program leaders, who run things like GLAM, Education Program, Affiliates, etc. These groups often need learning as well as technical products We listen to the community. Then, we propose ideas, and we use a diverge/converge method (CPS); where generate a bunch of ideas and then build on those. Once we have lots of ideas, we begin to assess which ones are most feasible or most important We look at our skills and capacity, contrast them with community members / communities' needs and understand what we are able to build. We develop tools and resources, but we don't have a process for iteration. We are strong at ideation, and we don't have a proper feedback process. Some difficulties that could bring this up: we don't have enough resources / capacities; our projects are not based on a 2-week model. For example, we have conferences that we can only iterate on once per year. We are going through this now. We are actually going to be inviting stakeholders who we serve, such as program leader in the community, affiliates, other staff, to feedback on our process and let us know if we are on the right track Communications! We rely on having a communications plan for much of our work, that involves thinking of the concept or idea to promote, imagine possible responses, and what comms formats and release dates we look forward to having. We use a combination of social media, blogs, mailing lists and meta portal to outreach community members and generate engagement. Our work has two aspects to maintain, tools that we create and documentation for learning. Much of the documentation might change from one year to the next.
Community Resources What kind of support and resources to grantees and committees need and want? Define the two types of personas: grant applicants and committee members. Motivation and needs of both. How do goals of WMF and CR stakeholders intersect? Create a shared understanding of the needs and limitations of both groups. -community consultations to better understand processes like grant applications; -have multiple converations with stakeholders about the meaning of impact; - future scenario: a world where we understand each region in its peculiarity and not as monoliths Process evaluation after each grant report; survey to grantees. And after a few years we might do a big redesign of the grants programs overall to meet needs
Front-end Engineers Understand what the feature is. Implement analytics to gather user data. What tools (libraries, plugins...) do already exist? Can we use existing tools? How can we avoid wasting effort/duplicating work? Helping understanding of non-devs technical constraints around concepts

Maybe even extending ideas with small technical refinements, add-ons. What tools do we select? Select them

Write code, write tests, iterate Coding conventions, coding guidelines, make sure implementation respects all the acceptance crtieria, verify security, verify performance Ensuring documentation is reflecting released product Fix bugs
UX Engineers Understand the issue (need & challenge), look if issue has been come up elsewhere before. Look to gather all perspectives on it. Work with design to make sure concepts are not too far from reality without stopping early concepts Early addressing UI/UX/tech pitfalls, that might end up in product dead ends later. How can we build this prototype fast? Build prototypes, AB testing and other evaluation Consult with stakeholders, users, analytics, and determine what works and what doesn't Announce features, results and reports on UX-related channels, listening to feeback, stepping back & then ensuring that product is documented well. Iterate
Research and Data Literature review; Exploratory analyses (scale & frequency); Ethnography (casual -- usually); Document on Meta. Sketching; Story telling; Proposal writing; Exploratory analysis (correlations); Document on Meta; Review of available technologies; Replication of past analyses; Design and review methodology; Discuss plans with other researchers, stakeholders, and specialists; Document on Meta; Calibrate measurement strategy; Take measurements; Run simulations; Develop technological intervention; Design & run field experiment; Recruit participants; Perform offline analyses; Compare methodologies; Document on Meta; Provide data and present findings in a way and in language that can support decision-making. Be heard. Iterate on explanations for observations. Propose future work. Perform followup analyses (proposed by stakeholders). Document on Meta. Provide data and present findings in a way and in language that can support the release; Write up manuscript for future reference; Publish in peer-reviewed journals. Give talks to broad audiences -- encourage others to explore "Future work" for us. Upload datasets to public repositories; Document on Meta. Record data and do longitudinal analysis; re-experiment as more data comes in and verify initial/preliminary findings; Summarize supported hypotheses into Theory. Support external researchers to replicate/extend our work. Document on Meta.
Communications Help define audience segments

Build understanding of audience media habits - What do they like to read/watch? - What media outlets and personalities are trusted? Assess Wikipedia Brand Awareness and Perceptions (familiarity, trust, concerns, hopes, etc.)

Help identify audience needs and develop "user challenge" narratives (like Community Liasions "user stories") Help validate potential concepts through social channel publishing

Evaluate communications resources needed to support product launch and support

Continue A/B Messaging testing on social channels to build robust understanding of audience reactions

Run product messaging/positioning workshop to guide ongoing communications Brainstorm Creative Materials and Language to succinctly communicate product attributes Ensure language localization of messages (and identify problem areas in non-translatable terms) Work with ComCom list to storytell across regions

Identify pathways to audience based on known social media and press preferences

Develop communications strategy needed to reach audience on key channels

Support product launch with multichannel communication strategy, which may include:

- announcement/media outreach - social media promotion - supporting collateral: video, product descriptions, FAQs

Continue messaging to reinforce product needs and continue collecting user feedback
Team Practices Group Help teams execute, keep work coordinated with cross functional stakeholders, promote general communication, and sharing metrics.