5 Year NASA Wikiversary

This page is an example of how analytics of a set of wikis can educate and motivate wiki users. This page was taken directly from NASA's wikis. It was originally created as a look at the growth of the NASA wikis over the first 5 years. This page was shared here as an example for others looking to motivate wiki users.

Pigeons celebrating the 5th birthday of the woo....wooo.....Wiki!

5 years of wiki

On November 8, 2011 the Extravehicular Activity (EVA) Wiki was created and the first revision was recorded. 5 years later there have been over 250,000 revisions made by over 525 users across wikis representing 17 organizations. With over 25,500 articles and 19,000 uploaded files we have changed the way information is stored, used, and shared. Of the 2452 people who have ever viewed the wiki, 525 have made at least one contribution and 151 are currently active contributors.

Why are so many people joining the wiki way? Because we shouldn't struggle to find the information we need. Because outdated and unreliable information is unacceptable. Because our information should work for us. Our mission is making reliable information easy to find. We do this by fighting duplication, integrating data silos, exposing discrepancies through transparency and semantic relationships, making documentation review more agile, and encouraging everyone to be bold.

This article explores how this revolution in knowledge management is succeeding.

Let's start with a look at how it all began.

Our grassroots beginning

Read more about our beginning in story form at the Wikimedia Blog and in technical paper form via EVA Wiki – Transforming Knowledge Management for EVA Flight Controllers and Instructors. We also have a couple videos of presentations: History of the EVA Wiki and Meeting Minutes & Group Communication (SMWCon Fall 2014).

Data for plots in this section were obtained through two different MySQL queries.

Stacked bar graph showing the number of EVA Wiki articles organized by category along with category descriptions (as of mid-2016).

Our story starts with a few crew instructors and flight controllers in the EVA group. They struggled with finding the information necessary for their jobs. When they did find info, it was often duplicated across multiple files in different locations with names like "Rev B", "Rev 2", and "Rev Steve". There were many discrepancies between things like console handbooks and lesson plans. With awesome tools like Google and Wikipedia in the public Internet, why didn't NASA have an equivalent for its data?

In an unadvertised guerrilla project, these few people installed MediaWiki (the core software behind Wikipedia) on a test server to see if it could work. The EVA wiki grew organically through reading required for proficiency training. Instead of simply reviewing the content, the content was added to the wiki. Because of this transparent single-source method, discrepancies and gaps in knowledge were found in lesson plans and hardware pages (based on content from lesson plans, console handbooks, Operational Control Agreement Databases, etc). As they grew a critical mass, others in the group began to take notice and they wanted in.

Contributions

• 
  Contributions to the EVA Wiki by Group per Month
• 
  Contributions to the EVA Wiki by Group per Month (% of Total)

The two stacked area graphs above show the growth of the EVA wiki as represented by revisions per month split across three groups: CX3 wiki admins, CX3 non-admins, non-CX3 (CX3 is the NASA org code for the ExtraVehicular Activity group in the Flight Operations Directorate). By comparing the two plots, you can see that the initial bootstrapping of the wiki was done by the few wiki admins. Over time their volume of contributions have remained steady, though it is interesting to see the periodic fluctuations in amplitude. Note that there are a few spikes such as the one in late 2013 that are due to bulk imports of content.

In late 2012, we see the first contributions from non-admins in CX3. Over time, the volume of contributions from CX3 non-admins continues to grow. As a percentage of the whole, you can see how the rest of the CX3 group steadily increased their activity and became the primary contributing group. Now the admins can focus on templates, extensions, and broad-scope changes that help the rest of the group.

It's also worth noting that a small amount of the contributions to the EVA wiki come from people outside CX3. Since there haven't been any incidents resulting in excluding contributions from non-CX3 users, this is evidence that collaboration across org codes is possible (and good).

Viewership

• 
  Views of the EVA Wiki by Group per Month
• 
  Views of the EVA Wiki by Group per Month (% of Total)

In the next two plots (above), we see the same types of plot but for views. Note that we didn't start tracking this data until late 2013. At that time, we saw about 10,000 views per month. In mid-2016 we peaked over 40,000 views. From the second plot, you can see how about 20% of our views now come from outside CX3. The spike in late 2015 was from JSC's Chief Knowledge Architect scraping our wiki to test new text analytics tools.

The people contributing

• 
  Unique Contributors to the EVA Wiki by Group per Month
• 
  Unique Viewers of the EVA Wiki by Group per Month

These next two plots focus more on the unique users contributing to and viewing the EVA wiki per month. These plots are colored by group, showing the number of unique users per month. The first plot is for contributions and the second is for views.

For the contributions, note the distinctive jump in CX3 non-admin contributions in late 2012. That number has steadily grown such that most of our 47 CX3 users are contributing each month. Note the slight, but steady increase in the green non-CX3 area. This means more people outside of the group are contributing to this knowledge sharing system.

For the views, it's easy to see that nearly every CX3 user views the wiki regularly. It's also very clear that the number of unique people viewing the EVA wiki content is continuously growing. In mid-2016 we peaked at over 400 unique people per month consuming content from the EVA wiki.

Diversification of contributions

Contributions to the EVA Wiki by Unique Users per Month (% of Total)

This plot is much more colorful than the others so far because it shows each unique EVA wiki user's contributions as a percentage of the whole. So at first, it was just a few people. Over time, the number of users has grown and the responsibility of adding and revising content is now spread across the entire set of users.

This is crowd-sourced knowledge sharing at work.

Steady growth of content

• 
  Unique Pages Revised in the EVA Wiki by Group per Month
• 
  Unique Pages Viewed in the EVA Wiki by Group per Month

In the above plots, we can see the number of unique pages in the EVA wiki revised per month, split by the groups discussed above. It appears these numbers are generally correlated to the number of active users and how many revisions they are making. Note that as of 2016, contributors are revising between 1,000 and 2,000 unique pages each month and viewers are visiting between 5,000 and 8,000 unique pages per month.

Does this amount of viewership and contributor-ship ensure that revisions are being reviewed? Read on to find out.

Watch analytics

For a more in-depth presentation of this section, watch Wiki Accountability and Culture Shift (2015 Houston MediaWiki Meetup).

After we had a critical mass of content and a growing number of active users, we assumed our content was under continuous review. To test this assumption, we built the Watch Analytics extension.

• 
  State of the EVA Wiki as of July 2014

The above plot shows the state of the EVA Wiki as of July 2014. The orange dots around the ellipse represent users. The blue dots represent wiki pages. The lines between each dot represents a user watching a page. By clicking the star at the top of any page near the search box, a user can be notified of future changes to the page. Being added as a watcher is also a default setting when a user revises a page. If the line is gray, the user has viewed the current revision of that page. If the page is red, the page has been revised since they last visited it. So red means there has been a content change on that page that they have not yet reviewed.

When we first saw this plot, it opened our eyes. We immediately recognized we had two major issues.

First, note all the pages that are clustered outside of the ellipse. Those are groups of pages only being watched by a single user. In most cases, that means the only person reviewing the content of those pages is the author. That's bad.

Second, the amount of red lines told us that people were not reviewing changes as we assumed. That's also bad.

To address this problem, we created the special Pending Reviews page (linked at the top-right of every page). This provides clear direction to users on which pages need to be reviewed based on who is watching them and how many other people have already reviewed any changes.

In order to encourage our users to watch more pages (to reduce the number of pages with 0 or 1 watchers), we added another feature to the Pending Reviews page. Once you work your way through the list of changed pages, the wiki offers "Watch Suggestions". Similar to how Amazon offers products based on your activity, the wiki suggests pages that you might be interested in watching.

So did all this work?

• 
  State of the EVA Wiki as of April 2015

This next plot shows the state of the EVA wiki after almost a year. It's pretty clear that these new features helped our situation. Now the question is whether we can sustain this behavior.

• 
  State of the EVA Wiki as of October 2016

In this third plot, we see the state of the EVA wiki as of October 2016. Clearly the wiki is growing and we still have troublesome spots to address. So our work is not done. But an important point to acknowledge here is that with the wiki we now have the power to quantify and measure the state of our knowledge management system. Can you name any other system that provides such insight and feedback to its users?

If you'd like to see the state of any wiki, go to the Special:WatchAnalytics page (in that wiki).

From ones of wikis to tens of wikis

We've spent enough time talking about a single group. Let's take a look at how other groups have joined the wiki way.

• 
  Start date of each wiki

The above graphic shows the date of the first revision for each wiki. After other groups starting hearing about (and in some cases using) the EVA wiki, they asked to create wikis of their own. Mission Systems (CD) was first to join and they made use of the Semantic MediaWiki feature to catalog and analyze their data. The Flight Operations Directorate (FOD) and ROBO wikis were next. The FOD wiki was created to house any general information useful to FOD that wasn't under export control or other data protection. It wasn't until much later that we put serious thought into how data protection could dictate our overall wiki architecture. We'll talk about that in a later section. With the ROBO group, they already had some content in a SharePoint wiki. James Montalvo found a way to programmatically import all of that content into our MediaWiki format. This bootstrapping gave the ROBO team the same asset they had before, but now with the power of Semantic MediaWiki and other features available in our platform.

Growth of content as each user group joined

• 
  Unique content pages across all wikis
• 
  Unique content pages across all wikis (stacked area)
• 
  Unique content pages of the 3 largest wikis (EVA, OSO, and ROBO)
• 
  Unique content pages of the smaller wikis

The above plots show the number of unique content pages in different ways. Some groups like EVA have steadily grown their knowledge base over time while some groups have clearly benefitted from mass imports of data. For example, the OSO team imported everything from their KB system so the information would be integrated with the rest of their knowledge base.

More and more contributors

• 
  Active contributors across all wikis
• 
  Active contributors across all wikis (stacked area)
• 
  Active contributors for each wiki

These next plots show the number of active contributors by month.

Wiki User Revision Scatterplot

This last plot shows every revision for each person, color-coded by wiki. At initial glance, it's just another way to show different groups of people coming onboard, contributing to separate wikis. But if you look closely, you can see how some users cross these wiki boundaries. You can also find some cases where a user was once active and no longer participates. In some cases, these are co-ops contributing during their tour. It's important that we acknowledge those users who have continued their contributions with the same passion as when they started.

Our collective impact

• 
  Views and revisions across all wikis
• 
  Unique user views and revisions across all wikis

Finally, let's take a look at views and revisions across all wikis. Our increasing participation by active contributors is constantly growing our collective knowledge base. Currently, that content is being consumed (viewed) 100,000 times per month, spread out across about 800 unique viewers.

How do we compare with Wikipedia?

Wikipedia started in 2001, but how does our first 5 years compare to their first 5 years? Keep in mind that Wikipedia is open to the public Internet with the scope of the sum of all human knowledge while ours began small, only focused on International Space Station (ISS) EVA operations knowledge and eventually growing to include the sum of all ISS knowledge.

• 
  Unique Content Pages Across All NASA Wikis
• 
  Unique Content Pages in Wikipedia
• 
  Active Contributors Across All NASA Wikis
• 
  Active Contributors in Wikipedia

The plots above compare our number of content pages and our number of active contributors to those of the first 5 years of Wikipedia. It's interesting to see how similar our growth patterns match Wikipedia. While we are certainly not 1-for-1 in volume, there are definite similarities in the slopes. If we look at specific numbers we can compare ratios. At the 5-year mark Wikipedia had about 835,000 content pages and 21,111 active contributors. That's a ratio of about 40 pages per user. At our 5-year mark we have about 30,000 content pages and 151 active contributors for a ratio of about 200 pages per user. Our wiki contributors have been busy!

We're doing really well with productivity of active contributors, but how well are we doing when comparing page views per active contributors? English Wikipedia currently averages 8 billion page views per month and 30,000 active contributors per month. First of all, think about the numbers here. For the billions of people who use English Wikipedia, there are only 30,000 active contributors updating the information. Now if we divide the 8B page views by 30k active contributors per month, that's an average of 266,667 page views per active contributor. For the NASA wikis, we average 88,000 page views per month and we have 151 active contributors per month. Keep in mind that these wikis currently only represent a portion of FOD and a few groups outside FOD. That's an average of 583 page views per active contributor. So on one hand, you could say Wikipedia is getting a lot more use out of the content they curate. But on the other hand, this means our contributors are much more active keeping our information up-to-date and we have a higher percentage of users who feel compelled to contribute. We'll talk about that more in the next section.

Other fun metrics

The 1%

In Internet culture, the 1% rule is a rule of thumb pertaining to participation in an internet community, stating that only 1% of the users of a website actively create new content, while the other 99% of the participants only lurk. Variants include the 1-9-90 rule (sometimes 90–9–1 principle or the 89:10:1 ratio), which states that in a collaborative website such as a wiki, 90% of the participants of a community only view content, 9% of the participants edit content, and 1% of the participants actively create new content.

Let's look at the numbers for our network of wikis:

• 2452 people who have ever viewed the wiki (lurkers)
• 525 people who have made at least one contribution (contributors)
• 376 people who have created content pages in the wiki (creators), a subset of contributors

{{#vardefine: Lurkers|82.364796775277 }} {{#vardefine: Contributors|5.0050386294928 }} {{#vardefine: Creators|12.63016459523 }} {{#subobject:|Data=5 Year NASA Wikiversary|Has number={{#var: Lurkers }}|Has label=Lurkers}} {{#subobject:|Data=5 Year NASA Wikiversary|Has number={{#var: Contributors }}|Has label=Contributors}} {{#subobject:|Data=5 Year NASA Wikiversary|Has number={{#var: Creators }}|Has label=Creators}}

Wiki User Revision Scatterplot

If the proportions of the pie chart look strange, it's because creators are actually a subset of contributors. Anyone who has created new pages is inherently also a contributor.

In comparison, we can see how our community has nearly twice as many contributors as typical Internet communities. Additionally, most of those contributors are bold enough to create new pages.

It's 5 o'clock somewhere

• 
  Heatmap of revision activity by hour
• 
  Distribution of revisions by hour
• 
  Distribution of revisions by hour, split by each wiki's % of the whole hour
• 
  Diversity of revisions by hour (users)

This series of plots looks at activity grouped by the hour of day. In the heatmap, you can see that most people make revisions during "normal" work hours. You can also see which wikis are more active than the others, with the EVA wiki ablaze in red. In the monochromatic histogram, we see again how most people contribute during the normal 9-5 work hours. Note the lunchtime lull, though. Perhaps wiki contributors are afraid of getting sauce on their keyboards.

The next two plots are much more colorful. These still show revisions grouped by hour, but the colors are significant. The first plot, where colors represent different wikis, shows an interesting red blob in the early hours of the day. Which group makes up the majority of the contributions between 2 and 5 AM Central Time? The Houston Support Group does, because of their time spent in Russia! The last plot, where colors represent different users, tells us that the HSG blob is largely the result of a single, dedicated user spending his time abroad contributing to his group's knowledge base.

Diversity keeps us honest

• 
  Diversity of revisions by users
• 
  Diversity of revisions by year

These two plots help to show the diversity of our wikis. The first is split into bars by wiki. Each color within a bar represents a different user in that wiki. The size of each color block is based on how much that user contributes relative to the rest of the users. So you can easily see how the older, larger wikis have become more diverse in their contributor base. Younger wikis still rely on the few motivated users adding content until they reach a critical mass for group adoption.

The second plot shows each year represented by a bar. Each year's bar is split by users, where the size is based on that user's contribution compared to the rest. As you may expect, our wiki contributor base gets more and more diverse each year. You might notice the large purple block in 2016. That's due to a single user importing a massive amount of data from another database into the wiki, where it can be better integrated with all the other information.

The evolution of a wiki page

One more thing, just because it's cool...

Over the course of 5 days, the US EVA 34 page evolved from a Current ISS Issues page to an EVA planning page and then to a historical documentation of the event, all through 44 revisions. Below is an animation showing how the page grew and evolved over those 5 days. This is just one example of how our knowledge capture can grow organically with the right participation.

The future

Our number of wikis is growing. As we spawn new wikis, each of those groups benefit from the power of the wiki but as a whole our knowledge remains segregated by org code. We are evaluating options for the best wiki architecture for the ISS Program. This might include creating a large ISS wiki to hold the bulk of each group's knowledge that is not under special data protection. This might also include developing software to automatically transfer data between a network of wikis.

See Wiki Architecture for more information.

Other cool stuff

Recognition from Wikipedia

Youtube link

Our friends at Wikipedia made this promotional video because they think it's cool that NASA uses their software. Because we find their software so useful, we have engaged the Wikimedia Foundation in conferences, hackathons, and of course the online community. We give back to the open source project through community activity, improving their software extensions, and writing and sharing our own extensions.

Read more about our story:
“It pays itself back in dividends”: NASA and MediaWiki, a natural pairing

Explaining the why and the how

Youtube link

In this recorded presentation to the FOD OSO group, we provide discussion of the FOD Wikis, why we can trust the data found there, and why it is an excellent resource unlike the other knowledge management systems we've all seen. Presenters make the case that the wiki is the future for our content management.

Testimonials

When I first started using the wiki I felt a little lost, like I wanted to understand the "folder structure" of it, but then I thought to myself - "Why do I not feel this way when I Google for something?", and realized that I just needed to look at it in a different way. It's not a hierarchical structure, it's a web. It's all about making sure you name pages in a way that most people are likely to find them, creating redirects for alternate search terms, and being smart about what you type into the search box - just like on any browser search page.

I wish we had had this since the beginning of ISS assembly!

It's WAY BETTER than sliced bread! Truly, it is an incredible tool - in the EVA world, we often ask ourselves "how did we do our job before DOUG and the VR Lab?", and the wiki is the same - it's getting to the point that people ask "how did we ever get along before the wiki?"

It provides an excellent place to post knowledge so that once a person in the group has learned something others don't have to dig up the information the same way -- they have an easier way to search for it.

One important feature is the searchability of meeting minutes. Meeting agendas are sorted by date on outside websites. The wiki allows for the minutes to be searched allowing the agenda and the presentation to be found by people who have not memorized the date or the board the topic went to.

Wiki is now the first place I go to look for information.

The wiki has DEFINITELY improved information sharing within our group.

It has made incremental updates to documentation easier. This lowers the barrier to keeping documents up to date since you don't have to make big updates and then go through a lengthy approval process which discourages the updates. You can work on them around your other responsibilities.

The more everyone contributes the better the information becomes and the less work everyone needs to do to dig up the same info over and over again.

This will help your team stay in sync when information changes frequently.

It's NOT that hard, just do it! If you see information that is incorrect, and you have the knowledge to correct it, this is what the wiki is all about. The wiki is where your team goes to find information, put your work out there for others to utilize.

The wiki is "our" wiki. The more you can contribute the better we are as a group. The wiki is a platform to eliminate information and knowledge gaps within our organization. If you are already a user of the wiki (you take information from it) please do your part and contribute information back to the wiki. The wiki is stronger with a diversity of contributors.

The wiki is only as good as what people put into it. The unique information that you have captured in your brain will never make it there unless you put it there.

The wiki is such a powerful tool that is not hard to learn and use, such that it can help us identify gaps in our knowledge and training that we never would have otherwise knew existed.

Just do it. Jump in. Submit some articles. There is a real gratification that comes from watching people use and develop something that you created.

The wiki becomes more and more useful the more involved and invested people are.

How great is the internet - when we want to know some obscure and probably unimportant fact, it's just seconds away. The wiki can be that for us here at work - except for important information! We are the wiki builders and process integrators that will facilitate the evolution of the next level flight controller, instructor, and analyst, having quick access to important knowledge - like an extension to our brain. In the short term, it will take all of us to contribute and continue to maintain the wiki until it becomes a normal part of our everyday process.

Wiki is a great way to pool disparate knowledge. You will find answers to questions you never knew to ask.

Every person has knowledge that is unknown to others, and by sharing that knowledge you are participating in the betterment of the group. By spending a short amount of time adding to the information contained within the wiki, you are essentially providing 24-7 training to your team without having to be present for that training.

Now, whenever someone learns something of use to others, it doesn't get buried in email or meeting minutes. Even when mentioned in those forums, someone always says "Add it to the wiki!", or just adds it themselves. We constantly refer other groups to the wiki and invariably they find it immensely helpful. It is fantastic for documenting information, researching history (in as far as the wiki goes back), and linking related things together, as well as performing custom queries and other features.

There's very little that isn't helpful to be on the wiki - even if it's just a link to the official place for the information, because the wiki is easy to search unlike most other NASA websites. For example, I wanted to remind myself how to find out when I could expect my next computer refresh, and boom, "There's a wiki page for that!"

Sometimes it's like magic. You go put a "clarify" tag or some other question out there, and sometimes within just moments you get a notification that someone has updated that page with the answer! It's a fabulous repository for information with seemingly endless flexibility for the needs of displaying different types of information, and yet it doesn't seem to suffer from the complexity/user frustration/too many options of other highly flexible tools (think Fox). Much of what makes this work so well is the expertise and passion of the wiki administrators who can relatively quickly create templates and other tools that make these things easy.

I could immediately see the benefits of it to fill a huge knowledge capture gap that we've been struggling with for years.

How do I start contributing?

The strength of our wikis comes from active participation of the users. Without continuous contributions and qualified reviews from everyone in the community, our wikis will become just another dumpster of outdated and unreliable information. But unlike other antiquated data silos burdened with long review cycles, you have an opportunity to contribute to the most accountable knowledge management system available.

Visit Wiki Fundamentals if you're interested in joining the movement.

References

See how this page was made.