MediaWiki

Wikimedia Developer Summit/2018/Embracing Open Source Software

< Wikimedia Developer Summit‎ | 2018

Embracing Open Source Software

Structured agenda

  • Consider challenges and opportunities to frame the work towards working on strategic solutions (1 hour)
    • Discuss ideas for challenges and opportunities guided by the 4 topics
  • Discuss next steps for the conversation (30 minutes after the break)
    • Summarize the general topics
    • Who should be involved? How do we encourage diverse involvement? How do we recognize proper stakeholders to participate?
    • Where should it take place?
    • Can we recognize drivers for the process? For pieces of it?

DevSummit event edit

  • Day & Time: Tuesday, 2:00 pm – 4:00 pm
  • Room: Kinzie
  • Facilitator: Deb
  • Notetaker(s): Niklas, Trizek, Santhosh, Brion, Moriel,

Session Notes: Introduction by Moriel: There are many angles we can look into this. Basically four angles in the introduction (look at this etherpad). Some of these topics were already touched in previous sessions. If anyone has any other angle we missed here, please add. In next 6-9 months the foundation will start working on a strategy.


The WMF (movement?) should commit to using open source internally even when inconvenient

  • Matanya: Usage of open source is out of scope. Does anyone think we shouldn't be using open source?
  • Brian: I have seen comments from others that are ... against using open source.
  • Giuseppe: we use Google docs, even in the preparation of this summit itself. Also a question about dogfooding.
  • Kunal: I disagree that everything should be prefixed(?) with WMF. For example the Toolserver allowed proprietary software, and we were unable to liberate them during the shutdown and migration to Toolforge. This should apply to the entire movement and not just the Foundation.
  • Subbu: Clarification on WMF or WMF projects / movement required -- they are not the same thing.
  • Moriel: Good question. We might want to have that conversation. We could start by looking at Foundation run projects.
  • James Montalvo: These kind of questions come up on wikitech.
  • Tgr: Everyone agrees that the software used to run the website should be using open source. There is a vocal group that foundation using closed source things by foundation for tasks like google doc. thinks that other software does not need to be open source. It is a cultural issue.
  • Moriel: I agree but I want to focus a bit more. Focus on OSS communities is in the strategy. How the WMF behaves on it? What will the WMF will do compared to what it will promise.
  • Matt: I agree we need to separate the feature stack we develop and use(?) that we use to run website and other software we use. We have been very strong on open source but no longer 100%. We use prop. services like machine translation. There is no difference between a proprietary API (SaaS) and proprietary in-house software. I'd like to focus on the feature stack first (web sites, apps, and entire stack, e.g. Apache, all the way up) and go back to 100% open source for this. We should keep working on the office IT in the background (and avoid additional proprietary software), but clearly we're already behind on that....
    • Giuseppe: In our infrastucture almost everything is open source, but networking gear is an exception. That cannot be fixed without causing huge issues.
    • dchan: Please no rule against proprietary machine translation. For languages which have no open source MT, we are part of the solution - initial use of proprietary MT will help us build a corpus that can be used to build open source MT. (cf Richard Stallman's willingness to use proprietary software in order to build its replacement)
      • Matt: First, I don't think using proprietary software for the feature stack is consistent with our precedent and values. I also don't think it's possible to redraft our values and explain in a coherent way why this is an exception, without creating a slippery slope. This is also the first time I've heard this argument (we're only using proprietary MT temporarily until we can replace it) despite being on the Global Collaboration team and having talked about it in the past. So I'm not sure the Foundation as a whole is committed to eventually dropping proprietary MT (though I would welcome that).
        • dchan: Ok, then let me rephrase my request as "please no blanket prohibition against all proprietary MT" - I'm happy for the use cases to be considered carefully.
  • Matanya: ... is not only a WMF concern (?)
  • Brion Vibber: It is important to consider when to use open or closed source. One key point to look at is the transition cost away from the closed source software. For example Google Docs, you can download everything, though you may lose metadata/ownership/sharing/links. Cost of getting out is more important than the cost of getting in when picking a solution.
  • TheDJ: so far those questions have been focused towards WMF. How important it is towards our users. I'm an OSS supporter and developers since 18 years. The mission of Wikipedia is bigget than open source software. I want our movement to succeed.and ready for compromises. what context we should collect amongst our users. what important is for them? What is their definition of freedom? ...?
    • Moriel: I think we are starting to repeat ourselves a bit. So let's move to the next angle. Will being more open souce put the movement more as leader for...

The movement should commit to being truly open source (easy install, good docs, easy to contribute)

  • Moriel: Yes, the movement, not the WMF.
  • Kunal: We should use free tools. Free IRC client was challenge for google code-in. Should recommend which free software to use. Brainstorming documents have to be moved away from google docs once ready. Email is ok. Also, Github is not free.
  • Tpt: 1/ to get more community contributors 2/ have a software easy to install and to use - Enables a federation based wikimedia sites, not all content stored by the same actor.
  • Mark: There are other open source entities, Mozilla is an example. They are great (?) but people still say they are not good enough (for open source citizens?).You cannot please everybody anyway
  • Matanya, DR to Kunal: It is about awaraneses and mindset. Once you have the mindset, ... the actual tooling is less important. Like Mark said, not everyone is going to be happy all the time. We should tell people what we plan to do and let them deal with it.
  • Moriel also DR: users will always find the place to complain. ?? to use more open source, or make open source better. I think we can move to the next one.


The movement should commit to encouraging downstream usage of WMF libraries

  • Moriel: In the ticket there are some resources to libraries we maintain. We have solved things that are commonly problematic in the internet? but our solutions only work inside MediaWiki. Couple of examples in top of this etherpad. Low hanging fruit: mw.msg (has jquery.i18n on JavaScript side; Intuition in PHP). 2/ Should we take OOJS and make it widely available.
    • Santhosh: Yes we should encourage downstream users. That will expand our engineering capacity. By making our code these libraries we got more contributions. 2/ We are getting more users and thus more eyeballs. We have these advantages, but we also have limitations. Do we have separate topic for limitations? [Moriel: no]. We should listen to the downstream users and accept their changes. Many of our open source libraries are run as volunteer effort. Many were started as pet projects.
  • Benoit: We should do the same for needs we have: use libraries that exist made by OSS friends. [Moriel: that's the topic of the next section] Let's not create a new library that replicates something that already exists.
    • Moriel: We shouldn't just advertize things indiscriminately. We solved a lot of problems that other people can definitely use. People who don't want to use all of OOUI, they can still see the solutions for specific issues like working around the browsers bugs. We have a lot of knowledge that we can share.
  • Kunal: Few years ago Bryan Davis, .. and me in the MediaWiki core team made some of code as independent composer libraries. The main thing these days is for people to take time to review it. We could improve the tooling to make it easier to create libraries. https://blog.wikimedia.org/2015/01/29/modernizing-mediawiki-with-libraries/ AND https://www.mediawiki.org/wiki/Library_infrastructure_for_MediaWiki
  • Tgr: Some frequently used code like Status or Message is not librarized. It would be good to get more resource for that and not do it volunteer basis. 2/ Promote libraries so people are actually aware they exist.
  • Sam: The mediawiki packages in packagist is a mess. https://packagist.org/packages/mediawiki/
  • Madhumitha: Putting code in github is not same as opensource. You need maintainers, people listen to people's issues, blogs, evangelization etc required
  • David: Some organizations maintain a list of their open source libraries, and perhaps we should as well.
  • Matanya: Installing from distribution packages doesn't really work for MediaWiki.
  • Moriel: We already talked how to encourage newbies. Challenge: we need to install full MediaWiki environment to fix a bug. But when we libraritize things, it will be easier. Luckily, because embracing open source is already a part of the strategy, .... We seem to have the ground work done and a lot of goodwill.
  • Corey: Libraries open other communities to us.
  • Moriel: Any more points before we move on? I LOVE this topic.

The movement should commit to being involved with upstream projects

  • Moriel: Should we be more involved? We talked a bit this in the previous session from eyes of expanding the contributor base. Should we be more active and encourage participation in upstream code.
  • Matt: I want to connect with previous session. We discussed about collaborating with other projects. I would like us to broaden our vision. Currently, we often have the mindset that either something is an in-house project (like MediaWiki core) or an upstream project (like HHVM) that we contribute to somewhat. I would like us to consider another model of major partnerships where we commit serious resources in our annual planning, and possibly help start new joint projects. This can help with major challenges like Video, 3d,... these are too big to do on our own.
  • TheDJ: Even when we contribute to other projects we never talk about it. We made 18 technical posts on the blog in the past ? months. We should talk about this contributions. We never explained to the people in the big picture the value of it. Doing that would actually attract a lot of other people that would be interest to solve our problems. What did we get from other projects on 2017? Discourse wrote about this: https://blog.discourse.org/2017/12/discourse-gives-back-2017/
  • Tgr: Open source is just not a license. It is also the ethics of paying back and not just freeriding. If everyone freerides, opensource dies. WMF is not a small organization anymore. It should support things like PHP by time or money.
  • Moriel: Any more thoughts it means to be open source?
  • Santhosh: Do we want to ask what challenges people see? To solve these issues.
  • Moriel: Let's add that.

List of challenges begin/using FLOSS:

  • Matt: We need to integrate this into the planning. If we want to do bigger projects, it needs to clearly stand in the annual plan with resources committed to it.
  • Roan: One of our most succesful libraries is CSSJanus maintained by Timo. https://www.mediawiki.org/wiki/CSSJanus We ported it to PHP and JavaScript while working on the resource loader. We did that on foundation time. Resource loader is sorta maintained by the performance team.
  • Matanya: CSSJanus flips directions for right to left languages. With it, I can use a normal Internet, in Hebrew.
    • Moriel: I've been going to the Unicode conference. Anyone there found CSSJanus super exciting but they were not aware of it because we don't promote it.
  • TheDJ: One of the challenge here is: we (especially the Foundation of which I'm not part) continuously need to explain this to certain layers of the organization: as a group of developers we need to write this down in a manifesto why it is beneficial to us. The fact we are not doing some of this stuff is because of opportunistic business thinking of getting away with it. If we just had good libraries with good documentation, that would speed up new employee boarding.
    • Deb: let's hear from people who haven't spoken yet?
    • Moriel: We can create that manifesto as a blog post. That would be great if we write what are the benefits. The code hierarchy when developing libraries... it will make the code better.
  • Benoit: Communities (as in final users on Wikipedia, Wiktionary...) will support a commitement to use FLOSS. Anytime someone proposes to use non-free software there is a big pushback from people. Example with CX using Yandex.
  • Santhosh: One weakness we have is reinventing the wheel. We cannot just take a library as is, some customization is required. But this is not a reason to write something from scratch.
  • James Montalvo: Not familiar with CSSJanus and RTL languages... it sounds like there a lot of other resource in the internet that are not up to speed in the way WMF is doing things.
    • Moriel: I have been public speaking how we solved the RTL problem. E.g. http://rtl.wtf/ - See 3 Lectures in sidebar. Big companies like Facebook, Google have big issues with RTL because libraries are not known or documented. It should not be a difficult choice because our libraries are open source (and some are MIT).
  • Timo: One reason we have made our own solution is that we have unique problems nobody else need to solve. Especially about languages and this is where we solved many problems. Using open source strategy not just for libraries but in general is something I'd like to see more.
    • TheDJ: I like Timo's comment about "We should hold ourselves to the same standard as we require from others[?]"
  • Moriel: When we come back, we will focus on the "now what??!!11!".

[photo, tea break]

Now what?!!

  • Matanya: [joking: I know it is not fun signing up, it's more fun to just discuss, but apparently we should do it.] Identify actionables.
  • Deb: people who haven't spoken yet, please speak up
  • Kunal: there is mw page on 'Developing libraries'. Like to see it expanded. Include blog posts, code reviews
  • Matanya: Is our code locked in gerrit? pros? cons?
    • Matt: it depends on which project it is. ?? I prefer to keep the open source. We have an obligation.
    • Moriel: How to improve gerrit experience?
      • Max: Only by throwing it away. But you can't.
    • Kunal: We discussed about improving opensource projects- so consider gerrit as one such project. Have also more people.
    • Matt: We do not need to stick to Gerrit. It seems Differential is better than Gerrit. There are also other open source tools out there, such as Gitlab, which we should explore.
    • https://phabricator.wikimedia.org/T167547 (declined)
  • Tgr: Currently we are doing not very good job of maintaining the libraries in GitHub. See https://phabricator.wikimedia.org/T136863
    • Matt [not spoken]: I support mirroring our code to GitHub (and everywhere else, it's open source, put it everywhere) and announcing it on Facebook, etc., it's just a question of what infrastructure we use.
  • Addshore: Source code browsing in Phabricator is pretty nice. We should stop telling people to use git review. It's just two commands in git that is needed.
    • https://gerrit-review.googlesource.com/Documentation/user-upload.html#push_create
    • Matt [not spoken]: We need to radically shorten and consolidate our Gerrit and git-review documentation (including how to install). I haven't had problems with git-review, though I know a few other people have. Also, git review -d is nice.
    • Addshore [not spoken]: +1 to consolidating documentaiton regarding gerrit / git-review.
  • Tpt: Better discoverability.
  • David: What do other open source communities use?
    • Most: Github.
      • Why are they comfortable using GitHub?
        • Matt [not spoken]: Not every organization has the same commitment to open source we do. Most projects can't afford to host their own projects or even pay for open source hosting.
        • Matt [not spoken]: As a major player in the open source world that people trust and support, we have the resources and obligation to go beyond what smaller projects do and build up the open source world.
          • Can we offer project hosting to projects that aren't directly related to Wikimedia? That'd be cool!
            • GitLab does this for free and their software is open source. But we don't trust them enough for our code?
              • We declined the task: https://phabricator.wikimedia.org/T167547 -- with the comment about invesetigating it "some time ago [...] the feature list is incredibly lacking". Is that still the case? ¯\_(ツ)_/¯ Feel free to reopen it.
    • Drupal: Open Source Custom System (built on Drupal) that is a patch-based workflow built on comments. Example: https://www.drupal.org/project/drupal/issues/2765959 The system also does attributions to individuals and organizations https://www.drupal.org/organizations who provide the patches.
    • WordPress: Subversion-based system built and hosted in-house (source not available, I think), Trac for bugtracking, deployments are automatically built from SVN tags.
  • Max: Last time I checked Phabricator also required separate tool to commit code (arcanist). One problem I notice is everything is a plugin in phabricator. No streamlined experience. I dont like phabricator as code hosting platform. can we use github? or gitlab?
  • Brion: Re: git-review. In my experience all git based tools have different tools/ways of pushing changes. It's just matter of documentation and usability. 2/ I second gitlab being an interesting project. Recently GNOME migrated to gitlab in order to improve usability while retaining their own infrastructure. Deciding to move such things may be out of scope, but something to consider.
  • Tgr: ... File a task and start discussing user experience of Phabricator
    • Giuseppe +1
    • Matt [not spoken]: Callsigns (rMW) are already optional, there are some improvements that can be made to the code browser, but it looks and works reasonably well.
  • Giuseppe: What we can say here is to give a strong mandate to make it better.
  • ??: ... Github has a search engine. People can discover our code that way.
  • Sam: There is no reason to stop replicating to GitHub.
  • Matanya: I would like us to get back to the main topic. Are there actions or next steps we should take?
  • Moriel: we need owners. Please add your name.
  • Matanya: Do people think we should have discussion about ...? For example PHP.
    • Kunal: We employ Stas, but I don't know whether he writes code for upstream.
    • ?: Should we fund such upstream developers? +1
  • Giuseppe: ?
  • Moriel: There are a lot of companies there that do working groups across other companies. We could do that.
  • Matanya: I want to ensure the action items are clear. Does anyone have comments about promotion?
  • Moriel: I think it could be split.
  • Matt: ?
  • Tgr: We have zero action items on the two other questions. Is that okay?
  • TheDJ: When we talk about how to assign responsibility of promotion. It adds to stress and it gets muddied quite quickly. The blog post is not only about promotion, but also to have time internally to evaluate. We need to build stuff and we are really busy. But if there is a process that shows us whether we have done nothing or have made progress.
  • Moriel: I am gonna summarize. We went over a lot of things today. I think we came up with pretty decent action items. Some of them are easier and some require more dedication. We definitely should continue this conversation. Please, after this session, go over the notes, and if you see action item missing, do add it. Please stay tuned to follow-ups in next couple of months!

[20 minutes left]

  • Tony Sebro (New WMF Deputy General Counsel) introduces himself: happy to be here.
    • Giuseppe: ...?
    • Tony: the fights about terminology can be exhausting.
    • Giuseppe: we have an issue with our smaller projects: lot of them are posted in some git repository without proper licenses. E.g. operations
    • Tony: Work hard to get documentation correct. It makes it easier for 3rd parties to come in. Decide what your risk tolerances are.
    • Mark: How much work it would be to setup a MediaWiki foundation.
    • Tony: Legally not much. Setting up new 503(c) orgs used to be problematic with the IRS but not anymore. What would be the funding model and its overhead?


Action items, next steps, and owners:

  1. write a blog post of our floss depnedencies and our interaction with them - Matanya
    1. Promotion [of libraries? which? -> who?]
      1. Send people to other conferences / share our work and see what others do
    2. https://phabricator.wikimedia.org/T185613 Collect list of channels where new Wikimedia software libraries could be promoted
  2. Write a blog post about the importance and benefit of making/maintaining libraries from our code - Moriel
    1. write a standard for writing code the open source way
    2. update https://www.mediawiki.org/wiki/Manual:Developing_libraries page/guide about how to write libraries (related to the blog post)
    3. Collect a list of all of our already available libraries
      1. <- needs owner!
    4. Investigate gerrit usability / alternatives
      1. <- needs owner!
    5. Investigate tactical changes to repositories so they have readme, etc
      1. <- needs owner!
    6. File a task to examine gitlab and/or alternatives to gerrit
      1. https://phabricator.wikimedia.org/T167547 (declined)
    7. Improve usability / discoverability of out GiHub mirrors:
      1. https://phabricator.wikimedia.org/T136863 - Gergő
    8. Find meetups and/or run meetups/workgroups with other companies/organizations that share mission/code etc
      1. <- needs owner!
Retrieved from "https://www.mediawiki.org/w/index.php?title=Wikimedia_Developer_Summit/2018/Embracing_Open_Source_Software&oldid=2709901"