Architecture meetings/WMF Engineering All-Hands 2013/Intro

Lightly-reformatted copy of email I sent to introduce this in 2013. I think it's worth rereading now as we discuss the WikiDev17. -- RobLa (talk) 20:25, 16 February 2016 (UTC)[reply]

Forwarded message ----------

 From: Rob Lanphier <robla@wikimedia.org>
 Date: Fri, Sep 6, 2013 at 5:01 PM
 Subject: Architecture discussion next week
 To: Development and Operations Engineers <engineering@lists.wikimedia.org>

Hi everyone,

I’d like to bring you up to speed on what we’re thinking about for the Architecture discussion at the tech days next week. I have a long background written up below, but I’d like to lead with the agenda.

The overall session is 90 minutes, and a lot of ground to cover:

15 minutes - Background: Basic explanation and clarifying discussion of the process
15 minutes - Role of our architects, senior developers, WMF in the process
15 minutes - Running through a couple example RFCs (no decisions, but highlighting the discussion; see below if you want to volunteer)
15 minutes - Brainstorming on RFCs that don’t exist yet, but should
10 minutes - How do we move things faster (e.g. open invite weekly meetings?)
10 minutes - Architecture summit in January
10 minutes - Summarize next steps; assign action items

Below is the way-too-long explanation about all of this. I’ve broken this up into sections which mostly mirror the agenda above.

Background

The way big architecture decisions get discussed and eventually implemented for MediaWiki is highly dependent on context. Much of the work is very driven by big features; for example, we made changes to the way the job queue works in service of Echo (with many benefits to other aspects of the system, such as how we handle video transcoding). Other work is driven by community development; for example, IAlex’s work on the context object. There’s still other work that people would like to embark on, but may not see a clear path forward, so they work around problems rather than address them. In all of these cases, developers are often unclear about what they “should” do, so they muddle through going with what worked for them in the past.

At the Amsterdam Hackathon, we started the process documenting our architectural guidelines (meetings). A goal of this documentation is to have some written guidelines to compare design proposals against, rather than basing our decisions on the design aesthetics of whichever reviewer happens to be looking at a big change in Gerrit. It has served as a means of having discussions some seemingly intractable disagreements and has defined a way forward for experiments in more controversial areas. For example, the idea that we should break up objects into value objects and logic objects is something we’re still divided on, if our discussion in Hong Kong is any guide. However, rather than continuing this argument in the abstract, we agreed that a specific example will be helpful, so Daniel Kinzler is working on an example based on the Title object.

Our defined process for making big changes to MediaWiki is the RFC4. We’ve had this process for a number of years, but until recently, we didn’t have a concrete commitment to move anything forward. Even now, the process is still a bit fuzzy, but things are moving forward. Tim Starling did a very preliminary assessment of all of the RFCs in the queue, and some have moved forward.

We’re now at the point where we need to have a conversation about how to make this really work for us, both as WMF Engineering and as a larger development community. We seem to have general agreement that MediaWiki could use more coherence in its design, but we’re still quite far from agreement on exactly what we should aspire to.

Role of our architects, senior developers, WMF in the process

We’ve gotten big enough as an organization that having just one or two people (e.g. Tim and Brion) make all of the major decisions about MediaWiki architecture just doesn’t scale, at least not the way we’re doing it today. But, as noted above, it’s also important that we strive for some level of coherence in how we do things, and a very common strategy is to centralize the decision-making in a single BDFL (e.g. Linux or Python). We need to have a discussion about what makes sense for the way we do development.

Example RFCs

I haven’t yet identified which RFCs would be best to talk about. The Front End folks are planning to talk about the LESS RFC in an earlier session, which might be useful for them to report on rather than talk about in detail. I’m open to suggestions, especially if you are planning to be here and have one that you feel comfortable quickly presenting to the group (lightning talk style).

Brainstorming on RFCs that don’t exist yet, but should

I think there’s a lot of things we know need to change about MediaWiki, but isn’t actually documented in RFC form. Additionally, there’s probably work that’s already underway that could stand a design review. Let’s just have an etherpad open and hammer out a list together of things that need RFCs. We can maybe put the result of that on a page titled “RFC/Incubator” or something like that.

Moving things faster

The architects generally have wanted to make the time to get together, but circumstances have intervened keeping that from happening. It may be helpful to have a more open process to make review happen faster (e.g. a weekly open-invite meeting). Please come prepared with your thoughts on what might work, and more importantly, what you personally are prepared to volunteer for to make things move more quickly.

Architecture summit in January

We’ve spent some time this summer in hazy planning mode for an architecture summit in January. The basic structure of the architecture summit would be:

Put out a call for RFCs for badly needed architectural changes, with a deadline some small but reasonable number of weeks beforehand
Cherry pick the most important RFCs for discussion resolution
Spend a reasonable amount of time discussing each RFC, attempting to come to consensus about acceptance of the work, or at least about next steps.

The goal of all of the above is to have the architectural conversations that are just too hard to have over email and IRC, which are manifold. This will be the place where we can chart a multi-year course for MediaWiki development. This is somewhat similar to what (I think) the Linux kernel developers do at the Linux Kernel Developers Summit (I haven’t attended, so I’m mainly relying on third-party accounts).

One thing that’s been difficult for us to figure out is what the right way of picking the attendees for this. This is far from a settled question, and I think it’d be healthy for us to discuss how this should work next week.

Conclusion

Not much more to say here, other than linking to the agenda, still subject to change: https://www.mediawiki.org/wiki/Architecture_guidelines/Meetings/WMF_Engineering_All-Hands_2013

I’m looking forward to talking more about this next week!

Rob