Wikimedia Release Engineering Team/Book club/Bugs and priority

EngProd Bookclub 2020-04-15

Currently Reading: Bugs and Priority

Random Notes

• thcipriani's random notes
• Brennen's notes, in entirety: I buy the premise that priority is fuzzy, I do not buy the proposed solution.
  • JDF: +1

Discussion

• ET: (1) Highly relevant to what I'm doing (2) Nothing particularly new there classic
• LIW: Classic/common -- misses the point and is next to useless -- whenever I try to have priorities like that I end up with a long bug backlog
• LIW: The things I look at:

   - Is it clear what needs to be done, then figure out what needs to be done to make it clear
   - Is this blocked
   - Is it clear when it's fixed
   - Is there a consensus with stakeholders that it needs to be fixed
   - Can I fix this? Or can James?

• ET: We have different views on bugs
• LIW: What does a priority mean? Why do we triage bugs at all? To decide what we need to work on next. Priority is not the way to do that.
• ET: Author is talking about priority variables, which is a flaw: What type of defect and what is the frequency of defect. Those questions have a very different weight.
• JDF: Fundamentally the problem with this document is that humans are squishy and now well categorised, and proports to fix that by adding more process. Adding all these categories is hugely tendentious.
• Weekly bug triage meeting where we published: https://www.mediawiki.org/wiki/Editing_team/Current_priorities#Decision_tree
• It didn't work very well because no one agreed on objective reality. The more process you add the more you exclude rather than include, and the more time you spend working on process.
• BB: Priority is subjective adding more process and pretending that you are quantifying it, doesn't really help
• JDF: I am reminded of the law when we try to do this in tech. Actual vs Grevious bodily harm – How do we know which applies? 200 years of case law and binding precedent.
• BB: I like the idea of case-law and binding precedent for handling things like this
• LIW: Mental models are useful. I am not fond of the article's chart. There are too many categories. I'd like to reduce this to a very small number. Is this something that we have to work on *right now*?
• BB: In phabricator there is a 1 bit of information that I care about in terms of priority. That carries real information in terms of case-law. UBN means that someone is really concerned.
• JDF: If you write down the heirarchy that will communicate it sufficiently. It is not a bad thing to try to rank and priorties. Declaring site emergency there is an agreement.
• ET: what is the use of documented processes? There are benefits to structural proceedures.
• JDF: Bugzilla migration -- severity and priority. I pushed for a 3rd prioritization level; i.e., how much we care vs how bad the bug is. Bugzilla gave us a large impact for a small number of people. The problem is there is this "one weird trick". This is a reliability/operations request.
• JDF: severity, frequency, and impact. Bugzilla has 2, Phab has 1. Reality is messy.
• Z: in phab, having boards and columns in those boards solves some of these problems. Waiting, or Doing, or Someday. You can also do this on a personal level.
• ET : "Dynamic triaging" - if something's happening, I'd communicate it immediately - people might take it as an immediate action, or they might demote priority. This happens constantly but isn't necessarily modeled.
• BB: I think that gets at the fundamental problem with priority which is -- who's priority? issue trackers are a tug of war between reporter and the person who is going to fix the bug. This is not something you can necessarily model.
• JDF: Bugs where we're blocked by SRE: we may have different priorities there
• LIW: Launchpad models some of that. Every bug can affect a large number of projects where each project can set its own priority.
• LIW: One of the fundamental rules I have for bug triage is that the backlog should be manageable in size. Debian has several 100,000s of open bugs. Having them open is a hendrance to people working on them. How do I find good bugs? How do I know if bugs is already reported? Scale makes everything worse.
• Z: I worked at a task tracking software place. How do you solve a big backlog? Don't have a big backlog. A good search is helpful, but if there are a lot of tasks, you will be unlikely to review them. The only solution is projects and boards. Closing all bugs has been suggested many times and is rejected each time.
• JDF: I think that a successful project is successful because it is unbounded in scope. ITIL -- management framework used in EMS for managing problems in prod. Issues vs Bugs vs Changes -- outside of the scope of features. Allows different teams to prioritize different types of work.
• ET: there must be someone to decide on priority
• Z: on closing all bug: on a year-old bug -- no activity in a year -- add a comment where you say, "we're thinking about closing this" -- that gives folks a chance. I'm sure anecdotally there would be 10-20% where no one will say anything.
• BB: As a user of large software systems I am perpetually encountering broken things and eventually I land on a bug that is closed. There are 50 people who say "this still doesn't work".
• JDF: Z is seeing a bug tracker as something used by devs to track software; BB is seeing it as a public bug tracker. In open software you see everything. There's a github project where a bot auto-rejects any patch submitted in the last 30 days that hasn't been touched.
• Z: I agree with both of you on some levels. This is a problem with open bugs as well as closed bugs. In case of a closed bug: at least there is some signal.
• BB: I think, as with priority, open/closed is modeling too many things.
• LIW: It is possible to close bugs without saying we don't care. We can't repro, we are busy we can't investigate, others aren't having this problem.
• ET: sometimes reproducing is half-a-day of intense work trying to reproduce. It can be time consuming
• JDF: This morning, I marked 5-6 bugs invalid, because we removed the future, but this is still somewhere in use in the universe.
• LIW: Debian has a way of tracking which version of software contains the problem. It's hard to understand but it works most of the time. But the system is complicated. Clearly the best solution is to not have problems.
• JDF: I think it's useful to think about modeling priorities. The more time you spend modeling, the less time you spend communicating.
• LIW: I concur.
• Z: Who suggested this
• T: me

16:37:47 brennen@inertia:/tmp ✵ dict tendentious
1 definition found

From WordNet (r) 3.0 (2006) [wn]:

  tendentious
      adj 1: having or marked by a strong tendency especially a
             controversial one; "a tendentious account of recent
             elections"; "distinguishing between verifiable fact and
             tendentious assertion" [syn: {tendentious},
             {tendencious}]