Reading/Web/What to do with an UBN

If a task is an Unbreak Now! (UBN), you should drop everything to fix it.

A task is usually UBN if code has reached production that breaks things or will soon do so.

Locate problem

The first thing to do is obviously locate the patch that caused the problem. `git blame` can be used for more complicated matters. Breakages may have occurred due to changes in core or a recent update to a library. Sometime it's not always easy to locate the problem!

Fix problem

When you have located the problem, fix it. A general rule is that if the fix is going to take substantial effort (e.g. longer than 20 minutes) you should consider reverting the problematic patch.

Deploy fix

You can make use of one of the many SWAT windows by adding something to the Deployment calendar. However if the next window is very far away (or it's Friday where there are no deploys) you'll need to be a little more proactive. Begin by asking in #wikimedia-operations ^connect and #wikimedia-releng ^connect IRC channels. Your best bet is to locate a deployer in the RelEng team - @Reedy @thcipriani or @RainbowSprinkles, @greg-g are usually good people to contact directly. If all else fails, you'll want to e-mail releng@wikimedia.org.