Phabricator/Migration/status

2014-05-19

The Phabricator RfC was finished with support for moving to Phabricator. Hence also the review of Wikimedia's Project management tools is finished. At the Zürich Hackathon 2014 three Phabricator related sessions took place (general introduction to Phabricator by Shahyar, Phabricator's code review concept and tool by Shahyar, discussion planning Day 1 of Phabricator in production). Another public IRC office hour about Phabricator took place on May 14th (log). Andre summarized the process of moving to Phabricator so far in a blogpost. Regarding overviews and keeping track, the project has a central general information page, a planning board for Wikimedia Phabricator Day 1 in Production, a team defined for the required work, and an upstream task board for planning.

2014-05-monthly

Mukunda Modell is currently addressing authentication and access restrictions for security tickets with upstream. Chase Pettet and Daniel Zahn of Wikimedia Operations are spearheading the Phabricator installation. Andre Klapper has upstreamed some issues, commented on numerous tickets, and identified further tasks related to migration. An overview board of tasks to solve for the first day of Phabricator in production is available. Furthermore, once Wikimedia SUL authentication is sorted out, it is investigated to launch the Phabricator production instance first with very limited functionality to provide a Trusted User Tool.

2014-06-10

Apart from commenting on / discussing / upstreaming Phabricator tickets, Andre gave a short Phabricator presentation at WMF's Monthly Metrics meeting and started thinking about on organizing sprints, releases, iterations in Phab and data migration from Bugzilla.

2014-06-monthly

Apart from discussions on how to implement certain functionality and settings in Phabricator among team members and stakeholders, Mukunda implemented a MediaWiki OAuth provider in Phabricator (Gerrit changes: 1, 2; related ticket) and Chase created a Puppet module for Phabricator.

2014-07-09

Mukunda Modell implemented WMF SUL Authentication for Phabricator. Chase Pettet deployed Legalpad, a tool to manage trusted users, on a separate server (workflow to be further defined with the Legal team). Sean Pringle and Chase put a data backup system for Phabricator in place. Mukunda wrote code to restrict access to tasks in a certain project, which can already be tested on fab.wmflabs.org. Chase upgraded the dedicated Phabricator server to Ubuntu Trusty. Andre Klapper listed the Bugzilla bug report elements to tackle in a script to import from Bugzilla and dropped his thoughts about Priority and Keyword migration. In upstream development, umbrella projects and subprojects are being worked on, which will influence our plans on handling release planning in Phabricator.

2014-07-22

Mukunda finished packaging using pkg-php-tools/dh_php5, worked on refactoring project access restrictions into a custom herald action (ticket and patch), showing the wiki account URL on Legalpad's signature list view, and file access restrictions (ticket and patch). Chase has been working on configuring exim for mail and discussing accessibility of file attachments with upstream. Andre commented on tickets about handling keywords, severity etc. and sent a summary email to wikitech-l communicating which 'regressions' we might see.

2014-07-monthly

Phabricator's "Legalpad" application (a tool to manage trusted users) was set up on a separate server. This instance provides WMF Single-User Login authentication. Mukunda implemented restricting access to tasks in a certain project which can be tested on fab.wmflabs.org. As a followup, he investigated enforcing security policy also on files and attachments and replacing the IRC bots by Phab's chatbot. Chase worked on initial migration code to import data from Bugzilla reports into Phabricator tasks (and ran into missing API code in Phabricator), investigated configuring Exim for mail, set up a data backup system for Phabricator, and upgraded the dedicated Phabricator server to Ubuntu Trusty. Quim started documenting Phabricator. Andre helped making decisions on defining field values and how to handle certain Bugzilla fields in the import script and sent a summary email to wikitech-l about the Phabricator migration status.

2014-08-11

Upstream Phabricator developers implemented granular file permissions and upload defaults, with making file data to be inaccessible (not undiscoverable) still to resolve (see related task, Mukunda investigates). Chase throughly tested the current state of file access security (prior to having a canCDN flag implementation which will require more testing), plus worked on supporting Bugzilla keyword conversion into separate Phabricator tags plus general improvements in the import script. In upstream, Mukunda added API to create projects. Chase added support for mailing lists as watching users in upstream and case sensitivity in project URLs is now handled by normalizing to lower case (see related task).

2014-08-20

Chase worked on and tested the data migration logic (attachments and security access) and ran into a Bugzilla WebService API issue. Mukunda worked on getting MediaWiki OAuth merged into upstream and merged a CustomField extension that adds a "MediaWiki Userpage". Chase and Mukunda also worked on the Project Policy Enforcer action for Herald, providing a user-friendly dropdown menu to restrict ticket access when creating the ticket. We also identified that we want to purchase an alternative domain first for content hosted on the Phabricator production instance.

2014-08-28

For a better overview, the Wikimedia Phabricator Day 1 project was split into three projects: Day 1 of a Phabricator Production instance in use, Bugzilla migration, and RT migration. Furthermore, the overall schedule was clarified. in A separate domain for user content was purchased (which still needs a certificate). Mukunda worked on getting the MediaWiki OAuth provider merged into upstream. More testing of the security implementation took place (with regard to making restricted data not only undiscoverable but also inaccessible). Chase also worked on the scripts to export and import data between the systems and support for external users in Phabricator and the related mail setup.

2014-08-monthly

The project is getting close to Day 1 of a Wikimedia Phabricator production instance. For better overview and tracking, the Wikimedia Phabricator Day 1 project was split into three projects: Day 1 of a Phabricator Production instance in use, Bugzilla migration, and RT migration. Furthermore, the overall schedule was clarified. In the last month, Security/permission related requirements got implemented (granular file permissions and upload defaults, enforcing that policy, making file data inaccessible and not only undiscoverable). In upstream, Mukunda added API to create projects and Chase added support for mailing lists as watching users. Chase worked on and tested the security and data migration logic. Mukunda continued to work on getting the MediaWiki OAuth provider merged into upstream. Chase and Mukunda also worked on the Project Policy Enforcer action for Herald, providing a user-friendly dropdown menu to restrict ticket access when creating the ticket. A separate domain for user content was purchased. Chase also worked on the scripts to export and import data between the systems and support for external users in Phabricator and the related mail setup. Chase and Chad also took a look at setting up Elasticsearch for Phabricator.

2014-09-09

Restricting access to Phabricator tasks based on project membership was implemented by its last patch merged. Phabricator now lets you interact with external (non-Phabricator) users via email which was a requirement for migrating RT tickets and shows the associated MediaWiki.org account on the Phabricator user page. An issue with uploading files was fixed. A first version of a 'Phabricator' on-wiki template was made available and Helder worked on making the 'Tracked' template support linking to Phabricator. Migrating content from fab.wmflabs.org to the Phabricator production instance started this week which required taking down the Phabricator testing instance on Wikimedia Labs. The team can be reached via the #wikimedia-devtools IRC channel on Freenode. Furthermore, work continues this week on upstreaming the MediaWiki OAuth provider (upstream ticket), getting a certificate for phab.wmfusercontent.org, configuring inbound email for phabricator.wikimedia.org, and determining the license of content submitted to Phabricator.

2014-09-17

The Phabricator production instance with tickets imported from now-defunct fab.wmflabs.org has been set up but remains read-only until Operations has finished setting up SNI on misc-web-lb and making it work with nginx (related patches: 1, 2, 3) and the SSL certificate (related patches: 1, 2, 3, 4, 5), Legal has blessed the required footer, and upstream has merged the Mediawiki OAuth provider. Daniel and Yuvi set up a new Phabricator test instance on https://phab-01.wmflabs.org/ (after allowing SSL) that anybody can play with. Daniel also added patches (1, 2) for redirecting http to https. In addition, work is going on to improve the Phabricator documentation and help.

2014-09-30

The certificate for phab.wmfusercontent.org for file uploads is in place. Daniel and Chase fixed an error when trying to log in via HTTP, the footer required by Legal was set up with its license, terms of use and correct links. Chase configured inbound email, fixed an issue with not being able to set the profile picture, customized some strings, made the reference in a ticket to its previous ticket system read-only. Daniel added IPv6 access to phabricator.wikimedia.org, Reedy added phabricator to the interwiki map, and Helder ported the 'tracked' gadget on MediaWiki to query the Phabricator API. In addition, Quim improved the Phabricator documentation and help. Andre showed the very basics of Phabricator in a video. Once the remaining dependencies of T463 are resolved and the custom OAuth provider is deployed, registration will be open for everybody.

2014-09-monthly

phabricator.wikimedia.org got set up with tickets imported from the previous Labs instance (public registration will be enabled once all remaining tasks have been sorted out). Restricting access to Phabricator tasks based on project membership was implemented. Inbound email was configured so Phabricator lets you interact with external (non-Phabricator) users via email. A certificate for phab.wmfusercontent.org for file uploads was set up and Operations set up SNI on misc-web-lb, made it work with nginx, added IPv6 access, and fixed an error when trying to log in via HTTP. The legal footer required was set up with its license, terms of use and correct links. Reedy added phabricator to the interwiki map, and Helder started to the 'tracked' gadget on MediaWiki to query the Phabricator API. Many further smaller fixes took place. Furthermore, Quim improved the Phabricator documentation and help. Andre showed the very basics of Phabricator in a video. Daniel and Yuvi set up a new Phabricator test instance on https://phab-01.wmflabs.org/ that anybody is welcome to test.

2014-10-15

After LDAP and SUL were successfully set up, the registration was enabled for everybody on phabricator.wikimedia.org. Bugzilla users are encouraged to create an account in Phabricator. Phabricator now allows interacting with external (non-Phabricator) users via email (like provided by RT). Regarding notifications, Valhallasw made phawikibugs run on Tools Labs and investigates related announcement rules and IRC bot to report Phabricator activity, and Phabricator now sends mail to the wikibugs-l mailing list for all task updates, similar to Bugzilla. Dealing with templates on wikis, Helder updated the 'tracked' template to accept Phabricator parameters and Quim ported the 'Extension' template to link to Phabricator in its Bugzilla section. Furthermore, Phabricator user profiles show which MediaWiki.org account a Phabricator account is associated with and have an IRC alias field, Phabricator's applications enabled by default were defined, the footer received a small fix, HSTS max-age was raised to one year, Phabricator's timezone now defaults to UTC. Christian is working on creating a Gerrit plugin to notify Phabricator tasks about related patches and Christopher is working on burndown charts functionality. Quim started discussions to plan migrating code review from Gerrit to Phabricator. Based on the finished proof of concept for the Bugzilla migration, the next big step is to set up a preview instance showing how tickets imported from Bugzilla will look like in Phabricator.

2014-10-31

The Phabricator preview instance was set up on https://bugzillapreview.wmflabs.org/ to show how tasks migrated from Bugzilla look like. After announcing, the community provided valuable feedback in the bugzilla-preview project. Chase and Mukunda worked on fixing many reported and outstanding issues. Mukunda continued to work on the security extension, access to private tasks for their authors and redirecting old bugzilla.wikimedia.org URLs. Chase finalized the logic to import Bugzilla data into Phabricator and handling Bugzilla comment metadata. Entering a real name is now optional. Phabricator now uses the official MediaWiki logo. Andre removed the Triagers project, informed users of Tags in Bugzilla that they will be removed. Quim created project creation guidelines. QChris made Gerrit create notifications in Phabricator tasks about related patches in Gerrit. Christopher implemented a free alternative to Google Charts for Phabricator burndown charts. Valhallasw documented creating tasks by email. Chase renamed the 'ext_ref' field to 'Reference'. Upstream developers fixed URLs in HTML emails. Documentation received improvements and enhancements. Furthermore, many smaller changes and fixes by Daniel (1, 2, 3, 4, 5, 6, 7), Chase (1, 2, 3, 4, 5, 6, 7), Faidon (1, 2) and Chad (1) went into Puppet.

2014-10-monthly

After LDAP and SUL were successfully set up, the registration was enabled for everybody on phabricator.wikimedia.org. Bugzilla users are encouraged to create an account in Phabricator. Phabricator now allows interacting with external (non-Phabricator) users via email (like provided by RT). The Phabricator preview instance was set up on https://bugzillapreview.wmflabs.org/ to show how tasks migrated from Bugzilla look like. After announcing, the community provided valuable feedback in the bugzilla-preview project. Chase and Mukunda worked on fixing many reported and outstanding issues. Mukunda continued to work on the security extension, access to private tasks for their authors and redirecting old bugzilla.wikimedia.org URLs. Chase finalized the logic to import Bugzilla data into Phabricator and handling Bugzilla comment metadata. QChris made Gerrit create notifications in Phabricator tasks about related patches in Gerrit. Christopher implemented a free alternative to Google Charts for the future Phabricator burndown charts. Phabricator now sends mail to the wikibugs-l mailing list for all task updates, similar to Bugzilla. Regarding on-wiki templates, Helder updated the 'tracked' template to accept Phabricator parameters and Quim ported the 'Extension' template to link to Phabricator in its Bugzilla section. Furthermore, Phabricator user profiles show which MediaWiki.org account a Phabricator account is associated with and have an IRC alias field. Entering a real name is now optional. Andre informed users of Tags in Bugzilla that they will be removed. Quim created project creation guidelines and started discussions to plan migrating code review from Gerrit to Phabricator.

2014-11-17

Based on community feedback on the Phabricator preview instance with a set of Bugzilla tickets imported, further smaller issues were fixed by Chase (Contents of URL field should not be lowercased, newline after "See Also", General/Unknown become General-and-Unknown, should be -or-, Use dashes/hyphens instead of underscores for project names for better project proposal results, remove wikibugs-l@ as default assignee from imported Bugzilla tickets, Bugzilla asterisks interpreted as bullet markup). Phabricator was updated to a recent upstream version including some database scheme changes. More metadata of Bugzilla tickets, such as the creation date of imported tickets, is preserved. Mukunda worked on redirects for Bugzilla URLs. Yuvi and Valhallasw made the IRC bot report Phabricator activity like wikibugs, follow channel rules, and ignore Subscribers/CC field changes. Apart from more preparation and testing of the migration starting on November 21st, Quim continued to work on common project management guidelines, Christopher continued to work on Burndown charts (work in progress can be seen here), Andre sent a second and last migration announcement via email to the 850 most recent Bugzilla users, and work continued on improving the general documentation.

2014-11-22

The defunct previous test instance fab.wmflabs.org now redirects to phabricator.wikimedia.org. Diffusion (the Phabricator application to browse code repositories which will repace gitblit in the future) and the notification feature in the top bar of Phabricator were enabled. On November 21st, the migration from Bugzilla to Phabricator started.

2014-11-29

The migration from Bugzilla to Phabricator was successful - all tasks and accounts (which need to get claimed by their users) were imported into Phabricator. Details and a list of steps performed are available. Bugzilla accounts that were default CC or default assignees of components in Bugzilla were contacted to please join their corresponding projects in Phabricator. Next on the Phabricator migration list is the migration from RT to Phabricator, followed by Mingle and Trello. Furthermore, Diffusion (for hosting and browsing repositories, to replace gitblit) got enabled in Phabricator, fab.wmflabs.org redirects to phabricator.wikimedia.org, and the Gerrit notification bot creates notifications in a corresponding Phab task even when the commit message contains a link to a Bugzilla ticket.

2014-11-monthly

Wikimedia migrated from Bugzilla to Phabricator for issue tracking. All Bugzilla tasks and accounts (which need to get claimed by their users) were imported into Phabricator. URLs for Bugzilla reports are redirected to the corresponding tasks in Phabricator. (Details and a list of steps performed for the Bugzilla migration are available.) Bugzilla accounts that were default CC or default assignees of components in Bugzilla were asked to join their corresponding projects in Phabricator. Furthermore, Diffusion (for hosting and browsing repositories, to replace gitblit) got enabled in Phabricator, fab.wmflabs.org redirects to phabricator.wikimedia.org, the Gerrit notification bot creates notifications in a corresponding Phab task even when the commit message contains a link to a Bugzilla ticket, and the notification feature in the top bar of Phabricator got enabled. Work continues on common project management guidelines and providing burndown charts for sprints. Next on the Phabricator migration plan is RT, followed by Mingle and Trello.

2014-12-18

After the migration from Bugzilla to Phabricator, apart from other followup work, the Sprint extension was deployed. On December 18, RT was migrated to Phabricator (except for the access-requests@ and procurement@ queues).

2014-12-monthly

On December 18, Wikimedia's RT instance was migrated to Phabricator (except for the access-requests@ and procurement@ queues). RT users were added to Phabricator's WMF-NDA group. The Sprint extension was deployed, allowing teams to visualize progress. Best practices for Project management, working with workboards, working with the upstream Phabricator team, and our Phabricator Security setup were documented. The Language Engineering, Multimedia and Analytics teams started migrating to Phabricator. The Legalpad application was enabled and configured. A maintenance window for updating our instance was defined.

2015-01-16

Phabricator was updated which fixed more than a dozen of bugs and requests in Phabricator itself. The Sprint extension and the Security extension received larger changes; authors of private tasks migrated from Bugzilla can now always access these tasks. The Multimedia team and the Fundraising Tech team started to use Phabricator for sprint planning. Future maintenance of Phabricator was clarified.

2015-01-monthly

Phabricator was updated which fixed more than a dozen of bugs and requests in Phabricator itself. The Sprint extension and the Security extension received larger changes; authors of private tasks migrated from Bugzilla can now always access these tasks. The Multimedia team and the Fundraising Tech team started to use Phabricator for sprint planning. Future maintenance of Phabricator was clarified. S worked on the migration script from Trello. Furthermore, Quim Gil and Andre Klapper gave a presentation about the Phabricator migration at FOSDEM conference, Brussels.

2015-02-monthly

Phabricator was upgraded and received several fixes such as improved matching in the "Projects" dropdown proposals. Also, project links by default show the workboard (if set up) instead of the project summary, received a project bar on the left, and some URLs changed. The Sprint extension shows avatars on cards, is invoked by setting a checkbox in the project settings instead of using an "§" in the name, the Burndown chart does not depend on Maniphest task status anymore, and supports setting "Can Edit Task Status" and "Can Assign Tasks" policies (currently not enabled). The full list of changes is available and corresponding documentation was updated. Regarding migration of teams to Phabricator, the Collaboration team migrated from Trello (thanks to improvements on the Trello migration script) and Language Engineering team migrated from Mingle. Phabricator's search was reverted from ElasticSearch to the MySQL backend, fixing numerous issues. Furthermore, the separate legalpad.wikimedia.org was removed (superseded by phabricator.wikimedia.org), work continued on turning Bugzilla into a static version. The "Needs Volunteer" priority was renamed to "Lowest", Gerrit repository ownership permission handling moved fom the wiki to Phabricator, the documentation about the "Security" dropdown got improved, Gadget-BugStatusUpdate.js got adapted to Phabricator, email task creation instructions were clarified, the "Export to Excel" button renamed, and guidelines for tracking tasks versus projects were written.

2015-03-monthly

General Roadmap & Reporting was converted to using Phabricator. The Volunteer NDA process was implemented in in Phabricator. The login page was improved. Fundraising members can modify policies for their tasks. Tickets of the (historical) Growth team were imported from Trello.