Data Platform Engineering/Data Platform SRE/Priorities
Here are the high level priorities of the DPE SRE team. The detailed backlog can be found on our main Phabricator board. Our current work can be followed on our "milestone" Phabricator board (there is no stable link to the current milestone, but it can be found as a link in the menu of our main board).
Current main projects
editTo simplify operations and increase availability, we are migrating Airflow to k8s.
Links
edit- Main phab task: T362788
- Design doc
To support the deprecation and removal of Graphite
Links
edit- Main phab task: T359033
To support work by the Search Platform team. In particular, DPE SRE is focused on migration of the internal WDQS clients and the operational support of the underlying servers / platform.
Links
editArchiva is our current solution for artifact hosting for Java / Scala projects and mirroring of external Maven repositories. It is unsupported and as a critical piece of our development and deployment infrastructure needs to be replaced. Gitlab is a component that provides the functionality that we need and is already deployed in our infrastructure, it is the obvious solution.
This project is driven by DPE SRE, but most of the implementation work is done by Search Platform, Data Engineering and Data Products. It is prioritized on top of the usual work for those teams and thus is slow moving.
Links
edit- Main phab task: T367315
- Decision brief
Usual operational work
edit- Incidents
- Various minor software upgrades
- Access requests
- SPARQL Federation requests
High level backlog of projects
edit- Migration of the Search cluster from Elasticsearch to OpenSearch: T370147
- Kafka upgrade: design doc
- Mutualized OpenSearch cluster: T362105 & design doc
- Hadoop upgrade: T379385
- Kubernetes upgrade
- Spark upgrade
- Migration of additional services to k8s
- Presto
- JupyterHub