Continuous integration/Jenkins job builder

Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally stores configuration of jobs in an XML format. JJB instead maintains jobs as simple descriptions in YAML format, which are then expanded to XML and uploaded to Jenkins through its HTTP API.

Upstream documentation is available at: https://docs.openstack.org/infra/jenkins-job-builder/.

Note: which jobs are triggered for which events is handled by Zuul via zuul/layout.yaml.

How it works

All jobs for Wikimedia Jenkins are managed by JJB. To create or change jobs, edit the YAML files in the https://gerrit.wikimedia.org/g/integration/config repository (submit a change, have it reviewed/merged by someone else).

Deployment must be done from the reviewer's local workstation. (Unlike mediawiki deployment, JJB does not run on the server.)

Install JJB

The recommended way is to use JJB, is via a Python virtualenv managed by the integration/config.git repository itself. This way you will always use the correct JJB version. You will need to have Python 3 (e.g. from Apt or Homebrew) and tox installed (e.g. pip3.11 install tox, avoid Python 3.12+ until upstream bug is fixed).

The ./jjb-* scripts in the repository will automatically use Tox and Python's virtualenv feature to install JJB on the first invocation.

$ git clone https://gerrit.wikimedia.org/r/integration/config.git
$ cd config
$ ./jenkins-jobs --version
Jenkins Job Builder version: 4.3.0

Configure JJB

JJB requires a basic authentication file with Jenkins credentials. Authentication is only needed if you intent to deploy changes to Jenkins. For local testing and validation, no authentication credentials are needed (just create a dummy file with empty user and password fields).

Our main installation at https://integration.wikimedia.org/ci/ only allows users in the ciadmin LDAP group to upload jobs.

To obtain an API token, log into Jenkins and open the "Configure" tab of your account (JENKINS_URL/user/USERNAME/configure; e.g. https://integration.wikimedia.org/ci/user/Krinkle/configure). Click the "Add API Token", give it a name (eg. "jjb"), and then copy the your new API token.

Create a jenkins_jobs.ini file within your integration/config.git checkout and ensure to make it readable only by your user chmod 0500 jenkins_jobs.ini:

[job_builder]
allow_duplicates=True

[jenkins]
user=USERNAME
password=API_TOKEN
url=https://integration.wikimedia.org/ci/
query_plugins_info=False

Troubleshooting

A few common problems that may occur when installing JJB.

error: externally-managed-environment

$ pip install tox
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to install.

This can happen if you use Homebrew to install Python 3.12 or later. Use Python 3.11 instead, and then use that to install tox:

$ brew install python@3.11
…
$ pip3.11 install tox

ERROR:stevedore.extension: module 'pkgutil' has no attribute 'ImpImporter'

ERROR:stevedore.extension:Could not load 'delete': module 'pkgutil' has no attribute 'ImpImporter'
ERROR:stevedore.extension:Could not load 'list': module 'pkgutil' has no attribute 'ImpImporter'
ERROR:stevedore.extension:Could not load 'test': module 'pkgutil' has no attribute 'ImpImporter'
ERROR:stevedore.extension:Could not load 'update': module 'pkgutil' has no attribute 'ImpImporter'

This is due to the popular "setuptools" Python package (which JJB depends on) not yet supporting Python 3.12 or later. Use Python 3.11 instead.

This bug is tracked upstream at https://bugs.launchpad.net/python-jenkins/+bug/2038855

pyconfig.h

 ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory
  #include "pyconfig.h"
                      ^
 compilation terminated.

You may be missing Python's development dependencies. In Debian / Ubuntu that is:

sudo apt-get install python-dev

Modifying jobs

See also Install JJB - You need to have it installed locally before you can test or validate config files.

• Modify jobs by editing the YAML files in the integration/config.git repository.
• Read the documentation to learn what features are available (JJB internally maps the YAML format to the XML format for the Jenkins API, properties might have slightly different names, or might work differently, or might not be supported).
• Use an editor with YAML syntax highlighting and ideally YAML linting as well. The YAML format relies on indentation for its structure (similar to Python); it's easy to make mistakes.
• Once done editing, run the following in your integration/config directory:

  $ ./jjb-test -o output/
  

  this internally translates to the following command:

  $ tox -e jenkins-jobs -- --conf jenkins_jobs.ini test jjb/ -o output/
  

  This verifies everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
• Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.

Output directory should now contains files like this:

$ ls -l output/
mediawiki-phpunit
operations-puppet-validate
...

Deploy changes

See also Install JJB - For deploying changes, JJB must be installed, configured, and authenticated

Check out the relevant change for integration/config from Gerrit. Ensure the change is rebased onto latest master to avoid accidentally undoing other changes.

integration/config$ git review -d 139035

Now use the JJB "test" command to validate the YAML files and generate the XML file that represent the Jenkins job.

integration-config$ ./jjb-test 'composer-package-php80-docker'

You can also generate multiple jobs (or all jobs) by passing no name or a wildcard name instead:

integration-config$ rm -rf output/ && ./jjb-test 'composer-*' -o output/
integration-config$ cat output/example-jenkins-job
<?xml ..
...

If the jjb-test command results in error, there is either a YAML syntax error or invalid an JJB structure inside the YAML file. The config patch will need amending in that case. Do not deploy!

If there are no errors, review the XML file for your job:

When you upload your Gerrit change for the integration/config repository, we automatically generate the above output for all jobs before and after, and show you a diff you can review. Browse for the output of the integration-config-jjb-diff Job for the full diff!

If the job looks correct, upload the updated Jenkins job configurations to Jenkins.

Do not merge changes in integration/config until after you have locally generated the affected jobs and deployed ("jjb-update") them to the Jenkins. It is customary for the repository to reflect the deployed state. Given that deployment happens locally, it is customary to +2 afterwards, instead of the other way around.

The ./jjb-update command accepts multiple (variadic) arguments, each of which may use the fnmatch patterns.

# Deploy one job
$ ./jjb-update 'example-jenkins-job'

 .. INFO:root:Updating jobs in jjb/ (['example-jenkins-job'])
 .. INFO:jenkins_jobs.builder:Reconfiguring example-jenkins-job

# Deploy multiple jobs at once with a pattern
$ ./jjb-update '*-example'
 ..

# Deploy multiple jobs at once with multiple patterns
$ ./jjb-update '*foo*' '*bar*'
 ..
# The jjb-update script translates to the following command:
$ tox -e jenkins-jobs -- --conf jenkins_jobs.ini update jjb/ 'example-*'

Debugging JJB

• You can enable debugging output by passing -l debug
• Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file ~/.jenkins_jobs_cache.yml to force the update.

Deleting a job

If you are deleting a job, the regular jjb update command will not work (since it only updates jobs it's aware of). To delete jobs, you can use the jjb-delete script in the integration/config repository.

Synchronizing all jobs

This might overwrite things you don't intend to. You usually do not want to do this.

Known issues:

• Updating all jobs does not allow one to control the order in which jobs are uploaded. Given our Jenkins instance is actively used, this will lead to problems where job A depends on job B, but job B is still absent or outdated.

Deploying changes to zuul config

Changes to zuul/layout.yaml are deployed using fabric, by running the following command:

$ ./fab deploy_zuul

See Continuous integration/Zuul#Update configuration for detailed instructions.

Deploying changes to dockerfiles

Changes to dockerfiles/* are deployed using fabric, by running the following command:

$ ./fab deploy_docker