Facebook Open Academy/Cron

Tracked in Phabricator
Task T59613

A common requirement in infrastructure maintenance is the ability to execute tasks at scheduled times and intervals. On Unix systems (and, by extension, Linux) this is traditionally handled by a cron daemon. Traditional crons, however, run on a single server and are therefore unscalable and create single points of failure. While there are a few open source alternatives to cron that provide for distributed scheduling, they either depend on a specific "cloud" management system or on other complex external dependencies; or are not generally compatible with cron.

Requirements

The Wikimedia Labs has a need for a scheduler that:

Is configurable by traditional crontabs;
Can run on more than one server, distributing execution between them; and
Guarantees that scheduled events execute as long as at least one server is operational.

The ideal distributed cron replacement would have as few external dependencies as possible.

Architectural notes

Research

Some interesting avenues of investigations have already been mentioned in the related Bugzilla (which see), as well as possible alternatives and counterarguments.

What are the current solutions that exist and what lessons can be learned from them?

Chronos
- Supports dependencies between jobs
- Will retry failed jobs
- One of multiple nodes is elected a master
- Has many dependencies including Apache Mesos and Zookeeper
Cronie
- If I understand the man page correctly:
- It only allows jobs to be executed on one chosen server at a time
- Must manually switch the chosen server if it goes down
- Requires a network-mounted share for the directory containing the shared crontabs
  - (FYI, that requirement is met for Lab's particular use case, but would normally indeed be considered onerous) — Coren ^{(talk)/(enwp)} 06:22, 4 February 2014 (UTC)[reply]

Jenkins
- Meant for continuous integration not job scheduling
- Only allows one master (single point of failure)
Gearman
- Framework for distributing tasks
- Has fault tolerance and job retries
- Would still require a scheduler and worker application for the APIs to be written
- Seems to be better suited to executing jobs at arbitrary times rather than being scheduled

Decisions taken

Language to use

Set to Python by fiat for expediency

Widely distributed, well known by a large development base
High availability of libraries
Many (most) Linux distributions default to it for system scripts
Striving to work properly for Python 2.7 and Python 3+

Coding style

We have chosen to use the PEP 8 code style. Be sure to follow this style guide as closely as possible and the use of a style checker (ex. pep8) is encouraged (note that it doesn't catch all issues).

http://legacy.python.org/dev/peps/pep-0008/

What is the difference between a job and a schedule?

A job represents an entry in a crontab (ex: */5 * * * * echo "megacron"). A schedule is a specific instance of that job. In the previous example, a schedule would be created every five minutes for the job.

Should servers that run jobs be decoupled from servers that handle scheduling?

No, every server can potentially become a scheduler. This makes it easier for the users of our project because it abstracts the distinction between a worker and scheduler away from them.

How are the schedules and jobs stored and distributed between servers?

We are depending on a database being available and accessible to every server. The user will be able to choose from a few types of supported databases.

How is the worker that runs the schedule chosen and how is that information synchronized?

The server that has been running the longest will become the scheduler. A server will know that it has been running the longest if it has the oldest Id in the database. This approach is by far the simplest to implement and it seems to be working fine (so far).

How late can a job run?

If a run of a job is missed, it will be run at the next possible opportunity. If multiple runs are missed, only one will be run.

How bad is it if a job runs multiple times?

This should never happen.

What libraries should we usewe use to solve subproblems?

croniter for parsing crontab intervals
setuptools for packaging the project

How do we tell if a worker goes down?

A worker will periodically "check in" by updating it's heartbeat timestamp in the database. If the timestamp becomes older than a predetermined value, the worker is then considered to be down.

How does the scheduler distribute schedules?

Whenever it creates a schedule, it will specify a time to run and a worker to run it. Workers are assigned in a round-robin fashion. Workers then fetch a list of jobs assigned to them, and run them at the appropriate time.

What user executes the schedules?

The UID of the user that created the job is stored. The schedules are then executed as that user.

Decisions that need to be taken and associated discussion

How bad is it if a deleted job still gets ran?

If a server gets isolated and the crontab is updated, it'll run all the jobs on the old table in perpetuity for as long as it cannot connect to any of the other servers. GLM
If a job is deleted after its start time, it should run (unless deleted refers to "aborted"). Such cases would need to be handled manually by an administrator. JT
In a common case of deleting a job from the crontab during the job's idle period, it should not run on non-isolated servers (this definition depends on the technique we use for server selection). Otherwise, there is essentially no delete functionality provided. FC

Implementation details

API

Functions
- get_jobs() [Obtains a list of jobs that correspond 1:1 to each entry in every user's crontab]
  - returns [Jobs]
- get_jobs_for_user(user_id) [Obtains a list of jobs that correspond 1:1 to each entry in a single specified user's crontab]
  - returns [Jobs]
- set_jobs([Job], user_id) [Removes existing jobs for the specified user and puts the list of Jobs into the datastore]
- set_job_time(Job) [Sets the last time run for the job to the current time]
- get_schedules(Worker) [Obtains a list of schedules that are currently assigned to the worker]
  - returns [Schedule]
- add_schedules([Schedule]) [Puts a list of schedules into the datastore]
- add_schedule(Schedule) [Puts a schedule into the datastore]
- remove_schedule(Schedule) [Removes a schedule from the datastore]
- get_heartbeat(Worker) [Obtains the value of the heartbeat timestamp associated with the worker]
  - returns timestamp
- update_heartbeat(Worker) [Updates the worker's timestamp to the current time]
- get_next_worker() [Returns the next available worker and moves it to the end of the queue]
  - returns Worker
- get_workers() [Obtains a list of all workers]
  - returns [Worker]
- create_worker() [Adds a new worker to the datastore and returns it]
  - returns Worker
- destroy_worker(Worker) [Removes a worker from the datastore]
- get_crontab(user_id) [Gets the crontab for the given user]
  - returns crontab
- set_crontab(crontab, user_id) [Adds a crontab as a string to the datastore for the given user]
Objects
- Job [A single parsed crontab row representing a single command to be run]
  - String : interval [Ex: "* * * * *"]
  - String : command [Ex: "foo.py"]
  - String : user_id
  - Datetime : last_time_run
- Schedule [A single element extracted from a Job, in a format that a worker can run]
  - Worker : worker
  - Datetime : time_to_run
  - Job : job
- Worker [A machine which Scheduler is a specific instance of, capable of performing crontab tasks]
  - Datetime : heartbeat [timestamp of last contact]

Where can the code be tested in an isolated environment?

With appropriate credentials, you can ssh into all of the three following servers:

megacron-one.wmflabs.org

megacron-two.wmflabs.org

megacron-three.wmflabs.org

You should have root access here using sudo. A NFS shared filesystem is mounted on each server at "/data/project/".

Packaging Instructions

To build a new Python Source Package:

python setup.py sdist python setup.py register python setup.py sdist upload

To update Python Source Package:

Down the road, after you’ve made updates to your distribution and wish to make a new release:

1. Increment the version number in your setup.py file,

2. Update your CHANGES.txt file,

3. If necessary, update the “Contributors” and “Thanks also to” sections of your README.txt file.

4. Run python setup.py sdist upload again.

To download via pip:

sudo pip install megacron sudo pip install megacron --upgrade

To build a Debian package from an existing Python Package:

py2dsc Megacron-0.1.tar.gz

sudo debuild -S -rfakeroot -kDE117DE3

sudo debsign megacron_0.1-1_amd64.changes -kDE117DE3

sudo dput ppa:johntanner/megacron megacron_0.1-1_amd64.changes

Add to your /etc/apt/sources.list:

deb http://ppa.launchpad.net/johntanner/megacron/ubuntu precise main

deb-src http://ppa.launchpad.net/johntanner/megacron/ubuntu precise main

To download via apt-get:

sudo apt-get update

sudo apt-get install python-megacron