Help:Wikifunctions/Implementation ordering and choosing

Note: When you edit this page, you agree to release your contribution under the CC0. See Public Domain Help Pages for more info.

When a function call is requested in the user interface, using the Evaluate a function call page (Special:EvaluateFunctionCall) or Try this function on a function or test page, Wikifunctions needs to choose one of that function's implementations to run the call. This choice is made by taking the first implementation listed in the function's property Z8K4/implementations. Z8K4/implementations contains all of the function's currently connected implementations, which have been ordered according to the strategy explained below. (If Wikifunctions determines that the first implementation is invalid, it will move on to the second implementation, but this situation arises very rarely.)

Implementations are also selected in this way when a function is called as part of the execution of a larger composition, and when a function is called via the public API. (Implementation choosing is different for test runs, however. Wikifunctions ensures that each implementation gets used with each test, so in this context the ordering of Z8K4/implementations is irrelevant.)

If you want to call a function and ensure that a particular implementation is used, you can request the function call from the page of that implementation, using the Try this implementation box found there.

The order in which implementations appear on a Function page is not based on the ordering of Z8K4/implementations; there is no visual indication on Function pages of that ordering.

Implementation ordering strategy

Ordering of implementations is based on their number of test failures and average runtimes. When implementations are ordered for a given function, Wikifunctions derives that information from the Function call metadata of the most recent test runs involving the currently connected implementations and tests. (Each test run involves one implementation running one test; thus, if there are N connected implementations and M connected tests, the ordering algorithm will employ metadata from N x M tests.)

Implementations having fewer test failures appear earlier in the ordering than those with more test failures. Within the group of implementations having the same number of test failures, those with smaller average runtimes appear earlier.

Calculation of average runtime

For each test run, its runtime is simply the elapsed time between when the orchestrator receives the run-function request, and when it returns the result. (This time is reported as Orchestration Duration in the Function call metadata.) For each implementation, then, its average runtime is just the average of its most-recent runtimes for all the currently-connected tests.

Timing of implementation re-orderings

Reordering of implementations happens under the following conditions:

test results have just been retrieved for a function (i.e., a user has just loaded, created, or edited a page that displays test results)
all connected implementations and testers are included in the results, and
at least one result is live (i.e., the function was actually run to produce the result; it was not obtained from the test results cache). This happens when the function has been edited, or an implementation or test has been created or edited.

For some functions, the average runtimes of the two fastest implementations may be close. In such a case, there's a risk of reordering happening too frequently --- that is, the first-ranked position could alternate frequently between the two implementations, resulting in lots of pointless page updates. (Any new value for any property of a function causes a new revision of that function's wiki page to be written.) In addition, updating a function page for implementation-ordering changes that do not involve the first position is not useful.

In order to prevent re-ordering from happening too frequently, Wikifunctions follows the following guidelines when it recalculates the implementation ordering:

If the first-ranked implementation has not changed, the function's Z8K4 property is not updated (even if other positions in the list have changed).
If reordering calculations indicate a different implementation than before is now performing best, the Z8K4 property is only updated if the average runtime for the new best performer is significantly better than the average runtime for the previous best performer (and the new best performer has no more test failures than the previous best performer). "Significantly better" is defined as an average runtime that is less than 80% of of the average runtime of the previous best performer.

Because of these factors --- particularly because of (2) above --- it may sometimes be the case that the current ordering does not precisely reflect the metadata displayed for the most recent test runs.