User:Dnessett/Parser Tests/MacGyver Tests

Each section (except the first) contains one of the quick and dirty analysis programs developed to get better insight into how well parserTests exercises the parser. These programs are called MacGyver tests in reference to a mid-80s to early-90s television program named after the protagonist (Angus MacGyver). In each episode MacGyver would get out of some difficulty by improvising tools from objects lying around. By naming these MacGyver tools, I want to emphasize their practical rather than aesthetic value. They are not examples of beautifully designed and well thought out programs. They were thrown together in about 2 days. They met the needs I had when I wrote them, but I had no intention of creating something with a long lifetime. So, anyone using them should do so at their own risk and with the understanding that parserTests may have changed between the time they were written and the time of their use - in which case, they may need some maintenance.

Test Information

Each test has some assumptions built into them that affects their operation. Both RunParserTests and AnalyzeCodeCoverage assume they are installed the /phase3/maintenance/ directory of a MediaWiki software distribution. Both write files into this directory. RunParserTests outputs two files: 1) parserTestsOut.txt, which contains the information generated by parserTests and normally written to StandardOut, and 2) parserTestsCodeCoverage.txt, which contains the code coverage information generated by Xdebug. AnalyzeCodeCoverage uses the information in parserTestsCodeCoverage.txt to generate code coverage analysis information. It writes the file parserTestsCoverageAnalysis.txt, which contains the analysis information. More details on these tests are found in the sections below.

RunParserTests.php

RunParserTests comprises a single class (RunParserTests) with one method (run()). It assumes the Xdebug extension is installed. It calls xdebug_start_code_coverage() with the XDEBUG_CC_UNUSED option. This generates a two-dimensional array indexed by file path name as the primary key and line number as the secondary key (see Xdebug documentation). Use of the XDEBUG_CC_UNUSED option generates information about code lines visited and code lines not visited. In the former case the value assigned to the line number is 1 and in the latter case it is -1. The sum of line numbers with either of these values assigned represents the total number of executable lines in a file. The percentage code coverage is the number of lines visited in a file divided by this total (times 100). RunParserTests is run from the terminal and takes one optional parameter, --run-disabled. This parameter is passed to parserTests. It controls whether tests marked "disabled" are run.

It must be noted that Xdebug has an unusual definition for an executable line of code. Understandably, comments are not counted as executable, but somewhat strangely Xdebug does not judge lines that have variable initialization to be executable. The documentation on Xdebug code coverage is very sparse and figuring out which class of lines are counted and which are not requires looking at some analyzed files and using inductive logic.

The Program

<?php

// Define MEDIAWIKI so DefaultSettings doesn't croak
define( "MEDIAWIKI", true );

require_once( dirname(__FILE__) . '/../LocalSettings.php');
require_once( dirname(__FILE__) . '/parserTests.inc');

$handle = 0;

function file_callback($buffer)
    {
    global $handle;
    fwrite($handle, $buffer);
    }

class RunParserTests
    {

    /**
     * @assert == 0
     */

    public function run()
        {
        global $wgParserTestFiles, $handle, $options;

        // Start code coverage analysis
        xdebug_start_code_coverage(XDEBUG_CC_UNUSED);

        if (file_exists ( 'parserTestsOut.txt' ))
            {
            unlink('parserTestsOut.txt');
            }

        $handle = fopen('parserTestsOut.txt','w');
        ob_start('file_callback');

        $wgTitle = Title::newFromText( 'Parser test script' );
        $tester = new ParserTest();
        $files = $wgParserTestFiles;

        // Print out software version
        $version = SpecialVersion::getVersion();
        echo( "This is MediaWiki version {$version}.\n\n" );

        $ok = $tester->runTestsFromFiles( $files );

        ob_end_flush();
        fclose ( $handle );

        // Dump code coverage information
        if (file_exists ( 'parserTestsCodeCoverage.txt' ))
            {
            unlink('parserTestsCodeCoverage.txt');
            }

        $handle = fopen('parserTestsCodeCoverage.txt','w');
        ob_start('file_callback');
        echo( "<?php\n\n" );
        echo( "// parserTests code coverage for MediaWiki version {$version}.\n\n" );
        echo('$version = "' . $version . '";' . "\n");
        echo( '$CodeCoverage = ');
        var_export(xdebug_get_code_coverage());
        echo( "\n\n?>" );
        xdebug_stop_code_coverage();

        ob_end_flush();
        fclose ( $handle );
        return ($ok ? 0 : 1);
        }
    }

  $runtest = new RunParserTests;
  $status = $runtest->run();

AnalyzeCodeCoverage.php

AnalyzeCodeCoverage uses the file parserTestsCodeCoverage.txt located in the /maintenance/ directory (by executing an include with it as parameter). This file is generated by RunParserTests.php. AnalyzeCodeCoverage counts the total number of executable lines in the files referenced by parserTests (as defined by the Xdebug code coverage logic) and the total number of executable lines visited. It divides the latter by the former to get a grand total percentage code coverage and then generates other grand total statistics. It then uses the per file code coverage statistics to generate a histogram of percentage code coverage and two file lists. The first is ordered by the percentage of code covered in each file and contains the total number of executable lines in the file, the number of lines visited and left unvisited and the percentage visited. The second file list contains the same per file information with the list ordered alphabetically by file pathname. All of this information is output to the file parserTestsCoverageAnalysis.txt

AnalyzeCodeCoverage takes one optional parameter, --file. If absent, the statistical information is formatted for display on a wiki page. If present, the statistical information is formatted for display in a file. In order to strip the prefix from the file pathnames and root their names in the distribution directory, the variable $path_prefix is set to the full pathname of the distribution. Users of this program must modify this string to the valid value on their machine.

The Program

<?php

$handle = 0;
$path_prefix = "/Volumes/Macintosh 2nd HD/DataBucket/Wiki Work/Code Development/LatestTrunkVersion";

include_once( 'commandLine.inc');
include( 'parserTestsCodeCoverage.txt' );

function sort_order_filename($x, $y)
    {
    if($x['filename'] == $y['filename'])
        {
        return 0;
        }
        else if ($x['filename'] < $y['filename'])
        {
        return -1;
        }
        else
        {
        return 1;
        }
    }

function sort_order_percentage($x, $y)
    {
    if($x['percentage'] == $y['percentage'])
        {
        return 0;
        }
        else if ($x['percentage'] < $y['percentage'])
        {
        return -1;
        }
        else
        {
        return 1;
        }
    }

$FileNotWiki = isset( $options['file'] );

function file_callback($buffer)
    {
    global $handle;
    fwrite($handle, $buffer);
    }

foreach ($CodeCoverage as $key => $value)
    {
    $suffix = str_replace($path_prefix , "", $key);
    $ShortNames[$suffix] = $value;
    }
unset($CodeCoverage); // Release $CodeCoverage since both arrays take up a lot of space

$GrandUncovered = $GrandPercentage = $NumberFiles = 0;
foreach($ShortNames as $file => $CountArray)
    {
    $Count['covered'] = $Count['uncovered'] = 0;
    foreach($CountArray as $line => $line_count)
        {
        if( $line_count > 0)
            {
            $Count['covered']++;
            } else {
            $Count['uncovered']++;
            }
        }
    $total = $Count['covered'] + $Count['uncovered'];
    $percentage = ( (float) $Count['covered'] / (float) $total ) * 100.;
    $CodeStatistics[$NumberFiles] = array('filename' => $file, 'covered' => $Count['covered'], 'uncovered' => $Count['uncovered'],
                                    'total' => $total, 'percentage' => $percentage);
    $GrandTotal = $GrandTotal + $total;
    $GrandCovered = $GrandCovered + $Count['covered'];
    $NumberFiles++;
    }

$GrandUncovered = $GrandTotal - $GrandCovered;
$GrandPercentage = (int) (( (float) $GrandCovered / (float) $GrandTotal) * 100);

// Output statistics

if (file_exists ( 'parserTestsCoverageAnalysis.txt' ))
    {
    unlink('parserTestsCoverageAnalysis.txt');
    }
$handle = fopen('parserTestsCoverageAnalysis.txt','w');
ob_start('file_callback');

if( $FileNotWiki == true)
    {
    $NL = "";
    $Indent = "\t";
    $BeginPad = "";
    $EndPad = "";
    $HistSpace = " ";
    $HistSingleDigitPad = "  ";
    $Hist100Pad = "   ";
    } else {
    $NL = "<br/>";
    $Indent = ":";
    $BeginPad = "<math>";
    $EndPad = "</math>";
    $HistSpace = "\ ";
    $HistSingleDigitPad = "\ \ \ ";
    $Hist100Pad = "\ \ \ \ \ ";
    }

echo("Code coverage statistics for parserTests using version $version at " .date('H:i, jS F Y') . $NL . $NL . "\n\n");
echo("Number of files exercised: " . $NumberFiles . $NL . "\n" );
echo("Number of lines in those files: " . $GrandTotal . $NL . "\n");
echo("Lines covered: " . $GrandCovered . $NL . "\n");
echo("Lines not covered: " . $GrandUncovered . $NL . "\n");
echo("Percentage covered:  " . $GrandPercentage . "%" . $NL . "\n\n");

// Output Histogram

echo("Code Coverage Percentage Histogram\n\n");

$HistCount = array();
for ($i=0; $i < $NumberFiles; $i++)
    {
    $HistCount[(int) ($CodeStatistics[$i]['percentage'])] =
        $HistCount[(int) ($CodeStatistics[$i]['percentage'])] + 1;
    }

for ($i=0; $i < 101; $i+=5)
    {
    $IntervalCount = 0;
    for ($j=$i; $j < ($i+5); $j++)
        $IntervalCount = $HistCount[$j] + $IntervalCount;
    if($i < 10) $Offset = $HistSingleDigitPad ; else $Offset = "";
    echo($Indent . $BeginPad . $i);
    echo(($i < 100 ? "-" . ($i+4) . ": " : ":" . $Hist100Pad )  . $Offset);
    for ($k=0; $k < $IntervalCount; $k++)
        echo("*");
    echo($HistSpace . $IntervalCount . $EndPad);
    echo($NL . "\n") ;
    }

if( $FileNotWiki == true)
    {
    echo("\nSource file statistics\n"); // Label the output in the file version
    }

echo("\n"); // Put some space between the global and file specific statistics

// Sort in percentage order and output per file statistics in that order

usort($CodeStatistics, 'sort_order_percentage');

echo("Percentage order" . $NL . $NL . "\n\n");

for ($i=0; $i < $NumberFiles; $i++)
    {
    echo ("File:  ..." . $CodeStatistics[$i]['filename'] . $NL . "\n");
    echo ($Indent . "lines covered: " . $CodeStatistics[$i]['covered'] . $NL . "\n");
    echo ($Indent . "lines uncovered: ". $CodeStatistics[$i]['uncovered'] . $NL . "\n");
    echo ($Indent . "total lines: ". $CodeStatistics[$i]['total'] . $NL . "\n");
    echo ($Indent . "percentage coverage: ". (int) $CodeStatistics[$i]['percentage'] . $NL . "\n");
    if( $FileNotWiki == false)
        {
        echo("\n"); // If we are creating wikimarkup, put a new line in to make it readable
        }
    }

// Sort in filename order and output per file statistics in that order

echo("\nFilename order" . $NL . $NL . "\n\n");

usort($CodeStatistics, 'sort_order_filename');

for ($i=0; $i < $NumberFiles; $i++)
    {
    echo ("File:  ..." . $CodeStatistics[$i]['filename'] . $NL . "\n");
    echo ($Indent . "lines covered: " . $CodeStatistics[$i]['covered'] . $NL . "\n");
    echo ($Indent . "lines uncovered: ". $CodeStatistics[$i]['uncovered'] . $NL . "\n");
    echo ($Indent . "total lines: ". $CodeStatistics[$i]['total'] . $NL . "\n");
    echo ($Indent . "percentage coverage: ". (int) $CodeStatistics[$i]['percentage'] . $NL . "\n");
    if( $FileNotWiki == false)
        {
        echo("\n"); // If we are creating wikimarkup, put a new line in to make it readable
        }
    }

ob_end_flush();
fclose ( $handle );