<<

NAME

fix_test_suite.pl -- remove failing tests from test suite until all tests pass.

SYNOPSIS

  fix_test_suite.pl -p project_id -d suite_dir [-f include_file_pattern] [-v version_id] [-s test_suite_src] [-t tmp_dir] [-A] [-D] [-L]

OPTIONS

-p project_id

The id of the project for which the generated test suites are analyzed. See Project module for available project IDs.

-d suite_dir

The directory that contains the test suite archives. See Test Suites.

-f include_file_pattern

The pattern of the file names of the test classes that should be included (optional). Per default all files (*.java) are included.

-v version_id

Only analyze test suites for this version id (optional). Per default all test suites for the given project id are analyzed.

-s test_suite_src

Only analyze test suites originating from this source (optional). A test suite source is a specific tool or configuration (e.g., evosuite-branch). Per default all test suite sources for the given project id are considered.

-t tmp_dir

The temporary root directory to be used to check out program versions (optional). The default is /tmp.

-A

Assertions: Try to remove failing assertions first, before removing the entire test method (optional). By default failing test methods are entirely removed.

-D

Debug: Enable verbose logging and do not delete the temporary check-out directory (optional).

-L

Logging: Enable logging of runtime information (optional). By default no database is used to keep the runtime information generated by this script.

DESCRIPTION

Runs the following worflow for each provided test suite (i.e., each test suite archive in suite_dir):

1) Remove uncompilable test classes until the test suite compiles.

2) Run test suite and monitor failing tests -- remove failing test methods and repeat until:

• The entire test suite passes 5 times in a row.
• Each test method passes in isolation (TODO: not yet implemented!).

If a test suite was fixed, its original archive is backed up and replaced with the fixed version. The results of the fix are stored in the database table suite_dir/TAB_FIX.

Test Suites

To be considered for the analysis, a test suite has to be provided as an archive in suite_dir. Format of the archive file name:

project_id-version_id-test_suite_src(\.test_id)?\.tar\.bz2

Note that test_id is optional, the default is 1.

Examples:

• Lang-11f-randoop.1.tar.bz2 (equal to Lang-1-randoop.tar.bz2)
• Lang-11b-randoop.2.tar.bz2
• Lang-12b-evosuite-weakmutation.1.tar.bz2
• Lang-12f-evosuite-branch.1.tar.bz2

Logging

This script logs all test-compilation steps information to fix_test_suite.compile.log, all test-execution steps to fix_test_suite.run.log, and it also logs summary information (e.g., how many tests were removed) to fix_test_suite.summary.log in the test suite directory SUITE_DIR.

<<