Simplifying the Checkbox project

"One checkbox to rule them all"

The current checkbox test harness infrastructure has a somewhat convoluted structure: there are at least four versions of checkbox (vanilla or community, compatibility, certification and oem). Each includes extensions to base checkbox, as well as a set of appropriate tests.

Historically this has led to dependency problems, confusion as to which version of checkbox contains which test, test conflicts and overlapping, and a coupling of tests with the harness which complicates use, maintenance of tests and support of checkbox and its derivatives.

The goal here is:

1- To simplify the tool and repository structure. Ideally there should only be one checkbox tool, and one test repository. These should be separated so that the test repository can be upgraded without needing to upgrade the tool too, *unless* a test requires a specific version of the tool (which can be nicely expressed with apt dependencies).
2- To manage the different kinds of tests that are done (oem, qa, certification, sru, bootchart, etc) via a whitelist mechanism. The existing whitelisting facilities should be revised to make this easier than it currrently is (i.e. auto-whitelisting of dependencies, possibly a nice GUI to select sets of tests, etc).
3- To clearly document test development and tool usage, enabling both Canonical and the community to more easily contribute tests to the repository.

As a bonus, some work could be done on checkbox to address usability and ease-of-use concerns. This will improve the life of test engineers (who have to spend a lot of their time using checkbox) as well as entice community members to do some more testing (via an easier, nicer, more pleasant-to-use tool).

Benefits: Ideally, checkbox users (QA, Hardware Certification and OEM) will have an easier time writing and running tests, and a consolidated test repository will make it easier to see which tests already exist, helping reduce duplication of work currently caused by isolation between versions of checkbox. A simpler structure will also ease debugging of test-related problems. Finally, a better-structured, friendlier checkbox infrastructure will hopefully entice community to contribute in both creating and running tests.