Expanding the isotracker testcase management capabilities

This is deeply related to what is proposed at https://blueprints.launchpad.net/ubuntu/+spec/qa-q-iso-testing-process . If we are confident about the basic pillars that lead from an ISO to a working first boot, ISO-Testing seems redundant. It will be impossible to ever have enough test-cases to effectively test the OS in all of it's aspects.

Please consider the infrastructure mentioned at https://blueprints.launchpad.net/ubuntu/+spec/qa-q-qa-testing-cadence

What to test, how to test, everyday, should be addressed by a special repo and page. It only works it targeted at an organized team, that can follow the process and produce valuable reports. Otherwise, it's just more bug reports. We don't need that. We need less bugs.

About reporting results: Launchpad is already a great platform for reporting bugs. But the ever growing flow of new bug reports makes things confusing. Developers can easily not see an important bug report. What we need is is to think a little about tags. Reports coming from experienced testers and teams , from LiveTesting sessions and scheduled testing sessions should not be triaged and moved at will. A tracker only makes sense to me if it helps bugs get fixed. The only way I imagine it happening is if a small amount of high added-value bug reports done by experienced testers is directed towards and easily accessible by the respective developers. That can be done immediately and with no new investments if we agree on tags for specific testing efforts, and instruct triagers to not alter such reports.

Thus by filtering LP with these special tags and target packages, we will be able to have a comprehensive view on testing status, testing results, packages quality. Testing teams should also be encouraged to document their daily testing activities and results. That is to be managed by testing teams, not by QA. [Effenberg0x0 at Ubuntu.com]

AGENDA:

    Summarize iso tracker capabilities

    Used for iso tracking

    links to a wiki for testcases

    http://testcases.qa.ubuntu.com/

    api available

    is it stable / published in archive?

    Current testcase landscape

    wiki

    http://testcases.qa.ubuntu.com/

    checkbox-application-testing

    http://bazaar.launchpad.net/~nskaggs/checkbox/checkbox-app-testing-qt/files/head:/jobs/

    moztrap

    https://github.com/mozilla/moztrap

What do we need in testcase managerment system?
Test case management
- Add test cases
  * Test Case ID
  * Dependancies
  * Target packages: what is this test case `
  * Actions/expected results
  *
- Remove/modify test cases

    Allow for authenticated users to edit test cases (

    Allow anyone to view test cases

    Allow for authenticated?including flavors?)

    Should allow for flavors to have seperate testcases

    link testcases to series

    re-think the status of testcases

    accessibilty test cases added

    test case management system is accessible

    testcase->testsuite->series

    allow granularity for tracking testcases and linking with results; eg, the testcase result on monday shows the version of the tescase from same time/date

gema> Test Case ID Issue: It may be solved by adding tagging capabilities for test cases to the tracker. Suggested by kate based on another app, let's discuss it next week

Work items:
[stgraber] Update DB schema to have testcase entries be one-to-many to testsuite and have testsuite be many-to-many to milestone_series: DONE
[stgraber] Never allow modifications to qatracker_testcase: DONE
[stgraber] 2: Figure out how to manage users roles (per product roles) (use lp teams, grant isotracker roles based on team): DONE
[stgraber] add link to report "bug" against problems found on a testcase (bug #684404): DONE
[stgraber] 2: come up with development plan for cycle: DONE
[stgraber] add testcase id that stays unique and persists across revisions: DONE
[stgraber] Let the team know when (iso.qa.dev.stgraber.org) is ready for testing: DONE
[stgraber] implement reading LAVA format on the tracker (http://linaro-dashboard-bundle.readthedocs.org/en/latest/index.html): POSTPONED