Integration testing for the bootloader

We would like automated integration testing of our bootloader for 12.10. Foundations will probably need to write these tests, with QA team providing guidance on the kind of tests to run.

* what infrastructure do we have for testing (QA team & Co)
 * real machines would have to be connected to a KVM to allow simulating pressing keys
 * technically possible, never tried as yet ([UPDATE] the KVM we use, Raritan Dominion KXII does not allow this type of access -- it uses a proprietary Java program to interface. It might be possible with a *serial* console, and we -- QA -- are getting it ready to use & try).
 * hardware vs. VMs: firmware-specific tests and anything that involves graphics will probably need hardware; other tests could be run on VMs
 * currently easiest to install from ISO with preseeding; can then feed in an updated test script
 * might make sense to write some autopkgtest tests for grub2, although these can only run on VMs
 * get BIOS images from vendors, use this within KVM (possibly more realistic in future with UEFI)
 * simulate rubbish video tables with SeaBIOS and/or our own UEFI builds

* what test cases do we want/need/require

== GRUB ==

   - Firmware type: UEFI, Mac EFI, BIOS
   - Various file systems: ext[234], FAT, NTFS, XFS, btrfs
   - Wubi: Jean-Baptiste had some automation for this, but needs to be ported to disk images
   - flicker-free (cameras?! video snapshots)
    - problems with native panel resolutions (either missing correct resolution, or broken)
   - intercepting VGA is troublesome because some systems will behave differently when emitting LCD vs. VGA; and it's not what the user is watching.
   - which resolution are we booting at (testing on multiple resolutions?)
   - multi-device (installing from USB, installation location, etc.) - best handled by installer unit tests
   - LVM, RAID, crypto?
   - regular boot, recovery, previous boot failed, booting alternate OS
   - UEFI: verify that escaping from top level returns to previous UEFI application
   - can be configured to read input from serial port
   - PXE support, including chainloading (pxelinux vs. grub2) -- doesn't MAAS test this a lot? Yes, and we also use COBBLER in the lab. Well, right now maas-provision *is* Cobbler.

=== Future work ===

 * Reserve some memory (or use UEFI variables) to pass GRUB instrumentation into kernel for later interrogation

== u-boot ==

 * Scriptable with a shell
   - Supports pre and post u-boot env run
 * serial port?
   - Generally the default for ARM
   - Can be tested using expect as well
 * Reading kernel from different storage types (micro-SD, USB)
   - Types: SD, eMMC, USB, SATA
   - Fs formats: ext[234], FAT
 * PXE support
   - Should also be tested by MAAS
   - Netboot install should probably be enough
 * Updating bootloader
   - Flash-kernel supports updating the boot loader itself, but it's generally not recommended
   - Update might be a good tthing to have once we start using PXE more and more over the time
 * /boot/boot.script (generating boot.scr)
 * device-tree support
   - Making sure the dt file is available for the kernel at the right location

Work items:
[cjwatson] Write initial sample test (BIOS, default installation, does it boot at all): POSTPONED
[apw] Investigate kernel patch for pre-boot performance instrumentation (GRUB -> kernel): POSTPONED
[rsalveti] Check how Linaro is planning to test the u-boot support at ARM: POSTPONED
[canonical-qa] Set up test harness based on initial sample - use kernel sru or bootspeed test systems as a basis: POSTPONED
[hggdh2] try Raritan console scripting for testing in one of the kernel SRU machines (not possible with Raritan, may be possible with a the serial console interface): DONE
[cjwatson] Write test for recovery mode including whether the kernel was told not to do VT handoff: POSTPONED
[cjwatson] Write last-boot-failed test: POSTPONED
[canonical-foundations] Write lots more boot loader tests: POSTPONED
[ogra] Placeholder work item for u-boot test scripting: POSTPONED
Ensure that u-boot was installed properly (including setting various bits of firmware data; such as CPU speed data, device tree in place etc): POSTPONED
[canonical-qa] investigate screen capture/video recording solutions (screen flickering detection): POSTPONED
[cjwatson] Get information from jk on UEFI VM setup and prepare initial test harness: POSTPONED