RedHat Enterprise Linux upgrade

Please pardon the lack of specific company names. Much of our work is performed under NDA, so while generals can be discussed, any information which could provide advantage to a competitor or expose weaknesses is obscured.

Fleet-wide RedHat Enterprise Linux upgrade, Major Retailer:

A major retailer had let their Linux fleet run for several years without consistent patching — 700+ x86 machines running various versions RedHat Enterprise Linux 3 through 5. Furthermore, these machines were running different levels of system board and HBA firmware. Over six weeks, AnderSand designed and implemented a set of upgrade scripts which leveraged their existing (but underutilized) Satellite infrastructure. In addition to upgrading all machines to the most recent revision, HBAs were flashed with updated firmware, vendor-specific support packages were upgraded, and system BIOS flashed. Rolling upgrades were massively successful, eliminating inconsistencies among machines and greatly reducing associated troubletickets. As an added bonus, all RHEL4 and RHEL5 machines were configured with HP SIM (System Insight Manager), allowing phone-home capability on detection of hardware errors.