A Framework for Test Environment Management
Part one in a series on test environment management offers a new test framework called agent control engine
(Note: This article is part one of a two-part series. Part two will appear in an upcoming issue.)
As the IBM* System i* platform capabilities continue to increase, so do demands on the Software Final System Test (FST) team. FST is an independent test group that conducts the final test phase in the i5/OS* release process. Its mission is to test each new release of i5/OS in a customer-like environment before it's available externally. This testing simulates an enterprise environment with a network of systems, including CPU-intensive applications and interactive computing. During test execution, this environment runs as a true 24-7 production situation. FST tracks CPU use for targeted test systems to ensure goals for stressing the systems are met. It's increasingly important to target and monitor system indicators for other areas as well, such as memory and I/O.
To manage the growing number of test applications, allow cross-tester sharing and enable the team to better monitor system performance, FST developed a new test framework called the agent control engine (ACE). This internal IBM framework can be used to control any application (e.g., batch jobs, test cases, etc.) and can be enabled to display relevant system parameters. ACE better manages FST's evolution to more complex, varied and demanding testing.
The ability to adjust agent parameters in real time means testers also need access to real-time data for decision making. ACE was designed to provide centralized access to key system indicators for all systems under test and historical trend data for problem investigation and test assessment.
The flexibility of the ACE framework makes it of potential interest for other testing or as an interface for a variety of applications. Since an agent is generic to ACE, the ACE framework could be useful for running anything from batch jobs to utilities. For example, FST has created an application to change user passwords across multiple systems in the test environment. Designed to be platform independent, the ACE framework could also facilitate testing on other platforms.
In this first of two articles, we'll expand on the FST environment and test strategy, explain the needs that drove the development of ACE, identify formal ACE requirements and provide an overview of the ACE architecture for readers interested in developing a similar framework.
The FST Environment
Figure 1 shows the complexity of the FST environment. This environment consists of many i5/OS partitions, as well as Microsoft* Windows*, AIX* and Linux* partitions, and becomes more complex with each release.
A portfolio of applications, agents and background workloads run across the systems and networks. These applications run simultaneously on targeted systems and emulate customer applications. Their main objective is to uncover system problems caused by application interactions. Agents differ from applications in that they're intended to stress a specific area of the system and go beyond the demands of normal customer use. They're designed to drive out defects in the target area or through the stress they add to the system overall. To ensure that the load on the system is realistic and sufficiently stressful, FST runs simple, self-executing background workloads. Traditionally, background workloads have targeted only CPU use, but this is changing.
The following examples illustrate the differences among applications, agents and workloads:
• Example application - The virtual crime and punishment (VCP) application simulates a data-warehousing environment. VCP consists of a set of courthouses and a data-warehousing system. Each courthouse stores information pertaining to the court hearings held within that location. This information includes data such as the hearing date, judge, offender, list of charges, prosecuting attorney, police officer, sentence, verdict, etc. During the evening, the data from each courthouse is extracted, transformed and loaded to a staging file. Once all uploads are complete, the data is moved into the data warehouse. Analytical queries are then executed over data in the data warehouse. A star schema database model is used within the data warehouse.
• Example agent - The DB2 agent is an on-demand database test application that generates dynamic, complex SQL statements. The application focuses on different areas, including the types of SQL statements, number of joins, length of time to run, etc. The agent provides a last-minute sanity check on complex database functionality.
• Example background workload - Swamp is a simple database application that performs inserts, updates and deletes to a data file set resident to the application.