The overall process of workload development is shown in Figure 2. Our goa l was to represent typ
ica l timesharing environments for the d i fferenr ta rget markets . The entire strategy consisted of
• Identifying typical real s i tes
• Col lecting data on resource u t i l i zat ion and i mage usage patterns
• Deriving a packaged workload ro represent the real s i te environ ment
• Va lida t i ng the workl oads by comparing the resource u t i l i zation of the workload aga inst the resource u t i l i zation at various cusromer sires and modifyi ng the workloads as req u i red
65
--- Performance Evaluatio n of the VA X 6200 .�)'stems
R EA L SYSTEM
R ESOU R C E U T I L I ZATION DATA
l
R ES O U R C E U T I L I ZATION DATA
-TER M I NAL ACTIVITY -USER CHARACTER ISTICS -USER M I X
1
-VAXRTE SCRI PTS - U S E R CHA RACT E R I STICS - U S E R MIX
STANDALONE SYSTEM
Figure 2 Interactive Multiuser Workload Development
In the fol l owing sections, we describe how we used this strategy to develop two mul t i user workloads: the engineeri n g workload , which rep·
resents an Electronic Computer-Aided Engineer
i ng environment (ECAE) ; and the Software Deve l opment Environment Workload (SDEW) .
Data Collection
Two Digital sires were chosen to represen t the ECAE and SDEW environments . I n ternal sites were chosen i n i t i a l l y to faci l itate the data col lec
tion process. Both s i res had c lustered environ
ments that consisted of a variety of VAX systems along with some workstations.
We collected i n formation on these c lustered systems to capture thei r behavior u nder the l oad generated by the environment over a period of one week . VAX SPM software was used to col lect resource util ization data (CPU, 1/0. and memory uti l i zation) on a l l the systems at both user level and system leve l . VMS I mage Accou n t i ng was used to obtai n resource u t i l i za tion data on a n i mage basis. Usi n g t h e SET HOST/LOG Digital Command Language ( DCL) command , we col lected log files of user sessions to study user habi ts . Other user characteristics, such as think t i me and type rates, were obtai ned through inter
views and observa tions .
Data A nalysis
The performance team studied t he c luster-wide resource util ization profil es in order ro select the t i me when the i n teractive activities were pre
dom i nant. We compared resource u t i li zation profi l es of i nd ividual systems agai nst r he c l
uster-66
w ide average over a week's accumu lation of data . Based on t h is comparison , we selected a typical day and a typical syste m . O ne hour was c hosen from the typical system on a typical day d u ri ng the period of peak i nteractive use to c haracterize t he system at fu l l load .
Further, based on the user profiles, we classified users accordi n g to computer usage, that is, heavy or l ight computing (for ECAE workload) and heavy, med iu m , or l ight comput·
ing (for SDEW workload) . We then used the i mage accoun t i ng data and user log files to c l as
sify users according to the type of activity t hey performed .
Once several user c lasses were identified, the number of users in each class, or user m i x , was determi ned. We defined the user m i x by l ooki ng at ( 1 ) t he n u m ber of users i n each c lass at the
Table 3 ECAE and SDEW User Mix
Type of User Eng ineer: H eavy Eng ineer: Light
Type of User
ECAE User Mix
SDEW User Mix
Heavy software development Light software development Secretary
Technical writer
No. of Users 3
3
No. of Users 1
3
Digital Technical journal No. 7 A u/lust 1 988
one-hour pea k , and ( 2 ) the organization struc
ture at t he rea l sites. Tab l e 3 shows the user m i x for ECAE a n d SDEW workloads . I n addition to interactive users, t hese workloads a lso have batch jobs runn i ng in the background.
Developing the Workload
Having identified the user c lasses and activities, we then deve l oped an imermediate workload using DCL command p rocedures. This i nter
mediate step a l l owed easier translation to the final workload , which was based on VAXRTE (VAXfVMS Remote Terminal E mu lator) scripts.
I ndividual user scripts were developed and val i dated . We then packaged t h e entire workload by i ntegrat i ng a l l of the user scripts and the batch jobs. Once development was complete, the workload was val idated at both system a nd user levels against the rea l i nternal site. Further val i dation was done at the user l evel agai nst Digita l 's customer sites.
Workload Validation
This section describes the workload val idation process using the ECAE work load as an example resource uti l i zation of the real system during the typical hour. System - and process-level resource u t i l izat ion data of severa l different resources which was considered tO be wel l within accept
able l i m i ts. Val idation of the 010 rare was made
Table 4 User Resource Utilization for Real Internal System and ECAE Workload
System-level validation - For system - l evel val idation , we compared t h e system-level usage of during the develop ment of the workload t he CPU and d isk 1/0 u t i li za t ion of subprocesses was added tO the resource u t i l ization of t he parent process. Although the workload represents the work done by t hose subprocesses and the load p laced on CPU and disk 1/0 resources, the workload does not represent the additional mem
ory requ i red by those subprocesses . As w i l l be described in subsequent sections, the lower memory u t i l i zation of the workload d id not con
stitute a problem.
67
1 00%
Figure 3 CPU Utilization for Real Internal System and ECAE Workload for 1 Hour
"' :::
Figure 4 DIO Utilization for Real Internal System and ECAE Workload for 1 Hour
1 00%
Memory Utilizatio n for Real Internal System and ECAE Workload for 1 Hour
A summary of the comparisons of t he average was to determ ine i f the workload was representa
tive of the l oad placed on systems by Digita l 's customers .
Two semiconductor manufacturers i n Cal i for
n i a were used as va l idation sites for t he ECAE workl oad . I n i ti a l ly, it was determ ined that t here were sign i ficant d i fferences between t he work performed at these customer sites and the work performed at the i n ternal Digital site. The Digital internal VAX systems were used for l ogic design of gate arrays , c i rcu i t boards, and systems; software used by external sem iconductor deve l opers. DECSIM simulations requ i re very basis in the workload a nd at customer sites.
The ECAE workload fal ls withi n the range of u t i l i zat ions observed at these customer sites for both disk and memory u t i l i zations . The workload is sl ightly (approxi mately I 0 percent) more CPU i ntensive on a per-user basis than was observed at
Digital Technical Journal No. 7 A ugust 1 <)88
Table 5 System-Level Resource Utilization for Real Interna l System and ECAE Workload
Table 6 Comparison of Resource Utilization on Customer System and
in ECAE Workload
Resource Utilization Customer ECAE
per Hour Sites Workload
formance numbers sl ightly conservative for the computer-aided e lectrical engineering market.