First data are presented from the tests. Next data are described about game performance. Finally data are presented about the use of additional and standard tools and the relationship between the use of these tools (based on the logfiles) and test scores as well as game performance.
Focus questions and lessons learned
5.3.1 Test results
Table 5-1 shows the average scores (and standard deviations) on the pre-test and post-test for the total group and the groups with the standard support tools and with the additional assignments. On the pre-test students on average gave a good answer on 38.4% of the items. On the post-test this was 48.7%. Although the score on the post-test is still low, there is evidence for a small learning gain. The difference between the post-test and pre-test score was significant for both experimental groups (paired samples t-test Additional assignments group: t=-3.26, df=14, p=0.006;
Standard tools: t=-3.43, df=10, p=0.006). The gain of knowledge especially occurs on the explicit knowledge items. The difference between the post-test and pre-test score on these items was significant for both experimental groups (paired samples t-test Additional assignments group: t=-3.46, df=14, p=0.004; Standard tools: t=-3.16, df=10, p=0.01). The difference on the implicit knowledge items was not significant.
There are no significant differences between the two experimental groups on the pre-test, post-test as well as on the transfer test.
Table 5-1. Average number of correct answers (and standard deviations) on the concepts (con), events (ev), knowledge management model (km), and relationships (rel) for the total group and for the groups with standard tools and with additional
Several types of questions were included in the explicit knowledge items: questions about concepts, events, steps in the knowledge model and questions about the relationship between interventions and indicators. In Table 5-2 the average number of correct answers is presented for each of these types. This indicates that the average knowledge gain on the explicit items of 2.4 (see Table 5-1) is mainly due to a gain of conceptual knowledge about knowledge processes and the like. The difference between pre- and post-test scores is significant for both experimental groups (paired samples t-test Additional assignments group: t=-4.01, df=14, p=0.001;
Standard tools: t=-2.67, df=10, p=0.024). And to a lesser degree to a gain in knowledge about the relationship between interventions and indicators (only a significant difference for the standard tools group: paired samples t-test: t=-3.55, df=10, p=0.005). This last type of knowledge could have been induced by the
Is it all in the game?
students themselves, but also was available to students by means of the feedback that was given and indirectly by the advice functionality that was available. A third source was the interventions handbook. The advice actually refers to certain parts of the interventions handbook. The differences between the two experimental groups on the four different types of items were not significant.
5.3.2 Game performance
Table 5-3 shows the starting and end values (after quarter 12 is finished) for some of the business indicators. It shows that it was difficult to improve market share. Most of the students managed to improve it during the game, but at the end market share went down again. Most of the other indicators have been improved substantially.
There were only small differences between the two experimental groups.
Further research showed that market share went down because of a combination of events that had a negative influence on market share. This also had an influence on profit and turnover.
* MS = Market share, CSI = Customer satisfaction index, Profit = Total profit in 3 years in millions, JSI = Job satisfaction index, PQI = Product quality index, ATM = Average time for new product to market, Turnover in last quarter in millions.
Table 5-4. Average level of competence in three knowledge domains for the total group and for the groups with standard tools and with additional assignments. At the bottom row the starting values. Scale from 1 to 10.
Competence in within the Coltec company at the end of the game together with the starting values.
Level of competence in the marketing and in the research and development domain has been improved considerably. In the production domain only a small improvement has been achieved. This is due to the fact that indicators related to the knowledge processes of gaining and development in the production domain were very low. For instance the average value of the effectiveness of knowledge gaining was 2.7 and of knowledge development was 4.3. In the R&D domain these values were 6.1 and 9.5 and in the marketing domain 8.2 and 8.2. During game play it became clear that the
Focus questions and lessons learned
KM Quest environment did not contain enough interventions to influence these two knowledge processes in that particular domain.
5.3.3 Relationship between test results and game play
From the sections above it is clear that players gain knowledge and that they are able to improve business and knowledge indicators. It is interesting to see whether there is a correlation between the scores on the tests and the values of game indicators. Table 5-5 shows that the correlations between pre-test and post-test score, knowledge gain (from pre-test to post-test) and some of the business and knowledge indicators for both groups are very low. None of them was significant. So there are no indications that a high level of prior knowledge leads to “better” game play or that being able to perform well in the game also means that one has a high post-test score or high knowledge gain.
Table 5-5. Correlation between total pre-test and post-test score, knowledge gain and the average values for some important business and knowledge indicators at the end of the game for the groups with standard tools and with additional assignments.
MS* CSI JSI PQI CM CR CP quality index, CM= Level of competence in marketing, CR= Level of competence in R&D, CP= Level of competence in production.
5.3.4 Use of additional assignments
As stated before the players in the condition with additional assignments could use these assignments, but their use was not enforced by the system. It was possible to ignore them. As it turned out 5 students (out of 16) never formulated a lesson learned and 4 never answered a focus question. The average number of quarters in which a lesson learned was formulated was 5.50 (standard deviation is 4.43) and the average number of quarters in which a focus question was answered was 5.31 (standard
Is it all in the game?
If a comparison is made of the score on the explicit items of the post-test (see Table 5-6) between the group that formulated lessons learned less than the average (5 quarters or less) and those who did more often than the average (6 quarters or more) it shows that the first group has a score that is significantly higher than the second group (Anova: F=6.70, df=1, p=0.021).
The same holds for those students who answered the focus questions less than the average and those who did more often (see Table 5-7). The first group has a significantly higher score on the explicit knowledge items (Anova F=9.02, df=1, p=0.009) and total test score than the second group (Anova F=5.36, df=1, p=0.036).
These two measures are not independent. Analysis showed that students who used focus questions often were the same as those who formulated lessons learned often (except for one student). (maximise market share, customer satisfaction and profit) and to a lesser degree on underlying principles and ideas or the knowledge management model. To illustrate this, some of the lessons learned formulated by different players are presented below.
“I have to give more attention to customer satisfaction.”
“Market share had to be improved and I succeeded in doing that”
“Keep your customers satisfied!”
“Training people was a good idea, but turned out to be bad for the profit”
“To improve things you have to spend money, but you have to look how many years you have to do with your money. In my case I have spend too much money, because I have to do 9 quarters with the same money as I spend in 3 quarters”
5.3.5 Access of standard tools
Learning and reflection in the KM Quest environment was supported in several ways.
Several resources were available that could give the learners additional information when needed, like an Intervention handbook, Indicator handbook, what, how and help files, and visualisation tools. Players could look at past performance by means of historic data about their own actions and about the status of the business model indicators (history). Furthermore, they could get feedback on the previous events (type of event, and possible interventions), and they could get advice (alerts that
Focus questions and lessons learned
indicate that certain indicators are low and that give hints about the types of interventions that could be implemented).
Table 5-8 gives an indication about the frequency of access of the support tools available in the system. The table shows that advice is accessed frequently by most of the players: in 83% of the quarters where advice was available it actually was consulted (students pressed the advice button in the menu bar to display information). The Intervention handbook, the feedback and the visualisation tool were also accessed frequently (in about half of the 12 quarters). The other support tools/information are rarely accessed. It is striking that the history information about one’s own behaviour (as well as system and business model behaviour) is hardly used at all. Although there are differences in the use of tools between the two experimental conditions, none are significant. Within the additional assignments group there were also no significant differences between the subjects who accessed the focus questions and lessons learned often and those who did not.
Table 5-8. Average number of quarters in which a selection of the standard support tools were used by the learners for the total group and for the groups with standard tools and with additional assignments.
Feedb Adv* Intv HB Indc HB Hist What How Help Visual divided by the number of quarters advice was actually available in the system)