Identifying Experts & Establishing Inclusion Criteria

‘Expert’ is a term frequently used but infrequently operationalized in everyday life. For example, it is fairly common to encounter and act upon the advice of experts without enquiring deeply into the basis for the expert’s designation as such. The need to identify experts or competent performers in a domain is, however, of central importance to researchers working within an expertise framework (Jääskeläinen, 2010), even when the research adopts a relativist approach.

In many domains, the identification of expert performers via objective criteria is far from straightforward—consider, for example, the complexities of aviation, medicine, and music, just three of the performance domains discussed in Ericsson, et. al. (2006). Indeed, Ericsson &

Charness (1997:7) recognize that “in most domains it is easier to identify individuals who are socially recognized as experts.” Identifying experts solely on the basis of social criteria (i.e., status, peer recognition) is problematic, however. Not all individuals identified as experts in a given domain will necessarily perform better than novices, and experience (time on task) and knowledge are not always reliable predictors of performance (Ericsson & Charness, 1997:7;

Ericsson, 2006a, 2006b; Jääskeläinen, 2010). Yates (2001:24) echoes this theme, calling

attention to the notion of “experienced incompetence” and warning scholars to be wary of seeing experience as equivalent to expertise. Similarly, Jääskeläinen (2010, citing Sirén & Hakkarainen, 2002) points to studies in which novices outperformed supposed experts at translation tasks, and advocates for differentiating between expertise and professionalism in translation studies, noting that “while all experts are professionals, not all professionals are experts” (p. 215). She also cautions against confounding the notions of specialization (e.g., in a specific type of

translation/interpreting, such as medical or legal) and expertise.

Approaching expertise research from a relativist stance does not obviate the need to

operationalize the definition of ‘expert’ or ‘competent performer’ for a given research project or domain. When the criteria for designating ‘expert’ and ‘novice’ groups vary greatly from study to study it becomes difficult, if not impossible, to make valid comparisons across studies (Köpke

& Signorelli, 2012). This is also true when the inclusion and exclusion criteria are not clearly described by the researcher. Despite the many expertise-related studies in the Interpreting Studies literature, there are no widely agreed upon criteria for inclusion in ‘expert’ or ‘novice’

groups. The studies discussed in Section 3.4 employ a wide range of criteria that are not always clearly defined or explicated. The following examples are illustrative of the diverse inclusion criteria for expertise-related studies of interpreting; note that the referenced publications are not always clear as to whether the author is describing preestablished criteria that were used for inclusion/exclusion in the study or providing a post hoc report on the participants’ demographic information:

• Ivanova (1999, 2000): The expert group had an average of 9 years of experience. The novice group was composed of students who had three months of simultaneous conference interpreting study/training.

• Moser-Mercer, et al. (2000): The group of professional interpreters had 5-10 years of experience. The novice group was composed of students beginning their first year of (postgraduate) study of conference interpreting.

• Liu (2001): The professional group had at least one year of formal training and at least two years of experience with at least 40 working days per year. The advanced novice group had 1.5 years of training in simultaneous conference interpreting. The novice group had just begun training in simultaneous conference interpreting.

The research described in this dissertation involved two groups (termed ‘novices’ and

‘experts’ in the results and discussion; see Section 3.2 for further discussion of the approach to expertise studies taken in this dissertation; see also Section 9.3.1 for a further note on the groups) with differing levels of training and experience in order to gain insight into performance at multiple points on the developmental trajectory. Following Chi’s (2006a) recommendation of

academic training, years on the job, and recommendation by peers in the domain as criteria for identification of competent performers, I established educational and experience criteria for inclusion in the study (peer recommendation was not used as an inclusion criterion, given the need to preserve confidentiality; however, peer networks were draw on to identify potential participants).

Liu (2001) recommends consideration of formal training in interpreting as a criterion for inclusion in expertise-related studies of interpreting, noting that individuals (in general, not necessarily interpreters) with both experience and training in a given domain have been shown to outperform those who have experience but no training. Köpke & Signorelli (2012) also stress that the type of training and the quality of professional experience are important factors to consider in identifying participants for research due to the possibility that differences in these factors may influence performance (and, thus, research findings). This consideration is especially relevant to this study, given the realities of the professional landscape of dialogue interpreting in the geographical area in which it was conducted: there are working interpreters with many years of experience who have little formal training. There are also recent graduates of academic programs that have considerable training but little experience. For this reason, separate criteria were established for training and experience for each group. The criteria and rationale for each one are listed below.

For the novice group, the inclusion criteria were as follows:

• Maximum of 40-60 hours of formal training.

o Some level of training was considered necessary, as the study was focused on trained interpreters rather than ad hoc interpreters or untrained bilinguals.

o The majority of private training programs that prepare interpreters to work in the field are 40 or 60 hours long, so this was established as the training cutoff for novices.

• Less than 200 hours of interpreting experience.

o Hours of work was considered a better measure than days, weeks, or months, given that many interpreters work as freelancers: a ‘day’ might include anywhere from an hour of work to eight or more hours. The quantity of hours was chosen arbitrarily but was thought to be sufficiently low (the equivalent of 5 weeks of full-time work; although novices are not likely to work full time as interpreters) to avoid automation of processing and development of well-established schema.

For the expert (‘competent performers’) group, the inclusion criteria were as follows:

• Graduate of college-level training program.⁴⁴

o Completion of a college-level training program ensures a distinct contrast in level of training in the two groups.

• At least 5 years of interpreting experience.

o Performance improvement is rapid at first, but generally slows dramatically after a certain level of skill is reached (Anderson, 2015). Five years of experience seemed sufficient time for participants to have developed fairly stable schema and automated processing, as well as to have settled into patterns with regard to their use of control mechanisms, etc.