“Alice started to her feet, for it flashed across her mind that she had never before seen a rabbit with
3.9 Advantages and limitations of the research design
3.9.2 Main challenges
However, these conclusions, discussed in Chapter 5 Discussion, were limited by a number of challenges that had to be faced, especially in relation to the interdisciplinary perspective included in the study. The research aimed at situating university students’ motivational profiles against the backdrop of the local economic milieu in order to investigate if such environmental factors were reflected in their motives and attitudes. This required, first of all, a theoretical synthesis of two, so far fairly distinct fields of study, those of L2 motivation and language economics. As I argued in the first chapter, economic considerations are inherent in many of the mainstream motivational concepts used in the study. However, as they tend to be represented covertly, once the challenge of building a theoretical bridge between the two fields was overcome, there were a number of practical issues to address.
Primarily designed to comply with research traditions in psycholinguistics and applied linguistics, the project illustrates some of the distinct characteristics of quantitative inquiries in
these domains. The concepts included in the questionnaire originated from previous research on foreign language learning motivation and the statistical tools used were also influenced by these earlier studies. Thus, while underlying economic aspects of participants’ motivation and attitudes were treated as an important element of their profile, these were examined in the light of the sources presented in chapters Chapter 2, Theory and context, and Chapter 4, Results.
These sources contributed the data necessary to compare and contrast the results of the motivation questionnaire with those of language economics research. Consequently, in some cases the scarcity of relevant research findings posed a considerable challenge and restricted the applicability of the results.
Nevertheless, the most significant obstacle to overcome presented itself during the data collection phase. As I stated above, one of the numerous advantages of internet questionnaires is easy access to participants. However, even when ‘distributing’ the instrument means transmitting an invitation message containing the link to the questionnaire via email, permissions are required, and regulations need to be observed. Therefore, it was decided that formal approval of the Deans of the four faculties would have to be obtained and faculty administration would handle the distribution of the message. At some of the faculties, this process proved to be lengthier and more difficult than previously estimated. Nevertheless, with the sole exception of the faculty of SES, the permissions were finally granted, and administrators were very helpful in circulating the invitation. In order to reach out to SES students, various student associations were contacted. Once their assemblies had discussed the question, most decided to circulate the message among their members. Despite their help, SES students’ participation (Section 3.7) fell behind compared to the rest of the faculties, a clear result of these difficulties.
One last difficulty to be mentioned here concerns the quantitative techniques used in the study.
Despite the abundance of literature and long tradition of survey studies in L2 motivation, experts begin to question the suitability of mainstream statistical tools for the investigation of complex processes of foreign language learning (Dörnyei et al., 2015a). Although quite refined in their mechanics and capable of highly reliable results, statistical procedures employed in traditional applied linguistics research tend to produce conclusions that are rather limited in their applicability. By being generalizable and thus describing a large group of individuals, the results can be argued, in fact, not to describe any one particular participant at all. In addition, quantitative findings often seem to only scratch the surface of certain phenomena, which is
apparent in the large number of questions they raise for each one they answer. These shortcomings of traditional quantitative techniques have prompted researchers to turn to novel statistical tools, conduct mixed studies or develop new methodologies to reach finer conclusions (ibid.).
3.9.3 Limitations
In line with these considerations, it is important to point out that the study described in this chapter followed a methodology that was, at the level of empirical data collection, strictly quantitative. As a result, its findings are subject to the issues discussed above. One example of these often-criticized aspects of quantitative studies is the lack of additional contextual information as regards the individuals in the study. In order to tackle that difficulty and so as to provide a more extensive background to the findings, the study also relies on a detailed description of the social and economic context of the Geneva setting. As I argued above (3.8.3), this approach allowed a better interpretation of the results and helped mitigate the absence of qualitative information. Nevertheless, as I’ll explore in more detail in Chapter 6 Conclusion, there is great potential in future qualitative and mixed-method projects in expanding on the results of the study.
Furthermore, while earlier I outlined the main advantages of electronic surveys, they are also characterized by a number of drawbacks. First of all, the digital layout of the questionnaire, in addition to facilitating participation, at certain points restricted the functionality of the instrument. For instance, it was partly to avoid such restrictions that all of the items were rendered compulsory, which can nonetheless be considered as a constraint. In turn, this all-obligatory policy might also be linked to the rate of dropouts discussed in Section 3.6.1.
Dropouts and self-selection not only impact sample size, but also the qualities of the sample, potentially playing a role in its representativeness.
In addition to dropouts and self-selection, two related problems that often characterize quantitative surveys, the findings of such studies are also curbed by the self-report nature of the data. The latter refers to the increased role of participants’ interpretation of the question and their own thoughts and feelings when it comes to answering a particular item. Therefore, the quality of the data is greatly influenced by the personality of the respondents and by their willingness to share the information. In sum, despite all care taken during the creation of the instrument, it is important to bear in mind that most of the information shared by participants
is a product of their view of the world and themselves and can thus be considered as subjective rather than factual.
Finally, as one of the main obstacles to the generalizability of the results, I would like to briefly draw attention to the question of representativeness, addressed in more detail in Section 3.7.9.
In that section, I concluded that the issue affects many studies in SLA, and even in social sciences in general, as often a more randomly selected sample is beyond the possibilities of the research. As shown above, different measures were utilized to minimize somewhat the drawbacks associated with opportunistic sampling techniques. At the same time, in the analysis and discussion of the results, presented in the following chapters, I exercise caution when it comes to drawing broader generalizations and overarching conclusions.