39 This case study has shown that LDMs account for lexical progression in terms of diversity 9: there is a clear increase in vocabulary diversity r om the fi rst to the second production, and r om the second to the third one; even if productions were written with only one month of distance. This may suggest that the growth of lexical diversity in the fi rst stages of language learning is rather fast. However, replicating this study with more data, and especially, with waves of productions more spaced in time, would help proving this hypothesis.
40 Another possibility for the assessment of vocabulary progression would be to use VGCs (Vocabulary Growth Curves – Baayen, 2010), with “cumulated” or
“aggregated” uses of the lexicon on a given production wave 10, whereas LDMs stand for individual scores. Up to a point, VGCs are an emulation of competence of the whole group, whereas metrics refl ect on individual performances. Psycholinguistic crowdsourcing experiments on fi rst language acquisition, as the one conducted by Emmanuel Keuleers and his colleagues at the Center for Reading Research (Keuleers et al., 2015), provide interesting perspectives for similar longitudinal inves-tigations of lexical acquisition in foreign languages. In their massive (300,000 Dutch subjects) investigations, they studied what they called “word prevalence”, defi ned as
“the percentage of a population knowing a word” (Keuleers et al., 2015: 1665). This concept of “word prevalence” could help us connect the individual performances
9. Most current existing lexical sophistication metrics have been developed for English, relying on r equency inventories or corpora that have been designed for English, but not for French. Other metrics, such as Lexical Sophistication Feature: Age of Acquisition (a feature available in CTAP [Common Text Analysis Platform] – Chen & Meurers, 2016), rely on large-scale psycholinguistic investigations that we cannot replicate. To the best of our knowledge, an alternative tool has been elaborated for spoken data, the LOPP (Lexical Oral Production Profi le), but the r equency bands acknowledged in the paper are meant to assess spoken production, not written essays (Lindqvist et al., 2013). Similarly, the metrics available through TAALES (Tool for the Automatic Analysis of LExical Sophistication) as detailed in Kyle and Crossley (2015) rely on English r equency lists.
10. With our data, it is possible to compute vocabulary growth rates and VGCs for “pooled texts” for each progression level (see Ballier & Gaillat, 2016), but the limited size of the texts in our corpus would not allow us to do so at the individual level, i.e., for each text.
as expressed in the individual texts and the knowledge of the group as it can be measured if we pool texts for each production wave. This kind of experiment works on individual versus collective representations when commenting on the methodological diff erence: individual computations of performances versus overall estimation/stimulation of a batch competence. This would give us a real angle for performance/competence and individuals versus longitudinal group assessment.
Interestingly, they showed that the vocabulary growth throughout life follows a logarithmic progression similar to the one observed in Herdan’s Law (Baayen, 2001), i.e., the growth of number of types follows the number of tokens observed in text corpora. It remains to be seen whether this large-scale experiment based on L1 acquisition can be replicated with learners, but it seems to be the case that lexical decision tasks should also be used with learners to get a more precise insight on their vocabulary knowledge.
41 In the longer run, fi ner distinctions may be needed for the investigation of lexical competence as analyzed with learner data. The emergence of more sophisticated data capture systems may lead to make distinctions between the various lexical items mobilized by learners. Some rarer words may be captured by LDMs (as new tokens) but not their corresponding cognitive cost. It could be interesting to measure the time needed by learners to process the lexicon, or at least to retrieve the words they have used in their productions. This is even more important in studies in which, like ours, data collection was not monitored: students wrote their essays on their own at home and had no time limitation. A system like input log or similar key log capture systems would give us evidence of the time required to process linguistic data, provided students type their productions using that system.
References
Alexopoulou , T., Michel , M., Murakami , A. & Meurers , D. 20 Task Eff ects on Linguistic Complexity and Accuracy: A Large-Scale Learner Corpus Analysis Employing Natural Language Processing Techniques. Language Learning 67 (S1):
180-20
Arnold , D., Wagner , P. & Baayen , H. 20 Using Generalized Additive Models and Random Forests to Model Prosodic Prominence in German. In F. Bimbot , C. Cerisara , C. Fougeron , G. Gravier , L. Lamel , F. Pellegrino & P. Perrier (eds.), INTERSPEECH 2013, 14th Annual Conference of the International Speech Communication Association (Lyon, France, August 25-29, 2013) . Baixas: International Speech Communication Association (ISCA): 272-27 Available online: https://www.
isca-speech.org/archive/archive_papers/interspeech_2013/i13_027 pdf .
Baayen , R.H. 200 Word Frequency Distributions . Dordrecht – Boston – London: Kluwer Academic Publishers.
Baayen , R.H. 200 Analyzing Linguistic Data: A Practical Introduction to Statistics Using R . Cambridge – New York – Melbourne: Cambridge University Press.
Ballier , N. & Gaillat , T. 20 Classifi cation d’apprenants r ancophones de l’anglais sur la base des métriques de complexité lexicale et syntaxique. In Actes de la conférence conj ointe JEP-TALN-RECITAL 2016 . Avignon – Paris: Association r ancophone pour la communication parlée (AFCP) – Association pour le traitement automatique des langues (ATALA). Vol. 9: ELTAL : 1- Available online: https://jep-taln20 limsi.
r /actes/Actes%20JTR-2016/V09-ELTAL.pdf .
Balyan , R., McCarthy , K.S. & McNamara , D.S. 20 Combining Machine Learning and Natural Language Processing to Assess Literary Text Comprehension. In X. Hu , T. Barnes , A. Hershkovitz & L. Paquette (eds.), Proceedings of the 10th International Conference on Educational Data Mining (EDM) . Wuhan: International Educational Data Mining Society: 244-24 Available online: http://educationaldatamining.org/
EDM2017/proc_fi les/proceedings.pdf .
Bentz , C. & Buttery , P. 20 Towards a Computational Model of Grammaticalization and Lexical Diversity. In A. Lenci , M. Padró , T. Poibeau & A. Villavicencio (eds.), Proceedings of the 5th Workshop on Cognitive Aspects of Computational Language Learning (CogACLL) . Stroudsburg: Association for Computational Linguistics: 38-4 Available online: http://aclweb.org/anthology/W14-050
Breiman , L. 200 Random Forests. Machine Learning 45 : 5-3
Chen , X. & Meurers , D. 20 CTAP: A Web-Based Tool Supporting Automatic Complexity Analysis. In D. Brunato , F. Dell’Orletta , G. Venturi , T. François &
P. Blache (eds.), Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC) . Stroudsburg: Association for Computational Linguistics: 113-1 Available online: http://aclweb.org/anthology/W16-41
Chipere , N., Malvern , D. & Richards , B. 200 Using a Corpus of Children’s Writing to Test a Solution to the Sample Size Problem Aff ecting Type-Token Ratios. In G. Aston , S. Bernardini & D. Stewart (eds.), Corpora and Language Learners . Amsterdam – Philadelphia: J. Benj amins: 139-14
Cotterell , R., Müller , T., Fraser , A.M. & Schütze , H. 20 Labeled Morphological Segmentation with Semi-Markov Models. In Proceedings of the 19th Conference on Com-putational Language Learning (CoNLL) . Stroudsburg: Association for ComCom-putational Linguistics: 164-17 Available online: https://aclweb.org/anthology/K/K15/K15-10 pdf . Covington , M.A. & McFall , J.D. 20 Cutting the Gordian Knot: The Moving-Average
Type-Token Ratio (MATTR). Journal of Quantitative Linguistics 17 : 94-100.
Crossley , S.A., Salsbury , T., McNamara , D.S. & Jarvis , S. 20 Predicting Lexical Profi ciency in Language Learner Texts Using Computational Indices. Language Testing 28 : 561-580.
DeBarr , D. & Wechsler , H. 200 Spam Detection Using Clustering, Random Forests, and Active Learning. In CEAS 2009: The 6th Conference on Email and Anti-Spam (Mountain View, California, July 16-17, 2009) .
De Vel , O., Anderson , A., Corney , M. & Mohay , G. 200 Mining Email Content for Author Identifi cation Forensics. SIGMOD Record 30 : 55-6
Fergadiotis , G., Wright , H.H. & Green , S.B. 20 Psychometric Evaluation of Lexical Diversity Indices: Assessing Length Eff ects. Journal of Speech, Language, and Hearing Research – JSLHR 58 : 840-85
François , T. & Fairon , C. 20 An “AI Readability” Formula for French as a Foreign Language. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) . Stroudsburg: Association for Computational Linguistics: 466-47 Available online:
http://aclweb.org/anthology/D12-104
François , T., Gala , N., Watrin , P. & Fairon , C. 20 FLELex: A Graded Lexical Resource for French Foreign Learners. In N. Calzolari , K. Choukri , T. Declerck , H. Loftsson , B. Maegaard , J. Mariani , A. Moreno , J. Odijk & S. Piperidis (eds.), LREC 2014: 9th International Conference on Language Resources and Evaluation . Paris:
European Language Resources Association: 3766-377 Available online: http://www.
lrec-conf.org/proceedings/lrec2014/pdf/1108_Paper.pdf .
Gala , N., François , T., Bernhard , D. & Fairon , C. 20 Un modèle pour prédire la complexité lexicale et graduer les mots. In P. Blache , F. Béchet & B. Bigi (eds.), Proceedings of TALN 2014 (Volume 1: Long Papers) . Paris: Association pour le traitement automatique des langues (ATALA): 91-10 Available online: http://aclweb.org/
anthology/F14-100
Greenwell , B.M. 20 “pdp”: An R Package for Constructing Partial Dependence Plots. The R Journal 9 : 421-43 Available online: https://journal.r-project.org/
archive/2017/RJ-2017-016/RJ-2017-0 pdf .
Gregori-Signes , C. & Clavel-Arroitia , B. 20 Analysing Lexical Density and Lexical Diversity in University Students’ Written Discourse. Procedia – Social and Behavioral Sciences 198: 546-55
Gu , Z., Gu , L., Eils , R., Schlesner , M. & Brors , B. 20 “circlize” Implements and Enhances Circular Visualization in R. Bioinformatics 30 ⒆ : 2811-28
Jarvis , S. 200 Short Texts, Best-Fitting Curves and New Measures of Lexical Diversity.
Language Testing 19 : 57-8
Jarvis , S. 20 Capturing the Diversity in Lexical Diversity. Language Learning 63 (S1):
87-10
Johansson , V. 200 Lexical Diversity and Lexical Density in Speech and Writing: A Developmental Perspective. Wori ng Papers in Linguistics 53: 61-7 Available online:
https://journals.lub.lu.se/LWPL/article/view/2273/184
Johnson , W. 194 I. A Program of Research. Psychological Monographs 56 : 1- Kettunen , K. 20 Can Type-Token Ratio be Used to Show Morphological Complexity
of Languages? Journal of Quantitative Linguistics 21 : 223-24
Keuleers , E., Stevens , M., Mandera , P. & Brysbaert , M. 20 Word Knowledge in the Crowd: Measuring Vocabulary Size and Word Prevalence in a Massive Online Experiment. Quarterly Journal of Experimental Psychology 68 ⑻ : 1665-169
Khmaladze , E.V. 198 The Statistical Analysis of a Large Number of Rare Events [technical report MS-R8804]. Amsterdam: Centrum Wiskunde & Informatica (CWI). 21 p.
Kyle , K. & Crossley , S.A. 20 Automatically Assessing Lexical Sophistication: Indices, Tools, Findings, and Application. TESOL Quarterly 49 : 757-78
Layton , R., Watters , P. & Dazeley , R. 20 Recentred Local Profi les for Authorship Attribution. Natural Language Engineering 18 : 293-3
Levshina , N. 20 How to Do Linguistics with R: Data Exploration and Statistical Analysis . Amsterdam – Philadelphia: J. Benj amins.
Lindqvist , C., Gudmundson , A. & Bardel , C. 20 A New Approach to Measuring Lexical Sophistication in L2 Oral Production. In C. Bardel , C. Lindqvist & B. Laufer (eds.), L2 Vocabulary Acquisition, Knowledge and Use: New Perspectives on Assessment and Corpus Analysis . European Second Language Association (EUROSLA): 109-12 Available online: http://www.eurosla.org/monographs/EM02/getfi le.php?name=Lindqvist_etal . Lu , X. 20 Automatic Analysis of Syntactic Complexity in Second Language Writing.
International Journal of Corpus Linguistics 15 : 474-49
Lu , X. 20 The Relationship of Lexical Richness to the Quality of ESL Learners’ Oral Narratives. The Modern Language Journal 96 : 190-20
Malvern , D.D., Richards , B.J., Chipere , N. & Durán , P. 200 Lexical Diversity and Language Development: Quantifi cation and Assessment . New York: Palgrave Macmillan.
McCarthy , P.M. 200 An Assessment of the Range and Usefulness of Lexical Diversity Measures and the Potential of the Measure of Textual, Lexical Diversity (MTLD) . Doctoral dissertation. University of Memphis.
McCarthy , P.M. & Jarvis , S. 200 “vocd”: A Theoretical and Empirical Evaluation.
Language Testing 24 : 459-48
McCarthy , P.M. & Jarvis , S. 20 MTLD, vocd-D, and HD-D: A Validation Study of Sophisticated Approaches to Lexical Diversity Assessment. Behavior Research Methods 42 : 381-39
Palomino-Garibay , A., Camacho-González , A.T., Fierro-Villaneda , R.A., Hernández-Farias , I., Buscaldi , D. & Meza-Ruiz , I.V. 20 A Random Forest Approach for Authorship Profi ling. In L. Cappellato , N. Ferro , G.F. Jones & E. San Juan (eds.), Wori ng Notes of CLEF 2015 – Conference and Labs of the Evaluation Forum (Toulouse, France, September 8-11, 2015) . CEUR-WS.org: 1- Available online: http://ceur-ws.
org/Vol-1391/72-CR.pdf .
Read , J. 2000. Assessing Vocabulary . Cambridge: Cambridge University Press.
Sichel , H.S. 197 On a Distribution Law for Word Frequencies. Journal of the American Statistical Association 70 (351): 542-54
Tack , A., François , T., Ligozat , A.-L. & Fairon , C. 20 Modèles adaptatifs pour prédire automatiquement la compétence lexicale d’un apprenant de r ançais langue étrangère. In Actes de la conférence conj ointe JEP-TALN-RECITAL 2016 . Avignon – Paris: Association r ancophone pour la communication parlée (AFCP) – Association pour le traitement automatique des langues (ATALA). Vol. 2: TALN : 221-23 Available online: https://
jep-taln20 limsi.r /actes/Actes%20JTR-2016/V02-TALN.pdf .
Tagliamonte , S.A. & Baayen , R.H. 20 Models, Forests, and Trees of York English:
Was/Were Variation as a Case Study for Statistical Practice. Language Variation and Change 24 : 135-17
Tanaka-Ishii , K. & Aihara , S. 20 Computational Constancy Measures of Texts – Yule’s K and Rényi’s Entropy. Computational Linguistics 41 : 481-50
Toolan , M.J. 200 Narrative Progression in the Short Story: A Corpus Stylistic Approach . Amsterdam – Philadelphia: J. Benj amins.
Treeratpituk , P. & Giles , C.L. 200 Disambiguating Authors in Academic Publications Using Random Forests. In F. Heath , M.L. Rice-Lively & R. Furuta (eds.), Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries . New York: Association for Computing Machinery (ACM): 39-4
Treffers-Daller , J. 20 Measuring Lexical Diversity among L2 Learners of French:
An Exploration of the Validity of D, MTLD and HD-D as Measures of Language Ability. In S. Jarvis & M. Daller (eds.), Vocabulary Knowledge: Human Ratings and Automated Measures . Amsterdam – Philadelphia: J. Benj amins: 79-10
Tweedie , F.J. & Baayen , R.H. 199 How Variable May a Constant Be? Measures of Lexical Richness in Perspective. Computers and the Humanities 32 : 323-35 Vajjala , S. 20 Automated Assessment of Non-Native Learner Essays: Investigating
the Role of Linguistic Features [arXiv.org preprint]. 1-2 Available online: https://
arxiv.org/pdf/16 0072
Van Ek , J.A. 197 The Threshold-Level. Education and Culture 28: 21-2
Verhelst , N., Van Avermaet , P., Takala , S., Figueras , N. & North , B. 200 Common European Framework of Reference for Languages: Learning, Teaching, Assessment . Cambridge:
Cambridge University Press.
Way , D.P., Joiner , E.G. & Seaman , M.A. 2000. Writing in the Secondary Foreign Language Classroom: The Eff ects of Prompts and Tasks on Novice Learners of French.
The Modern Language Journal 84 : 171-18
Wolfe-Quintero , K., Inagaki , S. & Kim , H.-Y. 199 Second Language Development in Writing: Measures of Fluency, Accuracy, and Complexity . Honolulu: University of Hawaii.
Yu , G. 20 Lexical Diversity in Writing and Speaking Task Performances. Applied Linguistics 31 : 236-25
Zipf , G.K. 194 Human Behavior and the Principle of Least Eff ort: An Introduction to Human Ecology . Cambridge: Addison-Wesley Press.
Tools
Baayen , R.H. 20 “languageR”: Data Sets and Functions with “Analyzing Linguistic Data:
A Practical Introduction to Statistics” (Version 0). URL: https://www.rdocumentation.
org/packages/languageR/versions/ 0 .
Hothorn , T., Hornik , K., Strobl , C. & Zeileis , A. 20 Package “party”. A Laboratory for Recursive Partitioning (Version 3-3). URL: https://cran.r-project.org/web/
packages/party/party.pdf .
Michalke , M. 20 Package “koRpus”: An R Package for Text Analysis (Version 0.06-5).
URL: http://reaktanz.de/?c=hacking&s=koRpus .
R Core Team 20 R: A Language and Environment for Statistical Computing (Version 1 [2016-06-21]). Vienna: R Foundation for Statistical Computing. URL: https://
www.R-project.org/ .
Schmid , H. 199 Treetagger: A Language Independent Part-of-Speech Tagger. University of Stuttgart: Institute for Natural Language Processing. URL: https://www.ims.
uni-stuttgart.de/forschung/ressourcen/werkzeuge/treetagger.en.html .
Appendix
Production 1 Production 2 Production 3
Mean SD Mean SD Mean SD
HDD 3 90 51 3 39 26 3 03 48
Herdan’s C 0.89 0.02 0.91 0.01 0.90 0.02
Maas a 0.22 0.02 0.21 0.01 0.21 0.02
Maas lgV0 21 0.38 55 0.34 48 0.43
MATTR 0.64 0.05 0.68 0.03 0.68 0.04
MSTTR 0.64 0.03 0.68 0.03 0.68 0.04
MTLD 5 12 19 60 34 6 98 2 97
MTLD-MA 5 69 07 6 66 79 6 61 2 87
Root TTR 76 0.92 10 0.80 09 0.76
Summer 0.85 0.03 0.87 0.02 0.86 0.02
TTR 0.60 0.07 0.65 0.04 0.63 0.06
Uber index 06 88 2 08 51 2 33 68
Yule’s K 15 36 3 69 16 61 4 28 12 16 3 47
Table 3 – Mean values for all LDMs across the whole dataset
P1 P2 P3 P1 P2 P3 P1 P2 P3 P1 P2 P3
P1 12 2 11 11 2 11 12 2 11 11 2 11
P2 5 13 1 5 13 2 5 13 2 5 13 2
P2 6 6 8 7 6 7 6 6 7 7 6 7
n trees 5k 50k 100k 1M
Table 4 – Investigating the role of the number of trees ( n trees)
Figure 8 – Conditional importance scores of LDMs
Figure 9 – Number of correct predictions with 1,000 random labels to predict