HAL Id: hal-02366673
https://hal.archives-ouvertes.fr/hal-02366673
Submitted on 16 Nov 2019
HAL
is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire
HAL, estdestinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Language resources and visual communication in a deaf-centered multimodal e-learning environment: issues
to be addressed.
Elena Antinoro Pizzuto, Claudia S. Bianchini, Daniele Capuano, Gabriele Gianfreda, Paolo Rossini
To cite this version:
Elena Antinoro Pizzuto, Claudia S. Bianchini, Daniele Capuano, Gabriele Gianfreda, Paolo Rossini.
Language resources and visual communication in a deaf-centered multimodal e-learning environment:
issues to be addressed.. Proc. VII Int. LREC Workshop “Supporting e-learning with language
resources and semantic data”, ELRA, pp.18-23, 2010, Proc. VII Int. LREC Workshop “Supporting
e-learning with language resources and semantic data”, 2-9517408-6-7. �hal-02366673�
L a n ~ o a q e Resources and Visual Communicatiorm in a Deaf-Centered aultiniodal E-Learning Environment: Issues to be Addressed
Elena Antinoro ~ i z z u t o ' , Claudia S. Bianchini Daniele Capuano Gabriele Gianfreda '. '
,Paolo ~ o s s i n i '
'
Istituto di Scieiize e Tecnologie della Cogiiizioiie. Corisiglio Nazionale delle Ricerclie. Via Nonieiitana. 56 - 001 61, Roma. 'Uiiiversité Paris 8, Rue de la Liberté, 2-
93526 St.Deiiis CEDEX 1 DIFILILE. Università di Perugia, PiazzaMorlacclii, I - 06 123. Perugia, 1tal y. Università di Roma "Sapieiiza7'. Dipartiiiiento di Informatica. Pictorial Computiiig Laboratory, Via Salaria 113, 00198 ~orna.%ipartimeiito di Scienze dell'Educazioiie e della Formazione,
U~liversità di Macerata, P.le Bertelli, 1. C.da ValIebona, 62 100 Macerata - Italy,
This paper examiiies some of tlie rnajor problems linked to tlie task of designiiig appropriate iiiultiliiigual e-learning eiivironments for deaf leariiers (DL). Due to their liearing disability most DL experieiice drainatic difficulties in acquiriiig appropriate literacy skills. E-leat-iliiig tools could in priiiciple be very useful for facilitatiiig access to web-based lcnowledge and pronioting literacy developnient in DL. However, designing appropriate e-learning enviroiimeilts for DL is a complex task especially because of tlie different liilguistic baclcground aiid experience DL inay have, a ~ i d of the inultimodal language resources that need to be provided and iiitegrated (e.g. lailguage produced iii the visual-gestural or sigiled inodaiity, i11 writteii texts, closed captioiiiiig for vocal laiiguage iiiforinatioii). The purpose of tliis paper is hvofold: (1) describe and discuss issues we believe iieed to be addressed, focusiiig oli tlie Iiinitatioiis that appear to cliaracterize several e-learning platforiiis that have beeii proposed for DL; (3) preseiit and discuss ongoing research aiined at overcoinirig these liinitatioils.
It is widely kiiowi~ that al1 over tlie world deaf cliildreii aiid. later, adults, experience dramatic difficulties in achieving appropriate receptive and expressive skills iiot only in ora1 or vocal laiiguage ( V L ) but also i11 written laiiguage. Tlie vast rnajority of deaf leariiers (DL) achieve literacy levels tliat are niarkedly below tliose proper of their heariiig peers (see among otiiers Caselli.
Maragiia & Volterra, 2006; Garcia & Derycke. 2010;
Garcia & Perini, 2010). As a result, iii their scl-iool years through adultliood, DL experieiice equalIy draniatic difficulties i11 accessiiig tlie vast body of knowledge. aiid tlie ricli learning eiiviroiimeiits made available by advanced rnultiniedia technologies, inost iiotably e-leariiiiig eiiviroiiinents. Appropriate written language slcilis are i11 fact unquestioiiably a pre-requisite for exploitiiig tlie possibilities arising fiom su ch mul tiniedia a ~ i d multiinodal leariiing eiivironments.
In Italy as al1 over tlie world' the situatjon of DL is especially coinplex due to the very difrerent language backgrouiid aiid experieiice deaf persons may Iiave depeiiding upon the language tliey use as their priiiiary or preferred iiieans of commuiiicatio~~, or L I . It is in fact necessary to distiiiguisli two groups: ( 1 ) those wlio use Italiaii Sign Iangiiage (LIS), tlie visual-gestriral,
face-to-face laiiguage of the ltalian deaf commuiiity (LIS-LI); ( 2 ) those wlio prefer to use spokeii aiid writteil Italiaii (Italiaii-LI ). It is importaiit to stress that. oli the whole, bofh g-l-ozips q f D L experience severe difficulties in achieving appropriate literacy levels - though of course 'exceptional leariiers' wlio overcome tliese difficulties cali be found witliin each group.
With respect to sigiiers. tlie followiiig must be iioted. Since the modei-11 study of signed laiiguages (SL) began with Stokoe's (1960) pioiieeriiig work o11 Aniericali Sigli language (ASL). world-wide research lias led to describe. aiid to recognize as full-fledged liumai~
natura1 languages, a very large nuniber of national SL.
iiicluding LIS aiid al1 the other inajor Europea11 signed languages. The use of SL for instructional purposes has been esplicitly recoinmended by the European Parliaiiieiit (see Resolutioil 17-6-1 988, arr. D).
Bilingual educati011 programs that offer sigiied and orallwritteii language iiistruction to deaf students have been developed in several coiintries, including ltaly where they have been applied for the most to Elenientary school cliildrei~. As reported by Caselli & al (2006). it is uiiquestioi~able tliat the use of a SL. eveii if limited to its usual, face-to-face- form. cali play a veiy iii~portarit role iii fostering DL's genera1 liiiguistic competence.
The iriclusion of SL witliin e-leariiing platfor~iis designed for DL has come as a natura1 developinerit of
'
For reasoiis liiiked t0 iiie deinography of deafiiess aiid to ihe tlie advai~cemeiit that have been made iii our kiiowledgecompier: socioliiiguistic aiid cultura1 properties of sigiied of SL alld of deaf sigliers. H ~as ,ve poiiit out ili ~ ~ ~ ~ ~ . laiigiiages tlie obseivatioiis we make liere with respect to Ital~v
cm easilx esteilded across natiolls aild c i i l ~ s e s . wjth the ~ e c t i o l l ~ 2 t0 4 below. nlail)l SeCeilt ai-id cuirent atrenlptS 1lecessai-y changes coiicemiiig tiie iiatioiial sigiied aiid to develop appropriate e-learning e~ivirolime~its for DL vocallwritten laiiguages.
exliibit, aildlor implicate some ma-joo conceptual, iiiethodoIogical ai~d practical liil~itations.
Irl section 5 we preseilt and discuss ongoing research aimed at overcoming tl~ese liinitations.
2 . S o m genera1 problems eormcerniwg exhstiwg e-learning pl&iion.ms for DL
For- t l ~ e purposes of this paper, we liri-iit our attentioiis to e-learniilg piatforrns designed for youiig or adult DL. Ai1 overview of severa1 such platforrns reveals the following ma-jor iimitations. First, the griidelines for deveiopinp the desired pjatforrns are often just "sketched7', aiid provide fairly genera1 suggestions conceriiiiig, for exainple: - tlle ii~clusioil of SL videos with SL tsai1siatioiis or explanations of the written texts found in a specific e-leariling platforin;
-
the deveIopinent of automatic tools (Le. avatars) for tsaiislatiitg written texts into SL; -the use of cooperati011 tools such as video conferencing'. Secoild.many existing or plaiined platforms appear to be designed primarily for DL who kilow SL. but s e e n ~ lo neglecr fl7e needs o f D L ~ v h o pr-efer to use YL.
011 tile whole. there appears thus to be a general ti-eild towards creating aild ii~cluding SL materials for impleineiitiilg writteil text-based ei-iviroilinents. Tlze contents eiicoded in written lar~g~iage are made more accessibie to (signing) DL via SL translations and explanations. Otller esamples are tlle platforn~ created within the project DEAL
'
for teacliiiig foreigil vocal-written lailguages to DL, or the one designed by Drigas SC. Koureinei~os (2005) for vocational and general educational training.A fairly large body of work has been dedicated to the developinent of signing avatars to be added to tl-ie users' interface. replacing SL materials preserited by rea1 sisners (see for exainple Efthimiou & Fotinea, 2007;
2008; I<arouzis, Caridaliis, Fotinea BL Efthirniou, 2007, or also the recent Italian project "ATLAS"
9.
Maily pro-jects for realiziilg signiilg avatars exhibit however, ii-1 our view, a rather surprisiilg Iimiration: they appear to have a uniclil-ecfional, FZ-cer?ier.edyer-spectii-e.
They start, for the inost, fiom V L written texts atid aiin at producing avatars that cari tsailslate siich written texts iilto individua1 signs and sigiied sequences. These pro-iect tl~us igilore or underestimate the problem of tr.ar~slatirzg Jho17~ s i p to i~ocal/ii~ritte~7 rexts. There are o111y few
projects tliat explicitly aim at realizing sigiling avatars firnctioning in both directions, i.e. frorn sign to speech aiidlor also written tests, aiid from speech and writteil
'
See for ex.: Ir~divrclitois iiho are Deaf or- I-inrii o f Hearrilg.Center for Assistive Technologj and Ei~viroiiinei-~trtl Access (CATEA ). T t %-- L " 3 z ' G L ? 1: l". . :
_
-- .,, . IAlS gziidelzr7es for D e ~ ~ e i o p i i ~ g ..lccessibiIe Lenr~~rng ..;I~~plicatlotzs. I M S Global Leai-niilg Coi~sortiurn.i .'- > --,- , < , I ,, - * < pt , .
.
-L -; ; Gel~eral gir~delir.res.for. Inclztstve Onlrne Cliltz~ral Contelir.
--
Ca11adiai-i Netwosk for Ii~clusi~ie Cultura1 E ~ c h a i ~ g e .
*('i9 % i I - i I
texts to sign. One exaiiiple is "Sigi~speak" (see also the pro-ject "Dicta sign")
'
3.
SL eomwnhanèeatio~a and instiirretitonal mateiiials: what
modegsof Sk to adopt?
Irrespective of wliether real sigilers or signing avatars are used, one additional limitatio~~ of inaiiy curreilt efforts towards integratiilg SL materials into e-lear~iing platforms concerns a failiire to recognize iinportant differences between SL arid vocaliwritten languages. and the probleins posed by the dramatically iiisufficient refereilce tools, and overall liilguistic descriptioils, that are cui-i-eiltly available for SL.
It must first be recalled that al/ SL are languages
~~'itliozit a -ir)ritfer7 tracfition. More importantly fsom a research standpoii1t. and even thougl~ almost 50 years have passed since tlie i~lodern study of SL has begun.
researcl-iers still have not found a11 agreement on: (a) what are the coiistituei-it elernents of SL; ( b ) what graphic systenis cail be used for representing SL in written form aiid. o11 this basis, develop appropriate i'eference tools (e.g. dictionaries, grammars, usage-based corpora etc) that are u~iquestio~lably Ilecessary for both the coinlilui1ities of sigiiers, and the exploitatioil of SL for educational aild instructional purposes (see Cuxac &
Antinoro Pizzuto, 2010; Garcia, 2006; 2010; Garcia Rr.
Deiyclte. 30 10).
It is not trivial to stress that, although oui.
ki1owledge of SL has considerably advanced, we still do not have aily moilolingual dictioilasy or gramrnar, for any of the SL that has been to date iilvestigated
-
not eveil for ASL.Iii this coiitext, one could expect that well-grounded proposals ain-ied at exploitiilg SL for iilstsuctional purposes would dedicate particular care in inal;iiig explicit the models of SL elements and discourse they adopt. This appears especially necessary becaiise. as recalled hereafter: there are at present two ina-jor ciasses of models for describing SL. 111 agreemenl with Ciisac dC Sallailcire (2007) we will refer to tllese il~odels as
"assiiniliatioi~ist" vs. "noil assin~iliationist": the first type of models highligl~t primarily the structural similarities betkveeii SL a11d VL, while the secoild oiles underscore that, in additi011 to important similarities there are equally relevant differeilces between SL aild VL.
Within the Iiinits of the preseiit context, w e iliiistrate some of the criicial differeiices behveeii these
&vo types of inodels in relati011 to the problein of defining what are the constituent elemei~ts of SL.
In substantial agreemeilt with early, very influential descriptiolis of ASL provided by Stokoe ( 1960) and subsequeiitly Iiiima & Bell~igi (1979), assiinilatioilist models assume that SL coi~stit~ients units are essel7tial[-i.
co117parnOle io T.2 1r~or.d~: and are yr-i~ilnr.ifi, seqz~er7tiall-i~
o~gnrzked i11 ti117e. These models al-e still largel>7 prevailiilg in curreilt research o11 SL and have beeil foi- the inost acritically adopted i11 educational applications
of differeilt types, including e-learning piatforms.
111 contrast, 11011 assiiniliationist models, based o11 extensive analyses of SL discoitrse, show tliat SL constituent elen~eitts cannot be easily assin~ilated to VL units. In addition to word-Iike eleinents. SL possess complex, liiglily iconic sh-uctures (HIS) that are sinlultaneously orgailized in a rnultilinear fashioii that has no parallel i11 VL. The differences between word-lilte aiid non-word-like units are rnarlted by no12 inailual aild manual ai-ticulators. rriost notably by rnodality-specific eye-gaze patteriis: wheri producing word-like uilits. the signer's gaze is directed towards the iilterlocutor, whereas when producing HIS the signer's gaze is typically directed away f o r ~ n the iilterlocutor.
Figure 1 below provides just two illustrative exainples of a word-iike unit ( l a ) and a noli-word-like HIS ( 1 b) that are coin~nonly found i11 SL discourse. The examples are talteli froln LIS discourse but a wealth of siinilar exaniples can be fouild in al1 SL (for relevant disc~issions, see especially Cuxac, 2000: Cuxac 5(:
Ailtinoro Pizzuto, 2010; Pizzuto, Pietrandrea & Simone, 2007; Garcia & Derycke; 201 0).
Figure 1 : Word-Iilte sign ( l a) and HIS ( 1 b)
T11e point we wish to stress here is the followii~g:
HIS are very fieclueilt in SL disco~~rse, rangiilg fioni 30%
to as much as 9090 (depeilding o11 discourse geisre) of the constituent elernents that cali be identified and parsed in SL discourse (Boutet. Sallaiidre & Fusellier-Souza, 20 10;
Cuxac & Ailtinoro Pizzuto, 2010; Sallandre, 2003; Di Reiizo & al, 2009). It should thus be evident that SL descriptioiis of aily sort, including modelisations via sigiiing avatars, caiinot disregard as "inargiiial" these structures that appear uilique of SL (see Cuxac & Dalle.
2007). E-Iearniilg materials based oli the assumption that SL eleinents are for the most "ust Iike VL words" thus exhibit severe Iimitations that need to be recognized.
critically discussed and, hopef~lli], amerided.
sorts to be "attended to" and processed, and other information stemmiiig from the interaction with other fellow studeilts a~idlor with a tutor (e.g. in excl~ailges ta1;inp place in actual classroo~ils or iil videocoiiferences).
Since deaf persons inust use their sight. and accordiiigly orient their visual attention, to process bot1-i itinds of iiiforinatioil, tile two tasks caiinot be carried out at the sarne tinie: DL cannot simultaneo~~sly look at teaching or explanatory materials displayed o11 the coinputer screen ar~d at Iiiiguistic, interaction-based iiiformation giveil on the saine inaterials which they mttst aIways decode primarily via visi011 (e.5. by lip-reading spolien utterances, processi11g a message i11 SL, reading subtitles).
Tliis is inuch u11lil;e what happens, in the sanie cooperative leariiii~g situatioii, for hearing learners wllo cali simultaileously process co~nmunicative inessages coilveyed through souiids and freely orient their visual atteiltioii to other types of iilforinatioil coining fioin tlle coinputer screeil. Devising ai1 appropriate e-learning environment for DL thus requires accurate analyses of the ways in which tliese learners use and distribute their visual attenti011 when performing differeilt leariiiiig taslts, and how this cali influente the leanling process.
5. Towsar-ds deaf-centered multilingua1 and multimoda1 e-iearrpiirag platforms
Figure 2 scliematically illustrates a mode1 for an e-learning platform prototype (ELPP) prototype we are currently developing wit1lii.i the fraiiie of a ilatioilal project which pursues two major, iilterrelated objectives:
(1) irnproving muItilingua1 I inulti~nodal e-Iearning enviro~iineiits for DL (High Scliool and Uiiiversity students): ( 2 ) proinoting their Iiteracy skills ".
ITALIAN i.lritt?n and SPOKEn
. . t
4 -
DEAF
LEARldiQS -+
NEEDS +.---- ( ~ 1 5 ~ 1 and
Itaiian-Li)
- 1 -
--
P DEAF LCARNCRS
VISUAL ATTENTI014 PATTERNS li4
HCI
EEAF-CEIITERED E-LEARFIING ENVIRONMEIIT
PROTOTYPE PLATFORM -- i L---
l
v-
DEAF WORLD VIEVd
- L
MULTINEDIA TECHWOLOGIES
Ai1 appropriate e-Ieariiing ei~vironinent for DL at Iarge, Figure 3 : A mode1 of deaf-centered e-learilinp platform i.e. for both sigilers arid non-sigiiers. inust talte in due
account a coilstraiiit that can be easily observed aild yet. The ELLP model illusti-ated i11 Figure 2 aims at to our Iinowledge, has ilot beeil carefully investigated in
previorts research. Wheil working with a computer, the 6
The prqject iil\~olves five research teains providing iiiter- arid
~ i s ~ l a ' atteiition Patteriis Proper of DL "lariiedi~- differ ~alls-discip]iilayy co1ilpeteiices across tiic fie]ds ofi -CL 5oni those observable i11 hearixig Iearners. This is n-ue liriguistics; -special and bilingual educati011 for DL;
especially ill situatioll of cooperative learning wllere the -11iultimedia tools foi- DL aild hearing learners:
-11uinali-coiilputei- interaction a i d visual leanling i11 e-Ieai-iling studeilrs silllu'taileous'y envisonmenrs: -foreigii laiip~iage teachiiip rneliiodolopies in iilformatioil concerilir-ig writteil materials of different both traditional aiid e-leariiing en~~iroilrneilts.
overcoiniiig tlie Iimitatioils proper of many e-leariiing platforins (see discussion above) in different ways. Each of tlie iiiajor L'co~iceptual coiiipoiieiits" of tlie iiiodel is at the salile tirne rilotivutrd Z7j*, and necessn~y~fòr designing a deufcer7ter'ed e-leurnir~g yla<forril. The platforii~ is grouiided upoii the idea that research aiined at creating useful products for deaf itsers needs to be developed, fioni tlie very start, ivith deaf persons, not just,for, or or?
deaf people. Accordingly, and ratlier differeiitly fio111 what is reported for many past and ongoing pro-jects directed to deaf persoiis, this idea guides our actuaI 'project inaiiagemeiit' practice. The project-leader team includes six deaf colleagues who ynrricipule as yroiugor~ists i11 t l ~ e plalzrzilzg urzd ar'ticz~lation of tl7e erlrire resec1rc11 VI-oject, not only as "end users" or "eild evaluators" of the language resources and didactic tools to be produced or in1plemented. Al1 oiir deaf colleagues are highly proficieiit iii LIS: three leariied to sign in infaiicj~, witliiii deaf signiiig families, three ad different ages, as it liappens to iiiost deaf sigiiers (see Cuxac &
Antinoro Pizzuto, 201 0); they possess different degrees of lcnowledge of spolieii /writteii Italian wliich mirror in part their educational backgrou~id
'.
We give here just few practical exaiiiples of tlie cruciaf iiivolvenient of our deaf colIeagues. The choice of the "coiitents" we will focus on for developiilg tl-ie ELPP~, and of tlie different forms in which such contents will eveiitrially be presented to DL 011 the ELPP (e.g.
spoken aiid writteii texts, speech-to-text captions, SL traiislations and explanations, zraphic illustrations), was made following extensive discussions, arnoilg the deaf aild the hearing meinbers of tlie pro-iect, of different, alternative possibilities. Our deaf colleagues are coiinibuting to the preparati011 of ad-hoc questionnaires ai~d to a thorough examination aiid evaluation of language taslis, materials, multirnedia techiiologies we are using andlor are currently developing (iilcluding for ex. the ELPP iiiterface). In short, the active involvement of deaf colleagues ensmes that the eiid products of our project be. on one hand, consistent with tlie "dec!fir~orId i~ieiil", (see Figure 3 ) - i-e. a complex configuration of experiential and conceptuaI knowledge that is strongly grouiided in visi017 (see ainong others Lane, Hoffineister
& Bahail, 1996). and: oli the other haiid, effectively
respond to DL needs (see Figure 3).
Oile otlier irnportant elemeiit of the deaf-hearing collaboratioii we are prornoting within the pro-ject is the following: al1 the lieariiig members of the project-leader teain possess a good or advaiiced kilowledge of LIS; four of tlie five (hearing) young researchers of the otlies research teains involved in the pro-ject are currentiy atteiiding classes to iearn LIS. We are also seeliing Spohen/writteii 1aiiguai.e proficiency iii deaf liersons is liighl:
variable aild oilly partially Iiilked to t l ~ e educatioilal leve1 achieved. Ous deaf colleagues iilclride one doctoral studeilt. one college grad~iate. one Uiiiversitj studei~t. three higli school graduates.
For space limits \ve can only inention 11ei-e tlle 'geiieral coilteilts' o f tlie ELPP: we \vi11 focus on tlie histoi-)'. evolutioil aild ilse of writiilg. and compare orol/sigiied vs. graphic/\~~itteil Soril~s of coinmunicatioil.
furtlier collaboratioiis with deaf experts w l ~ o use Italian (rather tlian LIS) as their preferred language.
As iioted above, ~ n o s t e-leariiiiig platforiiis for DL appeas to be desig~~ed or~ljl~fbr sigr7ir7g DL. I11 coiitrast. as shown in Figure 2. our ELPP aims at addr.essir7g the needs o f botli sig11ir7g (LIS-LI) and 1lon sigr~irlg (Ifalian-LI) DL. In fact, as also iioted above, botli such groups of DL experielice dramatic diffjculties in literwq?
develoyr~~erzl. Our research aims at ascertainiiig tlie specific comniunicative-linguistic needs of each group of DL alid tlie extent to which tliese are, or are not comparable. We expect tliat the results of our investigatioiis will provide: ( a ) novel, relevant information oli the liiiguistic-cognitive profile of tlie two grorips of DL, clariQiiig also wliether, andlor how knowledge of LIS as L1 inay. or nlay not, interfere with tile acquisition and Lise of spokeii/written ItaIiaii; ( b ) important indications oli how \ve may iieed to differentiate tlie niultili~lgual aiid iiiultiinodal materials to be created fos promoting literacy developnient in DL with LIS-LI as compared to DL witl-i Italian-Ll. For exainple, recalling what iioted in sectioii 3, it worild be plausible to hypothesize that, for DL with LIS-L1, the siinuItaneous1y organized, inultilinear liiiguistic sb-uctures that are higl-ily specific of tlieir SL, naniely HIS, niay negatively interfere with the learniilg of more sequeiltially organized liiiguistic structures that are proper of writteil langiiage. It would be equally plausible to hypothesize that tliese potential negative iiiterferences sl~ould be abseiit in DL witli ItaIian-LI. However, tliese Ilypotheses can be evaluated oniy by coinparing the liiiguistic-cog~iitive profiles of tlle two groups of DL, as we plan to do in our project.
A substantial novelty of tlie multilingua1 / iiiultiiiiodal ELPP e-learniiig etiviroiiment we are designing coiicerns the use. presentati011 (hence. by the saine tolien. explicit modeling and representatioli) of tlie two n~ajor types of Iai7gz~nge resozlrces tliat will be employed for pedagogica1 purposes. iianiely: Italia11 aiid LIS. What is nove1 in our niodel is that. as illustrated in Figure 2. writteii texts \vi11 be provided not only in 11lritte17 Italia11 (the target language in which we aiin to proinote DL literacy developinent), but also in ii~ritfen LIS - a laitguage resource whicli: to our l~nowledge, has never beeii experiniented in e-leariiing platforins for DL.
Spolcer~ Iraliun aiid jucr-~o-fi~ce LIS (the latter in tlie foi.111 of digital videos) will also be used (sec Figure 2).
For the iiistructioiial ~naterials to be provided in
~i~r'itien Italian, guided easificatioil procedures will be used to facilitate DL's access to textual materials;
speech-to-text captioning tools will grant visual accessibility to materials given in spoken Ituliar~;
Iing~iistic accessibility to the contents and forilis of Italian-eiicoded instructional niaterials will be enhanced, for DL witl~ LIS-LI, via appropriate videos providing tra~lslations and explanations in (face-to-face) LIS. Due to space limits, no further details are given here on tiiese three types of Ianguage resources. which wili be iiiipleinented driving o11 a co~~solidated experielice in
bilingual educati011 for DL (Caselli & al, 2006). and more generally i11 language teaching inetIiodoIogies, as detailed in our grant proposal. We describe briefly the rationale. einpirical grounds, and major ainis of our nove1 experimentatiori of written LIS .
As ~ioted i11 section 3. al1 SL are at present without a written tradition. For DL with LIS-LI, the lack of a writteit form of their own SL inay well be one of the obstacles oli the road towards achievilig appropriate literacy skills i11 a language - lilie Italia11
-
that 11ot only does have a writteii tradition but is also typologically very different fì-om their own (see especially our remarks above on SL HIS). Recent research shows that Italian signers can profitably use Sign Writing (SW). a grapliic system proposed by Sutton (1999) for writiiig SL, for:-transcribirig LIS face-to-face productioils;
-
creating, for tlie first tiine in the history of this SL: texts conceived directly in written LIS (SI47 llas beeii adapted for tliese purposes to LISI. More importantly for the present discussioil, tllis research shows tllat, relyiiig on S W-encoded LIS texts, signers cail autonoinously perforin ineailiiigful con~parisons between LIS and spoken/written Italian, at al1 struckiral levels-
lexical, moi-pliological, syiitactic, textual, pragniatic.011 tllis basis, signers can foriiiulate metacognitive and metalinguistic reflections on the strutture of LIS as coinpared to spokeil/writtei~ Italian, and more generally ori the relations between "orality" or face-to-face vs.
written cornrnuilicatioil. i11 a way that has never been possible. for thein. without relying on a written representation of their SL (see ainong others Di Renzo &
al. 3006; 2009; Gianfreda & al, 2009; Pizzuto & al 3006;
Aiitinoro Pizzuto & al, 2008). Takiiig in due account the crucial role t11 at metacognitive and metalinguistic skilIs notoriously play in the deveiopment of literacy sliills, these research findings have motivated us to further experiment written LIS, on our ELPP, as a potentially very p o w e r f ~ ~ l pedagogica1 tool for proinoting literacy abilities. S\V-encoded: writteil representatioils of LIS have also proven t e be extremely useful for advancing i11 tlle lii~guistic analysis of tlle language (,41iti11oro Pizzuto BI al. 2008), paving the way for more appropriate modelisations whicli niay be i ~ s e d for botli genera1 descriptive purposes, and for iinplementing the use of LIS as a linguistic resource on e-learning platforms.
We iloted in section 4 that DL 's viszral aitentioii, patter-ns i17 HCI inay significantiy differ from tllose of hearing leariiers. One other additional novelty of our project conceriis the use of eye-tracking equipineilt for ailalyzing DL's visual attentiori patterns, and compare them with tl~ose of hearing learners', duriilg Iearniilg taslts which dernaild tlie sim~iltai~eous processing of Ianguage resources along wit1-1 visual infonnation of different sol-ts. Prelirniilary results of a pilot study we have conducted indicate that. in processing rilultimodal / niultilanguage materials, the gaze patterns of DL with LI S-L 1 inarliedly di ffer fiotn those of hearing learners (Capuano. Levialdi & Antinoro Pizzuto. subinitted). We trust tllat the inore exteilsive investigations 011 this topic
we plan to develop witliin our project will provide us much n eeded, nove1 information for a better understaiiding of how visual ii~forination needs to be spatially and ternporally structured in e-leaniing eiivironments for DL, as compared to hearing users.
These analyses will also alIow to us ascertain whether there are (or not) relevant differences between sigiiing VS.
non-signiiig deaf students, when these DL with different language bacliground access and rise visually grounded infor~nation, of both linguistic and non-lingiiistic type.
Finally, recalling the crucial importailce of vision in the 'deaf world view', we think that web-based ri~ztltiri~edin 1echr.io1ogies and learning tools for a deaf-centered ELPP may be sigilificantly improved implelnenting a visually-based graphic interface.
Drawiiig on ongoing research 011 tlie topic (Capuano & al, submitted), we aini at designing an iilterface that DL can access and use easily a i ~ d 'ii~tuitively' because textual inforination (which is difficult for thenl) is significantly reduced, or even entirely replaced by rnostly non-textual ( icoilic) iiiforniation. This entails the need of creating a new, graphic way for browsing web pages, and interacting with the ELPP.
For the natural, deaf-peculiar visual wlay of grasping information to be exploited in our pIatfori11, we are going to use a new interactioil paradigin based on t l ~ e theories of embodied cogniiio\~ arid sfor~~zelli~7g (Lakoff
& Johiisoil, 1980; Johiison. 1987; I ~ n a z 8ci Benyon. 3007).
Wit1iii-i this paradigm, tlie learning process can be n~etapl~orically represented as a story that iacludes the Liser as tile rnain character. Accordingly, the user 'lives' the learning process by pliysically experiencing it - in the vii-tua1 space of the ELPP - as a path with a starting place, a sequence of severa1 Iear~iing steps, and a final goai. Such a metaphor seerns to be a very intuitive way of representing the learniiig environn~ent. h4oreover, it seelns to be ari adequate iiiteraction paradigin especially for deaf users, since it exploits the visual cl-iailiiel as the
~nain source of iilfornlation.
The preparation of this work was partially funded by -the Italian Ministry of Educati011 aild Research (MIUR-FIRB), Project E-learxirig, Deafness nnd TPi.itte17 Langunge / TrISEL -RBNE074TSL (3009-20 l?), see:
l : / / . i .C Ii ; -The Association Progelfi Felicità, Pro-ject "Writiilg L1S and SignWriting" Rome, (2005 -1. JVe thanlì our colieagues Stefano Levialdi, Marilena De Marsico, Ailila Labella and Alessio Di Renzo for helpfiil coinrnents.
7 . References
Antinoro Pizzuto, E., Chiari. I . & Rossini, P. (3008). The represeiltation issue and its inultifaceted aspects in constructing sign language corpora: questions.
answers, furtiier problems. P~.oceedir?gs qf the 3rd Worlishop 071 1l7e Repr~esenic~tior ar~d Pr~ocessing qf
Boutet. D., Sallandre, M.-A.. & Fusellier-Souza, I.
(3010). Gest~ialité huinaine et langues des signes:
entre continuuiil et variations. 111 B. Garcia et M.
Derycke (eds.), Sozfrds e1 lat7gzle n%s sigues. Aior.117e et i-aiaiufions, Langage et Sociéié. no 13 1, mars 201 0, 55-74.
Cap~iano. D.. Levialdi, S. & Aiitinoro Pizziito, E.
(submitted). Embodied vis~ial learning and deafiless: a concept paper.
Caselli, M.C., h~laragna. S. & Volterra, V. (2006).
Lingztaggio e sor.dità. Bologna: I1 blu lino,
Cusac. C. (2000). La Lai7gzte des Signes Fra~quise: /es Tbies de I'icorqicité, Faits de Langues i1°15-16, Paris:
Op111ys.
Cuxac. C. & A~ltinoro Pizzuto, E. (3010), Emergeilce, norme et variati011 dans les langues des signes : vers une redéfinitioil ~~otionilelle. In B. Garcia & M.
Derycke (eds.), Sozlr-ds e1 lcrngzie des signes. Norr71e e / i~ar.iafioi7s, Lungage et Sociéié, n" 13 1 , inars 30 1 0.
37-53.
Cuxac, C., Dalle, P. (2007). Problérnatique des chercheurs e11 traitement aiitomatique des laiigues des sigiies. 111: Tr.aifenzeizf ,-luioiitutiqzle des Lai7gztes, (-4 T4 L-4) - Traiien1e17t aztloma fiqzte des langzres des
sigizes, Vol. 48, N . 3, 15-30
- 4
/ s f 3 , b > t"s:~-Lt&2-:Y;"-' \. L;),. ;Lr -. .- l - iF\ z -2-L
Cuxac. C. et Sallandre, M-A. (3007). Iconicity and arbitrariness in French Sign Language: Izighiy iconic sti-uctures, degenerated iconicity and diagraii~niatic iconicity. in E. Pizzuto, P. Pietrandrea & R. Simone
( eds), T éi+bul and Sig~zed Langzlages, C0117pur.i11g Stl.~tcfztres. Constrztcts nr7d Adethodologies, Mouton d e Gruyter, Berlin-New I'ork, 13-33.
Di Renzo, A., Gianfreda, G., Lamano, L., Lucioli' T., Igeilnacclii~ B.. Rossini, P.. Bianchiiii C.S., Petitta G..
Antinoro Pizzuto- E., (2009). Representatioil - Ai~aly sis
-
Representatioil: nove1 approacl~es to the study of face-to-face and written iiarratives in Italia11 Sign Language (LIS). Papes Presented at the CILS Interr7aiional Conferente on Sig17 Langziages, Nainur, Belgium. November 16-20,3009.Di Renzo. A., Lamano, L-, Lucioli, T., Pennacchi. B., Porizo, L.. (3006). Italia11 Sign Lang~iage: Can we write it and transcribe it witl-i Sign Writing ? 111 ELRA (eds.). LREC 2006, TiTlor.kshop Pr.occzedin,ar (W--15):
Second l$brlishop o17 fhe Repr-esenfation and Processing of Sigt7 Langurges
.
1 1-
1 6.Drigas, A.S. & I<ourerneilos, D. (2005). Al1 e-Leanling Managen~eilt System foi- the Deaf people. Proceedings q f JTrSEE+lS Tr~ansacfions or7 Adsnnces i17 Ei7gineer.ii7g En'ztcation, VOI I, 2.20-24.
Efthimiou. E. & Fotinea. S. (2007). Ai1 Envirotlment for Deaf Accessibilit). to Educational Content.
Proceedings of the Firasi Irifer*nationnl Co~?fei.ence on i/?for.n1atio17 a17d Co117r1izinicatior7 Tecl717olog~~ cz17u'
Accessibili~q,
Efthimiou, E. & Fotinea, S. (2008), Tools for Deaf Accessibility t0 ai1 eGOV Environinent, in Lec1zli.e Aiores in C0177pziter- Sciei7ce (LNC'S). Vol. 5105, 446-452.
Garcia. B. (2006). Tlie ~~~ethodological. linguistic and sen~iological bases for t l ~ e elaboration of a written fori11 of LSF (French Sign Language). In ELRA (eds.), LREC 2006 - JI.or-liLshop P~.oceedings ( /V-1.51, Second
TdurI1~170p 017 tl7e Repl'esentafio~ a17d Processing of
Sigu Languages, 3 1-36.
Garcia. B . (20 10). Sott/.ds, szir.dilé, lungz~e(s) des signes ei épisiémologie des scieilces d z ~ languge.
Probléi17aiiqzies de la scr-ir)tzrr-isniioi7 e; n ~ o d é l i s n f i o ~ ~ des bas nii?eazcs en Langile des S i g ~ e s Frar7quise (LSF). Mémoire d'Habilitation à Diriger les Recherches, Université Paris 8-Saint-Denis.
Garcia, B. & Deryclie, M. 12010). Introductioil. In B.
Garcia & M. Deryclie (eds.), Sozlrds e f lnngzre des signes. IVoriiie e1 1-nr-iatio17s. Lai7gage e1 Sociéti, no
131, nlars 2010, 5-17,
Garcia. B. et Perini. M. (2010). Normes en jeu et jeu des nonnes dans les deux Iangues en préseiice chez les sourds locuteurs de la Langue des Signes Francaise (LSF). 111 B. Garcia et M. Deryclie (eds), Sozirds et la17gz~c des sig17es. A;or.li7e e1 ~ai*iafior7s, Languge e f Société, 11@ 131, mars 201 0. 75-94.
Gianfi-eda, G., Petitta. G., Biancl~ini, C.S., Di Renzo A., Rossini. P., Lucioli, T., Peiinacchi, B., Lamailo. L.
(3009). Dalla 1nodatit6 faccia-a-faccia a una liilgua scritta emergente: nuove prospettive su trascrizione e scrittura deila Lingua dei Segni Italiana (LIS). In C.
Consani, C. Furiassi, F. Guazzella & C. Pel-ta (eds.), J t f i del 9" Cor~gresso dell2ssociazione Italiana di Lil2gziislica ,4ppiicnta - 01-alitiì I' Sc1.ittlrru. I17 nzemo~~ia di Giorgio Rainzo~clo Cardoi7n. Perugia: Guerra Edizioni. 41 3-437.
Iinaz, h4. & Benyon. D (2007). Desigr7ii1g ~rlith blends.
Tlie MIT Press.
ICarpouzis, K., Caridaltis, G.. Fotinea, S.-E.. & Efthimiou, E. (2007), Educational resources and implenientation of a Greek sign language synthesis arcl-iitecture.
Conrpzlters R- Edilcatiorz, 49, 54-74.
KIima E.S. et Beliugi, U. (1979). The S i p s ofLungzrngc.
Cambridge, M A : Harvard Uiiiversity Pi-ess.
.Tohnsoil. M. (1987). Tl7e ho&' i17 the nzir7d. University of Cliicago Press.
Lakoff, G. & Joliiisoil, M. (1980). ibfetaphor. Tlé Live BJ:
University of Chicago Press.
Lane, H., Hoffnleister, R.J. & Bahan, B. (1996). A journey into the Deaf-World. Sa11 Diego, CA:
DawnSign Press.
Pizzuto, E.. Pietrandrea, P., & Simot~e, R. Introdiiction.
In E. Pizzuto, P. Pietraildrea & R. Simolle (eds.) (3007), J/érbal n17d Signed Langrrages - Conyarir7g strnctirres, coi7s~rrcfs ar7d n~ethodologies, Berliil / New York: Mouton De Gruyter, 1-10.
Pizzuto, E. Rossini, P. aC Russo. T. 12006). Representing signed languages i11 written form: questions that need to be posed. 111 ELRA (eds.). LREC 2006 - T.Ior-lsl7op Proceedir~gs ((11-1.7): Second Worksl7op on 1ke Representafion a17d Pr-ocessing of Sigr? Latlguages,
1-6.
Sallandre, M.-A. (3003). Les Z I M ~ T C S du discozliire efi
Langzre des Sig17es Fr.ar7qaise (LSF) - Tenfative de de cntegorizatio~7 duns le cadi-e d'ztne gra177171air.e de l'iconiciié. Thèse de Doctorat en Sciences du Langage, Paris, Université Paris 8.
Stolioe. W.C. (1960) Sign Lang~iage Strutture. in Sfztdies in Lingztisfics - Occasiouial Paper- n. 8, 1960 (rev. ed.
Linstoli Press, Silver Spring. MD, 1978).
Sutton. V. (1 999). Lessons in Sign JT?iting. Textbooli &
JTor-lcbook. La Jolla, CA: Deaf Actioil Comtnittee for Sign Witing (2nd editioil, 1st edition 1995).
