Image & Narrative, Vol 11, No 1 (2010) 118
Multimodality and its Modes in Novelizations
Heidi Peeters
Abstract (E)
This essay presents a dissection of the concept of multimodality, both at the level of the text and at the level of the experience this texts provides. In this way it demonstrates how the concept offers both a complex and extremely useful alternative to the notion of multimediality. Multimodality, transcending the material boundaries of the text, thus is not merely a feature of new or digital media, but through its functioning in the realms of cognition and emotion, emerges in older texts as well, especially if these are part of a network in which newer media function as well. This exploration of multimodality is able to shed light on seemingly outdated “monomodal” cultural practices. The cultural practice under scrutiny will be the novelization, the novel that is based on a film, with special attention for the role of illustrations.
Abstract (F)
Cet essai présente une analyse du concept de multimodalité, au niveau d’ l’objet médiatique comme à celui de l’expérience du média. Il démontre que ce concept offre une alternative complexe, mais extrêmement utile, à la notion de la multimédialité. Comme la multimodalité dépasse les frontières matérielles du texte et fonctionne aussi au niveau cognitif et émotionnel, elle n’est pas limitée au medias numériques, mais se manifeste aussi dans des médias plus anciens, particulièrement quand ceux-ci sont intégrés dans un réseau de media plus récents. Cette exploration de la multimodalité propose donc aussi un retour sur des pratiques culturelles en apparence datées, comme la novellisation, soit le roman rédigé à partir d’un film. Cette pratique culturelle sera examinée ici plus en détail, avec une attention particulière pour les illustrations de ce genre de romans.
Keywords
Image & Narrative, Vol 11, No 1 (2010) 119 Multimodality is a term that has recently been coined within the realms of media studies and narratology to account for textual forms in which the borders between traditional media or between different sensorial channels are transcended.1 On the one hand, at the level of the text, it designates a tendency towards the integration of a variety of semiotic systems (verbal, visual, kinaesthetic), on the other hand, at the level of the subject experiencing these texts, it designates the simultaneous engagement of different senses (seeing, hearing, touching, etc.). Even though the notion of multimodality might be rather new and largely unknown, it seems to denote one of the primordial qualities of postmodern multimedial texts, while in accounting for sensorial experiences, it provides insightful alternatives to this notion of multimediality. Furthermore, through its focus on the effects of texts on the public, the concept of multimodality might illuminate the function of older media that are seemingly at odds with the current move towards multimediality.
In order to investigate the phenomenon of multimodality, while stressing its divergence from the concept of multimediality, I propose to focus on exactly such a “monomedial” cultural form, namely the novelization; the novel or the literary text that is based on a film. Although the form of the novelization has existed almost as long as the medium film itself, it is especially within the current horizontally integrated media environment, that it is able to both challenge and illuminate the notion of multimodality.
Over the last century, many films, especially the big-budget productions of the New Hollywood from the 1980’s onwards, have resulted in the production of novelizations, always integrated within the marketing strategies surrounding the release of a film. In order to clarify the novelization’s relation to the new-found concept of multimodality, the novelization of The
Sixth Sense might serve as a prototypical example of the phenomenon as it circulates within
today’s society. The original 1999 film, M. Night Shyamalan’s third, was both a commercial and a critical success, being nominated for six Oscars, maintaining its place in the top 50 worldwide box-office list and providing its director with instant fame among contemporary audiences.2 The novelization is presented as a small paperback novel consisting of 170 largely
1Eija Ventola, Cassily Charles & Martin Kaltenbacher (eds., 2008) Perspectives on Multimodality. John
Benjamins, Amsterdam; Gunter Kress & Theo van Leeuwen (2001) Multimodal Discourse: The Modes and
Media of Contemporary Communication, London: Arnold. Marie-Laure Ryan (ed.) (2004) Narrative across Media: The Languages of Storytelling, Lincoln: University of Nebraska Press.
2 The Sixth Sense was listed n° 39 in the international box office list of the imdb-site in January 2010, it was
listed n° 131 in its list of viewer ratings, while the Rotten Tomatoes site at the same time indicated 85 percent of the selected critics to have been enthusiastic about the movie. Among the six academy awards for which the film
Image & Narrative, Vol 11, No 1 (2010) 120 fonted pages and the front cover, like the majority of novelizations, features one of the film posters, in this case a blurry six-shaped figure behind the young protagonist walking towards the lens. Furthermore, like most novelizations, this volume is explicitly presented as such (“a novelization”, “based on the film”), the cover provides an extensive list of film credits (director, studio, etc.) and the film’s tagline (in this case the already legendary quote “I see dead people”) is used to lure potential readers into buying the book. The back cover displays a film still of the two protagonists, a short description of the story, further film credits and an advertisement for more novelistic Sixth Sense tie-in products, while an interview with the director is inserted at the end of the novel. The middle of the volume contains eight pages of film stills, accompanied by descriptive captions and dialogue lines, broadly following the film’s chronological development, while the book’s length seems calculated for a reading duration approximating the time of a cinematic viewing session. The above description of the novelization of The Sixth Sense learns two things about novelizations in general. First of all, it learns that novelizations, like most novels, apart from the illustration on the cover or a few film stills that might be inserted within the volume, mainly use verbal modes and exclusively seem to engage the sense needed for reading, namely vision. Secondly, it teaches that through all sorts of peritextual markers, the novelization is explicitly linked to the film on which it is based and is hence not programmed to be read in isolation from this film. Even though both observations seem obvious and non-contradictory, the first presents the novelization as a monomodal medium, while the second opens paths for multimodality, as will be developed further on.
The Rise of the Image, the Fall of the Word
The above-mentioned phrase is the title of an insightful volume by Mitchell Stephens, in which he analyses the historical evolution of verbal and visual media and urges the humanities to accept the visual “mode” as the emerging dominant form of expression.3 Rather than fighting the evolution towards ever more visuality, literary departments, according to Stephens, should embrace, analyze and teach the literacy of the web - where moving and still images are combined with other sign systems - or the literacy of film - an inherently multimodal medium, combining moving images, sound, music, and digital animations with was nominated in 2000, were the categories Best Picture; Best Director; Best Screenplay Written Directly for the Screen and Best Editing.
Image & Narrative, Vol 11, No 1 (2010) 121 plain text in subtitles or in the credits. Without using the term, rather than defending visual literacy, Stephens defends multimodal literacy with a visual dominant.
The rise of such an imagistic multimodality obviously echoes the teachings of Jay David Bolter and Richard Grusin, as presented in their Remediation theory.4 Multimodality with the visual mode as its privileged component does mimic the mediatic evolution towards ever more hypermediacy, a concept that designates the exponential expansion of mediation in texts, with different sign systems and hence different modes being combined. Seemingly contradictorily to the above mentioned quality, transparency and immediacy as used by these authors indicate the capacity of a medium to “disappear” and to generate the impression that the presented material is not mediated at all, but plainly reality as such. Strangely enough, these last qualities can also be attained through multimodality, since in order to epistemologically create a mediated reality close enough to reality as such, all or at least most senses of the beholder should be effectively stimulated. The multimodal turn hence turns out to be a logical step in a teleological evolution towards ever more “realistic” texts.
If Stephens, Bolter and Grusin are to be taken seriously, texts the likes of novelizations appear to be at odds with the current development towards multimodal vizuality and immediate hypermediacy, as the right teleological path would lead to the adaptation of novels into movies, DVD’s or television series and to the supersession of written texts by hypermedial conceptual websites or videogames. Within the logic of Bolter and Grusin, novelizations could be explained away and neutralized as literature’s repurposing in a filmic environment, as literature’s desperate and somewhat parasitical attempt to survive in a multimodal landscape.
In this respect, the novelization of The Sixth Sense could be conceptualized as one of literature’s attempts to participate half-heartedly in the move towards multimodality, integrating filmic posters and film stills into its peritext and thus achieving a second-degree multimodality, in the margins of the film phenomenon.
Multimodality and its Modes
4
Jay David Bolter & Richard Grusin (1999) Remediation: Understanding New Media, The MIT Press, Cambridge, MA.
Image & Narrative, Vol 11, No 1 (2010) 122 The macro-structural field of media turns out to be far less well-organized than teleological theories would have it. While the mediatic landscape of the past might be archaeologically traceable and can be structured through neat diachronic schemas, as in Bolter and Grusin’s and Stevens’s case, a clear overview and analysis of the synchronic mediatic reality is more problematic. The current media field is highly hybrid, complicating neat divisions between multimodal and non-multimodal texts and obstructing neat allocations of particular texts to particular media, as a number of texts dangle in the spheres of intermediality or even transmediality. A webpage, containing verbal texts, images and video commercials, could be argued to either belong to only one medium - the internet - and hence to be monomedial, to be an intermedial phenomenon in its combination of the above mentioned media, or even to hover over these media and to become transmedial. The digital paradigm does indeed enable combinations of diverse cultural practices and experiments with hybridity, making multimediality a highly flexible concept.
While analyses at the level of media become more confusing due to the instability and inherent hybridity of today’s mediatic field, it makes sense to focus on the level below the medium, namely on the level of the text and on the modes that are exhibited within the text. Without having to come up with bold statements about macromediatic evolutions and the institutionalization of certain practices, one can simply concentrate on the text itself and analyze it as such. Unfortunately, modality is not as easy a concept as it may seem and apparently, the text cannot be isolated from the mediatic network in which it functions after all.
The crux of the concept of multimodality is that it is Janus-faced, with two intertwined dimensions. On the one hand, at the level of materiality, of the signs, it designates the different semiotic systems that are being used within a particular text. On the other hand, it designates immaterial effects, dimensions manifesting themselves at the level of reception and perception, at the level of the text’s sensorial effects on the public. Hence, on the one hand, multimodality contains the modes of the text “an sich”, on the other hand, it entails the modes in which that text is being perceived by its users. A multimodal text thus provides a multimodal experience and even gains its multimodal status from it, while the multimodal experience in its turn is determined by the materiality of the text. Multimodality consequently emerges from the interaction between the semiotic and the sensorial realm.
Image & Narrative, Vol 11, No 1 (2010) 123 Since multimodality can manifest itself at two different levels (semiotic and sensorial), this may, according to the rules of combinatory logic, result in a variety of forms. A multitude of sign systems, combined in one text can generate a multitude of sensations in the perceiver, but these sign systems may as well, perhaps secondary to a first perception, be grasped as one unity. In the first case, we face what I would call dissonant multimodality, as happens in web-pages, where pictures, texts, pop-ups and the irritating noises of free smileys are fighting for the surfer’s attention. This multimodality is dissonant in the sense that the different modes do not add up to one unified experience. In the second case, however, the different sign systems function harmoniously, adding up to one textual gestalt, a cognitive unity or reality, integrating the different modal parts. This mode of modality could be coined integrative multimodality. The perfect example would be a filmic text, where edited images, dialogued voices, sounds, music and subtitles are attuned to such a degree that the viewer does not notice them as separate entities anymore, but rather sees them as part of the reality of the text. It should be noted that the difference between dissonant multimodality and integrative multimodality is only absolute on a theoretical level and that the contingencies of reception can make texts potentially flip back and forth between both poles, as if they were Escher drawings. It should be noted as well that this dyad of modality in texts runs symmetrical with Bolter and Grusin’s division between hypermediacy and immediacy at the level of media. It is a cheap trick, but it is effective.
Nevertheless, the possible modes of multimodality are not exhausted yet. It is conceivable that a single sign system could trigger a multitude of sensorial modes in the receiver. There are, again, multiple ways in which this effect could be achieved. Broadly, we can distinguish between three categories: cognitive multimodality, transpositive multimodality and
intermodality. First of all, a sign system could trigger cognitive multimodality, which means
that the multimodality would be solely constructed within the perceiver’s mind. A printed, literary text, for example, could be defended to speak to all senses. This form of multimodality would on the one hand be inherent to the text, but on the other hand, it would also be highly dependent on the reader’s imagination and tendency to visualize - or rather sensorialize. Kafka’s novels are said to be highly visual, while a novel such as Patrick Süskind’s Das Parfum might be called extremely olfactory. Nevertheless, the way in which texts will be sensorialized, is to a large degree up to the reader. Thus, even though all literary texts are potentially multimodal on a cognitive level, the fluctuations in the actual realization of this form might be too big for it for it to be accepted among the other, more stable and
Image & Narrative, Vol 11, No 1 (2010) 124 materially based forms. A somewhat more stable mode of multimodality would be
transpositive multimodality, in which the textual mode would be an actual transformation of
another mode and hence cognitively evoke the earlier sign systems along with the current one. A cubist painting, for example, though materially static, should entail an evocation of movement. A last form of modality arises in the case of Millais Ophelia, which is based on Shakespeare’s play, so that the Shakespearean text shines through the brush strokes. A film likewise may evoke a novel from which it is adapted, along with the sign systems and sensory experiences used within and produced by the novel. In the case of the cubist painting, multimodality is structurally inscribed within the materiality of the perceived text, in the other two cases, multimodality only emerges from the connection between different texts and hence requires secondary knowledge from the perceiver (even though one might argue that secondary knowledge is also needed to some degree in the onlooker of the cubist painting). Millais’ painting and the film adaptation hence belong to a final mode of multimodality, coined intermodality, where a perceived text is connected to a cluster of different interwoven texts, which it in this way may evoke. A further and more elaborate example would be the “text” of the birth of Christ, a “happening” that is not tied to one single material carrier or cultural practice, but to a multitude of paintings, woodcarvings, biblical texts, stained-glasses, poems, stories, plays, children’s drawings, nativity sets, illustrated books and Feast of Epiphany cakes. One of these texts will evoke a set of others, based on the material network present within culture, but also on the sensorial, emotional and cognitive effects theses texts have on the public.
Cunning minds might note that not only the mode of cognitive multimodality, but the other two modes as well depend on the whims of the viewers, viewers who might possibly not “see” the movement in a cubist painting, just like they might not read Shakespeare through Millais. This could be so, but that does not render the texts themselves non-multimodal. The receivers in these cases are culturally blind to the multimodality, just like someone who has never seen a film will not be able to perceive the integrative multimodality in it at first. The last three modes nevertheless do depend strongly on the knowledge and aptitude of the viewer.
Novelizations and Illustrations
Novelizations, which are often conceived as the opposite of the more widely known filmic adaptations of novels, provide interesting cases of multimodality. Though they appear to be
Image & Narrative, Vol 11, No 1 (2010) 125 phenomena of little cultural significance, in the margins of the filmic industry, the practice is flourishing - many Hollywood films are still being adapted to the novelistic format- and far more interesting than most people would believe it to be. Novelizations function as merchandising items for the films they adapt, but at the same time they are commercially successful, proving there to be a public that buys and actually reads these novels. Novelizations have a reputation for being poor verbal transpositions of the much more multimodal film, a rich kinetic audio-visual construction out of which the narrative component and the dialogues are being distilled and rendered verbally on the page. This current, as mentioned above, seems counter-natural in the mediatic teleology, since film has convincingly been defended as the more multimodal and more up-to-date medium, while the monomodal texts of literature are supposed to be on the brink of extinction. As if to make up for a lack of multimodality, the movie poster is printed on the cover and film stills are being inserted.
Nevertheless, novelizations are way more multimodal than would appear at first sight, at least if we are willing to include the other modes of multimodality, transcending the mere textual ones. In this respect, illustrations prove more than merely extra-modal side-dishes to the actual text, but turn out to be gates through which different multimodality types are able to infiltrate in the text and in the reader’s mind. The movie poster on the cover and the inserted illustrations are part of what Gerard Genette calls “the paratextual” of the actual literary text, meaning that which “surrounds” it, both literally and figuratively, that which is a part of it, but only marginally belongs to it.5 Such a marginal role for the cover illustration, as well as for the inserted ones, however, does not do justice to their importance. More than a mere “cover” in the sense that it offers protection, it might be the incentive for the reader’s interest in the first place. An attractive cover tempts the potential reader into becoming an actual reader, all the more so in the case of the novelization, where it is actually the filmic connection, recognizable through the cover illustration and through obvious slogans, that cajoles the potential reader into picking up the book. Before having read the novel, the cover rather than the text itself shapes the novelistic experience to the potential reader. Illustrations, in such a first acquaintance with the novelization (and for that matter with any novel), stand in a relation of dissonant multimodality to the text, in proximity to it, but not adding up to one unified experience.
5
Gerard Genette (1997) Paratexts: Thresholds of Interpretation, Cambridge University Press, Cambridge (first published in French in 1987).
Image & Narrative, Vol 11, No 1 (2010) 126 In a second phase of acquaintance, a new sort of multimodality can emerge, a multimodality still in tune with the way regular illustrated novels would function. During the reading process, inserted illustrations will appear to be “transpositions” of the verbal text: passages rendered photographically as if to visualize them to the reader. This of course is a form of transpositive multimodality; while looking at the images, the verbal passages encountered in the text are being evoked and colored. The cover-illustration functions in a similar way, condensing the verbal story into a visual form. This form of multimodality, however, works both ways, since after having encountered the illustrations and the cover picture, the reader will, during the reading process, project them over (or under) the text, so that the mental images that the text evokes will also be based upon the illustrations. Through this bidirectional multimodal transposition, the visual will gradually infiltrate into the verbal experience and visa versa, both adding up to a form of integrative multimodality. This aspect materializes the form of cognitive multimodality that is inherent in all literary texts, but turns it into something less fluid and ephemeral, as the inserted illustrations will indeed unify the mental images evoked by the reading process for all readers. This mental process of unification is exactly the reason why Gustave Flaubert detested illustrations, as he interpreted unified and universalized cognitive multimodality as a nuisance to the independent imagination:
“A woman drawn resembles one woman, that’s all. The idea, from then on, is closed, completed, and all the sentences are useless.”6
The images, however, may be mentally animated during the reading process, while the text itself will be colored by them. Of course, illustrations do not wipe out personalized interpretations or visualizations of the text, but still they unify the multimodal experience generated by the text in comparison to non-illustrated novels.
All the above-mentioned aspects of multimodality need not be specific to novelizations, but could simply apply to all novels with illustrations. It is true that novelizations most often do have highly visual cover illustrations and they frequently contain inserted illustrations, but in
6 Flaubert in a letter to Ernest Duplan (Gustave Flaubert (1964) Oeuvres complètes. 2 volumes, Seuil, Paris),
quoted by Ricardou in Problèmes, p. 79 and paraphrased in English by Wolfgang Iser in The Implied Reader, Baltimore 1974, p 283.
Image & Narrative, Vol 11, No 1 (2010) 127 comparison with other novels, this only constitutes a difference in degree, not in kind. There is however a fundamental aspect to novelizations that separates their modality from all other novels and even sheds a new light on the above-mentioned forms of multimodality. This form, of course, is intermodality. Novelizations, even though they function independently to some degree, are part of an intertextual filmic network, not only on a marketing level, but also on the level of the text itself and most importantly on the level of the reader’s experience. This has far-fetching repercussions for the status of the illustrations and for the way in which the texts are being read. The filmic text to which novelizations refer is not so much a “pretext”, both figuratively in an economic sense and literally in a medial one, as it is a “supertext”, hovering above and behind the words. The interaction with the filmic supertext is essential to novelizations, as can be gathered from the way in which they are marketed (“based on the screenplay by”, “now a major motion picture”). It is impossible for the reader to ignore the hyperbolic link with the film and readers that buy and read these novels mainly do so, because they know and like the film the novel is based on. Either they will have seen the film, or they will be acquainted with it through other channels in the network (the trailer, reviews, previews) and will want to “see” the film through a different medium.
Novelizations promise to provide access to the universe of the multimodal film and to add something extra. In this respect, illustrations do not only stress the novel’s connection with the film, but they help the reader to access the film again. They are not random images or pictures, but either film stills - direct excerpts from the movie itself - or set photographs and actor portraits. The cover illustration, as mentioned above, most often is simply one of the movie posters, which must help the projection of the film over the text and the expansion of the film’s universe through the text. Illustrations are hence portholes into the filmic universes. The cover illustration refers to the film in its entirety, while the inserted illustrations guide the reader through the filmic text, while reading the literary text. The integrative multimodality the images provide in combination with the text is still a guided imaginary one, but also a remembered multimodality, grounded in the material multimodality of the film, or in the fragments of the film encountered in trailers, reviews and excerpt.7 Where the cognitive multimodality of traditional novels can become unified through illustrations, the cognitive multimodality of novelizations takes this unification one step further, as the exact
7 In this respect, see Victor Burgin’s take on filmic culture today, where film is no longer seen as a singular text,
but as a series of fragment, dispersed through culture. Victor Burgin (2006) “Possessive, Pensive and Possessed” in Stillness and Time: Photography and the Moving Image (eds. David Green and Joanna Lowry), Photoforum and Photoworks, Brighton.
Image & Narrative, Vol 11, No 1 (2010) 128 “sensorializations” (i.e. the visual and auditory representations) of the reading processes of novelizations already exist materially in the films themselves. Of course, there is no absolute stability in this respect. First of all, the movie is always a remembered one (or in some cases even a memory of a film seen through fragments from other media), and it is now known that the human brain is no reliable storage device, hence liable to blurring and distortion. Secondly, stability is impossible because the original film text in most cases simply does not exist anymore, but has become scattered over a series of different versions on different devices (the working version, the director’s cut, the version with an alternative ending). The novelization in this way is able to “replay” the remembered movie (distortedly), and even to expand it. The cover and the inserted illustrations function as triggers to visualize (among other modes) the original text again. The novelization and other tie-in media are hence not mere after-products, satellites of the original film text. All tie-in media participate in the filmic experience and in this way interact with it and with each other in a net of intertextual multimodality.
A New Multimodality
Though multimodality may seem an easy concept at first, it turns out to be far more complex. Rather than being a manageable quality at the level of the postmodern text, multimodality transcends its material boundaries and spreads over other and older texts and practices, flowing over into the cognitive and the cultural spheres. Multimodality is not merely a feature of new or digital media, but apparently emerges in older texts as well, texts like novels and novelizations. Nevertheless, in our time, the presence of multimodality seems to have increased hyperbolically with the growing importance of media networks. We can thus still not do away with the media concept for now, complex and confusing as this might be. Novelizations and their illustrations, though seemingly odd and anachronistic at first, do provide insightful examples of how multimodality can emerge in different, hidden and maybe unexpected ways. Against all odds, they prove to be true, if neglected, texts of our times.
Image & Narrative, Vol 11, No 1 (2010) 129
References
Victor BURGIN (2006) “Possessive, Pensive and Possessed” in Stillness and Time:
Photography and the Moving Image (eds. David Green and Joanna Lowry), Photoforum and
Photoworks, Brighton.
Jay David BOLTER & Richard GRUSIN (1999) Remediation: Understanding New Media, The MIT Press, Cambridge, MA.
Gerard GENETTE (1997 [1987]) Paratexts: Thresholds of Interpretation, Cambridge University Press, Cambridge.
Wolfgang ISER (1974) The Implied Reader, John Hopkins UP, Baltimore.
Friedrich KITTLER (1999 [1986]) Gramophone, Film, Typewriter, Stanford University Press, Stanford.
Gunter KRESS en Theo VAN LEEUWEN (2006) The Grammar of Visual Design, UCL Press, London.
Gunter KRESS & Theo VAN LEEUWEN (2001) Multimodal Discourse: The Modes and
Media of Contemporary Communication, Arnold, London.
Lev MANOVICH (2001) The Language of New Media, The MIT Press, Cambridge, Massachusetts.
Luc PAUWELS, Jan Marie PETERS (2005), Denken over beelden: Theorie en analyse van de
beeldcultuur, Acco, Leuven.
Marie-Laure RYAN(ed.) (2004) Narrative across Media: The Languages of Storytelling, University of Nebraska Press, Lincoln.
Mitchell STEPHENS (1998) The Rise of the Image, the Fall of the Word, Oxford University Press, Oxford.
Eija VENTOLA, Charles CASSILY & Martin KALTENBACHER (eds.) (2008) Perspectives
on Multimodality. John Benjamins, Amsterdam.
Heidi Peeters is a postdoctoral researcher at the University of Leuven. She wrote her PhD thesis on novelizations and her main fields of interest gravitate towards the semiotics of new media, issues of adaptation, transmedial storytelling, image-building and the construction of identities.
