Word and Image: Intermedial Aspects

The concepts of ‘word’ and ‘image’ are not synonymous with ‘verbal’ and ‘visual communication’ although they are often restricted to these modalities of sign use. Words and images are cross-medially related, and there are many overlaps. By ‘words’ I mean language, verbal texts or discourse, more generally: verbal communication. By ‘images’ I mean pictures and more generally visual communication, not mental images nor verbal images. Words are communicated both via the acoustic and the visual channels. The acoustic channel is the one of spoken words, but words are also communicated visually in writing. Even in acoustic (oral) communication, words mostly occur in a visual setting, i.e., in the context of gestures, facial, and eye communication, and in a situational setting of objects to which the words refer. The concept of ‘image’, on the other hand, does not necessarily exclude words as they occur in oral or written communication. In spoken language, we use ‘verbal images’, which are, of course, not visual, but mental images. Even in writing, we come across visual images, for example in pictography, where pictures are used to represent words visually, or in picture poems, where a written text takes the shape of a picture, or in iconic interpretation of letters, as in words such as U-turn or T-shirt. In contrast to such broader implications of the concepts of ‘word’ and ‘image’, we will have to restrict ourselves in this paper to words in the form of writing and to images in the form of pictures. There are not many media which are restricted to the use of written words or pictures only. The radio, telephone, telegram, letter, book, or e-

mail are media in which communication may or must take place by means of words only. Paintings, drawings or photography are media typically restricted to pictures. However, in most of the typically verbal media we find the presence of pictorial elements, and in most of the pictorial media we find the presence of verbal elements. Words are combined with pictures to a multimedial message. Furthermore, on the radio, words go together with music or other acoustic signs. In newspapers or newsmagazines they combine with pictures, and in the movies, television, or the hypermedia, words combine with pictures, music, and visually represented nonverbal (gestural) communication. Moreover, words are also able to represent pictures in a verbal description. In short, communication by words and by pictures overlap in a plurality of intermedial contexts.

. Approaches to word-image relationships
The relationships between words and images have been described from many points of view. From a semiotic perspective, syntactic, semantic, and pragmatic approaches may be distinguished.

Syntax
From a syntactic point of view, the combinations between words and images are described as to their relation in time or space. Temporally, the word-image syntax is either one of simultaneity or one of succession. Simultaneity of words and images predominates in the print media, when words illustrate a picture on one and the same page, but there is also simultaneity in the film, when the pictures show the actors speaking. Succession can be found in books, when the picture follows or precedes the verbal text to which it refers on a different page. It was the typical word-image relationship in the silent movies, where the words either preceded or followed the pictures to which they were related. Succession is also a typical relationship between literary texts and the visual arts. With years of distance, e.g., paintings succeed ancient works of literature, whose scenes they depict, and in the literary genre of ekphrasis, where a poem describes an earlier classical painting. Two main types of spatial relationship between word and image are contiguity and inclusion. Verbal texts with pictorial illustrations or photos with explanatory legends are examples of the contiguity type of spatial syntax. The inclusion of words in pictures is mainly of four kinds: 1. representation of words in pictures, as for example in a photo which includes the picture of a page of writing, 2. pictorialization of words, where words lose their character as verbal signs and become elements of the picture, 3. inscription, where the picture merely serves as a writing space, and 4. indexical inscription, where the words are inscribed in the picture as indices referring to depicted objects.

Semantics
Semantic studies of the relationship between words and images investigate the contribution of the pictorial and the verbal elements in the combination of both to a complex message. Most studies of word-image relationships have been concerned with such questions, and therefore I can restrict myself to a typology of word-image relationships from the semantic point of view. Five kinds of relationships between pictures and words in texts can be distinguished: complementarity, dominance, redundancy, discrepancy and contradiction.
∑ Complementarity is the ideal mode of combining words and pictures.
Word and image are complementary when both are equally necessary to the understanding of the message. ∑ Dominance can mean dominance of the picture, as in books on paintings, or dominance of text, as in illustrations of a novel. ∑ Redundancy is the extreme counterpart of dominance. In the context of a picture, a verbal message is redundant when it only repeats what you see anyhow. ∑ Discrepancy and contradiction are forms of mistaken or poetically deviant word-image combination. Word and image do not fit together. The text and the picture are juxtaposed by editorial negligence, a mistake of the reader, who does not see that both do not belong together, or because of a poetic device with the aim of creating a surprising contradiction between the verbal and the pictorial messages in order to make the reader think further about a possible solution of this enigmatic contradiction. 4 / 8

Pragmatics
While syntactic and semantic aspects of text-picture relationships have been investigated in many studies, less attention has been given to the pragmatic aspects of this relationship. When words are used to direct the readers' attention to the picture, especially to certain parts of it, or when pictures are used to direct the readers' attention to a specific verbal message, the word-image relationship is predominantly a pragmatic one. Prototypical of a pragmatic word-image relationship is a verbal message that says "Look at the upper right hand corner of this picture", or a picture that draws the readers' attention to, and arouses their interest in, an otherwise rather dull verbal message, as in advertising. The relationship between word and picture is in both cases an indexical one.

. A semiotic approach
In the following, I would like to suggest a new approach to the study of word-image relationships based on the Peircean trichotomy of the iconic, the indexical, and the symbolic sign. For readers acquainted with the fundamentals of semiotics, it is unnecessary to repeat that pictures are predominantly iconic and words are predominantly symbolic signs. However, it is necessary to underline that our topic is not the extratextual (referential) relationship between verbal or pictorial signs on the one and their referential objects on the other hand, but with the intratextual relation between words and pictures in one multimedial message, i.e., with the way words relate to the pictures in juxtaposition and pictures to words. My argument is that the trichotomy of icon, index, and symbol also applies to the intratextual relationship between words and images.

Intratextual iconicity
Iconic signs are signs based on a similarity between the sign and its object. There is intratextual iconicity between words and pictures when the verbal text conveys the same message as the picture. The picture is hence an icon of the text, and the text is an icon of the picture. In fact, a redundant illustration of a text or a redundant verbal paraphrase of a picture are the clearest examples of intratextual iconicity. With Peirce, we can furthermore distinguish between images, diagrams, and metaphors in the intratextual word-image relationship. The homology between text and picture is of the type of an image when both evoke the same mental image without any other semiotic mediation. It is of the type of a diagram when one of the two messages represents the other by means of merely structural relationships, and the relationship is metaphorical when there is a semiotic mediation via a third sign. between word and picture serves as an index that connects the verbal with the pictorial sign. The message is simply: this verbal text refers to that picture (and not to any other picture on the same page). Traditionally, the text appears below or above the picture to which it pertains (as in a caption or legend), but in advertising any space in the vicinity of the picture is being used. 4. indexicality by pars-pro-toto relationship: the pictorial message represents only a part of the message conveyed by the verbal message, or vice versa. For instance, the verbal message advertises "New York", but the picture only represents the Statue of Liberty. 5. exemplification: the picture gives only an example of what the verbal message refers to (and frequently vice versa). For instance, a supermarket advertises only one of its products without mentioning any of the other products for sale. Exemplification is closely related to ostension and to pars-pro-toto indexicality, and there are many other overlaps between the various subclasses of indexicality. 6 / 8

Intratextual symbolicity
While indexical relationships between words and images have received much attention in the study of word-image relationships and the iconic type of relationship can hardly be questioned, the possibility of intratextual symbolicity between words and images seems to be a paradox at first sight. After all, an essential feature of symbols is their arbitrariness, and arbitrariness is certainly not an efficient way of connecting a text with an image. However, arbitrariness is not actually Peirce's first criterion of symbolicity. In contrast to the icon, which represents its object because of its own sign quality, and the index, which is a sign because of a hic e t nunc relationship between sign and object at a given locus in time and space, the symbol, according to Peirce, is associated to its object because of a habit of sign interpretation. To be symbolic, word-image relationships would therefore have to depend on habitual associations. Habits of relating images to words and words to images exist indeed. We acquire the habit of associating a verbal and a visual message because of seeing both repeatedly in juxtaposition, because of an instruction to associate both, and after learning to associate one message with the other. Such processes of creating and learning habitual associations between verbal and pictorial messages are quite frequent in the media. We recognize by habit the pictures of prominent politicians and film stars in the media not because we discover any similarity between the photos and the real persons which they represent (but which we have never seen in real life), but because of having learnt to associate the pictures of these people with their names in previous messages conveyed by the media. The average newspaper reader, for example, does not recognize a picture of President Clinton because of the similarity which the photo has with the politician as a living person (and hence with the referential object of the picture), but because of having been told previously by the media that the man shown in the picture is President Clinton. In advertising, the Camel or the Marlboro campaigns make use of pictures that we associate habitually with brand names. Habitual association means that we no longer need not to be reminded of the name at all, when we see the picture. The pictorial message, in the end, does not need the verbal message any more. Camel and Marlboro advertisements, in fact, have been so long around that the campaigns now begin to present the pictures alone, omitting the words completely. Notice, however, that a habitual association between word and image cannot be created intratextually, i.e., within one text. Only as a result of a process of learning from earlier messages in which the word first appeared indexically connected with the picture did we acquire the habit of associating the brand name with the pictures, 'Marlboro' with the pictorial myth of the Wild West. In other words, symbolic word-image connections arise from intertextual habits of interpretation. Their origin is always an indexical sign relation.

. Evolutionary considerations
Let us briefly consider the relevance of the three modes of word-image relationship to the study of message propagation in the media. It may already have become apparent that the iconic, the indexical, and the symbolic word-image relationships refer to different strategies and phases of the propagation of messages: first, there is an indexical phase, a symbolic phase comes last. How does the iconic enter this evolutionary process? For lack of time, I must restrict myself to advertising campaigns. A sequence index-icon-symbol is clearly evident in three phases which we can observe in the evolution of advertising campaigns: 1. The indexical word-image relationship predominates in the phase of epiphany, the phase in which a product is introduced in the market. Since the product is new, it must be shown, pointed at, and indexically related to its name. 2. The second phase is the phase of repetition and affirmation.
Prototypical of this phase are advertisements of the type "Omo is Omo", or "Persil is Persil". The image repeats the message of the text, and the text says nothing new about the product. The image is an icon of the text, and the text duplicates iconically the message of the image. The intratextual repetition of the same message makes the message in the end autoreferential or tautological. 3. Only few advertising campaigns advance beyond these first two phases of indexical and iconic word-image relationships to a symbolic phase, where, as described above, word and image evoke each other by a habit of the interpreter. This takes place in a phase that we might call the phase of habitualization. The consumers have now become completely acquainted with the product and need only to be reminded of its existence. Products that can be advertised in this way, as in the Marlboro campaign, must have attains a climax of familiarity with