Multimodal Communication Modes

• Multimodal texts can be categorized into simple and complex types, with examples of simple multimodal texts including comics, picture books, newspapers, and social media, and complex digital multimodal texts including live action films, animations, and web pages.

• Teaching multimodal literacy is essential as it enables students to understand how authors combine different modes to convey meaning and how meaning is "orchestrated" through the selection and use of different modes in various combinations.

• Multimodality involves the complex interweaving of word, image, gesture, movement, and sound, including speech, to convey meaning.

• A multimodal text is defined as one that combines two or more communication modes to convey or make meaning, with communication modes including linguistic, visual, audio, gestural, and spatial.

• Linguistic mode involves the vocabulary, structure, and grammar of oral and written language, conveying written meaning through handwriting, printed page, and screen.

• Spoken mode conveys meaning through spoken language via live or recorded speech, with choices around mood, emotion, emphasis, fluency, speed, volume, tempo, pitch, rhythm, pronunciation, intonation, and dialect.

• Visual mode conveys meaning through choices of visual resources, including still and moving images, with visual resources such as framing, vectors, symbols, perspective, gaze, point of view, color, texture, line, shape, casting, saliency, distance, angles, form, contrast, lighting, and camera movement.

• Audio mode conveys meaning through sound, including choices of music, ambient sounds, noises, alerts, silence, natural/unnatural sounds, and use of volume, beat, tempo, pitch, and rhythm.

• Gestural mode conveys meaning through choices of body movement, facial expression, eye movements and gaze, demeanor, gait, dance, acting, and action sequences.

• Spatial mode conveys meaning through design of spaces, using choices of spatial resources including scale, proximity, boundaries, direction, layout, and organization of objects in the space.

• Spatial meaning extends from design of the page in a book to the design of a room, architecture, streetscapes, and landscapes.

• Types of multimodal texts include paper-based multimodal texts such as picture books, graphic novels, and posters, live multimodal texts such as dance and performance, and digital multimodal texts such as film, animation, and web pages.