The Multimodal Journey in Advertising

The concept of multimodal communication—conveying messages through multiple modes of communication, such as visual, linguistic, spatial, aural, and gestural modes—stands as fundamental to a comprehensive text that interacts with readers who hold many differentiated identities. Despite the prevalence of multimodal communication, many of our decisions in creating multimodal text are unintentional (Arola et al. 3).

Multimodal communication operates in complex manners that can connect with all five senses despite the process not always being visible to the reader. For instance, examine the features of the advertisement below, a 1.7-metre-tall banner placed on a board on central campus grounds, which I decided to analyse. Despite the numerous modes present throughout the text, I did not initially consider these whilst making the advertisement.

Figure A. A highly visual advertisement with minimal linguistic modes present. Several call-to-actions exist (Casual Gaming website hyperlink and group chat hyperlink) to redirect readers to more linguistic modes containing further information about the subject.

When I initially drafted this advertisement, I drew my intentions toward implementing a highly visual mode of communication. What this example particularly conveys, though, is that some modes may surface and showcase their visibility much more than others—in this case, the visual mode is highly present. Although one could argue that most comprehensive forms of texts contain multiple modes compromising with each other simultaneously to puzzle-piece together a panoramic message, often one mode will outweigh others in their presence throughout the text.

In this marketing material, we can scrutinise the elements found in each mode and their effects on readers. Examine the spatial mode of the content: we can easily absorb what takes up roughly three-quarters of the text, which is the visual imagery that also contains the gestural mode of the person’s pose. The entire imagery allows for  multiple interpretations of the content, implemented originally to connect with the reader’s sense of humour and feelings.

‘What a peculiar poster. This individual is positioned in a strange manner. I wonder what this is advertising? Casual Gaming Club. This organisation looks informal, they seem to have a… unique sense of humour—it kind of looks like that they embody a ‘meme culture’? I kind of want to meet this guy, he looks peculiar. Is the rest of the organisation’s members like this too? Maybe I would enjoy seeing what the organisation is like and meeting some people. I could go to the website to see when the next event is, maybe it’ll be fun.’

Multimodal text functions in strange, yet complex ways, ultimately instilling readers to create a journey for themselves to embark on. With so many different literary ingredients infusing into a single concoction of meaning, the mixture of content challenges, and thus encourages readers to paint a clear definition themselves of the message that the abstract text conveys. Although not all planned journeys by the writer can go as planned once the text is shared, some deviations from the structured path allow for creative opportunities for the reader to imagine something beyond the scope of the writer’s originality. This allows for a collaborative moment for the reader to unknowingly work with the writer to complete the multimodal text with the reader’s own interpretations.

Part of this journey is also observing the intertwining interactions between modes that complement what the reader sees initially. For instance, after the imagery attracts the reader to focus on the advertisement, the reader will be in search of more information, scanning peripheral content in hopes of alleviating his/her desire for a comprehensible solution. The text reading ‘Casual Gaming’ is then the next piece of information the reader views to direct a conclusion to the reader’s information search. By fluidly connecting the visual text to point the reader toward the linguistic mode, this simultaneous interaction activates the reader’s ability to link the emotions and opinions extracted from the visual mode to the detailed call-to-action linguistic mode of the organisation’s name, and finally the website landing page shown below. In terms of spatial mode, the placement of the visual mode over the linguistic mode encourages the reader to understand the content top-to-bottom order as intended, leading them from the highly visual mode-based advertisement to the more informational, linguistic mode-based website.

Figure B. Website landing page from the advertisement. With more linguistic modes present on the website, interested students should be able to find more information beyond simply a vague image from the initial banner.

And, despite the lack of an intentional aural mode, the reader’s imagination can act as the catalyst to create their own modes. Maybe the reader recognises the backdrop of the first example’s imagery (which displays a local, Chinese tea café on campus) and then invents ideas of how that environment sounds or feels: bustling conversations, the buzzing of the AC and its cool, breezy breath on the reader’s arms whilst holding a cold, cup of tea. Perhaps the reader even knows the individual on the banner, and then imagines interacting with the individual, or perhaps even reading the linguistic mode vicariously through this person’s tone, pitch, and accent of voice.

What if one doesn’t know any information about the text? Everyone’s identities come into play when interpreting text. For instance, the reader might assume the person’s identity as a male gender and an East Asian ethnicity. All these assumptions may then lead up to biases or stereotypes that affect the reader’s interpretation and drive the text in different directions. Therefore, an important rule of thumb is to always take into account the different identities that any reader could hold in understanding the text.

Taking into account the importance of modes beyond linguistic representations, as I continue my writing experimentation for my papers, understanding the intrinsic value of all modes is highly important. Particularly when I am trying to connect and empathise with my readers, providing multiple access points for the reader to grasp the text is vital. With this in mind, I may consider supplementing my heavily linguistic text with more visuals.


Works Cited

Arola, Kristen L., et al. Writer/Designer: Guide to Making Multimodal Projects. Bedford/St. Martins, 2018.

Day-to-Day Multimodality

Out of the many texts I encountered this week, I chose four that qualify as very different types of media. The first text, a music video for Lawrence’s “Do You Wanna Do Nothing With Me?, uses all five modes. The linguistic mode is crucial to the music video because the words of the song convey specific meanings and emotions, and the aural mode, since the sounds experienced by the listener come from the musicians’ musical choices. The spatial and gestural modes encompass the artists’ movements, facial expressions, dancing and arrangement in the room, helping Lawrence create a relaxed, casual and welcoming ambiance. Lastly, the visual mode plays a major role in the music video, because the living room setting and the funky, casual outfits of the musicians contribute to the overall feel of the video and the message of the song.

Another text I selected is Julia Alvarez’ In the Time of the Butterflies, a fictional novel about the four Mirabal sisters, who opposed the Trujillo dictatorship in the Dominican Republic. Since it’s a novel and the author’s meaning is conveyed through words, the text relies heavily on the linguistic mode. It also employs the spatial mode because the arrangement of the text affects the reader’s experience; each chapter is written in the perspective of a different sister.

This week, I also encountered a text in the form of a speech by Professor Joan Greve at my initiation into Beta Mu Epsilon, a professional Biomedical Engineering fraternity. Professor Greve spoke very briefly, touching on the mission of the fraternity and of biomedical engineering in general. She encouraged us to work dedicatedly in our field of choice. Professor Greve used the linguistic mode, selecting vocabulary that made her speech inspiring and engaging, as well as the aural mode, since we were listening to the speech and she had to choose which words and phrases to emphasize volume-wise. I think Professor Greve’s speech also employed the gestural mode because her facial expressions, stance and hand motions helped communicate her excitement and seriousness at different parts of the speech.

Lastly, I decided to analyze the written directions for an EECS 215 problem set. This text uses the linguistic mode, since it’s written, and the visual mode, since it included visual diagrams and examples to help students understand the content. It also uses the spatial mode because the order of the questions and pictures follows the order in which class material was taught.

The four texts I chose are very different in terms of form and purpose. Overall, I think the problem set and music video are the most different from one another, since they have such different audiences and messages. The one similarity I can identify is that the music video, speech and homework set are all very recent texts, while the novel was published in 1994. The two written texts, the novel and the homework set, are alike in their reliance on the linguistic and spatial modes to get the majority of the information across. The other two texts, the music video and speech, incorporate other modes to create meaning and impact the audience.

I find it interesting how all four texts use the linguistic mode, but in very different ways. As in the music video and speech, the meaning behind spoken or sung language is often supported by visual and gestural components, whereas authors of written texts need to rely more heavily on the linguistic mode to evoke emotion or communicate a message.

I particularly appreciate how the music video uses every single mode to create an ambiance. All five modes play into the upbeat, relaxed, low-key feel of the video. I realize that music videos allow for a lot of creativity because they can benefit from the affordances of all five modes, although I don’t think using fewer modes necessarily stifles creativity.

A Multimodal Friday

I decided to conduct an experiment on multimodality on just another regular Friday afternoon. Over the course of my day, I recorded my encounters with a variety of texts which all incorporated modality differently.

On my way to work, I encountered the first multimodal text of the day: a text message from one of my best friends.

My first instinct, of course, was to note down this text’s utilization of modality. The incorporation of linguistic and visual modes was immediately obvious, as the text used both words and a picture. Gestural mode was used in conjunction with visual mode: the way the dog is sitting is very indicative of the “energy” my friend is referring to.

Once I was at work, I ran across my second multimodal text: a recently published research paper that I was tasked with summarizing.

Again, linguistic mode was easy to recognize. The tone of this paper was very different from my friend’s text message, as this was a research paper and had to not only give background on a fairly complex topic, but also cater to a large audience of academics. The paper organized itself into paragraphs, used appropriate grammar, and did not have any photos of adorable dogs.

Visual and spatial modes are also important parts of this text. Looking at the font, spacing, and margins of this paper immediately categorizes it as a research paper. The authors chose to highlight an important point made in the paper by putting the pull quote in a different font and color, using both visual and spatial modality.

The final multimodal text of my Friday was a conversation with an old friend over dinner. In terms of modality, gestural mode came into the picture first. Things started off a little awkward, as we hadn’t seen each other in over two years. After ten minutes though, we were back to our normal selves, and that was clear from our body language. We went out to dinner at Sava’s, which immediately influenced the spatial mode of our interaction – we were seated across from each other at a high-top table. I was surprised to see aural mode make its way into our interaction as well: she had picked up a slight accent and different vocal tics from living abroad for so long.

One thing this experiment taught me is how interconnected all the modes are – most  modes, if not all of them, overlap with each other. Maybe for that reason, I had trouble finding a medium which used all five modes in distinct ways. I think the coolest thing about modes is how they are combined with each other, depending on the purpose of the text. Going forward, I am definitely going to be more cognizant of why an author decided to make use of a certain mode over another – what were they trying to accomplish? Who are they trying to reach?