Keynote Abstract

One important contribution commonly ascribed to hypertext is the ability to combine different forms of expression, and so be considered ‘multimodal’ (or, at least, ‘multimedial’). On closer analysis, however, theorizing just what this entails has remained limited. Similarly to the situation that long held concerning ‘written’ texts, it is too easily assumed that different modalities, sometimes labelled with terms such as ‘text’ or ‘image’, combine ‘naturally’ and so users should be able to follow such combinations with relative ease. Research on literacy, particularly with respect to contemporary media configurations, has shown this assumption to be false.

Constructing coherent interpretations of combinations of modalities can be far from straightforward, even when supported by good interface design; with poor design, which from the perspective of displayed ‘documents’ is unfortunately rather common, finding intended interpretations can present significant challenges. Now, when translated to the even more complex medial environment of hypertext, these potential problems are magnified considerably. Moreover, traditional considerations of where the ‘boundaries’ of hypertext might lie are now being redrawn as hypertext and the increasingly ‘hyper’-connected medial world become increasingly permeable. The entire multimodal world of social media and participatory digital cultures might then be considered from a hypertext perspective, but research on hypertext itself lacks conceptual tools with the power necessary to engage with that world. Simple ‘extensions’ of traditional notions of hypertext are likely to prove insufficient for a full-blown account of multimodality.

In this talk we address these concerns from the perspective of current developments in multimodality studies, where the starting point is communication as such, regardless of the expressive forms that are used for that communication and whether communication is mediated computationally, via interlinked artefacts and pathways, or by cross-linked practices of digital and non-digital use. In short, current medial practices demand that hypertext be seen not simply as, for example, a shift from page-based documents to video, but as a further computationally supported environment for the development and deployment of core multimodal theoretical constructs such as semiotic modes, media and genres. We introduce these concepts and show several practical examples of processing from ongoing projects with a variety of media. [1–3]

Speaker Biographies

Prof. Dr. John Bateman

Professor for Applied Linguistics, Bremen University, Germany

Figure 1: John Bateman

Over the past 15 years Bateman’s research has focused increasingly on communication that combines diverse forms of expression, including verbal language, images, moving images, image sequences, sound and movement, in all cases applying and extending models of textuality generalised from linguistics and semiotics. 2017 saw the publication of a broad transdisciplinary textbook on the theory and practice of multimodality research as a field of study in its own right (with co-authors Janina Wildfeuer and Tuomo Hiippala). His current work includes expanding notions of semiotics to combine discourse and embodiment (in the context of Human-Robot-Interaction in the ‘Everyday Activity Science and Engineering’ SFB of the university), broadening notions of textuality and narrative for diverse media (for example in the university’s interdisciplinary and collaborative research platform ‘World of Contradictions’ Lab on multimodal narrative), applying principles of formal ontology to the modelling of semantics for language and other modalities, and pursuing more robust empirical connections between data and theory for diverse media.

He studied linguistics and computer studies at Lancaster University and obtained his PhD at Edinburgh University in Artificial Intelligence within the ‘Epistemics’ postgraduate programme combining AI, linguistics, psychology, and philosophy. After working and researching on computational linguistics, multilingual and multimodal document generation, and formal ontology at the universities of Kyoto, Southern California, Saarbrücken, and Stirling and the Gesellschaft für Mathematik und Datenverarbeitung (GMD) in Darmstadt, he joined Bremen University in 1999.

Assist. Prof. Dr. Tuomo Hiippala

Assistant Professor in English Language and Digital Humanities University of Helsinki, Finland

Figure 2: Tuomo Hiippala

Tuomo Hiippala has been an Assistant Professor in English Language and Digital Humanities (tenure track) in the Department of Languages at the University of Helsinki, since 2018. He is also a member of the Digital Geography Lab, where he worked as a post-doctoral researcher in 2017, and affiliated with the Helsinki Institute of Sustainability Science and Helsinki Institute of Urban and Regional Studies. He is a member of Young Academy Finland (2018–2022). In 2015–2016, he was a post-doctoral researcher at the Centre for Applied Language Studies at the University of Jyväskylä. He holds a PhD in English philology from the University of Helsinki (2014), supervised by Eija Ventola.

His research focuses on multimodality, that is, how multiple modes of expression interact and co-operate with each other in different communicative situations. To exemplify, spoken language, gestures, posture and gaze are constantly coordinated in face-to-face interaction, while magazines, newspapers, websites and other page-based texts organize written language, photographs, illustrations, diagrams, information graphics and other modes of expression into coherent layouts. In both cases, communication builds on appropriate combinations of different modes of expression. Theories of multimodality attempt to explain how such appropriate combinations are formed and how they become understandable in context. He is particularly interested in how theories of multimodality can inform research on artificial intelligence, and conversely, how artificial intelligence can support empirical research on multimodality. In addition, he is also interested in applications of natural language processing and computer vision in the humanities and beyond.

References

John Bateman, Janina Wildfeuer, and Tuomo Hiippala. 2017. Multimodality: Foundations, research and analysis–A problem-oriented introduction. Walter de Gruyter GmbH & Co KG.

John A Bateman. 2017. “Triangulating transmediality: A multimodal semiotic framework relating media, modes and genres.” Discourse, Context & Media 20 (2017), 160–174.

Tuomo Hiippala and John A Bateman. 2021. “Semiotically-grounded distant viewing of diagrams: insights from two multimodal corpora.” arXiv preprint arXiv:2103.04692 (2021).