After a two-day, cross-faculty multimodality event featuring a joint keynote presentation by Gunther Kress and Theo van Leuven, as well as their much lauded hands-on workshop with PhD and early-career researchers, the multimodality satellite strikes again. On Weds 24th May 2017, we were lucky enough to get a sneaky preview of the paper Dr Martin Thomas (Centre for Translation Studies, LCS) will be giving at the Conference Corpus Semiotics: Reassessing Context, taking place at the University of Manchester on the 5th June. The presentation was entitled “Documentness and Representation” and addressed the question of how corpora could account for the graphic realisation of various kinds of texts.
The topic is relevant to us as a group because it highlights the need to go beyond traditional corpus accounts of purely verbal features and find ways to account for co-text, which is crucial for the successful comprehension of many genres of text and yet is still predominantly disregarded in current practices of corpus-making.
Martin discussed the importance of incorporating pragmatic features in the creation of both spoken and written corpora, as well as some issues corpus linguists face when transcribing texts into linear formats. He also reflected on how the genealogy of all texts is bound to the constraints both of production for and consumption by their intended audience. For example, when moving from the English to the Chinese version of a text, its font size usually needs to be increased by a predictable amount. A particularly clear and interesting case-in-point is that of gardening manuals, the appearance and format of which was revolutionised by the introduction of the printing press.
Caption: Dr Martin Thomas discussing how character printing changed the face of gardening manuals: the integration of text and graphics, which was very common in the Middle Ages, became more difficult and quite expensive after the introduction of the letter press, which meant a shift to predominantly text-based manuals.
Graphic realisations such as tables, diagrams and images are central to many text types e.g. instruction leaflets, manuals or bills, and yet it remains tricky to represent these features satisfactorily in an automated way when creating a corpus. Certainly, isolating linguistic elements from their co-textual graphic realisations and considering the latter “noise” that is of no interest – as it often happens in (corpus) linguistics – can result in a loss of meaning. Such loss can potentially be detrimental to understanding, especially where texts are particularly visually informative, and therefore calls for the creation of multimodal and multimedia corpora that connect the pragmatic and semantic properties of the meaning-making modes involved.
Caption: Martin’s presentation addressed the need to create alternative coding systems for multimodal corpora, where a combination of video, audio and textual resources organically combine to represent human discourse.
Examples of both typographic modulation and segmentation were presented as ways to reinforce meaning-making. Graphic features such as headings, paragraphing, colouring, punctuation, framing and reading direction were all discussed as aids for the document creator to convey meaning, as well as for reader to interpret the message being conveyed. In other words, different documents look different, and we do use such ‘documentness’ both to create and decode meaning. Martin also argued that the more resources used to make meaning, the more potential issues; for example, if said resources are not internally coherent. Sometimes genres can also borrow typographies from each other, and here the audience was invited to reflect on the resemblance between modern online journalistic articles and gas bills, where the newspaper-style headlines of the latter implicitly refer to the former, suggesting to the target audience that the reading strategies deployed when scanning (and possibly – but not necessarily – then reading in more detail) an article also work when reading a bill.
Caption: The resemblance between modern online newspaper articles and gas bills as an example of how typographies can be borrowed between textual genres. [reproduced with permission from the author]
In what felt like a very exciting new proposal, the presentation suggested a shift in viewpoint from transcribing the verbal elements of a text first and then encoding their graphic features, to describing all graphic features, including verbal elements, in terms of frequently occurring configurations in a two-dimensional space. Martin talked about two recent attempts to build annotation schemes, GeM and PAGE. The guiding principles, advantages and disadvantages of both were addressed. After the presentation, this was also the main topic of an engaged group discussion, where we talked about future developments as potential ways of integrating particularly useful features belonging to one but not the other annotation scheme.
This type of events where researchers give a rehearsal of a paper are extremely useful, not only to the presenter, who can ‘feel the lie of the land’, so to speak, i.e. see how their presentation is received, collect feedback, and obtain interesting pointers on their draft presentations, but also to the attendants, who get a chance to get to know other colleagues’ work, learn about the dazzling array of applications of multimodality, and be the first to hear about original research conducted across the UoL. And this is where the real power of the multimodality satellite lies: in its intrinsic interdisciplinarity, which lends itself so well to broadening views and research scope, whilst simultaneously creating spaces where interesting collaborations can be born and flourish. Needless to say, we are very much looking forward to welcoming and facilitating many such events in the future.
Words by: Valentina Ragni
Pictures by: Elisabetta Adami