Post-cognitivist HCI: Graphical Interaction, the Cultural Interface and Embodied Interaction (Part 5)

Graphical Interaction and the Cultural Interface

Instead of predominantly linear textual interaction, the graphical bitmap screen and the graphical user interface (GUI) brought the revolutionary possibility of using a two-dimensional surface for interaction with the computer. Dourish writes that since the arrival of the GUI the management of information has become the management of space, a space filled with the specific elements of the visual culture of the computer: scroll bars, buttons, drop-down menus or overlapping windows. The new visual layer of the interface turned away from the sequential thinking that we found in the preceding symbolic as well as textual interaction. The visuality of the interface further developed the declarative way of communicating with the computer into what can be called the “direct manipulation” of GUI objects. For since the days of the SAGE radar system, and later also of the engineer Ivan Sutherland’s Sketchpad project, it has become a matter of course that by clicking, dragging or selecting elements in virtual space we activate a technical assemblage invisible to us and change the contents of memory and other parts, which generate the action we have requested.

With the increasing power of technologies, designers began to have the freedom in this space to make use of digital elements that resemble what we commonly know from everyday life. The GUI paradigm of the “desktop”, created by Xerox, and of other objects that we would commonly find in an office, is a metaphor that cleverly draws on the repertoire of our existing knowledge of the real world. Another example is the visual strategy of skeuomorphism, which became very popular in Apple’s mobile and desktop operating systems. Skeuomorphism imitates the appearance of physical objects as faithfully as possible: the calendar, for example, looks like a real diary and even has its stitching depicted. And its declared advantage is that in entirely new software the user quickly recognises the intended affordances, or rather the available actions and controls, for the GUI follows relatively precisely how the object behaves in the physical world.

The theorist and practising interaction designer Alan Cooper spoke out against the excessive use of visual metaphors imitating the physical world in his book About Face. Because digital space need not follow physical and other limits, Cooper argues that the dependence of visual metaphors on the workings of the physical world is rather a limitation. Visual metaphors do not scale well, and their supposed intuitiveness of use depends heavily on sociocultural factors. Often it is not easy to find a visual metaphor for abstract technical processes. Instead, Cooper proposes the use of idioms, which a person does have to learn, but which, when suitably chosen, are quick and easy to remember¹.

According to Dourish, the GUI, besides using the power (and the weaknesses) of visual metaphors, also engages our cognitive skills, which we use in our ordinary coping with everyday visual information: pattern recognition and spatial thinking. Just as in interaction with the real world, the GUI brings problems with it too: designers must work out how to display visual information suitably which, by its density, competes for our attention. Fortunately for designers, the theory and practice of film, the visual arts, graphic design and other fields offer tried and tested patterns and practices that can be taken over into the digital environment.

Some mechanisms of working with two-dimensional visual space reach deep into the history of science and art. Lev Manovich argues that we can find precursors of the GUI and of the computer screen as early as the painting and architecture of the Renaissance. Manovich mentions the Italian polymath Leon Battista Alberti and his frame with a grid, which helped painters with linear perspective and which, according to Alberti, was a window onto the world². But what kind of window can the frame with a grid and the GUI be? They are certainly not a transparent window, even though that is the ideal of many designers who feel the need to conceal „zprostředkující povahu technologie“³ and to give users an authentic experience in using technologies. The media theorist Erkki Huhtamo is sceptical of the ideal of transparency and authenticity, for in his view technology „stává druhou přirozeností, vnějším i internalizovaným prostředím a předmětem touhy. Neexistuje již žádná potřeba činit ji transparentní, jednoduše proto, že není pociťována v rozporu s ‚autenticitou‘ zkušenosti“⁴.

For the screen and the GUI do not represent a window onto the real world, but rather onto the already mentioned space of the unreal and of simulacra, that is, of constructed images without an original of their own⁵. That is why the GUI resembles the abstract visual art of the twentieth-century avant-garde more than mimetic (imitative) painting, to which Renaissance painting belonged. For it forces the observer to look at the world in a certain way, one that separates out only a certain part of the chaotic and continuous nature of the world. According to Manovich, the interface is an aggressive medium, which filters reality and pushes everything outside itself into the background. This filtering is subject to the logic of a protocol, which imposes a certain mental model of interaction and communication. As Manovich goes on, such communication is mediated by a medium that influences, deforms or in some way translates the communication. The way in which the GUI, as a medium of a certain form (specifically, a rectangular screen with visual interactive elements), mediates interaction with the computer creates a model, system or ideology that stands in opposition to Shannon and Weaver’s linear model of communication. For they are not a transparent model of communication, but impose on the user a certain way of acting, which Manovich identifies as the operations of selection, compositing and teleaction⁶.

According to Manovich, these operations are indeed part of software, but they reflect a much broader social reality. The communication between software, the computer and the larger social world is a two-way communication between these participating entities, which influence and enrich one another. The methods used for working with data become, for one thing, general cognitive strategies for understanding how the human mind works, and software design and HCI then reflect “a larger social logic, ideology, and imaginary of the contemporary society”⁷.

This means that interactions and interfaces do not arise in a vacuum. Whether consciously or unconsciously, they contain embedded cultural codes and reflect the culture, society and other contextual factors that influence designers, but of course also the users themselves. Let us take Manovich’s last principle of new media (discussed in more detail in Part 16): the principle of transcoding. It denotes that interactive relation between society and technology which, for Manovich, materialises and is set in motion through the interface. In order to distinguish this view of the interface from the traditional conception of the interface in (cognitivist) HCI, however, Manovich introduces the neologism cultural interface, by which he expresses that in today’s information society there is no escape from the computer mediation of every part of our lives. It is precisely the interface that mediates our access to cultural data, or, in short, to culture and society, with which the Czech HCI researcher Jan Brejcha also agrees⁸, conceiving, on the basis of a semiotic analysis, every sign in the interface as filled with cultural content, or rather as having a semantic part influenced by the meaning systems of language and culture.

How can this analysis be reconciled with the earlier and seemingly contradictory claim that the interface is a window onto a non-existent, virtual space of simulacra? The simulacrum that, according to this analysis, the interface is supposed to (re)present is not a “desert of the real”, but simply something that has no reference in the physical world and takes on its form only as the more or less stable and uniform result of a creative synthetic or “poietic” act, as we see it in Aristotle as τέχνη (technē) or in Heidegger’s essay The Question Concerning Technology as making present and bringing out of concealment into unconcealment⁹. This act combines and brings together technological, human and cultural elements. The interface thus mediates between these diverse “components” and translates between their incompatibility, which stems from technical, cognitive, social, spatio-temporal and other limits. Just as a language interpreter overcomes misunderstandings between speakers of different languages, so the interface too, as an intermediary or medium, translates according to Manovich’s operations of selection, compositing, teleaction and many others. We can unite all these operations under the category of “filters”, for on the basis of a chosen strategy, which can be taken as an ideology, the interface enables some communication and interaction and not others.

If I write that the interface is a medium and that it translates, I will argue later, in the section on postphenomenology, that the medium already implicitly includes within itself the translation and transformation of information. Van Den Eede mentions that Marshall McLuhan often uses the words interface and translation synonymously. According to him, every medium is by nature a certain transformative barrier and constitutive force, and translation by means of a medium creates a new relation that would otherwise not be possible. That is why the universal situation in which the medium finds itself is precisely the “interplay” of disparate elements and the mediation between them¹⁰. In McLuhan’s ontology it is more correct to look at the medium and the interface not as a fixed entity, but as a dynamic process of mediation, which stabilises according to which entities or other processes it translates between. That is also why he prefers to analyse media as a constitutive force that influences its surroundings. This force, however, needs entities, just as entities need a relation between them for their identity. McLuhan’s relational ontology of media amounts to a critique of substance ontology. In the latter, the relations between relatively fixed identities of entities were merely an epiphenomenon, something secondary, whereas for McLuhan they are at least as important as the entities themselves, for it is the relations and interactions between entities that actively constitute the identity of entities.

If we return to Paul Dourish’s division of interaction, to the principle of transcoding and to Lev Manovich’s concept, we find that the interface must be a dynamic process, for besides relatively static material entities, the interplay of elements mentioned above is joined by cultural and social influences and, according to Dourish (and later also to post-cognitivist HCI theory as a whole), by the influences of the concrete environment and situation in which the interaction takes place. Dourish captures this in his concept of embodied interaction, which he understands as an umbrella term for the phenomena of tangible and social computing.

Social and Tangible Computing as Embodied Interaction

Although Dourish published his book on embodied interaction in 2004, that is, still several years before the massive rise of smart mobile devices, his theoretical frameworks of social and tangible computing are just as relevant today, if not more so.

Thus, for example, today we can clearly see the fulfilment of the assumptions of tangible computing in the approaches of ubiquitous interaction and the Internet of Things, which represent a deep integration of computing devices into the fabric of our everyday lives. Since the main principle of these approaches is the use of physical technological objects equipped with computing power and environmental sensors, these objects can react in real time to contextual data coming from the place where they are situated. Especially in the convergence of information technologies, architecture and urban planning, with the blessing of politicians, too, as we see in the efforts of “smart cities” to make the city a platform of digital services, it is possible to trace signs of how the world is becoming one big interface. Which is not merely poetic exaggeration, for besides the deep integration of technology into the physical world itself, tangible computing describes a new paradigm of interactions in which, instead of the graphical user interface, technological objects and the world around us themselves become the space for physical or, according to Dourish, direct interaction.

Unlike the mediation of our actions by the classic desktop paradigm of the GUI, with the mouse as input device, Dourish’s direct interaction presents data and technological objects that we can “take into our own hands” and manipulate directly. The principle of direct interaction developed into an entirely new paradigm of the natural user interface, or NUI¹¹. NUI should be complementary to the GUI and in many respects more suitable than it, for it uses the modalities of human senses and skills for input: NUI stems from the conviction that technologies should better mirror human capabilities, optimise the path to expert, apply to given contexts and tasks, and fulfil our needs¹². Control by touch or by voice, for example, requires no new knowledge of the user, and so it can be said that this is a natural way of interacting with technology. Concrete examples of NUI might be rotating a photograph with multi-touch gestures on a touch display, the Microsoft Kinect controller, which captures our movements, or the recently introduced headsets for mixed (augmented) reality, which use the physical world as an interface: a wall or part of a room can at any moment become the surface of a scene in a computer game designed for mixed reality.

According to Dourish, tangible computing does not define a concrete form of the interface or of its functionality. It is an umbrella term whose general characteristic is the trend of distributing computation across devices that are sensitive to their physical surroundings and possibly also to other nearby devices¹³.

Dourish further named and introduced the concept of social computing. As with tangible computing, here too the reference is to the real world, but the emphasis is placed on what we might imagine as the “social layer” of technology. The more we use technology, the more it becomes involved in social processes, by which it is influenced and which it in turn influences. The functions, meanings and individual signs of the interface and of technology become parts of the network of the social and cultural background of individual users. Precisely because technology integrates so deeply into our lives, we can say that it co-constructs our lives, our thinking and our acting. According to Dourish, but also to Jan Brejcha, interaction with the interface of technology means, besides the physical acts of clicking and touching, also the communication of social and cultural signs, or in other words, the interface manifests within itself a material and a semantic layer¹⁴, where meaning is not an inherent part of technology but is socially constructed. Socially constructed meaning presages a certain instability (later in this thesis, multistability), even though the materiality or physical layer of the object has remained essentially unchanged. Dourish regards interaction as a dialogue which, much like a real dialogue between two people, has no prescribed sequence of events, but arises as it goes along, so to speak. To understand how interactions and interfaces really work, it is not possible to isolate them from social, cultural and spatio-temporal contexts. Dourish calls this immersion of technologies and interfaces in the real world embodied interaction. Its theoretical core is the fact that people and technologies are not abstract entities somewhere in a Platonic world, but exist in the physical world, and their function, behaviour and acting are influenced by the world. The use and design of technology, interactions and interfaces should thus reflect how the real world works, or rather the ways in which we experience the world¹⁵.

Dourish’s concept of embodied interaction is at odds with what is known as naive Cartesian cognitivism¹⁶, according to which the human mind and its functions can be separated from the body, and the world appears as “a largely stable collection of objects and events to be observed and manipulated according to the internal mental states of the individual”¹⁷. Cartesian cognitivism implicitly claims that a disembodied individual, without participating in any society and culture, can think in the same way as an ordinary person.

Embodied interaction rejects such a view of our mind and of the role of the body, culture and society, and instead ascribes to them a participatory role in the constitution of human action and of the meaning of objects. In other words, the way in which people will use our technical and design creations cannot be determined in advance from the “designer’s armchair”, for the meanings and functions of objects arise only in interaction, and do so on several levels. Dourish distinguishes three levels of meaning of a technological object, a category into which the interface also fits:

1. the meaning of an object given purely by its physical or material constitution,
2. socially constructed meaning
3. and the meaning belonging to some system of practice, or rather to how a certain group of people uses the given object for its activity. The third way is referred to in the literature by a term from anthropology, “community of practice”, that is, a community of people who share the common practices of some activity.

The concept of embodied interaction is not entirely new; it draws on the philosophical heritage of phenomenology, which is concerned with the analysis of human experience. As I will show further on in the section where I analyse post-cognitivist HCI theories, the main contribution of embodied interaction and of the use of phenomenological analysis is the coupling of action and meaning, for according to these approaches meaning is not in the technology or interface itself, but is rather found in the way in which they are used and in how we interpret the functions and meanings of objects on the basis of our previous knowledge as well as of the current context. Embodied interaction and phenomenology thus declare a close coupling of human and world and a rejection of the disembodied user, who is not an objective, even mathematising observer of the world, but is rather essentially immersed in the world, in which they must practically cope with everyday problems at hand. For HCI this means above all a shift of attention away from technologies and objects themselves, viewed statically and abstractly, towards their concrete way of being used, or — and, given the declared interest of the field of HCI, this ought to sound almost tautological — towards the real interaction of human and computer (technology). I will now offer the reader a somewhat more conventional view of the historical development of HCI.

Notes

17 notes

1. 1↑

   COOPER, Alan, Robert REIMANN and Dave CRONIN, 2007. About face 3: the essentials of interaction design. Rev. ed. Indianapolis: Wiley, xxxv, 610 pp. ISBN 978-0-470-08411-3, pp. 271–275.

2. 2↑

   MANOVICH, Lev, 2002. The language of new media. Cambridge: MIT Press. ISBN 0-262-63255-1, p. 80.

3. 3↑

   BOLTER, Jay David and Richard GRUSIN, 2010. Imediace, hypermediace, remediace. Kapitoly z dějin a teorie médií. Praha: Akademie výtvarných umění v Praze, Vědecko-výzkumné pracoviště, pp. 69-93. Edice VVP AVU. ISBN 9788087108161. In English: “the mediating nature of technology”.

4. 4↑

   Cited in: BOLTER, Jay David and Richard GRUSIN, 2010. Imediace, hypermediace, remediace. Kapitoly z dějin a teorie médií. Praha: Akademie výtvarných umění v Praze, Vědecko-výzkumné pracoviště, pp. 69-93. Edice VVP AVU. ISBN 9788087108161, p. 86. In English: “becomes second nature, an external as well as internalised environment and an object of desire. There is no longer any need to make it transparent, simply because it is not felt to be at odds with the ‘authenticity’ of experience”.

5. 5↑

   If I claim that the GUI is a window onto constructed images without an original of their own, I do not mean to deny that the GUI, and the interface in general, can contain, for example, photographs that refer to existing things in the world. My argument is that even though some individual elements of the interface may refer to something existing, the interface as a whole has no original, for, as I comment further on in the text, it is a synthetic, creative act which, to put it in Heidegger’s terms, makes present the non-present.

6. 6↑

   MANOVICH, Lev, 2002. The language of new media. Cambridge: MIT Press. ISBN 0-262-63255-1, p. 76, p. 118.

7. 7↑

   MANOVICH, Lev, 2002. The language of new media. Cambridge: MIT Press. ISBN 0-262-63255-1, p. 118.

8. 8↑

   BREJCHA, Jan, 2015. Cross-cultural human-computer interaction and user experience design: a semiotic perspective. CRC Press. ISBN 9781498702577, p. 2.

9. 9↑

   HEIDEGGER, Martin, 2004. Věda, technika a zamyšlení. Praha: OIKOYMENH. Knihovna novověké tradice a současnosti. ISBN 8072980831.

10. 10↑

    EEDE, Yoni Van Den, 2012. Amor technologiae: Marshall McLuhan as philosopher of technology: toward a philosophy of human-media relationships. Brussels: VUBPRESS. ISBN 9789057181870, pp. 167–168.

11. 11↑

    WIGDOR, Daniel and Dennis WIXON, 2011. Brave NUI world: designing natural user interfaces for touch and gesture. Amsterdam: Morgan Kaufmann. ISBN 978-0-12-382231-4.

12. 12↑

    WIGDOR, Daniel and Dennis WIXON, 2011. Brave NUI world: designing natural user interfaces for touch and gesture. Amsterdam: Morgan Kaufmann. ISBN 978-0-12-382231-4, p. 9.

13. 13↑

    DOURISH, Paul, 2004. Where the action is: the foundations of embodied interaction. Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-04196-6, p. 15.

14. 14↑

    Cited in: BREJCHA, Jan, 2015. Cross-cultural human-computer interaction and user experience design: a semiotic perspective. CRC Press. ISBN 9781498702577, p. 3.

15. 15↑

    DOURISH, Paul, 2004. Where the action is: the foundations of embodied interaction. Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-04196-6, p. 17.

16. 16↑

    I will deal with cognitivism in more detail in the section Cognitivism in HCI.

17. 17↑

    DOURISH, Paul, 2004. Where the action is: the foundations of embodied interaction. Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-04196-6, p. 18.

Literature

8 sources

• BOLTER, Jay David and Richard GRUSIN, 2010. Imediace, hypermediace, remediace. Kapitoly z dějin a teorie médií. Praha: Akademie výtvarných umění v Praze, Vědecko-výzkumné pracoviště, pp. 69-93. Edice VVP AVU. ISBN 9788087108161.
• BREJCHA, Jan, 2015. Cross-cultural human-computer interaction and user experience design: a semiotic perspective. CRC Press. ISBN 9781498702577.
• COOPER, Alan, Robert REIMANN and Dave CRONIN, 2007. About face 3: the essentials of interaction design. Rev. ed. Indianapolis: Wiley, xxxv, 610 pp. ISBN 978-0-470-08411-3.
• DOURISH, Paul, 2004. Where the action is: the foundations of embodied interaction. Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-04196-6.
• EEDE, Yoni Van Den, 2012. Amor technologiae: Marshall McLuhan as philosopher of technology: toward a philosophy of human-media relationships. Brussels: VUBPRESS. ISBN 9789057181870.
• HEIDEGGER, Martin, 2004. Věda, technika a zamyšlení. Praha: OIKOYMENH. Knihovna novověké tradice a současnosti. ISBN 8072980831.
• MANOVICH, Lev, 2002. The language of new media. Cambridge: MIT Press. ISBN 0-262-63255-1.
• WIGDOR, Daniel and Dennis WIXON, 2011. Brave NUI world: designing natural user interfaces for touch and gesture. Amsterdam: Morgan Kaufmann. ISBN 978-0-12-382231-4.