Post-cognitivist HCI: A Brief History and Theory of Interaction: Electrical, Symbolic and Textual Interaction (Part 4)

A Brief History and Theory of Interaction

Since this thesis is concerned with the interface as one of its main themes, it will be fitting to take a brief look at the historical development of the interface and of interaction. I am inspired by Lev Manovich and his classic work The Language of New Media, in which he describes the development of the screen, of the interface and of software in general as a history of increasing abstraction¹. Here I also follow the strategy of the HCI theorist Paul Dourish, who sees the history of interaction as “a gradual expansion of the range of human skills and abilities that can be incorporated into interaction with computers”². In both cases I thus continue the narrative about the interface that I began in the previous section with an analysis of the first pioneers of HCI: the interface as augmenting human abilities and also as a reduction of complexity, which is masked by an increasing abstraction of how we access the functions of the computer.

From a historical point of view, the difficulty of operating a computer has indeed always fallen dramatically whenever the real complexity of the hardware, and later also of the software, was masked: the rewiring of mechanical connections was replaced by machine code, and low-level assemblers added an abstract symbolic layer, but were still too tightly bound to specific hardware. Only high-level programming languages, the text-based user interface of the command line and finally graphical, auditory, tactile or, in the case of game consoles and VR, kinaesthetic user interfaces abstracted away transistors and operating systems sufficiently. We should regard all these stages of controlling the computer as interfaces. Or, alternatively, as a new layer of abstraction of interaction that came closer to human thinking, thereby also modifying their mutual relation and bringing new user experiences.

Rather than describing in more detail the individual elements that made up, or make up, the user interface, Dourish describes the development of the interface according to the fundamental interactions that the interface made possible. According to Grudin, whom Dourish cites, the development of the interface and the history of interaction had an identifiable direction of development: from a technical focus, interactions turn more and more towards incorporating the physical and social world, as the context in which every user is inevitably situated³.

Dourish distinguishes five stages in the development of interactions: electrical, symbolic, textual, graphical and, finally, the pair of tangible and social computing, which in his work Dourish treats as two sides of the same coin, a coin he calls Embodied Interaction. Influenced by Heidegger’s philosophy, he argues that what they have in common is that they draw on our knowledge of the everyday world, specifically the world of physical artefacts and of social interaction, in which we are not objective, abstracted or disembodied observers, but rather an integral part of the physical as well as the social world. Dourish introduced the notion of embodied interaction both as a new stage of interaction and as a theoretical and philosophical critique of the existing practice of designing interactive systems.

Electrical Interaction

The Analytical Engine of Charles Babbage and Ada Lovelace made use of the movements of mechanical parts. In the first half of the 20th century, purely mechanical machines were replaced by the first analogue computers. Thanks to the discovery of Shannon, who proposed the possibility of simulating the principles of Boolean algebra in the movement of electricity directed by components such as vacuum tubes, relays, capacitors and later also transistors, there was a gradual shift from the first analogue computers to digital computers.

The first digital computers, however, suffered from the same ills as the analogue ones before them: they were machines designed to compute specific operations, for example partial differential equations. Reprogramming them to compute other tasks was either not possible at all, or it was necessary to reconfigure the very physical components that made up the computer. Let us recall the room-sized ENIAC computers, hard to imagine today, where programmers had to rewire the cabling and reset various switches by hand.

Of the relation between human and computer it can be said that it was uneven and that technical aspects wholly predominated. There could be no question of communication with the computer; the human was not a user of the computer, but rather its operator, indeed its servant, for interaction consisted in physically rewiring the internal parts of the computer. Today’s division into hardware and software did not yet exist. If we can speak of interaction at all, it was very difficult and awkward, for the human could not rely on a more abstracting interface and had to interact with the “naked electro-mechanical body” of the machine.

Symbolic Interaction

It was only the first computer program stored in the computer’s internal memory and the integrated instruction set of the processor that heralded the arrival of the computer for truly general use. At the same time, according to Dourish, this marks a turning point in the evolution of interaction with computers. Using the computer was no longer so bound to the physical implementation of the machine. With the arrival of programs stored in memory and of instruction sets supplied by processor manufacturers, the operators of the computer became programmers, for they were now provided with a layer of abstraction. The programmer could control the registers and accumulators of the processor without having to deal with how they physically work, first by writing strings of binary or hexadecimal characters and later also by using “assemblers”, or assembly languages. People are given certain symbolic codes (instructions) that can move or add values between registers. The advantage of machine language and assemblers was, thanks to abstraction, a looser tie to the physical implementation of the machine.

It was Grace Hopper who was instrumental in creating COBOL, one of the first high-level programming languages. High-level programming languages further abstracted the control of the computer’s internal mechanisms into the writing of specific operations that were much closer to natural language, specifically English. Other high-level programming languages followed which, although they moved away from natural language, introduced their own grammar and vocabulary in the form of conditions, loops, jumps, variable declarations, classes, objects, instances and so on.

The use of programming languages began to demand of people a greater emphasis on mathematisation and algorithmisation, thereby necessarily disciplining human thinking. With the spread of digital technologies, these skills became as important and prestigious in society as the use of natural language itself, if not more so, as Vilém Flusser argues in his essay⁴. We can also see efforts by politicians and experts to get programming into the curricula of primary and secondary schools.

Dourish writes that the transition to symbolic interaction and programming meant that interaction with computers now began to draw on other human skills, ones in which we are, after all, more adept. Symbolic interaction was cleansed of the physical layer of the computer and began to resemble communication in natural language.

Textual Interaction

According to Dourish, however, symbolic interaction did not represent real interaction with the computer, for that requires the “interactive loop” of two-way communication, in which the computer responds to the user’s commands in real time. For according to Dourish, programming probably cannot be regarded as interactive, because the code must first undergo a transformation into machine language, which, depending on the type of programming language, is handled by compilers or interpreters. And that takes time. At least it did when Dourish was writing his book. If the speed of the computer’s response is the main condition by which we divide symbolic and textual interaction, I would venture to argue that, at the time of writing this thesis, Dourish’s condition and division may no longer hold. Programmers of web applications routinely use online tools in which they write code in a programming language of their choice (for example JavaScript in online tools such as CodePen⁵) and immediately receive feedback on the commands of the code.

By textual interaction Dourish has in mind, rather, the prototypical configuration of a text terminal and a keyboard as the input device used for interacting with computers. He writes that textual interaction was explicitly brought to the fore, and the user no longer defined the internal structure of commands, but declared the desired functions that the computer was to carry out by itself: for example, filtering the files in a folder by name, answering a question asking to confirm the deletion of a file, and so on. I think that the dichotomy of the imperative and declarative programming styles, taken from the field of computer science, is apt here as an analogy for the difference between symbolic and textual interaction: in the imperative style the programmer must define the behaviour of the computer precisely, whereas in the declarative style the programmer uses ready-made functions and does not care how they are implemented, but merely declares what is to be done. This is, once again, a further layer of abstraction. The user now works with a new grammar and a lexicon of commands made of English words and phrases, which reduce the complexity of using the computer and hide the concrete implementation of their behaviour.

After the arrival of the graphical user interface, textual user interfaces were retrospectively given the name Text-based User Interface (TUI). Among concrete examples of this type of interface we can count, besides the command line of the console, the once widely used two-pane file managers and, by Dourish’s criteria, perhaps also computer games belonging to the genre of interactive fiction.

Notes

5 notes

1. 1↑

   MANOVICH, Lev, 2002. The language of new media. Cambridge: MIT Press. ISBN 0-262-63255-1, p. 115.

2. 2↑

   DOURISH, Paul, 2004. Where the action is: the foundations of embodied interaction. Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-04196-6, pp. 16–17.

3. 3↑

   DOURISH, Paul, 2004. Where the action is: the foundations of embodied interaction. Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-04196-6, p. 5.

4. 4↑

   FLUSSER, Vilém, 2013. Alfanumerická společnost. BOSÁK, Petr and Robert JANSA. Proto: grafický design a současné umění. V Praze: Tranzit.cz, pp. 73-81. ISBN 9788087259221.

5. 5↑

   www.codepen.io

Literature

3 sources

• DOURISH, Paul, 2004. Where the action is: the foundations of embodied interaction. Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-04196-6.
• FLUSSER, Vilém, 2013. Alfanumerická společnost. BOSÁK, Petr and Robert JANSA. Proto: grafický design a současné umění. V Praze: Tranzit.cz, pp. 73-81. ISBN 9788087259221.
• MANOVICH, Lev, 2002. The language of new media. Cambridge: MIT Press. ISBN 0-262-63255-1.