Post-cognitivist HCI: The First Pioneers of HCI: Bush, Licklider, SAGE and Engelbart (Part 3)

Introduction

In this chapter I describe the roots of the historical relation of the field of HCI to cognitive science and psychology, as well as the ideas of the first pioneers in HCI, who regarded their work as augmenting human intellect. I then move on to a deeper analysis of which axiomatic assumptions HCI adopted under the influence of the paradigm of cognitivism, and of how this showed in its conceptualisation of the human being and of human cognition. Next I analyse a selection of important critical responses to cognitivism, which will be followed by the core of this section, an analysis of post-cognitivist HCI theories, among which I count phenomenology, postphenomenology and the theories of extended and distributed cognition.

An Unconventional History of HCI: The First Pioneers of Extended Cognition and of Interface Theory

Vannevar Bush and Extended Memory

The first pioneer is Vannevar Bush, with his influential article As We May Think of 1945. In it Bush introduced a hypothetical device, the Memex (MEMory EXtended), which would make it possible to edit and quickly find any information stored on microfilm and linked by what are known as trails. The proposal to link documents by non-linear links that would simulate the associative thinking of the human brain was a revolutionary idea and inspired other researchers. Bush is widely cited in the literature on the history of hypertext and the beginnings of the web, and according to the Czech information scientist Jiří Cejpek he had an undeniable impact on the field of information science¹. Yet in the article we also read about a first sketch of a new way of interacting not only with the device itself, but also with information. Bush proposed a visual interface, resembling today’s tablet with a stylus as its input device, through which a person could work and interact with a vast database of knowledge, unimaginable for the human mind and memory. That is also why Bush conceived his Memex as an extension of human cognition and “an enlarged intimate supplement to his memory”².

J. C. R. Licklider and Man-Computer Symbiosis

J. C. R. Licklider built directly on Bush. Originally a researcher in psychoacoustics, Licklider studied the way in which the brain converts the mechanical vibrations of the air into an electrical signal that we interpret as sound. The calculations of his research became too demanding, and so he decided to focus on how to automate routine calculations and repetitive operations. He answered his question in 1960 by publishing the article Man-computer symbiosis, in which he presented an expected development in cooperative interaction between men and electronic computers, which, taking his inspiration from living nature, he called a symbiotic relation. Licklider assumed that in the near future computers would noticeably surpass humans in operations that can be reduced to algorithms. Such operations should therefore be left to them, and the role of the human would shift to defining goals, formulating hypotheses and evaluating results. According to Licklider, handing over a certain part of thinking was also meant to answer questions about how human cognition works. Compared with Bush, however, Licklider rejects the idea that computers should be merely a human extension, that is, an amplification of what humans can already do; they should rather give humans entirely new abilities. By symbiosis Licklider stresses the living together in intimate association of two dissimilar organisms. In this human–computer relation both sides will contribute to the activity in equal measure, so that it will be difficult to separate them neatly in analysis³. Dissimilar organisms, however, need some kind of mediator in order to communicate. For computers work in real time at an entirely different frequency. The task of masking and balancing out this difference falls to the interface. Such a function of the interface proved indispensable in the American project of the SAGE radar system, in which Licklider took part as an expert on human factors.

The Interactive SAGE System in the Context of Cognition Research

The Semi-Automatic Ground Environment, or SAGE, was the US Air Force’s defence system of the mid-1950s against a potential attack by the Soviet Union. It was the first computer network and an assemblage of all the American radar stations and of the first interactive screens, on which the raw data from the radars were displayed as dots, as visual information about the position of enemy aircraft. The operator of such a display could use a light pen to order the computer to track a given dot. Lev Manovich notes that the SAGE system contained all the main elements of the modern human-computer interface, since, besides displaying “reality” and information in real time, the display was the vehicle for directly affecting reality⁴. The operator of the SAGE system sat in front of a screen that was a window into a massively distributed military-technological complex, in which the human components “were fully integrated into the mechanized circuit of detection, decision and response”⁵.

Paul Edwards⁶ maps the historical context of the time in which Licklider wrote about man-computer symbiosis. The SAGE system was one of many efforts by the American military and by academic intellectuals within the discourse of the political hegemony of the USA in the “closed world”, in which the main strategy of containment would be realised in a possible total and centralised defence against the enemy. Edwards argues that the striving to control the integration of human and computer for military purposes drove research in artificial intelligence and psychology⁷. Edwards’s analysis further shows that the cybernetic understanding of human and machine as equal components in a system, the influence of Shannon’s information theory of communication and the need to integrate “human factors” into military-technological systems led to research into the human mind under the new paradigm of what is known as cognitivism. As I will show later in this thesis, research into human cognition rejected the behaviourist model of stimulus and response and began to study cognition on the basis of internal representations and the information processing of symbolic information, whereby human thinking came to be viewed through the metaphor of the computer.

Douglas Engelbart, Augmented Intellect and the Interface

Of Engelbart, another of the first pioneers of HCI, the literature usually mentions that he was the designer of the first computer mouse or the creator of the oN-Line System (NLS) software, which he presented to the public during his time at the Stanford Research Institute (SRI). The NLS system was responsible for many firsts: besides using the computer mouse as an input device, it contained a graphical user interface, displayed on a bitmap display, that used “windows” for multitasking and for displaying several applications at once. It was also the first software designed for several users to work together, one that contained hypertext link technology, and it brought many other firsts⁸. Although the NLS system rightly deserves the attention of historians of hypertext and the web, it belongs only to a later stage of Engelbart’s endeavour to augment human intellect and to enable people to cope with the complexity of the problems that are part of modern life.

For if we look at the article Augmenting Human Intellect: a Conceptual framework⁹ of 1962 (published two years after Licklider’s article), we see in it the whole theoretical foundation that Engelbart tried to realise from then on. For this thesis, what is most relevant in the article is the description of the influence of technology and language on human cognition, and of the function that the interface will play in augmenting human intellect.

Right at the beginning of the article, Engelbart explains that by “augmenting human intellect” he means increasing the capability of a man to comprehend and solve, more rapidly and better, situations that had previously been too complex. According to Engelbart, our culture gives us several means of augmenting the intellect. Besides learning and training, and certain methods of reaching a goal, our intellect is augmented by language and (technological) artefacts. Engelbart sees these four elements, completed by the human, as parts of a single integrated system, which he calls H-LAM/T after the initial letters of its parts. Because these parts of the whole are interconnected and interact with one another, Engelbart writes that a change in one part can call forth latent properties and capabilities in another part, and thereby fundamentally change the whole system. For Engelbart, the place of great potential for augmenting human intellect is precisely that part, the “artifacts”, for which the arrival of the digital computer and the Internet means a potentially great leap.

Engelbart did not attempt to speculate on whether natural human intelligence itself can be increased, but he believed that the symbiotic whole and notional human-machine cyborg, or H-LAM/T system, would in the end be more than just its parts. As far as intelligence is concerned, Engelbart on the one hand unambiguously situates human intelligence in the neurons of the brain, but at the same time writes that this is the lowest layer of a whole in which each higher layer provides greater abstraction and “phenomenological sophistication”. At the same time, it is in the interplay and synergy of these layers that, in the final analysis, we should look for intelligence. In other words, intelligence is distributed across the components of the H-LAM/T system, and is thus an emergent property arising from the interactions of the individual components of the system.

Engelbart’s conception of the H-LAM/T system included the human and the technological artefact as its most important parts, which have their own, incompatible processes and thereby form two separate domains. In order for them to interact and cooperate with each other, which is Engelbart’s condition for the whole H-LAM/T system to function, it is necessary to introduce a mediator that would balance out their incompatibility. In H-LAM/T this mediator is the interface. The term “interface” had already been used earlier in the context of technology, under the name “man-machine interface”. Perhaps also because at that time there was not yet such a dominance of the graphical user interface (GUI) as we see today, Engelbart could (or even had to) look at the interface much more abstractly and universally. As is evident from the diagram below, taken from the 1962 article, Engelbart did not understand the interface as something that is part of the artefact as such, as our present-day myopic vision, influenced by the “screen” and the visuality of the interface, would suggest, but rather as a boundary or limit “across which energy is exchanged between the two domains”¹⁰, or as something in between, a mediator, or better, a medium.

Figure 1 Diagram of Engelbart’s H-LAM/T system

What exactly does the interface mediate? According to Engelbart, the interface creates a relation and couples the components of the human and of the artefact, which have special “matching” processes that abstract their inner complexity. The interface is thus the relation of these surface and abstracting layers, which, however, are themselves the result of much deeper processes. By way of explanation, Engelbart gives the example of a finger and the key of a classic typewriter. He writes that the finger and the typewriter key are the resulting “matching” processes of deeper processes: in the human, of those responsible for producing the text in question, and in the artefact, of those that print the letter in ink on the paper.

By preferring the term “man-artifact interface”, Engelbart distances himself from the idea that computers or other machines are the only examples in which the interface plays a role. Instead, he defines the interface universally as the relation between the human and an artefact (for example a pencil, paper, a computer, a power station, the particle accelerator at CERN).

Since the existence of human artefacts reaches back into prehistory, it is open to question whether we really can analyse our relation to artefacts, mediated by the interface, in such general terms. Perhaps that is why Engelbart qualifies his definition by saying that, between the “components” of the human and the artefact, the interface mediates a relation that leads to higher, composite processes.

But what are composite processes? A composite process is any activity that can be carried out only with the successful cooperation of both components, the human and the artefact. Engelbart does not comment on what we should regard as a composite process and what not, and so we might perhaps at least mark out some bounds of the spectrum within which his definition makes sense. At one end of the spectrum there might be, for example, the operation of a particle accelerator, where we cannot do without technological artefacts and the interface; at the other end there is, say, walking across a field, where we can do without the artefact (shoes), even if it will not be pleasant. For our purposes let us regard everything else as lying in a grey area and as subject to the philosophical interpretation of the given situation, which may, surprisingly, mean that in a certain situation a quite clearly non-composite process such as human thinking can be problematised: not only with a diagnosis of Alzheimer’s disease, but also when we are rushing to work, a smart mobile device can serve us as an artefact without which we would not manage the task in question (calling the right number).

Whether this example is a proper instance of augmenting human intellect, or whether we can regard a smart mobile device as part of our cognitive system, is a question that points to the current theoretical debate in the fields of cognitive science and the philosophy of mind, on which this thesis will draw in the following sections. Specifically, in the section devoted to post-cognitivist HCI theories I will present theories that have greater potential to explain what role artefacts play in our cognition in the context of current research. And not only artefacts. Already from Engelbart’s diagram and from the article itself it is clear that he regarded human cognition and his H-LAM/T system not as what in the terminology of systems theory and cybernetics¹¹ is called a closed system, but as an open system, which receives information from its context. And that context is the external world, by which we should understand not only the geography of nature, but also social and cultural influences. For Engelbart’s project socioculture was very important, for it was precisely socioculture that generated the various means of augmenting our intellect, among which language and artefacts affect us the most.

A Summary of the First Pioneers of HCI and the Computer Interface

Each of the pioneers mentioned gave the field of HCI, which did not yet exist at the time, ideas and inventions that we take for granted today. They are rightly written about as pioneers of hypertext, the graphical user interface, software for team collaboration and so on. Amid all these achievements, however, the main motivation that led them to devote themselves to the interaction of humans with computers may remain hidden. In my view, a reading of the original articles shows that all of them saw the various input devices, the graphical user interface and technology “merely” as intermediate steps on the way to augmenting human cognition and increasing our capacity to work with information in an age of information overload. This should then help us to deal with the growing complexity of everyday life, but also with far-reaching scientific and social questions, among which today we would count climate, ecological and humanitarian problems.

It was above all Engelbart who, in the context of HCI, worked out the idea of augmented human intellect, in which the biological side of the human being and the human brain are part of a cybernetic system or cyborg organism, in which the human interacts with technology and with the external environment. For this interaction and “exchange of energy”, the interface, which smooths over the differences between the biological and the artificial components, is indispensable. In this picture the interface becomes a cognitive mediator, which necessarily has to reduce and abstract the real complexity of the world or of the internal mechanisms of technology. For it is precisely this reduction and veiling of reality that is, seemingly paradoxically, the prerequisite for our gaining augmented skills that would otherwise, without the reduction of complexity, not be possible. This is an idea that will appear several more times in this thesis.

Notes

11 notes

1. 1↑

   CEJPEK, Jiří, 2005. Proměny let devadesátých: knihovnictví na prahu informačního věku - děje, myšlenky a názory. Praha: Svaz knihovníků a informačních pracovníků ČR. Aktuality SKIP. ISBN 80-85851-15-6, pp. 28–29.

2. 2↑

   BUSH, Vannevar, 1945. As we may think. The Atlantic [online]. July 1945, (JULY 1945 ISSUE) [cited 2018-05-05]. Available at: https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/

3. 3↑

   LICKLIDER, J. C. R., 1960. Man-Computer Symbiosis. In: IRE Transactions on Human Factors in Electronics I. HFE-1. pp. 4-11, my emphasis.

4. 4↑

   MANOVICH, Lev, 1995. An Archeology of a Computer Screen. Kunstforum International. Germany. Available also at: http://manovich.net/index.php/projects/archeology-of-a-computer-screen

5. 5↑

   Cited in: MANOVICH, Lev, 1995. An Archeology of a Computer Screen. Kunstforum International. Germany. Available also at: http://manovich.net/index.php/projects/archeology-of-a-computer-screen

6. 6↑

   EDWARDS, Paul N., 1996. The closed world: Computers and the politics of discourse in Cold War America. Cambridge, Mass.: MIT Press. ISBN 978-0262050517.

7. 7↑

   EDWARDS, Paul N., 1996. The closed world: Computers and the politics of discourse in Cold War America. Cambridge, Mass.: MIT Press. ISBN 978-0262050517, p. 20.

8. 8↑

   In the article A lifetime pursuit (Engelbart, 1986), Christina Engelbart describes a total of 24 features that the NLS system was the first to introduce. Besides those already mentioned, we find among them, for example, hypermedia publishing, hypermedia email, version control of electronic documents, teleconferencing with screen sharing, context-sensitive help, or the fact that the NLS system established the first “internet”.

9. 9↑

   ENGELBART, Douglas C., 1962. Augmenting Human Intellect: a conceptual framework. Director of Information Sciences Air Force Office of Scientific Research,. Washington 25, D.C. Available also at: http://www.dougengelbart.org/pubs/papers/scanned/Doug_Engelbart-AugmentingHumanIntellect.pdf

10. 10↑

    ENGELBART, Douglas C., 1962. Augmenting Human Intellect: a conceptual framework. Director of Information Sciences Air Force Office of Scientific Research,. Washington 25, D.C. Available also at: http://www.dougengelbart.org/pubs/papers/scanned/Doug_Engelbart-AugmentingHumanIntellect.pdf, pp. 20–21.

11. 11↑

    VON BERTALANFFY, Ludwig, 1968. General System Theory: Foundations, Development, Applications. New York: George Braziller. ISBN 0-8076-0453-4.

Literature

8 sources

• BUSH, Vannevar, 1945. As we may think. The Atlantic [online]. July 1945, (JULY 1945 ISSUE) [cited 2018-05-05]. Available at: https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/
• CEJPEK, Jiří, 2005. Proměny let devadesátých: knihovnictví na prahu informačního věku - děje, myšlenky a názory. Praha: Svaz knihovníků a informačních pracovníků ČR. Aktuality SKIP. ISBN 80-85851-15-6.
• EDWARDS, Paul N., 1996. The closed world: Computers and the politics of discourse in Cold War America. Cambridge, Mass.: MIT Press. ISBN 978-0262050517.
• ENGELBART, Christina, 1986. A Lifetime Pursuit: a brief history of Doug Engelbart's work by Christina Engelbart [online]. [cited 2018-05-06]. Available at: http://www.dougengelbart.org/history/engelbart.html
• ENGELBART, Douglas C., 1962. Augmenting Human Intellect: a conceptual framework. Director of Information Sciences Air Force Office of Scientific Research,. Washington 25, D.C. Available also at: http://www.dougengelbart.org/pubs/papers/scanned/Doug_Engelbart-AugmentingHumanIntellect.pdf
• LICKLIDER, J. C. R., 1960. Man-Computer Symbiosis. In: IRE Transactions on Human Factors in Electronics I. HFE-1. pp. 4-11.
• MANOVICH, Lev, 1995. An Archeology of a Computer Screen. Kunstforum International. Germany. Available also at: http://manovich.net/index.php/projects/archeology-of-a-computer-screen
• VON BERTALANFFY, Ludwig, 1968. General System Theory: Foundations, Development, Applications. New York: George Braziller. ISBN 0-8076-0453-4.