(special edition, Clarisse Siekenius de Souza, Ed.).
Amsterdam: Elsevier Science Publishers. Vol. 14/8, 5 December 2001, pp. 437-440
Monsieur Jourdain: And this, the way I speak. What name would be applied to…?
Philosophy Master: The way you speak?
Monsieur Jourdain: Yes.
Philosophy Master: Prose
Monsieur Jourdain: It’s prose. Well, what do you know about that! ?These forty years now, I’ve been speaking in prose without knowing it.
(Molière, The Bourgeois Gentleman)
Abstract:
As the digital becomes part of the underlying structure of human existence and activity (just as electricity was integrated in the infrastructure), human interaction will be less and less direct. Mediated through various interfaces, human interaction via all kinds of networks becomes increasingly an expression of the semiotic condition of the human being in the post-industrial age. Professionals dedicated to human-computer interaction (HCI, a more suitable acronym than CHI) and semioticians must realize that they would benefit mutually if they would collaborate better than they have until now. Therefore, this paper submits working hypotheses to HCI practitioners and to semioticians dedicated to aspects of human interaction.
A position paper on human-computer interaction, beginning with a quote from literature is unusual. Instead of starting with bandwidth and chip spec considerations, and instead of joining the lamentation over I/O limitations, I submit that the problems of HCI are not primarily technological, but human. This is why the acronym HCI is better suited than CHI, and will be used in this text. As an aside, I suggest the HCI be adopted because this formulation establishes the cognitive framework: Humans initiate interaction. Through interactions, computation returns information and knowledge to the human being. This is why all those involved in HCI need to understand what goes on in it. More precisely, we do not know enough about forms of human interaction mediated by entities such as written commands, images, sounds, etc. that qualify as signs. All those involved in human-computer interaction “speak” semiotics, whether they are aware of it or not. In paraphrasing Paul Watzslawick’s famous axiom – One cannot not communicate – I submit (again) to the HCI community that one cannot not interact. And because interaction is based on signs, one cannot not “semiotize”; that is, one cannot avoid semiotics. In other words, unlike prose, as a label for our common use of language, semiotics affects interactions the way our awareness of a process affects our understanding or that process and our ability to control it, that is, to optimize, augment, or fine tune it. Indeed, we express ourselves through various signs; we interpret them, and thus become part of the infinite sign process. Our practical experiences (work, learning, various transaction, etc.) are more and more semiotic in nature, i.e., more and more mediated by signs, visual or of other types. The computer is a semiotic engine. After all, it is not electrons that users are interested in, but the information processed and the new interpretations made possible. This is why the computer accelerated the semiotization of human life, including the semiotization of the interaction between humans and machines as well as among individuals involved in distributed activities.
What the quote from Molière should afford the reader is a very simple understanding. Yes, we all speak prose. And even those who are not aware of this can be good at it. The meta-level (definition of prose) and practical experience (speaking to each other) are not co-dependent. If we take Monsieur Jourdain to be an HCI engineer, we face a relatively similar situation: The meta-level (definition of HCI) and the practical experience (computation) are, however, co-dependent. In order to make effective computation possible, the CHI professional has to provide means of interaction that do not stand in the way (“Learn the interface”), that do not become a cognitive “overhead”, but rather a cognitive support. Computational experiences should not be dependent upon the meta-level of HCI awareness. The case I am making through the formulation “One cannot NOT interact” is actually one in favor of computational ubiquity [1]. As is the case with every broad statement, it means close to nothing unless it is substantiated. Here are the arguments for my axiom:
Interactions can be direct: raising one’s hand in greeting, pounding on a table, pulling a lever, cracking an egg, for example. Or they can be mediated: showing a picture in reference to a physical phenomenon, touching a soft-button on a copy machine in order to execute a desired operation, or clicking on an icon in order to connect to a Web address. A simulation is a mediated interaction through which we settle in the representation of what is simulated under the assumption that some coherence is preserved between the real and the represented. This is the underlying cognitive assumption of semiotics.
Our interaction with computers is mediated by various I/O means, methods, and procedures. There is no way to avoid this condition of semiotic mediation. In the analog world, direct interaction is still sometimes possible. In the digital realm, a language (of two digits, 0 and 1) and a grammar (Boolean logic) stands between the user and the hardware.

The intersection area – where and how the mediation takes place, i.e., how interactions occur – is of interest from the perspectives of many disciplines: psychology, cognitive science, technology, education, communication theory, design, economics (how effective is the interaction; what costs does it entail?), culture (to what extent does it fit the cultural context), among others. Each of these disciplines can affect our understanding of HCI and can, within the framework of methodological investigation, support the implementation and evaluation of user interfaces and other aspects of interaction. Some of these perspectives are more important than others; some are easier to consider when proceeding with the task of making HCI possible and effective. The only characteristic impossible to eliminate is the semiotic characteristic, because regardless of the nature of the interaction considered, it always takes place through the intermediary of signs (more precisely, sequences of 0 and 1, according to rules of encoding and decoding expressed in the adopted format, e.g., of 32-bit machines, 64-bit machines, etc.). This intermediary appears to the user in forms close to our culture: words, actions (e.g., clicking on a mouse, that is, sending an interrupt signal), images, sounds. All these form the object of semiotics. They constitute an underlying element similar to language (but not reducible to it), in the absence of which no interaction is possible.
Monsieur Jourdain had spoken prose all his life without having the slightest inkling of it. People around him understood what he had to say. The same thing holds true of HCI in respect to semiotics, but only in respect to describing how a certain sign works. A minority of those involved in the field is aware of semiotics above and beyond an elementary understanding of how a word or an image is used in order to facilitate an action. Without exception, HCI professionals want to gain access to the secret of designing effective means and methods of interaction. Monsieur Jourdain confessed, “I am in love with a lady of great rank and quality and wish to ask your help in writing her a note.” Not unlike him, HCI professionals are dedicated to (and sometimes in love with) the computer and would like all the help they can get in order to make the interaction better and more intuitive. They want to support a variety of interactions corresponding to the variety of users. Eventually they try to reach that evasive goal (but a very assuring slogan!) of user-centered interaction, also known (in Norman’s vocabulary [2]) as human-centered design.
Short of understanding how signs are constituted and how interpretations emerge in a pragmatic context (i.e., what is called sign process); moreover, short of understanding what the relation between the machine, the program, and the user is, the HCI community will at best continue to provide solutions that in fact reduce the effectiveness of the computation it is asked to support instead of enhancing it. Actually, the HCI community is aware of that fact that only a small fraction of what is possible through computation is realized. And again, this is not because of technology (which is ahead of us), but rather because of HCI that is not yet well conceived. The user does not need to know the meta-language (semiotics). However, the HCI professional cannot avoid learning it. In view of all this, I argue for the following practical goals:
1. The necessity of a semiotic foundation.
As long as the complexity of computation was very low (due to the fact that computers are in our day still in a very early stage of development) and the interaction limited, the HCI community has been able to get away with even rudimentary semiotic notions or with an intuitive procedure. The following assessment refers to both semioticians and HCI professionals: All kinds of dilettantes and peddlers of ideas picked up during a conference coffee break have been successful in promoting metaphors, visual conventions, and I/O devices whose packaging was better than their real performance. Let’s face it, after the desktop metaphor, which was conceived on a semiotic foundation, almost nothing of similar impact has emerged. As the complexity of computation increases, richer interactions are possible. Especially as networking poses new challenges irreducible to the artistry of self-declared information architects and screen designers, interactions take new forms. Henceforth, the necessity of a semiotic foundation will make it impossible for HCI professionals to avoid studying the various aspects of interaction that networking brings about. Let me point to the relation between the represented and the representation, the dynamics of representation, visual codes, interpretation, multimedia, to name only a few of such aspects. (For more details, see Nadin, 1988 and the descriptions under Course of Study at the Website www.code.uni-wuppertal.de.)

My Computational Design Program [3] and Joseph Goguen’s program at UCSD are dedicated to the exploration of the semiotic aspects of interaction. The Group at the Pontificia Universidade Catolica in Rio de Janeiro pursues semiotic engineering as a well-defined field of applied semiotics (focused on communication) and information science. In Denmark, the Centre for Human-Machine Interaction (CHMI) is making progress along the line I suggested. The Workshop on Semiotic Approaches to User Interface Design, CHI 2000 provided a framework for exchanging ideas and experiences based on this work.

2. Integration of semiotic considerations at the level of operating systems.
Since an operating system is the program through which we address the computer (hardware, software), it follows that HCI considerations can no longer be limited to the cosmetics (or illustration) of the operating system’s functionality, but need to start with the design of the operating system. This is where interaction is pre-defined. At this high level, semiotic considerations should go hand in hand with technical considerations. Whether multi-user, multiprocessing, multitasking, or multithreading, operating systems are high-level semiotic entities that act as intermediaries between applications and users. Regardless of what kind of representations are used (line commands, icons, dial controls, etc.), these are signs mediating the expression of practical goals with the aim of using the knowledge embodied in the machine (at the hardware level and in software expression) for certain practical purposes. This is why the integration of semiotic considerations at the level of operating systems is fundamental for improved human-machine interaction.
3. Semiotic foundation of academic programs in HCI.
In order to optimize the use of computation, we need to address the complexity of HCI. This interaction is not reducible to technological innovation, but essential to its success. Thus, it follows that there is a need for establishing academic programs dedicated exclusively to the study of interactions in all possible computing environments. These must integrate, as their foundation, the scientific and technological aspects of computation, as well as the semiotics of computation. In particular, design, which also has a semiotic foundation, has to become a subject matter for all computer science programs. Interactions can be enhanced and made richer by design; they can be customized and adapted to individual characteristics. The issue is not screen design, as some tend to see it, but rather communication in a multimedia environment that supports HCI.
4. Complement usability with semiotic adequacy evaluations.
Last but not least: The obsession with user performance, expressed in usability studies, has to give way to a balanced understanding of what we evaluate. Now is the time to complement usability with semiotic adequacy evaluations (concerning types of signs, context and meaning, cultural characteristics). Some concrete aspects (communication, to give the example that Sickenius et al [cf. Prates, 4] focused on) of semiotic adequacy should be approached systematically. We need to look into how various signs (visual, aural, tactile, etc.) complement each other. If we learn about semiotic synergy, we might be able to optimize means and methods of interaction in order to minimize the computational cost of interfacing, while at the same time improving interaction.
As the digital becomes part of the current underlying structure [5] of human existence and activity – just as electricity was integrated in the underlying structure of industrial society – human interaction through networks becomes increasingly an expression of the semiotic condition of the human being. Even if, ad absurdum, the HCI community will continue to resist semiotics (regardless of reasons such as ignorance, laziness, captivity to models of interaction pertinent to pre-computation experiences, etc.), semiotic awareness will eventually help in freeing ourselves from the hodge-podge of unsystematic, obscure, and half-baked formulas still dominating the field. In not too distant a future, very few will be as surprised as Monsieur Jourdain was in finding out that he spoke prose. Not unlike those musicians who cannot read a musical script but who are, at times, very good performers, HCI professionals will have to study “music,” i.e., semiotics. In the concert of a great pianist, the many hours of study and exercise totally disappear in the genius of the performance. Semiotics, if understood and practiced with the same attention to precision and expression as logic and mathematics, disappears in HCI. But it makes possible interactions that would not arise without it.
In asking whether the HCI community is willing to study semiotics with the same dedication that it studies mathematics, logic, computer science, etc., we set an expectation that might be unrealistic. The alternative is the “art” model: great designers do something only they know how and why, and which by some miracle works. This romantic alternative is in fact none. HCI cannot be reduced to “nice” images, to “attractive” screen designs. Aesthetics is important, but only once the design of the interaction is right. The design of interaction is actually the “script” of a sign process (semiosis). And as such, this semiosis is anticipatory: a future state (one of many possible, for instance, in designing an application) of the system defines the current state. To design this kind of interaction is not possible without understanding how signs are perceived, interpreted, and used in a given context.
As computation becomes more ubiquitous, languages of interaction will emerge and become part of our general culture. The complexity behind an electric switch is no less impressive than that behind a computer application. But in the switch, we see a semiotic experience embodied, one that is probably comparable to a semiotic experience embodied in a nuclear power plant. HCI should not set its goals lower than that. The complexity and effectiveness of computation depend on HCI. And HCI is depends upon our ability to provide means of interaction that are both precise and expressive.
This work was partially supported by a grant from the Deutsche Forschungsgemeinschaft (DFG, German Research Association).
References
[1] Nadin, Mihai. The best computer is invisible, in Computer Art Faszination. 1996. Dr. Dotzler Media Institute: Frankfurt/Main, pp. 210-212
[2] Norman, Don. The Invisible Computer. Cambridge: MIT Press, 1998.
[3] Nadin, Mihai. Computational Design. Design in the Age of a Knowledge Society — HTML , in formdiskurs, Journal of Design and Design Theory, 2, I/1996. Frankfurt/Main: Verlag Form, pp. 40-60.
[4] Prates, R., C de Souza, S. Barbosa. A Method for Evaluating the Communicability of User Interfaces, in Interactions, January + February 2000, pp. 31-38.
[5] Nadin, Mihai. The Civilization of Illiteracy. — HTML Dresden: Dresden University Press, 1997.
Bibliography
Andersen, Peter Bogh. A Theory of Computer Semiotics. Cambridge: Cambridge University Press, 1990.
(1995). The force dynamics of interactive systems. Towards a computer semiotics, Semiotica 103-1/2, 5-45.
Nadin, Mihai. On the meaning of the visual: 12 theses regarding the visual and its interpretation; and The discipline of interdisciplinarity (Introduction), in Semiotics of the Visual: On Defining the Field. (Mihai Nadin, Ed.) = Semiotica, vol. 52, no. 3/4 (1984). Amsterdam: Mouton, 1985
Visual semiosis applied to computer graphics, in Annual Conference Proceedings of the ASEE. Hanover, PA: The Sheridan Press, 1986.
The meaning of the image, in Semiotics 1982 (J. Deely, J. Evans, Eds.). Lanham, New York, London: University Press of America, 1987.
Interface design and evaluation, in Advances in Human-Computer Interaction, vol. 2 (R. Hartson, D. Hix, Eds.). Norwood, NJ: Ablex Publishing Corp., 1988.
Design and Semiotics, in Semiotics in the Individual Sciences, vol. II (W.A. Koch, Ed.). Bochum: Brockmeyer, 1990. pp. 418-436.
What Difference Does Digital Make? in New Media Showcase, 1:1, 1991, pp. 5-8
Nadin, M and Zakia, R. Visual Aducation, in Semiotic Theory and Practice, (M. Hertzfeld, L. Metazzo, Eds.). Berlin: Mouton de Gruyter, 1988, pp. 1237-1243.
Nake, Frieder (1994) Human-computer interaction: signs and signals interfacing, in Languages of Design 2, 193-205.
Prates, R., C de Souza, S. Barbosa. A Method for Evaluating the Communicability of User Interfaces, in Interactions, January + February 2000, pp. 31-38.