The middle road leads nowhere

Visible Signs – the language of Multimedia

Signs and Systems. A semiotic introduction to Systems Design, Cambridge: University Press, Summer/1997.

Signs and Systems. A semiotic introduction to Systems Design, Cambridge: University Press, Summer/1997.
Prof. Dr. Mihai Nadin
Artikel in: Signs and Systems, A semiotic introduction to Systems Design,
Cambridge: University Press, erscheint im Sommer 1997
Within homogenous types of computation – from number crunching to symbolic processing – it was possible to infer from previous communication knowledge (e.g., linguistics, logical formalism, symbolism) to computer processes.
With the advent of multimedia, data types and communication processes multiply to the extent that prior communication knowledge no longer adequately explains meaning constitution. Time-independent (also called discrete) data, such as texts or graphics, and time-dependent data, such as sound and animated pictures, fuse into a new entity. Synchronization mechanisms are essential for the integration of the continuous and the discrete.
Multimedia editors allow us to manipulate and eventually integrate multimedia data types. At the intersection between multimedia-integrated data and non-sequential data structures, represented by the notion of hypertext, an entity called hypermedia emerges.
Fig.1 Multimedia, hypermedia and hypertext
This chapter argues in favor of a semiotic perspective centered around a high level generality sign concept. Of no less importance is the encompassing notion of infinite semioses (sign processes). The sign concept in question is not reducible to the particular signs making up the integrated multimedia data. Neither is it reducible to the known data structures (sequential or non-sequential).
In order to deal with the unity between data types and structures, a dynamic sign concept has to be introduced. In a way, we start from the end, i.e., from semioses (sign processes) expected to be infinite. This expectation reflects the intrinsic characteristics of the computational process through which we merge defined data (sound, text, image, motion) in order to achieve a level of multimedia abstraction and, hopefully, platform independence. Based on these premises, the dynamics of communication specific to multimedia results from semiotic interactions best designed by means of storyboards similar to those that shaped modern cinematography.
As opposed to any form of sequential closed communications, hypermedia requires means for and ways of generating an infinity of meaningful interpretations. A nonlinear structure is, after all, a graph constituted from nodes and links. The semiotic level of such nodes and links is quite abstract, but without a good understanding of these communicational entities, we will never exercise an efficient command of the process of generating the infinity of meaningful interpretations just mentioned. For this to happen, interfaces should provide a transparent language of interaction, while simultaneously protecting the novelty of relations among various data types. In other words, the pragmatic condition of multimedia is such that what is made transparent (navigation and retrieval) actually obfuscates what eventually will be found and retrieved. The tension between the how of the multimedia interaction and the what of conjuring meaning is expressed in the aesthetics of multimedia.
As an identifiable underlying level of the multimedia communication process, aesthetics indeed is its formative component. Through design geared towards multimedia integration, aesthetics provides a unifying strategy that does justice to the variety of data types and ensures the open character of communication processes. To program for multimedia is not to simply mix sound and text and animation and other elements, but to see how and under what circumstances they augment each other or can substitute for each other.
Hotel reservation system
With the advent of networking, and especially under the influence of the semiosis of hypermedia, Web publishing made it all the more critical to understand the semiosis of multimedia. Here again, what defines the approach is the pragmatics of distributed tasking and cooperative conjuring of meaning.
A case study for an international hotel reservation system located on the Internet provides insight into how the pragmatics defines all other semiotic levels (semantics and syntax) and how sign processes are rendered effective by aesthetic devices such as associations, contrasts, comparisons, or by the estasblishment of powerful metaphors that direct the navigation.
Fig. 2 Model of the user interface of Hotel Reservation System.
In simple terms, the object of enterprise is to design an interaction space. The pragmatics of the application is straightforward: define the interaction between information stored in multimedia formats regarding available hotels and the query of a person searching for a lodging defined by expectations (subjective and objective) selected from among many possible choices: price, location, comfort, etc.
The assignment translates semiotically into the constitution of nodes (supersigns) representing an ideal desired hotel, and links between such nodes and data representing what is actually available at each moment in time.
The sign process of search is designed in order to take advantage of the multiple data types (pictures, video sequences, text descriptions, icons), but also to allow for choices in a less rich environment of possibilities. In fact, the semiotic decision to design a limited but expressive choice space followed a storyboard strategy. The narration of finding a hotel has its parallel lines corresponding to the way sound, motion, light, color, and other characteristics contribute to the cinematographic synthesis.
Fig.3 The narration of finding a hotel.
Once this diagram emerged, the technical aspect of connecting an existent database to the Internet hypermedia structure proved to be quite simple. The underlying aesthetics is not reducible to how we illustrate search criteria. Its main elements are the links, entities of almost cinematographic quality, but within a non-sequenti-al film, i.e. each person looking for a hotel room is writing his or her own film. That aesthetics becomes implicit in multimedia semiosis is probably the concluding observation of this case study.
Serghei Eisenstein. The Film Sense. 1943
Mihai Nadin. Negotiating the World of Make-Believe: The Aesthetic Compass, in Real-Time Imaging, January, 1996 – .
The Art and Science of Multimedia, in Real-Time Imaging (P. Laplante and A. Stoyenko, Eds.) Piscataway: IEEE Publishers. 1995
Terry Winograd. Understanding Computers and Cognition. A New Foundation for Design. Norwood NJ: Ablex Publishing Corp. 1986

Posted in Downloadable Articles, Downloads, Semiotics

copyright © 2o19 by Mihai Nadin | Powered by Wordpress