Anticipation lies at the foundation of the human being's entire cognitive activity.

Interactive Multimedia Navigation

(with C. Lango) lCVIR Workshop (Content Visualization and Intermedia Representations), University of Montreal, Canada, 15 August 1998

(with C. Lango) lCVIR Workshop (Content Visualization and Intermedia Representations), University of Montreal, Canada, 15 August 1998


Mihai Nadin and Clemens Lango


We understand knowledge as an infinite sequence of associations between what we know and what we are in the process of acquiring as knowledge. The most important cognitive processes supporting knowledge acquisition are associations. How associations come into being is difficult to describe because association processes occur in open systems. Associations that have taken place can be documented, refined, or, if some turn out to be irrelevant, discarded. Associations are almost always multimedial, i.e., we associate texts, sounds, pictures, motion, etc.

The structure of an interactive, multimedia encyclopedia based on associations includes

1. a knowledge space/domain,
2. an associative search procedure,
3. a function for storing associative traces.

Obviously, we no longer support learning as a form of information consumption, but rather as a form of knowledge generation.

Interactive Multimedia Navigation
(Navigation Tools for Knowledge Dissemination via Interactive Multimedia)

Until now, problems of navigation have in the main been reduced to the research of interface. Aspects of metaphor constitution, of the role of visual language, of the understanding of feedback mechanisms, as well as of the role of back channels, were pushed into the background. Moreover, questions pertinent to knowledge acquisition, expression, understanding, and application are still only marginally approached. Consequently, navigation is defined for applications pertaining to homogenous sets of data, such as numerical representation, images, and texts. Indeed, relatively stable navigation methods and tools support text-based dissemination of knowledge, applications of mathematical intent, and simple ways to handle visual information by means of zooming or by animation procedures.

Although the community of scientists dedicated to the design and implementation of distributed collaborative learning systems is aware that the real potential of the digital dissemination of knowledge lies in media integration, very little has been done to specifically deal with the integration of image (produced as computer graphics, photographs, film, animations, videos), text, audio, and other media. Even less has been done in developing navigation concepts, methods, and instruments appropriate to this complex task.

Until now, navigation has been predominantly considered independent of forms and means of representation. Representation, however, is a constitutive dimension of gnoseological activity, not just a theme of formal description and focus. Accordingly, those involved in the cognitive aspects of learning realized that it is essential to conceive, design, and optimize navigation tools in respect to representation models, and not only to what is still naively defined as content. Various possibilities and restrictions are intrinsic to a chosen representational mode – assuming that the requirement of adequacy is effectively met. It is not the same thing when navigation takes place in a 3-dimensional space as when it takes place in 2-dimensional text display. We can read of molecular docking or we can experience it.

The representation chosen (text description, graphical illustration, or virtual reality context) affects to a high degree the nature and the outcome of the learning experience. Texts can be disseminated in various forms (from e-mail to scholarly publications). Graphic representations can range from simple diagrams to animated sequences. However, these forms are interactive in a limited sense only. They can be integrated in Web interactions, but not in cooperative forms of learning. A virtual reality environment can be distributed and accessed from many parts of the world, and experienced in a cooperative/collaborative effort.

In recent times, with the emergence of channels and active desktops, the dynamic characteristics of navigation tools have again changed. We know “pull” methods (learner finds a site and pulls a subject in some format varying between a PS download of text to a chat room), as well as “push” methods. These apply to “subscriptions” made in the network by those pursuing a specific course of learning. In both cases – push and pull – the subject of navigation must reflect the need for customization (learning is an individual experience regardless of where and when it takes place).

Our research at the Computational Design Program is focused on the concept of mind ergonomics. This term exactly addresses the expectation of customization, i.e., the individual nature of learning based on how minds, as opposed to systems, work. The three aspects that make up the object of our research are:

1) the expression of knowledge through interactive multimedia;
2) navigation tools corresponding to the expression of knowledge through interactive multimedia;
3) mechanisms for updating both the body of knowledge represented through interactive multimedia methods and the appropriate navigation tools.

A multimedia system for digital dissemination of knowledge usually consists of editors, search tools, and browser. Within this model, those using it still face shortcomings such as getting “lost in hyperspace;” “spaghetti appearance” (interference of data streams); and desynchronization (between navigation methods and new knowledge, between the representation and inner connection of this new knowledge).

Our alternative mind ergonomic approach addresses the following aspects:
the constitution of vast knowledge spaces, the best example being the Web;
the evolution of communication models (one-to-one, one-to-many, many-to-many);
the variety of forms of access, from the traditional question-answer model to hyperlinks and time scaling.

This approach suggests the following data-mining procedure: the constitution of associative links and the automatic generation of associative maps corresponding to different types of learning and investigation. In order to accomplish this goal, we suggest a multimedia database with powerful indexing and classification functions, eventually with resident intelligent procedures. This database is complemented by an intelligent search procedure able to handle multimedia representations of knowledge.

The underlying information is provided with a gnoseological map of known interpretations, as well as with an indexing utility in which the learner or researcher advances new hypotheses or new interpretations as these arise through the continuous progress of knowledge acquisition and discovery.

In developing the navigational tools, we addressed the constitutive elements of the mind-ergonomic structure. Specifically, this refers to the variety of cognitive types of learners, exemplified, for instance, in question formulation through text queries or through visual or aural queries.

Answers, although independent of these cognitive types, are made available in a variety of formats corresponding to those types. They are collected and submitted to further evaluation through the program and through the users themselves.

The major cognitive process on which the project focuses at this time is association. The difficulty in dealing with these processes is that they constitute an open system. Once constituted, associations can be documented, refined, discarded if they are irrelevant, or submitted for future evaluation by the learner and by researchers. In terms of programming, associations require the involvement of fuzzy models and a fully supported interactive multimedia environment. (A text can be associated to sound, image, motion, etc., and combinations thereof).

In the knowledge space that we define, learners can pursue a one-dimensional inquiry, the classic example being historical analysis, i.e., the gnoseological sequence. Or they can pursue a multi-dimensional inquiry, for instance, the relations between a theory in physics and the broader context of knowledge and action, including identifiers such as aesthetic value, dominant cognitive characteristics, the state of technology, etc. We are motivated in this approach by the notion that knowledge dissemination cannot and should not be reduced to retrieval mechanisms.

Learning minds, i.e., people interacting in the process of learning in a digitally supported environment, are actually defined in a process of discovery. The active component of the act of learning is what we in fact pursue. Accordingly, the navigation tools developed are tools for making such interaction possible. Especially in a context of information overload, learners have to be able to prune what appears to them as the seemingly endless knowledge trees provided by digital systems.

The result of learning is expressed through the associative map generated by the system as the learner/researcher gets closer to understanding the subject researched. After several inquiries, the learner/researcher will “own” several such associative maps and will be able to continue the learning process at the level of generality or abstraction at which new knowledge can be generated.

Obviously, we no longer support learning as a form of information consumption, but rather as a form of knowledge generation

Posted in Downloadable Articles, Downloads, LectUbiquitous, Lectures/Presentations, Ubiquitous Computing & Digital Media

copyright © 2o16 by Mihai Nadin | Powered by Wordpress

Locations of visitors to this page