Overview of this page
This page is a fast-navigation toolset for getting information about modalities, centered around three modality taxonomy tree representations and a list of generator modalities. All modalities are described following a common template.
Modalities are different ways of representing and exchanging information among people and between people and computers. Modalities have very different properties that make them apt for very different purposes of information representation and exchange.
The static graphics conceptual diagram below (Tree 1) shows a taxonomy of all modalities in the media of graphics, acoustics and haptics, organised into linguistic, analogue, arbitrary, and structure modalities.
Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.
Legendum: the abbreviation lab.-keyw. means labels or keywords. An arrow (» ) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, the generic level launches 20 numbered modality families whose descendancy lines are being kept track of using simple alphanumeric notation.
Taxonomy tree organised by medium [back to top, Tree 1, Tree 2, Tree 3, List]
The taxonomy tree below (Tree 2) has the same contents as the original taxonomy above but a different structure because it has been organised into modalities belonging to the media of graphics, acoustics, and haptics, respectively.
Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.
Legendum: the abbreviation lab.-keyw. means labels or keywords. An arrow (» ) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, the generic level launches 20 numbered modality families whose descendancy lines are being kept track of using simple alphanumeric notation.
The taxonomy sub-tree below (Tree 3) extends the gesture part of the taxonomy to sub-sub-atomic level.
Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.
Legendum: the abbreviation lab.-keyw. means labels or keywords. An arrow (» ) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, descendancy lines are being kept track of using simple alphanumeric notation.
Generator modalities are those that (i) have been used to generate the numbered modalities in Trees 1, 2, and 3, that is, all modalities at generic, atomic, sub-atomic, and sub-sub-atomic level, and which (ii) do not themselves appear as numbered modalities in those trees. The reason generator modalities exist is that each numbered modality has been systematically generated from a series of distinctions between basic properties, like static/dynamic or graphic/acoustic/haptic. Each basic property defines a class of generator modalities.
For example, to generate modality 6a, spoken discourse, we need the concepts of acoustics as a medium, of linguistic modalities, of static and dynamic modalities, and of discourse modalities. None of these appear as numbered modalities in Trees 1, 2 and 3, and only some of them appear at super level in Trees 1 and 2. Rather, they are what the numbered modalities are made of. Thus, the parent of spoken discourse is acoustic language or speech (modality 6) which can be generated from the acoustic medium, the linguistic modalities, and the static/dynamic distinction. To generate spoken discourse from acoustic language, we need an additional distinction among modalities, i.e., the one between discourse, labels/keywords, and notation.
The List below shows all generator modalities. Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.
When looking up a modality from one of Trees 1, 2, or 3, or from the List, the modality's generator modalities, if any, are shown in the first entry, i.e., Inherits properties from.
Legendum: An arrow (->) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, descendancy lines are being kept track of using simple alphanumeric notation.
Template for describing modalities [back to top, Tree 1, Tree 2, Tree 3, List]
All modalities are described following a common template. The current template has five entries:
Inherits properties from:
- lists the modalities from which the modality inherits properties. Inherited properties are not included in the description of the modality but are retrieved by looking up the modality's ancestors and generators in the taxonomy. For instance, speech inherits its property of being omnidirectional from the fact that speech is an acoustic modality. Music, for instance, although it is not speech, is also omnidirectional because it is acoustic. Therefore, the omnidirectionality of speech is not listed under acoustic language (modality 6) but under this modality's ancestor, acoustic modalities.
- Note that sibling modalities, such as spoken discourse, spoken labels/keywords, and spoken notation (modalities 6a, 6b, and 6c), inherit the same set of properties.
Specific properties:
- describes the specific, non-inherited properties of the modality. These properties make the modality different from its siblings, if any, and determine the modality's particular usability for purposes of information representation and exchange.
Aptitude:
- points out example conclusions that follow from the specific properties above, about the modality's aptitude and inaptness for information representation and exchange.
- A modality is apt at expressing a particular kind of information in context if (i) it can express the information; (ii) does so usefully and useably overall; and (iii) is the most useful and useable modality around for the purpose, or is among the most useful and useable modalities for the purpose. In other words, although it is often the case that several modalities can express a certain kind of information in context, these modalities may be more or less apt at it. Everything equal, we always want to choose the most apt modality in context. It is only when, for one reason or another, we cannot use a more apt modality that we have to choose a less apt one for expressing the information.
Descendants:
- lists the modality's descendants, if any, in the current version of the taxonomy;
- discusses any other candidate descendants that have not been included in the current version of the taxonomy, for instance because they are rarely used, have been fusioned with related modalities, or cannot be clearly classified at this point.
Multimodal combination:
- describes whether the modality can be used unimodally, or stand-alone, or requires specific other modalities in order to be used in user-system interaction.
- Note that all modalities can be useful in multimodal combination, i.e., in combination with some, but generally not all, other modalities.
Not a modality. The highest level of abstraction for distinguishing between
modalities. Different categorisations exist at this level, resulting in the differently structured taxonomies shown above.
Not a modality. The second highest level of abstraction for distinguishing
between modalities. This level currently includes members from 20 numbered modality families, many of which have descendants at lower levels.
Not a modality. The third highest level of abstraction for distinguishing
between modalities. This level currently includes members from 9 numbered modality families, some of which have descendants at lower levels.
Not a modality. The fourth highest level of abstraction for distinguishing
between modalities. This level currently includes members from 4 numbered modality families, some of which have descendants at lower levels.
Not a modality. The fifth highest level of abstraction for distinguishing
between modalities. This level currently includes members from 3 numbered modality families.
Inherits properties from: none.
Specific properties: Acoustic modalities,
- are modalities expressed in the acoustic medium. The physical carrier of information is sound and the sensor system is hearing;
- can be perceived by humans (unaided) in the 18 to 20.000 Herz band;
- are transient: you either catch the information the moment it's there or it's gone;
- are omni-directional, i.e., can be perceived in all directions from their source;
- typically do not require visual or limb (including haptic) activity to be perceived;
- when produced by people, are produced by using the body or extensions of the body, such as musical instruments;
- in low-noise environments, are salient to the sense of hearing;
- being perceived at a distance, enable the instant building of a perceptual overview of all currently audible sound (although sounds may drown one another, analogous to occlusion in vision);
- are largely dynamic, but representation can be static as well, allowing freedom of perceptual inspection;
- can often be made "interactively static" by giving the user control over whether to hear acoustic information again;
- can harm users' hearing if too loud for too long;
- have available the following information channels of the acoustic medium, at least: start and end time, loudness (intensity), pitch, duration, temporal order, rhythm, timbre, direction in 1D, 2D or 3D, distance in 1D, 2D or 3D.
Aptitude: Some example conclusions are that acoustic modalities
- are apt at attracting attention and conveying information in low-noise environments;
- are apt for representing and exchanging information with people whose limbs and eyes are occupied;
- are apt for representing and exchanging information with the blind and hard-of-seeing;
- are apt as input by those having hand-arm motor disabilities;
- are inept as output to the deaf and hard-of-hearing;
- must be prevented from hurting and harming users.
Descendants:
- See Tree 2.
- At generic level, acoustic modalities split into four families of linguistic (modalities 2 and 6), analogue (modality 10), arbitrary (modality 14), and structure modalities (modality 18), respectively. No strong reason has been found for generating separate static and dynamic acoustic modalities of any of those. Thus, each acoustic modality in the current version of the taxonomy includes both static and dynamic instances of the modality.
- Similarly, the descendants of acoustic language modality 6 (i.e., modalities 6a, 6b, and 6c) include any contributions from descendants of acoustic language modality 2.
- Due to the reversible pragmatic fusions just described, the current number of acoustic modalities from super level to sub-sub-atomic level is 21 (=1+5+8+2+5) rather than, as it would have been in the most extreme case, 47 (=1+10+22+4+10).
Multimodal combination: Acoustic modalities,
- unless carefully crafted into, e.g., background and foreground acoustics, like in movies, tend to drown one another, like when we cannot hear one another speaking because the music is so loud; or when we can hear neither what is being said in the original speech nor in its simultaneous spoken translation.
Inherits properties from: none.
Specific properties: Analogue modalities,
- represent information through similarity between representation and what it represents, like when an image of a cow has similarities to the cow it represents. If a representation is purportedly a drawing of a cow but does not resemble a cow, then we have the right to ask if what is being represented in the drawing is really a cow.
- need to have at least one measurable perceptual aspect in common with what they represent, like when a larger bar graph representation means a larger amount of something but doesn't perceptually reveal the nature of that something - might be crude oil exports or babies born, the graph would look the same.
- have specificity, i.e., literally share a greater or smaller amount of properties with what they represent;
- in the aspects in which they are analogue, have limited or no interpretational scope: what is being represented is more or less as perceived from the representation;
- are unfocused, don't express in any strictly focused way what the point of the representation is, so could be intended to represent innumerable different contents;
- are limited in their expressive power by specificity and lack of abstraction focus;
- don't constitute a system in the way linguistic representations do, hence are non-linguistic;
- work through similarity, hence are non-arbitrary;
- can typically be judged as to how correct the representation is.
Aptitude: Analogue modalities
- are apt at representing specific properties of everything real or imagined in space and space-time - objects, events, and processes;
- are apt at representing perceptually rich properties of objects, events, and processes;
- are necessary for immersive interaction - the more perceptual richness in the greater number of media, the more immersiveness;
- are apt at supporting the building of overviews of large data sets based on perception.
Descendants: see Tree 1. At generic level, the analogue sub-tree is split between static and dynamic analogue graphics, while such a split has not been found necessary at this point for the media of acoustics and haptics. Analogue acoustic representation is mostly dynamic, analogue haptic mostly static, whereas graphic representation is far more evenly balanced between the static and the dynamic. At atomic level, the analogue sub-tree splits into 4 times 5 modalities of the same type. Among these types, images are by far the broadest and most comprehensive form of representation, whereas maps, compositional diagrams, graphs, and conceptual diagrams have been invented and developed for specific purposes in each case, and originally for the graphics medium.
Multimodal combination:
Inherits properties from: none.
Specific properties: Arbitrary modalities,
- represent information through having been defined ad hoc at their introduction, i.e., using ad hoc-created links between representation and meaning rather than using representations from an already existing system of meaning. For instance, it is arbitrary representation to use grey shading to mark visited web links, or to use randomly generated character sequences for passwords. Arbitrary representation is in contrast to all other
modalities of modality theory, which rely on the systems of shared meaning that we already have;
- often exploit the information channels of some medium, such as using a particular shading to represent visited web links;
- stop being arbitrary when a representation becomes part of the shared meaning of some population, like when the green (bottom) traffic light comes to mean "go";
- if their meanings must be remembered or learnt, impose a learning overhead which increases with the number of arbitrary items to be learned. If users have the slightest chance of forgetting what a particular arbitrary representation means, they will.
Aptitude: Arbitrary modalities,
- should be avoided and should only be used if they can be shown to be preferable to non-arbitrary representation, such as when the latter is too clumsy to use in context or too difficult to perceive;
- if they are to be remembered by people without having to consult an external memory each time, are apt for assigning, temporarily or otherwise, ad hoc meaning to a single representation or to a very small number of representations only. This is especially true when non-literal meaning connotations are being used to help users learn and remember, like when using green colour to represent the most ecological or organic products shown in a graphic bar graph (but see the colour issues below);
- to be perceived and memorised by the intended users, should take into account properties of people, such as colour blindness, colour deficiencies, ability to verbalise perception, and other perception, memory, cognition, and media-related deficiencies and issues;
- to avoid confusion, should never make use of conflicting shared meaning, like when stipulating ad hoc, in a children's game, that "yes" means "no" and vice versa. Calling a company "Apple" and giving it a schematic apple for image logo, on the other hand, is not conflict but merely arbitrary use of "apple", which then confers onto the company the many non-literal meaning connotations that attach to apples;
- by default, should be backed up by external memory explaining their meaning, like when the legendum to a static graphic figure explains the meaning of the different kinds of shading used, assuming that shading cannot be replaced by a better, non-arbitrary, alternative, such as labelling.
Descendants: see Tree 1. Arbitrary modalities (i) have descendants at generic level only, and (ii) only the graphics daugthers have been split into static and dynamic. (i) is because arbitrary modalities appear to be relatively simple in each medium. (ii) is due to the assumption that, so far, the largest number of arbitrary representations by far have been created in the graphic medium, most of the time as static representations but also sometimes as dynamic representations.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image, gesture.
Specific properties: Baton gesture modalities
- are typically made through rhytmic, more or less repetitive movement of the body or some of its parts, like when moving the left or right hand/arm down repeatedly to emphasise points made in speech, or like the music conductor who conducts the orchestra using one or both arm/hands, one hand holding a baton;
- are typically used for emphasis during discourse and for following, or even conducting, the rhythm of a musical performance;
- exist in the media of graphics, acoustics, and haptics;
- are pointless when made stand-alone unless their meaning has been fixed by the discourse context or is apparent from considering the environment in which the behaviour takes place. If someone makes, e.g., head or hand batons out of any apparent context, look for the MP3 player!
Aptitude: Baton gesture modalities
- are apt for making discourse more lively, perceptually rich, and emphatic.
Descendants: See Tree 3 which distinguishes between baton gestures in the media of graphics, acoustics, and haptics.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image.
Specific properties: Body action modalities
- represent virtual or physical, non-communicative action of the body on the world, including other people, during user-system interaction, like when searching for some object in 3D virtual haptic space, doing physical exercise under the eye(s) of a computer tutor, fighting, or making love.
Aptitude: Body action modalities
- are apt for showing how something real or imagined is, was, will, can, or should be done.
Descendants: see Tree 3. Body action is largely haptic and visual (or graphic), since it is quite limited how much we can act acoustically without those actions being acts of communication in the acoustic medium. Apart from screams that shatter glass, the closest we come to acoustic body action may be the groans and puffs we produce during physically exerting body action.
Multimodal combination:
Compositional diagram modalities [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, static/dynamic.
Specific properties: Compositional diagram modalities
- are "analytical images" that provide an analytical account of their spatio-temporal subject-matter, rather than of its mere appearance;
- typically use reduced-specificity images to show the (de-)composition of objects, processes, and events. Reduced specificity helps focus the representation on its analytical message.
Aptitude: Compositional diagram modalities
- have developed for, and are apt for, representing the (de-)composition of spatio-temporal objects, processes, and events, and relationships among parts and between parts and whole.
Descendants: See Tree 1 at atomic level under analogue modalities. The current version of the taxonomy distinguishes between compositional diagrams in three media, and between static and dynamic graphic compositional diagrams.
Multimodal combination:
Inherits properties from: analogue, static/dynamic.
Specific properties: Conceptual diagram modalities
- represent analytical (de-)composition of abstract entities, such as organisations, families, theories, classifications, conceptual structures, models, functionality collections, or taxonomies like the modality taxonomy trees, using various spatio-temporal representation elements.
Aptitude: Conceptual diagram modalities
- have developed for, and are apt for, representing the (de-)composition of abstract entities, and relationships among parts and between parts and whole.
Descendants: See Tree 1 at atomic level under analogue modalities. The current version of the taxonomy distinguishes between conceptual diagrams in three media, and between static and dynamic graphic conceptual diagrams.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image, gesture.
Specific properties: Deictic gesture modalities
- use some part of the body to refer to something in space, such as an object, process, or event, or a group of these, like when using hand/arm stretched and index finger pointing, gaze, head nod, or finger touch, or several of these together, for the purpose;
- exist in the media of graphics and haptics but are rather far-fetched and contrived in the acoustic medium - like referring to particular things of types of things by making them vibrate or even crash when exposed to sound of a certain frequency;
- are typically pointless when made stand-alone unless their meaning has been fixed by the discourse context.
Aptitude: Deictic gesture modalities
- are apt for complementing discourse for making more or less precise spatial reference.
Descendants: See Tree 3 which distinguishes between deictic gesture in the media of graphics, acoustics, and haptics.
Multimodal combination:
Inherits properties from: analogue, static/dynamic.
Specific properties: Diagram modalities
- are analytical analogue representations which show the (de-)composition of spatio-temporal or abstract matters, such as objects, processes, events, concepts, or conceptual structures;
- exist in all media and can be static or dynamic.
Aptitude: Diagram modalities
- have developed for, and are apt for, representing subject-matter decomposition, relationships among parts and between parts and whole.
Descendants: See Tree 1 at atomic level under analogue modalities. The current version of modality theory distinguishes between two types of diagram, compositional diagram and conceptual diagram.
Multimodal combination:
Inherits properties from: linguistic, static/dynamic.
Specific properties: Discourse modalities
- have evolved to serve the purpose of interactive, situation-dependent linguistic communication in which the interlocuters share space, situation and time;
- exist in the media of graphics, acoustics, and haptics;
- are mostly dynamic but can be static as well;
- tend to have a strong rhetorical potential due to information channels that can be used;
- exist in many different tongues in acoustics and graphics, at least;
- is the maximally expressive form of situated language, corresponding to text for non-situated language. Part of the differences between discourse and text, grammatical and otherwise, seem explained by the fact that the former has developed for situated communication, the latter for non-situated communication.
Aptitude: Discourse modalities
- are apt for situated communication;
- are less apt for less situated communication hybrids between discourse and text, in which space, time, or situation are not being shared, like when speaking over the phone or doing spoken or sign language video-conferencing.
Descendants: See Tree 1 under linguistic modalities. The current version of modality theory distinguishes between spoken-acoustic, spoken-visual, and sign language discourse, but does not at this point have a separate entry for the much more rare modality of haptic discourse.
Multimodal combination:
Inherits properties from: none.
Specific properties: Dynamic modalities,
- as opposed to static modalities, do not allow freedom of perceptual inspection: a dynamic representation must be perceived and identified more or less immediately or else it's gone;
- exist in all media;
- can sometimes be brought under the user's control, so that the user can stop the information flow and get already presented information repeated, representation thus becoming "interactively static".
Aptitude: Dynamic modalities
- are generally apt for representing situated linguistic information or discourse;
- are apt for representing not-too-complex, not-too-fast, not-too-slow information to the ordinary human user;
- are apt for representing information that does not need to be studied carefully by the ordinary human user;
- may require experts to be decoded in detail in real time, like when a trained referee perceives far more of what happens in a sports match than does a novice spectator;
- typically must be made "interactively static" to be perceived in full detail, for instance by asking the presenter or actor to repeat, or by recording the presentation.
Descendants: See all trees. Note that a modality tree could be made which classifies all modalities at super level as being either static or dynamic. However, we haven't made such a tree representation of the modality taxonomy.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image, gesture.
Specific properties: Emblem gesture modalities
- are equivalent to linguistic expressions, or, more specifically, an emblem is equivalent to an equivalence class of linguistic expressions that have the same meaning, like the two-finger-palm-forward "V" for "Victory!"; or like when a wolf whistle is equivalent, in current US culture, to the exclamation "how beautiful" and its equivalents (Massaro et al. 2005);
- can be used stand-alone, e.g., without accompanying discourse;
- exist in the media of graphics, acoustics, and haptics;
- may differ in meaning from one culture to another (same body movement or non-linguistic sound, different meaning);
- are more numerous and more frequently used in some cultures rather than others.
Aptitude: Emblem gesture modalities,
- since they are equivalent to linguistic acts of expression, can sometimes be used when no corresponding linguistic act is feasible, proper, or wanted, like when the noise level at the football stadium is way too high for shouting and only graphic gesture emblems work.
Descendants: See Tree 3 which distinguishes between emblem gesture in the media of graphics, acoustics, and haptics.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image.
Specific properties: Expression modalities
- are what make the body and its parts express a person's mental (cognitive, volitional, emotional) and physical state through body and head posture and movement, face, eyes, voice, and touch;
- exist in all media;
- can be static or dynamic.
Aptitude: Expression modalities
- are apt for expressing a person's mental (cognitive, volitional, emotional) and physical state.
Descendants: See Tree 1 under analogue modalities. The current version of the modality taxonomy distinguishes between graphic, acoustic, and haptic expression at sub-atomic level.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image.
Specific properties: Gesture modalities
- are movement of any body part, several body parts, or of the body as a whole to convey, non-linguistically, the shared meaning that can be associated with the movement or with the non-linguistic sound produced by the movement;
- exist in the media of graphics, acoustics, and haptics;
- are typically dynamic, rarely static;
- can be "extended" by, e.g., holding an object in the hand and making the object part of the gesture, like when the music conductor uses a baton;
- are of very different, distinct types all of which exist in all media in which gesture is known to exist. Deictic acoustic gesture is probably the only unlikely gesture type/medium combination, cf. modality 10a1b;
- of type emblem, in particular, tend to differ significantly from one culture to another in number, contents, and frequency of use;
- are being used to very different degrees by different individuals, some people gesturing a lot, others little or not at all;
- are closely related to discourse modalities but in very different ways depending on gesture type.
Aptitude: Gesture modalities,
- have very different aptitudes, depending on the type of gesture (see Tree 3, sub-sub-atomic level). Some, like deictic gesture and sometimes iconic gesture, are near-indispensable for representing certain kinds of information in a discourse context; others, like emblems, are rather like substitutes for discourse; and yet others, like batons, iconic gesture most of the time, and metaphoric gesture nearly all of the time, are primarily apt for making discourse more lively, perceptually rich, imaginative, and emphatic;
- always contribute an element of liveliness to discourse.
Descendants: See Tree 1 under analogue modalities. The current version of the modality taxonomy distinguishes between graphic, acoustic, and haptic gesture at sub-atomic level, and - see Tree 3 - between five types of gesture per medium at sub-sub-atomic level.
Multimodal combination:
Inherits properties from: analogue, static/dynamic.
Specific properties: Graph modalities
- represent quantitative information through analogue means that typically bear immediately recognisable similarity to the subject-matter or domain of the representation. The bars in a bar graph, for instance, don't show what the quantities they represent are quantities of;
- typically represent statistical information or numerical data that may be collected empirically or generated from theories, models or functions;
- exist in the media of graphics, acoustics, and haptics;
- are static or dynamic, the original type of graph being the static graphic graph;
- are informationally equivalent to tables that contain the same information;
- are of several different types, see on Descendants below.
Aptitude: Graph modalities
- having been developed for the purpose in the graphics medium, are collectively apt for representing statistical information and numerical data for purposes of spotting trends, flow, profiles, global relationships, history and projections;
- are perceptually different from tables that contain the same information: tables are preferable for consulting exact numerical information, whereas graphs, in particular static graphic and haptic graphs, are preferable for the purposes mentioned in the previous point.
Descendants: See Tree 1 at atomic level under analogue modalities. Graphs are of many types although no agreed-upon typology has been established yet. The terminology in the field is rather confusing and heavily influenced by the traditional prevalence of graphic graphs, in particular 2D static graphic graphs. Possible types include (1) bar graphs which represent relatively small numbers of individual quantities and enable comparison and the spotting of relationships; (2) pie graphs which represent small numbers of parts of a whole with a view to comparing them in terms of quantity; (3) line graphs ("fever graphs", "curve charts") in which a line connects a series of discrete values collected over time or otherwise, enabling the spotting of trends; and (4) scatter graphs ("scatter plots", "dot charts") showing the distribution of discrete data points and enabling the spotting of flow, profiles, trends, history and projections of time-related and other data. Each of (1) through (4) have several sub-types in addition to existing in both 2D and 3D. All types and sub-types, it would seem, can be realised in both graphics and haptics, whereas acoustics poses problems for most of them, problems which so far have not been addressed much.
Multimodal combination:
Inherits properties from: none.
Specific properties: Graphic modalities
- are the modalities expressed in the graphics medium. The physical carrier of information is light and the sensor system is vision;
- can be static or dynamic, allowing freedom of perceptual inspection or not;
- can harm users' vision if too intense for too long;
- have available the following information channels of the graphics medium, at least: dimensionality 1D, 2D, 3D, and time, resolution, intensity, position, shape, size, texture, colour, distance, direction, orientation, heading, time of occurrence, velocity, duration, rhythm, spatial order.
- Native English-speaking readers should note that ordinary text like the present one is a graphics modality, so the usual distinction between "text and graphics" does not apply in modality theory.
Aptitude: Graphic modalities,
- are apt for enabling the building of an instant perceptual overview of broad segments of space due to the character of vision of being perception at-a-distance;
- are apt for users who can see well enough for the purpose at hand;
- are apt for the deaf and hard-of-hearing who can see;
- should be prevented from hurting or harming users.
Descendants: See Tree 2. At generic level, graphic modalities split into linguistic, analogue, arbitrary, and structure varieties, all of which are described separately in their static and dynamic forms.
Multimodal combination:
Hand-written language modalities [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static graphic language/dynamic graphic language.
Specific properties: Hand-written language modalities
- consist of human hand-made graphic written language;
- can be static or dynamic;
- are typically written in the personalised style of the individual writer;
- are more or less legible by others.
Aptitude: Hand-written language modalities,
- since some people write less legibly than others, are less apt than typed text for ensuring legibility;
- are apt for adding a personal touch to the communication.
Descendants: See Tree 1 under linguistic modalities, sub-atomic level, which shows the static hand-written modalities text, labels/keywords, and notation (modalities 5a2, 5b2, 5c2). The corresponding dynamic hand-written modalities are not shown in the current version of the taxonomy but could be easily added as modalities 8a2, 8b2, 8c2. In general, the use of dynamic hand-written language is not recommended, because the issue of legibility of hand-written language gets compounded by the issue of not having freedom of perceptual inspection.
Multimodal combination:
Inherits properties from: none.
Specific properties: Haptic modalities,
- are modalities expressed in the haptic medium. The physical carrier of information is mechanical impact, and the sensor system is touch by means of some part of the body;
- can be static or dynamic, allowing freedom of perceptual inspection or not;
- being accessed through perception-by-contact, could sometimes be likely to hurt or harm users if too pointed, too sharp, too forceful, too hot, etc.;
- have available the following information channels of the haptic medium, at least: dimensionality 1D, 2D, 3D, and time, resolution, position, shape, size, texture, force, hardness, malleability, adhesiveness, liquidity, temperature, distance, direction, orientation, heading, time of occurrence, velocity, duration, rhythm, spatial order.
Aptitude: Haptic modalities
- are apt for conveying information on how objects feel to the sense of touch;
- are apt for users whose sense of touch works well enough for the purpose at hand;
- are apt for the blind and hard-of-seeing who have a sense of touch;
- are not apt for enabling humans to build an instant perceptual overview of broad segments of space. This is due to the character of touch of being perception-by-contact: we cannot build an instant overview of more than we touch with our body and its parts;
- should be prevented from hurting or harming users.
Descendants: See Tree 2. At generic level, haptic modalities split into linguistic, analogue, arbitrary, and structure varieties. No strong reason has been found for generating separate modalities for static and dynamic haptic representations of any kind. Each haptic modality in the current version of the taxonomy includes both static and dynamic instances of the modality.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image, gesture.
Specific properties: Iconic gesture modalities
- are more or less sketchy renderings of objects, processes, or events, like caricaturing how someone else walks or speaks, or simply showing with arms stretched and fingers bent how long was the proverbial fish we almost caught;
- exist in the media of graphics, acoustics, and haptics;
- are typically used in, and can only be reliably interpreted in, a discourse context or in combination with observed events in the environment;
- are being executed in highly personalised ways, more or less - and sometimes very - different from one person to another;
- are being used frequently by some people, and far less, if at all, by others.
Aptitude: Iconic gesture modalities
- are apt for showing aspects of objects, events, and processes that can be hard to explain in language;
- are apt for showing a certain lively style of communication.
Descendants: See Tree 3 which distinguishes between iconic gesture in the media of graphics, acoustics, and haptics.
Multimodal combination:
Inherits properties from: analogue, static/dynamic.
Specific properties: Image modalities,
- have developed for the purpose of recognisably representing real and imagined objects, processes and events in all media;
- represent specific information that must be picked up through perception of the representation itself;
- can be judged as to how sketchy or perceptually rich the representation is.
Aptitude: Image modalities
- are apt at representing specific properties of everything real or imagined in space and space-time - objects, events, processes;
- are apt at representing perceptually rich properties of objects, events, and processes;
- that are perceptually rich, are necessary for immersive interaction and quasi-realistic simulation - the more perceptual richness in the greater number of media, the more immersiveness, the more quasi-realism;
- when perceptually rich, are useful for supporting identification of specific objects, events, or processes.
Descendants: See Tree 1 at atomic level under analogue modalities. Images, both static and dynamic, are an extremely important modality in graphics, which is why the taxonomy distinguishes between static and dynamic image modalities, whereas no distintion is being made yet between static and dynamic haptic and acoustic image modalities. The hypothesis is that both static and dynamic haptic images, and dynamic acoustic images, are extremely important to people as well, which we are gradually realising as we create immersive environments and simulations. It would seem that the distinction between perceptually rich images and sketches is valid across media.
Multimodal combination:
Inherits properties from: linguistic, static/dynamic.
Specific properties: Label or keyword modalities
- are meaningful units adopted from some spoken or written natural language or notation and used to convey isolated bits or chunks of meaning, like the use of isolated written words and phrases in GUI menus;
- exist in the media of graphics, acoustics, and haptics;
- can be static or dynamic;
- are meant to convey meaning when brief, focused information is at a premium, and are common in all media;
- are inherently ambiguous when used, as is typically the case, outside a specific linguistic text, discourse, or notation context;
- exist in many different natural language tongues.
Aptitude: Label/keyword modalities,
- are more or less apt for conveying chunks of meaning outside a specific linguistic text, discourse, or notation context, depending on the extent to which the provided context that is actually being provided helps reduce their ambiguity. For instance, it is often a hard problem to attain unambiguity for each label/keyword entry in a pull-down menu sitting among other pull-down menus accessible from a menu line in a classical GUI, each label/keyword describing abstract functionality;
- if contextually ambiguous and used in quantity for non-professional users, constitute a serious threat against the usability of the system, because most users are likely to use only a small fraction of the system's functionality: they never figured out what all the rest was about;
- are only apt for recipients (people and machines) who can see and understand the particular natural language or notation used or who wish to learn it.
Descendants: See Tree 1 under linguistic modalities.
Multimodal combination:
- menu lines, graph annotations, conceptual diagrams, etc., or when the computer cannot handle natural and spontaneous input and requires spoken or written commands instead
-
Inherits properties from: none.
Specific properties: Linguistic modalities
- express information in a way which is focused and has an open interpretational scope as to the specificity they represent. Focus at any level of abstraction, and limited specificity account for the typical, strong but limited expressive power of linguistic representation. For instance, language focuses with equal ease on all colours or on colour in general, on this particular specimen of colour that we are perceiving right now on this object in these ambient light conditions, and on everything and anything in between. What is not in focus is left to interpretational scope. For instance, the descriptive phrase "a brightly coloured house" may be interpreted as referring to any bright colour and even to any combination of bright colours;
- represent information as compositional syntactic-semantic systems of meaning according to which the meaning of the whole, like the meaning of the sentence "John loves Mary", can be systematically constructed from the meaning of its parts in an approximately rule-based manner;
- are all based on the natural languages, or naturally evolved languages, which come in thousands of varieties, like English, Chinese or Spanish, some of which are lacking a written language and some a current spoken language.
Aptitude: Linguistic modalities,
- are apt at expressing everything that requires abstraction focus, like "a man" or "colour", is abstract, like negation, disjunction, conditionals or variables, or is non-descriptive, like orders or questions;
- are in practice limited in expressiveness by to their cumbersome way of representing specificity, the specific details of objects, processes, and events;
- typically have interpretational scope and are therefore unsuited for specifying spatial detail;
- when situated, have evolved to serve communication when participants share time and location, i.e., physical context or situation, and can perceive and act on the same objects, processes and events. Discourse is the basic form of situated language;
- when non-situated, have evolved to serve communication when participants do not share time and location, physical context or situation, and cannot perceive and act on the same objects, processes and events. Text is the basic form of non-situated language;
- are equally apt at representing information whether or not they do so by means of analogue units of expression. Linguistic modalities that use analogue units of expression, like written Chinese, essentially work like the linguistic modalities that use non-analogue units of expression, like written English. So the fact that languages are syntactic-semantic systems of meaning overrides the use of analogue units more or less completely;
- may gain marginally in rhetorical potential and ease of early learning by having sub-sets of analogue units of expression, cf. the descriptions of modalities 1-4.
Descendants:
- See Tree 1.
- The important modality families are 4a-c and 5-8, but not 1-3 and the rest of 4.
- Regarding modality families 4 through 8, each family has one or several maximally expressive modalities called either text or discourse, and less expressive modalities called labels/keywords or notation. The distinction between discourse and text and reflects whether the modality has evolved to serve the purposes of either situated or non-situated communication.
- The linguistic modalities in the taxonomy do not include various transformations of text and discourse that do not change their nature, such as discourse rendered in writing or text read aloud.
Multimodal combination:
- Some high-expressive unimodal linguistic modalities have evolved for being used stand-alone for a multitude of purposes, in particular the non-situated modalities static graphic text (5a) and static haptic text (7a).
- By contrast, the high-expressive situated language modalities, in particular graphic sign language discourse (4a) and spoken discourse (6a, 8d), are inherently multimodal. Among these, spoken discourse is being used, successfully and unimodally, over the phone for various purposes.
- In general, the natural complement and supplement to linguistic representation is analogue representation (Tree 1), the latter supplying the representation of specificity that is the main weakness of linguistic representation.
- Two different ways of combining linguistic and analogue representation are in illustration and annotation. In illustration, analogue representation has the auxiliary role of illustrating the abstract and general information presented linguistically, typically in text, discourse or notation modalities. Analogue representation typically involves image modalities. In annotation, linguistic annotation has the auxiliary role of establishing focus on selective aspects of the analogue representation. Linguistic annotation is often done using keyword or notation modalities, whereas the analogue representation being annotated could be an image, diagram or graph modality.
Inherits properties from: analogue, static/dynamic, compositional diagram.
Specific properties: Map modalities
- are, in fact, a sub-type of compositional diagram defined by their particular domain of representation;
- provide geometric or topological information about real or virtual physical objects and focus on the relational structure of objects, processes, and events, in order to present location information about parts relative to one another and to the whole;
- exist in the media of graphics, acoustics, and haptics, although acoustic maps (modality 10b) remain to be explored to some important extent;
- can be static or dynamic.
Aptitude: Map modalities,
- having been developed for the purpose in the graphics medium, are apt for supporting navigation in space;
- however, if haptic or acoustic, tend to be far more time-consuming to explore initially than if represented in the medium of graphics.
Descendants: See Tree 1 at atomic level under analogue modalities.
Multimodal combination:
Inherits properties from: analogue, static/dynamic, image, gesture.
Specific properties: Metaphoric gesture modalities
- typically consist of a very sketchy analogue rendering of some source domain in order for it to be interpreted as being about some different target domain;
- exist in the media of graphics, acoustics, and haptics;
- are typically used in, and can only be reliably interpreted in, a discourse context or in combination with observed events in the environment;
- are highly personal and different from one person to another;
- are being used rather often by some people and not at all by others.
Aptitude: Metaphoric gesture modalities
- are apt for showing a certain lively and imaginative style of communication.
Descendants: See Tree 3 which distinguishes between metaphoric gesture in the media of graphics, acoustics, and haptics.
Multimodal combination:
Inherits properties from: linguistic, static/dynamic.
Specific properties: Notation modalities
- are systematic sets of expressions from ordinary language or elsewhere, created for expressing compact and precise information in particular domains, like base 10, propositional logic, chemical notation, dance notation, musical score, or just a simple system for pointing towards north, south, east, and west. In the latter case, systematicity may involve (i) including all of the four main compass directions and (2) carrying out all acts of pointing in a similar way;
- often but not always have specialist users, i.e., people with special training in the domain of the notation, including in how to use the notation itself;
- can sometimes be developed for a new purpose from scratch and within a realistic time frame;
Aptitude: Notation modalities
- are apt, in principle, for expressing compact and precise information in particular domains;
- if well-established, can be assumed to be apt for their special purposes;
- are only apt for recipients (people or machines) who can perceive and understand the particular notation used or who wish to learn it.
Descendants: See Tree 1 under linguistic modalities. Notation has atomic-level descendants for sign language, audio-visual spoken language, and graphic and haptic written language.
Multimodal combination:
Inherits properties from: none.
Specific properties: Static modalities
- allow freedom of perceptual inspection, i.e., all the time in the world for perceptually scrutinising the representation before deciding what to do: it doesn't go anywhere, nor change significantly, in the meantime;
- exist in all media;
- include representations that are being presented repeatedly and with a reasonably short time between each presentation, like a blinking icon, and which are meant to stop only when the user stops them. These representations allow full freedom of perceptual inspection even though they come and go.
Aptitude: Static modalities
- are apt for representing complex or critical information that needs to be studied carefully by the user;
- are apt for supporting careful reading, scrutiny, deliberation, or decision-making;
- are apt for representing analogue information about static entities and snapshots of dynamic entities.
Descendants: See all Trees.
Multimodal combination:
Inherits properties from: none.
Specific properties: Structure modalities
- use perceived similarity and difference in space and time to represent relationships of order, grouping, likeness, difference, and correlation among representations in some modality belonging to the same medium as the structures themselves;
- basically rely on our ability to perceive structure immediately. For instance, when two flocks of geese are crossing the sky, spectators might disagree about all sorts of things, such as the species of birds up there, the number of birds in each flock, a flock's speed, distance, direction, purpose in travelling etc. - but the spectators are unlikely to disagree that there are two flocks, not one, three, or five up there in the first place;
- often reinforce and highlight the represented relationships by using explicitly represented containers, separators, colour and shading, haptic texture, beeps, and other information channels, like for the correlations shown in a graphic or haptic table grid, the functional differences shown in a standard GUI replete with nested containers, or in the use of pause length to indicate topic change in speech.
Aptitude: Structure modalities
- are apt for representing order, grouping, likeness, difference, and correlation among representations in any modality in ways that can be immediately perceived.
Descendants: See Tree 1. Structure modalities (i) have graphic, acoustic and haptic descendants at generic level only, and (ii) only the graphics have been split into static and dynamic. (ii) is due to the assumption that, so far, the largest variety of structure representations has been created in the graphic medium, most of the time as static representations but sometimes as dynamic representations.
Multimodal combination: Structure modalities
- are typically multimodal, i.e., bi-modal or higher. It's hard but not impossible for structure modality representations to be unimodal, like a list of lists, or a table whose cells show different kinds of tables.
Inherits properties from: linguistic, static/dynamic.
Specific properties: Text modalities,
- or written language modalities, are maximum-expressive, non-situated, visual and, at least potentially, tangible natural language;
- exist in the media of graphics and haptics, and, in acoustics, as read-aloud text and dictated text. Text can also be performed in sign language;
- have evolved to serve the purpose of situation-independent linguistic communication. The recipient of the communication would typically be in a different place, situation, and time when decoding the written message compared to the context in which the author wrote the message. This explains some of the differences between text and situated discourse, such as that text is the more rigorously rule-based and formal, more explicit and elaborate, and less rich in information channels, of the two;
- exist as the written form of hundreds of the world's natural languages (although numerous natural languages are still missing written form);
- continue to appear in new, hybrid forms of communication that combine elements of prototypical text and prototypical discourse, such as e-mail, www chat, sms, and audio-recorded formal statements and reports;
- can be static or dynamic, both forms are widespread.
Aptitude: Text modalities
- are apt for non-situated communication;
- are more apt than discourse for preserving personal distance, like when chatting over the Internet with unknown people.
Descendants: See Tree 1 under linguistic modalities.
Multimodal combination:
Inherits properties from: linguistic, graphic, static graphic language/dynamic graphic language.
Specific properties: Typed language modalities
- use a particular graphic unit design (of font or otherwise) to ensure uniformity and reproducibility;
- can be static or dynamic.
- It is virtually always possible to find a unit design that is easily legible by the intended readers.
Aptitude: Typed language modalities
- are apt for ensuring legibility by readers who can see.
- are less apt than hand-written language for adding a personal touch to the representation.
Descendants: See Tree 1 under linguistic modalities, sub-atomic level, which shows the static typed language modalities text, labels/keywords, and notation (modalities 5a1, 5b1, 5c1). The corresponding dynamic typed language modalities are not shown in the current version of the taxonomy but could be easily added as modalities 8a1, 8b1, 8c1.
Multimodal combination:
1. Static graphic language using analogue units [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, analogue, graphic, static.
Specific properties: This entry discusses the special case of static graphic languages that use analogue static graphic elements. The general case of static graphic language is discussed under modality (5). Language does not depend on whether its basic units of expression are analogue or non-analogue. Some written natural languages, however, use or include analogue units of expression as opposed to, e.g., the non-analogue latin letters used in the present text.
Static graphic language using analogue units
- expresses information by means of analogue static graphic elements used as units of linguistic expression, like ancient Egyptian hieroglyphs or Chinese characters.
Aptitude: Static graphic language using analogue units
- whose analogue origins are largely forgotten by the language users, is hardly more apt at linguistic expression, or apt in different ways, compared to languages using non-analogue units. The ordinary Chinese is hardly able to explain the analogue origins of more than a couple of Chinese signs, and thus would seem unlikely to be influenced by the analogue origins of Chinese signs at all. The aptitude of such languages is thus the same as that of their non-analogue counterpart (modality 5);
- whose analogue aspects are vividly present to visual perception by the language user, like the hieroglyphs once were, might have some special aptitude potential. First stages of learning the language might be facilitated, for instance. So far, however, we haven't identified any current languages of this nature.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination: See static graphic language using non-analogue units (modality 5).
2. Acoustic language using analogue units
[back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, analogue, acoustic, static/dynamic.
Specific properties: This entry discusses the special case of analogue acoustic elements in acoustic languages. The general case of acoustic language is discussed under modality (6). Language does not depend on whether its basic units are analogue or non-analogue. However, many, if not all, spoken natural languages include a subset of words or other elements that have some acoustic similarity to what they represent.
Analogue acoustic linguistic elements
- are analogue spoken elements used as units of linguistic expression, like the English word "slush" which sounds similar to the sound of walking through a layer of melting snow or ice. As language users, we may tend to notice those similarities without being fully conscious of them.
Aptitude: Analogue acoustic linguistic elements,
- provided that people notice their analogue aspect somehow, may have a small aptitude potential over and above that of non-analogue spoken language (modality 6). For instance, if something is wet, using "slushy" terms about it or pronouncing the terms used "slushily" is likely to have the rhetorical effect of increasing immersiveness of presentation.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination: See spoken language using non-analogue units (modality 6).
3. Haptic language using analogue units [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, analogue, haptic, static/dynamic.
Specific properties: This entry discusses the special case of analogue elements in haptic languages. The general case of haptic language is discussed under modality (7). Language does not depend on whether its basic units are analogue or non-analogue. However, haptic language might conceivably include words or other units that have some haptic similarity to what they represent.
Haptic language using analogue units
- expresses information by means of analogue haptic elements used as units of linguistic expression. We are not aware at this point of any analogue haptic linguistic elements. Braille, for instance, converts alphabets into non-analogue haptic units, see modality 7.
Aptitude: Analogue haptic linguistic elements
- might play a role in haptic languages for the blind-deaf-mute, for instance by facilitating fast initial language learning, cf. modality (7).
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination: See haptic language using non-analogue units (modality 7).
4. Dynamic graphic language using analogue units [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, analogue, graphic, dynamic.
Specific properties: Dynamic graphic language using analogue units
- expresses information by means of analogue dynamic graphic elements used as units of linguistic expression. We distinguish between two distinct varieties:
- the first variety is merely a dynamic version of static graphic language modality (1), such as film subtitles or hieroglyphs or Chinese signs scrolling across the display. It is not clear if the analogue nature of the units makes any aptitude differences compared to the use of dynamic non-analogue graphic elements as described for modalities 8a, 8b and 8c;
- the second variety is sign language, primarily the family of gesture-based languages that have been developed to enable the deaf and hard-of-hearing conduct purely visual situated communication, without the need for speech; but also a range of less expressive, more specialised languages. Sign language is produced by means of gesture made by hands, arms, face, head and other body parts.
- Note that not all gestures used in a sign language have analogue origins. In fact, it is an open question if sign languages, given that they tend to include both analogue and non-analogue elements, should belong to modality (4) as they now do, or to modality (8).
- Note also that sign language expressions can be used statically just like spoken expressions can, as in a repeated short message or a sign language icon, cf. modality (6).
Aptitude: Graphic sign language
- is apt for situated communication with the deaf and hard-of-hearing and anyone else who have some skills in interpreting or generating a particular sign language, or who wish to acquire or improve those skills;
- is apt for communication when speech must be avoided because it has high saliency and easily catches unwanted attention, like when commando soldiers proceed by stealth;
- is apt for communication when speech is ineffective, for instance when the noise level is too high, like on an active aircraft carrier deck.
Descendants: This generic-level modality has three descendants in the taxonomy: sign language discourse (4a), sign language labels/keywords (4b), and sign language notation (4c).
Multimodal combination: Graphic sign language,
- being a situated language, is inherently multimodal but apt for being used unimodally (or stand-alone) for various purposes.
4a. Graphic sign language discourse [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, analogue, graphic, static/dynamic, dynamic graphic language using analogue units, discourse.
Specific properties: Discourse is maximum-expressive situated natural language. Graphic sign language discourse
- is full-fledged use of the world's sign languages developed for situated communication with, and among, the hard-of-hearing;
- has the same strong expressiveness as spoken discourse (modality 6a).
Aptitude: Graphic sign language discourse
- is apt for situated communication with the hard-of-hearing who have some skills in visually interpreting, or visibly generating, a particular sign language, or who wish to acquire or improve such skills.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
4b. Graphic sign language labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, analogue, graphic, static/dynamic, dynamic graphic language using analogue units, labels/keywords.
Specific properties: Graphic sign language labels/keywords
- are meaningful units adopted from some graphic sign language and used to visually convey isolated bits or chunks of meaning analogous to the way in which isolated written words or phrases are being used in GUI menus;
- being labels/keywords, are inherently ambiguous.
Aptitude: Graphic sign language labels/keywords
- whether static or dynamic, are more or less apt for conveying isolated bits or chunks of meaning depending on the extent to which the context helps reduce their ambiguity.
- Note that, normally, the recipient of the information, user or machine, must know, or wish to learn, the meaning of each label/keyword in their sign language of origin.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
4c. Graphic sign language notation [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, analogue, graphic, static/dynamic, dynamic graphic language using analogue units, notation.
Specific properties: Graphic sign language notation
- is a systematic set of sign language expressions developed to convey meaning for some special purpose typically involving specialists, such as visual communication with a taxiing aircraft pilot.
Aptitude: Graphic sign language notation
- if well-established, can be presumed to be apt for its special purpose. It is realistically possible to develop new sign language notations.
- Note that, normally, the recipient of the information expressed in the notation, user or machine, must be familiar with the notation or wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
5. Static graphic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static.
Specific properties: This entry discusses static graphic language in general, except for the small contribution made under modality (1) on analogue static graphic elements.
Static graphic language
- using static graphic non-analogue characters or other non-analogue units is illustrated by the present text in latin alphabet characters. Other examples are the Greek and Cyrillic alphabets, or the Braille and Morse code alphabets as visually perceived;
- has developed for non-situated communication with, and among, the seeing, like in mail letters, emails, books, newspapers, or advertisements.
- Note that it doesn't matter whether the static graphic language experessions are electronic, on paper, stones on a mountainside, or otherwise, as long as they are meant to be visually perceived.
Aptitude: Static graphic language
- is more or less apt, depending on the sub-modality chosen, for non-situated communication in letters, articles, books, emails, advertising, or otherwise. People strongly tend to prefer situated language modalities, such as speech (modality 6) or sign language (modality 4) for situated communication;
- has developed into several hybrids in the form of more or less situated uses of static graphic language, such as instant exchanges of email, SMS, and text chat.
Descendants: This generic-level modality has three descendants in the taxonomy: text (5a), labels/keywords (5b), and notation (5c).
Multimodal combination:
Inherits properties from: linguistic, graphic, static, static graphic language, text.
Specific properties: Text is maximum-expressive non-situated natural language. Static graphic text
- is maximally expressive, visual use of the world's written natural languages and related "artificial" languages, like esperanto;
- has developed for non-situated communication with, and among, the seeing;
- exists in many different tongues.
Aptitude: Static graphic text
- is apt for maximally expressive non-situated linguistic communication among seeing people and between seeing people and seeing computers;
- unless well translated, is only apt for recipients (people and machines) who understand the particular natural language used or wish to learn it.
Descendants: This atomic-level modality has two descendants in the taxonomy: static graphic typed text (5a1) and static graphic hand-written text (5a2).
Multimodal combination:
5b. Static graphic labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static, static graphic language, labels/keywords.
Specific properties: Static graphic labels/keywords
- are meaningful units adopted from some graphic written language (natural language or notation, cf. modalities 5a and 5c) and used to convey isolated bits or chunks of meaning, like the use of isolated written words or phrases in GUI menus. In fact, the proliferation of static graphic labels/keywords is one of the hallmarks of the classical GUI;
- being labels/keywords, and hence used outside a linguistic text, discourse, or notation context, are inherently ambiguous;
- exist in many different natural language tongues.
Aptitude: Static graphic labels/keywords
- are more or less apt for conveying isolated bits or chunks of meaning depending on the extent to which the provided context helps reduce their ambiguity. For instance, it is often a hard problem to attain unambiguity for each label/keyword entry in a pull-down menu sitting among other pull-down menus accessible from a menu line in a classical GUI, each label/keyword describing abstract functionality;
- if contextually ambiguous and used in quantity for non-professional users, constitute a serious threat against the usability of the system accessed through the GUI, because most users are likely to use only a small fraction of the system's functionality: they never figured out what all the rest was about;
- if exhibiting short-term repetitive change (and hence static in our sense), provide saliency in an otherwise unchanging static graphic environment;
- are only apt for recipients (people and machines) who can see and understand the particular natural language or notation used or who wish to learn it.
Descendants: This atomic-level modality has two descendants in the taxonomy: typed labels/keywords (5b1) and hand-written labels/keywords (5b2).
Multimodal combination:
Inherits properties from: linguistic, graphic, static, static graphic language, notation.
Specific properties: Static graphic notation
- is a systematic set of static, visually accessible written language expressions developed to convey meaning for some special purpose typically involving specialists, such as for describing a branch of mathematics or natural science, a programming language, music, or dance;
- may include not only parts of ordinary static written text, symbols, etc. but also arbitrary static graphic elements, analogue and non-analogue, see modality (13), and static graphic conceptual diagram elements, see modality (9e).
Aptitude: Static graphic notation
- is apt for expressing compact, precise, high information load contents because of the freedom of perceptual inspection offered;
- if well-established, can be assumed to be apt for its special purpose;
- can be realistically developed for new purposes;
- is only apt for recipients (people and machines) who who can see and understand the particular notation used or who wish to learn it.
Descendants: This atomic-level modality has two descendants in the taxonomy: typed notation (5c1) and hand-written notation (5c2).
Multimodal combination:
Inherits properties from: linguistic, graphic, static, static graphic language, text, static graphic text, typed language.
Specific properties: Static graphic typed text
- uses a particular unit design (of font or otherwise) to ensure uniformity and reproducibility.
Aptitude: Static graphic typed text
- is apt for ensuring visual legibility. It is virtually always possible to find a unit design that is easily legible for the intended readers.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
5a2. Static graphic hand-written text [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static, static graphic language, text, static graphic text, hand-written language.
Specific properties: Static graphic hand-written text
- is hand-made by the individual writer who typically follows, more or less, some personalised style of unit design or other.
Aptitude: Static graphic hand-written text
- since some people write less legibly than others, is less apt than typed text for ensuring legibility;
- is apt for adding a personal touch to the communication.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
5b1. Static graphic typed labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static, static graphic language, labels/keywords, static graphic labels/keywords, typed language.
Specific properties: Static graphic typed labels/keywords
- use a particular unit design (of font or otherwise) to ensure uniformity and reproducibility.
Aptitude: Static graphic typed labels/keywords
- are apt for ensuring visual legibility. It is virtually always possible to find a unit design that is easily legible for the intended readers.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
5b2. Static graphic hand-written labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static, static graphic language, labels/keywords, static graphic labels/keywords, hand-written language.
Specific properties: Static graphic hand-written labels/keywords
- are hand-made by the individual writer who typically follows, more or less, some personalised style of unit design or other.
Aptitude: Static graphic hand-written labels/keywords
- since some people write less legibly than others, are less apt than typed labels/keywords for ensuring legibility;
- add a personal touch to the communication - which, however, is generally pointless because there is little use for labels/keywords in personal communication. As a result, use of hand-written labels/keywords is likely to be viewed either as a draft of the final result or as an expression of lack of resources or professionalism.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
5c1. Static graphic typed notation [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static, static graphic language, notation, static graphic notation, typed language.
Specific properties: Static graphic typed notation
- uses a particular unit design (of font or otherwise) to ensure uniformity and reproducibility.
Aptitude: Static graphic typed notation
- is apt to ensure legibility. It is virtually always possible to find a unit design that is easily legible to the intended readers.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
5c2. Static graphic hand-written notation [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static, static graphic language, notation, static graphic notation, hand-written language.
Specific properties: Static graphic hand-written notation
- is hand-made by the individual writer who typically follows, more or less, some personalised style of unit design or other.
Aptitude: Static graphic hand-written notation
- since some people write less legibly than others, is less apt than typed notation for ensuring legibility;
- adds a personal touch to the communication - which, however, is probably pointless because there is little use for notation in personal communication.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
6. Acoustic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, acoustic, static/dynamic.
Specific properties: This entry discusses spoken language, or the speech modality, in general, except for the small and particular contribution made by analogue words (se modality 2). Spoken language
- is language expressed through speech and meant to be perceived primarily through hearing (but see also modalities 8d, 8e, and 8f);
- has developed for situated communication among, and with, the hearing;
- is natural to the hearing in the sense that it has been learnt from a very early age, possibly starting before birth;
- exists in many different natural language tongues;
- in native and known languages, have very high saliency to humans;
- has available the following information channels, in addition to those of acoustics more generally, at least: voice quality, stress, intonation, dialect, accent, personality.
Aptitude: Some example conclusions based on the properties of spoken language and its ancestors in the taxonomy, are that speech
- is apt for situated communication with the hearing who have skills in generating or interpreting a particular spoken language, or who wish to acquire or improve such skills;
- is apt for linguistic communication with the very large majority of hearing people, including the billion or more who are illiterate;
- is apt for linguistic communication when hands and eyes are occupied;
- is apt for linguistic communication when graphic or haptic language materials are unavailable, inaccessible (no light, gloves on), or impractical, like during many outdoor activities;
- in native and known languages, is apt for attracting human attention even in relatively noisy environments, and hence not apt at all if attracting attention should be avoided;
- is apt for being understood only when a single individual is speaking at a time.
Descendants: This generic-level modality has three descendants in the taxonomy: spoken discourse (6a), spoken labels/keywords (6b), and spoken notation (6c).
Multimodal combination: Spoken language
- generally combines well with non-acoustic, non-linguistic modalities, i.e., non-acoustic analogue, arbitrary, and structure modalities;
- only combines well with other linguistic representations in the same or different modalities, if the contents represented is the same and presented simultaneously, like in subtitling: humans are bad at doing several linguistic or language-like tasks at the same time.
Inherits properties from: linguistic, acoustic, static/dynamic, acoustic language, discourse.
Specific properties: Spoken discourse
- is full, high-expressive use of the world's spoken languages developed for situated communication with, and among, the hearing;
- is the most widespread modality for situated communication among humans.
Aptitude: Spoken discourse
- is apt for situated communication with the hearing who have skills in interpreting or generating a particular language, or who wish to acquire or improve such skills;
- is apt for discussion, negotiation, explanation, instruction, practical communication at work or elsewhere, conversation, chat, and much else that does not require painstaking attention to complex detail for which written language may be more apt;
- has strong rhetorical potential;
- is a salient way of presenting information in a low-acoustic environment;
- can be static in the limiting case, like when repetitively issuing a short message.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: linguistic, acoustic, static/dynamic, acoustic language, labels/keywords.
Specific properties: Spoken labels/keywords
- are meaningful units adopted from some spoken language (natural language or notation, cf. modalities 6a and 6c) and used to convey isolated bits or chunks of meaning;
- being labels/keywords, and hence used outside a linguistic text, discourse, or notation context, are inherently ambiguous;
- being labels/keywords, have an idiosyncratic aspect: people, even in the same culture and using the same first language, do not classify the world using the same labels/keywords. So if you search someone's directory for cabs, you may not find any because they are called "taxis" instead.
Aptitude: Spoken labels/keywords
- are more or less apt for conveying isolated bits or chunks of meaning depending on the extent to which the provided context helps reduce their ambiguity. The Treasure Hunt game illustrates a powerful context for reducing ambiguity: when the users know that they are navigating a townscape, they have no problem correctly understanding the system's spoken keywords "house", "door", "temple ruins" etc.;
- when designed by the system designers, may not match those that a large fraction of the intended users would spontaneously use;
- are a salient way of presenting information in a low-acoustic environment;
- are only apt for recipients (people and machines) who can hear and understand the particular natural language or who wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: linguistic, acoustic, static/dynamic, acoustic language, notation.
Specific properties: Spoken notation
- is a systematic set of spoken words or phrases developed to convey meaning for some special purpose typically involving specialists, such as for dentists
dictating pyorrhoea alveolaris parameters;
- has limited and specialised expressiveness compared to spoken discourse (modality 6a);
- is dynamic but can be static as a limiting case.
Aptitude: Spoken notation
- if well-established, can be assumed to be apt for its special purpose;
- can be realistically developed for new purposes;
- being typically dynamic, is generally more error prone than static notation modalities;
- is only apt for recipients (people and machines) who can hear and understand the particular natural language tongue in which the notation is being expressed or who wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
7. Haptic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, haptic, static/dynamic.
Specific properties: This entry discusses haptic language in general, both static and dynamic, except for the small contribution made under modality (3) on analogue haptic elements.
Haptic language is of at least three very different types:
- The first type might be called haptic input language and includes all forms of non-situated language produced haptically through hand-writing, typing, morse key tapping, or otherwise: graphic hand-writing and typed language for the seeing (modality 5); haptic output language for the blind and hard-of-seeing (see below).
- The second type is situated haptic language and includes haptic counterparts of situated languages like speech (modality 6) and sign language (modality 4). This relatively rare form of linguistic expression is primarily aimed at someone who is blind and deaf-mute at the same time, and currently does not form what would otherwise have been descendant modalities 7d, 7e and 7f of modality (7) due to its rarity.
- The third type is non-situated haptic output language meant for non-situated communication through tactile perception, primarily among, and with, the blind and hard-of-seeing.
- Haptic output language (non-situated), using non-analogue units of expression, has been developed for communication among and with the blind and hard-of-seeing, for instance as Braille, an equivalent to alphabetic graphic written language (modality 5).
Aptitude: Haptic output language (non-situated),
- if established, like in the case of Braille, can be assumed to be apt for non-situated communication for the blind and hard-of-seeing;
- is only apt for recipients (people and machines) who understand the particular natural language or notation used and have skills in decoding haptic expressions, or who wish to learn that language or those skills.
- Note that the percentage of the blind and hard-of-seeing population who are familiar with Braille varies widely from one country to another, from under 10% to above 80%.
Descendants: This generic-level modality has three descendants in the taxonomy: haptic text (7a), haptic labels/keywords (7b), and haptic notation (7c).
Multimodal combination:
Inherits properties from: linguistic, haptic, static/dynamic, haptic output language, text.
Specific properties: Text is maximum-expressive non-situated natural language. Haptic text,
- like Braille, is maximally expressive haptic use of the world's written natural languages;
- is traditionally static and on paper, but is being made dynamic (under user control) for computer output.
Aptitude: Haptic text
- is apt for maximally expressive non-situated communication among, and with, the blind and hard-of-seeing who understand the language that is being used and have the skills to decode haptic expression.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: linguistic, haptic, static/dynamic, haptic output language, labels/keywords.
Specific properties: Haptic labels/keywords
- are meaningful units from some written natural language or notation, expressed in haptic output language, like Braille, and used to convey bits or chunks of meaning outside their standard textual or notational context, analogous to the way in which isolated written words or phrases are being used in GUI menus;
- being labels/keywords, are inherently ambiguous.
Aptitude: Haptic labels/keywords
- are more or less apt for conveying bits or chunks of meaning depending on the extent to which the context helps reduce their ambiguity;
- are only apt for recipients (people and machines) who understand the meaning of each label/keyword in its language of origin or who wish to learn it, and who have the skills to decode haptic expression or wish to learn those skills.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: linguistic, haptic, static/dynamic, haptic output language, notation.
Specific properties: Haptic notation
- is a systematic set of haptic expressions developed to convey meaning for some special purpose, such as mathematics, and often involving specialists, such as divers operating in inky water;
- has limited and specialised expressiveness compared with haptic text (modality 7a).
Aptitude: Haptic notation,
- if static, is apt for expressing compact, precise, high information load contents for the blind and hard-of-seeing, because of the freedom of perceptual inspection offered;
- if dynamic, is apt for expressing changing parameter values for the blind and hard of seeing, e.g., for process monitoring purposes;
- if well-established, can be assumed to be apt for its special purpose;
- can be realistically developed for new purposes;
- is only apt for recipients (people and machines) who understand the particular notation used or wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
8. Dynamic graphic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, dynamic.
Specific properties: Dynamic graphic language
- is language as expressed graphically and dynamically. It does not seem to matter whether the units of expression used are analogue or non-analogue, cf. modality 4.
- Ignoring graphic sign language (modalities 4a, 4b and 4c), language is primarily expressed dynamically and graphically in two ways. The first, dynamic graphic writing, is simply the dynamic form of static graphic language (modality 5), like in text scrolling down-screen. The second is the visual aspect of spoken language, or visual speech, consisting of movements of mouth, tongue, lips and teeth.
Aptitude: Dynamic graphic writing
- is apt when the user's undivided attention can be assumed, when the information is not overly complex or is not very important anyway, or when the user has some usable way of getting what was missed repeated. An example could be parameter monitoring in process control;
- is only apt for recipients (people and machines) who understand the particular language or notation used or wish to learn it.
Aptitude: Visual speech
- is rarely, if ever, fully apt because, for all or most spoken languages, some phonemes cannot be distinguished through visual speech observation (or "lip reading") alone but must be distinguished either acoustically or through contextual inference that doesn't always work;
- has the dual role of being a modality that (i) may be used when the alternatives are even less apt, and (ii) supports spoken language perception and understanding. All humans who can hear and speak use it in noisy conditions, and the hard-of-hearing use it always to support their acoustic perception of what was said;
- is only apt for recipients (people and machines) who understand the particular language or notation used or wish to learn it.
Descendants: This generic-level modality has six descendants of two different kinds: (i) dynamic graphic writing, consisting of dynamic text (8a), dynamic labels/keywords (8b), and dynamic notation (8c); and (ii) visual speech consisting of graphic spoken discourse (8d), graphic spoken labels/keywords (8e), and graphic spoken notation (8f).
Multimodal combination:
Inherits properties from: linguistic, graphic, dynamic, dynamic graphic language, text.
Specific properties: Dynamic graphic text,
- is a dynamic version of the classical static graphic text (modality 5a) which has developed for non-situated communication with, and among, the seeing;
- is maximally expressive, dynamic visual use of the world's written natural languages and related "artificial" languages, like esperanto;
- is typically dynamic because it scrolls across the screen horizontally or vertically, or because it appears for a limited amount of time, without being repeated;
- being dynamic, requires current user attention in order not to be missed, and forces the viewer to read the text in more or less the temporal order in which it is being presented;
- exists in many different tongues.
Aptitude: Dynamic graphic text
- is apt for representing changing information when spoken presentation of the information (i) conflicts with other modalities, like in speech subtitling, (ii) disturbs other people or may attract their unwanted attention, (iii) is infeasible, or (iv) does not fit the intended users (who may be, e.g., hard-of-seeing);
- is decreasingly apt as a number of other factors grow: number of other user tasks to be done at the same time or in the same time frame, on which users must spend attentional resources; number of other purposes on which users must spend attentional resources; information amount, complexity, and importance; and lack of some way for the user to get missed information repeated, so that it becomes "interactively static": it's dynamic all right, but it is within the user's control to make it get repeated at any time;
- is only apt for recipients (people and machines) who understand the particular language used or wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. However, dynamic graphic text exists in at least two different varieties, typed and hand-written, cf. static modalities 5a1 and 5a2.
Multimodal combination:
8b. Dynamic graphic labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, dynamic, dynamic graphic language, labels/keywords.
Specific properties: Dynamic graphic labels/keywords
- may be thought of as a dynamic version of the static graphic labels/keywords so familiar from GUIs (modality 5b);
- are meaningful units adopted from some graphic written language (natural language or notation, cf. modalities 8a and 8c) and used to dynamically convey bits or chunks of meaning;
- require current user attention to be decoded;
- being labels/keywords, and hence used outside a linguistic text, discourse, or notation context, are inherently ambiguous;
- exist in many different natural language tongues and can be based on many different notations.
Aptitude: Dynamic graphic labels/keywords
- are more or less apt for conveying isolated bits or chunks of meaning depending on the extent to which the provided context helps reduce their ambiguity;
- are apt for visually representing state change and category information when the possible states or categories can be assumed to be known by users or unambiguous in context, like labels on aircraft showing up on a fighter airplane's heads-up display;
- is apt for visually representing process parameter information;
- are only apt for recipients (people and machines) who can see and understand the particular natural language or notation used or who wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. However, dynamic graphic labels/keywords exist in at least two different varieties, typed and hand-written, cf. static modalities 5b1 and 5b2.
Multimodal combination:
Inherits properties from: linguistic, graphic, dynamic, dynamic graphic language, notation.
Specific properties: Dynamic graphic notation
- may be thought of as a dynamic version of static graphic notation (modality 5c), like presenting program code dynamically line-by-line;
- is a systematic set of visually accessible, written language expressions used to dynamically convey meaning for some special purpose typically involving specialists;
- typically requires substantial cognitive load to follow, like a mathematical proof;
- being dynamic, requires current user attention in order not to be missed, and forces the viewer to read the text in more or less the temporal order in which it is being presented.
- Provided that a purpose can be found, it is realistically possible to develop new dynamic graphic notations.
Aptitude: Dynamic graphic notation,
- unless very simple, or merely aimed to be superficially scanned, is hardly apt at all because representations should not carry high cognitive load and be dynamic at the same time;
- at the very least, should be interactively static, so that the user can stop the presentation and get anything repeated anytime;
- if used nevertheless, should be used when a maximum of user attention is available;
- is only apt for recipients (people and machines) who can see and understand the particular notation used or who wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. However, dynamic graphic notation exists in at least two different varieties, typed and hand-written, cf. static modalities 5c1 and 5c2.
Multimodal combination:
Inherits properties from: linguistic, graphic, static/dynamic, dynamic graphic language, spoken discourse.
Specific properties: Visual spoken discourse
- is spoken discourse (modality 6a) viewed purely graphically and focusing on mouth, tongue, lips and teeth;
- is dynamic except in the limiting case;
- exists in many different tongues;
- requires hard training to decode stand-alone to the extent possible.
Aptitude: Visual spoken discourse,
- is probably never fully apt stand-alone because inherently ambiguous, cf. modality (8);
- may still be used stand-alone to train, and convey information to, "lip readers";
- should never be used unimodally except for a target user population of "lip readers";
- in synchronous combination with acoustic spoken discourse, is apt to support speech recognition and understanding in noisy conditions and for the hard-of-hearing;
- is only apt for recipients (people and machines) who can see and understand the particular language used or who wish to learn it.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
8e. Visual spoken labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: linguistic, graphic, static/dynamic, dynamic graphic language, spoken labels/keywords.
Specific properties: Visual spoken labels/keywords,
- are meaningful units adopted from some spoken language (natural language or notation, cf. modalities 8d and 8f) considered purely visually as conveying bits or chunks of meaning through the movements of mouth, tongue, lips and teeth;
- are static or dynamic;
- exist in many different tongues;
- are doubly ambiguous: (i) they cannot unambiguously express all spoken labels/keywords because of the phonemic ambiguity problem, see modality (8); and (ii) labels/keywords, being used outside a linguistic text, discourse, or notation context, are themselves inherently ambiguous. This double ambiguity probably explains why little use is being made of graphic spoken labels/keywords stand-alone;
- unless few and specially selected for being comparatively easy to recognise, require hard training to decode stand-alone to the extent that decoding is possible at all.
Aptitude: Visual spoken labels/keywords,
- might be used stand-alone provided that labels/keywords are hand-selected to avoid phonemic ambiguity (see modality 8). The resulting labels/keywords are more or less apt for conveying isolated bits or chunks of meaning depending on the extent to which the context helps reduce their ambiguity;
- are only apt for - typically specially trained "lip reading" - recipients (people and machines) who can see and understand the particular visual spoken language used or who wish to learn it.
- in synchronous combination with acoustic spoken labels/keywords (modality 6b), are apt to support speech recognition and understanding in noisy conditions and for the hard-of-hearing.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: linguistic, graphic, static/dynamic, dynamic graphic language, spoken notation.
Specific properties: Visual spoken notation
- is a systematic set of spoken words or phrases considered purely visually by looking at the movements of mouth, tongue, lips and teeth, and developed to convey meaning for some special purpose typically involving specialists;
- has limited and specialised expressiveness compared to visual spoken discourse (modality 8d);
- might exist in many different tongues but seems to be quite rare;
- requires hard training to decode stand-alone to the extent possible.
Aptitude: Visual spoken notation,
- is apt stand-alone for "lip readers" provided that the phonemic ambiguity problem (see modality 8), is carefully avoided;
- in synchronous combination with acoustic spoken notation (modality 6c), is apt to support speech recognition and understanding in noisy conditions and for the hard-of-hearing.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, static.
Specific properties: Analogue static graphics
- express specific information through static analogue representation in the graphic medium;
- allow unrestricted freedom of visual inspection of the analogue information represented;
- express information in the graphic medium subject to limitations of dimensionality, specificity, sketchiness, aspects of similarity, and resolution of the representation.
Aptitude: Analogue static graphics are apt for visualising information in 1D, 2D and 3D (scale model) spatial while preserving full freedom of perceptual inspection, representations serving as snapshots of real or imagined objects, processes, and events or aspects of these. Several snapshots can be sequenced into a cartoon to represent causality and change over time.
Descendants: This generic-level modality has five descendants in the taxonomy: static graphic images (9a), static graphic maps (9b), static graphic compositional diagrams (9c), static graphic graphs (9d), and static graphic conceptual diagrams (9e).
Multimodal combination:
Inherits properties from: analogue, graphic, static, analogue static graphics, image.
Specific properties: Static graphic images
- are static graphic representations which generally have many aspects of similarity with what they represent;
- represent information by rendering visual specificity of real or imagined objects, processes, and events, and may, in the limit, make it appear as if the viewer is in direct visual contact with the actual entity represented by the image. Generally speaking, images constitute the most natural and realistic analogue static graphic modality;
- allow unrestricted freedom of visual inspection of the properties of objects, processes, and events represented.
Aptitude: Static graphic images
- are apt for representing the visual specifics of real or imagined objects, processes and events for free perceptual inspection;
- can visually represent objects, processes, and events that are not perceivable by humans, for instance because of being too small, too large, too remote, too slow, too fast, transparent, beyond the human visual sensory thresholds, or normally hidden beneath some exterior;
- can visually represent processes and events that take place in other physical media, such as presenting acoustic and haptic information graphically;
- when rendered with reduced specificity, are semi-apt for visually representing ideal types rather than particular individual objects, processes, and events, like the basic shapes of geometry;
- when rendered with high specificity and resolution, are apt for representing the visual uniqueness of individual objects, processes, and events, as in high-resolution photographs of people's faces, facilitating their identification.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Some commonly referred-to prototypes include sketches, photo-realistic images, and semi-abstract images whose specificity has been reduced in order to highlight identity, like the basic shapes of geometry in textbooks. Static graphic image icons are an essential component of the classical WIMP (Windows, Icons, Menus and Pointing) graphical user interface.
Multimodal combination:
Inherits properties from: analogue, graphic, static, analogue static graphics, map.
Specific properties: Static graphic maps
- constitute our established and, in several respects, well-developed modality for representing spatial layout. This is, in fact, why maps are a modality in the taxonomy because, otherwise, a map is just a kind of compositional diagram, i.e. modality (9c);
- typically represent a location or area in 1D, 2D or 3D, using reduced specificity and scale to focus on points of interest and generate an easy overview.
Aptitude: Static graphic maps
- are apt for presenting large amounts of information on spatial location, direction, distance, and other spatial layout properties and relationships for free visual inspection;
- are apt for supporting navigation in space.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. The prototypical static graphic map is the roadmap, a drawn, reduced-scale, birds-eye 2D schematic representation of part of the surface of the earth, showing selected features (or points of interest), such as roads, rivers, mountains and cities, and having been designed to enable travellers to find the right route between geographical locations. Note that this map is typically bi-modal because points of interest are labelled using labels like those of modality (5b).
Multimodal combination:
9c. Static graphic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, graphic, static, analogue static graphics, compositional diagram.
Specific properties: A diagram is an analytical image. Static graphic compositional diagrams
- represent a visual view of the spatial decomposition of objects, processes and events in 1D, 2D or 3D space. A sequence of diagrams can show temporal order of decomposition stages or steps.
Aptitude: Static graphic compositional diagrams afford freedom of visual inspection of analytical information about spatial and spatio-temporal decomposition.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. In fact, however, static graphic maps (modality 9b) are a descendant because maps are a kind of compositional diagram.
Multimodal combination:
Inherits properties from: analogue, graphic, static, analogue static graphics, graph.
Specific properties: Static graphic graphs
- have been developed over centuries in order to improve our means of representing quantitative information visually in a 1D, 2D or 3D graph space as (static) data graphics;
- are typically labelled using labels/keywords, images or otherwise, making typical static graphic graphs at least bi-modal.
Aptitude: Static graphic graphs
- aptly provide freedom of visual inspection of data profiles and for making comparisons, spotting trends, and discovering new relationships among large quantitative data sets;
- are generally inept at representing exact numerical information - use, e.g., tables instead.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Some familiar graph types are line graphs, bar graphs, pie graphs, and scatter plot graphs.
Multimodal combination:
9e. Static graphic conceptual diagrams [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, graphic, static, analogue static graphics, conceptual diagram.
Specific properties: Static graphic conceptual diagrams
- constitute the traditional modality for representing abstract, conceptual information in 1D, 2D and 3D analogue form;
- provide a static, spatially structured view of the composition of abstract entities using a limited number of analogue means, such as spatial grouping, direction, and order, to represent, e.g., closeness of relationship, time, temporal order, and subsumption;
- are at least bi-modal because the conceptual entities represented must be labelled by means of labels/keywords, images, or otherwise, like in the taxonomy conceptual diagrams.
Aptitude: Static graphic conceptual diagrams
- are apt for exposing to free visual inspection the structure, composition, and internal relationships of abstract, conceptual entities, such as organisations, theories, classifications, conceptual models, and other conceptual structures.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
10. Analogue static-dynamic acoustics [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, acoustic, static/dynamic.
Specific properties: Analogue acoustics
- express specific information through analogue representation in the acoustic medium;
- largely consist of dynamic representation although having static acoustic representation as a limiting case, such as a repetitive sound that can only be turned off by the user;
- can to some extent represent 2D and 3D spatial information, such as different sound sources distributed in 2D or 3D space.
Aptitude: Analogue acoustics are apt for auralising information about real or imagined objects, processes, and events or aspects of these.
Descendants: This generic-level modality has five descendants in the taxonomy: acoustic images (10a), acoustic maps (10b), acoustic compositional diagrams (10c), acoustic graphs (10d), and acoustic conceptual diagrams (10e).
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, image.
Specific properties: Acoustic images
- represent information by rendering aural specificity of real or imagined objects, processes, and events, and may, in the limit, make it appear as if the viewer is in direct aural contact with the actual entitities represented by the image. Generally speaking, images constitute the most natural and realistic analogue acoustic modality;
- unless static or controlled by the user, allow no freedom of aural inspection of the properties of objects, processes, and events represented.
Aptitude: Acoustic images
- are apt for representing the aural specifics of real or imagined objects, processes and events, contributing to immersive interaction;
- can aurally represent objects, processes, and events that are not perceivable by humans, for instance because of being too remote or beyond the human visual sensory thresholds;
- when rendered with high specificity, are apt for representing the aural uniqueness of individual objects, processes, and events, as in high-quality recordings of people's voices, facilitating their identification.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Earcons (acoustic image icons) are common in graphical user interfaces as, e.g., user action feedback and alerts.
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, map.
Specific properties: Acoustic maps
- are a particular kind of acoustic compositional diagrams (modality 10c) which support navigation in space by means of acoustic cues;
- are, by analogy with static graphic maps (modality 9b), representations in 1D, 2D (stereo), or 3D (3D sound) of the soundscape of a location or area. Representations would typically use reduced specificity and reduced scale to focus on points of interest and generate an easy overview. Points of interest would typically be labelled, using labels/keywords, images or otherwise, making typical acoustic maps at least bi-modal.
Aptitude: Acoustic maps
- hold a far from fully explored interaction potential, primarily in support of navigation when vision cannot be relied upon very much, if at all, but even for enhancing ordinary visual maps (modality 9b). Bats, for instance, and submerged submarines navigate in space very much based on acoustic cues. Acoustic cues might be apt for this kind of navigation to the extent that the environment is acoustically mappable, i.e., includes relatively permanent and relatively stationary sound sources having a relatively constant acoustic profile. If number, location, and profile of sound sources change unpredictably, the environment probably isn't acoustically mappable. Conversely, an environment might be acoustically "rigged" in a way that would enable it to convey spatial information in minute detail.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. We are not aware of any prototypical acoustic maps.
Multimodal combination:
10c. Acoustic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, compositional diagram.
Specific properties: A diagram is an analytical image. Acoustic compositional diagrams
- show the decomposition of complex 1D, 2D and 3D sound entities, such as of a running car engine or a musical harmony.
Aptitude: Acoustic compositional diagrams
- are apt for showing the decomposition of complex 1D, 2D and 3D sound entities. Components are typically labelled, using labels/keywords, images or otherwise, making typical acoustic compositional diagrams at least bi-modal.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, graph.
Specific properties: Acoustic graphs
- express quantitative information acoustically, like when a standard Geiger counter continuously detects ionizing radiation and emits a click per pulse generated;
- could use, in principle, either a 1D, 2D or 3D acoustic space;
- would typically be labelled, using albels/keywords, images, or otherwise, making typical acoustic graphs at least bi-modal.
Aptitude: Acoustic graphs
- are apt for representing quantitative information in real time for the purpose of providing a rough and inexact overview. Data acoustics may be useful when, e.g., data needs to be monitored continuously and the operator is blind or has eyes occupied;
- typically have to be presented one set of data at a time, thus are not apt for making comparisons, spotting trends, and discovering new relationships among quantitative data sets;
- are generally inept at representing exact numerical information - use, e.g., tables instead.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. The simplest and most obvious type seems to be a graph that represents changes of a single parameter over time.
Multimodal combination:
10e. Acoustic conceptual diagrams [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, conceptual diagram.
Specific properties: Acoustic conceptual diagrams
- represent abstract, conceptual information in 1D, 2D and 3D analogue acoustic form but having very few analogue means of representation available, such as temporal order and duration;
- are at least bi-modal because the conceptual entities represented must be labelled by means of labels/keywords, images, or otherwise.
Aptitude: Acoustic conceptual diagrams
- have so limited expressiveness that their aptitude, if any, is quite limited as well.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture.
Specific properties: Gesture is movement of a body part, several body parts, or the body as a whole to convey shared non-linguistic meaning. Acoustic gesture
- is gesture expressed for acoustic perception;
- is typically dynamic, rarely static.
Aptitude: Acoustic gesture
- consists of sub-types that are apt for very different purposes.
Descendants: This sub-atomic level modality has five descendants in the current version of the taxonomy: acoustic batons (10a1a), acoustic deictic gestures (10a1b), acoustic emblems (10a1c), acoustic iconic gestures (10a1d), and acoustic metaphoric gestures (10a1e).
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, expression.
Specific properties: Acoustic expression
- is the acoustic part of the "mirror of the soul", or of how people express their mental and physical states through their vocal apparatus, like when the voice is tired, happy, or panicky, or using extensions of the body, such as musical instruments;
- is often non-deliberate and involuntary, the expressing person not being aware of producing expression information.
Aptitude: Acoustic expression
- is apt for acoustically expressing the mental and physical states of actual and virtual people, and of robots.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy. Candidate classifications should not forget music and song in addition to tone of voice and other acoustic expressions.
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, baton.
Specific properties: A baton is a rhytmic movement, typically of the hand-arm, that typically accompanies speech for visible emphasis. Acoustic batons
- are batons expressed for acoustic perception;
- typically are sounds rhytmically produced to accompany speech for audible emphasis or otherwise;
- cannot be produced by the vocal apparatus while speaking at the same time, but may be produced by, e.g., making audible haptic batons (modality 11a1a).
Aptitude: Acoustic batons
- don't seem to have any obvious aptitude and rather appear to reflect personality or a slightly peculiar personal communication style.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, deictic.
Specific properties: Acoustic deictic gesture
- identifies something in space and is perceived acoustically;
- typically must be accompanied by complementary speech which explains what is being referred to;
- is easy to make by people unaided when identifying themselves, like the child who shouts "I am here" during a game;
- is hard to make by people unaided when they have to identify something different from themselves, like the soprano who makes her voice shatter a glass, thereby identifying it.
Aptitude: Acoustic deictic gesture
- only seems apt for self-deixis, i.e., for pointing out oneself by any acoustic means agreed upon.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, emblem.
Specific properties: Emblems are gestures that are more or less equivalent to conventional linguistic expressions. Acoustic emblems
- express emblems for being heard, like the wolf whistle that, in some cultures, means "How beautiful!";
- are salient in low-noise environments;
- are expected to show important differences in shared-meaning emblem repertoire from one culture or sub-culture to another.
Aptitude: Acoustic emblems
- are apt for expressing, subject to cultural constraints, what corresponds to linguistic expressions when the recipient is not necessarily looking at the producer of the emblem.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, iconic.
Specific properties: Iconic gesture is like very sketchy analogue rendering of objects and events. Acoustic iconic gesture
- is iconic gesture meant to be heard;
- typically consists in making a sketchy analogue rendering of something or someone by means of the vocal apparatus, like of a bird's voice, but might be made by other body parts as well;
- is highly personal - it's like everyone's doing his or her best to draw a simple sketch of something, which normally produces very different sketches, some of which may not recognisably be sketches of the same motif, whereas others are elegant and precise;
- is typically strongly dependent upon the accompanying speech for its correct interpretation;
- is being used to very different extent by different people, some never using it, others using it rather frequently. Acoustic iconic gesture thus represents style of communication rather than a necessary component of the contents of communication.
Aptitude: Acoustic iconic gesture
- is apt for increasing immersiveness and liveliness of interaction by adding a simple audible sketch to a narrative.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
10a1e. Acoustic metaphoric gesture [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, metaphoric.
Specific properties: Metaphoric gesture makes a very sketchy analogue rendering of some source domain and wants it interpreted as being about some different target domain. Acoustic metaphoric gesture
- is metaphoric gesture meant to be heard;
- typically consists in making a sketchy analogue rendering of something or someone by means of the vocal apparatus while suggesting through speech that the rendering should be interpreted metaphorically as being about something else, like when we say that someone blew his top and add a personal acoustic image of something blowing up;
- is highly personal and strongly dependent on the accompanying speech for its correct interpretation;
- is being used to different extent by different people. Some never use acoustic metaphoric gesture, which thus represents style of communication rather than a necessary component of the contents of communication.
Aptitude: Acoustic metaphoric gesture
- is apt for increasing immersiveness and liveliness of presentation by adding metaphoric use of a simple acoustic sketch to spoken language narrative.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
11. Analogue static-dynamic haptics [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, haptic, static/dynamic.
Specific properties: Analogue haptics
- are haptic representations in 1D, 2D, 3D and, possibly, time, which have some similarity with what they represent;
- express specific information in the haptic medium, subject to limitations of dimensionality, specificity, sketchiness, aspects of similarity, and resolution of the representation;
- include both static and dynamic haptic representation;
- mostly consist of static information representation for the time being, i.e., representations that allow unrestricted freedom of tactile inspection of the analogue information represented.
Aptitude: Analogue haptics
- primarily but not only if static, are apt for representing the detailed tactile "feel" of real and imagined objects, their physical state, texture, structural detail, changes and patterns of movement, and otherwise;
- if static, are semi-apt for representing overviews of objects as well as event and process snapshots in 1D, 2D and 3D spatial, because haptic overviews must be built step by step through exploration and cannot be acquired at a glance like when built through vision;
- are generally semi-apt for object and event/process classification: large and complex objects can be unwieldly to inspect; haptics miss some of the information channels used by vision, such as colour; and dynamic haptic objects, events and processes, in particular, can be very difficult to inspect.
Descendants: This generic-level modality has five descendants in the taxonomy: haptic images (11a), haptic maps (11b), haptic compositional diagrams (11c), haptic graphs (11d), and haptic conceptual diagrams (11e).
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image.
Specific properties: Haptic images
- are haptic representations that generally have many aspects of similarity with what they represent;
- represent information by rendering haptic specificity of real or imagined objects, processes, and events, and may, in the limit, make it appear as if the viewer is in direct haptic contact with the actual entity represented by the image. Generally speaking, images constitute the most natural and realistic analogue haptic modality.
Aptitude: Haptic images
- primarily but not only if static, are apt for representing the detailed tactile "feel" or specifics of real and imagined objects, their physical state, texture, structural detail, changes and patterns of movement, and otherwise, like the feel of a piece of cloth;
- if static, are only semi-apt for representing overviews of objects, scenes, and event and process snapshots, because haptic overviews must in general be built step by step through exploration and cannot be acquired at a glance like when built through vision;
- when rendered with high specificity and resolution, are only semi-apt for representing the haptical uniqueness of individual objects, processes, and events, because much of their uniqueness resides in proportions and other relations among properties, which can only be perceived by building overviews;
- when rendered with reduced specificity, are rather apt for haptically representing ideal types rather than particular individual objects, processes, and events, like the basic shapes of geometry.
Descendants: This atomic-level modality has two descendants in the taxonomy: haptic gesture (11a1) and haptic body action (11a2).
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, map.
Specific properties: Haptic maps
- mainly if static, are a kind of haptic compositional diagram (modality 11c) specialised at representing spatial scenes and layout;
- are not being used much yet but may be expected to typically represent a location or area in 3D, using reduced specificity and scale to focus on points of interest and enable overview creation.
Aptitude: Haptic maps
- if static, are semi-apt for representing spatial scenes and layout, including information on spatial location, direction, distance, and other spatial layout properties and relationships for free visual inspection. The semi-aptitude is due to the fact that haptic overviews must in general be built step by step through exploration and cannot be acquired at a glance like when built through vision;
- are generally inferior to static graphic maps in terms of speed of access.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Nor is there any prototypical instances yet, like the 2D roadmap is a prototype of a static graphic map (modality 9b). However, a typical haptic map may be expected to include linguistically labelled points of interest, using, e.g., spoken or Braille output labelling, making the map at least bi-modal.
Multimodal combination:
11c. Haptic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, compositional diagram.
Specific properties: A diagram is an analytical image. Haptic compositional diagrams
- if static, represent a haptic view of the spatial decomposition of objects, processes and events in 1D, 2D or 3D space. A sequence of diagrams can show temporal order of decomposition stages or steps.
Aptitude: Haptic compositional diagrams
- when static, provide freedom of haptic inspection of analytical information about spatial and spatio-temporal decomposition;
- when static, are inferior to static graphic compositional diagrams (modality 9c) in the sense that vision can provide a diagram overview at a glance whereas a haptic overview must be painstakingly built through haptic exploration.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. In fact, however, haptic maps (modality 11b) are a descendant because maps are a kind of compositional diagram.
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, graph.
Specific properties: Haptic graphs
- if static, may perhaps at this point be thought of as haptic versions of static graphic graphs (modality 9d) which have been developed over centuries in order to improve our means of representing quantitative information visually in a 1D, 2D or 3D graph space as (static) data graphics;
- are typically labelled using labels/keywords, images or otherwise, making typical static haptic graphs at least bi-modal;
- if dynamic, might express quantitative data dynamically through, e.g., dynamic force feedback.
Aptitude: Haptic graphs
- if static, aptly provide freedom of haptic inspection of data profiles and for making comparisons, spotting trends, and discovering new relationships among large quantitative data sets;
- are generally inept at representing exact numerical information - use, e.g., tables instead;
- if static, are inferior to static graphic graphs in the sense that haptics, as opposed to graphics, does not allow at-a-glance overview, so overview of a static haptic graph must be built painstakingly through graph exploration, and access speed suffers as well;
- if dynamic, might be apt for representing real-time dynamic quantitative information for continuous monitoring when the operator is blind or has eyes occupied, i.e., as an alternative to acoustic graphs (modality 10d), and for use when acoustic graphs don't work because of too much noise.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Some familiar graph types are line graphs, bar graphs, pie graphs, and scatter plot graphs.
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, conceptual diagram.
Specific properties: Haptic conceptual diagrams
- if static, may be thought of as a haptic counterpart to static graphic conceptual diagrams (modality 9e) which (i) constitute the traditional modality for representing abstract, conceptual information in 1D, 2D and 3D analogue form; (ii) provide a static, spatially structured view of the composition of abstract entities using a limited number of analogue means, such as spatial grouping, direction, and order, to represent, e.g., closeness of relationship, time, temporal order, and subsumption; and (iii) are at least bi-modal because the conceptual entities represented must be labelled by means of labels/keywords, images, or otherwise, like in the taxonomy conceptual diagrams;
- if static, are inferior to static graphic conceptual diagrams in the sense that haptics, as opposed to graphics, does not allow at-a-glance overview, so overview of a static haptic conceptual diagram must be built painstakingly through diagram exploration, and access speed suffers as well;
- if dynamic, MORE;
Aptitude: Haptic conceptual diagrams
- are apt for exposing to free haptic inspection the structure, composition, and internal relationships of abstract, conceptual entities, such as organisations, theories, classifications, conceptual models, and other conceptual structures.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture.
Specific properties: Gesture is movement of a body part, several body parts, or the body as a whole to convey shared meaning non-linguistically. Haptic gesture
- is gesture expressed for haptic perception;
- is gesture produced by using the body or its extension to touch a real or virtual object or person, like when - in many but possibly not all cultures - patting someone on the shoulder or pointing at a virtual object by touching a tactile screen;
- is typically dynamic, rarely static.
Aptitude: Haptic gesture
- consists of sub-types that are apt for very different purposes. One thing they have in common, however, is that they can be used when visual and acoustic communication is undesirable or infeasible, like in the dark and in other conditions of bad visibility, when stealth is required, or if the noise level is high.
Descendants: This sub-atomic level modality has five descendants in the current version of the taxonomy: haptic batons (11a1a), haptic deictic gestures (11a1b), haptic emblems (11a1c), haptic iconic gestures (11a1d), and haptic metaphoric gestures (11a1e).
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, expression.
Specific properties: Haptic expression
- is the haptic part of the "mirror of the soul", of how people express their mental and physical states through their bodies, in this case by touching others, like in a friendly handshake or a joyful embrace;
- differs in terms of, at least, whether A touches B at all, how A touches B, and where A touches B;
- differs from one person to another depending on personality;
- differs from one culture to another;
- can be non-deliberate and involuntary;
- is typically dynamic.
Aptitude: Haptic expression
- is apt for expressing the mental and physical states of actual and virtual people, and of robots.
Descendants:
Multimodal combination:
Inherits properties from:analogue, haptic, static/dynamic, analogue haptics, image, haptic images, body action.
Specific properties: All representation and exchange of information, like speaking or showing a graph, requires physical action by humans or computers. Body action is physical action that does not issue any of the representations described in the taxonomy, from linguistic to arbitrary representations. Haptic body action
- is body action upon real or virtual objects and persons, like walking, creating a drawing, pocketing a mobile phone, helping someone who fell to stand up again, or navigating a 3D haptic environment without visual aid;
- is typically dynamic.
Aptitude: Haptic body action
- is apt for representing the haptics of actions performed by virtual people and robots.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, baton.
Specific properties: A baton is a rhytmic movement, typically of the hand-arm, that typically accompanies speech for visible emphasis. Haptic batons
- are batons expressed for haptic perception;
- involve touching something when performing the batons, like slapping a table surface while making each of several points in speech;
- typically exhibits a temper or expresses a sense of seriousness.
Aptitude: Haptic batons
- don't seem to have any obvious aptitude and rather appear to reflect personality or a slightly peculiar personal communication style.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, deictic.
Specific properties: Haptic deictic gesture
- is made by touching whatever is being pointed at, typically with finger-hand or some extension of these, like when selecting an object on a tactile screen.
Aptitude: Haptic deictic gesture
- is apt for unambiguous identification of objects and locations in space.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, emblem.
Specific properties: Emblems are gestures that are more or less equivalent to conventional linguistic expressions. Haptic emblems
- may be expressed by touching people, like slapping someone's back, meaning, in many cultures, "Congratulations, well done!", or objects, like knocking the table surface after someone's speech, meaning, in several Western cultures, "Congratulations, well spoken!"
- are typically used for expressing emotion, attitude and appreciation;
- show important differences in shared-meaning emblem repertoire from one culture to another;
- show strong cultural differences in how much people of the same or different gender and of different family relationships touch one another. For instance, some cultures do not allow women to touch male non-family at all, not even through the haptic emblem of shaking hands in greeting.
Aptitude: Haptic emblems
- are apt for expressing emotion, attitude and appreciation, subject to cultural constraints.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, iconic.
Specific properties: Iconic gesture is like very sketchy analogue rendering of objects and events. Haptic iconic gesture
- is iconic gesture meant to be felt haptically;
- consists in making a sketchy analogue rendering of something through touch, like taking the other person's hands and showing how "The piano teacher took and held my hands like this" or mock-biting a child's arm to show how the lion would eat the child;
- is highly personal - it's like everyone's doing his or her best to draw something, which normally produces very different drawings, some of which may not recognisably be drawings of the same motif;
- is typically strongly dependent upon the accompanying speech for its correct interpretation;
- seems rather rare in general.
Aptitude: Haptic iconic gesture
- is apt for increasing immersiveness by adding a simple body impact sketch to a spoken or sign language narrative.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
11a1e. Haptic metaphoric gesture [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, metaphoric.
Specific properties: Metaphoric gesture makes a very sketchy analogue rendering of some source domain and wants it interpreted as being about some different target domain. Haptic metaphoric gesture
- consists in making a sketchy analogue rendering of something through touch while explaining through speech or sign language that the rendering should be interpreted metaphorically as being about something else;
- is highly personal and dependent on the accompanying speech for its correct interpretation, see the explanation of haptic iconic gesture (modality 11a1d);
- seems so rare that only contrived examples can be found. See the far more realistic examples under iconic graphic gesture (modality 12a2d).
Aptitude: Haptic metaphoric gesture
- may not be apt for much because the haptic invasion of the other's intimacy may rarely be counterbalanced by any sufficient contribution of information. If, for instance, A tells B that, metaphorically speaking, "They strangled us", then it hardly adds worthwhile information for A to mock-strangle B.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic.
Specific properties: Analogue dynamic graphics
- express specific information through dynamic analogue representation in the graphic medium;
- express information in the graphic medium subject to limitations of dimensionality, specificity, sketchiness, aspects of similarity, and resolution of the representation.
Aptitude: Analogue dynamic graphics
- are apt for visualising specific information in 1D, 2D and 3D spatial when change and development over time is important to the visualisation.
Descendants: This generic-level modality has five descendants in the taxonomy: dynamic graphic images (12a), dynamic graphic maps (12b), dynamic graphic compositional diagrams (12c), dynamic graphic graphs (12d), and dynamic graphic conceptual diagrams (12e).
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image.
Specific properties: Dynamic graphic images
- are dynamic graphic representations that generally have many aspects of similarity with what they represent;
- represent information by rendering visual specificity of real or imagined objects, scenes, processes, and events, and may, in the limit, make it appear as if the viewer is in direct visual contact with the actual entity represented by the image. Generally speaking, dynamic images constitute the most natural and realistic analogue dynamic graphic modality and form the core of animation.
Aptitude: Dynamic graphic images
- are apt for representing the visual specifics of movement and change of real or imagined objects and scenes, processes and events;
- are apt for representing the visual specifics of virtual conversational characters, human-like and otherwise, and characters performing physical action;
- are apt for visually representing dynamic views of objects, scenes, processes, and events that are not perceivable by humans, for instance because of being too small, too large, too remote, too slow, too fast, transparent, beyond the human visual sensory thresholds, or normally hidden beneath some exterior.
Descendants: This atomic-level modality has three descendants in the taxonomy: visual facial expression (12a1), visual gesture (12a2), and visual body action (12a3). Dynamic graphic image icons are appearing in graphical user interfaces.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, map.
Specific properties: Dynamic graphic maps - a kind of dynamic graphic compositional diagram (modality 12c) -
- typically represent a location, area, or region of space in 1D, 2D or 3D, using reduced specificity and scale to focus on information of interest and generate an overview;
- often show development over time of what is being mapped, such as animated Earth plate techtonics showing the evolution of continents;
- are also often used as dynamic overlay onto a static graphic background map (modality 9b), like in weather model animation of, e.g., predicted precipitation, wind, or temperature, or as showing dynamic change in a static map context, like an expanding system of roads "growing" on the map.
Aptitude: Dynamic graphic maps
- are apt for providing an impression of change to, and within, a map context. However, to communicate to people the details of the represented changes, the representation should be interactively static, so that the user can move back and forth in it, frame by frame or otherwise;
- sometimes provide a more selective focus on phenomena of interest than corresponding dynamic graphic image videos composed of, e.g., satellite images of cloud cover or of the summer-melting of the arctic polar ice.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
12c. Dynamic graphic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, compositional diagram.
Specific properties: A diagram is an analytical image. Dynamic graphic compositional diagrams
- typically represent a dynamic visual view of the decomposition of objects, processes and events in 1D, 2D or 3D space and time, using reduced specificity and scale to focus on information of interest and generate an overview, like in an animation that "explodes" a wheelbarrow step by step into its components.
Aptitude: Dynamic graphic compositional diagrams
- are apt for providing an impression of decomposition processes. However, to communicate the process details to people, the representation should be interactively static, so that the user can move back and forth in it, frame by frame or otherwise.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. In fact, however, dynamic graphic maps (modality 12b) are a descendant because maps are a kind of compositional diagram.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, graph.
Specific properties: Dynamic graphic graphs
- typically show quantitative process variable changes in real time, such as the changes in ground motion produced by an earthquake and normally measured and recorded using a seismometer;
- are typically labelled using labels/keywords, images or otherwise, making typical dynamic graphic graphs at least bi-modal.
Aptitude: Dynamic graphic graphs
- are apt for showing and monitoring quantitative process variable changes in real time. However, to get more than a fleeting impression of development over time, a static graph record is needed, like a seismometer record, see modality 9d;
- are generally inept at representing exact numerical information - use symbolic (linguistic) information instead.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
12e. Dynamic graphic conceptual diagrams [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, conceptual diagram.
Specific properties: Dynamic graphic conceptual diagrams
- represent abstract, conceptual information in 1D, 2D, 3D and temporal analogue form, using a limited number of analogue means, such as spatial and temporal grouping, direction, and order, to represent, e.g., closeness of relationship, time, temporal order, and subsumption;
- use dynamic means to show conceptual diagrams in the graphic medium, like showing a family tree in a series of snapshots of the tree rather than as a whole, i.e., as a static graphic conceptual diagram (modality 9e);
- are at least bi-modal because the conceptual entities represented must be labelled by means of labels/keywords, images, or otherwise, like in the taxonomy conceptual diagrams.
Aptitude: Dynamic graphic conceptual diagrams
- do not appear apt for anything in particular, except for providing a fleeting impression of abstract structure and relationships;
- might perhaps help make mental processes and conceptual inferencing vivid in ways that static conceptual diagrams cannot.
Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture.
Specific properties: Gesture is movement of any body part, several body parts, or the body as a whole to convey shared meaning non-linguistically. Graphic gesture
- is gesture expressed for visual perception;
- often is all there is to perceive of a gesture, because the gesture is not being expressed acoustically (modality 10a1) or haptically (modality 11a1) at all, like when waving the hand in farewell;
- is typically dynamic, rarely static.
Aptitude: Graphic gesture
- consists of sub-types that are apt for very different purposes. One thing they have in common, however, is that they can be used when acoustic communication is undesirable or infeasible, like when stealth is required or the noise level is high.
Descendants: This sub-atomic level modality has five descendants in the current version of the taxonomy: graphic batons (12a2a), graphic deictic gestures (12a2b), graphic emblems (12a2c), graphic iconic gestures (12a2d), and graphic metaphoric gestures (12a2e).
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, expression.
Specific properties: Graphic expression
- is the graphic part of the "mirror of the soul", or of how, most of the time, people express their mental and physical states through their body;
- is primarily made by face and gaze but also through head and body posture, and movement of body and body parts;
- is often non-deliberate and involuntary.
Aptitude: Graphic expression
- is apt for visually expressing the mental and physical states of virtual people and robots.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, body action.
Specific properties: All representation and exchange of information, like speaking or showing a graph, requires physical action by humans or computers. Body action is physical action that does not issue any of the representations described in the taxonomy, from linguistic to arbitrary representations. Graphic body action
- is visually perceived body action upon real or virtual objects and persons, like walking, creating a drawing, pocketing a mobile phone, or helping someone who fell to stand up again;
- is typically dynamic.
Aptitude: Graphic body action
- is apt for visually representing actions performed by virtual people and robots.
Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, baton.
Specific properties: A baton is a rhytmic movement, typically of the hand-arm, that typically accompanies speech for visible emphasis. Graphic batons
- are batons expressed for visual perception, like when a politician lists a series of promises in a public speech, or emphasises each word, or every important word, in a key statement;
- are used with very different frequency by different people, from none at all to batons accompanying most of the discourse made by a particular individual, thus representing style of communication rather than a necessary component of the contents of communication;
- may be used with different frequency in different cultures;
- may be used with different frequency among different professions within the same culture.
Aptitude: Graphic batons
- are apt for representing a particular style of spoken discourse, in which rhytmic hand-arm movement accompanies the speech for visible emphasis, for contributing visible structure (cf. modality 20), or otherwise.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, deictic.
Specific properties: Graphic deictic gesture
- is made to visually point at something, typically using finger-hand or some extension of these, and/or gaze, and/or gaze + head nod;
- typically must be accompanied by complementary speech which explains what is being referred to.
Aptitude: Graphic deictic gesture
- is apt for complementing speech for making more or less precise spatial reference depending on the context.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, emblem.
Specific properties: Emblems are gestures that are more or less equivalent to conventional linguistic expressions. Graphic emblems
- express emblems visually, like the two-finger "V" which in several cultures means something like "We won!", the eye blink (or wink) that, in many cultures, means "Just kidding", the head shake which means "No" in many cultures, etc.
- often show important differences in shared-meaning emblem repertoire from one culture or sub-culture to another.
Aptitude: Graphic emblems
- are apt for expressing, subject to cultural constraints, what corresponds to linguistic expressions without acoustically speaking, for instance when the noise level is too high, when speaking would inconvenience others, or when stealth is required.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, iconic.
Specific properties: Iconic gesture is like very sketchy analogue rendering of objects and events. Graphic iconic gesture
- is iconic gesture meant to be seen;
- typically consists in making a sketchy analogue rendering of something or someone by means of hand-arms, other body parts, or the body as a whole;
- is highly personal - it's like everyone's doing his or her best to draw a simple sketch of something, which normally produces very different sketches, some of which may not recognisably be sketches of the same motif, whereas others are elegant and precise;
- is typically strongly dependent upon the accompanying speech or sign language for its correct interpretation;
- is being used to very different extent by different people, some never using it, others using it quite frequently. Graphic iconic gesture thus represents style of communication rather than a necessary component of the contents of communication.
Aptitude: Graphic iconic gesture
- is apt for increasing immersiveness and liveliness of presentation by adding a simple visual sketch to a spoken or sign language narrative.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
12a1e. Graphic metaphoric gesture [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, metaphoric.
Specific properties: Metaphoric gesture typically makes a very sketchy analogue rendering of some source domain and wants it interpreted as being about some different target domain. Graphic metaphoric gesture
- is metaphoric gesture meant to be seen;
- typically consists in making a very sketchy analogue rendering of something or someone by means of hand-arms, other body parts, or the body as a whole, while explaining through speech or sign language that the rendering should be interpreted metaphorically as being about something else, like when we give our personal rendering of how something blows up and say "And then he blew up!";
- is highly personal and strongly dependent on the accompanying speech or sign language for its correct interpretation;
- is being used to different extent by different people. Some never use graphic metaphoric gesture, which thus represents style of communication rather than a necessary component of the contents of communication.
Aptitude: Graphic metaphoric gesture
- is apt for increasing immersiveness and liveliness of presentation by adding metaphoric use of a simple visual sketch to a spoken or sign language narrative.
Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: arbitrary, graphic, static.
Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already, like creating a blue team and an orange team prior to playing some game. Arbitrary static graphics
- express information through static graphic representations defined ad hoc.
Aptitude: Arbitrary static graphics
- are apt for adding information contents to representations in static graphic modalities, for instance by exploiting unused or under-used information channels, such as colour or texture, for instance for assigning different meanings to the differently coloured lines in a static graphic line graph (modality 9d).
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
14. Arbitrary static-dynamic acoustics [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: arbitrary, acoustic, static/dynamic.
Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already, like agreeing to whistle once if everything is calm and three times if there is danger. Arbitrary acoustics
- express information through acoustic representations defined ad hoc;
- has static repreentation as a limiting case, i.e., short-term endlessly (unless stopped by user) repetitive arbitrary acoustics.
Aptitude: Arbitrary acoustics
- are apt for adding information contents to representations in acoustic modalities, for instance by exploiting unused or under-used information channels, such as pitch, loudness, or time;
- are apt for providing salient information in a low-noise environment.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
15. Arbitrary static-dynamic haptics [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: arbitrary, haptic, static/dynamic.
Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already. Arbitrary haptics
- express information through static or dynamic haptic representations defined ad hoc.
Aptitude: Arbitrary haptics
- are apt for adding information contents to representations in haptic modalities, for instance by exploiting unused or under-used information channels, such as texture, or time.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: arbitrary, graphic, dynamic.
Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already, like creating a blue team and an orange team prior to playing some game. Arbitrary dynamic graphics
- express information through dynamic graphic representations defined ad hoc.
Aptitude: Arbitrary dynamic graphics
- are apt for adding information contents to representations in dynamic graphic modalities, for instance by exploiting unused or under-used information channels, such as colour, texture, or time.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: structure, graphic, static.
Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Static graphic structure
- structures static or dynamic graphic information contents by using static graphic information channels to spatially separate individual entities, create spatial groups and relationships etc., like in graphic toolbars or tables.
Aptitude: Static graphic structure
- is apt for expressing the existence of relationships among static or dynamic graphic entities - of similarity, difference, correspondence etc.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
18. Static-dynamic acoustic structure [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: structure, acoustic, static/dynamic.
Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Acoustic structure
- structures static or dynamic acoustic information contents by using the acoustic information channels to separate individual entities, create groups and relationships etc., like when pausing before proceeding to a different topic or separating groups of acoustic entities in 3D acoustic space.
Aptitude: Acoustic structure
- is apt for expressing the existence of relationships among acoustic entities - of similarity, difference, correspondence etc.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
19. Static-dynamic haptic structure [back to top, Tree 1, Tree 2, Tree 3, List]
Inherits properties from: structure, haptic, static/dynamic.
Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Haptic structure
- structures static or dynamic haptic information contents by using haptic information channels to spatially or temporally separate individual entities, create spatial or spatio-temporal groups and relationships etc., like in haptic toolbars or tables
Aptitude: Haptic structure
- if static, is apt for expressing the existence of relationships among static haptic entities - of similarity, difference, correspondence etc.
- raises interesting issues about the aptitude of dynamic haptic structure and haptic structuring of dynamic haptic entities, in particular for the blind and hard-of-seeing who, unless aided by technology, are only able to constantly monitor what they touch.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Inherits properties from: structure, graphic, dynamic.
Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Dynamic graphic structure
- structures static or dynamic graphic information contents by using information channels of dynamic graphics to spatially or temporally separate individual entities, create spatial or spatio-temporal groups and relationships etc., like when, visibly, different groupings keep forming dynamically among a population of static or dynamic graphic representations.
Aptitude: Dynamic graphic structure
- is apt for expressing, or signalling, the existence of dynamically changing relationships among entities - of similarity, difference, correspondence etc.;
- is apt for creating a dynamically changing focus, especially in static graphics.
Descendants: This generic-level modality has no descendants in the current version of the taxonomy.
Multimodal combination:
Taxonomy modifications compared to book version [back to top, Tree 1, Tree 2, Tree 3, List]
Modality theory is work in progress. Preparation of the Modalities part of this website has led to the following extensions and modifications of the taxonomy presented in the book:
- Super level: The term "explicit" has been removed from the structure modalities 17, 18, 19 and 20.
- Generic level:
- Atomic level:
- Sub-atomic level:
- Sub-sub atomic level:
- Super level:
- Generic level:
- Atomic level:
- Sub-atomic level:
- Sub-sub atomic level:
