In his analysis of direct manipulation, Nielson says that the computational device (which reminds me of Don Norman’s ‘information appliances‘) is the “ultimate in directness”: Not only is there a direct mapping from commands to goals, but a direct mapping from physical actions to commands. Direct manipulation interfaces, such as desktop GUIs, only give the first part of this mapping.
Today’s computer input devices and today’s computer user interfaces are analogous: What you do with a mouse, or a tablet stylus, has a direct correspondence to what happens on the screen; likewise, what you do with the little pictures on your screen translates into (notionally) corresponding operations on the computer’s state. Broadly, these correlations are intended to make using a computer familiar and thereby easier. However, there is a disconnect: Although what you have in your hand behaves as a real object, what appears on screen doesn’t, and not only because interface designers make some odd decisions. There is an indirection between the input gestures (e.g., moving the mouse) and commands in the interface (deleting a file), which is that you have an avatar in the interface world—the mouse pointer—as an intermediary between syntax and semantics.
Why have a mouse pointer at all? A scene in the film Minority Report featured an operator manipulating digital objects on a screen using hand gestures (though the awkwardness of the invented gestures had geeks everywhere clutching their tendons in sympathy). Recently, a group at NYU created a working prototype of a similar system. The syntax of these interfaces is immediately accessible, and the semantics follow the syntax according to how closely the digital objects simulate real objects. The syntax is, however, strictly limited to what is simulated.
Haptic input devices like the Sensetable also obviate intermediaries. With the AudioPad, pucks with some electronic kit inside represent microphones and samples. Move a sample puck closer to the microphone puck and the sample is played louder. Now one has a command language where the physical attributes of the input device (e.g., position) are the syntax. Further, the abstract object now has the noisy analogue behaviour that real objects have, like sliding and spinning, which expands the vocabulary of commands and smears it into a continuum.
The advantage of indirection is that it provides a shield of abstraction: my mouse is always a mouse but the mouse pointer can be a cursor, a file being moved, a finger for pressing buttons. It can also be different objects, because when I set down something I’m manipulating it goes back to being just digital state and I can manipulate something else. This kind of multiplicity isn’t possible without the indirection. There is a halfway position, which is to have generic input objects that can assume the identity of particular digital objects. This is actually what AudioPad does—a puck can ‘be’ any sample at a point in time.
Here is where we meet up with consumer electronics. While in the past we’ve largely had tools that embody verbs, nowadays we have devices that embody nouns. Your iPod is your music collection. I think we’ll soon see a convergence whereby computer input devices become more specialised as their nouns are manifested as real objects (take a look at the wacky game controllers available), and consumer electronic devices shift in sympathy, and as technology allows, to having richer, nuanced vocabularies.