I’m a big fan of NUI applications, which makes working at InterKnowlogy a great fit. Here at IK we’ve done extensive work with touch applications, Microsoft Surface, and Kinect, for example. There is not always a clear definition, however, of what makes a UI a NUI. What exactly do we mean when we say “Natural” User Interface? The term NUI is usually used to indicate the use of more natural interaction with the virtual objects displayed on the screen. Rather than using the abstraction of a mouse to click on objects, we might instead use a touch screen and let a user actually touch the object with a finger, as if it were real. Or we might let Kinect observe a user’s motions, driving interaction and interpreting complex gestures. New methods of physical interaction with the computer are certainly part of making a user’s experience more natural, but just as important are ways of strengthening the analogy of the virtual objects to real world counterparts.

In the early days of computing, the conceptual objects of an application were clearly distinct from any real world counterpart. Applications were text based, and the user was left to use his imagination to picture these concepts as real objects. Applications eventually became more visual, and the conceptual objects within them became increasingly represented visually as real world counterparts. Documents looked like paper, recipes were shown on virtual index cards, and reminders were shown as post-it notes. With the advent of XAML-based UIs, designers have been given incredible power to visualize data in creative ways. Data can now be represented elegantly, sometimes modeling concrete real world objects, and sometimes modeling more abstract concepts. But in either case, a data object is frequently treated as if it has physical presence, even if only on the screen. Users increasingly expect to be able to interact with the objects similarly to how they would interact with real world objects. Give a toddler an iPad, and he’ll quickly figure out the basic modes of interaction, because the objects react as you would expect them to. This is one way that I would define NUI: the objects in the application act and react as you would expect them to, based on your experience in the real world.

A topic that has captured my interest is the importance of animation within effective NUI. Animation used inappropriately is annoying at best. But used effectively, it can bring an application to life, making the objects behave more naturally. The best use of animation, in my opinion, is when it aids in keeping a user oriented within the application. In the real world, objects don’t disappear from one location, reappearing elsewhere instantly. Instead they move smoothly from one location to another, and their progress through time helps us to understand the transition. In applications, however, we routinely expect a user to understand what has happened when objects have jumped instantly from one spot to another. (Unfortunately, this is still typically true of applications that are described as NUI apps.) Instant changes to data layout are even more disorienting when they are driven by sources other than the user (perhaps by a remote user with whom you’re interacting, or by live changes to the data from an external feed). If the user makes a change himself, he at least will understand why the layout of data suddenly changed, even if he is not entirely certain what moved where. At best, instant changes to data layout cause a user’s experience to lose some of the NUI feel, because the illusion of working with real objects is shattered.

WPF offers some nice support for animations, from the use of storyboards to drive dependency properties, to Visual State Managers for organizing and automating the transitions required to morph data from one visual representation to another based on logical states. But WPF is lacking other basic support for NUI animations. It’s still the norm to see data jump instantly from one location to another, whether as a result of moving from one collection to another, or from the re-ordering of data within a collection, or from the introduction or deletion of data within a collection, or as the change in a detail view when selection in a master view is changed. Animating these types of changes is still often difficult, with limited support provided. And yet it is this type of animation that is, in my opinion, most important for a good NUI experience, because these are the most common changes to the data themselves. It is data that the user is most interested in staying oriented to, so we should strive to give visual cues that make changes to data understandable and relatable. Ideally, adding support for smooth (“fluid”) transitions of data layout would be simple, with the appropriate XAML simply indicating this preference declaratively.

There have been a few great starts towards this goal in XAML, but much more work is required. (Specifically, the FluidMoveBehavior and FluidLayout features in Blend are very powerful. See here for a great summary.) In future posts I will describe the attempts that Microsoft has made towards supporting such functionality, summarizing their strengths and limitations. (Examples of limitations are the inability to produce expected results when dragging-and-dropping data, and the lack of support for rotations in the transitions.) I will also discuss my own ongoing attempt to write a Fluid Layout library that provides a much richer solution to this challenge. Check back for future posts, and hopefully I’ll have some useful discussion and code to share soon!