From: Phil Cohen [pcohen3@mindspring.com]
Sent: Sunday, January 30, 2000 7:41 AM
To: albrecht@teco.uni-karlsruhe.de; Phil Cohen
Subject: Re: Situated Interaction in Ubiquitous Computing

POSITION PAPER

Phil Cohen, David McGee, and Sharon Oviatt
Center for Human-Computer Communication
Oregon Graduate Institute


This workshop presupposes that the device itself is aware
 of its location and situation of use.  Granted
that this will be possible soon (if not now).  It will be able
 to adapt its behavior to suit
the situation, task, and user.   However, just as strongly,
the user will adapt his/her usage
patterns to suit the situation and location.   For example,
user studies in our laboratory and others  have shown that
users of multimodal systems will prefer to speak in some
situations, but not others;  will prefer to handwrite/draw/gesture
in some environments and for some contents,  will alter their
behavior if one modality is failing (e.g., noisy environment), or if they
suspect that they will be unable to perform adequately
(e.g., not know how to pronounce a foreign surname).
Likewise, some user responses to the
environment will lead to much worse system performance,
as when a user adjusts his/her volume and speech characteristics
to noisy environments (e.g, Lombard speech).   In such environments,
multimodal systems will be able to provide both alternative modalities,
as well as modalities that when used in combination can overcome the
weaknesses of one with the strengths of another.

In the case of our handheld multimodal QuickSet system at OGI (Cohen et al.,
1997), we have  demonstrated that such mutual disambiguation of modalities
(here, speech and pen-based gesture) can enable
robust performance in the face of noisy environments,
and in response to foreign accents  (Oviatt, 1999).    We will discuss how
mutually compensating multimodal systems can be built, which will be able to
adapt to their environment, and to their user's patterns.  However, we need
to be careful in not adapting too quickly, since many of the environmental
factors
are transitory (e.g., a plane passing by).  Too quick an adaptation could
cause the system to be unpredictable.

BACKGROUND:

At OGI, we have built QuickSet, a wireless, collaborative, agent-based
multimodal system that is used to control distributed applications, such as
control of virtual environments, medical informatics, emergency planning,
and military simulations.   The system runs on computers ranging
from wearables to wall-sized.  Recently, it has been extended to support
multimodal interaction with paper-based interfaces.   Research is
underway with Columbia Univ. to develop a multimodal wearable augmented
reality system.

A demo can be given.

References:

Cohen, P. R., Johnston, M., McGee, D., Oviatt, S.,
   Pittman, J., Smith, I., and Chen, L.,
   "QuickSet: Multimodal interaction for distributed applications,"
  Proc. of the 5th Int'l ACM Multimedia Conf., ACM Press, New York 1997,
31-40.

Oviatt, S. L.,  "Mutual disambiguation of recognition errors in a multimodal
architecture," Proc. of Human Factors in Computing Systems (CHI'99),
ACM Press, New York, 576-583.