Short Paper as a Proposal to attend the Workshop on Handheld CSCW at CSCW '98
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
The Pebbles project is creating applications to connect multiple hand-held Personal Digital Assistants (PDAs) to a main computer such as a PC. We are currently using 3Com PalmPilots because they are popular and widespread. We created the "Remote Commander" application to allow users to take turns sending input from their PalmPilots to the PC as if they were using the PCs mouse and keyboard. "PebblesDraw" is a shared whiteboard application we built that allows all of the users to send input simultaneously while sharing the same PC display. We are investigating the use of these applications in various contexts, including co-located meetings.
Keywords: Personal Digital Assistants (PDAs), PalmPilot, Single Display Groupware, Pebbles, Amulet.
Note: This is an extended abstract of a paper to appear at CSCW'98.
The full paper is also available in Postscript (about 4.4 megabytes) or Adobe Acrobat (pdf) (about 400K).
There are certain kinds of meetings, including design reviews, brainstorming sessions, and organizational meetings, where a PC is used to display slides or a current plan, and the people in attendance provide input. When a PC is used as part of the presentation, often different people will want to take turns controlling the mouse and keyboard. For example, they might want to try out the system under consideration, to investigate different options, or to add their annotations. With standard setups, they will have to awkwardly swap places with the person at the PC.
We observed that at most meetings and talks, attendees do not bring their laptops, probably because they are awkward and slow to set up, the batteries may not last long enough, and there is a social stigma against typing during meetings. Today, however, many people are taking notes on their hand-held "Personal Digital Assistant" (PDA), such as a PalmPilot or a Windows CE device. A PalmPilot is a small PDA from 3Com with a 3¼ inch diagonal LCD display screen which is touch sensitive, and a small input area for printing characters using a special alphabet (see Figure 1). PalmPilots are small, they turn on instantly, the batteries last for weeks, and notes are taken by writing silently with a stylus. With the new palm-size PCs from Microsoft just starting to be available, this type of device will become even more ubiquitous. Since many people are already using a PDA, we developed a set of applications to explore how these could be used to allow everyone to provide mouse and keyboard input to the PC without leaving their seats. We call this the "Remote Commander."
Once the PDAs are connected to the PC, many other intriguing applications become available. In addition to the Remote Commander, we also created a shared drawing program, named PebblesDraw, that allows everyone to draw simultaneously. In creating PebblesDraw, we had to solve a number of interesting problems with the standard widgets such as selection handles, menubars and palettes. Since this is a Single Display Groupware (SDG) application, all users are sharing the same screen and therefore the same widgets. Palettes, such as those to show the current drawing mode or the current color, normally show the current mode by highlighting one of the items in the palette. This no longer works if there are multiple people with different modes. Furthermore, the conventional menubars at the top of the window or screen are difficult to use in multi-user situations because they may pop up on top of a different users activities. The conventional way to identify different users is by assigning each a different color, but in a drawing program, each user might want to select what color is being used to draw, causing confusion between the color identifying the user, and the color with which the user will draw. We therefore developed a new set of interaction techniques to more effectively support Single Display Groupware.
This research is being performed as part of the Pebbles project. Pebbles stands for: PDAs for Entry of Both Bytes and Locations from External Sources. We will be presenting a full paper on this topic at CSCW'98. This document summarizes the CSCW'98 paper, includes a brief description of a new system we have created to support Chat, and then discusses why I would like to attend the workshop.
Figure 1. One version of the 3Com PalmPilot running the Pebbles Remote Commander application.
The PalmPilot is a small inexpensive hand-held "Personal Digital Assistant" (see Figure 1) formerly sold by USRobotics (which was bought by 3Com) and also now sold by IBM as the "WorkPad." The PalmPilot sold over one million units in its first 18 months, and many people in our academic community are using them to take notes in meetings. Although our research is using PalmPilots because they are easy to program and popular, the research would equally apply to any PDA. We will soon port the software to palm-size PCs running Microsoft Windows CE.
One of the most important reasons the PalmPilot is so popular is that it connects very easily to a PC (and also to a Macintosh or Unix workstation) for synchronization and downloading. Each PalmPilot is shipped with a cradle and wire that connects to a computers standard serial port. Software supplied with the PalmPilot will synchronize the data with the PC. It is also easy to load new applications into the PalmPilot. Pebbles takes advantage of this easy connection to a PC.
The first application from our Pebbles project is the Remote Commander. This allows strokes on the main display area of the PalmPilot to control the PCs mouse cursor, and for Graffiti input to emulate the PCs keyboard input. The important point is that this works with all existing PC applications, without requiring any modifications to the applications themselves. For example, multiple people can use PowerPoint, Word, Excel, Illustrator, etc., from their PalmPilots.
We believe in distributing the results of our research, to help collect useful feedback and aid in technology transfer. The first version of the Remote Commander software was released for general use on February 17 (see http://www.cs.cmu.edu/~pebbles) and was downloaded about 3000 times in the first six weeks. It works with Windows 95 or NT, and current versions of the PalmPilot.
In designing this application, we wanted to make sure that it would fit in with the PalmPilot style. Therefore, it needed to be natural and non-intrusive. It should be easy to switch between using the Remote Commander and using other PalmPilot applications, and it should take advantage of the Graffiti gestures that the users had already learned. We also had to deal with the PalmPilots limitations: a small, grayscale screen, and a fairly slow processor. Furthermore, the users focus should normally be directed to the PCs screen, so it should be possible to operate the Remote Commander without looking at the PalmPilot, just as the mouse can be used without looking.
In the current version of Remote Commander, we rely on social protocols to control whose turn it is, which is not difficult since everyone is in the same room. Everyones inputs are mixed together. We decided that a more formal "floor control" mechanism would obstruct the fluid sharing and collaboration that is typical in meetings. This is supported by experience and informal experiments with other groupware systems.
When designing the Remote Commander, we had a number of uses in mind. The first is as discussed above: for multiple people in a meeting to take turns controlling the mouse and keyboard. Remote Commander is designed to support meetings where everyone is in the same room. (Remote participants might use other CSCW applications, such as Microsoft NetMeeting.) Remote Commander might be used so different people can have a turn controlling an application, to support multiple annotations on a work product, or so everyone can enter their personal data into a shared document. Having a single, shared visual display is important for coordination and collaboration, and people will take turns or all work together on these displays. A second use for Remote Commander is to support small, informal meetings. These arise when two or three people are discussing something that is running on a PC, such as a demo situation, or for small impromptu design and debugging discussions.
Figure 1 shows the PalmPilot running the Remote Commander. The user can move the stylus in the upper area to move the PC's cursor, and the user can write Graffiti characters to input characters. There is also an on-screen keyboard that can be used (see Figure 2). To press the mouse buttons, the user can tap on the cursor area, or use the up and down physical scroll keys in the bottom-middle of the PalmPilot (see Figure 1). Full details of the operation are in the CSCW paper.
Figure 2. The Remote Commanders on-screen keyboard. Clicking in the "abcdef" area gets this keyboard.
To use the PalmPilot to control the PC, they need to be connected to each other. Our goals for the connection technology include: minimal interference with the use of the PalmPilots, continuous connections while users are writing on the PalmPilots, multiple PalmPilots operating at the same time in the same room, and minimal power consumption.
Ideally, we would use some sort of wireless technology. The most popular short-range wireless technology is infra-red (IR), but this has many problems for our application. For example, the IR built into the new PalmIII version of the PalmPilot appears to transmit for only about 1 meter, and it must be aimed at the receiver (in the same way that a TV remote control must be aimed at the TV). One reason we want to attend the workshop is to see how others have used various wireless technologies.
In the meantime, we are using a wired solution, which has the advantage of being flexible, inexpensive, and easy to set up. It lets us explore the various user interface and systems issues, and is practical for use today.
The Remote Commander currently assumes the devices are connected using the built-in serial ports of the PalmPilot and the PC. The easiest way to do this is to simply use the cradle provided with the PalmPilot. However, using the PalmPilot while it is in the cradle is quite clumsy, so instead, we usually use the HotSync cable sold by 3Com, which does a nice job of staying attached to the PalmPilot without getting in the way. It is also inexpensive. In order to connect multiple PalmPilots to a PC at one time, the PC needs multiple serial ports. We bought a PCMCIA card for our laptops to provide this.
In addition to Remote Commander, we built another application, called PebblesDraw, to explore allowing all users to have their own cursors. PebblesDraw is a multi-cursor drawing program that is analogous to other "shared whiteboard" applications, with one important difference: here all the users share the same display. Thus, PebblesDraw qualifies as Single Display Groupware (SDG). PebblesDraw is implemented using the Amulet toolkit on the PC.
Figure 3. First version of PebblesDraw.
The initial design for PebblesDraw is shown in Figure 3. Clicking on the "Add User" button at the bottom allows the name, serial port number and the shape for that user to be entered. All active users are shown along the bottom of the window. At the left are the conventional drawing and color palettes. At the right is a button panel that contains the most common commands.
In creating this application, we discovered a number of important issues not addressed by previous SDG applications. In particular, the standard widgets, like selection handles, palettes, and pull-down menus, do not work for multiple users. Most of the issues are independent of the particular input device used, so they are not particularly relevant to the topic of the workshop. See the full paper for details.
We have not yet had much experience with using Remote Commander in meetings, and we have never tried using PebblesDraw. One of our plans for the fall is to perform a number of studies of using PalmPilots in various kinds of meetings, to see how they work in practice. We want to explore issues of effectiveness, various floor control mechanisms, and whether PalmPilots can replace a room-full of PCs as is needed for Group Decision Support Systems. There are many issues in the design of these experiments, and we will be interested in feedback from the workshop members.
Since the CSCW paper was submitted, we have created a number of new applications for using the PalmPilots along with a PC.
First, we created a variety of ways to use the PalmPilot to scroll the contents of the PC's screen, to help investigate one way that a single user might benefit from using a PalmPilot and a PC at the same time.
More relevant to the theme of this workshop is our Chat program (see Figure 4), which allows users to send side messages to each other. When various users are connected to the PC using the Remote Commander or PebblesDraw, then a user can switch to PebblesChat on the PalmPilot and send a message to all the other connected users or to a specific user by name. The receiver(s) will be notified of an incoming message and can reply.
Figure 4. The new PebblesChat program that lets one user send a message to another or to all others.
As mentioned above, an important focus of our future work will be studies to determine how well these PDA applications work in meetings. We would like to study some real meetings, in addition to the controlled meetings we will need for formal studies. We also want to explore various options for connecting the PDAs to the PC (maybe using the "BlueTooth" radio technology), and we will continue to refine these applications and their architecture. We also plan to port these applications to Window CE PDAs.
Of course, we want to explore many other applications of these ideas to new domains. The CSCW literature contains many interesting programs designed for multiple computers, such as "Electronic Brainstorming" and "Structured Idea Generation Process" from Univ. of Arizona and Xerox PARCs Cognoter. These are often called "Group Decision Support Systems." We want to see which of these will be effective if used with PalmPilots and a single PC display, instead of a special room full of conventional computers.
Another direction is to investigate taking more advantage of the PDA's screen. Instead of having the users current modes shown in the cursor and home area as in PebblesDraw, they might be shown on each persons PDA screen. Then, the user could easily see and choose the desired mode. Once there is a protocol for applications to download their controls onto the PDA, this could be extended to allow the PDA to be a general-purpose controller for any device, not just a PC program. For example, the PDA could control house-hold appliances, light switches, copiers, and thermostats.
We have developed applications to investigate many interesting uses for multiple PDAs connected to a PC. The Remote Commander application allows multiple co-located users to take turns controlling the mouse and keyboard of a PC without leaving their seat. The PebblesDraw application demonstrates interaction techniques that allow multiple users to work on the same display simultaneously. These tools can be useful for formal and informal meetings. Using PDAs in this way takes advantage of technology that people already have and have already learned, and does not necessarily require changes in work habits. We are excited about the potential of this approach and the positive response we have received from initial users.
The Pebbles project is allowing us to investigate many interesting research questions in the area of Single Display Groupware, and how multiple people can use small hand-held computers synchronously in meetings. New interaction techniques and software architectures have been developed, and we have many interesting ideas for the future.
This research is directly in line with many of the topics of the workshop, and I believe the workshop attendees will be interested in the issues we have identified with using multiple hand-held devices while they are attached to a PC. I will be happy to give a demonstration at the workshop of the applications we have created so far, if there is interest. There seems to have been very little study about using hand-held devices while communicating with a PC, yet this will be increasingly important and relevant as new technologies like BlueTooth start to be deployed and make it easy to have multiple PDAs communicating with each other, with PCs and with other devices in the same room. There is a rumor that the next generation of PDAs will have rechargable batteries that recharge while the PDA is connected to the PC (with a wire), so this would further encourage applications like ours that use the PDA connected continuously to a PC.
We are hoping to find out at the workshop what other people are doing in this area, and what approaches people have been using, especially for communication and testing. Can IR be made to work effectively? What are the alternatives? Have others had success with hand-helds wired to other devices and machines? How can appropriate studies of these new technologies be conducted? What are appropriate and inappropriate uses for hand-held devices? etc. I would very much like to attend the workshop to discuss these and other issues.
Also working on the Pebbles applications have been Herb Stiel, Robert Gargiulo, Carl Evankovich, and Ben Bostwick.
Brad A. Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects including: User Interface Software, Demonstrational Interfaces, Natural Programming, and the Pebbles PalmPilot Project. He is the author or editor of over 175 publications, including the books "Creating User Interfaces by Demonstration" and "Languages for Developing User Interfaces," and he is on the editorial board of five journals. He has been a consultant on user interface design and implementation to 30 companies. Myers received a PhD in computer science at the University of Toronto where he developed the Peridot UIMS. He received the MS and BSc degrees from the Massachusetts Institute of Technology during which time he was a research intern at Xerox PARC. From 1980 until 1983, he worked at PERQ Systems Corporation. His research interests include User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility.