In emergency response, gathering intelligence is still largely a manual process despite advances in mobile computing and multi-touch interaction.  Personnel in the field take notes on paper maps which are then manually correlated to larger paper maps that are eventually digested by global information systems (GIS) specialists who geo-reference the data by hand.  Unfortunately, this analog process is personnel intensive and can often take twelve to twenty-four hours to get the data from the field to the daily operation briefings.  This required processing means that the information digested by personnel going into the field is at least an operational period out of date. In a day where satellite photography and mobile connectivity is becoming ubiquitous in our digital lives, we believe that we can improve this state of the practice for most disciplines of emergency response.

Our goal is to integrate multiple digital sources into a common computing platform that provides two-way communication, tracking, and mission status to the personnel in the field and the incident command hierarchy that supports them.  Advanced technology such as geo-referenced digital photography and ground robots are still limited to sending video only to the operators at the site, but this limitation is changing quickly. Recent advances in robotics, mobile communication, and multi-touch displays are bridging these technological gaps and providing enhanced network centric operation and increased mission effectiveness.

Our research in human computer interaction leverages these technologies through the use of several different types of multi-touch displays. A single-robot operator control unit and a multi-robot command and control interface has been created. It is used to monitor and interact with all of the robots deployed at a disaster response.  Users tap and drag commands for individual or multiple robots through a gesture set designed to maximize ease of learning. A trail of waypoints can provide specific areas of interest or a specific path can be drawn for the robots to follow. The system is designed as discrete modules allow integration of a variety of data sets like city maps, building blueprints, and other geo-referenced data sources. Users can pan and zoom on any area, and it can integrate video feeds from individual robots so you can see things from their perspective.

Manual robot control is achieved by using the DREAM (Dynamically Resizing Ergonomic and Multi-touch) Controller. The controller is virtually painted beneath the user's hands, changing its size and orientation according to our newly designed algorithm for fast hand detection, finger registration, and handedness registration.  In addition to robot control, the DREAM Controller and hand detection algorithms have a wide number of applications in general human computer interaction such as keyboard emulation and multi-touch user interface design.

Current work is moving from simulated robots to real robots. Having successfully controlled multiple robots with our three main multi-touch devices (Microsoft Surface, Apple iPad, and 3M Multi-Touch Displays) work is now being done to integrate these technologies to work together simultaneously.