In M. Smith, G. Salvendy & R. Koubek (Eds.) Design of Computing Systems: Social and Ergonomic Considerations (pp 79-82). Amsterdam: Elsevier, 1997.

Multi-agent human performance modeling in OMAR*

Stephen Deutsch

BBN Corporation, 10 Moulton Street, Cambridge, MA 02138, USA


Simulation has been used for many years as a tool to evaluate system performance, but it is only recently that attempts have been made to include realistic models of human operators in these evaluations. The Operator Model Architecture (OMAR) is a simulation system that addresses the problem of modeling the human operator. Its development focused first on the elaboration of a psychological framework that was to be the basis for the human performance models, and then on the design of a suite of software tools to support the development of these models. In using OMAR to build human performance models, particular attention has been paid to the representation of the multi-tasking capabilities of human operators and their role in supporting the teamwork activities of the operators.

The ability to model human operators and their interactions with other team players and with their target systems has opened up several new areas for investigation through simulation. Procedure development is an important one. Simulation can now have an impact on both operator-procedure and maintenance-procedure development. It will now be possible to pursue procedure development by evaluating procedures far earlier in the design cycle of a system than was previously possible. The evaluation of both operating and maintenance procedures as part of the design process can lead to significant improvements in system operability and reduce downstream maintenance costs.


The modern workplace is seldom the province of a single person. A person's work is most frequently part of a larger effort, linked more or less closely to the work of others, either at a nearby workplace or at a remote site. It is becoming more common for people to work with others at remote sites as the capabilities of networked systems improve to support this mode of operation. On the one hand, team members can be viewed as resources to assist in the accomplishment of a task, while on the other hand, they can be the source of interruptions to ongoing tasks as they seek assistance in meeting their own objectives. To address the goal of modeling the teamwork of human players, human performance models must be capable of accurately reflecting human multi-task behaviors and the accompanying shifts in attention that these behaviors demand.

Proactive activities require that attention be focused to support a given task. Similarly, reactive demands are made on attention by interruptions that may be auditory or visual. To improve the behaviors of the human performance models, it was important to examine the nature of teamwork and particularly its impact on attention, to improve those aspects of the model that represent multi-tasking, and to enable the human performance model to exhibit reasonable behaviors in the teamwork aspects of the tasks being executed.

2.1. An architecture for modeling human multi-task behaviors

The basic architectural components of the OMAR human performance model (Deutsch & Adams, 1995) are implemented at the symbolic level. They are patterned after the large-scale structure of the brain as outlined by Edelman (1987, 1989) and Damasio (1989a, 1989b). The performance of a particular capability is typically implemented through the participation of a small number of functional centers. Subsets of these centers, operating concurrently, typically have links between them operating in both directions -- reentrant signals as described by Edelman (1987). The operation of a set of "lower" level centers is coordinated by a "higher" center, or convergence zone in Damasio's (1989a, 1989b) terminology. Damasio's convergence zones are much like Minsky's (1986) hierarchy of agents in The Society of Mind.

The Simulation Core (SCORE) language, the OMAR procedural language, provides the basis for representing functional capabilities. A SCORE procedure is used to represent a simple capability. The procedure is typically in a wait state, pending activation based on a match to a particular stimulus pattern. Several such procedures may represent the components of a particular functionality. The procedures form a network along whose links pass the signals that each of them may generate. For any signal generated, one or more related procedures may be enqueued on it. Depending on the response of their respective pattern matchers, some procedures may be activated by a stimulus, while others may ignore it. The pattern of activation in a complex of procedures differs due to variations in the stimulus patterns.

Within any given complex, several procedures may be running concurrently, some representing automatic processing, others representing components of attended processing. In the several layers of concurrent processing, a sensory input may have initiated the "lowest" processing level, with each subsequent "higher" processing layer starting up at the behest of the initial "output" from the next lower level. The behaviors of the concurrent processes are based on those discussed by Jackendoff (1987) in Consciousness and the Computational Mind.

2.2 Attention

Within this architectural framework, attention is not simply one component or one complex of procedures. Following Neumann (1987), attention is a "generic term for a number of phenomena each of which is related to a different selection mechanism." In building the model of attention, a selected subset of these phenomena related to air traffic control and aircrew tasks were implemented. The focus was on auditory and visual processing, since they are the most important forms of attention in managing the air traffic control and flight deck workplaces.

The work on auditory attention addressed the verbal communications of air traffic controllers and aircrew members, either in person, via telephone, or over the party-line radio. The work on visual attention focused on the air traffic controller's use of the synthetic radar screen and visual support for, and coordination of, manual workplace tasks. Both visual and auditory attention have proactive and reactive components. Verbal communication is the basis for proactive coordination of flight-deck activities, while party-line radio communications are frequent interruptions to ongoing activities. In the visual domain, the appearance of a new aircraft icon on the radar screen may interrupt an ongoing activity, while proactive visual attention is required to accomplish simple manual actions such as setting mode control panel values.

The processing of auditory messages is modeled by a set of concurrent processes. At the lowest level is a "hearing" process that is initiated in response to the onset of the auditory input. Shortly after the hearing process is initiated, it in turn triggers a cognitive "message- understanding" process. Lastly, an "attended" cognitive process represents the hearer's attending and reacting to the spoken message. The nature of the communication over the party- line radio made it necessary to factor in another layer of complexity. From an aircrew's perspective, many air traffic controller messages are directed to other aircraft in the airspace and can be ignored at some level. Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy (1995) suggest that this is determined as soon as a verbal discriminator appears. The air traffic controllers identify the target aircraft for each message as the first utterance in a message, but it is clearly not the case that both the "understanding" and "attended" processing stop at this early point in message processing. An aircrew member will not initiate a verbal communication with another crew member while the "ignored" message is still in process. The speculation represented in the model is that the "understanding" process continues processing the incoming message, and it is this process that "flags" the end of the message, so that another verbal message resuming the interrupted person-to-person conversation among crew members may be initiated.

The presence of the party-line radio means that auditory interruptions are an expected occurrence. In particular, it must be possible to stop an intra-crew conversation at the onset of a radio message and resume the conversation at the completion of the interruption. SCORE priorities assigned to in-person and radio conversations are used to implement the processing of the interruption by a new radio message. The intra-crew verbal transactions are typically a statement-response pair. An interruption anywhere in the exchange will be resumed, not at the point of the interruption, but by the initial statement of the exchange being repeated.

Visual attention, as modeled in OMAR, also has proactive and reactive components. The proactive processes are executed primarily in support of related cognitive and manual processes. The reactive processes are concerned with the response to visual events. The air traffic controller's synthetic radar workplace is a visually rich environment. The visual events modeled include the appearance of a new aircraft icon on the radar screen as the aircraft approaches the air traffic controller's airspace, the movement of the aircraft icon across the screen as its position is updated, and a flashing light on the telephone to announce an incoming call from a neighboring controller. The initial response to a visual event is the simple act of identifying the event, followed by a sequence of signals that trigger the appropriate network of procedures to respond to the event. The events that occur are not unexpected and there is typically a goal that has set up and is governing the response to each event. The execution of the response at each stage is mediated by the priority associated with the response procedure and that of the other ongoing procedures. Visual attention in support of cognitive procedures takes several forms. Probably the most complicated activities take place as the air traffic controller "sits down" at the radar console to take over control of the airspace from the previous controller at the start of a shift. There are several aircraft in the sector and in neighboring sectors. Flight strips, arrayed at the side of the radar screen, provide flight-plan information on active and pending aircraft. The air traffic controller's initial acts are to "read" the active flight strips, associate each with the appropriate aircraft icon on the radar screen, and initiate a procedure for managing that aircraft's transit through the airspace. Implicit in the procedure for managing the aircraft is the memory of where the aircraft icon appears on the radar screen. The expert air traffic controller, like the expert chess player, maintains the knowledge of where the pieces are on the board by constantly revisiting them. In the human performance model, this memory is local to the procedure for managing the particular aircraft, rather than a slot in a global short-term memory resource.


The human players modeled in the air traffic control environment include the en route and approach controllers and the aircrews of the aircraft in their sectors. Recently developed scenarios have examined the ATC hand-off of aircraft from one controller to a neighboring controller in high traffic density conditions. Earlier studies examined aircrew and ATC procedures for approach and landing sequences, including top-of-descent negotiation and procedures in which radio-based voice communication was replaced by data-link for aircrew/ATC communication. The studies, typically based on high-density air traffic conditions, have required the simulation of large numbers of human players. The efficiency of OMAR in human performance modeling has made this possible.

The aircrew processing of an ATC directive is depicted in Figure 1. Following company- based policies, the pilot-not-flying (PNF) responds to the ATC directive, while the pilot-flying (PF) also attends to the communication and has responsibility for acting on the directive. The onset of the ATC communication may well have interrupted verbal communication between crew members on the flight deck that will have to be resumed. Radio communications to other aircraft in the sector are attended in a closely related manner and may similarly be the cause of interruptions to verbal communication to coordinate flight deck activities.

Expectations also play a critical role in the coordination and accomplishment of team tasks and their modeling is essential to the evaluation of operating procedures. In addition to the basic processing of the auditory message, the expectations that support cross-checking are important components of OMAR human performance modeling capabilities. Based on the expectation generated by the ATC directive, the pilot-not-flying will check that the pilot-flying has properly set the mode control panel (MCP) in compliance with the directive and intervene if the directive is not accurately executed in a timely manner.

Figure 1. Aircrew processing an ATC directive.


Teamwork, an essential component in the successful operation of many systems today, makes significant demands on the multi-tasking capabilities of individual team members. Additional demands on attention are made by the mix of proactive and reactive activities that force the interruption and resumption of tasks in the complex flow of activities to accomplish individual and team goals. By addressing the nature of teamwork and carefully modeling its impact on multi-tasking and attention, the behaviors of the OMAR human performance models were improved to better represent this elaborate set of human capabilities. These advances in human performance modeling have made it possible to develop and evaluate new operating procedures involving large numbers of team players in complex time-critical environments.


Damasio, A. R. (1989a). The brain binds entities and events by multiregional activation from convergence zones. Neural Computation, 1, 123-132.

Damasio, A. R. (1989b). Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition. Cognition, 33, 25-62.

Deutsch, S. E., & Adams, M. J. (1995). The operator-model architecture and its psychological framework. 6th IFAC Symposium on Man-Machine Systems. MIT, Cambridge, MA.

Edelman, G. M. (1987). Neural Darwinism: The theory of neuronal group selection. New York: Basic Books.

Edelman, G. M. (1989). The remembered present: A biological theory of consciousness. New York: Basic Books.

Jackendoff, R. (1987). Consciousness and the computational mind. Cambridge, MA: The MIT Press.

Minsky, M. (1986). The society of mind. New York: Simon and Schuster.

Neumann, O. (1987). Beyond capacity: A functional view of attention. In H. Heuer & A. F. Sanders (Eds.), Perspectives on perception and action (pp. 361-394). London: Lawrence Erlbaum Associates.

Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995) Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634.

* The research reported on here was conducted under USAF Armstrong Laboratory Contract No. F33615-91-D-0009.

Return to the OMAR Index Page