Marilyn Jager Adams
Bolt Beranek and Newman Inc.
10 Moulton Street
Cambridge, MA 02138
Abstract: The operators of complex equipment are frequently members of a team who must manage their control functions across numerous interruptions. They succeed, in part, because of their multi-tasking skills. The OMAR System includes a suite of representation languages as the basis for constructing models of these human multi- tasking behaviors. Prior to the development of the computational languages, a psychological framework was developed that attempts to identify key elements of the computational foundation for these behaviors. The psychological framework and the design of the representation languages for developing models of human multi-tasking behaviors are described.
Keywords: Human supervisory control, Man-machine interaction, Simulation languages, Knowledge representation
In investigating human operator performance in the management of complex equipment one must recognize that, it is often the case that, the operators are members of a team executing their tasks in the presence of frequent interruptions. Exploring this performance through simulation requires new levels of fidelity in human performance modeling. The Operator-Model Architecture (OMAR) is a new computational framework designed to support the development of simulation models of human agents interacting with other human agents, both simulated and real, in executing these complex tasks. While OMAR is intended to support the development of a broad range of perspectives vis-à-vis modeling methodologies, its development has been guided by a particular psychological framework. The development of this framework has been based on recent research in experimental psychology, cognitive science, neuropsychology, and computer science. It reflects the requirement to portray the human operator's complex capability to intermix thoughtful and automatic behaviors in addressing proactive and reactive situations.
In developing a psychological framework, it is important to identify and quantify the sources of human performance in a manner that makes it possible to develop a computational model of these behaviors. The literature on experimental psychology reinforces the basic premise that human performance capability is rooted in the ability to manage multiple "simultaneous", but raises many questions with respect to the structure of multi-task behaviors and automaticity. These are the factors that shape both novice and skilled performance, and are the sources of human error at each skill level. The neuroscience and computer science literatures have also been relevant. Gerald Edelman's reentrant and global maps have provided a neuroscience basis for the procedure network that is described later in this paper for modeling multi-task performance and automaticity. The representations of goals and plans, the data-flow model of procedure activation with data arriving from multiple sources, and the race/join semantics for specifying the parallel execution of network procedures have been adapted from computer science.
In the formulation of the psychological framework, the Memory Module effectively comprises a large number of process or event memories of varying degrees of complexity and works with a variety of data. These memories communicate with one another via a complex of excitatory and inhibitory interconnections. The strength of any given complex of connections reflects its frequency of use as well as its current level of activation, such that the strength of its response to a message is a joint function of learning, expectation or priming, and the compatibility of the message with its structure.
Complementing the Memory Module, a Cognitive Module is proposed. Within the Cognitive Module, the activities and interests of oneself as well as the impinging world are represented in terms of means-ends frames. This is consistent with Kant's (1787/1965) arguments that neither superficial similarities nor contiguities in space and time are enough to support rationality in an associative model -- a person's ability to sort and order experiences of the phenomenal world depend additionally on an understanding of causality. The knowledge structures of the Cognitive Module are envisioned to consist, in part, of goals with plan structures similar to those developed in the cognitive science and artificial intelligence literatures.
The work of the system is carried out through the interarticulation of the Memory and Cognitive Modules. Memory messages trigger means-ends frames of the Cognitive Module. Unresolved means-ends frames seek out relevant data from the Memory Module, selectively activating prior knowledge or priming the system for relevant messages from the environment. In addition, with particularization and resolution, the frames direct action and response by invoking and, hence, coordinating and monitoring the effects of the corresponding action knowledge in the Memory Module. Thus, it is the transactions between the Cognitive and Memory Modules that govern the systemπs receptivity while weaving its fabric of cognitive coherence and behavioral integrity.
Neuroscience and the Psychological Framework
In developing the psychological framework, the potential for the contribution of recent work in the neurosciences could not be overlooked. Edelman (1987) discusses the psychological functions of "development, perception (in particular, perceptual categorization), memory, and learning" and how they relate to the brain. Edelman (1989) extends his analysis to consider "perceptual experience -- the interaction of memory with the present awareness of the individual animal," that is, perceptual awareness and conscious experience.
Reentrant Maps, Global Maps, and Degeneracy: Neural maps refer to the ordered arrangement and activity of groups of neurons as distinct from single-neuron connections. They are highly and individually variant in their intrinsic connectivity. Changes in the behavior of the network are the result of changes within particular populations of synapses. "These structures provide the basis for the formation of large numbers of degenerate neuronal groups in different repertoires linked in ways that permit reentrant signaling" (Edelman, 1987, p. 240) where, in degenerate systems, functional elements in a repertoire may perform more than one function and a function may be performed by more than one element (Edelman, 1987, p. 57). Reentry is a basic mechanism suitable for synchronizing the neuronal activity across the mappings at diverse hierarchical levels. Global mappings have a dynamic structure that reaches across reentrant local maps and unmapped regions of the brain to account for the flow from perception to action. Motor activity, an essential input to perceptual categorization, closes the loop.
A Neuroscience View of Memory as Active: In contrast to the view of memory as storage -- as a data base -- Edelman, as did many who preceded him, views memory as process. For him, memory is the "ability to categorize or generalize associatively" (Edelman's italics, 1987, p. 241). Categorization occurs at the level of a global map and is degenerate. Edelman is well aware of the distinctions between declarative and procedural memory, but he is also quick to point out that these distinctions may be less than generally assumed. He suggests that there may be a procedural base supporting declarative memory.
In Edelman's view of memory as process, perception, categorization, generalization, and memory are closely linked. "Memory is a form of recategorization based upon current input; as such, it is transformational rather than replicative" (Edelman, 1987, p. 265). Memory is an active process of classification leading to recategorization and, thus, a partitioning of the world that is presented as one "without labels." Storage, to the extent that it exists, is one of procedures for mapping inputs to responses; hence, full representations of objects are neither stored nor required: "It is the complex of capacities to carry out a particular set of procedures (or acts) leading to recategorization that is recollected" (Edelman's italics 1987, p. 267). This view contrasts sharply with memory cast as data residing in a data base, where content is passive, references are made to it, it may fade with time, and in the case of short-term memory, new memories may reinforce or replace existing memories. In such schemes, something processes memory as data, reinforcing some of it and degrading other parts of it.
Related theories, focusing on the dependence of learning on the learnerπs actions or responses, have posited that even ostensively static, declarative knowledge is represented in memory only as it is embedded in the larger context, including the actions and events through which it was learned and is used (e.g., Neisser, 1976; Yates, 1985). That is, procedural knowledge is distinctly active. People have processes or procedures for doing things which they constantly modify and adapt to use in their everyday being. They take inputs from the world, process them and effect changes in the world around them. The processes work more or less well with each other depending on circumstances. In this sense, memory is procedural knowledge and, in particular, has an active component. Accepting that "declarative" knowledge is embedded in procedural knowledge, semantic memory also becomes active and procedure-like. Short-term memory is a cluster of actors re-enforcing or suppressing one another, modulated by conflict-resolution that is itself based on the class structure of the memory processes. Long-term memory and explicit and implicit memory are variations on this theme of memory as process. In this spirit, Edelman's (1987, 1989) reentrant and global mappings are, in part, a data-flow network of processes.
Multi-tasking, Automaticity, and the Psychological Framework
Human performance models are expected to represent human behaviors in the operation of complex computer-supported equipment. The range of behaviors to be modeled have perceptual, cognitive, and motor components with varying degrees of complexity in each category, and there are frequently occasions in which operators are required to execute several tasks "simultaneously." In these models, it is also necessary to represent a range of human behaviors from novice to expert along one dimension and from errorful to error-free along another dimension. The domain is complex, but there is a wealth of knowledge on these issues in the experimental psychology literature. Because the richness of the psychology literature also represents a diversity of views, the first task was one of establishing a psychological framework in which the more important threads in this research may be represented.
Dual-Task Performance: Psychological theories of task interference can be divided into two categories: In one, processing draws on resources in a graded fashion; in the other processing resources can only serve one task at a time. To be sure, many theories are hybrids, of which the following are representative examples:
The Resource Model (Wickens, 1984) may be characterized by:
Toward reconciling such differences, note that the single-user processors of the Postponement Model may be considered "resources" that are bimodal; that is, a task gets all of the resource or none of it. In adopting this perspective in the psychological framework, it is posited that contention among particular classes of tasks is based on dynamically computed metrics.
Automaticity: Automatic processes typically differ from those of nonautomatic processes in the following ways (Logan, 1988; Shiffrin and Schneider, 1977): They are fast, effortless, autonomous or obligatory, able to operate in parallel, consistent or stereotypic, and unavailable to conscious awareness. Significantly, some theorists have stressed that some of these are relative characteristics (Cheng, 1985). Indeed, it is largely by probing the relative expression of these characteristics across experience and training that psychologists have begun to disentangle their underlying nature. These explanations may be divided into two categories. In the first category, increases in automaticity are attributed to the increased efficiency of processes per se. In the second, they are ascribed to more comprehensive and complete memory support for practiced tasks.
Within the view of automaticity as a product of processing efficiency:
In the psychological framework, a set of functional human capabilities were identified. OMAR (Deutsch et al., 1993) was designed to facilitate the implementation of human performance process (HPP) models with these capabilities (see Young, 1992). OMAR encompasses a suite of representation languages, graphic language editors and browsers, and a simulation environment. The Simple Frame Language (SFL) provides the traditional role of declarative knowledge representation and simultaneously forms the bridge to the object-oriented implementation based on Common Lisp. A graphic editor provides a network view of the object definition hierarchy and a table view of the slots of individual objects. Appropriate editing functionality is available in each view.
Agent behaviors are represented in the Simulation Core (SCORE) language whose implementation is based on ACTORS (Agha, 1986) semantics. SCORE facilitates the representation of an agentπs goals and plans and the networks of procedures that implement the behaviors. The rule language was included to represent novice through intermediate decision-making and problem-solving skills. The ability of tasks to wait for and to signal events provides the data-flow interactions discussed earlier. The development of SCORE procedures is aided by the graphic browser for individual procedures, and for the network formed by the set of procedures for an agent. Network views are available across the procedure caller/callee and generate-signal/await-signal perspectives. The SCORE simulator efficiently executes agent behaviors represented as compiled goals, plans, and procedures. Scenarios may be developed and executed with recorders and a traveler (Manning, 1987) available to provide on-line and post-run insight to agent performance through task and event timelines.
Attributes of HPP Models and OMAR Tools
The psychological framework identified particular aspects of human behaviors that are necessary for HPP model development. Features of the representation language that are essential to express these agent behaviors are the focus of the following discussion.
Proactive Behaviors:The psychological framework requires a representation of goals as the basis for proactive human behaviors. A goal expresses the conditions necessary for its achievement and includes a plan of subgoals directed toward the achievement of the goal. The leaves of the plan are distinguished by their invocation of and response to one or more procedures or agent actions. A goal expresses what is to be accomplished, and a plan outlines the steps to achieve the goal. The actions or procedures direct agent behaviors. SCORE, a procedural language, provides the representation of goals, plans, and procedures as defined here.
Multi-tasking and Task Contention:The modeling of multi-task behaviors requires a formalism to represent the interactions between tasks contending to execute. The psychological framework suggests basing this competition on the accrual of activation credits vis-à-vis other tasks. As tasks are developed, they naturally form contention classes -- classes within which the competition to execute will need to be modeled. The unique feature identified here is the classification of tasks by the way in which they compete with other tasks to execute. Depending on their internal structure and processing requirements, tasks may compete with other tasks within their own class or with tasks from different classes. SFL, used to define the basic objects of the OMAR environment, is used as the basis for specifying the class membership of SCORE tasks. SCORE tasks are SFL objects.
Given the classification of tasks, it was necessary to provide a basis for developing the protocols for implementing the explicit contention among tasks. Generic functions were developed for managing the decision to suspend or resume a task, as well as the actual suspension and resumption of individual tasks. The accrual of activation credits for a task is typically driven by events in the environment or by a cognitive action that promotes the activation of a particular task. Additionally, provisions were required to enable a suspended task to dictate its own behavior during the period for which it is suspended -- perhaps by simply adjusting the level of activation of the task over time. A task may also impact the activation of tasks of a given class operating in parallel with it. As the activation levels of tasks are adjusted over time, these functions provide the basis for a task asserting its right to execute. A loss in activation level for an active task will open a window of opportunity for competing tasks, while a gain in activation level by a nonexecuting task will enhance its opportunity to execute. The means to establish restart points for resuming a task is also provided. In this environment, there is no explicit scheduler. Instead, agent behaviors emerge from the dynamics of the activation levels among competing tasks.
Reentrant Map Semantics:The psychological framework is realized, in part, by a network of procedures. Emerging agent behaviors are the result of goal-directed actions and data-driven events. Task activation, with inputs arriving from multiple sources, results in a data-flow architecture that resembles Gerald Edelman's (1987) reentrant and global maps as described in his Theory of Neuronal Group Selection. Here, procedures correspond roughly to reentrant map elements that are driven by multiple inputs and that generate outputs for multiple consumers. The data flow in the network imposes sequential execution of procedures within a data-flow path, while enabling parallel execution of procedures along parallel paths.
In the data-flow architecture, all arguments need not come from a single source; that is, one procedure may provide some of the arguments to a subsequent procedure, but additional arguments from another procedure may be required before the subsequent data-flow node may execute. In a breaking with a traditional data-flow architecture, not all the arguments need necessarily be present for a procedure to execute. The psychological framework does not endorse a pure data-flow architecture.
A typical SCORE procedure is much like a Lisp function in that it may return values to its calling procedure -- but this event simply does not happen in data-flow networks. Instead, procedures in the data-flow network typically use the SCORE loop-forever macro, sending out their results using signal-event as they near the completion of an iteration and return to the top of its loop-forever to await its next activation. The top of the iteration cycle is typically a wait for the arrival of one or more signals. A set of SCORE forms, asynch-wait, with-signal and with-multiple-signals, are provided for this purpose. As this and other issues were addressed, the implementation has evolved into reentrant-map semantics that support the implementation of the psychological framework.
Parallel Execution:The parallel execution of procedures must be handled, both at the language level and in the simulator. At the language level, parallel execution may be expressed at two levels of abstraction. At the procedure level, the forms race, join, and satisfy manage the execution of sub-procedures in parallel. Parallel procedures operating under race all complete when the first one completes. For join, every subprocedure must complete before subsequent procedures execute. Within these parallel forms a failing sub-procedure will cause the entire form to fail immediately. The satisfy form traps sub-procedure failures and completes execution of the form when the first successful sub-procedure completes. In the plan for a goal the same forms are used to manage parallel subgoal execution. The SCORE compiler expands the parallel forms into a run-time form emulating parallel execution as outlined here.
Rule-Based Behaviors: While it is likely that expert knowledge formulated as rules will not lead to expert performance (Holyoak, 1991), there are performance levels at which human behaviors are rule-based (Rasmussen, 1983; Dreyfus and Dreyfus, 1986). To model these levels of performance, a rule-based system was required as part of the OMAR tool set. Since the reasoning that is modeled frequently involves decisions on actions to be taken, the rule system may reference SCORE-based goal, plan, and procedure definitions, just as it may reference any SFL object. Rules are assembled in rule packets that focus on particular domain issues. A procedure that invokes a rule set defines the context in which the rule set operates -- a function that must usually be covered by additional if-clauses.
Skill Levels in Human Performance: Human skill level has a significant impact on measured performance. HPP models developed in the OMAR environment must be capable of exhibiting performance associated with selected skill levels. Across a range of skill levels, there will be qualitative as well as quantitative changes in the actions that an agent will take to complete a particular task. In modeling these actions as procedures, the quantitative changes will reflect the efficiency with which the tasks are carried out. The transition from thoughtful deliberate action to automatic behavior is one of the more notable qualitative changes in performance with improved skill levels. The performance of a given task at these very different levels of performance will be represented by distinct sets of procedures, frequently with very different structure.
A commercial air traffic control scenario was the first modeling effort undertaken using OMAR. HPP models were developed for the controllers and the two-person flight crews of each aircraft. In the scenario the controllers manage the airspace through radio conversations with the flight crews and telephone conversations with neighboring controllers. The flight crews attend to controller directives and maintain a dialogue to coordinate flight deck operations.
The authors wish to thank Michael Young the USAF Armstrong Laboratory and Carl Feehrer of BBN for their continued support in this effort. The research reported on was conducted under USAF Armstrong Laboratory Contract No. F33615-91-D-0009.
Agha, G.A. (1986). Actors: A model of concurrent computation in distributed systems. Cambridge, MA: MIT Press.
Cheng, P.W. (1985). Restructuring versus automaticity: Alternate accounts of skill acquisition. Psychological Review, 92, 414-423.
Deutsch, S.E., M.J. Adams, G.A. Abrett, N.L. Cramer, and C.E. Feehrer (1993). RDT&E Support: OMAR Software Technical Specification, AL/HR-TP-1993-0027. Wright- Patterson AFB, OH.
Dreyfus, H.L., and S.J. Dreyfus (1986). Mind over machine. New York: Free Press.
Edelman, G.M. (1987). Neural Darwinism: The theory of neuronal group selection. New York: Basic Books.
Edelman, G.M. (1989). The remembered present: A biological theory of consciousness. New York: Basic Books.
Holyoak, K.J. (1991). Symbolic connectionism: Toward third-generation theories of expertise. In K.A. Ericsson and J. Smith (Eds.), Toward a General Theory of Expertise: Prospects and Limits. Cambridge, MA: Cambridge University Press.
Kant, I. (1787/1965). The critique of pure reason. New York: St. Martin's Press.
Logan, G.D. (1988). Toward an instance theory of automatization. Psychological Review, 95, 492-527.
Manning, C. (1987). Acore: The Design of a Core Actor Language. Masters Thesis, Massachusetts Institute of Technology.
Neisser, U. (1976). Cognition and reality: Principles and implications of cognitive psychology. San Francisco: W.H. Freeman.
Pashler, H., and J.C. Johnston (1989). Chronometric evidence for central postponement in temporally overlapping tasks. Quarterly Journal of Experimental Psychology, 41A, 19- 45.
Rasmussen, J. (1983). Skills, rules, and knowledge; signals, signs, and symbols; and other distinctions in human performance models. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 257-266.
Shiffrin, R.M., and W. Schneider (1977). Control and automatic information processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127-190.
Wickens, C.D. (1984). Processing Resources in attention. In P. Parasuraman and R. Davies (Eds.), Varieties of attention. New York: Academic Press.
Yates, J. (1985). The content of awareness is a model of the world. Psychological Review, 92, 249-284.
Young, M.J. (1992). A Cognitive Architecture for Human Performance Process Model Research. AL-TP-1992-0054, Wright-Patterson AFB, OH.