We have developed three training and two test scenarios for
the Decision-Making Evaluation Facility For Tactical Teams
(DEFTT) simulator. These scenarios have been used in studies
for training metacognitive skills for Naval officers (Cohen,
Freemen, Wolf, Militello, 1995; Freemen, Cohen, 1996). The
scenarios were derived from existing DEFTT simulations, and
were enhanced to include tracks whose behaviors were modeled
after tracks discussed at length in the critical incident
interviews conducted as part of the TADMUS program. We are
currently translating these scenarios from DEFTT to the CIC
simulator. We have copies of scenarios developed by Sandra
Marshall, and will work with her and the other team to develop
a common set of scenarios for testing the learning characteristics
and performance of the various hybrid architectures, and for
comparison with human subjects.
Working with human subjects (16 officers with CIC or equivalent
experience) and the scenarios identified above, we have collected
data that includes assessments of likely intent, judgments
of confidence in those assessments, explanations and criticisms
of the most likely intent, and actions that officers consider
appropriate for the current situation. (The findings of this
study are summarized below.) These data are being used to
develop the training program for the hybrid architecture,
and will be used to compare its performance to that of the
human subjects in experiments this fall.
Related data from another subject pool (Surface Warfare Officers
School; 60 Naval officers, 92% with shipboard CIC experience)
is available, and may be analyzed if required to form a more
extensive basis for training. Further data regarding the domain
or metacognitive performance of human subjects will be collected
as required using Towne's CIC simulator.
Long-Term Knowledge Base
Using critical incident interviews collected as part of the
TADMUS program we have encoded a Long Term Knowledge Base
(LTKB). This encodes domain knowledge required to combine
evidence concerning the behavior of tracks in support of hypotheses,
e.g., 'search-and-rescue,' 'commercial air,' or 'hostile-intent',
and associates situation models structures with plan structures,
e.g., 'issue warning,' 'VID track,' or 'illuminate track.'
Working with the SHRUTI simulator, we are now verifying that
the LTKB can draw the inferences that support reflexive recognitional
responses and metacognitive reasoning.
Hybrid architecture design & implementation
We have constructed adaptive critic memories and performed
various tests concerning improved exploration methods, the
use of measures of confidence in estimates of expected value,
and the relative merit of various associative memory structures
and algorithms. We are currently integrating the various components
of the hybrid architecture (the CIC simulator, Shruti and
the LTKB, and the adaptive critic architectures) for machine
learning experiments this fall.
Year two: Metacognition organizes domain
learning
In the second year, we will explore how metacognitive behavior
organizes and influences domain learning. This occurs at two
time scales. (1) Evidence is received and reflexively combined
into stories (causal structures that organize evidence within
frames that, e.g., explain intent). Metacognitive skills critique
these reflexive results, and set the goals for adapting the
results (correcting). As the stories are adapted, new situation
models and plan structures are instantiated. These are then
stored back into the domain model (a la Case Based Reasoning),
where they are available for future recognitional pattern
matching. (2) At a slower time scale, metacognition provides
for the review of prior episodes. Here metacognition sets
knowledge acquisition goals that focus learning behavior.
At this level, metacognition can be sensitive to a number
of factors involved in developing a rich and diverse domain
understanding, including the abilities to explain conflicting
evidence, to identify unreliable evidence, and to complete
gaps in understanding. Incorrect, or inappropriate, explanations
are also identified.
Recent accomplishments
Two important findings have emerged over the
past 12 months in relationship to this work. In studies with
human subjects in Naval combat simulations, we have validated
(a) measures of metacognition and (b) methods of manipulating
metacognition via training, (c) we have collected data concerning
human metacognitive performance. Based on this work, we can
proceed to develop parallel (a) measure, (b) training programs,
and (c) performance criteria for the hybrid system. (2) An extension
to the SHRUTI architecture has been developed that supports
encoding and efficient reasoning over inconsistent knowledge.
This advance in SHRUTI's functionality will enable the hybrid
system to recognize inconsistant knowledge and use it, as human
decision makers do, to improve situation models. These findings
are summarized below.
Learning metacognitive skills
A study of training based on the Recognition
/ Metacognition model was conducted at the Naval Postgraduate
School (NPS) at Monterey, CA. Thirty-five officers with an average
of ten years of military experience participated in the study.
CTI's training was designed to help officers formulate situation
assessments in a highly realistic simulation of CIC AAW operations,
critique those assessments, correct them, and take action (that
is, ceasing critical thinking) in a timely manner. Subjects
received familiarization with the domain and the simulator,
performed a pretest, received training (involving lectures,
discussion, reading a training text, and performing exercises
using paper and the simulator), and then completed a posttest.
Results of the study indicated that training influenced metacognitive
skills, as it was design to do. Training enabled officers to:
devise more arguments in defense of their assessments
(an increase of 25%, p=.001; An assessment of argument quality
is in progress. In an earlier study for ARI, CTI found that
quality rose (ns) as the number of arguments rose)
identify more of the assumptions underlying their assessments
(an increase of 41%, p<.001)
recognize more of the evidence conflicting with a given
assessment (an increase of 58%, p<.001)
specify more of the exception conditions under which seemingly
conflicting evidence was consistent with the given assessment
(an increase of 28%, p<.001)
Training also had a positive effect on decision outcomes, this
is, on the bottom line. It enabled officers to:
make more accurate assessments of highly ambiguous, high-stakes
situations (an increase of 30%,p=.001 for one of the two
test scenarios, ns for the second scenario)
take actions that were more appropriate to the situation
(p=.039)
maintain their confidence (i.e., their ability to act)
even as they conducted more through critiques of their assessments
(confidence rose 12.5%, ns).
Officers rated the training positively, and were more likely
to do so the greater their tactical experience.
These results are significant for the hybrid architecture
project in several ways. (1) They demonstrate that we have
developed effective measures of several metacognitive skills
used by Naval officers in complex CIC scenarios. On this basis,
we can implement similar measures to monitor and test metacognitive
operations in the hybrid system. (2) We have developed training
(based on the Recognition / Metacognition model) that influences
human use of metacognitive skills and domain outcomes. This
training will be used to develop the training program for
metacognitive machine learning in the hybrid architecture.
(3) The results of the human training and the machine learning
can be directly compared in terms of the elicited metacognitive
measures.
Modeling inconsistent knowledge
Where traditional analytical processes simply
aggregate concordant and conflicting data, we have observed
that officers treat the two types of information quite differently:
Conflicting evidence (e.g., regarding intent) is used as a symptom
of erroneous assumptions and spurs efforts to find alternative
interpretations of cues or alternative hypotheses. (Cohen, 1986,
1989). Metacognitive correcting actions attempt to resolve the
three classes of problems identified by critiquing (incompleteness,
conflict, and unreliability). They do so by selecting operations
that will transform some part of the situation model, or cause
it to be abandoned in favor of a different one. Experienced
officers use several tools in their efforts to correct the deficiencies
of models. To represent and reason efficiently over inconsistent
knowledge it is crucial to model these human behaviors.
SHRUTI is a connectionist model of reflexive reasoning. It
demonstrates how connectionist networks can represent relational
structures and perform certain types of computations over
such structures in an efficient manner. The SHRUTI architecture
is being utilized in this project to encode the domain model
and plan structures, and to support recognitional elaboration
of situation models and plans in response to evidence. The
results instantiated in the SHRUTI network are reviewed by
metacognition (criticized and corrected), subject to available
time, stakes, uncertainty and novelty.
Recent work has extended SHRUTI such that it can now deal
with positive knowledge as well as negated facts and systematic
knowledge (rules) involving negated antecedents and consequents.
The extension only requires local inhibitory connections.
The extended model explains how an agent can hold inconsistent
knowledge in its long-term memory without being 'aware' that
its beliefs are inconsistent, but detect a contradiction when
two contradictory beliefs that are within a small inferential
distance of each other become co-active during an episode
of reasoning. Thus the model is not logically omniscient,
but detects contradictions whenever it tries to make use of
inconsistent knowledge in particular situations. The extended
model also explains how limited attentional focus or action
under time pressure can lead an agent to produce an erroneous
response. The extended SHRUTI model is therefore capable of
modeling a wider range of reflexive reasoning phenomena. Shastri
and Grannes (1995).
This work has implications both for training and for the
design of the hybrid architecture. Training may be improved
by focusing on bringing together logically inconsistent aspects
of the knowledge base and providing officers with improved
skills for detecting, attending to, and resolving conflict.
By explicitly modeling inconsistent knowledge, the hybrid
architecture is able to maintain alternative explanations,
to identify conflicting conclusions, and to activate metacognitive
processes for improving situation models.
References
Cohen, M; Freemen, J; Wolf, S; Militello, L
(1995) Training Metacognitive Skills in Naval Combat Decision
Making. Arlington, VA. CTI, Inc.
Freeman, J, Cohen, M. (1996). Training for Complex
Decision-Making: A Test of Instruction Based on the Recognition
/ Metacognition Model. Proceedings of the 1996 Command and
Control Research and Technology Symposium, Monterey, CA.
Cohen, M, Freeman, J, Wolf, S. (In press). Metarecognition
in Time-Stress Decision Making: Recognizing, critiquing, and
correcting. Journal of the Human Factors and Ergonomics Society.
