The Shruti model and implementation
have been extended in a number of ways, including: improved
abductive reasoning; multiple-consequent rules; soft / evidential
rules, facts and evidence combination calculus; priming and
recency effects; supervised learning via backpropagtion in
the structured network; and, most importantly, the model has
been augmented to integrate the propagation of "belief" with
the propagation of "utility". This integration will allow
the model's sense of utility to direct its search for answers
and explanations and focus its activation along paths that
promise to have a high utility. We have continued to develop
the connections between the recognitional and metacognitive
systems.
This work includes an experiment
in which we tested the efficacy of a metacognitive "intervention"
on the construction of a situation estimate and on learning
in the reflexive system. The machine learning experiment utilized
one of the core scenarios identified in the tactical decision
making domain of the hybrid computational architectures program
(Korea). In this scenario, some kinematic cues suggest hostile
intent (an aircraft flying at low altitude toward own ship).
Others, however, do not (speed is only moderate), and there
is also the possibility that the aircraft is being deployed
with the intent of defending another platform against U.S.
ships. We developed an event tree to reflect branching possibilities
of key events in this scenario. Among these possibilities
were hostility between the US and the other country, the appropriateness
of the approaching contact for use as an attack or rescue
platform, intent (to attack U.S. ships, or to protect the
other platform from the same country), and kinematic features
of the track (bearing, altitude, and speed). Probabilities
were assigned to the branches of this tree in order to create
a set of scenarios representative of causal / evidential relationships
in the real-world domain underlying the scenario. A randomly
selected half of these scenarios were used for training, and
half for testing. Backpropagation was used to adapt the weights
of the reflexive system in the training scenarios. In the
reflexive-only condition, each scenario was trained for 50
cycles of relaxation. In the metacognitive training condition,
a hint to "think about" (i.e., query) non-hostile intent was
inserted midway through these relaxation cycles.
In both learning conditions,
rules for concluding hostile intent from purely kinematic
cues (e.g., heading toward own ship at low altitude) were
reduced in strength. However, changes in the knowledge base
were more pervasive in the metacognitive learning condition,
and applied to inferences of hostile intent from contextual
as well as kinematic cues. For example, in the metacognitive
condition, the reflexive system learned that when tensions
between two countries are high, the intent of a military platform
may be to protect other platforms rather than attack own ship.
A series of four behavioral studies has examined the effect
of training metacognitive skills on tactical decision making
by 201 active-duty Naval officers in scenarios similar to
the core scenarios. The training emphasized four aspects of
metacognition, which were presented as the S.T.E.P. technique
for tactical decision making:
* Story-building: When faced with a situation in which events
are difficult to interpret, stakes are high, and time allows
for deliberation, formulate a story that motivates and explains
past and present events.
* Testing the story: Ferret out evidence that conflicts with
this story. Try to generate an explanation for this evidence
that preserves core aspects of the story, such as your assessment
of the intent of a suspect track.
* Evaluating the story: Judge the plausibility of the story
you have generated, bearing in mind assumptions you have made
to handle conflicting and missing evidence. If the story is
weak, build a new story, test it, and evaluate it.
* Planning against weak assumptions: Formulate contingency
plans to protect against the failure of key assumptions.
Officers in these experiments executed high-fidelity AAW
scenarios on computer-based simulators, and, in written tests,
assessed the intent of experimenter-selected tracks from each
scenario, critiqued assessments, and defended them. From these
data, we assessed the effects of training on tactical decisions
by comparing the accuracy of officers' assessments with those
of an SME. We evaluated the effects of training on metacognitive
processes with counts of the arguments officers made concerning
assessments.
In each of the four experiments, every reliable effect of
training on metacognitive processes was positive and large
(ranging from 23% to 94%). Furthermore, the mean score on
each process measures rose with training in every experiment,
and though some of these increases were not statistically
reliable, the pattern over all studies and all measures was
consistently positive.
Experiment |
#1 |
#2 |
#3 |
#4 |
Metacognitive
processes |
|
|
|
|
Story-building:
Defending preferred assessments |
+ |
+23%** |
+26%(p<.10) |
+ |
Story-building:
Defending non-preferred assessments |
+ |
+23%** |
+40%* |
+ |
Testing
stories: Identifying conflict |
+52%* |
+58%** |
+ |
+94%* |
Testing
stories: Explaining conflict |
+ |
+27%** |
+23%* |
+ |
Evaluating
stories: Generating alternative assessments |
+ |
+41%** |
+ |
+48%* |
Other |
|
|
|
|
Confidence
in preferred assessment |
+ |
+18%* |
+ |
+ |
Table #1: All reliable effects on measures of metacognition
were positive, and mean scores on all measures rose with training
in all experiments. ** = p < .01; * = p < .05; + = positive
but statistically unreliable improvement with training.
Metacognitive skills training also had positive effects on
the accuracy of tactical decisions. In every case in which
training had a statistically reliable effect on decision accuracy,
that effect was large and positive. The smallest effect on
accuracy was an improvement of 35%; in the largest case, training
produced accuracy rates of 83% vs. 16% among controls. In
addition, officers' confidence in their assessments rose at
the mean in every study. (The effect on confidence was reliable
in only one experiment, however).
In sum, training metacognitive skills appears to improve
decisions and the processes by which officers make their decisions,
and also to improve the process by which the hybrid architecture
adapts its beliefs.
|