Abstract: The application of pattern classification techniques to physiological data has undergone rapid expansion. Tasks as varied as the diagnosis of disease from magnetic resonance images, brain-computer interfaces for the disabled, and the decoding of brain functioning based on electrical activity have been accomplished quite successfully with pattern classification. These classifiers have been further applied in complex cognitive tasks to improve performance, in one example as an input to adaptive automation. In order to produce generalizable results and facilitate the development of practical systems, these techniques should be stable across repeated sessions. This paper describes the application of three popular pattern classification techniques to EEG data obtained from asymptotically trained subjects performing a complex multitask across five days in one month. All three classifiers performed well above chance levels. The performance of all three was significantly negatively impacted by classifying across days; however two modifications are presented that substantially reduce misclassifications. The results demonstrate that with proper methods, pattern classification is stable enough across days and weeks to be a valid, useful approach.
Abstract: Abstract We recently proposed an adaptive filtering (AF) method for removing ocular artifacts from EEG recordings. The method employs two parameters: the forgetting factor û and the filter length M. In this paper, we first show that when û = M = 1, the adaptive filtering method becomes equivalent to the widely used time-domain regression method. The role of û (when less than one) is to deal with the possible non-stationary relationship between the reference EOG and the EOG component in the EEG. To demonstrate the role of M, a simulation study is carried out that quantitatively evaluates the accuracy of the adaptive filtering method under different conditions and comparing with the accuracy of the regression method. The results show that when there is a shape difference or a misalignment between the reference EOG and the EOG artifact in the EEG, the adaptive filtering method can be more accurate in recovering the true EEG by using an M larger than one (e.g. M = 2 or 3).
Abstract: Fatigue is a well known stressor in aviation operations and its interaction with mental workload needs to be understood. Performance, psychophysiological, and subjective measures were collected during performance of three tasks of increasing complexity. A psychomotor vigilance task, multi-attribute task battery and an uninhabited air vehicle task were performed five times during one night's sleep loss. EEG, ECG and pupil area were recorded during task performance. Performance decrements were found at the next to last and/or last testing session. The EEG showed concomitant changes. The degree of impairment was at least partially dependent on the task being performed and the performance variable assessed.
Abstract: OBJECTIVE: We show that psychophysiologically driven real-time adaptive aiding significantly enhances performance in a complex aviation task. A further goal was to assess the importance of individual operator capabilities when providing adaptive aiding. BACKGROUND: Psychophysiological measures are useful for monitoring cognitive workload in laboratory and real-world settings. They can be recorded without intruding into task performance and can be analyzed in real time, making them candidates for providing operator functional state estimates. These estimates could be used to determine if and when system intervention should be provided to assist the operator to improve system performance. METHODS: Adaptive automation was implemented while operators performed an uninhabited aerial vehicle task. Psychophysiological data were collected and an artificial neural network was used to detect periods of high and low mental workload in real time. The high-difficulty task levels used to initiate the adaptive automation were determined separately for each operator, and a group-derived mean difficulty level was also used. RESULTS: Psychophysiologically determined aiding significantly improved performance when compared with the no-aiding conditions. Improvement was greater when adaptive aiding was provided based on individualized criteria rather than on group-derived criteria. The improvements were significantly greater than when the aiding was randomly provided. CONCLUSION: These results show that psychophysiologically determined operator functional state assessment in real time led to performance improvement when included in closed loop adaptive automation with a complex task. APPLICATION: Potential future applications of this research include enhanced workstations using adaptive aiding that would be driven by operator functional state.
Abstract: Abstract The electro-encephalogram (EEG) is useful for clinical diagnosts and in biomedical research. EEG signals, however, especially those recorded from frontal channels, often contain strong electro-oculogram (EOG) artifacts produced by eye movements. Existing regression-based methods for removing EOG artifacts require various procedures for preprocessing and calibration that are inconvenient and timeconsuming. The paper describes a method for removing ocular artifacts based on adaptive filtering. The method uses separately recorded vertical EOG and horizontal EOG signals as two reference inputs. Each reference input is first processed by a finite impulse response filter of length M (M=3 in this application) and then subtracted from the original EEG. The method is implemented by a recursive leastsquares algorithm that includes a forgetting factor (û=0.9999 in this application) to track the non-stationary portion of the EOG signals. Results from experimental data demonstrate that the method is easy to implement and stable, converges fast and is suitable for on-line removal of EOG artifacts. The first three coefficients (up to M=3) were significantly larger than any remaining coefficients.
Abstract: We studied 2 classifiers to determine their ability to discriminate among 4 levels of mental workload during a simulated air traffic control task using psychophysiological measures. Data from 7 air traffic controllers were used to train and test artificial neural network and stepwise discriminant classifiers. Very high levels of classification accuracy were achieved by both classifiers. When the 2 task difficulty manipulations were tested separately, the percentage correct classifications were between 84% and 88%. Feature reduction using saliency analysis for the artificial neural networks resulted in a mean of 90% correct classification accuracy. Considering the data as a 2-class problem, acceptable load versus overload, resulted in almost perfect classification accuracies, with mean percentage correct of 98%. In applied situations, the most important distinction among operator functional states would be to detect mental overload situations. These results suggest that psychophysiological data are capable of such discriminations with high levels of accuracy. Potential applications of this research include test and evaluation of new and modified systems and adaptive aiding.
Abstract: The functional state of the human operator is critical to optimal system performance. Degraded states of operator functioning can lead to errors and overall suboptimal system performance. Accurate assessment of operator functional state is crucial to the successful implementation of an adaptive aiding system. One method of determining operators' functional state is by monitoring their physiology. In the present study, artificial neural networks using physiological signals were used to continuously monitor, in real time, the functional state of 7 participants while they performed the Multi-Attribute Task Battery with two levels of task difficulty. Six channels of brain electrical activity and eye, heart and respiration measures were evaluated on line. The accuracy of the classifier was determined to test its utility as an on-line measure of operator state. The mean classification accuracies were 85%, 82%, and 86% for the baseline, low task difficulty, and high task difficulty conditions, respectively. The high levels of accuracy suggest that these procedures can be used to provide accurate estimates of operator functional state that can be used to provide adaptive aiding. The relative contribution of each of the 43 psychophysiological features was also determined. Actual or potential applications of this research include test and evaluation and adaptive aiding implementation.
Abstract: The issue of crewmember workload is important in complex system operation because operator overload leads to decreased mission effectiveness. Psychophysiological research on mental workload uses measures such as electroencephalogram (EEG), cardiac, eye-blink, and respiration measures to identify mental workload levels. This paper reports a research effort whose primary objective was to determine if one parsimonious set of salient psychophysiological features can be identified to accurately classify mental workload levels across multiple test subjects performing a multiple task battery. To accomplish this objective, a stepwise multivariate discriminant analysis heuristic and artificial neural network feature selection with a signal-to-noise ratio (SNR) are used. In general, EEG power in the 31-40-Hz frequency range and ocular input features appeared highly salient. The second objective was to assess the feasibility of a single model to classify mental workload across different subjects. A classification accuracy of 87% was obtained for seven independent validation subjects using neural network models trained with data from other subjects. This result provides initial evidence for the potential use of generalized classification models in multitask workload assessment.
Abstract: Three-hundred Fischer 344 rats and 300 C57BL/6 mice of each sex were divided into three treatment groups and exposed intermittently (6 hr/day, 5 days/week) to JP-4 jet fuel vapors at concentrations of 0, 1000, and 5000 mg/m3 for 12 months. At exposure termination, 10% of the animals were killed and those remaining were held for a 12-month postexposure tumorigenesis observation period. Pathologic findings in male rats revealed treatment-related renal toxicity and neoplasia consistent with the male rat unique alpha 2 mu-globulin nephropathy syndrome. Distinct JP-4-induced respiratory toxicity was not observed, and pulmonary neoplasms were not significantly increased in any treatment group. Benign hepatocellular adenomas were slightly increased in high-dose female mice, but the trend was reversed in male mice. Other pathologic findings were regarded as equivocal or compatible with expected biologic variation. The study did not demonstrate target organ toxicity or carcinogenesis which could be extrapolated to other species.
Abstract: Near-infrared (NIR) sensing in flight applications can provide critical objective indicators of crew state. By monitoring
oxy-hemoglobin concentrations, a NIR sensor can detect changes in flight crew physiology in response to both cognitive demands
and extreme conditions related to flight applications, including gravity-induced loss of consciousness (G-LOC) and hypoxia.
A custom NIR sensor was created for in-helmet monitoring of oxy-hemoglobin in flight. This wearable, wireless sensor addresses
requirements for flight applications and was applied to a case study that examines the raw optical signal and oxy-hemoglobin
response to Valsalva maneuvers performed at 1g.
Abstract: Performance was significantly improved in an Uninhabited Air Vehicle (UAV) task when individually determined task difficulty levels were used to present psychophysiologically controlled adaptive aiding. Previous research in our laboratory demonstrated that the benefit of adaptive aiding varied according to the operatorâs skill level when a common task difficulty was used for all operators. In the present study the difficult task level was determined for each operator. An average task difficulty level was also used. The best performance occurred when adaptive aiding was presented based upon psychophysiological data submitted to an artificial neural network when the implementation level was individually determined for each operator. Performance improvement using the mean difficulty level was lower as were the results when the adaptive aiding was randomly presented. Individual cognitive capability must be considered to achieve optimal performance via adaptive aiding.
Abstract: We recently proposed an adaptive filtering method for removing ocular artifacts from EEG recordings. In this study, the accuracy of this method is evaluated quantitatively using simulated data and compared with the accuracy of the time domain regression method. The results show that when transfer of ocular signal to EEG channel is frequency dependent, or when there is a time delay, the adaptive filtering method is more accurate in recovering the true EEG signals.
Abstract: Psychophysiological measures and artificial neural networks were used to determine how well higher levels of cognitive activity, such as executive function, spatial and verbal working memory and global workload, could be assessed. A complex uninhabited air vehicle simulator was used in which subjects were responsible for four vehicles simultaneously. The subjects had to evaluate visual images and maintain the status of the vehicles. The results showed that the cognitive states, derived from subjective reports, could be accurately classified. These results have application in human factors environments which demand higher level cognitive processing and may be useful when implementing adaptive aiding in these systems.
Abstract: A system is described which provides a means for improving the quality of information derived in a decision making process by weighing certain multiple and alternative information channels. The method is applied to data estimating the cognitive workload state of a human operator dealing with a complex task using noninvasive sources of physiological data as a basis
Abstract: The realization of optimal system performance is the goal of both system designers and users. One critical component in attaining this goal is proper operator functioning. In contemporary systems the functional state of the operator is not considered during system operation. Degraded states of operator functioning can result from the demands of controlling complex systems, the work environment and internal operator variables. This, in turn, can lead to errors and overall suboptimal system performance. In the case of mental workload, system performance could be improved by reducing task demands during periods of operator overload. Accurate estimation of the operatorâs functional state is crucial to successful implementation of an adaptive aiding system. One method of determining operator functional state is by monitoring the operatorâs physiology. In the present study, physiological signals were used to continuously monitor subjectâs functional state and to adapt the task by reducing the number of subtasks when high levels of mental workload were detected. The goal was to demonstrate performance improvement with adaptive aiding. Because adaptive aiding during high mental workload has not been previously implemented its benefit has not be demonstrated. Application of adaptive aiding techniques reduced tracking task error by 44% and resource monitoring error by 33%. These results demonstrate the utility of adaptive aiding using physiological measures with artificial neural networks to determine the appropriate time to introduce the aiding.
Abstract: Technological advances in helmet-mounted displays (HMDs) have permitted the design of "see-through" displays in which virtual imagery may be superimposed upon real visual environments. The utility of see-through displays in multitask environments remains uncertain, especially in environments that involve switching one's attention between those tasks represented in the virtual display and those existing in the real world. The present study was designed to assess the effects of a secondary visual monitoring task on performance and workload in a head-slaved tracking task. Participants attempted to center a reticle over a moving circular target using a Kaiser Electronics SimEye 2500 HMD while concurrently performing the visual monitoring task component of the Multi- Attribute Task Battery (MATB; Comstock & Arnegard, 1992), which was displayed on a computer monitor. Task difficulty for the head-slaved tracking task was varied by manipulating time delay. Results are discussed in terms of their implications for practical implementation of see-through HMDs in multi-task environments.
Abstract: Operator functional state assessment is a critical component of adaptive aiding systems. A combination of physiological and performance variables were used with a neural network to determine operator functional state. A multiple task battery provided three levels of mental workload. The data were randomly divided into two data sets, one was used to train a neural net and the other to test the accuracy of the trained neural net. The results showed an overall correct classification of 86.8% for the test data set. For the three levels of task difficulty the correct classification was 90.5% for low, 81.7% for the medium and 88.3% for the high. These results support the use of combined physiological and performance data to obtain high levels of operator functional state classification accuracy. The optimal approach to utilizing this information during system operation will have to be developed. With current technology the development of small, wearable operator state classifiers is possible.
Abstract: The ability to correctly identify an operatorâs functional state is essential to being able to provide an adaptive interface between the system and the operator. The main obstacle to successful implementation of aiding is acquiring accurate knowledge of the operatorâs functional state. Nineteen channels of EEG were recorded while seven Air Force air traffic controllers performed a TRACON simulation task. Difficulty was manipulated by varying the total number of aircraft to be managed during 15 minutes. Four levels of difficulty were used. The most difficult condition, overload, was designed to cause the controllers to "lose" the picture. FFTs from overlapping ten second EEG epochs representing the middle five minutes of each condition were separated into five bands. Sixty percent of the epochs trained a linear discriminant classifier while the remainder were used to test classifier accuracy. Overall, the four difficulty levels could be correctly classified with 85% accuracy and the overload condition was correctly classified 94.5% of the time. These results demonstrate that psychophysiological measures can be used to accurately determine the level of mental workload experienced by an operator performing complex tasks. The high degree of accuracy for the most difficult condition is especially notable since mental overload detection is the primary goal of adaptive aiding systems. With knowledge about the operatorâs functional state during high levels of mental demand, the operator and/or system could off-load lower priority tasks to the system thus enabling the operator to concentrate on the primary task. This should improve performance on the crucial components of the higher priority task. Once the period of high demand is passed the operator could resume the off-loaded tasks. These data suggest that nonintrusive and continuously monitored EEG can be used to accurately assess operator cognitive load.
Abstract: Fly-by-wire control systems in advanced cockpits provide an opportunity to simplify the manual control demands on the pilot. However, this simplification may be at the expense of distancing the pilot from direct contact with important sources of information. Control loading systems provide the opportunity for enhancing the capacity of the stick as an information channel, providing the pilot with information about the critical aircraft state variables required for control. In this study parameters governing the movement of the pilotâs control stick (i.e. the stiffness of a spring-centered stick) were dynamically adjusted to be proportional to moment-to-moment states of the simulated vehicle (i.e. roll velocity). The hypothesis was that the âÂÂfeelâ of the dynamically varying stick would provide control information leading to more precise control performance in a single-axis roll tracking task. RMS error results did not support this hypothesis. The result is discussed in the context of an ongoing research program to examine strategies for information integration in advanced cockpits.
Abstract: This research demonstrated the first closed-loop implementation of adaptive automation using operator functional state in an operationally relevant environment. In the Uninhabited Combat Air Vehicle (UCAV) environment, operators can become cognitively overloaded and their performance may decrease during mission critical events. This research demonstrates an unprecedented closed-loop system, one that adaptively aids UCAV operators based on their cognitive functional state A series of experiments were conducted to 1) determine the best classifiers for estimating operator functional state, 2) determine if physiological measures can be used to develop multiple cognitive models based on information processing demands and task type, 3) determine the salient psychophysiological measures in operator functional state, and 4) demonstrate the benefits of intelligent adaptive aiding using operator functional state. Aiding the operator actually improved performance and increased mission effectiveness by 67%.
Abstract: Fatigue is a well known stressor in aviation operations and its interaction with mental workload needs to be understood. Performance, psychophysiological, and subjective measures were collected during performance of three tasks of increasing complexity. A psychomotor vigilance task, multi-attribute task battery and an uninhabited air vehicle task were performed five times during one night's sleep loss. EEG, ECG and pupil area were recorded during task
performance. Performance decrements were found at the next to last and/or last testing session. The EEG showed concomitant changes. The degree of impairment was at least partially dependent on the task being performed and the performance variable assessed.
Abstract: Determining operator cognitive or functional state is a critical component of adaptive aiding systems. To determine cognitive state, we must decide which measured features from the human will assist in distinguishing different levels of mental activity. A battery of psychophysiological signals was collected for two levels of cognitive workload from which 43 measures were derived. Three feature-reduction methods, principal component analysis, a weight-based partial derivative method, and a weight-based signal-to-noise ratio were applied, and the results were used as inputs to an artificial neural network for training and classification. Average classification accuracies up to 89.7 percent were achieved and the number of input features required was reduced by up to 84 percent.
Abstract: Determining operator cognitive or functional state is a critical component of adaptive aiding systems. To determine cognitive state, we must decide which measured features will assist in distinguishing different levels of mental activity. Psychophysiological signals were collected for two levels of cognitive workload from which 43 measures were derived. Three feature reduction methods were applied, and the results were used as inputs to an artificial neural network for training and classification. Average classification accuracies up to 89.7% were achieved and the number of input features required was reduced by up to 84 percent.
