|
Computational
Analysis of Nonverbal Behavior in Adaptive Tutoring |
Whitehill,
Littlewort, Reilly, Phan, Movellan, Bartlett
There has been
growing recognition of the importance of adaptive tutoring systems that respond
to the studentŐs emotional and cognitive state. However little is known about
childrenŐs facial expressions during a problem solving task. What are the
actual signals of boredom, interest, confusion, or uncertainty in real,
spontaneous behavior of students?
The field also is in need of spontaneous datasets to drive automated
recognition of these states. To date, there is a paucity of
empirical data to support understanding nonverbal behavior in teaching at a
computational level. The goal of this project is to develop
computational methods to measure and model the nonverbal behavior and
interactive strategies observed during face-to-face teaching. To support these models, we are
collecting datasets of student-teacher interactions as well as nonverbal
behavior during problem solving. The computational models will serve as a
foundation for a new generation of embodied teaching agents that approximate
the benefits of face-to-face human tutoring. This
research builds the foundation for automated tutoring systems that sense the
state of the student and adapt accordingly. The
project will help advance the science of learning and teaching by improving our
understanding of the dynamics of nonverbal behavior in teaching at a
computational level, across multiple time scales: From low-level micro-expressions
in the timescale of tens of milliseconds, to cognitive and affective processes
with time scales of seconds, to higher level strategic behaviors operating at
longer time scales.
NSF / IIS / HCC Grant: Computational Analysis of Nonverbal
Behavior in Adaptive Tutoring. PI: Bartlett. Co-I: Reilly, Movellan.
8/1/2009-7/31/2013.
NSF Science of Learning: Temporal Dynamics of Learning
Center, NSF SBE 0542013
Workshops:
NIPS Workshop: Personalizing
Education with Machine Learning. Organizers: Mike Mozer, Javier Movellan,
Rob Lindsey, Jake Whitehill. Neural Information Processing Systsems, Lake Tahoe
CA, Dec 8, 2012.
TDLC Workshop: Optimal
Teaching. Organizers: Javier Movellan and Jacob Whitehill. University of
California, San Diego, May 4, 2012.
Papers and
Presenatations
Sathyanarayana S, Satzoda, Carini,
Lee, Salamanca, Reilly, Forster, Bartlett, Littlewort (2014) Towards Automated
Understanding of Student-Tutor Interactions using Visual Deictic Gestures. Proc.
Computer Vision and Pattern Recognition Workshops (CVPRW), p. 480-487.
Sathyanarayana S, Littlewort G,
Bartlett M (2013). Hand Gestures
for Intelligent Tutoring Systems: Dataset, Techniques & Evaluation. Proc. IEEE International conference on
Computer Vision, Workshop: Decoding Subtle Cues from Social Interactions. pg
769 – 776.
Paper on IEEE Xplore
Dykstra
K, Whitehill J, Salamanca L, Lee M,
Carini A, Reilly J, and Bartlett, M (2012). Modeling One-on-one Tutoring
Sessions. Proc. International Conference on Development and Learning /
Epigenetic Robotics, San Diego, CA. Download pdf
Salamanca
L, Carini A, Lee M, Dykstra K,
Whitehill J, Angus D, Wiles J, Reilly J, and Bartlett, M (2012). Characterizing
the Temporal Dynamics of Student-Teacher Discourse. Proc. International
Conference on Development and Learning / Epigenetic Robotics, San Diego, CA. Download pdf
Brian
S, Salamanca L, Whitehill J, Reilly J, Bartlett M, Angus D, and Wiles J (2012).
Using Recurrence Plots to Visualize the Temporal Dynamics of Tutor/Student
Interactions. Proc. International Conference on Development and Learning /
Epigenetic Robotics, San Diego, CA. Download pdf
Salamanca L, Lee M, Carini A, Dykstra K, Whitehill J,
Bartlett M, Reilly, J. (2012) Profiling Student-Teacher Eye Gaze Behaviors.
Poster presentation, Southern California Cognitive Neuroscience Meeting at San
Diego State, March 2, 2012.
Whitehill, J., Serpell, Z., Foster, A., Lin, Y.C., Pearson,
B., Barlett, M., et al. (2011). Toward an Optimal Affect-Sensitive
Instructional System of Cognitive Skills. Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition Workshop on Human Communicative
Behavior, (pp. 20 - 25). Download
pdf
Littlewort GC, Salamanca LP, Reilly JS, and Bartlett MS
(2011). Automated measurement of childrenŐs facial expressions during problem
solving tasks. Proc. IEEE International Conference on Automatic Face and
Gesture Recognition, p. 30-35. Download
pdf
Butko,
N.J., Theocharous, G., Philipose, M., Movellan, J.R., (2011). Automated facial
affect analysis for one-on-one tutoring applications.Ó Proceedings of the 9th IEEE
Conference on Automatic Face and Gesture Recognition, p.382-387.
Download
Serpell Z, Whitehill Z, Movellan J, Foster A, Lin Y, Pearson
B, and Bartlett M (2011). Sensitivity to Non-verbal Behavior Influences
Cognitive Training of Minority Students. Presentation at American Psychological
Association conference, May 2011, Washington, DC.
Phan, L., Meza, R., Filizardo, J., Littlewort-Ford, G.,
Bartlett, M., Movellan, J., Reilly, J. (2009). Behavioral indices of cognitive
processing in children.Ó Oral presentation, Multimode Conference, Toulouse,
France. Abstract pdf
Whitehill, J., Bartlett, M., and Movellan, J. (2008). Automatic
facial expression recognition for intelligent tutoring systems. Workshop on
CVPR for Human Communicative Behavior Analysis, IEEE Conference on Computer
Vision and Pattern Recognition, pg. 1-6.
Download pdf
Whitehill,
J., Bartlett, M., and Movellan, J. (2008). Measuring the perceived difficulty
of a lecture using automatic facial expression recognition. Intelligent
tutoring systems. Montreal, Canada June 23-27, 2008.