Computational Analysis of Nonverbal Behavior in Adaptive Tutoring
Whitehill, Littlewort, Reilly, Phan, Movellan, Bartlett
There has been growing recognition of the importance of adaptive tutoring systems that respond to the studentŐs emotional and cognitive state. However little is known about childrenŐs facial expressions during a problem solving task. What are the actual signals of boredom, interest, confusion, or uncertainty in real, spontaneous behavior of students? The field also is in need of spontaneous datasets to drive automated recognition of these states. To date, there is a paucity of empirical data to support understanding nonverbal behavior in teaching at a computational level. The goal of this project is to develop computational methods to measure and model the nonverbal behavior and interactive strategies observed during face-to-face teaching. To support these models, we are collecting datasets of student-teacher interactions as well as nonverbal behavior during problem solving. The computational models will serve as a foundation for a new generation of embodied teaching agents that approximate the benefits of face-to-face human tutoring. This research builds the foundation for automated tutoring systems that sense the state of the student and adapt accordingly. The project will help advance the science of learning and teaching by improving our understanding of the dynamics of nonverbal behavior in teaching at a computational level, across multiple time scales: From low-level micro-expressions in the timescale of tens of milliseconds, to cognitive and affective processes with time scales of seconds, to higher level strategic behaviors operating at longer time scales.
NSF / IIS / HCC Grant: Computational Analysis of Nonverbal Behavior in Adaptive Tutoring. PI: Bartlett. Co-I: Reilly, Movellan. 8/1/2009-7/31/2013.
NSF Science of Learning: Temporal Dynamics of Learning Center, NSF SBE 0542013
NIPS Workshop: Personalizing Education with Machine Learning. Organizers: Mike Mozer, Javier Movellan, Rob Lindsey, Jake Whitehill. Neural Information Processing Systsems, Lake Tahoe CA, Dec 8, 2012.
TDLC Workshop: Optimal Teaching. Organizers: Javier Movellan and Jacob Whitehill. University of California, San Diego, May 4, 2012.
Papers and Presenatations
Sathyanarayana S, Satzoda, Carini, Lee, Salamanca, Reilly, Forster, Bartlett, Littlewort (2014) Towards Automated Understanding of Student-Tutor Interactions using Visual Deictic Gestures. Proc. Computer Vision and Pattern Recognition Workshops (CVPRW), p. 480-487.
Sathyanarayana S, Littlewort G, Bartlett M (2013). Hand Gestures for Intelligent Tutoring Systems: Dataset, Techniques & Evaluation. Proc. IEEE International conference on Computer Vision, Workshop: Decoding Subtle Cues from Social Interactions. pg 769 – 776. Paper on IEEE Xplore
Dykstra K, Whitehill J, Salamanca L, Lee M, Carini A, Reilly J, and Bartlett, M (2012). Modeling One-on-one Tutoring Sessions. Proc. International Conference on Development and Learning / Epigenetic Robotics, San Diego, CA. Download pdf
Salamanca L, Carini A, Lee M, Dykstra K, Whitehill J, Angus D, Wiles J, Reilly J, and Bartlett, M (2012). Characterizing the Temporal Dynamics of Student-Teacher Discourse. Proc. International Conference on Development and Learning / Epigenetic Robotics, San Diego, CA. Download pdf
Brian S, Salamanca L, Whitehill J, Reilly J, Bartlett M, Angus D, and Wiles J (2012). Using Recurrence Plots to Visualize the Temporal Dynamics of Tutor/Student Interactions. Proc. International Conference on Development and Learning / Epigenetic Robotics, San Diego, CA. Download pdf
Salamanca L, Lee M, Carini A, Dykstra K, Whitehill J, Bartlett M, Reilly, J. (2012) Profiling Student-Teacher Eye Gaze Behaviors. Poster presentation, Southern California Cognitive Neuroscience Meeting at San Diego State, March 2, 2012.
Whitehill, J., Serpell, Z., Foster, A., Lin, Y.C., Pearson, B., Barlett, M., et al. (2011). Toward an Optimal Affect-Sensitive Instructional System of Cognitive Skills. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop on Human Communicative Behavior, (pp. 20 - 25). Download pdf
Littlewort GC, Salamanca LP, Reilly JS, and Bartlett MS (2011). Automated measurement of childrenŐs facial expressions during problem solving tasks. Proc. IEEE International Conference on Automatic Face and Gesture Recognition, p. 30-35. Download pdf
Butko, N.J., Theocharous, G., Philipose, M., Movellan, J.R., (2011). Automated facial affect analysis for one-on-one tutoring applications.Ó Proceedings of the 9th IEEE Conference on Automatic Face and Gesture Recognition, p.382-387. Download
Serpell Z, Whitehill Z, Movellan J, Foster A, Lin Y, Pearson B, and Bartlett M (2011). Sensitivity to Non-verbal Behavior Influences Cognitive Training of Minority Students. Presentation at American Psychological Association conference, May 2011, Washington, DC.
Phan, L., Meza, R., Filizardo, J., Littlewort-Ford, G., Bartlett, M., Movellan, J., Reilly, J. (2009). Behavioral indices of cognitive processing in children.Ó Oral presentation, Multimode Conference, Toulouse, France. Abstract pdf
Whitehill, J., Bartlett, M., and Movellan, J. (2008). Automatic facial expression recognition for intelligent tutoring systems. Workshop on CVPR for Human Communicative Behavior Analysis, IEEE Conference on Computer Vision and Pattern Recognition, pg. 1-6. Download pdf
Whitehill, J., Bartlett, M., and Movellan, J. (2008). Measuring the perceived difficulty of a lecture using automatic facial expression recognition. Intelligent tutoring systems. Montreal, Canada June 23-27, 2008.