Tulips 1.0 is a small Audiovisual database of 12 subjects saying the
first 4 digits in English. Subjects are undergraduate students from
the Cognitive Science Program at UCSD. The database was compiled at
Javier R. Movellan's laboratory at the Department of Cognitive
Science, UCSD. If you use these data for your research please refer to
it as the "TULIPS1 database, (Movellan, 1995)".

Movellan J. R. (1995) Visual Speech Recognition with Stochastic
Networks. in G. Tesauro, D. Toruetzky, & T. Leen (eds.) Advances in
Neural Information Processing Systems, Vol 7, MIT Pess, Cambridge.

To get a .ps copy of the paper you may go to movellan's WWW home page

http://cogsci.ucsd.edu/~movellan/

The "RawData" directory contains two directories: Tulips1.A and
Tulips1.V The "PreprocessedData" directory contains representations
used by different researchers on this database.

==================== RawData directory ========================
Tulips1.A/raw contains the audio files in .au format. The file names are
coded as follows: Anthony12e.au means that the subject was Anthony,
saying the digit "1" for the second time. Ben 41e.au meand that the
subject was Ben saying the digit "4" for the first time.  The format
of the .au files is as follows: The first 28 bytes in each file are
reserved for header information, using the standard .au format.
Signal information starts on byte number 29 (byte 28 using
zero-offset). Each byte encodes acoustic energy on each sample (1 byte
per sample). The sampling rate is 11127 Hz.


The header starts out with the character sequence ".snd"
After this character sequence there are 24 bytes of header information

Tulips1.A/cepstrals contains the cepstral processed audio files. The
processed audio files are in PGM format (so they can actually be
visualized) with 26 pixels arranged in the following order

12 cepstral coefficients 
1 log-power
12 cepstral derivatives
1 log-power derivatives



========
NOTE FOR SUN OWNERS: To play the files you may need to convert them to
2 byte per sample format. To do so use the following command (do not leave blanks between the commas)

audioconvert -i offset=28,encoding=linear8,rate=11.1k,channels=mono -F
-f linear16,rate=11.1k,mono -o outfile infile

To play use 

audioplay outfile
=========



Tulips1.V contains the video files in 100x75 pixel 8bit gray level,
.pgm format.  The file names are as follows:

Anthony12.00004 means this is the forth frame of subject Anthony
saying the digit "1" for the second time. Each frame corresponds to
1/30 of a second.


======= Results: ======= 

1) Human subjects untrained on lip reading: 89.93%correct. 
Sample size: 6 subjects.

Confusion matrix:


                   Digit Presented
                   1     2    3    4
Subject 
Response 1        125    3   17    9
         2        1     141  4     1
         3        15     0   121   3   
         4        3      0    2   131

2) Trained lip readers: 95.49% correct
Sample size: 3 subjects.

Confusion matrix:


                   Digit Presented
                   1     2    3    4
Subject 
Response 1        68     0    3    0
         2        0     72    3    0
         3        3      0    64   1
         4        1      0    2   71


3) Continuous density Hidden Markov Model (See Movellan, 1995)
Generalization score (obtained using jacknife method): 89.93% correct



Confusion matrix:


                   Digit Presented
                   1     2    3    4
Subject 
Response 1        24     1    3    2
         2        0      21   0    1
         3        0      1   20    0
         4        0      1    1   21


4) The best results up to date were obtained by Mike Gray (2-5-98) at
91.6667% correct , with the following confusion matrix


24 0 1 3
0 23 0 1
0 0 21 0
0 1 2 20

88 correct out of 96: 91.666667%


==========
Please let me know at movellan@cogsci.ucsd.edu of any published results
based on these data.


Details about the composition of Tulips1.V
==========================================
Movie     Number of frames
=====     ================
Anthony11 6
Anthony12 8
Ben11 11
Ben12 10
Candace11 6
Candace12 6
Cynthea11 13
Cynthea12 12
Don11 12
Don12 10
George11 8
George12 9
Isaac11 8
Isaac12 7
Jay11 10
Jay12 11
Jesse11 10
Jesse12 7
Oliver11 7
Oliver12 8
Regina11 11
Regina12 7
Simon11 9
Simon12 7


Anthony21 10
Anthony22 8
Ben21 10
Ben22 9
Candace21 7
Candace22 7
Cynthea21 10
Cynthea22 10
Don21 14
Don22 15
George21 10
George22 7
Isaac21 11
Isaac22 10
Jay21 12
Jay22 11
Jesse21 9
Jesse22 8
Oliver21 9
Oliver22 11
Regina21 9
Regina22 10
Simon21 10
Simon22 6

Anthony31 11
Anthony32 10
Ben31 8
Ben32 16
Candace31 9
Candace32 9
Cynthea31 10
Cynthea32 8
Don31 12
Don32 12
George31 12
George32 12
Isaac31 7
Isaac32 6
Jay31 9
Jay32 11
Jesse31 7
Jesse32 9
Oliver31 7
Oliver32 8
Regina31 12
Regina32 12
Simon31 8
Simon32 8

Anthony41 9
Anthony42 8
Ben41 13
Ben42 12
Candace41 9
Candace42 9
Cynthea41 12
Cynthea42 10
Don41 11
Don42 13
George41 13
George42 13
Isaac41 11
Isaac42 6
Jay41 13
Jay42 12
Jesse41 6
Jesse42 11
Oliver41 9
Oliver42 13
Regina41 11
Regina42 13
Simon41 9
Simon42 9



