Red Hen Lab: Dataset and Tools for Multimodal Human Communication Research
Jungseock Joo, Francis F. Steen, and Mark Turner.
Künstliche Intelligenz (2017). Supplementary material (under construction).
Click here for the video in which the computer repair tutor uses multimodal deictic gestures and words, with an analysis.
Click here for a video clip from The Big Short (2015) in which actress Margot Robbie uses a dismissive gesture to the left. This film concerns the collapse of the USA housing market. It presents occasional vignettes about complex financial instruments, including packages of mostly subprime mortgages. The characters in the plot of the film talk about and are involved in such transactions. Then Margot Robbie, an actress who has no character role in the movie, is presented by voice-over as "Margot Robbie." We are told that she will explain such packages. She is in a bubble bath, sipping champagne served to her by a butler. She looks at the camera, switching the direction of her gaze when the camera selection shifts. When she is finished with her elaborate explanation of financial juggling across wildly complicated investment instruments, which is meant of course to be hilarious from a naked beauty in a bubble bath, only because we think the naked bubble bath with champagne is exactly where everyone relaxes from game-theoretic aggression in the trading of financial instruments, she says, "Got it? Good." She sips the champagne, looks into the camera, and says, "Now, fuck off," with great prosody, flashing eyes, a movement of the head back to the left, and a raised index finger on the left hand which sweeps away farther left, up, and behind her. In classic joint attention, two or a few speakers are togther and communicating about something they can actually perceive. If someone says in that scene, "Now, fuck off," with the appropriate co-speech gesture, crucially, it does not mean at all that the hearer will suddenly have no perceptual access to the speaker. But this scene with Margot Robbie is a scene of blended classic joint attention, in which speaker and hearer are not together, in which there are additional technical and cinematic operations, and in which the claimed topic of attention (the financial instruments) is not actually perceptible. To understand this film clip, we are using a conceptual network for blended classic joint attention, and for that network, her multimodal communication means exactly that she will now disappear, without actually moving! We expect the immediate edit, and the disappearance of the image of Margot Robbie, the champagne, and the bubble bath.
A note on Argument Structure Constructions: Knowing a language means knowing patterns of relationships. The verb "throw" leads us to expect input for the agent, for the object thrown, and probably for the direction of the throwing. The English pattern Noun-Phrase Verb-Phrase Noun-Phrase Prepositional-Phrase is used for just these kinds of relationships, as in "He jumped the horse over the hurdle." This is called the "Caused-Motion Construction." Such a pattern, in which a verb, for example, is related systematically to other spots in the utterance is called an “Argument Structure Construction (ASC).” Linguists investigate which parts of the semantic interpretation come from the actual words (e.g. "jumped") and which parts come from the grammatical pattern. For example, "he" and not the horse is the subject of "jumped," even though we know that it is the horse whose body actually leaves the ground and returns to it. Our semantic interpretation is that he did something that caused the horse to move by jumping in a direction. Note, of course, that we do not know what he did: he might have been standing and signaled to the horse with his hand, or have been riding the horse and pulled up on the reins, or . . .
A note on epistemic stance and evidentials: Linguists have already analyzed the ways in which epistemic distance can be indicated grammatically and lexically. "Evidentials" are grammatical elements that indicate the source of the content. Many languages have evidentials that are morphological markers on the individual word. English does not have morphological evidentials of this sort, instead expressing them optionally through adverbial lexical items, such as "allegedly," "reportedly," "officially." However, human communication is not dependent on such grammatical and lexical means to express epistemic stance.
NewsScape summary statistics at 2017-09-10 6:58 GMT
Previous values from checkpoint at 2016-02-20 3:30 GMT Total networks: 51 Total series: 2,893 Total duration in hours: 342,310 (275,647) Total metadata files (CC, OCR, TPT): 898,874 (732,732) Total words in metadata files (CC, OCR, TPT): 4.16 billion, 4,162,316,851 exactly (3.40) Total caption files: 441,400 (355,338) Total words in caption files: 2.76 billion, 2,764,903,135 exactly (2.25) Total OCR files: 420,595 (344,734) Total TPT files: 36,879 (32,660) Total words in OCR files: 960.25 million, 960,253,097 exactly (758.65) Total words in TPT files: 437.16 million, 437,160,619 exactly (386.82) Total video files: 441,178 (355,155) Total thumbnail images: 123,231,458 (99,233,008) Storage used for core data: 120.26 terabytes (100.39)
CNN's Brooke Baldwin, Newsroom, 13 October 2014
11:00:20 >> WOLF, THANK YOU SO MUCH. 11:00:23 GREAT TO SEE ALL OF YOU ON THIS MONDAY AFTERNOON. 11:00:25 LESS THAN A WEEK AFTER LIBERIAN EBOLA PATIENT THOMAS ERIC DUNCAN 11:00:31 DIED INSIDE THAT HOSPITAL, ONE OF THE NURSES THAT TREATED HIM 11:00:34 IS INFECTED WITH THIS DEADLY VIRUS. 11:00:36 THIS NURSE WORKS AT TEXAS HEALTH PRESBYTERIAN. 11:00:39 HER CASE IS THE FIRST KNOWN TRANSMISSION IN THE U.S. 11:00:43 HOSPITAL OFFICIALS SAY SHE HAD A NUMBER OF CONTACTS WITH DUNCAN 11:00:46 AND THAT SHE FOLLOWED ALL OF THE PROPER PROTOCOLS. 11:00:49 THE CDC, HOWEVER, IS NOT SO CERTAIN. 11:00:52 IT BELIEVES THERE WAS SOME KIND OF BREACH 11:00:55 ALTHOUGH IT CANNOT PINPOINT THAT. 11:00:58 THAT WAS THE PHRASEOLOGY FROM THAT NEWS CONFERENCE YESTERDAY. 11:01:01 A SOURCE WITH DIRECT KNOWLEDGE OF THE CASE TELLS CNN THAT CDC 11:01:04 DETECTIVES TALKED TO THE NURSE SEVERAL TIMES AND FOUND 11:01:08 INCONSISTENCIES ABOUT THE KINDS OF PROTECTIVE GEAR SHE WORE, 11:01:11 HOW SHE PUT THIS GEAR ON, HOW SHE REMOVED IT. 11:01:14 BUT LET ME SHARE THIS WITH YOU. 11:01:16 JUST THIS MORNING CDC DIRECTOR TOM FRIEDEN CLARIFIED HIS 11:01:20 COMMENTS ABOUT THIS PROTOCOL BREACH THAT A NUMBER OF PEOPLE 11:01:22 WERE SAYING WAS ABSOLUTELY BLAMING THE NURSES WHO ARE 11:01:26 TREATING THESE EBOLA PATIENTS. 11:01:30 HERE HE WAS. 11:01:32 >> WHAT WE NEED TO DO