Experimentally controlling for anticipatory baseline effects in visual-world studies

Visual world studies investigate hypotheses of the form "Does constraint X influence the processing of linguistic fragment Y?" One necessary feature of such studies is that information corresponding to constraint X must be presented temporally prior to Y. Consequently, there may be effects of X on a participant's looking behavior independently of whether X imposes any constraint on the processing of Y. These anticipatory effects, when not appropriately controlled, can cloud the interpretation of results.

In this walkthrough, we will re-analyze data from Experiment 1 of Barr (2008).

[Postscript Dec. 2010: if I did this analysis again today, I would use random slopes for subjects and items and use permutation tests to get p-values. Your results may differ, dependning on which version of lme4 you are using (which is another argument for using permutation tests). I hope to be able to post something about this new technique soon.]

Participants saw displays containing four pictures (see above figure) and heard a speaker refer to one of the four pictures, the "target" picture (e.g., "click on the bucket"). There was also a "critical" picture that in one condition was a competitor—in other words, the first syllable or so of the name of the depicted object overlapped phonologically with the name of the target. See the right panel in the above figure, where the critical object is highlighed in red (of course, this highlighting was not seen by the subject). It was expected that the presence of a competitor would temporarily interfere with the identification of the target. In a second condition, the critical object was a noncompetitor (e.g., stepladder; left panel). In addition to this Competitor manipulation, I also manipulated whether the listener believed that the speaker knew the identity of this critical picture (independent variable of Ground). In the Privileged condition, the listener thought that the speaker did not know about this picture; in the Shared condition, the listener believed that the speaker did know about it. Obviously, speakers can't refer to things that they don't know about. My question was whether listeners would be able to use this knowledge to constrain the mapping of words to referents, or whether this mapping process was largely automatic. If they can, then a competitor that is privileged should produce less competition than one in that is shared. If they can't, then the competition should be roughly equivalent.

Now, telling listeners that a speaker doesn't know about some picture can make them look at it less than other pictures that the speaker is assumed to know about. So listeners may be less likely overall to look at the critical object when it's privileged, relative to when it's shared, regardless of whether or not it is a competitor. For example, they may attempt to restrict their focus of attention to things that are mutually known in anticipation of a forthcoming referring expression. But that by itself wouldn't answer the question of whether this knowledge is used in the referential process. It is possible that the mapping process proceeds relatively automatically and does not use this knowledge to constrain the binding of words to referents. I know that it may sound odd to say that at one level, listeners are taking into account the speaker's perspective, while at another level, they are not. But this kind of dissociation is possible in cognition. Fodor (1983) gives the example of visual illusions to demonstrate this point: we may know that a visual illusion is just an illusion but can't avoid falling for it anyway. I reasoned that something like this kind of automatic processing could underlie the so-called "egocentrism" that Keysar, myself and colleagues have seen in a number of studies (e.g., Keysar, Lin, & Barr, 2003).

The design of this study (which was based on Hanna, Tanenhaus, and Trueswell, 2003) is that it has a built-in control for anticipatory effects of Ground. Any anticipatory advantage for the critical object when it is shared compared to when it is privileged will be present independently of whether the critical object is a competitor or noncompetitor. The logic is to analyze the 'competition' effect, the likelihood of fixating the critical object when it is a competitor relative to when it is a noncompetitor. The competition effect for the privileged condition is given by comparing the privileged competitor to the privileged noncompetitor; the effect for the Shared condition is given by comparing the shared competitor to the shared noncompetitor. These competition effects will be 'free and clear' of any anticipatory effects. Thus, the main prediction was: If listeners can use common ground to constrain lexical processing, the competition effect in the privileged condition should be less than that in the shared condition.

Twenty listeners were included in the study, each of which completed sixteen experimental items (and an equivalent number of fillers). I defined an analysis window spanning from 180 ms after the onset of the target word (e.g., "bucket") until 180 ms after the end of the longest word in the study (which was 736 ms); in other words, a 736 ms window spanning from 180 ms to 916 ms after word onset. For each trial, I calculated the number of frames that the listener looked at the critical object, as well as the total number of frames in the window (which was always 184 frames, given that the sampling rate of the eyetracker was 250 Hz).

Here are what the curves look like for the competitor object in each of the four conditions:

As expected, listeners looked more at shared than at privileged objects at the onset of the analysis window (0 ms in the plot, which is 180 ms after word onset). But they did so equally whether the critical object was a competitor (buckle) or noncompetitor (stepladder).

In the MLR paper, I discussed how one can control for anticipatory effects by modeling the curves. In the current case, we are going to collapse over the time variable, since the experimental design allows us to control for anticipatory effects.

The data files can be downloaded either as a zip file archive or as a .tar.gz archive.

Let's load in the lme4 package and the data (file interf.RData), and then take a look at the data for the first two subjects:

library(lme4)
load(file="interf.RData")
subset(interf, SubjID==1 | SubjID==2)

   SubjID ItemID CompetitorT GroundT Competitor Ground CbyG   Y   N
1       1      1          CP       S        0.5    0.5  0.5 119 184
2       1      2          CP       P        0.5   -0.5 -0.5   0 184
3       1      3          NC       S       -0.5    0.5 -0.5   0 184
4       1      4          NC       P       -0.5   -0.5  0.5   0 184
5       1      5          CP       S        0.5    0.5  0.5 123 184
6       1      6          CP       P        0.5   -0.5 -0.5   0 184
7       1      7          NC       S       -0.5    0.5 -0.5   0 184
8       1      8          NC       P       -0.5   -0.5  0.5   0 184
9       1      9          CP       S        0.5    0.5  0.5 157 184
10      1     10          CP       P        0.5   -0.5 -0.5   0 184
11      1     11          NC       S       -0.5    0.5 -0.5   0 184
12      1     12          NC       P       -0.5   -0.5  0.5   0 184
13      1     13          CP       S        0.5    0.5  0.5  43 184
14      1     14          CP       P        0.5   -0.5 -0.5  55 184
15      1     15          NC       S       -0.5    0.5 -0.5  17 184
16      1     16          NC       P       -0.5   -0.5  0.5   0 184
17      2      1          CP       P        0.5   -0.5 -0.5 184 184
18      2      2          NC       S       -0.5    0.5 -0.5   0 184
19      2      3          NC       P       -0.5   -0.5  0.5   0 184
20      2      4          CP       S        0.5    0.5  0.5 128 184
21      2      5          CP       P        0.5   -0.5 -0.5   0 184
22      2      6          NC       S       -0.5    0.5 -0.5  63 184
23      2      7          NC       P       -0.5   -0.5  0.5   0 184
24      2      8          CP       S        0.5    0.5  0.5   0 184
25      2      9          CP       P        0.5   -0.5 -0.5  56 184
26      2     10          NC       S       -0.5    0.5 -0.5   0 184
27      2     11          NC       P       -0.5   -0.5  0.5 133 184
28      2     12          CP       S        0.5    0.5  0.5 104 184
29      2     13          CP       P        0.5   -0.5 -0.5   0 184
30      2     14          NC       S       -0.5    0.5 -0.5   0 184
31      2     15          NC       P       -0.5   -0.5  0.5   0 184
32      2     16          CP       S        0.5    0.5  0.5   0 184

The variable CompetitorT corresponds to whether the critical object was a competitor (CP) or noncompetitor (NC). The variable Competitor is the effect coded variable for the regression. GroundT corresponds to whether the critical object was privileged (P) or shared (S), and Ground is the corresponding effect coded variable for the regression. CbyG is the effect code for the interaction (sign of Competitor * sign of Ground * .5). Y is the number of frames in the window for which the point of gaze was on the critical object, while N is the total number of frames in the window.

Although for each trial we have made 184 observations, it is certainly not the case that these individual observations are independent (the eye position was sampled once every 4 ms). So we can convert the observations for a given trial into a single independent number by aggregating, either by computing a proportion (Y / N) or via the empirical logit function \( y = log\left(\frac{y+.5}{n-y+.5}\right) \) (McCullagh & Nelder, 1989). I will first show the results from a 'conventional' analysis on the proportional data and then do the same analysis using weighted empirical logit regression (quasi-MLR). These analyses give similar answers, but suggest different numerical trends for the interaction term.

Now we are ready to run the regression. Since we have only aggregated up to the level of the individual trial, we are able to fit a model with crossed effects of Subject and Item, thereby simultaneously accounting for random variance due to both factors (Baayen, Davidson, and Bates, 2008).

We first do an analysis on the proportional scale.

[Postscript Dec. 2010: if I did this analysis again today, I would use random slopes for subjects and items and use permutation tests to get p-values. Your results may differ, dependning on which version of lme4 you are using (which is another argument for using permutation tests).]

interf$p <- interf$Y / interf$N
interf.lmerP <- lmer(p ~ Competitor*Ground +
                     (1 | SubjID) +
                     (1 | ItemID),
                     data=interf)
summary(interf.lmerP)

We obtained t values (which we are treating as z) of 3.76 for the Ground (p < .01) and .87 for the interaction (p = .38).

Now let's do the quasi-MLR analysis using weighted empirical logit regression. We first need to compute the empirical logit as well as the weights. The we run basically the same linear mixed-effects regression as we did for the proportional case.

interf$elog <- log((interf$Y+.5)/(interf$N-interf$Y+.5))
interf$ewt <- 1/(interf$Y+.5)+1/(interf$N-interf$Y+.5)
interf.lmer <- lmer(elog ~
                    Competitor*Ground +
                    (1 | SubjID) + (1 | ItemID),
                    data=interf, weights=1/ewt)
summary(interf.lmer)

Barr, D. J. (2008). Pragmatic expectations and linguistic evidence: Listeners anticipate but do not integrate common ground. Cognition, 109, 18–40.

Fodor, J. A. (1983). The modularity of mind: An essay on faculty psychology. Cambridge, MA: MIT Press.

Hanna, J., Tanenhaus, M. K., & Trueswell, J. C. (2003). The effects of common ground and perspective on domains of referential interpretation. Journal of Memory and Language, 49, 43-61.

Keysar, B., Lin, S., & Barr, D. J. (2003). Limits on theory of mind use in adults. Cognition, 89, 25-41.

McCullagh, P., & Nelder, J. A. (1989). Generalized linear models. London: Chapman and Hall.