We replicate a study by Ladd and Lenz (2009) measuring the effects of shifting newspaper endorsements on the 1997 British election. Utilizing difference-in-differences design, as well as regression and matching techniques, we find that the endorsements had a significant impact on the election, in line with the authors’ findings. However, we use data on party identification to conduct a robustness test that raises questions about the impact.

Introduction

We replicate an article by Jonathan Ladd and Gabriel Lenz examining the impact of newspaper endorsements on the 1997 UK general election.1 Prior to the election, several prominent newspapers, including the Sun, switched from their traditional support of the Conservative party and endorsed the Labour party. The authors exploit this shift, as well as the consistent messaging of other newspapers, to measure the treatment effect of reading a newspaper that switched endorsements. They find a significant effect: a difference-in-differences test reveals a treatment effect of 8.6 percentage point increase in the percent Labour vote, and a 12.7 percentage effect for habitual readers of newspapers that switched.

The authors supplement their findings with numerous alternative tests. They incorporate a variety of covariates into their model, which increase the treatment effect to 12.2% and 23.1% respectively. As several of the covariates significantly differ between the treatment and control groups, they also utilize exact and genetic matching techniques on selected variables, both of which also increase the treatment effect from the bivariate model. As a placebo test, the authors analyze vote intentions in 1996 prior to the treatment and find little difference between the treated and control groups. Finally, they examine individuals who stated they read one of the switching newspapers but then stop prior to 1996 and therefore don’t receive the treatment; as expected, they find no evidence of a treatment effect for these individuals. Overall, the article is an empirically sound example of using observational data to test for causal inference.

Replication

For the first step of our replication project, we analyzed the appendix to the article and followed their steps for coding the variables used in the analysis as dichotomous variables. This allowed us to re-create their basic difference-in-differences test, in which the percentage of Labour vote is compared between 1992 and 1997 between the treatment and control groups, utilizing the parallel trends assumption. We find a slightly smaller treatment effect of 7.2 (compared to 8.6%), which appears to be a function of finding smaller proportions of voters supporting the Labour party in both 1992 and 1997. While we are unsure why we find this discrepancy, one possible explanation is that we only include people in the treatment if they read the newspaper in 1996, while the authors also measure paper readership based on an earlier year if no 1996 interview was conducted. Another explanation is that their proportion of Labour votes is from a smaller pool than ours, for example only out of combined Labour and Conservative votes. Overall, however, our findings are largely consistent with theirs.

Next, we created a balance table among covariates for the treatment and control groups. The findings are very similar to the findings in the paper, as the covariates are almost identical for both groups.

We next replicated their probit regression with an extensive list of covariates included in the model, each coded dichotomously. Like the authors, we again found a positive, significant coefficient for the treatment of 24% on the outcome of Labour vote in 1997, which is much larger than in the bivariate model. The largest positive coefficient was Labour vote in 1992, while the largest negative coefficients were Conservative vote in 1992 and being white.

We also replicate their test using exact matching of covariates, using the following variables: Labour Vote, Conservative Vote, Liberal Vote, Labour Identification, Conservative Identification, Liberal Identification, Labour Support, Conservative Support, and Political Knowledge. After matching on these covariates, we find an average treatment effect of 7.59%, which is similar to the treatment effect of 7.2% that we found using the difference in difference. However, this differed slightly from their finding of a treatment effect of 10.9 after matching; we believe this difference is due to our inability to accurately replicate the indexing of all of the variables, as they weren’t all fully specified.

Additional Robustness Test using Party Identification

Finally, we noticed that there was data available on party-ID for each year between 1992 and 1997, and used this information to test the parallel trends assumption that the treatment and control groups would have experienced the same rate of change in voting for the Labour party if the treatment never occurred. Incorporating information on Labour party-ID for the intervening years challenges this assumption. As can be seen in the graph, the percent of respondents who identify with the Labour party increased between 1992 and 1995 in both the treatment and control groups at levels higher than would be predicted by the parallel trends assumptions. Notably, after 1995 party-identification levels for both groups then stagnate, and there is almost no perceptible impact of the 1996 treatment on party-identification levels. These trends would seem to challenge their findings that the treatment had a major impact on increasing vote choice for the Labour party, and instead paint a picture in which Labour party-identification was already increasing, particularly amongst the treatment group between 1994-1995. This raises the possibility that other variables caused the increased differential in Labour vote among the treatment and control groups.

The authors do test the variable of intended vote choice in 1996 as a placebo test and find no significant differences between the treatment and control groups, which supports their findings because it was before the treatment was administered. However, they don’t present the exact results of this test, and we were unable to find the variable of intended vote choice in the dataset to replicate. Furthermore, while it could be argued that party-ID doesn’t necessarily entail voting for that party, the graph demonstrates that in both 1992 and 1997 the percent of party ID closely aligned with vote choice for both groups, indicating that it is a reasonable predictor. Therefore, judging by the graph, we would expect that a higher percentage of the treatment group would have voted for the Labour party in 1995 than in 1997, which runs counter to their theory. Overall, we believe this finding raises questions about the magnitude of the treatment effect of the newspaper endorsements.

  1. Ladd, J. M., & Lenz, G. S. (2009). a Rare Communication Shift to Document Exploiting the Persuasive Power of the News Media of Technology. American Journal of Political Science, 53(2), 394-410.↩