Lying with statistics – feat. ECR and Daniel Hannan

Yesterday the ECR Group announced that they would not nominate  a candidate for EC president because participating in the process is considered to be too “federal”. At the same time they want to take part in one of “leader debates” in the run-up of the elections…

Anyway, in order to back up their point of view they presented the results of a ComRes opinion poll. At the press conference Daniel Hannan said: “There is no evidence of popular demand for having more pan-European elected positions”. Unfortunately, ECR’s own opinion poll tells a different story that contradicts Mr Hannan’s assessment:  39% of the respondents agree with the idea that the European Parliament is choosing the next EC president “as this will make the winning candidate more legitimate” –  27% are against it.  (p. 8)

But the opinion poll is also a text book case study in how to lie with statistics. Unfortunately some journalists and tweeps (me included!) fell for it and wrote stuff like “65 % of Europeans who never heard of…” or similar snippets. Here is the methodological note of the survey:

ComRes interviewed 1 , 200 adults from the UK, France, Germany, Italy, Spain and Poland (200 per country) aged 18+ online between 5 th and 14 th February 2014. Data were weighted to be representative of all adults aged 18+.

So what are the problems? First of all, it is a very weak sample and it may not be entirely representative. There are also methodological problems when it comes to online surveys (who fills out online surveys?). But: 200 is a very small sample for each of the countries (other surveys would base their results on 1500 respondents for Germany alone!)  And why did they choose 200 respondents for each country despite huge differences in population size? It is also a bit unclear how the weighting  has been carried out (ex-ante or ex-post? only regarding age?)

The survey also fails to cover “Europe” and is – by all means – not representative for Europe or the EU.  So we should not talk about “Europeans” when citing the survey. The aim of the study was “to understand public’s attitude across Europe to the European Union and the upcoming European elections”. However, the study only covers 6 EU member states: So whatever results you find they only apply to those 6 countries and do not have any statistical validity for the rest of Europe.

ComRes must be aware of these shortcomings as they consistently refer to “respondents”  instead of talking about a broader category (for example “voters”, “Europeans” or “Germans”). It is a typical survey made for journalists with short attention spans. The sample design basically triggers certain (seemingly) logical associations such as “that’s what Europe thinks”, “In Europe, this is a problem…”, “A new opinion poll about what Europeans think…” – the problem is simple: All those statements are wrong, the ECR survey cannot be used to back up such claims.

1 Comment

  1. Weighting is ok if you want to discuss the population, however it remains unclear how it was done exactly. Also the second sentence in the “Methodology” section is misleading because in the “Summary” section they talk about respondents not about the population (in a statistical sense).

    If they would want to publish it in a journal, they would get a reject from the reviewers ;-)

Leave a Reply

© 2014 Kosmopolito

Theme by Anders NorenUp ↑