Polling bias

Individual polls are biased. That’s not to say that they consciously try and favour one party or another. Instead, the way that they sample and weigh their samples results in bias, with that bias being measured by how far the particular polling company is from the average of all polls.

Occasionally Erudite’s Poll of Polls tries to account for the bias of individual polling companies by adjusting each poll by how far they deviate from the industry average for each party. But does this go far enough? After all, if the polling industry average is too high or low for each party, any attempt at correcting bias won’t fix the industry-wide problem.

The political left tend to give great credence to Danyl McLauchlan’s bias corrected Poll of Polls. Mr McLauchlan calculates the polling bias by looking at how the polling industry did in picking each party’s 2011 election result and then calculating the discrepancy between their average final polling and that party’s actual result. As it turns out, the pollsters got Labour and the Greens roughly correct – Mr McLauchlan adjusts Labour down by 0.6% and the Greens down by 0.5%. It’s National and NZ First that are badly out of whack, with the final polling average being out by a whopping 3.1% for NZ First (too low) and 4.1% for National (too high). McLauchlan therefore adjusts his non-bias corrected Poll of Polls by those amounts, to get his bias corrected version.

To me, McLauchlan’s approach is fatally flawed. One needs to ask the question, ‘If the pollsters got Labour and the Greens almost right, why did they get National and NZ First so badly wrong?’ So what did happen in the final build up to election day. The obvious incident that springs to mind is the teapot tapes saga, with National giving Winston Peters vital oxygen in the dying last weeks of the campaign. The teapot tapes conversation occurred on 11 November 2011, with the election occurring 15 days later on 26 November 2011. Rumours slowly circulated about just what John Key and Don Brash had discussed, with Winston Peters tantalising media with leaked tidbits. NZ First went from being nowhere in the polls to breaking 6% on election day (6.59% to be precise), while National’s vote plummeted from the ‘governing alone’ predictions that had been swirling for months to its final result of 47.31%.

Is it more likely that the polls were inherently biased against NZ First by 3.1% and in favour of National by 4.1%, or is it a case of the polls not reacting to the sudden speed of the anti-National/pro-NZ First reaction that the teapot tape saga produced? Is the 2011 election an aberration or are the polls consistently over-representing National to that degree and under-representing NZ First?

Gavin White of UMR wrote a fascinating blog last month about the accuracy of companies’ final polling prior to MMP elections compared to actual election results. He concludes:

Counting all mainstream media polls since 2005 (i.e. excluding UMR but including TV3 and Fairfax / Research International polls in 2008 and 2011) leaves 14 polls, and an average error of:

    • National: 2.4% too high
    • Labour: 0.5% too low
    • Greens: 1.5% too high
    • NZ First: 1.1% too low.

That’s a significant difference compared to McLaughlan’s bias corrections. But it does show that the polling average for the left-right bloc gap is likely to be skewed in favour of the right by 1.4% (given that I place NZ First in the centre, separate and distinct from the centre left and centre right).

(It is, however, worth noticing that the Fairfax poll for 2011 was probably the worst of the bunch from the major polls, over-estimating National by 6.7%, under-estimating Labour by 1.5%, over-estimating the Greens by 0.9% and under-estimating NZ First by 2.6%. Fairfax now has a different pollster. I therefore wonder whether there’s any significant change if Fairfax’s historic pre-election poll results are removed from the mix. If I have time, I may try and replicate Gavin White’s analysis, with the removal of Fairfax.)

And Gavin White’s conclusions?

I think history suggests that:

    • If the total for Labour + Greens is within about 2% of the total for National and its allies (whichever of ACT, United Future and the Conservatives makes it into parliament), then it’s actually pretty much a deadheat. 
    • If NZ First gets 4% in most of the mainstream polls, then they’ll probably pass the 5% threshold on election day.

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s