Part 1 of this post was in response to a post on the Daily Blog by Frank Macskasy entitled “TV3 News on Polling Cellphone Users – Only A Year Late“. Part 1 dealt largely with the methodology issues that polling companies might have with calling cellphones, and the reality that the political preferences of non-landline households would have to be dramatically different from the rest of the population for there to be any substantial difference in polling results.
For this post, I’d like to pick up on a point previously made by Mr Macskasy, which was reiterated in his recent post:
“As the only pollster to call respondants’ [sic] cellphones, Roy Morgan is the most credible polling company and the one to watch.”
So, how does Roy Morgan fare in terms of its accuracy? If the other polling companies are producing skewed results (ie. pro-National) due to their failure to call cellphones, the Roy Morgan poll should be consistently more accurate than its competitors.
Being a polling company that runs a political poll must be tough. For nationwide political polls, there’s only one check of accuracy that matters – how did they poll just before election day?* Let’s take a look at how Roy Morgan has fared in recent elections, in terms of their success in picking the election day results of the four main political parties:
- National: 49.5% (compared to election result of 47.3%), which is closer than any of the other four major public pollsters.
- Labour: 23.5% (compared to election result of 27.5%), the worst of the pollsters by some margin. The next worst was 3News, which had Labour on 26%.
- Greens: 14.5% (compared to election result of 11.1%), the worst of the pollsters. The next worst was again 3News, with the Greens on 13.4%.
- NZ First: 6.5% (compared to election result of 6.6%. Basically spot on, and certainly the best of the pollsters.
That’s quite a mixed bag. Picking NZ First to within 0.1% is offset by badly underestimating Labour and overestimating the Greens. And though Roy Morgan came closest of the pollsters to National’s election day result, that’s still a 2.2% discrepancy.
- National: 42% (compared to election result of 44.9%). The only one of the major public pollsters to underestimate National’s result, with the next lowest poll result being 3News which had National on 46%.
- Labour: 34.5% (compared to election result of 34.0%), the best of the pollsters.
- Greens: 10% (compared to election result of 6.7%), the worst of the pollsters, although both One News and 3News had the Greens on 9%.
- NZ First: 4.5% (compared to election result of 4.1%. Close, but not quite as close as the Herald-Digipoll, which had NZ First on 3.9%.
Again, a mixed bag, having got to within 0.5% of correctly picking Labour and NZ First, but badly overestimating the Greens and significantly underestimating National.
Average deviance from election results
Looking at the deviance of the pollsters’ final pre-election polls from the actual election results, averaged across their results for the four main parties, we’d expect to see Roy Morgan doing better than the average of the other pollsters, if Mr Macskasy’s thesis is correct and Roy Morgan is the “one to watch”.
Unfortunately for Mr Macskasy, it just ain’t so.
For the 2011 election, Roy Morgan’s average deviation across the four main parties was 2.4%. This compared to an average of 2.2% across the average of the other polling companies.
In descending order, from best to worst, the results were:
- NZ Herald – 1.6% average deviation across the four main parties.
- One News – 1.7%.
- Roy Morgan – 2.4%.
- 3News – 2.7%.
- Fairfax – 2.9%.
For the 2008 election, Roy Morgan’s average deviation was 1.8%, bang on the average of the other polling companies. Again, from best to worst:
- 3News – 1.3%.
- NZ Herald – 1.6%.
- Roy Morgan & One News – 1.8%.
- Fairfax – 2.4%.
And going back one further election to 2005, Roy Morgan’s average deviation was 1.9%, compared to an average of 1.7% across the average of the other polling companies. Best to worst?
- 3News – 0.9%.
- One News – 1.4%.
- NZ Herald – 1.8%.
- Roy Morgan – 1.9%.
- Fairfax – 2.5%.
So, based on predicting election results, Roy Morgan sits just south of the middle of the field. Not consistently bad, but not exactly a pinnacle of precision when compared to the other pollsters. Further evidence that cellphone polling is simply not an instant panacea to polling inaccuracy.
* Which is not necessary a fair check of accuracy. The polling company’s polling may have been bang on for every poll leading up to election, but one strange result just before The One Poll That Matters can shred a reputation.