When Sir Owen Glenn announced in 2012 that he was pouring $2 million into an independent inquiry in child abuse and domestic violence, there was a mixed reaction. Negative reaction focussed largely on whether the inquiry would be truly independent, with concerns that Sir Owen, as wielder of the purse strings, would have too large an influence over the inquiry’s methodology and conclusions. When the inquiry’s executive director Ruth Herbert and operations director Jessica Trask resigned in May 2013, citing a “breakdown in the relationship” with Sir Owen, those concerns seemed well-founded. Now, the latest debacle involves the inquiry’s figures regarding the cost of domestic violence. The NZ Herald reports:
A report by economist Suzanne Snively and Wellington theatre student Sherilee Kahui, published by the inquiry yesterday, said family violence cost New Zealand between $4.1 billion and $7 billion a year – up from Ms Snively’s last estimate in 1994 of just $1 billion. But the higher figure of $7 billion was based on a claim that 23.6 per cent of women born in Christchurch in 1977 suffered intimate partner violence in the year leading up to interviews when they were 25 in about 2002. That figure in the original paper published in 2005 by the Christchurch Health and Development Study actually refers to the number of men as well as women who scored 3 or 4 points on a violence victimisation scale for intimate partner violence. Two-thirds of people in the study scored below 3 points and 9.4 per cent scored above 4 points. Those scoring 3 or 4 points were described in the original paper as “predominantly a group of individuals reporting frequent minor psychological aggression and occasionally severe psychological aggression”, but “none reported any of the signs of severe domestic violence [injury or fearfulness]”.
The $7 billion figure was a late addition to the study, which initially contained only a “low-end” estimate of $4.1 billion and a “moderate scenario” of $4.5 billion. So what was the justification given for suddenly messing with the numbers? Well, according to the Herald:
The high-end estimate was added after experts in Auckland and Wellington said they believed the true domestic violence victimisation rates were higher than the “moderate scenario” rates of 18.2 per cent for women and 1.9 per cent for men. “We were struggling to find empirical evidence of an estimate that would be higher than 18.2 per cent,” Ms Kahui said. “So it was about finding something higher.”
Really? The experts didn’t like the data they had, so they went looking for anything that would better suit the conclusion they wanted to reach? And when they found a different study, they misinterpreted it, but didn’t notice they’d made a mistake, because their mistake matched their gut feeling prejudices?
Which is a great pity, because the “low-end” and “moderate scenario” figures of $4.1 and $4.5 billion were already large numbers, which should surely be of concern to any public policy-maker.
Unfortunately though, when researchers have already admitted essentially massaging the data to fit what their in-built biases consider the numbers should have looked like, it certainly raises questions about any subsequent conclusions they might draw.