hautepop:

blackbeardblog:

If kids report that they’re transgender and have one leg and belong to a gang and have several children … take it with a grain of salt.

This is a good article on, basically, kids trolling surveys for a laugh. It happens a lot. If I was a kid in the age of online surveys, I’d do it too. Especially if it was one of those surveys where the sole purpose is a hand-wringing clickbait headline about how kids these days think the Earth is flat or would marry their iPhone or whatever. If you’re offered a ludicrous answer in a survey designed to confirm someone’s view of how crappy the modern world is or how dumb everybody is – well, it’s hardly surprising some people take the hint.

But as one look at the article summary tells you, this is also a real problem. What the industry euphemistically calls “hard-to-reach” populations – small minority populations, basically – are actually harmed by this kind of stuff. The article has a good example – a study that reported negative impacts of adoption turned out to show nothing of the sort when troll answers got taken out.

I am not part of any population that suffers from prejudice or bias – name a privilege and I benefit from it. But I am a researcher, so I see at reasonably close hand what happens to data. And it seems to me that data and representation have a treacherous relationship. Inevitably, since people find in data what they are looking for.

On the one hand, data can offer stark evidence of inequalities, different needs and priorities, and different experiences: numbers that can be vital in making a case for change. On the other, data can be the comfort blanket that tells decision makers that change isn’t important. Research can erase minorities by reducing them to the status of a statistical insignificance, or it can ignore the diversity of their experiences in favour of a data-enforced average. There is every reason for people to mistrust data and research.

And cases like the adoption study one introduce yet another such reason – the possibility that careless research will end up magnifying the voices of the mischievous (or, let’s face it, malicious) and endorse stigmatizing myths instead of revealing anything useful. The remedies outlined – dummy questions in particular – are ingenious, and this kind of internal check should be routine in any important survey. But the uneasy relationship between research and representation – at the analysis stage as well as the collection stage – is harder to solve.

A very good post by Tom Ewing – I’ve highlighted the bits I especially like.

(The pull-quote’s not that clearly worded, btw – the thing to “take with a grain of salt” are survey results about minority groups such as transgender children, not the existence of trans children themselves. Sadly the latter disbelief still exists.)

Thinking about it, this is basically an example of a “false positive” result. The Wikipedia article provides a really useful working out of this with regard to HIV testing. Essentially if you’re in a “low-incidence” population for HIV, a “yes” result on an HIV test may be more likely to reflect a failure of the HIV test rather than you actually being HIV+. (This is why health services will re-test a positive blood sample before actually telling the patient.)

The same is true for online surveys of low-incidence groups or behaviours, right? That the margin of error – the trolling, the fucking around – is able to entirely distort whatever truth might actually be in the data.

What we need to understand is the “error rate” of the survey – is it 2% of participants who mess around with answers or 20%? And do they mislead on 2%, 20% or 100% of the questions answered? Using this information we could create a measure of “human error” or “deliberate error” to be considered on top of the statistical sampling error, another type of predictable error in surveys which occurs because the subset sampled will never be exactly equivalent to the total population they represent.

There’s therefore a point at which the “human error” of the survey (e.g. let’s say it sums to 5% error on each question) is greater than the incidence rate of the minority group. At this point, any survey results are subject to the “false positive paradox” and need corroborating by other methods before you can read anything into them.

Postscript: This paper by Lynn Conway is useful for estimating the incidence of transexuality not just in the West but globally. Needless to say it’s not a survey methodology. The figure to bear in mind is 1 in 300 people (2011 update), or 15m people globally, equivalent to the population of Kazakhstan, Ecuador or Cambodia.