{"id":16811,"date":"2014-05-28T15:20:10","date_gmt":"2014-05-28T15:20:10","guid":{"rendered":"http:\/\/rafaelfajardo.com\/portfolio\/mischievous-responders-confound-research-on\/"},"modified":"2014-05-28T15:20:10","modified_gmt":"2014-05-28T15:20:10","slug":"mischievous-responders-confound-research-on","status":"publish","type":"post","link":"https:\/\/rafaelfajardo.com\/portfolio\/mischievous-responders-confound-research-on\/","title":{"rendered":"‘Mischievous Responders’ Confound Research On Teens"},"content":{"rendered":"

‘Mischievous Responders’ Confound Research On Teens<\/a><\/p>\n

\n

hautepop<\/a>:<\/p>\n

\n

blackbeardblog<\/a>:<\/p>\n

\n
\n
If kids report that they\u2019re transgender and have one leg and belong to a gang and have several children \u2026 take it with a grain of salt.<\/div>\n<\/blockquote>\n

This is a good article on, basically, kids trolling surveys for a laugh. It happens a lot. If I was a kid in the age of online surveys, I\u2019d do it too. Especially if it was one of those surveys where the sole purpose is a hand-wringing clickbait headline about how kids these days think the Earth is flat or would marry their iPhone or whatever. If you\u2019re offered a ludicrous answer in a survey designed to confirm someone\u2019s view of how crappy the modern world is or how dumb everybody is – well, it\u2019s hardly surprising some people take the hint.<\/p>\n

But as one look at the article summary tells you, this is also a real problem. What the industry euphemistically calls \u201chard-to-reach\u201d populations – small minority populations, basically – are actually harmed by this kind of stuff<\/strong>. The article has a good example – a study that reported negative impacts of adoption turned out to show nothing of the sort when troll answers got taken out.<\/p>\n

I am not part of any population that suffers from prejudice or bias – name a privilege and I benefit from it. But I am a researcher, so I see at reasonably close hand what happens to data. And it seems to me that data and representation have a treacherous relationship. Inevitably, since people find in data what they are looking for.<\/p>\n

On the one hand, data can offer stark evidence of inequalities, different needs and priorities, and different experiences: numbers that can be vital in making a case for change<\/strong>. On the other, data can be the comfort blanket that tells decision makers that change isn\u2019t important. Research can erase minorities by reducing them to the status of a statistical insignificance, or it can ignore the diversity of their experiences in favour of a data-enforced average<\/strong>. There is every reason for people to mistrust data and research.<\/p>\n

And cases like the adoption study one introduce yet another such reason – the possibility that careless research will end up magnifying the voices of the mischievous (or, let\u2019s face it, malicious) and endorse stigmatizing myths instead of revealing anything useful. The remedies outlined – dummy questions in particular – are ingenious, and this kind of internal check should be routine in any important survey. But the uneasy relationship between research and representation – at the analysis stage as well as the collection stage – is harder to solve.<\/p>\n<\/blockquote>\n

A very good post by Tom Ewing – I\u2019ve highlighted the bits I especially like.<\/p>\n

(The pull-quote\u2019s not that clearly worded, btw – the thing to \u201ctake with a grain of salt\u201d are survey results about minority groups such as transgender children, not the existence of trans children themselves. Sadly the latter disbelief still exists<\/a>.)<\/p>\n

Thinking about it, this is basically an example of a \u201cfalse positive\u201d result. The Wikipedia article<\/a> provides a really useful working out of this with regard to HIV testing. Essentially if you\u2019re in a \u201clow-incidence\u201d population for HIV, a \u201cyes\u201d result on an HIV test may be more likely to reflect a failure of the HIV test rather than you actually being HIV+. (This is why health services will re-test a positive blood sample before actually telling the patient.)<\/p>\n

The same is true for online surveys of low-incidence groups or behaviours, right? That the margin of error – the trolling, the fucking around – is able to entirely distort whatever truth might actually be in the data.<\/p>\n

What we need to understand is the \u201cerror rate\u201d of the survey – is it 2% of participants who mess around with answers or 20%? And do they mislead on 2%, 20% or 100% of the questions answered? Using this information we could create a measure of \u201chuman error\u201d or \u201cdeliberate error\u201d to be considered on top of the statistical sampling error<\/a>, another type of predictable error in surveys which occurs because the subset sampled will never be exactly equivalent to the total population they represent.<\/p>\n

There\u2019s therefore a point at which the \u201chuman error\u201d of the survey (e.g. let\u2019s say it sums to 5% error on each question) is greater than the incidence rate of the minority group. At this point, any survey results are subject to the \u201cfalse positive paradox\u201d and need corroborating by other methods before you can read anything into them.<\/p>\n

Postscript: This paper by Lynn Conway is useful for estimating the incidence of transexuality<\/a> not just in the West but globally. Needless to say it\u2019s not a survey methodology. The figure to bear in mind is 1 in 300 people<\/a> (2011 update), or 15m people globally, equivalent to the population of Kazakhstan, Ecuador or Cambodia.<\/p>\n<\/blockquote>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

‘Mischievous Responders’ Confound Research On Teens hautepop: blackbeardblog: If kids report that they\u2019re transgender and have one leg and belong to a gang and have several children \u2026 take it with a grain of salt. This is a good article on, basically, kids trolling surveys for a laugh. It happens a lot. If I was […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"link","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[10],"tags":[],"class_list":["post-16811","post","type-post","status-publish","format-link","hentry","category-words","post_format-post-format-link"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p6PWot-4n9","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/posts\/16811","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/comments?post=16811"}],"version-history":[{"count":0,"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/posts\/16811\/revisions"}],"wp:attachment":[{"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/media?parent=16811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/categories?post=16811"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rafaelfajardo.com\/portfolio\/wp-json\/wp\/v2\/tags?post=16811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}