At CHI 2014 a paper was presented (Liu et al., 2014) which sought to demonstrate, through an analysis of keywords specified in a large tranche of CHI papers (from the last 20 years), that HCI research is lacking in “motor themes”. According to the paper, a motor theme is a commonly addressed topic in a given academic discipline that defines the research “mainstream”. Motor themes themselves are in turn made from keyword clusters that emerge during a co-word analysis process performed on the collection of keywords, and are found to have particular values of centrality and density. (Note that co-word analysis was initially popularised by Michel Callon and other STS researchers for the study of scientific disciplines, based upon a conceptual backdrop of actor-network theory.) Keyword clusters would be things like “collaboration” and “handheld devices” amongst many others (see Liu et al. (2014) for the full list), while centrality and density metrics rate “how ‘central’ a theme is to the whole field” and “the internal cohesion of the theme” (Kostakos, 2015).
I attended the conference talk—delivered with panache by Vassilis Kostakos—and the paper received a curiously noisy reception; this was unusual for the normally serene audience at CHI, so I felt it must have touched a nerve for people, resonating with existing concerns some researchers were already occupied by in some way. There were audible gasps when Kostakos produced the punchline of the talk: a graph plotting the apparent absence of keyword clusters in the coveted “Quadrant I”, i.e., the motor theme zone (see image below). This graph was set in opposition to some published co-word analyses of other research areas (e.g., stem cell research, psychology, etc.), which provided more ‘healthy’ plots.
The tenor of the talk and the response it received have to be considered alongside other activities appearing at CHI in recent years—for instance, the “Interaction Science” SIG of 2014 along with the emergence of yearly events at CHI around ‘replication’ from 2011 onwards. It was this chain of events that made me start to wonder whether a somewhat-dormant cultural undercurrent at CHI (and in HCI more broadly) had been surfaced.
Before I continue I should strongly emphasise that it is extremely important Kostakos and others (e.g., Howes et al. (2014), Wilson et al. (2011), etc.) are discussing the ‘state of the field’. We should applaud them for bringing up this difficult conversation when it is far easier to continue with ‘business as usual’ and to shy away from mulling over what can be contentious arguments (which HCI and CHI in particular seems to prefer to avoid—often confusing challenges to research with attacks on researchers). While I have very different views on the topic, the emergence of a debate about the very idea of HCI, what it is and its seeming contradictions, seems like a valuable activity for us and probably long overdue. In many ways this reflects some of the preoccupations of early HCI manifest in the exchanges between Carroll and Campbell (1985), Newell and Card (1985), and others.
What follows below is a list of my objections, highlights of what I perceive as confusions, as well as agreements (in a strange way) with Liu et al.’s paper and Kostakos’s corresponding ACM interactions magazine piece, “The Big Hole in HCI Research” . There comes with this an implication that the discussion is also applicable to the broader cultural movement that I felt it represents in HCI.
Low-hanging fruit: On method
The easiest line of attack in academic work is usually ‘going for the methods’. Often this is a proxy for other problems that a peer reviewer can’t necessarily articulate. It’s also a strategy reviewers may employ when they have fundamental perspectival differences with the approach taken by the authors of the paper but are perhaps unable to step outside their own perspective for a moment (I would be the first to admit having done this in the past). However, in Liu et al.’s paper, understanding problems with method help us unpack what I see as deeper confusions around disciplinarity and the status of HCI in its relation to ‘science’ (which I shall relentlessly retain in quotation marks within this post, sorry!). So this is where I start: at the low-hanging fruit.
In principle co-word analysis (of academic papers) clearly has value in providing surveys of particular attributes of publication corpora. Yet we should also exercise some caution here: the claims made off the back of the co-word analysis must have attention paid to them, and therefore the basis upon which such claims are being made. First we should remind ourselves of these claims. The subsequent summary of the original CHI paper (Kostakos, 2015) states that the co-word analysis “considers the keywords of papers, how keywords appear together on papers, and how these relationships change over time”. Doing a co-word analysis, it is argued, therefore “can map the ‘knowledge’ of a scientific field by considering how concepts are linked”. To clarify, this is indeed a significant claim: firstly that co-word analysis of paper keywords is an adequate method for ‘mapping knowledge’ for a given discipline, and secondly that it can also be used to demonstrate ‘gaps’ in a knowledge space.
My problem with this methodologically is that there is no obvious sense in which co-word analysis of the keywords of papers can provide an adequate overview of “the ‘knowledge’ of a scientific field” (Kostakos, 2015) because it does not follow that keywords are being deployed by authors in order to provide some set of indices to some known-in-common ‘map’ of disciplinary knowledge in the first place. Instead one must subscribe to the idea that authors’ deployments of keywords are driven by some ‘hidden order’ which only becomes visible through the application of the method of co-word analysis. I want to question that claim.
In order to unravel why I might question this, I should explain that I think that some confusions are being made about just what is being ‘done’ in the writing of keywords for academic papers. These confusions touch on more fundamental misunderstandings about language. Firstly we have to consider how keywords are encountered in the situation in which CHI papers are read (i.e., consider the visual practices of paper-reading CHI-formatted research): they are placed on the first page of the paper, they are prominent under the abstract (where we might start reading) and they have their own headed section. (We don’t ‘read’ keywords, we ‘look at them’; I think there is a distinction.) The point is that their visual organisation sets them up to do particular kinds of work for the reader of the paper only as it is read, i.e., they cannot be easily removed from their situational relevance to the surrounding text and the manner by which we as readers encounter them.
For instance, authors may deploy keywords according to myriad of possible reasons, many of which may or may not pertain to some ‘hidden order’—i.e., the ‘knowledge map’ that is discoverable through co-word analysis. Here are some examples. Keywords may get deployed as terms for ACM Digital Library search optimisation. Keywords may be ‘signals’ for allying oneself to some sub-community of researchers or ‘sending a message’ to another. Keywords can be used as discriminators of novelty perhaps via the creation of new terms (and thus claiming prospective research spaces). Keywords can be referents to established corpuses of work, framing judgements on the work through the lens of an an existing tradition in order to tell reviewers that “this is one-of-those-papers, so judge it on those terms”. And, of course still possible, keywords may be indices or intellectual coordinates to some agreed-upon ‘map’ of HCI knowledge (although one might ask by which textbook those coordinates are even constructed). The point is that the practical purposes of keyword deployment get lost in the co-word analysis because all keywords are treated in the same way.
None of the above necessarily diminishes Kostakos’s notion of “The Big Hole in HCI” but it certainly exposes some problematic methods by which the claim is substantiated in the first place. Nevertheless, in order to provide a grounding for such a claim, Liu et al. must also conceptualise HCI as a discipline, which is what I turn to next.
On disciplines and disciplinarity
Both Liu et al.’s CHI paper and Kostakos’s interactions article refer to HCI on a number of occasions as a discipline. The main argument refers to a ‘lifecycle’ for motor themes and their role in disciplines. Disciplinary architecture here is described by various quadrants (see image below), tracking themes as they are born (“Quadrant III: Emerging or declining themes”), begin to stabilise (“Quadrant IV: Basic and transversal themes”), go mainstream (“Quadrant I: Motor themes”) and then die off (back to Quadrant III) or perhaps decline (“Quadrant II: Developed but isolated themes”). Themes may never reach Quadrant I or may go straight to Quadrant II, or perhaps get stuck in Quadrant IV or never make it past Quadrant III. But the basic idea is of the lifecycle and notions of a healthy movement of themes across the graph.
This description, of course, assumes HCI’s classification as a discipline and offers remedial advice for establishing its stability (see the discussion on implications for design below). Largely this assertion passes without comment; in fact this reference to HCI as a discipline is necessary in order to make sure it becomes comparable with other disciplinary objects that Liu et al. hold as reference points based on results of other researchers’ co-word analyses. (The comparison disciplines used in Liu et al. are psychology, consumer behaviour, software engineering and stem cell research.) These reference points can then be used to show the absence of “Quadrant I” keyword clusters in HCI compared to other disciplines and thus the disciplinary deficiencies of HCI.
Even if we take HCI as a discipline, the corresponding implication of Liu et al. that disciplines are somehow ‘comparable’ is itself contentious, I would argue. For instance, it is hard to see how, say, the activities of stem cell researchers have any bearing on the activities of HCI researchers, and it is not clear whether it is reasonable to assume their paper-writing practices, let alone their everyday research work practices, are similar. Or, perhaps, those of psychologists and software engineers. Instead I would suggest that each works with phenomena particular to them, and have methods of reasoning and research practices particular to them. What counts as relevant research questions in one has nothing necessarily to do with what counts in another. Further, it is also unclear with this disciplinary assumption in place why it might be that specialisms like stem cell research should be compared with all of psychology—a broad church to say the least—why not social psychology or cognitive psychology? Instead, co-word analysis may be just analysing how keywords (of whatever extraction method) get used, and it may be just that HCI’s use of keywords is different rather than deficient.
The very idea that HCI is a discipline at all is also itself certainly contentious. I think Yvonne Rogers is correct when she suggests that HCI is an “interdiscipline”. The implication (intended or not by her use of this term, I don’t know) is that in being an interdiscipline, HCI should indeed have “The Big Hole” Kostakos identifies, because the very nature of an interdiscipline would be an absence of a disciplinary core. If there were some essential disciplinary core to HCI it would struggle in its role as broker between disciplines (as pointed out by Alan Blackwell recently (Blackwell, 2015)). Even the earliest moments of HCI commenced as a meeting place between cognitive psychologists, software engineers and, to some extent, designers. In other words ‘we have never have been disciplinary’.
At its most basic the notion of a discipline is an attempt at finding a way of ordering knowledge (Weingart, 2010). It is not a ‘natural fact’ and we cannot treat ‘the discipline’ as transcendent features of a ‘hidden order’. ‘A discipline’ is (I’d argue) an epiphenomenon of the particular community of research practice. And it’s precisely because of this that the arguments made about the application of concepts borrowed from (broad brush) ‘science’ become difficult to handle. Onto which topic I turn next.
Accumulation, replication, generalisation: On ‘science’ and ‘the scientific’ in HCI
One of the key assertions in the interactions article is that “a lack of motor themes should be a very worrying prospect for a scientific community”. Kostakos suggests that remedies should be pursued “[if] we want to claim that CHI is a scientific conference”. I interpret this to mean that HCI has the potential for a scientific disciplinarity that may be established through the development of motor themes. Accordingly, a set of signature scientific procedures or ‘scientific qualities’, as I’ll label them, are described by Kostakos so as to achieve this; these are mentioned as 1. accumulation (science’s work is that of cumulative progress), 2. replication (science’s work gains rigour from replicability), and 3. generalisation (science’s cumulative work involves expansivity). Kostakos describes how “new initiatives have sprung up our field to make it more scientific in the sense of repeating studies, incremental research, and reusable findings”, which I take as reference to the replication (Wilson et al., 2011) and “interaction science” (Howes et al., 2014) agendas I describe above.
Yet making HCI “more scientific” is not really a new drive in HCI. HCI’s initial development was oriented strongly by many self-described scientists (going by Liu et al.’s scheme of labelling sciences) from psychology and cognitive science, both of which have often been at pains to demonstrate their scientific credentials through adherence methods presumed to be drawn from the natural sciences. So one could argue that the cultural foundations for HCI’s desire to be ‘scientific’ have always been present. In addition, attempts to reorder HCI back into accord with the ‘scientific qualities’ outlined by Kostakos have also been suggested before, such as notions from Whittaker et al. (2000) to develop standardised “reference tasks” in order to establish generalisation, and therefore ‘scientific’ legitimacy. These attempts have faltered, however.
The problem, I think, is firstly that can be very problematic to engage in deployments of ‘science’ as a concept. Secondly I think it is mistaken at least to imply (or not guard against an implication even if unintended) that these qualities are properties of ‘science itself’.
On the first point, ‘science’ is a linguistic chimera for HCI because the term is so diversely and nebulously applied, not only in Liu et al. and by Kostakos, but also in discourse within HCI more broadly. It is unhelpful for us because ‘science’ is often used to do very different things that we may well wish to avoid. For instance, this may be in establishing a kind of epistemic and / or moral authority, or an attempt to gain peer esteem for a research community in poor academic standing, or internally as a method for legitimising certain kinds of work and delegitimising others’ (i.e., categorisation between ‘science’ and ‘not science’) in the course of cultural wars. ‘Science’ then becomes problematic because such (rhetorical) uses can tend to be deployed in place of adequate assessments of research rigour on its own terms (what ‘own terms’ might mean is explored below).
This leads to my second point, the idea of the accumulation, replication and generalisation of findings as being intrinsic properties of ‘science’ rather than methodical practices conducted by a community of researchers (see Crabtree et al. (2013) and also Rooksby (2014) on this point). This latter view suggests that the standards of ‘what counts’ as a generalisation, ‘what is’ a relevant process of accumulation (which I take as the establishing within researchers’ discourse of particular motor themes), and ‘what motivates’ the conduct of replications, should be decided upon as a matter of agreement between researchers. It cannot be determined through adherence to an external and nebulous set of ‘scientific standards’ that are adopted from a notion of ‘science in general’ (e.g., what we might call ‘textbook’ understandings developed from formal descriptions of the natural sciences)—for no such thing really exists. Instead, if by ‘science’ we mean ‘demonstrating a rigour agreed upon by practitioners of the relevant and particular genre of reasoning the work pertains to’ then I might consider it a useful term. But it seems unlikely this is what is being meant (it’s definitely very unwieldy!).
This all said, I have a great deal of sympathy for the desire of Liu et al., Kostakos, Wilson et al., Howes et al., and others who seek to increase the rigour of the HCI community—such a motivation for the critique can only be encouraged. Yet, to reiterate, this cannot come at the expense of specifying singular-yet-nebulous approaches like making HCI ‘more scientific’, particularly when the model of ‘more scientific’ is based on classic tropes of what a mythical ‘science’ is said to be, rather than as a matter of how researchers engage in the various shared practices to establish agreement and disagreement over findings.
Instead, I think if we take the ‘interdiscipline’ challenge seriously we should be looking for two things of particular HCI contributions. Firstly, we should expect a rigour commensurate with the research’s own disciplinary wellsprings, whether this is (cognitive, social, etc.) psychology, anthropology, software engineering or, more recently, the designerly disciplines. Rare examples of such ‘internal rigour’ being taken to task is found in the ‘damaged merchandise’ (Gray and Salzman, 1998), ‘usability evaluation considered harmful’ (Greenberg and Buxton, 2008) or ‘ethnography considered harmful’ (Crabtree et al., 2009) debates (although in HCI they feel like more like ‘scandals’—which perhaps says something about the level of debate in HCI more than anything else). What this means is that the adoption of materials, approaches, perspectives, etc. from disciplines ‘external’ to HCI (and remember in this view, there is only ‘the external’) should not result in lax implementations of such imported concepts, approaches, etc. within the HCI community. The ‘magpie-ism’ of HCI research is a double-edged sword: increasing vigour and research creativity, yet often resulting in violence being done to the origins of imported approaches, concepts, etc. And without specialist attention, weak strains are sustained / incubated within HCI; the controversies outlined above are manifestations of this problem. Secondly, we should expect a rigour in the HCI research contribution’s engagement with the notion of being an ‘interdiscipline’. This is what ‘implications for design’ is all about (albeit quite a deficient form as pointed out by Kostakos and others); that is, an attempt to meet others at the interface of disciplines. But more on this next.
The curse of the interdiscipline: Implications for design
The interactions article builds upon Liu et al. by arguing that “the reason our discipline lacks mainstream themes, overarching or competing theories, and accumulated knowledge is the culprit known as implications for design”. The absence of HCI’s engagement with proper ‘scientific qualities’ like generalisation and accumulation is thus pinned to the perceived need to write “implications for design” sections in CHI papers in order to get them past peer review even when the rest of a paper is presenting a high quality of research work. Moreover, within the interdisciplinary community of HCI, it really can never be enough, as Kostakos rightly points out, just to vaguely target ‘relevant practitioners’.
While I have sympathy for this argument, I also think a reassessment has to be made as to why the ‘implications for design’ discussion has emerged in the first place, which I have hinted at above. We can use similar questions to those posed over keywords: what is ‘being done’ in the writing of ‘implications for design’? (Helpfully, Sas et al. (2014) have recently published a categorisation of the different kinds of uses ‘implications of design’ is put to.)
I would argue that ‘implications for design’ can be read as a gesture towards being an ‘interdiscipline’. They are typically an effort to answer the question “why should I (the reader) care about this work?”, a question that is in no way unique to HCI. It would be a mistake to assume that we need not be accountable to the ‘interdisciplinary other’ in HCI. And yes, often the gesture is poorly performed and poorly labelled.
Instead we should perhaps start considering ‘implications for HCI’ rather than ‘implications for design’ as a better sign of taking work at the interface of disciplines seriously.
Howes, A., Cowan, B. R., Payne, S. J., Cairns, P., Janssen, C. P., Cox, A. L., Hornof, A. J., and Pirolli, P. (2014). Interaction Science Spotlight. CHI 2014.
Weingart, P. (2010). A short history of knowledge formations. In Thompson, J. Klein and Mitcham, C. (eds.), The Oxford Handbook of Interdisciplinarity. OUP Oxford, pp. 3-14.
