Jo Bates Paul D. Clough Robert J¨aschke
Jahna Otterbacher (Eds.)
BIAS
Proceedings of the Workshop on Bias in Information, Algorithms, and Systems Sheffield, United Kingdom, March 25, 2018
Editors Jo Bates
The University of Sheffield jo.bates@sheffield.ac.uk Paul D. Clough
The University of Sheffield p.d.clough@sheffield.ac.uk Robert J¨aschke
Humboldt-Universit¨at zu Berlin robert.jaeschke@hu-berlin.de Jahna Otterbacher
Open University of Cyprus jahna.otterbacher@ouc.ac.cy
Typesetting: Camera-ready by author
Preface
More than ever before, data, information, algorithms and systems have the po- tential to influence and shape our experiences and views. With increased access to digital media and the ubiquity of data and data-driven processes in all areas of life, an awareness and understanding of topics, such as algorithmic account- ability, transparency, governance and bias, are becoming increasingly important.
Recent cases in the news and media have highlighted the wider societal effects of data and algorithms requiring we pay it more attention.
The BIAS workshop brought together researchers from different disciplines who are interested in analysing, understanding and tackling bias within their discipline, arising from the data, algorithms and methods they use. The work- shop attracted 14 submissions, including research papers and extended abstracts.
After a peer reviewing process in which each submission received three indepen- dent reviews, the following six papers were accepted and are included in these proceedings:
– Claude Draude, Goda Klumbyte and Pat Treusch: Re-Considering Bias:
What Could Bringing Gender Studies and Computing Together Teach Us About Bias in Information Systems?
– Christoph Hube, Besnik Fetahu and Robert J¨aschke:Towards Bias Detection in Online Text Corpora
– Vasileios Iosifidis and Eirini Ntoutsi:Dealing with Bias via Data Augmenta- tion in Supervised Learning Scenarios
– Serena Oosterloo and Gerwin van Schie: The Politics and Biases of the
“Crime Anticipation System” of the Dutch Police
– Alan Rubel, Clinton Castro and Adam Pham: Algorithms, Bias, and the Importance of Agency
– William Seymour:Detecting Bias: Does an Algorithm Have to Be Transpar- ent in Order to Be Fair?
The papers cover a wide range of research topics: from conceptual discussions of algorithmic transparency and fairness to empirical research and case studies.
William Seymour (page 2) discusses the relationship between the fairness of an algorithm and its transparency and the important distinction between process transparency and output transparency. For most effective machine learning al- gorithms we cannot hope to obtain process transparency, as their inner workings are beyond conscious human reasoning. Seymour argues that a viable alternative is to analyse the transparency of the outcome of an algorithm. He also presents two exemplary methods – local explanations and statistical analysis – that could help to understand the fairness of the outputs.
Alan Rubel, Clinton Castro and Adam Pham (page 9) address the notions of agency and autonomy with regard to algorithmic systems. While debates about biases in algorithmic systems often emphasise potential and actual harms, the
ii Preface
authors argue that our concerns about algorithms should not be limited to such issues. Moving the debate forward beyond interest in algorithmic harms, they argue that the ”moral salience” of algorithmic systems cannot be understood without also addressing their impacts on human agency, autonomy and respect for personhood.
Claude Draude, Goda Klumbyte and Pat Treusch (page 14) explore the po- tential for theoretical frameworks from gender studies – including Haraway’s
“situated knowledges” and Harding’s “standpoint theory” – to inform a better understanding of how bias emerges within information systems. With a particu- lar focus on issues of androcentrism, over/underestimation of gender differences and the stereotyping of gender traits in the workings of information systems, their paper considers how feminist insights might help to account for and pre- vent bias in information system design.
Christopher Hube, Robert J¨aschke and Besnik Fetahu (page 19) present a method for identifying language bias within textual corpora using word embed- dings, based on word2vec. This includes a two-stage process in which firstly seed words indicating bias are extracted from Conservapedia, a dataset that includes opinionated political articles. The second step uses word2vec to identify bias words involving the seed list created previously. The approach iterates to keep growing the list of bias words that could be used to form feature vectors for tasks such as supervised learning.
Vasileios Iosfidis and Eirini Ntoutsi (page 24) describe techniques for data augmentation (SMOTE and oversampling) to deal with cases of class imbalance where under-represented groups can affect data-driven methods, such as super- vised learning. Their experiments on the Census Income and German Credit datasets show that the classes can be more equally represented using data aug- mentation without affecting overall classification performance. This is particu- larly important when dealing with biases in datasets around certain attributes, such as gender and race, where the methods proposed in the paper can reduce classification errors for potentially discriminated groups.
Serena Oosterloo and Gerwin van Schie (page 30) walk us through a crime prediction system currently being used in the Netherlands, from a critical data studies perspective. Their paper illustrates various sources of inaccuracies in the system, including those that cannot be helped – because the necessary attribute cannot be measured with great precision in the offline world – as well as those that result from human biases (e.g., the choices made during the process of classifying a crime and the parties involved).
The workshop was opened by a keynote fromAnsgar Koene, University of Nottingham, (page 1) discussing socio-technical causes of bias in algorithms and systems and the role of policies and ethical standards.
Sheffield, March 25, 2018 https://ir.shef.ac.uk/bias/
Jo Bates Paul D. Clough Robert J¨aschke Jahna Otterbacher
Program Committee
Jo Bates The University of Sheffield Alessandro Checco The University of Sheffield Paul D Clough The University of Sheffield
David Garcia Medical University of Vienna and Complexity Sci- ence Hub
Maria G¨ade Humboldt-Universit¨at zu Berlin
Jutta Haider Lund University
Libby Hemphill University of Michigan Frank Hopfgartner The University of Sheffield Robert J¨aschke Humboldt-Universit¨at zu Berlin Ansgar Koene University of Nottingham Jochen L. Leidner Thomson Reuters
Kristian Lum Human Rights Data Analysis Group Jahna Otterbacher Open University of Cyprus
Elvira Perez Vallejos University of Nottingham Emilee Rader Michigan State University Kalpana Shankar University College Dublin
Claudia Wagner GESIS – Leibniz Institute for the Social Sciences Ziqi Zhang The University of Sheffield
Contents
Policy approaches to socio-technical causes of bias in algorithmic
systems – what role can ethical standards play?. . . 1 Ansgar Koene
Detecting Bias: Does an Algorithm Have to Be Transparent in Order
to Be Fair?. . . 2 William Seymour
Algorithms, Bias, and the Importance of Agency. . . 9 Alan Rubel, Clinton Castro and Adam Pham
Re-Considering Bias: What Could Bringing Gender Studies and
Computing Together Teach Us About Bias in Information Systems?. . . 14 Claude Draude, Goda Klumbyte and Pat Treusch
Towards Bias Detection in Online Text Corpora . . . 19 Christoph Hube, Besnik Fetahu and Robert J¨aschke
Dealing with Bias via Data Augmentation in Supervised Learning
Scenarios . . . 24 Vasileios Iosifidis and Eirini Ntoutsi
The Politics and Biases of the “Crime Anticipation System” of the
Dutch Police . . . 30 Serena Oosterloo and Gerwin van Schie
Policy approaches to socio-technical causes of bias in algorithmic systems – what role can
ethical standards play?
Ansgar Koene University of Nottingham
Despite warnings about Bias in Computer Systems going back to at least 1992, it has taken concerted efforts by groups of researchers like FAT/ML (Fair- ness, Accountability and Transparency in Machine Learning) and news and me- dia reports highlighting discriminatory effects in data driven algorithms (e.g.
recidivism prediction) to dispel the na¨ıve-optimistic myth that the logic/mathe- matics of computation would automatically result in objectively unbiased out- comes.
Starting from a brief overview of key challenges in the design and use of data driven algorithmic decision making systems I will focus in on the inherently socio-technical nature of real-world applications that give rise to concerns about bias.
Against this background I will discuss the use-cases and design framework currently under consideration in the IEEE P7003 Standard for Algorithmic Bias Considerations working group, with comparison to the findings coming out of the UnBias project on the lived experience and concerns of teenaged digital na- tives regarding their every-day interactions with algorithmically mediated online media.
I will conclude with a review of some of the policy making initiatives that were recently launched by professional associations (e.g. ACM principles; IEEE Global Initiative on Ethical Considerations in AI and Autonomous Systems), industry led organizations (e.g. Partnership on AI) and regional/national gov- ernment bodies (e.g. EC Algorithmic Awareness Building; TransAlgo in France;
UK parliamentary inquiries).
Ansgar Koene is Senior Research Fellow at the Horizon Institute for Digital Economy Research at the University of Nottingham. He is Co-Investigator on the UnBias project whose goal is to emancipate users against algorithmic biases for a trusted digital economy. Ansgar is chair of the IEEE working group for the development of the Standard for Algorithmic Bias Considerations.
Detecting Bias: Does an Algorithm Have to Be Transparent in Order to Be Fair?
William Seymour University of Oxford, Oxford, UK {william.seymour@cs.ox.ac.uk}
Abstract. The most commonly cited solution to problems surrounding algorithmic fairness is increased transparency. But how do we reconcile this point of view with the state of the art? Many of the most effective modern machine learning methods (such as neural networks) can have millions of variables, defying human understanding. This paper decomposes the quest for transparency and examines two of the options available using technical examples. By considering some of the current uses of machine learning and using human decision making as a null hypothesis, I suggest that pursuing transparentoutcomesis the way forward, with the quest for transparent algorithms being a lost cause.
Introduction
Recent investigations into the fairness of algorithms have intensified the call for machine learning methods that are transparent. Unless an algorithm is transparent, so the argument goes, then how are we to know if it is fair? But this approach comes with a problem: many machine learning methods are useful preciselybecause they work in a way which is alien to conscious human reasoning.
Thus, we place ourselves in the position of having to choose between a more limited (and potentially less effective) set of algorithms that work in ways that we can understand, and those which are better suited to the task at hand but cannot easily be explained. To clarify, this paper is concerned with the use of transparency as a tool for auditing and communicating decisions, rather than debate over the higher level ‘transparency ideal’, or harmful/obstructive uses of transparency as described by [1, 2].
This paper will discuss the arguments for and against transparency as a design requirement of machine learning algorithms. Firstly, I will break down what we mean when we talk about fairness and transparency, before considering arguments and examples from both sides of the discussion. I will cover two different black box techniques that provide interpretable explanations about algorithmic decisions—local explanations and statistical analyis—as well as some of the problems associated with each of these techniques. The techniques listed are by no means exhaustive and are meant to represent different styles that can be used to generate explanations. To conclude, there will be a discussion on the role that transparency might play in the future of machine learning.
What Do We Mean by Transparency?
Since transparency in this context is rooted in fairness, perhaps a better starting point would be to ask what we mean by fairness. A dauntingly complex question in itself, most people would consider approaches that ‘treat similar people in similar ways’ to be fair. These often coalesce along lines of protected characteristics (such as race and gender), as these are where the most glaring problems are often to be found. These characteristics are often expected to be excluded from the decision making process even if they are statistically related to its outcome.
But problems arise when a philosophical definition of fairness is translated into a set of statistical rules against which an algorithm is to be compared. There are multiple perpendicular axes against which one can judge an algorithm, and the best fit will vary based on the context in which the algorithm is used. Examples include predictive parity, error rate balance, and statistical parity to name a few [3]. To further muddy the waters, it is possible to draw a distinction between process fairness (the actual process of making a decision) andoutcome fairness (the perceived fairness of a decision itself) [4]. It is possible for an algorithm with low process fairness (e.g. including race as a factor in decision making) to exhibit high output fairness (e.g. ensuring similar levels of false positives across racial groups).
As for the term transparency, I refer to information available about an algorithm that details part of its decision making process or information about the decisions it makes, which can be interpreted by a human being. Depending on the context, this could be a data scientist, policy maker, or even a member of the public. Interpretability is a key requirement here, ensuring that published data do actually aid our understanding of algorithmic processes.
As we are concerned about investigating fairness, it makes sense to think of two types of transparency corresponding to those for fairness: process transparency (how much we understand about the internal state of an algorithm) and outcome transparency (how much we understand about the decisions, and patterns in decisions, made by an algorithm). This distinction is important, as while there exist tools that can achieve some level of outcome transparency for all algorithms, only certain types of algorithm exhibit process transparency.
Method I: Local Explanations
The first method we consider is a black box method of explaining individual decisions. Local explanations work by sampling decisions from the problem domain weighted by proximity to the instance being explained. These samples are then used to construct a new model that accurately reflects the local decision boundary of the algorithm. For non-trivial algorithms, the local model will be a bad fit for other inputs, as global decision boundaries will be of a higher dimension than the local one (see Figure 1).
An example of this would be online content moderation. If a user has submitted a post which is deemed by an algorithm to be too toxic, we might want to explain Detecting Bias: Does an Algorithm Have to Be Transparent? 3
Fig. 1.Depiction of a local decision boundary (dotted line) of the bold cross derived from sampled decisions (shapes) weighted by their distance from the decision being explained. The global decision boundary is represented by the pink and blue background.
Reproduced from [5] with permission.
to them which parts of their message caused the algorithm to reject it. For the input sentence
“idiots. backward thinking people. nationalists. not accepting facts. sus- ceptible to lies”1
a local explanation might reveal that the words “idiots”, and “nationalists” are the greatest factors contributing to the message being flagged as toxic. This is not to say that all messages containing the word “nationalists” are toxic, but that the word is considered problematic in this context.
Here we have produced an interpretable explanation without knowing anything about how the algorithm operates—we can say that local explanations provide evidence for outcome fairness. By looking at these explanations for decisions a system makes, we have enough information to conclude that a decision was unfair because it violates our definition of fairness as described above. This is a good start to our goal of auditing for fairness.
Moving From Local to Global
Local explanations do a good job of informing users of the main factors behind the decisions they are subject to, but they fall short of providing assurance that the system as a whole operates fairly. In order for this to happen, one needs to be able to create a mental model of the system which is functionally close enough to the original that one can predict what it will do (or at least believe that
1 Taken from the list of examples on the Google Perspective API home page at https://www.perspectiveapi.com/
4 William Seymour
its reasoning will be of sufficient quality). Because local explanations consider only facets of the current decision, they do not reveal much about the wider reasoning that pervades an algorithm. While of great use to an individual who is concerned about a decision concerning themselves, they are much less useful to an auditor who is seeking assurance that the algorithm as a whole is fair. A handful of randomly chosen samples being satisfactory does not give sufficient assurance that all answers will satisfy a set of fairness criteria. This highlights the distinction drawn earlier between local and global fairness guarantees.
Perhaps then, explanations for audits need to operate at a higher level than local explanations. But then we encounter the problem that the high dimensionality of non-trivial models means that global explanations must be simplified to the point of absurdity in order to be intelligible. If explanations can be thought of as “a three way trade off between the quality of the approximation vs. the ease of understanding the function and the size of the domain for which the approximation is valid” [6], then do we risk going so far towards the scale end of the spectrum that we must abandon our hopes of arriving at an answer which is also understandable and accurate?
Method II: Statistical Analysis
Given these problems it is perhaps questionable as to whether any scheme which only considers individual decisions can ever be sufficient to determine if an algorithm is fair or not. When considering higher level explanations of algorithms we find that statistical analysis can offer us the reassurance (or otherwise) that we desire about an algorithm, taking into accounts trends across entire groups of users rather than being limited to individual circumstances.
Statistical analysis is another black box method, and often takes the form of calculating information about particular groups of users and how they are dealt with by the algorithm. By comparing accuracies and error rates between groups it is possible to identify systemic mistreatment. Explaining these findings is often simple, given most people’s intuitive understanding of accuracy and false positives/negatives (see Figure 2).
Lies, Damned Lies, and Statistics
One trap that exists when performing statistical analysis is that due to the aforementioned multitude of ways one can express statistical fairness it is almost always possible to present evidence of compliance and noncompliance. This is because many types of statistical fairness are inherently incompatible with each other: altering the classifier to increase fairness along one axis will always decrease it in another.
In the wake of Machine Bias [7], ProPublica and Northepoint argued that the COMPAS algorithm was unfair and fair, respectively. Both parties were technically correct. These explanations are thus only valid when paired with background knowledge in data science and ethics, and may not suitable for Detecting Bias: Does an Algorithm Have to Be Transparent? 5
Fig. 2.A comparison of decile risk scores between black and white prisoners assessed by the a recidivism algorithm. A score of 1 represents the lowest risk, and 10 the highest risk. One can clearly see that white prisoners are much more likely to be classified as low risk. Generated from code published by [7].
presentation to the general public—doing so could lead to a reduction in trust of machine learning techniques, especially if the presented facts are used to support previously held beliefs which are incorrect [2].
Another issue is that all of the methods that provide interpretable decisions inevitably present reasoning that correlates with a decision making algorithm but is not causally related to its output. In these cases if the algorithms internals are indeed intractable then it will remain impossible to ever prove a causal link between the explanation system and the algorithm itself. This is not an insurmountable problem, by its nature all machine learning deals with correlations, but it needs to be understood that using black box analysis techniques is not enough to guarantee that a system is fair unless the entire problem domain is exhaustively searched. For any model big enough to require auditing this will be impossible.
Discussion
The point that becomes clear as we look at the realities surrounding transparency in machine learning is that exclusively pursuing understandable and/or open source algorithms is infeasible. When reviewing even a moderately-sized code base, it quickly becomes apparent that issues of transparency and interpretability cannot be resolved simply by making computer code available [8]. With a caveat for certain contexts, we need to be able to deal with algorithms that are not inherently transparent.
Put another way, industry players are incentivised to use the machine learning techniques that are best for profits, a decision which almost always favours efficacy over interpretability. Given this, we need to consider techniques that can 6 William Seymour
be applied to methods where the raw form of the model defies our understanding, such as neural networks.
The position I advocate for here is not that we should give up completely on pursuing transparency, but that we need to be clearer about what we are seeking.
By failing to differentiate between process and outcome transparency we run the risk of intractable algorithms being used as an excuse for opaque and potentially unfair decision making.
At the same time, it is important to understand the epistemological implica- tions that come from using correlation-based methods to provide transparency.
However, this is already something that is being dealt with when it comes to algorithmic decisions themselves. If the rest of the community can tackle the misguided use of algorithmic ‘evidence’, then it is surely also possible to do the same with transparency.
Ultimately it is up to us to decide in each case whether the correlation-focused evidence we can generate about an algorithm is sufficient to draw conclusions about its fairness or unfairness. It is helpful to frame the question in the context of the alternative, which is human-led decision making. It is no secret that decisions made by people can occasionally be opaque and prone to bias [8], and using this human baseline as a null hypothesis reminds us that the goal of our quest for transparency should be for machines exceed our own capabilities, not to obtain perfection.
A realistic approach would be to use both types of technique (white and black box) in tandem, analysing the inner workings of simpler components where possible and utilising second hand explanations and analysis otherwise. We should remember that transparency can appear as a panacea for ethical issues arising from new technologies, and that the case of machine learning is unlikely to be any different [9]. That it is difficult to analyse the inner workings of particular techniques will not slow or prevent their uptake, and it is increasingly clear that there is a public and regulatory appetite for more accountable machine learning systems. Therefore, going forward we need to be focussed on the attainable if we are to effectively hold algorithm developers and their algorithms to account.
References
1. Ananny, M., Crawford, K.: Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. new media & society (2016) 2. Flyverbom, M.: Transparency: Mediation and the management of visibilities. Inter-
national Journal of Communication10(1) (2016) 110–122
3. Chouldechova, A.: Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data5(2) (2017) 153–163
4. Grgic-Hlaca, N., Zafar, M.B., Gummadi, K.P., Weller, A.: The case for process fairness in learning: Feature selection for fair decision making. In: NIPS Symposium on Machine Learning and the Law. (2016)
5. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you?: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM (2016) 1135–1144
Detecting Bias: Does an Algorithm Have to Be Transparent? 7
6. Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explanations without opening the black box: Automated decisions and the GDPR. (2017)
7. Larson, J., Mattu, S., Kirchner, L., Angwin, J.: How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) (2016)
8. The Royal Society: Machine learning. Technical report, The Royal Society (2017) 9. Mittelstadt, B.D., Allo, P., Taddeo, M., Wachter, S., Floridi, L.: The ethics of
algorithms: Mapping the debate. Big Data & Society3(2) (2016) 8 William Seymour
Algorithms, Bias, and the Importance of Agency
Alan Rubel1, Clinton Castro1, and Adam Pham1
1 University of Wisconsin, Madison WI 53706, USA
Abstract. We argue that an essential element of understanding the moral salience of algorith- mic systems requires an analysis of the relation between algorithms and agency. We outline six key ways in which issues of agency, autonomy, and respect for persons can conflict with algo- rithmic decision-making.
Keywords: Algorithms, Bias, Agency.
1 Algorithms and agency
The last few years have seen growing interest in the uses, misuses, and biases of auto- mated, algorithmic information systems. One key area of inquiry concerns ways in which algorithms reflect various biases, for example in model choice, by reflecting existing social structures, and by reifying antecedent beliefs. The literature contains a number of arguments regarding how algorithms may cause harm, may discriminate, and may be inscrutable. There has been less scholarly focus on a different moral fail- ing, namely algorithms' effects on agency. That is our focus here.
Consider the 2016 U.S. case of Wisconsin v. Loomis [1]. There, defendant Eric Loomis pleaded guilty to crimes related to a drive-by shooting. The trial judge or- dered a presentence investigation report (or "PSI"), which in turn used a proprietary risk assessment tool called COMPAS. This tool is designed to make better decisions in allocating resources for supervision, and the company that developed it specifically warns against using it to make sentencing decisions. Nonetheless, the trial judge used the PSI and COMPAS report in his decision to sentence Loomis in the maximum range for the crimes to which he pled guilty.
Much of the literature about the use of algorithms recognizes that such uses of al- gorithms may discriminate by race, ethnicity, and gender and that because the algo- rithms are proprietary defendants cannot scrutinize their effects. But the Loomis case also presents a puzzle. It is plausible that, even though he received a lengthy prison sentence, Loomis was not harmed at all. That is, it is plausible that Loomis received exactly the sentence he would have received had the trial judge never ordered the PSI.
Moreover, because Loomis is white, and the algorithm appears to disadvantage black defendants, he likely did not experience discrimination on the basis of race or ethnic- ity. Nonetheless, Loomis may have been wronged (but not harmed). And it is Loomis that was wronged, not (only) others who suffer discrimination from the use of algo- rithms. But how so?
The wrong consists not of discrimination or of excessive sentence (at least not on the basis of the algorithm), but of a procedural wrong. Loomis—like anyone facing the criminal justice system—has a claim to understand the mechanisms by which he is subjected to incarceration and to use that understanding to make his case. Denying him that understanding is not a harm in itself (though it may or may not result in a harm), but a failure of respect for him as a person. There are, of course, numerous calls for algorithmic transparency. However, absent an explanation for why trans- parency matters, the criticism is not well-grounded.
Our contention in this paper is that many algorithmic systems are similar to the Loomis case in that they engender wrongs that cannot be reduced to harms. More specifically, we will argue that a complete picture of the moral salience of algorithmic systems requires understanding algorithms as they relate to issues of agency, auton- omy, and respect for persons.
2 Six conflicts between algorithms and agency
First, algorithmic systems may govern behavior or create rules that are not of the sort that any agent is capable of reasonably following. Cathy O'Neil provides the ex- ample of a school system using algorithms to evaluate (and fire) teachers, despite their model's inability to distinguish the effects of good (bad) teaching from back- ground noise [2]. The moral upshot (in addition to failing to do anything good for the schools) is that teachers are evaluated according to criteria that no reasonable agent could agree to—and this is true even for those teachers not fired [3].
Of course it cannot be the case that any algorithm that causes harm—i.e., makes someone worse off than they would have other been—is unreasonable. After all, peo- ple can be made worse off for justifiable reasons. Genuinely ineffective teachers might be made worse off by not having their contracts renewed, and that could justi- fied (assuming there are no other factors that demand renewal). What seems to matter is a combination of the seriousness of the harm (losing one’s job is a very serious harm), the trustworthiness of the algorithm in the decision (in O’Neil’s account, the algorithm appears quite unreliable), and whether one is able to control the outcome (a key problem in the teaching case is that background noise—outside of teachers’ con- trol) accounted for much of the evaluation. Having one’s livelihood be determined on the basis of an unreliable system in which one cannot exercise substantial control is a state of affairs to which one cannot reasonably agree. Where people do agree, it may be evidence of deeper injustices yet.
Second, is the issue of epistemic agency. For a person to have a reasonable degree of agency requires that they know where they stand, regardless of whether they can take action. The basis for this claim is the idea that people are reasoning beings, who plan, their lives, and who think of themselves as members of a broader community.
And where we stand in relation to other people and (more importantly) in relation to institutions that exercise power over us, matters to us. So, denying a person the ability to understand the reasons why they are treated as they are is a failure of respect for them as agents. This we can see in the Loomis case. The COMPAS algorithm is pro- 10 Alan Rubel, Clinton Castro and Adam Pham
prietary. Although Loomis (or anyone) can find out some basic information about the algorithm (e.g., its inputs), no one can access the algorithm itself. To the extent that it matters, then Loomis’s lack of access prevents him from understanding the basis for how he is treated by the state of Wisconsin.
Third, algorithmic systems can leave individuals with no recourse to exercise what we will call "second-order power" (or appeals) over outcomes. That is, where algo- rithms are opaque, proprietary, or inscrutable, individuals cannot draw on their epis- temic agency (described above) to take action and appeal decisions on the basis of reasons precisely because they are prevented from understanding the underlying rea- sons for the decision. In Loomis this problem appears as a failure of due process in a criminal case—Loomis cannot explain to the court why the COMPAS score is not (if it is not) appropriate for his case. But the issue arises in other contexts as well, for ex- ample in consumer scoring by credit agencies [4].
Fourth, algorithmic systems can fail to respect boundaries between persons. Algo- rithms can be used to make decisions about workers, such as when employers use al- gorithms to schedule employees in ways that frustrate their need for reasonable work hours, or about students, such as when advising systems nudge students towards ma- jors based on anticipated success. By necessity, algorithms generalize about individu- als, but doing so treats them as undifferentiated collections of work or credit hours.
This treatment may fail to account for important aspects of their individual needs as persons for reasonable work hours or course schedules that are a good intellectual fit.
A fifth issue concerns not the agency of those who are affected by algorithms but by those who deploy them. Using algorithms to make decisions allows a person or persons to distance themselves from morally suspect actions by attributing the deci- sion to the algorithm. The issue here is that invoking the complexity or automated na- ture of an algorithm to explain why the suspect action occurred allows a party to im- ply that the action is unintended and something for which they are not responsible.
So, for example, in late 2017 the news organization ProPublica discovered that Face- book's system for tracking user interests and selling advertisements based on those in- terests allowed others to purchase ads targeting Facebook based on anti-Semitism [5].
Facebook's response was not to admit that there was a wrong or that they perpetrated a wrong. Rather, it was to point to the fact that the user categories were generated au - tomatically, that the odious ones were only rarely used, and that "[w]e never intended or anticipated this functionality being used this way" [6]. This response serves to mask Facebook's agency in generating categories that can be misused by others using the Facebook platform.
Lastly, there's a bigger and deeper issue about the very nature of agency, autonomy, and individual liberty. In the recent philosophical literature on liberty and freedom, one important thread pushes back against notions of negative and positive liberty (roughly the freedom from constraint and the capability of acting to further one's in- terests, respectively) [7]. This view maintains that a person's freedom is a function of the quality of their agency, or that their desires, values, and interests are their own.
The recent literature on filter bubbles, fake news, and highly tailored advertising in social media suggests that algorithms are being extensively (and increasingly) used to manipulate persons' choice architectures to generate understandings, beliefs, and mo- Algorithms, Bias, and the Importance of Agency 11
tivations that are not persons' own (in some cases, and to some extent). In other words, the concern about filter bubbles, fake news, and tailored advertising is not merely that bad consequences will result (perhaps so, perhaps not). Rather, it is that they diminish quality of agency and, hence, freedom properly understood.
To be clear, concerns about algorithms are many and varied. They include harm and they include discrimination. But we cannot fully understand the moral salience of algorithms without addressing their effects on agency, autonomy, and respect for per- sons.
3 Discussion
After the presentation of the longer paper based on this abstract at the BIAS work- shop, participants raised a number of important points that are worth addressing here.
The first key question is whether criticizing the Loomis case on the grounds that an inscrutable algorithm played a role in the sentencing decision proves too much. After all, had the judge in the case simply issued a sentence for the charges to which Loomis pleaded guilty, the ultimate basis for the decision is also inscrutable. The judge can offer reasons—prior convictions, seriousness of charges and “read-in”
charges, prior incidents of violation, but those are simply inputs. We likewise know the inputs to COMPAS. In both the algorithm case and in the judge-only case, there is at root an inscrutable process. In the COMPAS case it’s the proprietary algorithm, in the judge case it is the psychology of the judge.
Our answer is two-fold. First, it is true that in some sense our argument is not unique to algorithmic decision systems. However, the fact that other inscrutable sys- tems may conflict with agency, autonomy, and respect for persons neither diminishes the concern with respect to algorithms nor treats non-algorithmic systems differently.
That is, if our arguments point to ways in which decisions by judges, or administrative agencies, or bureaucracies, are problematic then we should examine those systems more carefully with the concerns in mind.
Our second response is linked to our argument about agency laundering. Although (for example) a judge’s psychology is opaque much as COMPAS is, there is at least one important difference between human and algorithmic decision-makers. Specifi- cally, human decision-makers can be culpable. Machines, no matter how good their AI, can be causally effective in bringing about outcomes, but they cannot be morally responsible. That is, they are not agents. If COMPAS or another algorithm “gets it wrong” in some case or other, the moral responsibility for getting it wrong falls to the humans and groups of humans that developed and deployed the algorithm. The same is not true for a judge (or other human decision-maker). Had a judge come to a sen- tencing decision in the Loomis case without using COMPAS, she would be account- able to that decision. And the problem of algorithmic decisions is, in part, that their use can launder such exercises of agency. Hence, while inscrutability is a concern that is not unique to algorithms, there are key differences with respect to human decision- makers.
12 Alan Rubel, Clinton Castro and Adam Pham
Another participant asked about what should done about the use of COMPAS and similar risk assessment systems being put to use in criminal justice contexts. Unfortu- nately, we do not have a comprehensive answer. But there are a couple of considera- tions worth thinking through. One is whether the arguments we make get some of their force from the severe penalties and high incarceration rates in the United States’
criminal justice system. The unreasonableness (our first argument) of algorithmic de- cision systems, as we argue, is partly a function of their stakes. Where the stakes are higher, the ability of agents to reasonably abide a system’s decisions diminishes. So, one possibility may be wrapped up in criminal justice reform. Another possibility is that use of such systems demands that agents using the systems recognize that the sys- tems are tools, and only humans can make agential decisions.
A last comment addresses the nature and extent and agency laundering. For exam- ple, is the mere reliance on a system to make decisions enough to make it agency laundering [8]. We are currently developing a broader framework for understanding agency laundering, and our full account is beyond the scope of this extended abstract.
However, our sense is that mere reliance or delegation is not enough. Rather, an agent has to ascribe some morally relevant quality to the algorithm (neutrality, accuracy, functionality) such that the agent’s own role in a decision becomes obscured.
References
1. Wisconsin v. Loomis, 881 N.W.2d 749, 371 Wis. 2d 235 (Wis. 2016).
2. O’Neil, C., Weapons of Math Destruction: How Big Data Increases Inequality and Threat- ens Democracy, pp. 3-10 . Crown, New York (2016).
3. As various political philosophers have noted, a key element of a liberal democratic soci- ety--and for institutions within such societies--is that members will only reasonably partic- ipate where they can abide fair terms of social cooperation, provided others do as well.
See, e.g., Rawls, J., A Theory of Justice. Harvard University Press, Cambridge, MA (1971).
4. Pasquale, F. The Black Box Society: The Secret Algorithms That Control Money and In- formation, pp. 32-34. Harvard University Press, Cambridge, MA (2015).
5. Angwin, J., Carner, M., and Tobin, A. Facebook Enabled Advertisers to Reach ‘Jew Haters’. ProPublica, Sept. 14, 2017. https://www.propublica.org/article/facebook-enabled- advertisers-to-reach-jew-haters, last accessed 2017/12/30.
6. Sanberg, S. Facebook post, September 20, 2017.
https://www.facebook.com/sheryl/posts/10159255449515177, last accessed 2017/12/30.
7. Christman, J. Liberalism and Individual Positive Freedom. Ethics 101(2) (1991).
8. Thanks to Eric Meyers for this point.
Algorithms, Bias, and the Importance of Agency 13
Re-Considering Bias: What Could Bringing Gender Studies and Computing Together Teach Us About Bias in
Information Systems?
Claude Draude1, Goda Klumbyte2, Pat Treusch3
1 University of Kassel, Pfannkuchstraβe 1, 3412 Kassel, Germany claude.draude@uni-kassel.de
2 University of Kassel, Pfannkuchstraβe 1, 3412 Kassel, Germany goda.klumbyte@uni-kassel.de
3 TU Berlin, Marchstraβe 23, 10587 Berlin, Germany p.treusch@tu-berlin.de
Abstract. This contribution focuses on interrogating the definition of bias in in- formation systems. It serves as a discussion starter and asks what we could learn from approaches to bias analysis in the field of gender studies and how the methodologies developed in gender studies can be beneficial to understanding, analysing and managing bias in information systems. We look at two specific theories originating in gender and feminist science and technology studies –
“situated knowledges” (Haraway) and “strong/weak objectivity” and “stand- point theory” (Harding). Specific parameters of gender-related bias are taken in- to account: androcentrism, over/underestimation of gender differences, stereo- typing of gender traits and emphasizing dichotomies through research design.
Through these we show how the above-mentioned theoretical framework can be applied to develop a better understanding of the workings of bias in information systems. The paper closes with pointing out the possibilities of a societally shared accountability for biased systems.
Keywords: Bias, Information Systems, Situated Knowledges, Standpoint Theo- ry, Strong/Weak Objectivity, Gender.
As digitalization and the use of complex information systems as well as algorithmic tools gain speed, the question of bias in data, information and computational process- es becomes ever more present both in academic and public discourse. Such bias can articulate in different forms, for instance, through the seemingly neutral prediction of crime rates as performed by ADM (algorithmic decision making [1]), or through software based on language processing and analysis, as for instance in the case of software trained on Google News that completed the sentence “Man is to computer programmer as woman is to X” with “homemaker.”
The knowledge produced through these systems and tools – be it “prediction,”
analysis, pattern recognition, or other forms of output – is called into question espe- cially when it turns out to favour certain social groups (such as favouring white
against black populations in recidivism prediction) or when it reproduces stereotypes in social relations that are considered unfair or unethical (such as racial and economic discrimination patterns that are observable in predictive policing systems or sexual discrimination that until recently was observable in dominant search engines). The fundamental connection between algorithmic calculations and social relations has lately become more transparent through the work of critical math scholars such as O’Neil. She not only problematizes the role of computer-aided decision making, but also highlights the importance of choices that the developers make: “The math- powered applications powering the data economy were based on choices made by fallible human beings” [2]. While these choices became “opaque,” the effects of the
“encoded human prejudice, misunderstanding, and bias into the software systems” [2]
result in an increase of inequality. In this regard, the effect of producing inequalities is grounded in the mathematical models themselves, making it critically important to investigate the biases, but also the neutrality and objectivity of mathematical models.
Along these lines, however, it is rarely asked what exactly constitutes bias, how bias works and relatedly, what it would mean to generate non-biased or bias-free knowledge through computational systems. How does bias form? And: where to lo- cate the bias – as part of the developer, as part of the information system design or as part of the algorithm? This also includes posing questions of accountability for a cer- tain bias. What are the criteria for something to be considered biased? Is it even pos- sible to produce un-biased knowledge through technological means?
Gender studies as an academic field deals not only with relations among genders, but critically reflects on systems of classification as such (man/woman, nature/culture, human/animal, – to name a few) and asks how these systems (re-)produce social ine- qualities. The approaches developed in gender studies highlight that social categories intersect. Furthermore, gender studies analyse how these intersections influence the way objectivity and knowledge production are understood and carried out. In this regard, to “[b]reak[...] with prejudices and reconstruct[...] the object of research re- quires a different way of seeing, in the light of which common‐sense knowledge is reconstructed as a form of bias” [3]. As Agre [4] noted in his germinal work on criti- cal technical praxis in building AI systems, in technical fields the concepts – such as
“information,” “intelligence,” “knowledge” – are used both in specific mathematic, formalised terms, as well as in more colloquial terms. Thus, bias in information sys- tems has a double connotation. It can be viewed in technical terms, but it is also im- portant to interrogate in what ways the colloquial, everyday bias (and biased assump- tions) play a role in information systems development. Gender studies deliver the analytical tools as well as the conceptual framework for acknowledging both – the double/simultaneous meaning of concepts (formalised and colloquial) as methods of reconstructing how social relations, including bias, are encoded into mathematical models.
To interrogate the intersection between different notions of bias and information systems we rely on two theories that originated in gender and feminist science and technology studies, namely “situated knowledges” (Haraway [5]) and “strong/weak Re-Considering Bias: Bringing Gender Studies and Computing Together 15
objectivity” plus “standpoint theory” (Harding [6, 7]). In a nutshell, the theory of situated knowledges draws attention to how knowledge production cannot be cut off from the social and material positionality of a researcher, including their historical, conceptual, cultural, social, etc. context. According to Haraway, there is no “view from nowhere.” Thus, scientific claims are not universal. Instead, we need to re-think scientific knowledge production as valid from a specific perspective or position that operates always within certain figurations of time, space and artefacts; that is, situated knowledges. Harding’s theory of strong/weak objectivity and standpoint theory sug- gests that people involved in knowledge production must be attentive to relations of power that knowledge is always implicated in (whose perspective are we looking from? Who benefits from this perspective? Who loses?). We should, according to both theorists, aim not at “neutral” objectivity but rather at a kind of partial objectivity that acknowledges its perspective and is open about the benefits it produces to par- ticular groups (while possibly excluding others).
These theoretical approaches can lead to, first, a better understanding of the work- ings of bias in information systems and, second, open up the possibilities of a socie- tally shared accountability for biased systems. Specifically, we suggest that the per- spective of situated knowledges points to the understanding of knowledge as a prod- uct of a complex network, where human researchers, data, data structure, algorithms, and broader social, political, historical and scientific context all contribute to the spe- cific results that are produced (cf. [8]). This in turn invites to re-think bias as a com- plex phenomenon, distributed across the whole process of designing a particular in- formation system. For instance, while researchers have accepted the possibility of data being biased, a situated knowledges perspective points to the importance of inter- rogating biases in the ways data is being classified (how are the categories formed?
Which variables are being selected as important?) and processed (which kind of algo- rithms are built and used? How are different variables weighted in the process?), as well as to biases that occur in research design itself. In addition, this also means that bias is not a constant value, but rather that it can also change in relation to the catego- ries formed, the variables selected and the ways in which data is processed.
As a starting point, gender studies provide specific parameters that can be used raising awareness towards gender-related bias in information system development, such as
• androcentrism (un/conscious focussing on masculine/male perspective);
• over/underestimation of gender differences (gender differences are either paid too much attention where they would not generally play a big role, or they are left un- noticed where such attention is due);
• stereotyping of gender traits (ascribing certain values/expectations/character to different genders, often reproducing prejudices that are well established in society);
• emphasizing dichotomies (focussing on showcasing differences between genders where such differences are not of major significance).
16 Claude Draude, Goda Klumbyte and Pat Treusch
Since gender is always intersectional, a more exhaustive list of parameters needs to include stereotyping of racial traits and/or bias in regard to sexual orientation, reli- gion, age, socioeconomic status and dis_ability.
Furthermore, relying on Harding’s notion of strong/weak objectivity and stand- point theory, we argue that to develop better accountability standards and reduce bias- es occurring through the development process, attention should be paid to the purpose and the expected results of a particular information system. One way of doing that would be by developers and researchers paying a closer look at the concepts that are used to describe what information systems do, and how the meaning of those concepts shifts in different contexts. For example, the notion of “prediction” in algorithmic systems for developers might mean that the system analyses data and discovers pat- terns that express strong correlation. However, once such a system is used for the purposes of drawing policy suggestions (as in “predictive policing,” for instance), the notion of “prediction” acquires a different, more colloquial meaning, thus affecting the expectations of the user and the (mis)understanding of what kind of knowledge the system generates.
A more “distributed” understanding of bias as occurring throughout the process af- fects also where the accountability for bias should be located. Nissenbaum [9] sug- gests that responsibility in computerised society needs to be re-defined in a more dis- persed and nuanced way since the ownership of blame does not follow a clear, linear path but is instead more scattered through a network of actors. One recent example illustrates the need for such nuanced sensibility: in March 2016 Microsoft released the AI chatbot named Tay. However, “[t]he 19-year-old female chatbot was promptly co- opted by a series of internet trolls and within 24 hours became a neo-Nazi mouthpiece for racist and sexist epithets” [10]. How much responsibility for this biased outcome of Tay’s performance should be ascribed to software developers? How much are soci- ety and the users to blame? Tay is just one example that displays how strongly the question of bias in information systems relates to the question of accountability for how they are brought into use.
To sum up, this position paper is meant as a discussion starter and inspiration for further research into a more ethical and socially just information systems design by tapping into the interdisciplinary potential of gender studies. In short, we provided insight into gender studies approaches on knowledge production and asked how these approaches could be useful in accounting for bias in information systems. Under- standing knowledge production – and the occurrence of bias – as a complex, embed- ded phenomenon and interrogating not only the processes but also the purposes and expectations of building information systems, helps understand accountability as a distributed process.
Re-Considering Bias: Bringing Gender Studies and Computing Together 17
References
1. ADM Manifest, https://algorithmwatch.org/de/das-adm-manifest-the-adm-manifesto, last accessed 2018/01/19.
2. O’Neil, C.: Weapons of Math Destruction. How Big Data Increases Inequality and Threat- ens Democracy. Penguin Random House, London (2016): 3.
3. Oakley, A.: Experiments in Knowing: Gender and Method in the Social Sciences. Polity Press, Cambridge (2000).
4. Agre, P. E.: Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI. In: Bowker, G., Gasser, L., Star, L., Turner, B. (eds.): Bridging the Great Divide: So- cial Science, Technical Systems, and Cooperative Work. Erlbaum, New York (1997).
5. Haraway, D.: Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Feminist Studies 14(3), 575–599 (1988).
6. Harding, S: Objectivity and Diversity: Another Logic of Scientific Research. The Univer- sity of Chicago Press, Chicago (2015).
7. Harding, S.: Whose Science? Whose Knowledge? Thinking from Women’s Lives. Cornell University Press, Ithaca (1991).
8. Akrich, M.: The De-Scription of Technical Objects. In: Bijker, W. E., Law, J. (eds.): Shap- ing Technology/Building Society. Studies in Sociotechnical Change, pp. 205-224. Cam- bridge, Mass: MIT Press (1992).
9. Nissenbaum, H.: Accountability in a Computerized Society. In: Friedman, B. (ed.): Human Values and the Design of Computer Technology, pp. 41-64. CSLI Publications/Cambridge University Press, New York, Melbourne (1997).
10. Montalvo, F. L.: Debugging Bias. Busting the Myth of Neutral Technology. bitch 17, 36- 40 (2016).
18 Claude Draude, Goda Klumbyte and Pat Treusch
Towards Bias Detection in Online Text Corpora
Christoph Hube1, Robert J¨aschke2 and Besnik Fetahu1
1 L3S Research Center, Leibniz Universit¨at Hannover, Germany {hube, fetahu}@L3S.de
2 Institut f¨ur Bibliotheks- und Informationswissenschaft, Humboldt University Berlin, Germany
robert.jaeschke@hu-berlin.de
Abstract. Natural language textual corpora depending on theirgenre, often contain bias which reflect the point of view towards a subject of the original content creator. Even for sources like Wikipedia, a colla- boratively created encyclopedia, which follows aNeutral Point of View (NPOV) policy, the pages therein are prone to such violations, this due to either: (i) Wikipedia contributors not being aware of NPOV policies or (ii) intentional push towards specific points of views. We present an approach for identifying bias words in online textual corpora using se- mantic relations of word vectors created through word2Vec. The bias word lists created by our approach help on identifying biased language in online texts.
1 Introduction
To enforce neutrality and quality of the provided information, Wikipedia has established several guidelines and policies.Neutral Point of Viewpolicy demands Wikipedia editors to put aside their personal opinions on a topic and create objective content. Even for information sources that allow opinions or where opinions are part of the sources’ agenda (e.g. many news websites) it is helpful for the readers to understand the intrinsic bias of sources. Especially in the context offilter bubbles andecho chambers [1,7], bias detection plays an important role.
In this work we aim to detect automatically the use of explicit language bias, i.e. bias that is introduced through specific words and phrases. Language bias stands in contrast to more implicit bias that is introduced throughgatekeeping or coverage of specific topics [5]. As an example of language bias consider the following two sentences:
– “Barack Obama served as president of the United States.”
– “Barack Obamaunsuccessfullyserved as president of the United States.”
The first sentence follows a neutral point of view. In the second sentence bias is introduced by adding the word unsuccessfully.
We call words that typically introduce bias to a statement or are a strong indicator of bias in a statement bias words. Bias words can be grammatically
diverse with existing examples across nouns, verbs, adjectives, adverbs and more, and furthermore they may vary based on the context they occur. In this paper we present an approach for identifying bias words in online text corpora using the semantic relations of word vectors created through word embedding approaches likeword2Vec[2]. The resulting bias words can be used for bias detection in text.
Recasens et al. [4] tackle the language bias problem where they identify the most biased word in a sentence already knowing that the sentence is biased.
To do so, they rely on language features such as lists of factive, assertive and implicative verbs, and additionally make use of a bias lexicon extracted from a subset of Wikipedia revisions, in which the editor mentions the abbreviation POV (Point of View). In contrast to Recasens et al. [4], our approach differs in that we provide a comprehensive list of bias words with nearly∼10,000 words, and in that we make use of word embeddings, which capture semantics and syntactic relationships between words, to extract words that may indicate bias.
In Section 2 we introduce a semi-automated approach for seed word ex- traction (Section 2.1) and a fully automatic approach for extracting bias words given a set of seed words and a fitting text corpus, e.g. the latest Wikipedia corpus (Section 2.2).
2 Approach
In this section, we describe in detail the two main steps of our approach: (1) seed word extraction, and (2) bias word extraction.
2.1 Seed Word Extraction
Through empirical observations, we see that bias words often co-occur with other bias words in the word vector space. In order to identify these bias word clusters, we first need to extract a small number of bias words that we can use as seeds for our approach. The idea is to use word vectors that already have a high density of bias words since it will make the manual identification of bias words faster.
Therefore we use a corpus from which we expect to have a high density of bias words compared to Wikipedia.
Conservapedia1is a Wiki shaped according to right-conservative ideas inclu- ding strong criticism and attacks especially on liberal politics and members of the Democratic Party of the United States. Since no public datset is available we crawled all Conservapedia articles under the categorypolitics (and all subcatego- ries). The crawled dataset comprises of a total of 11,793 articles. We preprocess the data using a Wiki Markup Cleaner. We also replace all numbers with their respective written out words, remove all punctuation and replace capital letters with small letters. In the next step we use word2Vec to create word embeddings based on the Conservapedia dataset.
To achieve a high density of bias words, we explicitly pick words that are associated with a strong political split between left and right in the US (e.g.
1 http://www.conservapedia.com
20 Christoph Hube, Besnik Fetahu and Robert J¨aschke
media, immigrants, abortion) for the seed word extraction. We leave for future work to automate the process of seed word extraction, where approaches like [3]
can serve as a starting point, however, its use can be limited since clean and explicit labels (bi-partisan or more POVs) of the textual corpora is required, a task deemed to be very difficult considering the broad coverage in encyclopedias like Wikipedia.
For each word we then manually go through the list of closest words in the vector space using cosine similarity and extract words that seem to convey a strong opinion. For example among the 100 closest words for the wordmedia in the vector space we find words such asarrogance,whining,despises andblatant.
We merge all extracted words into one list. The final seed list contains 100 bias words.
2.2 Bias Word Extraction
Given the list of seed words, we extract a larger number of bias words using the Wikipedia dataset of latest articles2. We preprocess the dataset in the same way as we preprocessed the self-crawled Conservapedia dataset and create a word vector space using word2Vec. First, we split the seed word list randomly into n= 10 batches of equal size. In the next step we use the semantic relations of word vectors created to identify clusters of bias words. For each batch of seed words we sum up the word vectors of each word in the batch. Next, we extract the closest words according to the cosine similarity of the combined vector. By using the combined vector of multiple seed words we increase the probability of extracting bias words compared to the use of only one seed word. Table 1 shows an example of the top 20 closest words for the single seed word indoctrinate and a batch containing indoctrinate and 9 other seed words. Our observations suggest that the use of batches of seed words leads to bias word lists of higher quality.
We use the extracted bias words as new seed words to extract more bias words using the same procedure. Table 2 shows statistics for our extracted list of bias words. The list contains 9742 words with 42% of them tagged as nouns, 24% tagged as verbs, 22% tagged as adjectives and 10% tagged as adverbs. The high number of nouns is not surprising since nouns are the most common part of speech in the English language. To annotate the words with their part of speech we use the POS tagger[6]. We provide the final bias word list at the paper URL3.
3 Conclusion and Future Work
We introduced a new approach for extracting bias words using word2Vec from textual corpora like Wikipedia. We are planning to integrate bias word lists among other features into a machine learning classifier for bias detection. For a
2 https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages- articles.xml.bz2
3 https://git.l3s.uni-hannover.de/hube/Bias_Word_Lists
Towards Bias Detection in Online Text Corpora 21
Table 1.Top 20 closest words for the single seed wordindoctrinateand the batch con- taining the seed words: indoctrinate, resentment, defying, irreligious, renounce, slurs, ridiculing, disgust, annoyance, misguided
Rank Single seed word Batch of seed words
1 cajole hypocritical
2 emigrates indifference
3 ingratiate ardently
4 endear professing
5 abscond homophobic
6 americanize mocking
7 reenlist complacent
8 overawe recant
9 disobey hatred
10 reconnoiter vilify
11 outmaneuver scorn
12 helmswoman downplaying
13 outflank discrediting
14 renditioned demeaning
15 redeploy prejudices
16 seregil humiliate
17 unnerve determinedly
18 titzikan frustration
19 unbeknown ridicule
20 terrorise disrespect
Table 2.Statistics about the extracted bias word list POS tag # ratio
nouns 4101 (42%) verbs 2376 (24%) adjectives 2172 (22%) adverbs 997 (10%) others 96 (1%) total 9742
proper evaluation we will use crowdsourcing to generate a ground truth of biased and non-biased statements from both Wikipedia and Conservapedia.
Acknowledgments This work is funded by the ERC Advanced Grant ALEXAN- DRIA (grant no. 339233), DESIR (grant no. 31081), and H2020 AFEL project (grant no. 687916).
22 Christoph Hube, Besnik Fetahu and Robert J¨aschke
References
1. R. K. Garrett. Echo chambers online?: Politically motivated selective exposure among internet news users. Journal of Computer-Mediated Communication, 14(2), 2009.
2. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed repre- sentations of words and phrases and their compositionality. InAdvances in neural information processing systems, pages 3111–3119, 2013.
3. B. L. Monroe, M. P. Colaresi, and K. M. Quinn. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403, 2008.
4. M. Recasens, C. Danescu-Niculescu-Mizil, and D. Jurafsky. Linguistic models for analyzing and detecting biased language. InACL (1), pages 1650–1659, 2013.
5. D. Saez-Trumper, C. Castillo, and M. Lalmas. Social media news communities:
gatekeeping, coverage, and statement bias. In22nd CIKM. ACM, 2013.
6. K. Toutanova and C. D. Manning. Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. InProceedings of the 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora: Held in Conjunction with the 38th Annual Meeting of the Association for Computational Linguistics - Volume 13, EMNLP ’00, pages 63–70, Stroudsburg, PA, USA, 2000. Association for Computational Linguistics.
7. V. Vydiswaran, C. Zhai, D. Roth, and P. Pirolli. Biastrust: Teaching biased users about controversial topics. In21st CIKM. ACM, 2012.
Towards Bias Detection in Online Text Corpora 23
Dealing with Bias via Data
Augmentation in Supervised Learning Scenarios
Vasileios Iosifidis1 and Eirini Ntoutsi1 L3S Research Center, University of Hannover, Germany
{iosifidis, ntoutsi}@l3s.de
Abstract. There is an increasing amount of work from different communities in data mining, machine learning, information retrieval, semantic web, and databases on bias discovery and discrimination-aware learning with the goal of developing not only good quality models but also models that account for fairness. In this work, we focus on supervised learning where biases towards certain attributes like race or gender might exist. We propose data augmentation techniques to correct for bias at the input/data layer. Our experiments with real world datasets show the potential of augmentation techniques for dealing with bias.
1 Introduction
Nowadays, decision making systems tend to become fully automated by replacing human judgment with algorithmic decisions that rely solely or to a great extent on data. Since decision making systems are data-driven, they can be applied in a wide variety of applications from recommendation systems and insurance ratings to medical diagnoses and decisions on whether a patient should receive treatment or not.
However, concerns have been raised as these algorithms may under-perform if trained on pre-existing biases which lay inside data distributions. These concerns led to anti- discrimination laws which try to prevent different treatment of individuals or groups based on specific attributes (e.g ethnicity, gender), also named protected attributes.
Even without considering protected attributes in the learning process, algorithms can still be unfair towards specific individuals or groups. The reason can be explained by analyzing the data: in some cases, particular attributes, calledproxies, can reveal the value of a protected attribute (e.g., attribute “wife” or “husband” can reveal the protected attribute “gender”).
One of the main reasons which causes discrimination in the classification process is the under-representation of protected groups. For instance, medical treatment data may lack observations of a specific disease misjudging ill patients as healthy. Under-represented groups tend to be highly misclassified compared to over-represented groups. In this work, we focus on improving the correctly classified instances of a protected group without degrading the overall classification performance. To this end, we propose data augmentation techniques to increase the representation of the (minority) protected group.
The rest of the paper is organized as follows: Section 2 gives an overview of the related work. Section 3 presents data augmentation techniques which have been used for dealing with class imbalance. In Section 4 we present our experimental analysis.
Conclusions are given in Section 5.
