A very interesting article from La Quadrature du Net on the CAF algorithm used to "predict" who is likely or not to cheat among the beneficiaries (!).
Thus, CAF employees are no longer subject to moods when they have to deal with fraud: it is the algorithm that tells them who they should target.
Except that, as La Quadrature du Net rightly points out, the algorithm only processes data according to the program with which it was designed by human beings. So there is no such thing as a neutral, impartial, objective algorithm....
“This algorithm is interesting from this point of view since it was trained 'in the rules of the art', see the references above, starting from a database resulting from random checks. there is therefore no sampling bias a priori, as in the case of facial recognition algorithms. That being said, the algorithm repeats the human biases linked to the checks carried out on these randomly selected files (severity with people on social minima, difficulty in identifying complex fraud…) But above all, as explained in the article, it reflects the complexity of the rules for access to social benefits, which is a purely political subject that the algorithm only reveals.
La Quadrature du Net purely and simply requests the withdrawal of this discriminating algorithm from the CAF.
If you want to know more, you can contact La Quadrature du Net directly (which always does excellent, very serious work) here: contact@laquadrature.net
CAF: digital at the service of exclusion and harassment of the most precarious
Posted on October 19, 2022
For almost a year now, we have been fighting within the collective “Stop Controls” in order to oppose the effects of dematerialization and the use of digital technology by administrations for the purposes of social control. After having discussed the situation at Pôle Emploi, we are interested here in the case of the Family Allowance Funds (CAF). We will soon come back to the consequences of this fight in which we wish to fully engage in the coming months.
"Between CAF and you, there is only one click". This is what we could read on a CAF poster at the start of the year. And the subtitle leaves you dreaming: “Access to all CAF services 24 hours a day”. Vain promise of a digital facilitating access to social benefits, at any time of the day and night. Sinister slogan masking the reality of excessive computerization, a vector of calculated social exclusion.
While the generalization of online procedures is accompanied above all by a reduction in physical reception capacities, a mode of contact that is essential for people in precarious situations2, it is to an algorithm that the CAF leaves the care of predict which recipients would be “(un)trustworthy” and need to be checked3. Responsible for giving a score to each beneficiary, supposed to represent the “risk” that they benefit unduly from social assistance, this scoring algorithm serves a policy of institutional harassment of the most precarious
The Shame Algorithm
Fed with hundreds of data that CAF has on each beneficiary5, the algorithm continuously assesses their situation in order to classify and sort them, via the assignment of a score (“risk score”). This note, updated monthly, is then used by the teams of CAF controllers to select those to be subject to in-depth control6.
The little information available reveals that the algorithm deliberately discriminates against the precarious. Thus, among the elements that the algorithm associates with a high risk of abuse, and therefore negatively impacting the score of a beneficiary, we find the fact7:
– To have low income,
– To be unemployed or not to have a stable job,
– To be a single parent (80% of single parents are women)8,
– To dedicate a significant part of its income to housing,
– To have many contacts with CAF (for those who would dare to ask for help).
Other parameters such as place of residence, type of housing (social, etc.), mode of contact with CAF (telephone, email, etc.) or being born outside the European Union are used without that we do not know precisely how they affect this note9. But it is easy to imagine the fate reserved for a foreign person living in a disadvantaged suburb. This is how, since 2011, CAF has been organizing a veritable digital hunt for the most disadvantaged, the consequence of which is a massive over-control of poor people, foreigners and women raising a child alone.
Worse, CAF brags about it. Its director qualifies this algorithm as being part of a "constant and proactive policy of modernizing tools to fight against fraudsters and crooks". The institution, and its algorithm, are also regularly presented at the state level as a model to follow in the fight against "social fraud", a theme imposed by the right and the far right in the early 2000s.
How can such a profoundly discriminatory device be publicly defended, moreover by a social administration? It is here that the computerization of social control takes on a particularly dangerous character, through the technical alibi it offers to political leaders.
A technical alibi for an iniquitous policy
First of all, the use of the algorithm allows CAF to mask the social reality of the sorting organized by its control policy. Exit the references to the targeting of social minima recipients in the “annual control plans”. The latter now report “datamining targets”, without ever explaining the criteria associated with the calculation of “risk scores”. As a CAF controller said: “Today it is true that data makes things easier for us. I do not have to say that I will select 500 RSA beneficiaries. It's not me who does it, it's the system that says it! (Laughs). »12
The notion of “risk score” is also used to individualize the targeting process and deny its discriminatory nature. A CAF control officer thus declared in front of deputies that “More than populations at risk, we are talking about profiles of beneficiaries at risk, in connection with data mining”13. In other words, CAF argues that its algorithm does not target the poor as a social category but as individuals. A large part of the "risk factors" used to target recipients are, however, socio-demographic criteria associated with precarious situations (low income, unstable professional situation, etc.). This rhetorical game is therefore statistical nonsense, as the Defender of Rights reminds us:14 "More than a targeting of 'presumed risks', the practice of data mining forces the designation of populations at risk and, in doing so, leads to instil the idea that certain categories of users are more inclined to cheat”.
Finally, the use of the algorithm is used by CAF leaders to shirk responsibility for choosing the criteria for targeting the people to be controlled. They transform this choice into a purely technical problem (predicting which files are most likely to present irregularities) whose resolution is the responsibility of the institution's teams of statisticians. The only thing that counts then is the effectiveness of the proposed solution (the quality of the prediction), the internal workings of the algorithm (the targeting criteria) becoming a simple technical detail that does not concern politicians15. A director of CAF can thus say publicly: “We [CAF] do not draw up the typical profile of the fraudster. With datamining, we don't draw conclusions,” simply omitting to say that CAF delegates this task to its algorithm.
Early over-control of the most precarious
This is our response to officials who deny the political nature of this algorithm: the algorithm has only learned to detect what you have decided to target. The over-control of the most precarious is neither a coincidence nor the unexpected result of complex statistical operations. It is the result of a political choice of which you knew, even before the deployment of the algorithm, the consequences for the precarious.
This choice is as follows16. Despite CAF's communication about its new "fight against fraud" tool (see for example here, here or here), the algorithm was designed not to detect fraud, which is intentional, but indus (overpayments) in the broad sense17, the vast majority of which result from involuntary declarative errors18.
However, CAF knew that the risk of error is particularly high for people in precarious situations, due to the complexity of the rules for calculating social benefits concerning them. Thus, as early as 200619, a former director of the fight against fraud at the CAF explained that "the undus are explained […] by the complexity of the services", which is "all the more true for the services linked to precariousness (hear the social minima). He added that this is due to taking into account “numerous elements of the user’s situation which are very variable over time, and therefore very unstable”. Concerning isolated women, he already recognized the “difficulty of grasping the notion of “marital life””, a difficulty in turn generating errors.
Asking the algorithm to predict the risk of undue payment therefore amounts to asking it to learn to identify who, among the recipients, is dependent on social minima or is a victim of the conjugalization20 of social assistance. In other words, CAF officials knew, from the start of the targeting automation project, what would be the “risk profiles” that the algorithm was going to identify.
Nothing is therefore more false than to declare, as this institution did in response to the Defender of Rights' criticisms, that "the controls to be carried out" are "selected by a neutral algorithm" which obeys "no presupposition »21. Or that “the controls […] resulting from datamining […] leave no room for arbitrariness”.
Discriminate to profit
Why favor the detection of errors rather than that of fraud? Errors being more numerous and easier to detect than situations of fraud, which require the establishment of an intentional character, this makes it possible to maximize the amounts recovered from the beneficiaries and thus to increase the "yield" of controls.
To quote a former head of CAF's anti-fraud department: "We CAF, quite honestly, on these very big frauds, we can't be the leader because the stakes are beyond us, in a way." And to point out a little further on his satisfaction that in the last "objective and management agreement", a contract binding CAF to the State and defining a certain number of objectives,22 there is a "distinction between the rate recovery of undue fraud and undue non-fraud […] because the efficiency; is still more important on non-fraud industrials which, by definition, are of lesser importance”.
This algorithm is therefore only a tool used to increase the profitability of the controls carried out by CAF in order to feed a communication policy where, throughout activity reports and public communications, the harassment of the most precarious becomes a evidence of "good management" of the institution
Dehumanization and digital exposure
But digital has also profoundly changed the control itself, now turned towards the analysis of the personal data of the beneficiaries, whose right of access given to the controllers has become sprawling. Access to bank accounts, data held by energy suppliers, telephone operators, employers, traders and of course other institutions (employment center, taxes, national social security funds …)24: control has turned into a real digital stripping.
These thousands of digital traces are mobilized to feed a control where the burden of proof is reversed. Much more than the interview, personal data now forms the basis of the controllers' judgement. As a CAF controller said: “Before, the interview was very important. […] Now the control of information upstream of the interview takes on much more importance. »25. Or even, “a controller when he prepares his file, just by going to see the partner portals, before meeting the beneficiary, he has a very good idea of what he will be able to find”.
Refusing to submit to this transparency is prohibited under penalty of suspension of benefits. The “right to digital silence” does not exist: opposition to total transparency is equated with obstruction. And for the most reluctant, CAF reserves the right to request this information directly from the third parties who hold it.
The control then becomes a session of humiliation where everyone must agree to justify the smallest detail of their life, as this beneficiary testifies: “The interview […] with the CAF agent was a humiliation. He had my bank accounts in front of him and went through every line. Did I really need an Internet subscription? What had I spent these 20 euros drawn in cash on? »26.
The score assigned by the algorithm acts in particular as proof of guilt. Contrary to what the CAF wants to believe, which reminds anyone who wants to listen that the algorithm is only a "decision-making tool", a degraded risk score generates suspicion and severity during controls . It is up to the beneficiary to answer for the algorithmic judgment. To prove that the algorithm is wrong. This influence of algorithmic scoring on control teams, a recognized fact referred to as "automation bias", is even better explained here by a controller: "Given the fact that we are going to control a situation strongly scored, some told me that, well, there is a kind of – even unconsciously – not an obligation of results but to say to themselves: if I am there, it is because there is something so it is necessary that I find »
Dramatic human consequences
These practices are all the more revolting as the human consequences can be very serious. Psychological distress, loss of housing, depression28: the control leaves significant traces in the lives of all controlled. As a director of social action explains29: “You have to imagine that the undue payment is almost worse than non-recourse”. And to add: “You are in a mechanism for recovering undue payments and administrations which can also decide to cut you off all access to social benefits for a period of six months. Really, you find yourself in a black situation, that is to say that you made a mistake but you pay extremely dearly for it and this is where an extremely strong degradation situation begins which is very difficult behind to recover ” .
Requests for undue reimbursement can represent an untenable burden for people in financial difficulty, especially when they are due to errors or omissions that cover a long period. Added to this is the fact that overpayments can be recovered via deductions from all social benefits.
Worse, the numerous testimonies30 collected by the Defender of Rights and the Stop Control and Changer de Cap collectives report numerous illegal practices on the part of CAF (non-compliance with adversarial proceedings, difficulty of appeal, abusive suspension of aid, failure to provide the report investigation, no access to findings) and abusive re-qualifications of situations of involuntary error as fraud. These improper qualifications then lead to the filing of recipients identified as fraudsters31, filing reinforcing à in turn their stigmatization during future interactions with CAF and whose consequences may extend beyond this institution if this information is transferred to other administrations
Digital, bureaucracy and social control
Admittedly, digital technologies are not the root cause of CAF practices. As the “social” side of the digital control of public space by the police institution that we document in our Technopolice campaign, they are the reflection of policies centered around logics of sorting, surveillance and general administration of our lives32.
The practice of scoring that we denounce at CAF is not specific to this institution. A pioneer, the CAF was the first social administration to set up such an algorithm, it has now become the "good student", to use the words of a LREM MP33, which should inspire other administrations. Today it is thus Pôle emploi, health insurance, old-age insurance or even taxes which, under the impetus of the Court of Auditors and the National Delegation for the Fight against Fraud34, are working to develop their own scoring algorithms.
At a time when, as Vincent Dubois35 says, our social system is always tending towards "fewer social rights granted unconditionally [...] and more aid [...] conditional on individual situations", which "logically calls for more control », it seems legitimate to question the major projects for the automation of social assistance, such as that of « solidarity at the source » proposed by the President of the Republic. Because this automation can only be achieved at the cost of an ever-increasing scrutiny of the population and will require the establishment of digital infrastructures which, in turn, will confer ever more power on the State and its administrations.
Fight
Faced with this observation, we ask that the use of the scoring algorithm by CAF be put to an end. The search for undus, the vast majority of which are of the order of a few hundred euros36, can in no way justify such practices which, by their nature, have the effect of throwing precarious people into situations of immense distress.
To the remark of a CAF director saying that he could not "answer precisely as to the biases" that his algorithm could contain - thus implying that the algorithm could be improved -, we answer that the problem is not technical, but political. Since it simply cannot exist without inducing discriminatory vetting practices, it is the scoring algorithm itself that must be abandoned.
We will soon come back to the actions we want to take to fight, at our level, against these policies. Until then, we will continue to document the use of scoring algorithms in all French administrations and invite those who wish, and can, to organize and mobilize locally, like the Technopolice campaign run by La Quadrature. In Paris, you can find us and come and discuss this fight within the framework of the general meetings of the Stop Controls collective, whose press releases we relay via our website.
This fight can only benefit from exchanges with those who, at CAF or elsewhere, have information on this algorithm (the details of the criteria used, the internal dissensions that its implementation may have provoked, etc.) and want us to help combat such practices. We encourage these people to contact us at contact@laquadrature.net. You can also drop documents anonymously on our SecureDrop (see our help page here).
Finally, we would like to denounce the police surveillance to which the Stop Controls collective is subject. Making telephone contacts on the part of the intelligence services, allusions to the actions of the collective with some of its members in the context of other militant actions and over-presence of the police during simple towing operations in front of CAF agencies: as many of police measures aimed at the intimidation and repression of a social movement that is both legitimate and necessary.
Saisi
