Opinion: social credit systems currently being trialled in China use algorithms and data-mining to decide an individual's social status

Imagine if you had to show your passport or identity card instead of using your travel pass to board the train or bus. Imagine also if, depending on your behaviour over the past few weeks, your identity card would "tell" the automatic reader whether you deserved admission or not. If you have been naughty, then your permission to use public transport is denied. If you have been good and nice then you will be admitted and you can commute to work or visit the hospital as planned.

This may sound odd but such a social credit system is already being trialled in China. The idea behind social credit systems is to assign "trust" indicators to individuals by using autonomous algorithms to mine data and collate information.

Trust in the state?

Based on past activities and public reviews, an algorithm decides an individual’s social status. Collating data through largely innocuous means can serve as an indicator for how "deserving" or "undeserving" an individual might be and whether you can use public transport or even gain access to public education or public health services.

From RTÉ 2fm's Dave Fanning Show, The Irish Times' Clifford Coonan reports on the social credit rating system in China

While this grants autonomous algorithms worrying degrees of power and control over daily lives, the clear attraction for China in pursuing such a social credit system is to improve reliability and quality across a marketplace prone to food scandals, poor services and questionable goods. By increasing trust and quality, the business and tourist environment can be overhauled, regulated and improved.

We would be right to question the ethical implications. Who writes the code? Who can use the information and for what purposes? What happens to personal trust and physical interactions if our trustworthiness is managed by algorithms?

For some, it is a foolproof system afforded by a neutral judge vis a vis a few lines of unbiased code. But code is never unbiased and reflects particular design choices by particular groups of people. For others, it is a step too far by a state obsessed with social control.

From RTÉ Radio 1's The Business, The Irish Independent's Adrian Weckler discusses Apple's new facial recognition technology

Yet think of Ireland. The foundations for these systems are already in place, but a small liberal democratic government is not in control of the code. The code lies in the hands of private companies, often non-Irish and/or transnational entities, answerable to demands of the market or whims of the owner. If you ever bought items on Ebay, or used Airbnb or an Uber, you will already be familiar with the online rating of products and services. 

While social credit systems may not yet be a commonplace phenomenon in countries like Ireland, New Zealand, or Australia, we are slowly adopting and accepting key features of such a system almost by stealth. If, for example, you buy an item on Ebay you can rate the seller. As reviews grow, this amounts to an indicator of the seller’s trustworthiness. The lower the rating the lower their product will appear on Ebay’s product list. Good reviews reward good sellers and, similarly, the seller can rate you as a buyer.

For platforms such as Ebay, this peer-review system answers a problem of how to maintain and regulate quality. Many online platforms and services use such a scoring mechanism as a check against poor quality products and services that may otherwise break trust in the host platform. Taxi hire platform Uber similarly allows reviews for both the driver and the user, as does Amazon, Airbnb and even certain dating sites.

Looking to the future, we need to articulate a new method for engaging technological innovation

Now imagine a system where all these platforms plug in together so that you have one rating across multiple platforms. If you receive a bad rating on Ebay, it affects your rating on Uber and vice versa. Imagine your identity card links you to all these private services as well as to public services, health, transport, welfare.

Trust in technology

The tendency in technological innovation is to increase efficiency by merging different processes and services through technical capability. However, the parallel sacrifice is an obfuscation of agency and control and lines of responsibility and accountability become murky. Do we trust private companies to generate code that has societal interests at heart or might they only seek better control of their margins, their shareholders, their own trust indicators?

We are living in an age of transition and the hallmark of this transition is our increased use of technology in everyday life. The most powerful force in contemporary society is found through mundane social and ecological adaptation in everyday life rather than in the scientific laboratory of momentous discovery. Incrementally, technology becomes culturally embedded so that it is already mundane at its most potent. The unfamiliar – concealed by familiarity – escapes scrutiny and regulation, and gains purchase in our daily life.

We cannot, and perhaps should not, repeal the great luxuries that technology affords us. But looking to the future, we need to articulate a new method for engaging technological innovation. A method that brings the user and the elected politician into the design process, not merely as objects of user experience research but as active participants in democratic design architectures with the social good at its heart.

The views expressed here are those of the author and do not represent or reflect the views of RTÉ