CHAPTER TWO

Data-Driven Business Models

#surveillance capitalism  #value extraction  #harm

Data-driven business models come in many shapes and forms, and are characterised by having the processing of data as a core element. In this context we focus on data-driven business models that rely on data for creating revenue in the guise of data-driven marketing. Marketing being the broader concept that includes ‘advertising’, as discussed in Chapter 5 (marketing law).

In addition to being dependent on the processing of (large quantities of) personal data, data-driven business models offered to consumers often rely on personalisation and the extraction of value from users.¹ Often, data-driven business models comprise two separable products, where at least one product is provided free of charge.

These business models exist in both online and offline environments and usually rely on the monetisation of (1) surveillance, (2) attention and (3) behaviour modification by means of personalised marketing, including targeted advertising. The surveillance, attention and behaviour modification capabilities can be made available to third parties, including for both commercial and political purposes.

Loyalty and rewards programs are examples of data-driven business models that are offered both offline and online. This ‘service’ is ancillary to the trader’s selling of products, including goods (e.g. grocery shops) and services (e.g. airlines). The service may consist of providing discounts, bonuses and other benefits to the user. In essence, the program is a marketing technique that connects the consumer more closely to the trader, who may increase revenue and profits by better understanding their consumers—including their behaviour—and offering personalised marketing.

Social media services and other platforms are typical online examples of data-driven business models—the bazaar is an offline platform that is not data-driven. Platforms may monetise attention and data in many ways. Here we focus on revenue from personalised marketing, including targeted advertising. The ancillary product is offered to third parties (advertisers), but reasonable arguments can be made for the targeted advertising being the primary product, i.e. the users (consumers) are the real product being sold to advertisers.

1.1. Personalisation and surveillance

Personalisation is at the core of many data-driven business models, either because personalisation is required for delivering the primary product, or because it is a ‘convenient’ reason to justify the collection of personal data for the ancillary product, such as personalised marketing.

For instance, loyalty/rewards programs and social media services justify the collection and use of personal data—which includes surveillance of behaviour for one purpose—that subsequently may be used for personalised marketing.² A decision to personalise the primary product may thus be driven by a goal of increasing revenue from ancillary services, including data-driven marketing—often framed as ‘enhancing the product’ or ‘improving the user experience’.

1.2. Value extraction and behaviour modification

The idea of monetising attention is far from new,³ and the idea of the attention economy was introduced in the 1970s by Herbert A. Simon:⁴

‘in an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.’

Traders may monetise the attention of its users by displaying advertising. Advertising and other sorts of marketing techniques may be used to further commercial (products), political (parties/interests) and societal (e.g. safety) aims. Advertising constitutes a subset of behaviour modification techniques that is comprised in the broader concept of ‘marketing’.

The value of each user is directly proportionate to the amount of exposure. Advertising may be ‘targeted’ by means of personal data, or ‘contextual’ by making assumptions (possibly informed by data; personal or not) based on context. Advertising saddles in a forum for horse riders is an example of contextual advertising.

Advertising is valuable to the advertiser because it may influence the behaviour of the targeted users. The value of an advertisement may be increased by having more detailed information about the user: Advertising saddles to horse riders (outside forums for horse riders) may be more efficient than targeting motorbike riders.

The trader has an economic incentive to increase (1) the number of users, (2) their engagement (amount and nature of attention) and (3) knowledge about individual users (personal data). As we will discuss later, personal data coupled with insights from psychology and technology can be used to (a) increase the value of user experiences (through influence) and (b) increase attention and engagement, including by means of creating addictive behaviours.

We use the terms ‘behaviour modification’⁵ and ‘influence’ interchangeably and understand them to comprise both persuasion and manipulation. We use persuasion as the lawful form of influencing behaviour, and manipulation, which includes deception and coercion, as unlawful influencing.⁶ The distinction is normative and not always easy to establish, as we discuss in Chapter 5 (marketing law).

Other types of value extraction include benefiting from the users’ content creation, relationships, anxieties, psychological needs, etc. Some of these aspects are dealt with in Chapter 10 (human dignity and democracy).

1.3. ‘Free’

Often, data-driven business models rely on products that are offered free of charge, i.e. without the consumer having to make a ‘payment’. This makes it more likely that consumer’s will sign up for something like a loyalty program or a social media service. In Chapter 7 (persuasive technology), we introduce the concept of ‘friction’, which is removed when the user does not have to make a payment.

Some loyalty programs take the shape of a fee-based subscription service—like Amazon Prime—but nevertheless still rely on surveillance and behaviour modification. Other service providers offer paid-for commercial-free versions of their service.

2. Providing access to ourselves . . . and others

We leave traces when we act in social or societal contexts, whether online or offline. The most significant attributes of the traces we leave when technology is involved are the amount of data points and the (often unlimited) storage thereof. In addition to the scale, technology also allows for observing behaviour in the private sphere, as we discuss further in Chapter 7 (persuasive technology) and Chapter 10 (human dignity and democracy).

In contrast to traditional landline telephones, mobile phones are usually personal (one user per phone).⁷ With the advent of ‘smart’ (phones, watches, health devices, televisions, homes, cars, etc.), information technology has turned into a massive web of surveillance, tracking and recording devices. Also the use of cookies and other tracking techniques on the internet has allowed for the tracking of individuals across services, platforms, technologies, etc.⁸

By nudging users to access services through apps—instead of, e.g., a web-browser—the trader may obtain more data and gain more control over the user experience and, possibly, circumvent privacy measures implemented on the browser of the privacy-aware user.

Personal data may be deliberately revealed by the user or derived from behaviour, including interactions with other people. By means of algorithms, even simple data points may reveal much, as expressed by Jamie Bartlett,

‘[algorithms] can take your music preferences or your book preferences and extract from this seemingly innocent information very accurate predictions about your religiosity, leadership potential, political views, personality and so on.’⁹

Whenever we use information technology, we inevitably leave information that is necessary for the communication. But especially for the use of apps, we often provide a plethora of information that is not necessary for the communication. For illustration, when installing Facebook’s Messenger App on an iPhone, one can click through to the following explanation of data used for particular purposes (‘depending on feature use and age’):

Third-Party Advertising

• Purchases (purchase history)

• Financial Info (Other Financial Info)

• Location (Precise location, Coarse Location)

• Contact Info (Physical Address, Email Address, Name, Phone Number, Other User Contact Info)

• Contacts (Contacts)

• User Content (Photos or Videos, Gameplay Content, Other User Content)

• Search History (Search History)

• Browsing History (Browsing History)

• Identifiers (User ID, Device ID)

• Usage Data (Product Interaction, Advertising Data, Other Usage Data)

• Diagnostics (Crash Data, Performance Data, Other Diagnostic Data)

• Other Data (Other Data Types)

Developer’s [Facebook, Inc.] Advertising or Marketing, Product Personalisation, Other Purposes

[Same as under Third-Party Advertising]

Analytics [different from Third-Party Advertising in italics]

• Health & Fitness (Health, Fitness)

• Financial Info (Payment Info, Other Financial Info)

• User Content (Photos or Videos, Audio Data, Gameplay Content, Customer Support, Other User Content)

• Sensitive Info (Sensitive Info)

App Functionality [different from Third-Party Advertising in italics]

• Health & Fitness (Health, Fitness)

• Financial Info (Payment Info, Credit Info, Other Financial Info)

• User Content (Email or Text Messages, Photos or Videos, Audio Data, Gameplay Content, Customer Support, Other User Content)

• Sensitive Info (Sensitive Info)

By providing access to a number of features, including contacts, the user indirectly gives access to information about third parties that may not be using the services in question.¹⁰ Access to information about others can also be derived from access to information we share about others via platforms and other services.

We may also learn about how traders process data by announcements. For instance, in 2017 Google ceased its practice of scanning Gmail messages to sell targeted advertisements.¹¹ And Facebook announced in 2021 that it as of 19 January 2022 will remove ‘Detailed Targeting options that relate to topics people may perceive as sensitive, such as options referencing causes, organizations, or public figures that relate to health, race or ethnicity, political affiliation, religion, or sexual orientation.’¹²

On 9 September 2021 Ray-Ban and Facebook launched smart sunglasses (Ray-Ban Stories)—that resembles the abandoned Google Glass from 2011—‘that give you an authentic way to capture photos and video, share your adventures, and listen to music or take phone calls—so you can stay present with friends, family, and the world around you.’¹³ In the launch video, Mark Zuckerberg emphasises that ‘Ray-Ban Stories are an important step toward a future when phones are no longer a central part of our lives and you won’t have to choose between interacting with a device or interacting with the world around you.’ It is argued that the glasses will ‘let people live in the moment’ and ‘make people’s life better’ [by recording and sharing more data].

On 20 April 2021 Apple announced its AppTrackingTransparency feature, which allows users to choose what third-party tracking they will tolerate.¹⁴ This may be a (long-awaited) first step for markets to limit the current pervasive surveillance for marketing purposes. The fix only applies to Apple products (iOS), with roughly a quarter of the market. Google’s Android platform sits atop almost three quarters of the market for smartphone operating systems.¹⁵

2.1. Online and offline

Traders may make efforts to link activities in the online and offline worlds, e.g. by offering traceable discounts, using phone numbers as identifiers and/or involving smartphones in the off-line ‘purchase experience’. These efforts may also be disguised as benefits for registering products (extra battery, extended warranties, etc.) or joining a loyalty/rewards program, and as the encouragement to share ‘purchase experiences’ with the trader, friends or other people.

3. Artificial Intelligence (AI)

We use the term ‘Artificial Intelligence’ to include the use of big data, algorithms, machine learning and deep learning. AI systems rely on computer programs that can handle large quantities of data and are designed to achieve (optimise for) particular objectives.

AI systems are designed to interact with their surroundings, including human beings, by receiving and delivering signals. Data is the raw material used by AI systems to achieve their objectives. With the digitisation of our societies, computers have access to vast amounts of data—in terms of both scale and scope—that may be useful for behaviour modification.¹⁶

Correlations can be identified through pattern recognition to develop strategies, including in the guise of algorithms, which can be applied, tested and refined in the interaction with their surroundings. Causations are important to human intelligence and can be applied to the design of AI systems by humans. AI systems can ‘learn’ from patterns in observations and feedback from their actions. AI allows for pattern recognition beyond human perception,¹⁷ and therefore also includes an element of unpredictability.

Breakthroughs in AI have come from cheap parallel computation, big data and better algorithms,¹⁸ and AI systems are often sufficiently powerful to allow for real-time human–computer interaction, including by means of adaptive algorithms. Even though processing power does not equal intelligence,¹⁹ it does allow to build systems that may pass the ‘Turing test’, i.e. exhibit behaviour indistinguishable from that of a human.

In data-driven marketing, AI systems can be used to increase the effectiveness of marketing, and allow targeted advertising based on ‘how [consumers] will behave, what they will buy, and what they will think.’²⁰ Facebook may, for instance, predict future behaviour to allow advertisers to ‘target people on the basis of decisions they haven’t even made yet’, including the identification by means of so-called loyalty prediction to identify people ‘who are “at risk” of jumping ship from one brand to a competitor’.²¹

3.1. A proxy for understanding people

The overarching idea behind data-driven marketing is that models are trained to recognise and draw inferences from information and behaviour that, for instance, may reveal that riders of particular motorbike brand are actually likely to buy horse saddles. The information may be used to predict not only interests and preferences but also how best to persuade individual consumers, including by building a ‘persuasion profile’, as we discuss in Chapter 7 (persuasive technology).

From big data analyses of personal data, it is possible to obtain knowledge about how people are likely to behave; thus providing traders with more precise information regarding certain aspects of the individual’s preferences or expected behaviour. Such analyses may reveal correlations that are not likely to be discovered through mere logic. Actual causations are not necessarily important as long as the correlations in probabilistic terms provide sufficient insights to influence individuals.

In the 1960s, privacy concerns also revolved around the use of personality tests and other means to invade ‘psychological privacy’.²² It seems fair to assume that the current framework of pervasive surveillance is akin to an automated, real-time personality test at unprecedented scope and scale. The surveillance is likely to expand with new innovative technologies to record and share experiences (e.g., Ray-Ban Stories), measure biological states (smart health devices) and actually brain-to-text communication²³ that we may voluntarily adopt to improve skills and personal efficiency.

There is likely a need for more popular conversations about these issues as discussed in Chapter 12 (next steps). The following analyses aim to suggest how marketing law and data protection law can be interpreted in light of fundamental rights to mitigate harms from predatory data-driven business models.

4. Harms from attention extraction and behaviour modification

The Center for Humane Technology has identified and published the following harms from the ‘extractive attention economy’ relying on persuasive technology:²⁴

1. Digital Addiction: Digital slot machines occupy more and more space in our lives.

2. Mental Health: We constantly face a battle for our attention, social comparison, and bullying.

3. Breakdown of Truth: It’s become harder than ever to separate fact from fiction.

4. Polarization: Stronger ideological rifts make compromise and cooperation more difficult.

5. Political Manipulation: Creating discord through cyberwarfare is far more cost-effective than military action.

6. Superficiality: A social system built on likes and shares prioritizes shallowness over depth.

We will revisit these issues in Chapter 10 (human dignity and democracy) and Chapter 12 (next steps). In this vein, it could be appropriate to include the following quotation from the EDPS:²⁵

‘[…] the Digital Single Market cannot uncritically import the data-driven technologies and business models […]. The internet has evolved in a way that surveillance—tracking people’s behaviour—is considered as the indispensable revenue model for some of the most successful companies. This development calls for critical assessment and search for other options.’

The EDPS is the European Union’s independent data protection authority. Its general mission includes advising EU institutions and bodies on all matters relating to the processing of personal data, including being consulted by the European Commission on proposals for legislation etc., as well as intervening before the CJEU to provide expert advice on interpreting data protection law.²⁶

The importance of the regulation of data-driven business models can also be illustrated by the U.S. Federal Trade Commission settlement with Facebook resolving charges that Facebook deceived consumers by telling them they could keep their information on Facebook private, and then repeatedly allowing it to be shared and made public.²⁷ In July 2019, a violation of this settlement lead to a $5 billion penalty and ‘significant requirements to boost accountability and transparency’.²⁸

More recently, the European Parliament has in the context of the proposed Digital Services Act emphasised the need to ensure that ‘what is illegal offline is also illegal online’, and encouraged stricter rules on ranking, non-discrimination, understandable explanation of algorithms, and targeted advertising. The latter to favour ‘less intrusive forms of advertising that do not require extensive tracking of user interaction with content’.²⁹

In a society where ‘smart’ equals surveillance, the area of privacy law is facing a renaissance akin to that of intellectual property rights a few decades ago, when the digitalisation of products (music, movies, etc.) took off. This time, it is citizens—including their behaviours and experiences—that are being digitalised, which calls for consideration of human dignity among other fundamental rights as explored in Chapter 10 (human dignity and democracy).

In that chapter, we also take into account Frances Haugen’s revelations documented by The Wall Street Journal in their ‘Facebook Files’.³⁰ This whistleblowing is a commercial equivalent to Edward Snowden’s revelations of government surveillance.³¹ It may also compare to revelations about how the tobacco and oil industries knowingly have prioritised profits over public safety by suppressing and disputing evidence of harm.