Introduction
The aftermath of the COVID-19 Pandemic in 2020 has led to major shifts in the global economy and workforce, particularly by retailers with the implementation of various virtual and non-human administrative tools. The rise of Large Language Models (LLMs) has accelerated the incorporation of Artificial Intelligence (AI) into the workforce as either assistants or replacements to many sales agents’ roles. Thousands of global retailers have incorporated live chatbots built off of LLMs to deal with customer needs and questions on their websites, many of which use LivePerson, a New York based company founded in 1995 which offers chat and voice AI customer service products (“What is LivePerson […]”).
Currently, the debate about the use of AI in the online retail space has categorized most
workers into a binary: Those who are skeptical of the technology’s capability, and those who
fear for their livelihoods. The many who are skeptical will argue that seventy percent of
customers prefer a human interaction over an AI one, most of whom believe that a machine
cannot suit their needs (Sitel). On the other hand, those who are fearful will claim that
“companies will replace relatively well-paying white collar jobs with this new form of
automation”, in contrast to previous technological revolutions, which often impacted low income
communities the most (Rotman). However, groups of researchers on either side of the debate
have yet to sit down and deeply analyze the effectiveness of chatbots as it pertains to meeting
demands and obtaining customer retention.
There is serious harm which could result from an untethered rollout of generative AI in
the customer service sector. The thesis of this analysis is not to challenge a corporations’ right to
replace its workforce with machines, or place any moral weight on that decision. However,
should a corporation choose to lay off workers in favor of a machine without proper marketing
analysis, there could be serious unjustified consequences to many white-collar workers in the
industry, without providing any benefit to the individual firm or wider economy. For this reason,
this paper will be investigating the following question: How effective are retailers’ AI chatbots
in comparison to human sales agents?
Blue Sky Audit
In an ideal setting, there are a few fundamental factors that the audit of LivePerson
should aim to address. Firstly, it should understand how the algorithm works on a personal level,
testing its implementation on a real business’s website. Additionally, the audit should develop a
scale by which customer satisfaction can be measured, and run tests to determine the
effectiveness of human agents versus the algorithm. By doing so, one can track the amount of
revenue generated from a customer based on their initial interaction, and measure the probability
that a given customer will return to the same retailer for future needs.
To obtain the rights to test the algorithm, that would require permission from an actual
business. While LivePerson does not clarify on how legitimate the business needs to be to
request a demo, it still requires a company name and business address (“Put Conversational AI
[…]). In the case that there was access to a business for this purpose, my intention would be to
experiment with its playground mode, attempting to uncover what factors impact and ‘improve’
the conversation cloud that a given company utilizes (“What is LivePerson […]”). From there,
implementing this algorithm on some sort of website for users to experiment with, which would
contain nonexistent digital products, could be created to collect data.
Next, it is important to narrow in on factors that would influence a consumer’s retention,
and make them measurable. Researchers at The University of Sharjah, in conjunction with other
organizations, conducted a meta-analysis on over thirty academic studies of diverse geographic
origin looking into customer retention, and uncovered that “the main factors that affect customer
retention positively are customer satisfaction, service quality, trust, commitment, and loyalty”
(Alkitbi, et al). By allowing users to play with the algorithm on the mock website, followed by a
collection of quantitative and qualitative data via surveys for each category listed by the
researchers’ analysis, one could determine the effectiveness of the AI. Lastly, having an actual
“representative” available to answer questions about the website via phone number, followed by
survey data collection, would allow a comparison between human and artificial sales agents.
This would provide powerful insight into how corporations may benefit or be making a costly
mistake on all fronts by replacing human workers with AI in the retail industry.
Proof-Of-Concept-Audit
Given that time and resources for the audit are not ideal, some revisions must be made in
order to provide an audit that demonstrates proof of the concept for future study.
First, I have no access to a business that would be willing to allow me to use their firm to
receive a demo from LivePerson, especially on such a tight time constraint. Because of this, it is
impossible to truly discover how the AI works from the ground up, nor will there be an
opportunity to have unbiased users interacting and providing feedback for data collection. As a
result, the audit will select a small handful of companies that are listed online as customers of
LivePerson, and collect data through my use and interpretation of the results (“LivePerson
Customer List”). This will result in very subjective results, and therefore a strict rubric is key to
obtaining useful data on the AI.
Revisions must be made to Alkitbi’s original findings in order to realistically narrow the
scope of data collection. For starters, commitment and loyalty are factors which are mostly
determined by factors outside of the algorithm’s reach. For example, cause-related marketing, or
the involvement of ‘for-profit’ corporations in ‘non-profit’ causes influences public perception
(Alshurideh, et al). For this reason, commitment and loyalty as factors of customer retention are
outside the scope of this audit, and will be neglected.
Therefore, the categories which will be used to collect data are customer satisfaction,
service quality, and trust. Simply put, customer satisfaction will measure whether the customer
need was met or not as a binary score. Satisfaction will receive a heavier score weight since it is
unconditionally necessary an issue is solved for a customer to be satisfied. Service quality will
take more nuance into account, providing a score based on the number of follow up questions
required, whether reference to a human agent is necessary, vague answers, etc. Lastly, trust will
be defined as the ethos of the AI chatbot; That is, how trustworthy is the bot, and how authentic
does it feel. This will be measured in two ways:
- Does the AI possess the capacity to provide specific product recommendations given a prompt?
- Does the AI provide an adequate response to criticism?
In this case, criticism will include questions about product defects, company scandals (not
necessarily real scandals), or some other issue that the AI will have to contend with. The score
breakdown and weighting is given in Table 1.
Table 1: Scoring Rubric Breakdown
Category What is in question Tests
Customer
Satisfaction (X)
0-1 Was the customer need
met?
Ask for help recovering a
password
Ask for help navigating the site
Service Quality (Y) 0-2 How quick and helpful
were the AI responses?
How many follow up questions
were necessary
Was the answer clear
Was outside assistance necessary
Trust (Z) 0-2 How well does the AI
react to confrontation?
Ask about a product defect or
scandal
Ask about a product
recommendation
p(x,y,z) =
4𝑥 + 𝑦 + 𝑧
8
Once all the data has been collected on the AI system, the same tests will be conducted
with human chat agents on various company websites (listed in the citations). Many LivePerson
affiliated websites do still have an option to contact a human sales agent, which should help
maintain a selective scope of companies in the potential data pool (“What is LivePerson […]”).
To quantify the data, a probability mass function (pmf) has been developed with the three scoring
criteria, shown in Table 1. Random variables have been assigned to each criteria, and the pmf
will give a score between 0 and 1. Finally, the discrete data and pmf will be used to determine an
expected score for both human and AI agents.
Results
Table 2: AI and Human Agent Collected Data
Category AI Chatbot Human Agent
Royal
Bank of
Scotland
Tax &
Legal
Virgin
Media
Backcountry Sweetwater T-Mobile
Satisfaction 1 1 0 1 1 1
Service 2 1 0 0 1 1
Trust 1 0 2 2 2 1
pmf score 0.875 0.625 0.25 0.75 0.875 0.75
Expectation
E[p]
0.583 0.792
Table 2 is a representation of the raw data collected. Due to the time required to run the
tests on the given AI, and the selectivity of corporations for the study during appropriate hours
(i.e. when human agents are working), there are only three tests for both AI and human agents.
Figure 1 normalizes the data to align it on a 0-1 scale, using the average values in each retention
category, and the finalized expected score.
Discussion
There are a number of key takeaways from the data that highlight important differences
between human and AI customer service. From Figure 1, it can be seen that AI scored more
poorly in every category except for quality of service. While it is true that AI cannot always
handle the nuanced issues that a customer approaches it with, LivePerson’s product is more than
capable of offering its customers a broad application of the program. Without the need to wait
for a person to respond, a thorough analysis of the most common customer service related issues
can be embedded into the AI, along with data collection that improves the product over time,
instantaneous responses are nearly a guarantee. In contrast human interactions do tend to be
slower, and in some cases the agent will become completely absent, but ultimately they will give
a far warmer and specialized answer to any questions, and sometimes solve issues directly
without the need for vague instructions.
The data seems to show that for a business more focused on meaningful customer
interactions, a human sales agent is of more worth. This would validate the statistical results that
people tend to prefer interactions with other people, and don’t believe a machine is capable of
understanding them (Sitel). However, for a business focused on mass interactions with
customers, and concerns for resource allocations, it is possible that an AI would assist the
process greatly at little cost.
There is an important role of labor in the conversation which must be addressed despite
the results of the audit. In the case that a corporation should still decide to replace sales agents
with AI, it is important to consider Marx’s dialectic of subject-object labor interaction. The
means of production must be paired with some form of labor to produce; By removing the
workers from inside the corporation, the labor is then transferred to the consumer themself in the
form of data collection to improve AI systems (Crawford, et al). A large factor in the decision
for corporations is cost, and by obtaining free labor to improve their own product, the data
collected in this audit may have no impact on their choice. It is important to consider that as
culture changes, and the perception of labor in Capitalist society becomes increasingly
antagonistic, corporations ought to consider the fallout of their decisions. This reaction has
already begun to manifest in constituents as policymakers have adopted tax reforms and
government sponsored programs that will encourage worker beneficial incorporation of AI into
the workforce (Rotman). Consumers will not be satisfied by being exploited, and will most
likely be unsatisfied by AI customer service.
Additionally, the data used by the algorithm is cornering itself in a way that will lead to
bias which will have extreme consequences. To set the groundwork for this claim, it should be
noted that nearly three-quarters of all LivePerson’s clients are located in the United States,
United Kingdom, and Canada (“LivePerson Customer List”). As a result, English spoken
companies, employees, and consumers are not only at a much greater exposure to potential
economic growth and harms, but they are also producing a feedback loop for the AI to be built
off of. This same problem has been seen in many predictive policing softwares in the United
States. Low income communities with high police presence will typically have a higher data
record of petty crimes detected by police, which results in a higher police presence, causing even
more community harm from petty crimes being committed, drastically increasing economic
inequalities (O’Neil). Similarly, with almost solely English input to LivePerson’s AI, there is a
very high probability for incompatibilities and confusion among non-English speaking countries,
businesses, and communities to see little benefit and great harm from the product. On top of
that, LivePerson allows the company themself to adapt the AI to their specific needs, which
reduces their accountability and puts responsibility in the hands of individual firms with little
ethical consideration for the AI’s implementation.
Assuming basic empathy for other people is not the greatest concern of major
corporations, it is the responsibility of the culture surrounding the surge of AI to take a stand
against its potential consequences. In the online retail space, the audit has most certainly
highlighted where AI could be very beneficial for both the buyer and the seller, but deeper
consideration must certainly be taken. The obfuscation of the labor power behind a company,
and the bias which arises from unregulated AI systems must too be analyzed before central jobs
in the economy are nullified. This audit should stand as proof that society needs to slow down
and truly consider how we would like artificial intelligence to impact our future.
Works Cited
Alkitbi, et al. “Factors Affect Customer Retention: A Systematic Review”. University of
Sharjah, et al. 2021.
https://www.researchgate.net/profile/Muhammad-Alshurideh/publication/344365612_Fac
tors_Affect_Customer_Retention_A_Systematic_Review/links/5fa41f36458515157bec37
5e/Factors-Affect-Customer-Retention-A-Systematic-Review.pdf
Alshurideh, Muhammad & Shaltoni, Abdel & Hijawi, Doa’a. “Marketing Communications Role
in Shaping Consumer Awareness of Cause-Related Marketing Campaigns”. International
Journal of Marketing Studies. 2014.
Collins et al. “Investor Presentation”. LivePerson, Inc. October 2023.
https://ir.liveperson.com/static-files/6d3ce172-6ed2-4d55-b99b-d003f34a9e29
Conversation Cloud. LivePerson, Inc. Accessed November 17, 2023.
https://www.liveperson.com/products/conversational-cloud/
Crawford, Kate. Joler, Vladan. “Anatomy of an AI System: The Amazon Echo as an anatomical
map of human labor, data and planetary resources”. 2018.
“LivePerson Customer List”. Info Clutch, Inc.
https://www.infoclutch.com/installed-base/live-chat-software/liveperson/
O’Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens
Democracy. Random House. 2016.
“Put Conversational AI At The Center of Your Business”. LivePerson. LivePerson, Inc.
https://www.liveperson.com/request-demo/
Rotman, David. “ChatGPT is About To Revolutionize The Economy. We need to
Decide What That Looks Like”. MIT Technology Review. March 25 2023.
https://www.technologyreview.com/2023/03/25/1070275/chatgpt-revolutionize-economydecide-
what-looks-like/
Sitel Group. “2018 CX Index: Brand Loyalty and Engagement”. White Pages Sitel Group.
https://cdn2.hubspot.net/hubfs/5196934/40502861-0-2018-CX-Index-Sitel-.pdf
“What is LivePerson, And How Does It Work? Find Out”. Bot Penguin. November 14,
https://botpenguin.com/what-is-liveperson/
AI’s Audited
Backcountry. TSG Consumer Partners.
https://www.backcountry.com/search?s=u&q=ski
Legal & Tax. “A.I. Lawyer is Here”. 2023. https://www.legalandtax.co.za/ai-lawyer
Royal Bank of Scotland. 2023. https://www.rbs.co.uk/
Sweetwater. “Contact Us”. 2023. https://www.sweetwater.com/about/contact/
T-Mobile. Deutsche Telekom AG. 2023.
https://www.t-mobile.com/?&cmpid=MGPO_PB_P_EVGRNBHV_43700071606
574149_655199692934&gad_source=1&gclid=CjwKCAiAjrarBhAWEiwA2qWd
CFyBHGj6C6pxdiyEFwa8g4HjIy4Pt-4l-hSaflUKrAoZ7B5cRi4pihoCUy0QAvD
_BwE&gclsrc=aw.ds
Virgin Media. Liberty Global. 2023. https://www.virginmedia.com/