The slop of generative AI

by Enrico Nardelli

(versione italiana qua)

I am writing this article prompted by a very recent post by my excellent colleague Walter Quattrociocchi. It is – to be honest – yet another in a long series where he explains on social media the dangers posed by generative artificial intelligence tools (GenAI, hereafter) for those who lack a solid understanding of the subject they are asking about. In his post, Walter recalls two recent episodes of sloppiness resulting from the use of these tools.

The first involves a print newspaper article published including the text of that standard closing question GenAI tools offer at the end of their response, asking whether the user would like to expand or adjust the text for a different publication context (e.g., a popular science piece rather than an academic one).

The second is that a well-known international consulting firm included fabricated citations in a—highly paid—government report and was, consequently, forced to partially refund its fee.

Neither example is new: the inclusion of AI-hallucinated cases in legal briefs has been happening in the United States for at least a couple of years, and forgetting to remove boilerplate phrases generated by GenAI in scientific articles has also been happening with some regularity for quite some time.

What Walter rightly observes—and I completely agree with—is that the issue is now systemic, widespread across all sectors and at every level. In his words: "We have delegated judgment to a statistical language engine, mistaking textual coherence for factual truth. This is the heart of the epistemic problem: the confusion between linguistic consistency and epistemic truth." A point that resonates with what I wrote in April 2023, warning that the main danger of GenAI tools is that «They have no real understanding of the meaning of what they are producing, but (and this is a major problem on the social level) since they express themselves in a form that is meaningful to us, we project onto their outputs the meaning that is within us», and recalling, two years later, that «showing competence with the words that describe the world is not the same as having competence about the world itself».

Two recent studies offer further evidence of how GenAI tools fall short of their potential and why, therefore, great caution is needed instead of the uncritical praise hailing them as something everyone must absolutely know how to use.

The first is the State of AI in Business 2025 report by MIT, which aptly speaks of a “GenAI Divide”, the gulf between the high rates of corporate adoption of these tools and the low return on investment observed so far. According to the report, 95% of companies are seeing no return on their investment, despite widespread use of the tools themselves. At the individual level, there is an increase in productivity, but this is not translating into corporate profitability.

The causes of these failures are a lack of robustness in supporting work processes, a failure to learn context, and a misalignment with daily operations. The judgment on scalability is particularly interesting: «the core barrier to scaling is not infrastructure, regulation, or talent. It is learning. Most GenAI systems do not retain feedback, adapt to context, or improve over time». I had already addressed this difficulty three years ago in my book La rivoluzione informatica (published just as the first such systems were gaining traction). Writing more generally of "cognitive machines" (the term I use for all computing systems), I noted that they "lack both the flexibility and adaptability to change their mode of operation as contextual conditions shift." It is gratifying to see one's own assessments hold up over time.

The 5% of companies that are succeeding are doing so by asking vendors to tailor these tools to the specific needs of their workflows and their own proprietary data (as I have always advised those who have privately asked me how to use GenAI within their companies). They are also evaluating effectiveness not based on generic benchmarks (like passing standardized exams in various disciplines) but by considering the actual output produced.

A subsequent study by a team of researchers from Stanford and BetterUp Labs further investigated this situation. Specifically, 1,150 US workers from various sectors were interviewed to understand how GenAI output is used at work. It emerged that while some use these tools to refine work they have already produced, many use them to churn out low-quality output that then has to be corrected by someone else downstream. The term "workslop" is gaining currency – where "slop" is an informal word for watery, unappetising food – to describe AI-generated content that looks passable on the surface but lacks the substance needed to move a task forward effectively. More formally, we might call it "shoddy work," and the result is that corporate productivity suffers.

In this study, 40% of the workers interviewed revealed they had received "slop" in the last month that was practically unusable. While the phenomenon primarily concerns peer-to-peer interactions, in 18% of cases it also occurs hierarchically, in both directions. The most serious long-term impact is on trust between colleagues, which is eroded whenever someone receives shoddy work. As with all previous technological waves, what is needed first and foremost is a clear identification of the strategy through which these tools can improve work processes, followed by the selection of technology capable of supporting that strategy and appropriate workforce training. In short, GenAI does have potential; it all depends on how it is used.

The Roman orator Marcus Porcius Cato, known as Cato the Censor, is credited with the saying «rem tene, verba sequentur», i.e., master the subject matter, and the words will follow, an admonition about the importance of thoroughly mastering a topic to be able to argue its cause effectively. Similarly, we could say in this case: "rem tene, mens mechanica te sustinebit," where "mens mechanica" refers to what I call "mechanical intelligence" rather than the more common "artificial intelligence".

Only if you know your subject well will GenAI be able to enrich and enhance what you do. This is why I believe the foundations of any area of knowledge must still be acquired, as they always have been, by “the sweat of one's brow”. Only then will we be able to use GenAI tools in the best way possible.

--
The original version (in italian) has been published by "StartMAG" on 12 october 2025.