Shared from the 1/24/2023 Financial Review eEdition



This story has been illustrated using Midjourney, an AI-based image generator now being sued for ingesting artists’ work without their permission.

Artificial intelligence The rise of ‘‘generative AI’’ could both revolutionise and democratise entire fields of human endeavour or be a form of creative bankruptcy, writes John Davidson.

On the topic of how one goes bankrupt, Ernest Hemingway famously wrote that it happens two ways: gradually, then suddenly. The aphorism has come to be applied to other phenomena, like the rate of uptake of technology, or hair loss.

Or, now, artificial intelligence.

Since the rise of what’s become known as ‘‘generative AI’’ in 2022 and the appearance of ChatGPT on November 30 that year, artificial intelligence has transformed from a steady stream of research and development over a period of half a century, to a gushing fire hydrant of technological innovation that, depending on whom you ask, suddenly promises to revolutionise and democratise entire fields of human endeavour, or threatens to bankrupt them.

Or, quite probably, both.

‘‘I do think that ChatGPT marks the start of an ‘all-at-once’ moment in the human evolutionary journey with technology,’’ says Nicholas Therkelsen-Terry, the co-founder and CEO of Max Kelsen, an artificial intelligence and machine-learning software development company that for eight years has been using precursors of ChatGPT to create automated business applications for the likes of Domino’s Pizza and Johnson & Johnson MedTech.

‘‘Knowledge work is now far more under threat than it ever has been in history, and we’re soon going to see a lot of change in what our workforces look like very soon,’’ he says. It’s not a bad thing. ‘‘Things change and jobs change and the nature of work changes, and we as humans roll on,’’ he says. The big question is, where are we rolling? ChatGPT is a so-called ‘‘large language model’’ (LLM) machine-learning chatbot that, according to the chatbot itself, was created by analysing billions of sentences of text taken from the internet.

The analysis built a statistical model that’s able to create human-like conversation by predicting which words are most likely to follow a given prompt (such as the question ‘‘how does ChatGPT work?’’), and which word is most likely to follow or precede the word before or after it.

Large language models trace their roots back to the natural language processing (NLP) research in the 1950s and 1960s, but the progress in NLP began to shift from the ‘‘gradual’’ phase to the ‘‘sudden’’ phase in 2017, when a team of researchers at Google published a paper called ‘‘Attention is all you need’’, revolutionising the field and giving rise to a new breed of machine learning.

The paper proposed a way of analysing text, known as a ‘‘transformer’’, that rather than just analyse a word in the context of the word that came before it, could look at an entire input sequence (such as a sentence or a paragraph) and weigh up which parts of the sequence were most relevant to the output the NLP engine was asked to generate.

Suddenly, the model could create a correlation between words at the beginning of a sentence, and words in the middle or at the end, vastly improving the model’s performance.

Since then, LLMs with names like GPT (Generative Pre-trained Transformer) from Elon Musk’s OpenAI in 2018, ‘‘BERT’’ (Bidirectional Encoder Representations from Transformers) from Google in 2018, and RoBERTa from Facebook in 2019 have been released at a steady pace, typically with open-source licensing that allowed competing researchers to learn from each other.

‘‘BERT was underwhelming, to be honest,’’ Therkelsen-Terry told The Australian Financial Review.

‘‘It didn’t give us a huge boost over what we were already doing. But RoBERTa was massively transformative. Facebook took BERT, they dialled up the number of parameters, and overnight it was better than anything we’d ever seen before by a considerable margin,’’ he says.

Parameters can be thought of as the rough equivalent of neurons in an organic brain. They take an input (say, a photo of food), apply a statistical weighting to it (what’s the likelihood it’s shaped like a sausage?), and pass it along to the next parameter, which applies a new weighting (what’s the chance it’s red?), passing the image though parameter after parameter until, at the end, the model decides that the thing in the photo is, in all likelihood, a hot dog (but it might be a red pickle).

This type of AI, known as ‘‘deep learning’’, doesn’t deal in certainties, but is referred to by data scientists as ‘‘probabilistic’’, or ‘‘stochastic’’. There are other forms of AI that are more deterministic – that’s definitely a hot dog! – but it’s deep learning that is in the ascendancy right now.

(Deep learning is so popular among data scientists, Tesla has even adopted a completely probabilistic approach to autonomous driving, abandoning lidar sensors and other, more deterministic sensors and relying solely on in-car cameras and deep-learning-based computer vision. Critics of the purely deterministic approach, like the Emeritus Professor of Psychology and Neural Science at New York University, Gary Marcus, say this explains the infamous video of a Tesla Model Y driving itself into a private jet. Maybe there’s an object there, but then again … )

With each new generation of LLMs, the number of parameters has been ballooning, starting with 117 million in GPT and 340 million in BERT.

GPT-3, the LLM upon which OpenAI based ChatGPT, has 175 billion parameters. Google’s GLaM (Generalist Language Model) has 1.2 trillion.

The result is a kind of magic: machines that have ingested an internet’s worth of data, weighed up the relationships between things, and are able to generate content that appears to be new and original.

But transformers and parameters, it turns out, aren’t everything. Even LLMs with vast numbers of parameters can still make egregious errors, both of fact and of judgment.

If the data the model was trained on contained false information (which it does) or racist or sexist comments (which it does), then the LLM might still spit out erroneous or racist or sexist responses, if it judges those are the most likely responses to a given prompt.

In a paper on the dangers of LLMs, AI researcher Emily Bender described the models as ‘‘stochastic parrots’’: they can regurgitate what they’ve been taught, often in new and seemingly intelligent ways, but they don’t understand a word of it.

Hanno Blankenstein, the CEO and founder Unleash live, a company that creates AI business solutions in the field of computer vision, likens AI models to infants.

The ‘‘AI kids’’ have been exposed to all the data in the world, but they still need to be taught the rules of the world, he says.

One way to do this, he says, is through reinforcement learning administered by humans, who train the AI model with a regime of repeated ‘‘treats and penalties’’ until its outputs closely match the real world.

Creating a machine learning model that could help electricity providers lower the cost of maintaining powerlines, Unleash live used a generative AI to create millions of artificial photographs of cracked electrical insulators, and then used those images to repeatedly train a computer vision system until it could spot a crack in an insulator in a real photo, he says.

The result, he says, is a system that can trawl through countless hours of imagery taken from drones flying close to power-lines, and automatically identify problems that humans should look into.

OpenAI – the company behind ChatGPT and which has a strategic alliance with Microsoft – did something similar to train its LLM GPT-3 not to parrot hateful or harmful content in its responses to prompts.

In an effort to build a more ethical version of GPT-3 (the version that became known as GPT-3.5, which is used by the ChatGPT chatbot) OpenAI hired humans to painstakingly provide the LLM with the mathematical equivalent of penalties and treats, whenever it did or did not come up with harmful or hateful responses.

The ethics of OpenAI’s method for creating an ethical AI have since been questioned, however. A recent investigation by Time revealed that OpenAI hired an outsourcer that used low-paid workers in Kenya to teach GPT-3 how to identify toxic content, and avoid reproducing that content in its outputs.

The toxic content the workers had to deal with ‘‘described situations in graphic detail like child sexual abuse, bestiality, murder, suicide, torture, self harm, and incest,’’ Time reported.

And traumatising workers in order to generate ethical AI content isn’t the only complaint about the sudden explosion of generative AI.

Upon discovering that their artworks had been used without permission to train a generative AI to produces images in response to prompts, three artists recently commenced a class action against Stability AI and Midjourney, as well as the company DeviantArt.

The lawsuit alleged that the generative AIs ‘‘flood the market with an essentially unlimited number of infringing images (that) will inflict permanent damage on the market for art and artists’’.

Nick Cave, the Australian singer and songwriter, has voiced similar concerns about generative AIs that have been trained on the lyrics of his songs, and that can write new songs in the ‘‘Nick Cave style’’ when prompted to do so. ‘‘I understand that ChatGPT is in its infancy, but perhaps that is the emerging horror of AI – that it will forever be in its infancy, as it will always have further to go, and the direction is always forward, always faster,’’ he wrote on his blog.

Describing a song written by ChatGPT in his style, Cave wrote: ‘‘this song is bullshit, a grotesque mockery of what it is to be human, and, well, I don’t much like it’’.

‘‘The apocalypse is well on its way. This song sucks,’’ Cave blogged.

It’s possible, though, that the solution to all of this will simply be more of this.

Google is already working on LLMs that provide far more source-attribution to their outputs, helping to deal with the copyright infringement issues raised by artists, and helping to add more confidence to probabilistic responses to questions.

It might only be a coincidence, but source-attribution might also help Google sell advertising through generative AI responses, says Therkelsen-Terry: if a reply to a question is identified as coming from a business, that business might one day be hit up for an ad fee, he says.

And, in response to the problem posed by ChatGPT for schools and universities, enterprising individuals have already started to build AI models that can detect when an essay is written by other AIs.

Indeed, LLM makers such as OpenAI are planning to add ‘‘watermarks’’ to their machine-generated texts, in the form of words, letters and punctuation discreetly embedded in the text that show where it came from.

What we’re experiencing right now might merely be the growing pains as AI leaves its infancy and suddenly grows up; the initial snap of G-force as the technology moves from ‘‘slowly’’, to ‘‘all-at-once’’.

Or, as Cave intimates, it might be a form of creative bankruptcy from which humans may never recover.

Therkelsen-Terry, who uses AI in his job every day, says it’s the former.

‘‘If we have the tools to build this stuff, we have the tools to correct for the downside risk that it creates.’’ AFR

See this article in the e-Edition Here