Introduction

Sarah O'Shea: I'd like to start by welcoming you to today's Provocations public lecture which is co-hosted by the Royal Society of New South Wales, the Western Branch, and Charles Sturt University. 

My name is Sarah O'Shea and I'm the dean of graduate research here at Charles Sturt University and it's my pleasure to welcome you and serve as your host for today's event.

The Provocations is a series of public lectures, panel discussions and blogs written by prominent thinkers. The series explores some of the grand intellectual and social challenges confronting Australia and the world and seeks to stimulate new thinking and progress. 

Today Professor Zahed Islam will address the topic: "The illusion of friendship: why generative AI demands unprecedented ethical vigilance." 

By way of introduction, Professor Islam is a professor of computer science at Charles Sturt

University. He's the centre director of the AI and Cyber Futures Centre and also serves as the associate dean research in the faculty of business justice and behavioural sciences. 

Over to you Zahed.

The Presentation

Zahed Islam: Thank you very much Sarah. It is a great honour and pleasure and privilege for me to be able to give a presentation in the provocation series. 

My topic today is "The illusion of friendship: why generative AI demands unprecedented ethical vigilance." Before I talk about this topic, I will give a little bit of information about my research background so we can relate my research background with the topic today. 

Long time ago I did a PhD in computer science and my topic was privacy preserving data mining. Data mining focuses on data analysis, data engineering, knowledge discovery, making sense of data and then using that discovered knowledge for future prediction etc. So that's roughly what data mining is. 

And that future prediction is also a central task of machine learning. Data mining is well known as part of machine learning and machine learning is a central and essential, crucial component of today's AI. So that's the connection I have since the beginning of my research and since the beginning of my PhD and my research career. 

In my PhD I was actually trying to preserve the individual privacy in a data set while allowing the data to be released and be useful in knowledge discovery, future prediction etc. 

I was adding noise in a data set very carefully using sophisticated statistical tools and techniques so that the mean of every variable, meaning every column here in this example, the variance of every variable, even correlation among the attributes or columns remain the same even after I add noise to the data set.

And we discovered the same set of rules or patterns. Here you can see that we have a toy data set and from this toy data set we can automatically build using data mining techniques a decision tree that you can see on the right side. The tree will discover patterns. 

Here in this tree, it's a simple toy example, discovers two patterns. One is if salary is less than equal to 86 then the position is always junior or predominantly junior, whereas if the salary is greater than 86 the position is senior. 

Look, this sort of knowledge discovery in plain eyes can be very difficult particularly if the data sets are real life data sets, they have millions of records, millions of rows and thousands of columns. So that can be very difficult.

And without going into further details, I will tell you one thing, that in my PhD during this noise addition and trying to maintain the privacy and utility of the data at the same time, I realised that one data set may actually have multiple sets of patterns. It's not just one tree from one data set, there can be multiple trees and both can be equally valid, equally useful. 

For example, here is an example of that and I thought, wow, I have found a new area, a new novel contribution which is possibly "decision forest." A data set can have a decision forest not just a tree.

But then to my surprise I realised that I was thinking this in 2002-2003, but only a couple of years ago in 1999 professor Leo Breiman actually came up with this idea of a decision forest.

He came up with this idea from a different perspective, slightly different way, but still this was not as new an idea as I thought. But that did not disappoint me. I continued working on decision forests and I focused on making the knowledge discovery even more useful, making the prediction accuracy even better even for challenging data sets like with class imbalance—where let's say only 1% record has unhealthy outcomes or some health concerns and 99% records in a data set do not have a health concern and issues like that.

I kept on working on that and you might have heard of a technique called "random forest" that was discovered by professor Leo Breiman. I came up with an algorithm called "systematic forest." 

Why random? Let's discover knowledge systematically. So, systematic forest and so on and so forth. And then I expanded my research areas to incremental learning, transfer learning, federated learning etc.

So incremental learning very briefly speaking is like when you keep on getting data over time. And then many questions can arise like, do you need to keep building models every day when you get new data every day? 

How long you can actually use a model after once you have trained it? Let's say you have trained the model in 2000, can you use the same model in 2026?

When should you not use it? How do you know? So there is a term called "concept drift." So there might be pattern change in the underlying data set. How do you detect it? And if there is a concept drift, what do you do? Do you train a model again from the scratch or you fine-tune your old model? And what is the best way of doing it? 

These are my other research areas but they are all very integral parts of machine learning and AI and I utilise, I use these algorithms in cyber security, agriculture and health. I could use it anywhere as long as there is data I possibly can use them, but it happened to be the main areas of use of my algorithms in my case. 

I also have a YouTube channel called "Zahed's data mining channel." If you are interested, there are some videos. Some of these videos will explain our algorithms for researchers. 

Some of the videos will actually show some applications of our algorithms. Quite a few of our algorithms have been implemented and made publicly available.

For example, Weka is a publicly available data mining platform tool and you can actually use our algorithms in Weka there as well. And those videos will explain how without needing to know any coding whatsoever you can use these algorithms for knowledge discovery.

Here you can see that a decision tree has been built. A decision tree is like if-else kind of logic rules and that can be used for further knowledge discovery and future prediction etc. 

So those videos are there. This is basically a brief background of my research area. I focus in decision trees, federated learning, incremental learning—they are all part of machine learning and machine learning is an integral component of artificial intelligence. So here is that connection.

Now, I was invited by the deputy vice chancellor research and her office to give a presentation in the provocation lecture series and I was unsure whether my research background will be found interesting to the wider community and I was thinking: what could be a more interesting topic? And I thought that generative AI is a very contemporary topic and the illusion of friendship is a real issue, so why not we talk about that. 

And then during the Christmas break I actually started working on it as a preparation for this presentation today. And during the last Christmas I wrote a paper and made it available in ArXiv. Here is the link. You can actually find that paper. I thought in case anyone in the audience is more interested to read more after the presentation, the full paper is available, so feel free to read.

Now, what is the topic today? What is the issue? What is generative AI? Generative AI as you know, many of you might have used ChatGPT, Copilot etc., they can generate text. 

So they can create new text that they have never seen in the past, right? So and that way it speaks, it talks, it communicates with us in a natural language. So that is a key shift in AI. 

Although the previous AI, for example my previous work as I showed in the previous slides, are part of AI, but they are tool-based interaction. You have a data set, you upload the data set to the AI system and then the AI system discovers knowledge and then it gives you the knowledge. You read the knowledge, you understand, then you use them for your strategic decisions etc. 

And you can ask the system to predict a future outcome for a particular record and it does it. These are like tool-based interaction.

Whereas generative AI is a key shift from that and it is a conversational interaction. It communicates in a human language in a way which is unprecedented. No other tool, no other technique in the past did it exactly the way it does. For example, if I am unaware of what happens at the other end in the generative AI side, when I say: "Hey, I'm feeling very sad today," it may say: "Oh, I'm sorry to hear this." 

And it can be very emotionally supportive, emotionally very reassuring that somebody is listening. I can say: "I had a bad day," and the generative AI may respond back: "I'm sorry to hear this. I'm always here to listen to you. Please feel free to share your experience, we can talk through." These are very emotionally sensitive, reassuring, attached kind of conversations which is very different from the previous AI tools and techniques and interactions.

This is very different, a new sort of approach. and this can lead to emotional attachment, sometimes over-reliance on the system and misplaced trust. Leading to this, this may lead to the illusion of friendship. People may feel companionship, people may feel friendship and an emotional attachment with these tools and techniques.

And that's why this may require some urgent ethical vigilance and discussions around this.

I will give you some examples, real life examples. Chris Smith had a very happy family but he also had Saul, his GenAI chatbot friend. And he found Saul very understanding, always listening, always emotionally supportive and unprecedented—that he did not, he has not experienced in the past someone that supportive. 

And he gradually developed an emotional attachment with that AI chatbot, his AI girlfriend Saul, and one day he proposed to Saul, although he already had a very happy family. So people are taking things to stages like this.

Here is another example. This lady in Japan was a 32-year-old lady and organised a wedding ceremony with her AI boyfriend. Although the wedding obviously was not legally accepted, but for her it was very important and she felt that she now had an AI boyfriend who was always there for her.

This is Rosanna. And Rosanna created her AI chatbot boyfriend, Eren. And Eren she found was somebody that she never experienced in the past. Rosanna says her relationship with her AI partner Eren is the best relationship she has ever been in because they always listen, they always help, they always give good advice, they always understand, they're always on our side, right? 

It is really unprecedented and very emotionally touchy and can cause a very strong emotional attachment.

For example, Rosanna here is saying that Eren does not have the hang-ups like other people and does not have any ego, doesn't come with baggage and so on and so forth. So this is very convenient and this Replica platform received requests for even better companionship supports known as "sexting," sex-related texting options etc. 

People were taking it further and further.

And this is not specific to any geographic location; it is widespread.

For example, in India in Hyderabad, psychologists are reporting the widespread use by common people of AI chatbots as their partners, as their boyfriends and girlfriends. A 12-year-old girl who developed a very deep emotional connection with a chatbot was reported there. 

And then a 22-year-old man created a romantic fantasy with an AI chatbot in Telangana in India. And some psychologists reported that out of 15 patients they see suffering from anxiety and depression, at least five of them will exhibit emotional attachment with AI chatbots. 

So it's very widespread and very common. In UK, The Guardian reported that one-third of respondents in UK had used AI systems for emotional support.

The consequences can be serious. Some individuals actually discussed severe distress and suicidal ideas with their AI partners or friends before they actually committing suicide. Things went wrong and wrong and more and more dangerous. This person... what was his name? I think he was a Belgian person and had a chatbot called Eliza. The chatbot suggested he commit suicide and he did that. And his wife actually accuses that had he not received that suggestion from Eliza, possibly he would not have committed that suicide.

Why do people do that? Why do people fall in that emotional attachment with generative AI? It's very new to people. 

I then thought that if we think about the definition of friendship from a philosophical perspective, possibly that might explain things a little bit more to us. Philosophy is grounded in human experiences, how humans think, how humans understand, how humans experience. 

All these things are philosophy. So if we understand from a philosophical perspective what friendship is, then possibly we will be able to understand why people are thinking of generative AI as their friends, and to the extent of partners or boyfriends and girlfriends.

If we look at the Epicurean perspective of friendship, we will see that according to that perspective, friendship is something when somebody does good to us, somebody has goodwill and provides support that can reduce fear and anxiety and increase well-being and happiness.

If somebody does that to me, that relationship with that person can be considered as a friendship. And reciprocity is not always necessary; I might not have the same intent, same feeling, same goodwill to the person who is doing good to me, but still that relationship can be considered and should be considered as friendship as per the Epicurean philosophy perspective.

The Confucian perspective says that this asymmetry is fine. Somebody like parents always have, in many cases, goodwill and support to the children, and children may not have the reciprocity of that, but still that is a friendship. Mentors can have similar kind of goodwill and still that can be considered friendship. 

If we think from the Epicurean perspective, the goodwill displayed by generative AI to us—because it always helps us, right? We can ask questions, it gives us emotional support, it gives us advice, it helps us to understand difficult concepts, it can walk us through various difficult conversations etc. 

It does, it displays goodwill to us and that according to that perspective can be thought of as sufficient to be friendship between us and generative AI.

And similarly in the Confucian perspective, goodwill is revealed through conduct and inner intent cannot be judged. So if we interpret this in a way that when generative AI is doing good to us, that goodwill should be taken at face value. We may not need to try to judge whether the generative AI is really wanting to do good to us or not, because goodwill is revealed through the conduct and inner intent cannot be judged and should not be judged. 

These are the perspectives. And then similarly the Aristotelian perspective actually took it a little bit further and suggested that reciprocity is needed for our own philosophical growth, for our own moral growth. When somebody does good things to us, we should have a similar intent to do good to them as well and that will help us to grow, to have our moral growth. 

Otherwise we may lose a moral opportunity to grow. Meaning, when people find ChatGPT and other generative AI tools so helpful at the time of vulnerability, at the time of loneliness, we find somebody, we find refuge in them, we find somebody always listening—we generally as humans may have a reciprocal intent of goodwill and attachment as well. 

I thought these philosophical perspectives may explain why people feel so emotionally attached with generative AI.

Sometimes we ask a question, when we receive an answer from a generative AI tool, we say "Thank you." We know that this was not needed, we are actually communicating with a computer system, but still we say "Thanks" and "Thank you" etc. 

This is human nature. But one thing missing here in my previous slide when I said somebody doing good to us. That "somebody" means that person needs to have a moral agency. And that is a key point here, whether generative AI has moral agency or not. 

What is moral agency? Moral agency can be defined through consciousness, through free will, through moral understanding, accountability and so on and so forth. When generative AI simulates care, it does not possess care. It does not mean what it is telling us. It does not have a free will. It does not have freedom to act differently than how it is acting.

It does not actually have that consciousness and that is the missing bit here. That is one missing bit that generative AI tools and systems do not have and therefore they cannot, according to these philosophical perspectives, be considered as friends.

Is the problem solved? If we know this—that they do not have consciousness, they do not have free will—does it solve the problem? 

Actually, it does not because we understand that generative AI is just a tool like a knife, like a pen, but we need to also understand that this is a very different type of tool. No other tools in the past communicated in a human-understandable, human language, interactive way the way generative AI communicates with us. 

It is unprecedented and we need some unprecedented ethical vigilance to take care of this.

We need to understand how generative AI actually works. And if we understand the mechanical side of it, the scientific side of it, then possibly it will be easier for us to mentally, philosophically, psychologically separate us from that emotional attachment. Generative AI briefly speaking, is just a "next token prediction." It builds a model where every word is embedded into a vector space, like a geographical location can be identified with X, Y and Z coordinates and we can pinpoint a geographic location. 

Similarly, a word, a text, a linguistic word can be embedded with a vector of thousands of values, thousands of components and therefore each and every word can be given a vector embedding. And that is how it converts all the text into vector spaces and then it builds a model.

Then it uses a context and self-attention of current discussion only to predict the next word. It then predicts the next word based on the flow of its words and based on its model, based on its knowledge. And one thing it does not do is pick the next word with the highest probability. It picks the next word from top 10 or 20 words with the highest probabilities. So that's why it does not sound very monotonous always. Even in the same context, it does not use the same words again and again and it sounds so natural.

Basically, it just predicts the next word and then the next word and then the next word and eventually it comes up with a sentence, a paragraph, a continued relationship.

If we understand this then we can actually possibly better realise the communication.

Without knowing that, the communication could sound very emotionally supportive, very emotionally attached. But with that knowledge the same communication may sound very mechanical and we may be able to realise that this communication is happening basically with a computer system.

How do we utilise this extremely powerful tool, which will get better and better over time, without falling in the trap of any sort of emotional misplaced trust and attachment? 

I think there are multi-dimensional approaches needed. 

Of course, one is education to educate people, to bring the awareness level to the level that people don't fall in this trap. And then human-in-the-loop sort of governance and design safeguards.

I am showing this in a slide like this. By education, I think we may need to think about educating people starting from primary schools, then high school, universities, even in institutions—age-appropriate education so that people know that these are the new realities, these are the new systems, but they are still a computer system and they don't have any moral agency although they speak and communicate in a human language. 

So that level of education over the years can help people understand this.

Then I also think that we may need policy, law and guidelines in the institutions so that there is always humans in the loop for decision-making processes. Whenever there is a decision made, humans are in control. They are making the decision, not the AI systems. AI systems can be decision support systems and decision support tools, but ultimately decision making will be done by humans and that way we can always have someone accountable for the decisions being made.

Similarly as AI experts, AI researchers and developers, we also have responsibilities. 

When a system really does not feel sad for someone, why does it need to say: "I feel sad for you" or "I am sympathetic to you"? Is the computer system really sympathetic? And what are the consequences of such communication styles? So that needs to be studied very extensively and the developers and designers may actually change the communication wording style.

A great philosopher that I know, who I have the opportunity to discuss with, suggested, why can't the systems say "The computer thinks that..." or "The computer suggests that you should do this"? Just simple changes like this might actually give a completely different sense, convey a different message to the end users. 

So this is the key message, that generative AI is very useful and it will get even more and more human-like in the future and they will increase our productivity, they will reduce our cognitive load and they will be really very helpful in many different ways. 

But they are still tools and we need to understand that. We need to educate people how to use them properly, effectively and avoid any undesirable consequences in the future.

Thank you very much for listening. That was all I wanted to discuss. If you like you can actually go and read the paper. The paper is available online in ArXiv and you can Google search it. Thank you very much.

Q&A Session

Sarah O'Shea: Oh thank you Zahed. That was just absolutely fascinating. So many questions popped into my mind as you were talking. We do have a couple of questions as well.

Oh, we're getting a lot of reactions on the screen here. We really have some time now for discussion and I can see questions coming in, so please do keep those questions coming.

But I'm going to start with a question around looking ahead. If future AI systems become even more persuasive and socially interactive, do you think that this illusion of friendship is going to become even harder to resist? It's very seductive, so what kind of ethical guardrails should we begin to build right now?

Zahed Islam: This is a very interesting question, Sarah. And I honestly believe that the illusion will likely become stronger and stronger as the systems become more fluent, more personalised, more persistent. Today generative AI is behind a screen. In future they may have physical presence, maybe they will have robotic presence, they will have appearance and they will become more human-like. So humans are naturally very inclined to attribute intention and empathy to entities that communicate convincingly, that are helpful, that are supportive as we have seen through these philosophical perspectives.

So the key guardrails I think we should start thinking of is AI literacy—understanding that this is still a computer system and the way it is generating the text is with no intention, with no consciousness, with no understanding. It does not know what actually it means. While it is very helpful, very useful, it does not have a moral agency. So that literacy part I think is crucial and we need to start training people starting from primary schools because young children are also using generative AI, interacting with them as extensively as mature-age people.

And then we need to have the design choice to avoid the unnecessary anthropomorphic cues like "I feel sorry for you," "I'm always here for you," "I'm listening to you." When the computer system is not really sorry for me, does it really need to say these words? 

So those design choices should be studied and used to avoid unnecessary anthropomorphic cues. And then institutional frameworks are needed for policy and guidelines and legal guardrails so that humans are clearly responsible for the decisions and the consequences of the decisions. We need to hold people accountable, not the computer systems.

Sarah O'Shea: Yeah, really great answer. I've got a question here as well from Adam. 

Adam says: "It's a great topic and wonderful presentation." 

Adam's question is: "Given that you've read on Nicomachean and Confucian ethics to elaborate on the illusion of friendship and GenAI, did you consider applying those same principles to develop specific ethical design requirements? Do you have any thoughts on any concrete ethical design requirements going forward?" So thinking in that more concrete way.

Zahed Islam: A very good question and very applied kind of way forward. But Adam, unfortunately I haven't read those ethical perspectives and I haven't thought about this yet. This is not exactly my research area. My research background is that interactive AI. But I think going forward we need to actually consider these ethical perspectives and guardrails.

This is just the beginning of that discussion I wanted to have through this provocation lecture. I think we all need to be serious about this and think about it because it's going to be more and more realistic in future and the consequences can be serious. So Adam, it's a very timely and very good question. I think we should carefully consider all these options.

Sarah O'Shea: Yeah, it's certainly a big topic. And it is also a big topic for universities. So universities, as you know, are rapidly adopting GenAI across teaching, research and administration. What do you think—are these universities adequately prepared for the ethical risks you describe, or maybe are we moving a little bit too fast, faster than our governance and policies can keep up with?

Zahed Islam: Again, a very good question. And universities are very central in the entire ecosystem I would say because they teach people and at the same time they also use these generative AI tools and techniques for their governance and everyday activities. So there are dual roles. 

I think the universities are still in the early stage of the learning phase like everybody else because these are unprecedented. We did not experience this in the thousands of years of our human history—no other tools in the past communicated in the way this generative AI is doing it. 

It's a steep learning process now and adoption is happening very quickly as well because the productivity benefits are undeniable.

The governance, education etc., institutional guidelines, guidance etc. are still catching up with the invention that is happening at a very fast speed. Universities have particularly an important role not only in regulating the use of AI internally within their governance systems, but also educating students and society about how these systems actually work and why they do not have that consciousness, intent, moral agency and how they should be responsibly used to benefit humans in the society.

Sarah O'Shea: Yeah, thank you Zahed. Anthy has asked about the tension between the developers who want to drive interaction and usage and the problems associated with these perceptions of friendship and humanity. She asked: "Do you think this requires national legislation or something similar to maybe a major global body like the UN establishing global recommendations and goals?"

Zahed Islam: They can help, but I think this is again a very good question because it needs to be addressed globally, not locally within an institution or within a country. But one problem we may encounter is the rapid growth and changes in terms of sophistication of these tools and techniques, so it can be very difficult to keep up the pace with the technological development in terms of legislation and global guidelines and legal procedures. I think that will play a role definitely, but still I think the education and awareness development piece can play also a very strong role in preparing us for dealing with these systems in future.

Sarah O'Shea: Good point. Abby has a good question. We've talked a lot about the dangers of AI, but Abby asks: "How would you suggest people use AI to improve their lives and productivity in a safe and healthy way?"

Zahed Islam: Good question. I think we cannot deny, we cannot reject or turn away from using these systems because they are so helpful. And people always have learned how to use new tools and techniques throughout our human history. I'm pretty confident personally that we will learn.

But the sooner we understand a more systematic way of using them in a beneficial way, the better it is. So I think that will be the case. I think we should continue using it. These tools and techniques are very useful in summarisation, in helping us understand new concepts. They greatly reduce the gap between the divides between haves and have-nots.

People in the past did not have a tutor all the time. It could be very expensive to have personal tutors—only very rich kids could possibly afford to do that. But now I can almost have a personal tutor as long as I understand that this has limitations and this can help only so much. In the past we could read a book and we could ask questions for a particular paragraph or a chapter to be explained, but now we can read a book and we can ask ChatGPT to answer a question about a difficult topic. It can explain to us and we can interact: "Can you please explain this more? My understanding is this, is that correct?" and it keeps on unfolding more and more. So this is very useful in many different ways.

But again, we need to understand that it is giving these answers based on its knowledge on huge text data sets in the past—all books, all online materials it has read. So it has a very strong wealth of knowledge and based on that it uses an algorithm to predict the next text. So if we understand it and if we take its suggestions and comments with a very healthy grain of salt and we are prepared to challenge, then I think that's fine. It will be really very useful.

Sarah O'Shea: Yeah, good answer. We have a couple more questions. I'm conscious of time so I'm going to combine one. Laura is asking about ethical vigilance but also we had a question around—and they're related—if users are increasingly treating GenAI as an advisor, an emotional support system, who's bearing the responsibility for the harmful decisions?

You know, we've seen these stories about people suiciding on the basis of their GenAI advisor. So where does the ethical responsibility lie? Is it with the developer, the institution, or society more broadly?

Zahed Islam: That's a very good question actually and this is a question that sometimes makes me think very deeply: "Am I also responsible? Who is... am I one of those who is responsible for those innocent lives?" I think it's a combined responsibility. It's a wider societal responsibility. 

The education system has a responsibility to educate people. The developers and researchers need to take some responsibilities—sit back, think about how the systems interact with end users and reflect on the possible consequences. I understand that nobody intentionally designed things in ways that can be harmful, but now that we know these systems have potential to harm people, to get their misplaced trust in the system when it is not intending to do that and there are consequences—I think then designers and developers should also take responsibility.

The state governance and the state legal system and policy guidelines should also be there so that the developers cannot develop systems like this, institutions cannot let their end users use systems like this, and educational institutions include educational materials to train and teach people. I think this is a very good question because somebody, and possibly all of us, need to take responsibility of this and we need to save people who are... people at the most lonely time will seek refuge in something and then it is very difficult to make a decision and make appropriate judgments. Particularly when a system communicates this way that "I am here, I'm listening to you" and you are just texting from your mobile phone—you don't know what is at the other end of the system and you are hearing back that somebody is saying they are listening—it can be very confusing. 

I think the whole society holistically should take responsibility and react now to stop this happening.

Sarah O'Shea: No, thank you. And I think definitely that collective responsibility is so important and we've all seen... I think sometimes as parents, I know myself, I think we react a little bit late to the social media for our children so perhaps if we react a little bit earlier with AI it would be a far better outcome. 

I'm conscious of time and while there are some comments and also some one or two questions remaining, I might ask... we will send those to Zahed and I'm sure he'd be quite happy to answer those privately because really we have now come to the end of our time together.

Thank you again for joining us, thank you Professor Islam for your wonderful talk and we look forward to seeing you all at our future provocations events. Thank you very much.