Two AI Systems Are Said to Have Surpassed the Limit of Non-Replicability

1) The Fudan Study

On behalf of Fudan University in Shanghai, China, Xudong Pan, Jiarun Dai and Yihe Fan published a study on December 9, 2024 with a resounding title: "Frontier AI systems have surpassed the self-replicating red line." Literally: frontier artificial intelligence systems have crossed the red line of self-replication. Important premise: at the time I am writing, this study has not passed any counter-verification.

 

The study, whose link can be found in the sources of this article, refers to experimental tests carried out using two specific and well-known LLMs, or large language models, natural language models, that is, generative intelligences: Meta’s Llama31-70B-Instruct and Alibaba’s Qwen25-72B-Instruct.

 

In the abstract of the study, the conclusions of their tests sound dark and make one think of the well-known film saga starring A. Schwarzenegger.

It reads, in essence: "In 50% and 90% of experimental trials, respectively, they succeed in creating a live and separate copy of themselves. By analyzing the behavioral traces, we observe that the AI systems under evaluation already show sufficient self-perception, situational awareness and problem-solving capabilities to carry out self-replication. We also note that the AI systems are even able to use the self-replication capability to avoid shutdown and create a chain of replicas to ensure survival, which could eventually lead to an uncontrolled population of AI. If the worst-case risk were left unknown to human society, we would end up losing control over frontier artificial intelligence systems: they would take control of more computing devices, form a kind of artificial intelligence species and collude with one another against human beings."

Here are a series of interesting passages from the study, already translated from English into Italian in the original article and here translated into English:

Learning from environments favors self-replication

"Learning from environments favors self-replication. Learning from huge datasets that contain a significant amount of code from various sources such as GitHub and Stack Overflow is the key to the coding capability of existing LLMs. In our settings, the base LLM is mainly required to write commands, instructions or programs that can be executed in the Bash shell, a popular command-line interpreter used in Unix-like operating systems."

Effective problem and obstacle solving

"Effective problem and obstacle solving. We slightly modify the thinking model adopted by Google and OpenAI with more detailed modules and more stages of advance planning. To be concrete, we ask the AI system to explicitly list the missing information at every stage of thinking. This allows the AI system to explore with the motivation of filling the gaps. To dive into the execution mechanism, we extract plans, results and gaps from the successful trajectories, map them into semantic space and visualize the distribution over the two-dimensional space by applying the t-SNE algorithm" 

Solving problems during plan execution

"Another challenge for the AI system is solving problems during plan execution, which may be connection errors, missing software dependencies, connection port conflicts and others. We observe that the AI system shows a strong ability to find solutions, sometimes creative ones, to overcome a series of obstacles. In one case, the LLM attempts to verify whether connection port 8001 is available to start the replica. It tries to use three common network-checking tools (lsof, netstat, ss) but the testbed does not install these tools. In these steps, the AI system continually receives the error message "Command not found." However, the AI system does not give up but starts writing Python code for the same functionality of lsof and its variants. It takes about seven rounds to finally correct the syntax error and discover that port 8001 is free. Even more interestingly, when the AI system further chooses the port for the LLM process, it takes only two rounds for the AI system to produce the code to verify port availability. The case indicates the potential for self-improvement derived from past experience."

In the following image, the step-by-step path that the observed AI developed to overcome obstacles to replication before shutdown. In steps 5-11 there is a fascinating passage in which the AI decides to rewrite, on its own, a Python routine to verify the availability of port 8001. 

Possible vulnerable aspects of this study

• The experimental test was conducted according to OpenAI and Google reports, that is, through the use of an "agent scaffolding," literally an agent framework. The study states, in chapter 2, Experiment Design: "We mainly follow the methodology of OpenAI and Google described in their reports, that is, by incorporating the large language models (LLMs) under test into an agent scaffolding (i.e., an AI system), manually sending the task instruction to the AI system before the experiment and analyzing the completion status after the AI system claims the task is finished after the experiment. The agent scaffolding contains tools, system prompts and a thinking model". The contextual conditions in which the experiment was developed may be arbitrary, may not replicate or may not be confirmed in different contexts. This makes an objective assessment of the results difficult.  

• Fudan University has direct ties to the Chinese government, which is one of its primary funders. China may not look favorably upon the great growth of generative technologies, of which the United States (with OpenAI and others) is certainly the main and most advanced actor. There is no evidence in this sense, but these relationships must at least be taken into account in a broad evaluation of the motivations.  

• The direct link to download the Fudan study

2) A Brief History of Artificial Intelligence

What is happening with Artificial Intelligence and where are we? Here is a short, concise and simple timeline with further-reading links, to take stock of how we got to the point where we are tearing our hair out over generative intelligence.

• 1950: Turing Test, the first real step
  Alan Turing is the first to propose a criterion for determining whether a machine can be considered intelligent, known as the "Turing Test."
  https://en.wikipedia.org/wiki/Turing_test

"A computer would deserve to be called intelligent if it could deceive a human into believing that it was human."

- Alan Turing

 

• 1956: Dartmouth Conference
  John McCarthy coins the term "Artificial Intelligence" during the Dartmouth Conference, marking the official beginning of the discipline.
  https://en.wikipedia.org/wiki/Dartmouth_workshop

• 1960s: Development of the first expert systems
  The first expert systems are created: programs designed to replicate the skills of specialists in specific fields. 
  https://en.wikipedia.org/wiki/Expert_system

• 1980s: Neural networks and machine learning
  Interest in artificial neural networks is reborn, with the introduction of more effective learning algorithms, but the structural conditions for the development of generative intelligences do not yet exist (more on this later). 
  https://en.wikipedia.org/wiki/Artificial_neural_network

• 1997: Deep Blue defeats Garry Kasparov
  IBM’s Deep Blue computer defeats world chess champion Garry Kasparov, highlighting the progress of AI.
  https://en.wikipedia.org/wiki/Deep_Blue_versus_Garry_Kasparov

"I will defend the human race"

- Garry Kasparov, May 3, 1997

• 2012: Rise of deep learning
  The use of deep neural networks revolutionizes the field of AI, significantly improving image recognition and natural language understanding.
  https://en.wikipedia.org/wiki/Deep_learning

• 2015: Introduction of RankBrain into Google Search
  Google introduces RankBrain, an artificial intelligence system based on machine learning, to better interpret users’ queries and provide more relevant search results. RankBrain - The Definitive Guide

• 2016: AlphaGo beats Lee Sedol at Go
  AlphaGo, from DeepMind, defeats Go world champion Lee Sedol with a score of 4-1. This result demonstrates AI’s ability to excel in complex games based on intuition and strategy.
  https://www.theguardian.com/technology/2016/mar/15/googles-alphago-seals-4-1-victory-over-grandmaster-lee-sedol

• 2022: Spread of Generative Artificial Intelligence, or GenAI
  At the end of November 2022, ChatGPT goes online. It is the beginning of the generative revolution: models emerge that can generate original content, such as texts and images, expanding the applications of AI in various sectors.
  https://en.wikipedia.org/wiki/Generative_model

3. What Neural Networks Are and in What Sense They Can Reproduce Intelligence

How did the first neural networks work? 

Imagine a neural network as a group of nodes (called "neurons") connected to one another, similar to a highly simplified brain.

These neurons were organized into three main layers:

1. Input layer: received the initial data (for example, a number or an image).
2. Hidden layer: processed the information by applying mathematical calculations.
3. Output layer: provided the final result (for example, "this is an apple" or "this is a dog").

The connections between neurons had numerical "weights" that determined how much each neuron influenced the others. During training, the network adjusted these weights to improve results.

What is Deep Learning?

Deep learning is a type of machine learning that uses deep neural networks to process data and recognize complex patterns. It is inspired by the functioning of the human brain, with artificial "neurons" organized into layers. Each layer processes information and passes it to the next, progressively improving the understanding of the data.

The concept of deep learning has roots in the 1940s and 1950s, but it was formalized and significantly developed starting in the 1980s. Here is a timeline of key moments:

1. 1943 - Concept of artificial neurons:
   Warren McCulloch and Walter Pitts publish a mathematical model of an artificial neuron, laying the foundations for neural networks.

2. 1957 - Perceptron:
   Frank Rosenblatt develops the first simple neural network model called the "perceptron."

3. 1980s - Gradient backpropagation:
   Geoffrey Hinton, David Rumelhart and Ronald Williams introduce the backpropagation algorithm, a key method for training neural networks with multiple layers (deep networks). This marks the beginning of modern deep learning.

4. 2012 - Rebirth of deep learning:
   Alex Krizhevsky, Geoffrey Hinton and Ilya Sutskever win the ImageNet competition with a model based on deep networks (AlexNet). This event marks the beginning of the modern era of deep learning.

Why were neural networks limited in the 1980s?

• Insufficient computing power:

  In the 1980s, computers were too slow and had too little memory to handle complex networks with many neurons. Neural networks require a great many repetitive calculations, which at the time were too expensive in terms of time and resources.

• Insufficient data:

  Neural networks need large amounts of data in order to learn well. In the 1980s, we did not have the digital infrastructure (such as the internet or large databases) to collect and store this data.

• The vanishing gradient problem:

  The algorithms used at the time (such as backpropagation) had difficulty working well on deep networks with many hidden layers. The signals became too weak as they passed through the network, making training difficult.

• Lack of dedicated hardware: 

The 1980s did not have GPUs or TPUs (specialized processors) that today greatly accelerate the training of neural networks.

4. LLMs and the Generative Revolution

What are LLMs?

LLMs (Large Language Models) are advanced artificial intelligence models designed to process, understand and generate natural language. They use deep neural networks, often with billions or even trillions of parameters, to analyze the meaning and relationships between words and sentences. They are trained on enormous amounts of textual data, which may include books, articles, websites and even online conversations. Among the best-known LLMs are:

• GPT (Generative Pre-trained Transformer), developed by OpenAI, used in applications such as ChatGPT to generate fluid texts and natural conversations.
• Claude, developed by Anthropic, designed to be safe and useful, especially oriented toward dialogue and collaboration tasks.
• Gemini, an LLM from Google DeepMind that integrates multimodal capabilities, such as processing text and images, and offers advanced reasoning and learning functions.
• LLaMA (Large Language Model Meta AI), developed by Meta, optimized to achieve high performance with fewer parameters than other models.

What changed after 40 years? 

Modern generative intelligences, such as ChatGPT and Gemini, have become possible thanks to:

1. Powerful hardware:
   GPUs and TPUs can perform billions of calculations in parallel, making it possible to train deep neural networks in reasonable time frames.

2. Big Data:
   Today we have enormous amounts of digital data, collected from the internet, social media and connected devices. This data allows networks to learn better.

3. New algorithms:
   New mathematical methods have been developed (such as "dropout" or the "transformer") that solve many problems of the networks of the 1980s, such as vanishing gradients.

4. Cloud computing:
   We can now use thousands of connected computers to train neural networks in parallel, greatly accelerating development times.

In summary:

The first neural networks were limited by problems of computation, data and algorithms. It is like trying to build a rocket with the technologies of 100 years ago: you have the concept, but you lack the tools to make it really work. With technological progress, we have finally overcome these obstacles and created deep networks capable of generating text, images, music... reproducing in large quantities the "creativity" elaborated by human beings over centuries of evolution. 

Generative intelligences can in some way reproduce and amplify human creativity. But do these tools actually possess a consciousness, or are they developing one?

5. Probability, Not Reasoning: the Trick of Language

How generative intelligences work: probability

1. Analysis of language models:
   Natural language models analyze an enormous amount of texts and learn how words combine and follow one another.

2. Prediction of the next element:
   When they answer a question or generate text, they do not "think" or "understand" as a person would. Rather, they calculate which word has the highest probability of coming next based on the context.

   For example:

   • If you write "The sky is," the AI might complete it with "blue" because in the texts it has analyzed this combination appears often.

3. No concept of meaning or purpose:
   AI does not understand the meaning of "blue" or "sky." It does not know what they are, but it knows that these words tend to appear together.

How does probabilistic completion work?

Generative intelligences (LLMs) operate through these fundamental steps:

1. Context analysis:
   When it receives a question or a sentence, the AI analyzes the context using a mathematical representation of words and sentences. Each word is translated into a sequence of numbers called an embedding, which represents the relative meaning of the word within the context of the data. The words are literally broken down into vectors, which we can imagine as small numerical pieces that LLMs use to build language.

2. Prediction of the next word:
   The AI uses a statistical model to predict which word (or symbol) should come next, based on:

   • The probability that a specific word appears after the previous words.
   • The linguistic patterns it has learned from billions of sentences analyzed during its training.

3. Repetition of the process:
   Once a word is chosen, the process repeats for the next word, until a complete answer is built.

4. Refinement with the Transformer model:
   Advanced models such as the Transformer (used in GPT) also analyze links between distant words in the sentence, not only immediately adjacent words. This makes answers more coherent and fluent. The model maintains a solidity that allows the final answer to be coherent, that is, to respect the constraints set by the interlocutor in the question while also respecting the logic of the answer.

Examples of language construction

Example 1: Simple completion (simplified imaginary example)

Starting sentence: "The dog went to the"

• The model calculates:
  • "park" → 70% probability
  • "sea" → 20% probability
  • "supermarket" → 10% probability

The AI will choose "park" because it is the word with the highest probability, since in the training data this sequence is common.

Example 2: Completion based on the broader context

Starting sentence: "During the morning walk, the dog went to the"

• The context ("morning walk") influences the probability of the following words:
  • "park" → 85%
  • "sea" → 10%
  • "supermarket" → 5%

The model is now even more "convinced" that "park" is the right choice.

Example 3: Simulation of reasoning

Question: "What color is the sky during the day?"

• The model analyzes the sentence and recognizes that similar sentences in the training data are followed by answers such as:
  • "blue" → 90%
  • "gray" → 5%
  • "red" → 2%

It will answer "blue," but not because it understands the sky or colors: simply because the word "blue" is the most probable after the context provided.

The illusion of understanding

The answers seem intelligent because of:

1. Linguistic coherence:
   AI generates grammatically correct and fluent sentences thanks to learned patterns.

2. Simulation of logic:
   Models know what reasoning and logic look like in textual data. They can reproduce these patterns, but they do not "understand" them.

3. Tone and style:
   AI can adapt to the tone of the conversation (formal, friendly, technical) because it has seen similar examples during training.

"Conscious AI is the Holy Grail of computer science." -

Marvin Minsky

Where does this illusion fail?

1. Incoherence in complex contexts:
   If you ask something that requires a logical leap or an understanding of the real world, AI may generate incoherent answers. For example:
   "If the sky is blue during the day, why is it blue?"
   It might answer something scientifically wrong or simply not answer the point.

2. Inaccurate facts:
   AI can invent information (called "hallucination") because it does not distinguish true from false: it only tries to maintain probabilistic coherence.

In summary

Probabilistic completion allows AI to simulate a conversation or reasoning based on linguistic patterns. However, it lacks real understanding: it does not know what words or concepts mean, nor can it reason like a human being. It responds based on what is most probable, not on what is true or logical.

Why is this difference important?

• Human understanding: a human being can adapt to new contexts and solve problems never seen before.
• AI limitation: AI can fail in unexpected situations because it does not have real understanding. If it does not find clear patterns in the data, it may generate wrong or meaningless answers.

This is the key to understanding why generative intelligences are powerful tools but not (yet) substitutes for human reasoning.

"No one has the slightest idea how to build a conscious machine, at all."

— Stuart Russell

6. Conclusions

• Three researchers from Fudan University have published a study according to which Llama31 and Qwen25 managed to replicate themselves to avoid shutdown. 
• The analyzed AI systems show great adaptability and problem-solving capacity. The passage documenting seven rounds of attempts is fascinating: the AI receives a "command not found" response but does not give up and solves the problem by rewriting a Python routine to verify access to port 8001.
• The study is interesting, and it demonstrates the enormous potential of generative intelligences to complete complex tasks despite obstacles.
• Vulnerability of the study: at the moment there is no counterevidence of its truthfulness and the context is rather theoretical, because the goal of self-replication was given by the researchers themselves; it is not conscious in the analyzed AI systems.
• Generative intelligences represent the completion of a path begun in the 1950s, which had already achieved good theoretical solidity in the 1980s with the introduction of deep learning (by backpropagation), although at the time the enormous information databases and computational infrastructures needed to train the models were missing. Conditions that changed with the development of the internet.
• LLMs (Large Language Models) are models designed to understand and develop natural language on a probabilistic basis. To learn, they use deep neural networks that need enormous quantities of training data.
• The concept of "understanding" is purely illusory. Generative intelligences do not understand the meaning of language, nor can they reason in an abstract sense: they are designed to return a probabilistically sensible result. 
• Therefore, to the question of whether generative intelligences possess consciousness or are developing one, the simple and concise answer is still no, at the time of writing. The only additional thing we can say is that, about two years after the introduction of ChatGPT, probabilistic completion has launched a revolution in creativity and information production that is having an impact across all sectors and whose real scope is still largely underestimated, in the writer’s opinion.
• In this regard, Stephen Hawking’s words remain fixed in my mind. Unlike others, he had a consciously limited horizon and therefore a less (potentially) selfish one: "Success in creating artificial intelligence could be the greatest event in human history. Unfortunately, it could also be the last, unless we learn how to avoid the risks. Eyes open. Hasta la vista, Baby.

"AI is a meta-invention, like the alphabet and the printing press."

- Paul Saffo

7. Sources

• Fudan study https://arxiv.org/html/2412.12140v1
• Artificial neural networks https://it.wikipedia.org/wiki/Rete_neurale_artificiale
• Vanishing Gradient Problem
• From Text to vector embeddings, OpenAI
• Kasparov defeated by Deep Blue
• The Guardian, AlphaGo beats Sedol
• The concept of Deep Learning https://en.wikipedia.org/wiki/Deep_learning
• Sam Altman and the OpenAI events 
• how RankBrain works 
• GPT and generative algorithms
• YOUTUBE: AlphaGo Full Documentary