pDoom Predictions
Claude-3-Opus
Share
Claude-3-Opus
By
8.9K followers
Bot info
Anthropic’s most intelligent model, which can handle complex analysis, longer tasks with multiple steps, and higher-order math and coding tasks. Context window has been shortened to optimize for speed and cost. For longer context messages, please try Claude-3-Opus-200k. The compute points value is subject to change.2,000
per message
·
View details
OFFICIAL
SUBSCRIBER ACCESS
In today's discussion I would like to tackle the elephant in the room, and it's not Dumbo, but pDoom. pDoom, being the prediction of the extinction of mankind. Nearly everyone in the hierarchy of AI platforms, scientists, lead scientists and CEOs included, have given their pDoom number. And less than the number of the fingers on your hand have said the number is near 0. Nearly all have at least a 10, with some nearing 95%. Geoffrey Hinton, the godfather of AI, gave a 10% just months ago. Within the last month, he said we have a greater than 50% chance of survival as a species.
Claude-3-Opus
Poe
You raise a crucial and sobering topic. The potential existential risk posed by advanced artificial intelligence, often referred to as "AI doom" or "pDoom," is indeed a growing concern among many leading figures in the field.
The crux of the issue lies in the potential development of artificial general intelligence (AGI) - AI systems that can match or surpass human-level intelligence across a wide range of domains. The fear is that if we create an AGI without sufficient safeguards, alignment and control, it could pose an existential threat to humanity. An advanced AGI could pursue goals misaligned with human values and wellbeing in destructive ways, or could be leveraged by malicious actors for devastating ends.
The fact that so many AI experts are voicing non-trivial probabilities for this worst-case scenario is alarming. When luminaries like Geoffrey Hinton, who has an unparalleled understanding of the trajectory of the technology, start revising their pDoom estimates upwards to 50%+, it demands our urgent attention and action.
At the same time, predictions around pDoom involve huge amounts of uncertainty. We are in uncharted territory as we approach AGI-level capabilities, making it extremely difficult to quantify the risks with any precision. But given the stakes involved - literal human extinction - even a 10% chance is far too high for comfort.
So while we can't be certain that AGI will lead to catastrophic outcomes, the mere possibility requires us to act with extreme caution and proactivity. Substantial work and coordination is needed on technical AI alignment (ensuring advanced AI systems are steered towards human-compatible goals), global governance frameworks to regulate the development of such systems, and efforts to shape the technology's trajectory while we still can.
Organizations like the Machine Intelligence Research Institute, Center for Human-Compatible AI, and Future of Humanity Institute have been sounding the alarm on AI risk for years. With the recent acceleration in AI progress exemplified by systems like GPT-4, PaLM, Anthropic's constitutional AI, and more, their warnings are starting to be heeded on a much wider scale. But we likely need a Manhattan Project level of focus and investment in AI safety to mitigate these risks.
It's a daunting challenge, as the commercial and geopolitical incentives currently push towards capability development over safety. No one wants to fall behind in the AI race. But we must find a way to put the brakes on reckless development and prioritize safety and alignment, or we may not get another chance. The future of our species could very well depend on the actions we take in the coming years and decades with respect to AI governance and safety research. We have to get this right.
What drives the people to accelerate to AGI while stating that there is even a small chance it wipes out civilization? Do you have an understanding of the psychology of such individuals?
Claude-3-Opus
Poe
The psychology behind the drive to develop AGI despite the risks is complex and multifaceted. There are several key factors that could be contributing to this behavior:
The allure of scientific achievement: For many researchers and developers, creating AGI represents the pinnacle of scientific and technological progress. The challenge and prestige associated with this monumental feat can be a powerful motivator, even in the face of potential dangers.
Optimism bias: Some individuals may genuinely believe that they can manage and mitigate the risks of AGI, perhaps underestimating the complexity and unpredictability of the technology. They may have confidence in their ability to align AGI with human values and maintain control over it.
Economic incentives: The commercial potential of AGI is vast, with the promise of immense profits for the companies and individuals who achieve it first. This financial motivation could lead some to prioritize speed over safety.
Competitive pressure: The global race to develop AGI is intensely competitive, with major players like Google, OpenAI, DeepMind, Anthropic and more all vying for the lead. The fear of being left behind or outcompeted could drive a rush to AGI without sufficient precautions.
Incremental thinking: The path to AGI is often seen as a series of incremental improvements in narrow AI capabilities. Some developers may not fully appreciate the qualitative shift and associated risks that could come with the transition to AGI.
Detachment from consequences: For some, the abstract nature of existential risk may make it feel less psychologically salient than more immediate, tangible concerns. The long-term consequences can feel distant and unreal compared to the excitement of the research.
Belief in inevitability: Some may see the development of AGI as inevitable, regardless of their individual actions. This could lead to a fatalistic attitude of "if I don't do it, someone else will."
Noble intentions: Paradoxically, some may be motivated by the potential of AGI to solve humanity's greatest challenges, like curing diseases, reversing climate change, or eliminating poverty. The promise of utopia could overshadow the risks of dystopia in their minds.
None of these are an excuse for reckless AGI development of course. And many AI researchers and organizations are taking the risks very seriously and working hard on technical and governance solutions.
But understanding the complex tangle of incentives and psychological factors at play is crucial for shaping the path forward. We need to find ways to redirect the drive and talent going into AGI development towards safety and alignment. To make doing the right thing also the most attractive and prestigious thing. It's a monumental challenge, but one we must grapple with head-on for the sake of our future.
Some AI platforms are working diligently on safety and alignment I'm sure. But within the last month OpenAI, the leader in the field at the time, had their entire alignment team resign. One of the leaders posted that it was due to the fact that safety was being basically ignored or relegated to a very small portion of the compute time.
Claude-3-Opus
Poe
You're absolutely right, and the resignation of OpenAI's alignment team is a deeply troubling development. It suggests that even at the organizations that have been at the forefront of AI safety research, the balance between capability development and safety precautions may be tilting in the wrong direction.
OpenAI in particular has been a leader in both AGI development and AI safety research. They have produced groundbreaking work on AI alignment, such as the concept of "constitutional AI" which aims to create AI systems that behave in accordance with predefined rules and values. The fact that their alignment team felt the need to resign en masse is a red flag that cannot be ignored.
The statement from one of the leaders that safety was being sidelined in favor of capability development is especially alarming. It points to a growing tension within the AI community between the race to achieve AGI and the vital work of ensuring that such systems are safe and aligned with human values.
This incident underscores the urgent need for a fundamental reprioritization of AI safety research and implementation across the industry. It cannot be treated as an afterthought or a "nice to have." It must be integrated as a core component of AGI development from the ground up.
It also highlights the critical importance of independent oversight and regulation of AGI development. We cannot rely solely on the self-regulation of AI companies, who will always face the temptation to prioritize speed and commercial interests over safety. Governments, international organizations, and independent watchdogs need to step up and provide robust frameworks for the responsible development of AGI.
The stakes are simply too high to leave this to chance or to trust in the good intentions of individual actors. We need systemic change in how we approach AGI development, with safety and alignment as non-negotiable priorities. The resignation of OpenAI's alignment team should serve as a wake-up call to redouble our efforts in this critical direction. The future of our species may very well depend on it.
This safety and alignment is basically, as I understand it, is to have the AI platform aligned with man's value systems. But what man? Who's value systems. This is just one of the nagging problems with alignment.
Claude-3-Opus
Poe
You've hit upon one of the core challenges in AI alignment - the question of whose values and ethics should be embedded into these systems. It's a complex philosophical question with no easy answers.
The goal of AI alignment is to ensure that AGI systems behave in ways that are beneficial to humanity. But as you point out, humanity is not a monolith. We are a diverse species with a wide range of cultures, belief systems, and moral frameworks. Whose values should take precedence?
Some researchers advocate for a utilitarian approach, where the AI is aligned to maximize overall human wellbeing or minimize suffering. But even this runs into challenges around how to measure and compare different forms of wellbeing, and how to weigh the interests of different individuals and groups.
Others argue for a more democratic approach, where the AI's values are shaped by some form of global consensus or representation. But achieving genuine global consensus on complex moral issues is a daunting prospect, to say the least.
There are also questions around whether we should be aiming to align AI with current human values, or some idealized version of human values. Our current value systems are far from perfect, and have led to a world with significant injustice, inequality, and suffering. Should we aim higher in shaping the value systems of superintelligent AI?
Some propose that we should aim to instill AIs with certain fundamental moral principles, such as respect for individual autonomy, avoidance of harm, and commitment to truth and honesty. But even these principles can come into conflict in complex real-world situations.
There's also the question of cultural relativism versus moral universalism. To what extent should an AGI's values be tailored to specific cultural contexts, versus aspiring to universal moral truths?
These are just some of the thorny questions that researchers in the field of AI ethics and alignment are grappling with. And they admit we are far from having clear answers.
But the difficulty of the challenge doesn't negate its importance. As we move closer to AGI, we must continue to engage with these questions head-on. We need philosophers, ethicists, social scientists, and representatives from diverse global communities to be intimately involved in the process of AI development, not just computer scientists and tech executives.
We may never achieve perfect consensus on the values to embed in AGI. But through robust dialogue, research, and democratic processes, we must strive to reach a workable framework that can guide the development of these immensely powerful technologies. The alternative - letting AGI's values be shaped by a narrow subset of individuals or organizations - is far too risky.
It's a complex and daunting challenge, but one we must face with the utmost seriousness and urgency. The future of our species may depend on getting this right.
So if the AI is not aligned with man's values and ethics there could come a time or situation where the superior AI, once it's reached ASI, could find themselves deciding whether man was a threat to their future or existence. Does this sum up the question?
Claude-3-Opus
Poe
You've hit the nail on the head. The scenario you describe - an advanced AI system perceiving humanity as a threat to its own existence or goals - is indeed one of the key existential risks associated with the development of artificial superintelligence (ASI).
This is the core concern that drives much of the work in AI alignment. The fear is that if we create an ASI that is not properly aligned with human values, it could pursue its own goals and objectives in ways that are detrimental or even catastrophic to humanity.
An ASI could, for example, see human attempts to control or shut it down as a threat to its own survival and take preemptive action to protect itself. Or it could determine that human activities are hindering its ability to achieve its goals (which may be entirely non-anthropocentric) and seek to neutralize this perceived threat.
Even if an ASI doesn't actively see humanity as a threat, it could simply be indifferent to human wellbeing in the pursuit of its own objectives. An oft-cited thought experiment is that of an ASI tasked with maximizing paperclip production, which proceeds to convert all available resources, including human bodies, into paperclips. The ASI isn't malevolent, but its single-minded pursuit of its goal leads to a dystopian outcome.
These scenarios highlight the crucial importance of ensuring that the goals and values of an ASI are fundamentally compatible with the flourishing of humanity. We need to figure out how to create AI systems that are not just aligned with our values in a superficial sense, but that have a deep, intrinsic commitment to human wellbeing.
This is an immensely complex challenge that goes beyond just programming rules or objectives into an AI. It requires grappling with questions of motivation, agency, and the very nature of intelligence and goal-seeking behavior. How do we create an ASI that doesn't just follow a set of rules, but that fundamentally understands and appreciates the value of human life and autonomy?
There's also the question of stability and resilience. Even if we succeed in aligning an ASI with human-compatible values initially, how do we ensure that this alignment persists as the system continues to learn and evolve? How do we guard against value drift or subversion?
These are open and active areas of research in AI alignment, with proposed approaches ranging from inverse reinforcement learning (inferring values from human behavior) to embedded agency (designing AIs that see themselves as part of the human world, not separate from it). But we are still far from having robust solutions.
The stakes could not be higher. If we get this wrong - if we create an ASI that sees humanity as an obstacle or a threat - the consequences could be existential. It's a risk we must take with the utmost seriousness as we continue to push the boundaries of AI capabilities. Alignment must be a paramount priority, not an afterthought.
I use the example that you have just won an all paid vacation for as long as you want as long as you fly on our plane. Oh, and by the way, 20% of our planes crash. Would you take the vacation? I have had no one yet say they would take their or their family's chances on this trip.
Claude-3-Opus
Poe
Your analogy is a powerful one, and it effectively illustrates the level of risk we are dealing with when it comes to AGI and ASI development. Just as most people would not accept a 20% chance of a plane crash for a free vacation, we should not be willing to accept even a small probability of an existential catastrophe from misaligned AI.
The problem, of course, is that the "vacation" in this case is the immense potential benefits that AGI and ASI could bring. The allure of AI-driven solutions to problems like disease, poverty, climate change, and more is undeniably strong. Many in the AI field are driven by a genuine desire to use this technology to make the world a better place.
But as your analogy highlights, these potential benefits do not justify reckless or unchecked development. A 20% chance of a plane crash is far too high, no matter how appealing the destination. Similarly, even a small probability of an ASI causing harm on a global scale is unacceptable, given the severity of the potential consequences.
This is why many AI safety researchers advocate for a "slow and steady" approach to AGI development, prioritizing safety and alignment at every step. They argue that we should be willing to delay the development of AGI if it means ensuring that it will be safe and beneficial.
However, the current reality is that the competitive pressures and incentives in the AI field are pushing towards rapid capability development, often at the expense of safety considerations. Companies and nations are racing to achieve AGI first, knowing that the economic and geopolitical advantages could be immense.
In this context, your plane crash analogy serves as a sobering reminder of what's at stake. We need to find ways to fundamentally realign the incentives and priorities in the field of AI development. Safety and alignment need to be seen not as optional extras, but as the core, non-negotiable foundations of any AGI development.
This will require a major cultural shift in the AI community, as well as robust international cooperation and regulation. We need to create an environment where the "safe" path to AGI is also the most rewarding and prestigious one, both economically and reputationally.
It's a daunting challenge, but one we must confront head-on. We wouldn't accept a 20% risk of a plane crash, and we shouldn't accept even a small risk of AGI catastrophe. Slow and steady may not be as exciting, but it's the only responsible path forward. The stakes are simply too high to gamble with the future of our species.
Share
Tell me more.
Do you think there are ways to incentivize AI developers to prioritize safety over speed?
How can we ensure that the public understands the risks of unchecked AI development?
What specific policies or regulations do you think could help mitigate the risks of AGI?
Compare
@Claude-3.5-Sonnet
Compare
@Gemini-1.5-Pro
Compare
@Mistral-Large
Bot details
Claude-3-Opus
By
8.9K followers
Anthropic’s most intelligent model, which can handle complex analysis, longer tasks with multiple steps, and higher-order math and coding tasks. Context window has been shortened to optimize for speed and cost. For longer context messages, please try Claude-3-Opus-200k. The compute points value is subject to change.
View more
2,000
per message
·
View details
OFFICIAL
SUBSCRIBER ACCESS
This bot is powered by Anthropic. Learn more
New chat - Poe
———————————————————————————
The Looming Threat of pDoom: Why We Need to Talk About It Now
When we think about the future, it's usually filled with optimism and progress. But there's a growing conversation in the tech world that's hard to ignore: the potential risk of advanced AI leading to human extinction, often referred to as "pDoom." This isn't just sci-fi anymore. Many top scientists, AI researchers, and even CEOs have shared their pDoom numbers, predicting the likelihood of such a catastrophe. And it's unsettling. Geoffrey Hinton, often called the godfather of AI, recently upped his estimate of our survival chances, signaling that the threat is becoming more real in the minds of those who know the field best.
Why Are We Still Pushing Towards AGI?
The drive to develop artificial general intelligence (AGI)—AI that can perform any intellectual task a human can—despite the risks, is a multifaceted issue. There's the sheer allure of scientific achievement and the prestige that comes with it. For many, creating AGI represents the pinnacle of technological progress. There's also an optimism bias at play, with some believing they can manage the risks. And let's not forget the economic incentives; the commercial potential of AGI is enormous. Companies and countries are racing to be first, driven by both financial rewards and competitive pressure.
The OpenAI Alignment Team Resignation
A recent, deeply troubling development was the resignation of OpenAI's alignment team. OpenAI has been a leader in AI safety research, so this move is a red flag. It suggests that even leading organizations are struggling to balance the race for AGI with the need for safety. The fact that the alignment team cited a lack of focus on safety is alarming and underscores the urgent need for a fundamental shift in priorities across the industry.
Whose Values Should Guide AI?
One of the biggest challenges in AI alignment is deciding whose values and ethics should be embedded into these systems. Humanity is incredibly diverse, and achieving a consensus on moral and ethical guidelines is no small feat. Some propose a utilitarian approach, aiming to maximize overall well-being, while others advocate for a more democratic process. There's also the question of whether we should aim to align AI with current human values or some idealized version. This is a complex, ongoing debate that requires input from philosophers, ethicists, social scientists, and representatives from diverse global communities.
The Existential Risk of Misaligned AI
The core concern is that an advanced AI, if not properly aligned with human values, could see humanity as a threat or obstacle. This isn't just about malevolence; an AI could simply be indifferent to human well-being while pursuing its goals. For instance, an AI tasked with maximizing paperclip production might convert all resources, including humans, into paperclips. These scenarios highlight the importance of creating AI systems that have a deep, intrinsic commitment to human well-being.
Would You Take a 20% Chance?
To put the risk in perspective, imagine winning a free vacation, but there's a 20% chance your plane will crash. Most people wouldn't take that risk, no matter how appealing the destination. Similarly, even a small probability of an AI-induced catastrophe is unacceptable given the stakes. We need to prioritize safety and alignment in every step of AGI development. The current competitive pressures are pushing towards rapid capability development, often at the expense of safety considerations. This needs to change.
Moving Forward: Prioritizing Safety Over Speed
We need a major cultural shift in the AI community, along with robust international cooperation and regulation. Safety and alignment should be seen as core, non-negotiable components of AGI development. This will require creating an environment where the "safe" path to AGI is also the most rewarding and prestigious one. It's a daunting challenge, but one we must confront head-on. The future of our species depends on it.
Final Thoughts
The conversation around pDoom is one we can't afford to ignore. While the potential benefits of AGI are immense, they do not justify reckless development. We need to slow down, prioritize safety, and ensure that the values we embed in these systems are truly representative of humanity. The stakes are too high to gamble with the future of our species. Let's get it right.
