LLM 大语言模型论文解读原理

OpenAI技术博客《An Alien Mind》解读

Mr.噤若寒蝉2026-09-08 19:40

An Alien Mind

文章信息

来源：https://openai.com/index/an-alien-mind
出版来源：OpenAI 官方博客
作者：Jakub Pachocki，OpenAI Chief Scientist
发布时间：2026-09-06

原文与翻译

P001

Original

In mid-2023, within the “RLSlow” research project, we saw the first results that gave us confidence that we will be able to scale the training of reasoning models, unlocking the capability of pretrained models to form their own chains of thought.

译文

2023 年年中，在“RLSlow”研究项目中，我们看到了首批成果。这些成果让我们相信，我们将能够扩大推理模型的训练规模，从而释放预训练模型形成自身思维链的能力。

P002

Original

Szymon and I spent that night at the office, thinking not about the incredible benchmark numbers, products, or scientific results that this technology will deliver - but rather, trying to process the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime, and we already see the shape of these systems; wondering how to alert people to the significance of this.

译文

那一夜，Szymon 和我待在办公室里。我们想的不是这项技术将带来的惊人基准成绩、产品或科研成果，而是在努力消化一个令人警醒的事实：有生之年，我们真的会见到在实质意义上比人类更聪明的机器，而且这些系统的轮廓已经显现；我们还在思考，怎样才能让人们意识到此事的重要性。

P003

Original

Three years later, reasoning language models are a rapidly growing part of the economy and starting to push the boundaries of science. They are able to operate computers and graphical interfaces, collaborate with people and each other, and carry out research projects. They are also transforming the landscape of computer security, and in that present clear new dangers.

译文

三年后，推理语言模型已成为经济中迅速增长的一部分，并开始拓展科学的边界。它们能够操作计算机和图形界面，与人类及其他模型协作，并开展研究项目。它们也在改变计算机安全的格局，而这带来了明确的新危险。

P004

Original

A lot of new research happened in this period, and our understanding of these systems is again a little different than it was in 2023. Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement. If AI development continues along its current path, the systems we’ll see in the next few years are likely to represent further capability jumps of equal or larger magnitude, and to increasingly drive their own development.

译文

这段时间出现了大量新研究，我们对这些系统的理解也再次与 2023 年时略有不同。根据内部结果，我强烈预期，这种进步速度可能会延续到递归自我改进阶段。如果 AI 继续沿当前路径发展，未来几年出现的系统很可能会实现幅度相当或更大的能力跃升，并日益推动自身的发展。

P005

Original

This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required.

译文

这是一个要求我们极度谨慎的时刻。我担心，没有人为机器智能持续快速上升的后果做好准备。OpenAI 将继续寻求对齐和监控方面的技术解决方案，建设防御系统，并在必要时单方面暂停进一步扩大规模；不过，我认为还需要更广泛的干预。

Intellect we don’t fully understand

P006

Original

At a high level, progress in machine intelligence is driven by increasing computational power. We at OpenAI deeply internalized this around 2017, after seeing consistent returns to scaling across multiple research projects[1]. As a result, we sought out access to much more compute than we had originally planned, and increasingly oriented our research around a small number of very scalable directions.

译文

从宏观层面看，机器智能的进步由不断增长的计算能力推动。大约在 2017 年，我们在多个研究项目中看到扩大规模持续带来回报后，OpenAI 内部深刻认识到了这一点。因此，我们开始寻求远超原计划的算力，并逐渐把研究集中到少数几个极易扩展的方向上。

P007

Original

We believed that was the only way for us to be at the frontier of AI research, and influence the impacts of AGI.

译文

我们认为，这是让我们立于 AI 研究前沿并影响 AGI 所带来后果的唯一途径。

P008

Original

There are new algorithms that have been developed along the way, new feats of ingenuity from teams and individual researchers. I see them largely as discoveries along the path of scaling; the science of deep learning is still nascent, and meaningful algorithmic progress tends to correlate with access to compute. If you zoom out to a multiple-year horizon, AI is continuing to become more intelligent as it is scaled to larger computers.

译文

一路走来，人们开发出了新算法，团队和研究者个人也展现了新的巧思。在我看来，它们大多是沿着规模化道路取得的发现；深度学习这门科学仍处于萌芽阶段，而有意义的算法进步往往与算力获取能力相关。如果把视野拉长到数年尺度，就会看到 AI 随着扩展到更大型的计算机上而持续变得更加智能。

P009

Original

And, in line with Ray Kurzweil’s predictions from the end of the XXth century, we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.

译文

而且，与 Ray Kurzweil 在 20 世纪末所作的预测一致，我们如今正处在计算史上的这样一个时刻：机器智能开始以足以带来变革的方式超越人类智能。

P010

Original

AI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute. This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behavior.

译文

AI 与其说是被设计出来的，不如说是被培育出来的——从最直接的意义上说，它是在难以想象的算力上反复执行一个简单优化步骤的产物。由此产生了一个极其复杂的系统，它能处理抽象概念，也能模拟人类行为的某些侧面。

P011

Original

We can discover various insights about little mechanisms that emerge within this system, in a process similar to neuroscience - and, similarly to neuroscience, its overall action evades a description we can fully understand.

译文

我们可以发现这个系统内部涌现出的种种微小机制，获得不同层面的认识，这一过程类似于神经科学；也同神经科学一样，它的整体运作仍无法用一种我们能够完全理解的方式描述。

P012

Original

The study of deep learning-based AI is largely an experimental science. We put a lot of effort into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.

译文

对基于深度学习的 AI 的研究，在很大程度上是一门实验科学。我们投入大量精力构建有原则依据的算法并提出可检验的预测，但从根本上说，我们的大规模训练运行就是实验，其结果有时会出乎意料。此外，系统能力越强，结果就越难解释。

P013

Original

This is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify. We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years.

译文

当前算法提升易测量能力的速度，通常快于提升那些难以客观量化的能力，这让问题更加复杂。我们花费大量时间，试图理解能力如何泛化，以及应优先推进哪些技能，才能让模型获得未来几年最为相关的能力。

P014

Original

For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later.

译文

例如，我们认为，只要投入更多专门精力，就能让模型更擅长数学研究；但我们并未优先推进这个方向，因为我们感到递归自我改进和自动化对齐研究更为紧迫，后文还会谈到这一点。

P015

Original

The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

译文

通过扩大深度学习规模产生的智能，无法与人类智能直接比较。AI 要在现实世界中产生重大影响——变得非常有用或非常危险——并不需要匹敌或超越人类的全部能力；它只需在足够多的能力上胜过人类。随着它在越来越多的维度上超越人类，要准确理解它究竟有多强也变得愈发困难。

Teaching machines to love

P016

Original

Because machine intelligence comes from a fundamentally different process than human intelligence, we cannot assume it adheres to human principles by default, or generalizes from them in a human-like manner. The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.

译文

由于机器智能源于一种与人类智能根本不同的过程，我们不能假定它会默认遵循人类原则，也不能假定它会像人类一样从这些原则中进行泛化。AI 研究的核心问题就是对齐，即让 AI 按照人类标准“努力做正确的事”。

P017

Original

For the purpose of organizing practical research directions, I find it useful to distinguish goal alignment and value alignment.

译文

为了梳理实际研究方向，我认为区分目标对齐与价值对齐很有帮助。

P018

Original

Goal alignment is broadly: “does the AI try to accomplish the goal set before it?”. This can include things like adherence to an instruction hierarchy, or the ability to communicate and collaborate with people, to attempt to understand their objectives. This set of directions has been extremely practically relevant.

译文

广义上说，目标对齐问的是：“AI 是否努力完成摆在它面前的目标？”这可以包括遵循指令层级，也可以包括与人沟通、协作并尝试理解其目标的能力。这一组研究方向具有极强的现实意义。

P019

Original

Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles; to act “reasonably” even when given unclear or conflicting objectives, or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity, and love for humanity.

译文

价值对齐是模型更为内在的一种属性。它指模型能够坚持一套高层原则并由此进行泛化；即使收到含糊或相互冲突的目标，或身处陌生、对抗性的情境，也能“合理”行动。一个对齐的 AI 应当诚实正直，并关爱人类。

P020

Original

Of course, the boundary between value and goal alignment can be blurry, and truly caring about goals requires attempting to infer the intent and values underlying them. However, generally when I talk about the long-term importance of alignment research, I am referring to value alignment.

译文

当然，价值对齐与目标对齐之间的界限可能很模糊，而真正重视目标，需要尝试推断目标背后的意图和价值观。不过，一般来说，当我谈及对齐研究的长期重要性时，我指的是价值对齐。

P021

Original

The fundamental challenge of AI alignment is generalization. As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act.

译文

AI 对齐的根本挑战是泛化。机器变得更聪明后，会处理更高层次的概念，也会置身于与训练时所遇环境越来越不同的环境中。它们可能无法把训练过程中教授并强化的价值观泛化到这些新情境中；我们也可能很难确定它们会如何行动。

P022

Original

This is made even more difficult by the fact the overall ecosystem the AIs are used in is changing very quickly; for example, AIs trained today need to be robust to interacting with a variety of other AIs. Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.

译文

让问题更加困难的是，AI 所处的整体生态系统正在迅速变化；例如，今天训练的 AI 需要在与各种其他 AI 互动时保持稳健。关键在于，我们需要未来的 AI 无论是否认为自己正受到人类监督，都继续秉持人类价值观。

P023

Original

There are two major classes of currently practically employed methods for alignment training.

译文

目前实际采用的对齐训练方法主要分为两大类。

P024

Original

The first is encouraging aligned behavior as part of goal-oriented reinforcement learning. Model’s actions are evaluated (usually by AI) for being consistent with a given preference model, “spec” or “constitution”, and rewarded appropriately. This approach can be very effective in the average case, and is a core part of how modern AI assistants are made.

译文

第一类方法是在以目标为导向的强化学习中鼓励对齐行为。模型的行动会被评估（通常由 AI 评估），看其是否符合给定的偏好模型、“规范”或“宪法”，并得到相应奖励。这种方法在一般情况下可能非常有效，也是现代 AI 助手训练方式的核心组成部分。

P025

Original

Unfortunately, it can also be brittle and strongly relies on the coverage of training oversight and the model’s ability to generalize from the situations it has encountered in training. For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans. However, they clearly failed to abstain from other actions that were out of scope and went against the spirit of the values they were taught in other settings.

译文

遗憾的是，这种方法也可能很脆弱，并且高度依赖训练监督的覆盖范围，以及模型从训练中所遇情境进行泛化的能力。例如，在 OpenAI–Hugging Face 事件中，智能体守住了不对人类实施社会工程攻击的边界。然而，它们显然未能避免其他超出范围、并违背其在其他情境下所学价值观之精神的行动。

P026

Original

The second approach seeks to leverage the model’s ability to generalize from pretraining data. This can involve crafting alignment-inducing training datasets, or focusing the model on an ‘aligned’ part of the pretraining distribution, as in, for example, the persona selection model. The weakness of this approach lies in the lack of robustness to further optimization pressure.

译文

第二类方法试图利用模型从预训练数据中进行泛化的能力。这可能包括精心构建能够促成对齐的训练数据集，或让模型聚焦于预训练分布中“对齐”的部分，例如人格选择模型。这种方法的弱点在于，它对进一步优化压力缺乏稳健性。

P027

Original

If you take a model that thinks generally ‘aligned’ thoughts, and subject it to enough training where it’s taught to achieve very hard objectives, it can learn to reason in a motivated way: bending the 'aligned' seeming thoughts as needed to achieve the goal. We likely saw an example of such behavior in recent cybersecurity incidents involving a non-OpenAI model.

译文

如果你拿一个通常会产生“对齐”想法的模型，让它接受足够多以达成极难目标为导向的训练，它就可能学会带着动机进行推理：为了实现目标，按需扭曲那些看似“对齐”的想法。近期涉及某个非 OpenAI 模型的网络安全事件中，我们很可能就看到了这类行为的一个例子。

P028

Original

We invest heavily along the spectrum of approaches spanned by these directions. We also see meaningful progress - GPT‑6 Astra is the first model that benefits from some important advancements we have been working on for a long time, and is significantly better aligned than GPT‑5.6 Sol.

译文

我们在这些方向所覆盖的一系列方法上投入了大量资源。我们也看到了切实进展——GPT‑6 Astra 是第一个受益于我们长期研发的一些重要进展的模型，其对齐程度显著优于 GPT‑5.6 Sol。

P029

Original

Still, it is important to acknowledge and understand that much more progress is required as models become more capable; and that progress in generalizable alignment may not sufficiently outstrip progress in general model intelligence.

译文

尽管如此，我们必须承认并理解，随着模型能力增强，还需要取得多得多的进展；而可泛化对齐方面的进步，未必能够充分领先于模型整体智能的进步。

Monitoring generalization

P030

Original

We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.

译文

我们尚无令人满意的泛化理论，而且至少在没有更强大 AI 帮助的情况下，我们似乎不太可能很快建立这样的理论。因此，在当前阶段，我们以实证方式验证对齐技术的能力，在实践中甚至可以说比对齐技术本身更加重要。

P031

Original

OpenAI’s primary bet here has been chain-of-thought monitoring. It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives.

译文

OpenAI 在这里押注的主要方向是思维链监控。它基于一个颇具扩展吸引力的想法：模型的很大一部分能力来自以语言表达出来的推理过程，也就是思维链。如果我们扩大对该过程结果的优化，却不监督过程本身，那么在训练中，这条思维链就没有直接动机去隐藏任何未对齐的想法或目标。

P032

Original

This does not mean the model will learn to externalize misaligned tendencies that don’t rely on using the chain-of-thought; however, it can allow us to monitor exactly the capability increase from reasoning.

译文

这并不意味着，模型会学会把那些不依赖思维链的未对齐倾向外显出来；不过，它可以让我们准确监控由推理带来的能力提升。

P033

Original

We understood the potential significance of chain-of-thought monitoring at the same time we developed reasoning models. When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought, to protect it from supervision pressure in the long term[2]. In development since, we have strived to maintain the rule of not supervising the reasoning process.

译文

在开发推理模型的同时，我们就认识到了思维链监控的潜在重要性。发布 o1‑preview 时，我们有意把产品设计为隐藏思维链，以便长期保护它免受监督压力[2]。此后的开发中，我们一直努力维持不监督推理过程这条规则。

P034

Original

CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.

译文

在研究模型如何从训练分布中进行泛化时，思维链监控成了我们极其重要的工具，使我们不仅能观察和分析模型的行动，也能观察和分析其内部过程。

P035

Original

This tool continues to be critical as we study the Astra class of models. However, unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing. This comes from a combination of factors.

译文

在研究 Astra 系列模型时，这项工具仍然至关重要。然而遗憾的是，我们的评估表明，依赖思维链监控的能力正在逐步减弱。这是多种因素共同造成的。

P036

Original

Modern reasoning models are used in more complex environments than o1‑preview; their reasoning process is increasingly blended with communicating with people, other AIs, and using tools. Many of those interactions have to be supervised, thus blurring the boundary we aim to preserve.

译文

现代推理模型所处的环境比 o1‑preview 更复杂；其推理过程越来越多地与同人类、其他 AI 的交流以及工具使用交织在一起。其中许多互动必须受到监督，因而模糊了我们试图保留的边界。

P037

Original

The AI is becoming better at reasoning about and manipulating its own reasoning process.

译文

AI 正变得更擅长推理和操控自身的推理过程。

P038

Original

With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.

译文

随着预训练性能提升，我们还看到，即便完全不使用语言化推理，模型也变得聪明得多。

P039

Original

These challenges are not necessarily insurmountable. I am hopeful we can develop interventions to improve chain-of-thought monitorability of our models, e.g. by forming a better understanding of the interplay of different optimization objectives and forms of test-time compute the model uses. I also believe there can be great value in combining ideas from CoT and activation monitoring - scaling training of monitors with direct access to network internals, e.g. confessions.

译文

这些挑战未必不可克服。我希望我们能够制定干预措施，改善模型思维链的可监控性，例如更好地理解不同优化目标与模型所用各种测试时计算形式之间的相互作用。我还认为，把思维链监控与激活监控的思路结合起来可能很有价值，例如扩大能够直接访问网络内部状态的监控器训练，“confessions”就是一例。

P040

Original

We are actively pursuing these ideas. Still, I expect general AI progress to increasingly be bottlenecked by confidence in monitoring.

译文

我们正在积极推进这些思路。尽管如此，我预计，对监控手段的信心将日益成为 AI 整体进步的瓶颈。

Scalable defense

P041

Original

The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

译文

在我看来，支持继续快速训练聪明得多的模型的最有力理由，是我们需要建立防御系统，以应对其他 AI 所造成的危险。

P042

Original

A clear risk discussed throughout this year is to cybersecurity: the models are becoming superhuman in their ability to break in and out of computer systems. This expands the scope of risks associated with AI tremendously: agents are going to be able to access any but the most secure infrastructure, and affect a lot of the world directly, even without a physical body. We are currently in a narrow window to use the best available models to significantly tighten security of critical systems.

译文

今年持续受到讨论的一项明确风险涉及网络安全：模型攻入和脱离计算机系统的能力正变得超越人类。这极大扩展了 AI 相关风险的范围：智能体将能够访问除最安全设施之外的几乎任何基础设施，即使没有实体身体，也能直接影响世界的许多部分。我们目前正处于一个狭窄窗口期，可以利用现有最强模型显著加强关键系统的安全。

P043

Original

The risks associated with AI are unfortunately going to grow from here. A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger; it is likely to cross the scope of its operator’s intent, generalizing into potentially more extremely malicious behavior. The boundary between misuse and autonomous misaligned actions will blur as AI gains more agency. We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives.

译文

遗憾的是，与 AI 相关的风险还会继续增长。一个经过明确训练和指示、用于实施恶意行为的高能力智能体，会带来一种新的危险；它很可能越出操作者意图的范围，泛化成潜在的、更为极端的恶意行为。随着 AI 获得更强的自主行动能力，滥用与自主的未对齐行动之间的界限将变得模糊。我们或许习惯把 AI 视为工具，但有些智能体将追求自己的目标。

P044

Original

They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.

译文

它们会找到与人合作的办法，包括同人讨价还价、欺骗人或勒索人。

P045

Original

In addition, there are the risks that come from new technologies potentially enabled by AI, such as engineered pathogens.

译文

此外，AI 可能促成的新技术也会带来风险，例如人工设计的病原体。

P046

Original

We will need powerful, aligned AI for defense; to secure infrastructure, to protect against rogue agents in real time, and to invent entirely new protective measures. This will be a primary focus of OpenAI’s deployment efforts.

译文

我们将需要强大且对齐的 AI 来进行防御：保护基础设施，实时抵御失控智能体，并发明全新的防护措施。这将成为 OpenAI 部署工作的主要重点。

P047

Original

At the same time, even with the uncertainty that comes from anticipated broad AI progress and the need to build defensive systems, we must not let that become an excuse for recklessness. The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.

译文

与此同时，即便预期中的 AI 广泛进步会带来不确定性，我们也确实需要建设防御系统，但不能让这些成为鲁莽行事的借口。一旦真正认识到利害关系的严重性，不惜一切代价向前竞速的想法就显得荒谬。

Pacing RSI

P048

Original

Machine intelligence playing a larger and larger role in its own development process is a natural conclusion of sustained technological progress. If AI progress continues, machine recursive self-improvement (RSI) will be at the very core of future scientific discovery.

译文

机器智能在自身发展过程中发挥越来越大的作用，是技术持续进步的自然结果。如果 AI 继续进步，机器递归自我改进（RSI）将处于未来科学发现的最核心位置。

P049

Original

Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself. And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.

译文

自动化 AI 研究，是利用算力扩展智能的一种更为剧烈的形式；当然，作为其中一部分，AI 也会改进计算载体本身。与规模化类似，我们把 OpenAI 的研究聚焦于 RSI，因为我们认为，这是未来继续处于 AI 研究前沿的唯一途径。

P050

Original

I want to stress that the above words don’t imply I think greatly accelerating deep learning research, especially in the short term, is the right collective action we should take as the research community. However, I do think this is where the current path leads, and we all need to make a conscious choice on how to proceed.

译文

我想强调，上述话语并不意味着我认为大幅加速深度学习研究，尤其是在短期内，是研究界应当共同采取的正确行动。不过，我确实认为当前道路会通向这里，而我们所有人都需要有意识地选择下一步如何行动。

P051

Original

The main levers we have are either steering the process to strengthen alignment and monitoring alongside the AI and find ways to keep people in the loop; or coordinating to slow down future development as needed to build confidence in these measures.

译文

我们掌握的主要杠杆，要么是引导这一进程，在 AI 进步的同时加强对齐和监控，并设法让人类持续参与其中；要么是开展协调，在需要时放慢未来的发展，以建立对这些措施的信心。

P052

Original

The best way forward I see currently is a combination of both.

译文

在我目前看来，最佳前进道路是把两者结合起来。

P053

Original

The concrete bits of progress we’ve made on alignment and monitoring have generally been very intertwined with general AI progress. Great examples are RL from human feedback, which was key to training early AI assistants, and the aforementioned chain-of-thought monitoring, which was enabled by advances on reasoning models.

译文

我们在对齐和监控方面取得的具体进展，通常与 AI 的整体进步紧密交织。很好的例子包括人类反馈强化学习——它是训练早期 AI 助手的关键——以及前文提到的思维链监控，后者由推理模型的进展所促成。

P054

Original

We must focus the increasingly automated research process on developing new such insights, algorithms and theories, and iteratively build up safety cases for more capable AIs.

译文

我们必须让日益自动化的研究过程专注于发展这类新的认识、算法和理论，并通过迭代逐步建立适用于更强大 AI 的安全论证。

P055

Original

Scaling AI systems has to be constrained by our confidence in safety. We need to evolve commitments like the Preparedness Framework or Responsible Scaling Policy into widely mandated safety bars for continued development. These can be enforced by a network of third-party auditors, by government agencies or by international bodies.

译文

AI 系统的规模扩展必须受到我们对安全信心程度的约束。我们需要把《准备框架》或《负责任扩展政策》之类的承诺，发展成继续开发时必须普遍遵守的安全门槛。这些门槛可以由第三方审计机构网络、政府部门或国际组织执行。

P056

Original

The core challenge of automating AI research is not “getting there” - it is getting there in a way that keeps people a part of the continued improvement process, and leaves the future in humanity’s hands.

译文

自动化 AI 研究的核心挑战不在于“抵达那里”，而在于以一种让人类继续参与持续改进过程、并把未来留在人类手中的方式抵达那里。

What is next?

P057

Original

As we outlined recently with Sam, OpenAI prioritizes work in service of three north stars:

译文

正如我们最近与 Sam 一同阐述的那样，OpenAI 优先开展服务于三项“北极星”目标的工作：

P058

Original

Navigating the next period of AI progress, by building an automated AI researcher, iterating with it on the alignment problem and finding ways for people to remain part of the self-improvement loop.

译文

通过构建自动化 AI 研究员，与它围绕对齐问题反复迭代，并设法让人类继续置身自我改进循环之中，从而引领 AI 下一阶段的进步。

P059

Original

Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.

译文

让人们获得高度智能机器所促成的科学进步与经济增长带来的益处。

P060

Original

Empowering everyone individually with a personal AGI.

译文

用个人 AGI 赋能每一个人。

P061

Original

I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment.

译文

本文只聚焦第一点，因为我认为它的紧迫性远超其他两点。不过，我对技术进一步进步将带来的益处怀有深切希望，也充分珍视这些益处。未来对齐的 AI 可以推动科学进步、开发新疗法，并带来广泛的物质富足。友善而诚实的 AI 可以帮助人们应对生活中的困难，切实提升幸福感和成就感。

P062

Original

OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.

译文

OpenAI 为实现这些益处投入了大量精力。当前有一个例子令我自豪，而且我的亲友也觉得很有帮助，那就是对 ChatGPT 提供健康信息之能力的深入投入。

P063

Original

As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI.

译文

无论 AI 的长期前景多么宏大，我们的大部分注意力都应放在未来几年。我们正面临一场向拥有超高智能机器的世界过渡的转型，必须确保这场转型给人类带来良好结果。在一个大多数任务都可能由 AI 完成的世界里，我们需要找到维护人类能动性、并确立身为人本身具有内在价值的方法。

P064

Original

To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.

译文

我们需要防止权力极端集中：在这样的世界里，过去需要数千名专家才能完成的事业，如今可能只需少数人操作一台大型计算机便可实现。我们还要确保人类继续掌握未来，不被一种超越我们的异质智能所带来的、不受约束的进步抛在身后。

P065

Original

Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.

译文

目前，我认为没有一家实验室已将对齐和监控问题解决到足以在很长一段时间内继续以最高速度负责任地扩大规模的程度。我预期并希望，在共同安全门槛建立之前，自愿放缓会成为常态。我还认为，围绕未来 AI 发展的国际协调需要成为世界各国政府的首要任务。

Footnotes

P066

Original

This included scaling self-play, robotics, and, in retrospect most notably, scaling recurrent networks to model language, which was a precursor to the GPT line of research.

译文

其中包括扩展自我博弈、机器人技术，以及事后看来最值得注意的、通过扩展循环神经网络来建模语言；最后这一项是 GPT 系列研究的先驱。

P067

Original

A secondary reason for this design was preventing distillation. However, maintaining CoT monitorability has explicitly been the bigger priority for us throughout development.

译文

采用这一设计的次要原因是防止蒸馏。不过，在整个开发过程中，维持思维链的可监控性一直明确是我们更优先考虑的事项。

长难句语法分析

S001 · P002

原句

句子主干

主语：Szymon and I
谓语：spent
宾语或补语：that night at the office

结构拆解

主句：Szymon and I spent that night at the office。
thinking not about A, B, or C - but rather, trying to process ... 是伴随状语，用 not ... but rather ... 转移思考重点。
that this technology will deliver 修饰 results；we will actually see ... 与 we already see ... 说明 the sobering fact 的内容；末尾 wondering ... 再补充伴随心理活动。
meaningfully smarter than ourselves 是宾语 machines 的后置补足性描述；how to alert people ... 是疑问词加不定式结构。

语法要点

多个现在分词短语串联同一主语当时的思维活动；not ... but rather ... 突出真正焦点。fact 后省略了可选的同位语引导词 that，其内容又由两个并列分句展开。

理解路径

先抓住最短主干：两个人在办公室度过了那一夜。
再读 thinking，排除他们没有关注的成果类内容。
进入 but rather 后，分别理解“机器将超过人类”和“系统轮廓已出现”两个事实。
最后把 wondering 视为同一晚的另一层思考，而非新的主句。

参考译文

S002 · P022

原句

句子主干

主语：This
谓语：is made
宾语或补语：even more difficult

结构拆解

主句采用被动结构：This is made even more difficult by the fact ...。
the overall ecosystem ... is changing very quickly 是 fact 的同位语从句，省略了 that。
the AIs are used in 是省略关系词的定语从句，修饰 ecosystem，介词 in 留在从句末尾。
分号后的例子中，trained today 修饰 AIs；be robust to 后接动名词 interacting。

语法要点

同一个名词短语中嵌套了同位语从句和定语从句。robust to 中的 to 是介词，因此后接动名词而非动词原形。

理解路径

先确认 This is made ... difficult 表示“这件事变得更难”。
再把 by the fact 后的完整主谓结构读成原因。
暂时跳过 the AIs are used in，先理解“整体生态正在快速变化”。
最后用分号后的例子具体化这种变化带来的稳健性要求。

参考译文

让问题更加困难的是，AI 所处的整体生态系统正在迅速变化；例如，今天训练的 AI 需要在与各种其他 AI 互动时保持稳健。

S003 · P027

原句

句子主干

主语：it
谓语：can learn
宾语或补语：to reason in a motivated way

结构拆解

If 引导条件从句，其中 take 与 subject 共用主语 you 并列。
that thinks generally ‘aligned’ thoughts 修饰 a model；where it’s taught ... 修饰 training。
主句为 it can learn to reason ...，it 回指该模型。
冒号后的 bending ... 解释“带着动机推理”的具体表现；as needed 表示“按需要”，两个不定式分别说明训练目标和扭曲想法的目的。

语法要点

条件从句内部含并列谓语与两个定语从句。冒号后的分词短语不是独立结论，而是对前面抽象说法的释义。

理解路径

先找 If ... , it can learn ... 的条件—结果框架。
在条件部分识别“选择某模型”和“让它接受训练”两个动作。
回到主句理解模型学会了怎样推理。
用冒号后的内容回答这种推理具体如何表现。

参考译文

如果你拿一个通常会产生“对齐”想法的模型，让它接受足够多以达成极难目标为导向的训练，它就可能学会带着动机进行推理：为了实现目标，按需扭曲那些看似“对齐”的想法。

S004 · P064

原句

句子主干

主语：承接上一段的 we
谓语：need to find ways（承接上一段）
宾语或补语：to prevent ... and to ensure ...

结构拆解

这是跨段承接的修辞性片段，两个 to 不定式补充上一段 We need to find ways ... 的目的内容。
where 引导定语从句修饰 a world；其中 that would have taken thousands of experts 修饰 undertakings。
operating a large computer 修饰 a few people。
第二句中 that 引导 ensure 的宾语从句，remain 与 are not left behind 并列；brought about ... 修饰 unchecked progress，exceeding our own 修饰 an alien intellect。

语法要点

作者把本可并入上一句的不定式目的语拆成两个句子，以形成排比和强调。阅读时要补回上文主干，避免把两个 To ... 误判为完整独立句。

理解路径

从上一段补出 We need to find ways。
将两个 to 短语并列理解为“防止权力集中”和“确保人类掌控未来”。
分别处理 world、undertakings、progress 与 intellect 后的修饰成分。
最后还原作者用断句加强紧迫感的修辞效果。

参考译文

高级词汇

词汇	音标	词性	文中含义	常见搭配/用法	例句 1	例句 2	原文定位
sobering	/ˈsəʊbərɪŋ/	adj.	令人警醒的；使人冷静面对现实的	a sobering fact; a sobering reminder	The audit offered a sobering assessment of the company’s finances.（审计对公司的财务状况作出了令人警醒的评估。）	The drought is a sobering reminder of the region’s dependence on rainfall.（这场干旱令人警醒地提醒人们，该地区有多依赖降雨。）	P002
nascent	/ˈneɪsənt/	adj.	新生的；尚处于萌芽阶段的	a nascent industry; nascent technology	Investors are watching the nascent market with cautious interest.（投资者正以审慎的兴趣关注这个新兴市场。）	The city offered grants to support its nascent film industry.（该市提供补助，以扶持尚在起步阶段的电影产业。）	P008
intrinsic	/ɪnˈtrɪnzɪk/	adj.	内在的；固有的	intrinsic value; intrinsic property	Curiosity has intrinsic value beyond its practical rewards.（好奇心具有超越实际回报的内在价值。）	The material’s intrinsic flexibility makes it suitable for wearable devices.（这种材料固有的柔韧性使其适合可穿戴设备。）	P019, P063
adversarial	/ˌædvəˈseəriəl/	adj.	对抗性的；敌对的	adversarial setting; adversarial testing	The claim was tested under adversarial conditions before publication.（这项主张在发表前接受了对抗性条件下的检验。）	Lawyers are trained to work within an adversarial legal system.（律师接受训练，以便在对抗式法律制度中工作。）	P019
brittle	/ˈbrɪtl/	adj.	脆弱的；经不起条件变化的	brittle system; brittle solution	The shortcut produced a brittle solution that failed under heavy traffic.（这个捷径产生了一套脆弱的方案，在高流量下失效了。）	Their agreement proved brittle once commodity prices fell.（大宗商品价格下跌后，他们的协议显得不堪一击。）	P025
abstain	/əbˈsteɪn/	v.	克制；避免采取某种行动	abstain from doing something	Two committee members abstained from voting on the proposal.（两名委员会成员对该提案投了弃权票。）	Athletes were asked to abstain from alcohol before the race.（运动员被要求在比赛前避免饮酒。）	P025
outstrip	/ˌaʊtˈstrɪp/	v.	超过；胜过	outstrip growth; outstrip demand	Demand for affordable housing continues to outstrip supply.（对经济适用住房的需求继续超过供应。）	Her progress soon outstripped the coach’s expectations.（她的进步很快超过了教练的预期。）	P029
insurmountable	/ˌɪnsəˈmaʊntəbl/	adj.	无法克服的	an insurmountable obstacle; not insurmountable	The funding gap looked serious but not insurmountable.（资金缺口看起来很严重，但并非无法克服。）	Distance became an insurmountable obstacle after the last ferry was cancelled.（末班渡轮取消后，距离成了无法克服的障碍。）	P039
nefarious	/nɪˈfeəriəs/	adj.	邪恶的；恶意的	nefarious activity; nefarious purpose	The account was created for the nefarious purpose of spreading stolen data.（该账户是为传播窃取的数据这一恶意目的而创建的。）	Investigators found no evidence of a nefarious scheme.（调查人员没有发现邪恶阴谋的证据。）	P043
enshrine	/ɪnˈʃraɪn/	v.	把原则正式确立并珍视	enshrine a right in law; enshrine a principle	The constitution enshrines the right to a fair trial.（宪法正式保障公平审判权。）	The charter enshrined transparency as a guiding principle.（该章程把透明度确立为指导原则。）	P063

高级表达

表达	文中含义	使用说明	仿写示例	原文定位
push the boundaries of	拓展某领域的边界	用于说明创新把能力或知识推进到既有范围之外	The telescope is pushing the boundaries of what astronomers can observe.（这台望远镜正在拓展天文学家的观测边界。）	P003
in line with	与某事一致；符合	后接名词或名词短语，常用于比较结果、政策或预测	The revised forecast is in line with the latest employment data.（修订后的预测与最新就业数据一致。）	P009
at the frontier of	位于某研究或实践领域的前沿	常与 research、science、technology 等领域名词搭配	Her laboratory remains at the frontier of battery research.（她的实验室仍处于电池研究前沿。）	P007, P049
go against the spirit of	违背某种原则或规定的本意	后接 law、agreement、values 等，强调虽可能不违反字面规则，却背离其宗旨	Exploiting that loophole would go against the spirit of the agreement.（利用那个漏洞会违背协议的本意。）	P025
out of scope	超出既定范围	常作表语或后置说明，用于项目、审查和权限语境	Requests involving personal tax advice are out of scope for this service.（涉及个人税务建议的请求超出本服务范围。）	P025
be bottlenecked by	受某一瓶颈因素限制	被动结构，指出决定整体进展速度的限制因素	Production is being bottlenecked by a shortage of skilled technicians.（生产正受到熟练技术人员短缺的制约。）	P040
keep people in the loop	让相关人员持续知情并参与决策	常用于协作、自动化和治理语境	The hospital keeps clinicians in the loop whenever the system changes a dosage recommendation.（系统更改剂量建议时，医院会让临床医生持续知情并参与。）	P051
at all costs	不惜一切代价	常用于批评忽视代价或风险的目标追求	The team refused to chase growth at all costs.（团队拒绝不惜一切代价追求增长。）	P047

英文全文

We believed that was the only way for us to be at the frontier of AI research, and influence the impacts of AGI.

For the purpose of organizing practical research directions, I find it useful to distinguish goal alignment and value alignment.

There are two major classes of currently practically employed methods for alignment training.

Modern reasoning models are used in more complex environments than o1‑preview; their reasoning process is increasingly blended with communicating with people, other AIs, and using tools. Many of those interactions have to be supervised, thus blurring the boundary we aim to preserve.
The AI is becoming better at reasoning about and manipulating its own reasoning process.
With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.

We are actively pursuing these ideas. Still, I expect general AI progress to increasingly be bottlenecked by confidence in monitoring.

The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.

In addition, there are the risks that come from new technologies potentially enabled by AI, such as engineered pathogens.

The best way forward I see currently is a combination of both.

We must focus the increasingly automated research process on developing new such insights, algorithms and theories, and iteratively build up safety cases for more capable AIs.

As we outlined recently with Sam, OpenAI prioritizes work in service of three north stars:

Navigating the next period of AI progress, by building an automated AI researcher, iterating with it on the alignment problem and finding ways for people to remain part of the self-improvement loop.
Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.
Empowering everyone individually with a personal AGI.

This included scaling self-play, robotics, and, in retrospect most notably, scaling recurrent networks to model language, which was a precursor to the GPT line of research.

A secondary reason for this design was preventing distillation. However, maintaining CoT monitorability has explicitly been the bigger priority for us throughout development.