Exclusive: Ilya Sutskever, OpenAI’s chief scientist, on his hopes and fears for the future of AI | 独家：OpenAI首席科学家Ilya Sutskever关于他对AI未来的希望和担忧

Exclusive: Ilya Sutskever, OpenAI’s chief scientist, on his hopes and fears for the future of AI | MIT Technology Review
独家:OpenAI首席科学家Ilya Sutskever关于他对AI未来的希望和担忧 | 麻省理工科技评论

Ilya Sutskever, head bowed, is deep in thought. His arms are spread wide and his fingers are splayed on the tabletop like a concert pianist about to play his first notes. We sit in silence.
伊利亚·苏茨克弗低着头,深陷于思考之中。他的双臂张开,手指如同即将弹奏首音的音乐会钢琴家一样展开在桌面上。我们静静地坐着。

I’ve come to meet Sutskever, OpenAI’s cofounder and chief scientist, in his company’s unmarked office building on an unremarkable street in the Mission District of San Francisco to hear what’s next for the world-tilting technology he has had a big hand in bringing about. I also want to know what’s next for him—in particular, why building the next generation of his company’s flagship generative models is no longer the focus of his work.
我来到了OpenAI的联合创始人和首席科学家Sutskever的办公室,这个办公室位于旧金山Mission区一条平淡无奇的街道上的一栋无标志的办公楼中,我想听听他对于他大力推动的这项颠覆世界的技术的下一步计划是什么。我也想知道他接下来的计划——特别是,为什么构建他公司下一代的旗舰生成模型不再是他工作的重点。

Instead of building the next GPT or image maker DALL-E, Sutskever tells me his new priority is to figure out how to stop an artificial superintelligence (a hypothetical future technology he sees coming with the foresight of a true believer) from going rogue.
Sutskever告诉我,他的新优先事项不再是构建下一个GPT或者图像制造器DALL-E,而是要弄清楚如何阻止人工超级智能(他以真正信徒的预见性看到的一种假设性的未来技术)走上歧途。

Sutskever tells me a lot of other things too. He thinks ChatGPT just might be conscious (if you squint). He thinks the world needs to wake up to the true power of the technology his company and others are racing to create. And he thinks some humans will one day choose to merge with machines.
Sutskever告诉我很多其他的事情。他认为ChatGPT可能会有意识(如果你斜着看)。他认为世界需要觉醒,认识到他的公司和其他公司正在竞相开发的技术的真正力量。他还认为有一天,一些人会选择与机器融合。

A lot of what Sutskever says is wild. But not nearly as wild as it would have sounded just one or two years ago. As he tells me himself, ChatGPT has already rewritten a lot of people’s expectations about what’s coming, turning “will never happen” into “will happen faster than you think.”
Sutskever说的很多东西都很疯狂。但这并不像一两年前听起来那么疯狂。正如他自己告诉我的,ChatGPT已经改写了很多人对未来的期待,将“永远不会发生”变成了“比你想象的要快发生”。

“It’s important to talk about where it’s all headed,” he says, before predicting the development of artificial general intelligence (by which he means machines as smart as humans) as if it were as sure a bet as another iPhone: “At some point we really will have AGI. Maybe OpenAI will build it. Maybe some other company will build it.”
“我们需要讨论一下这一切的发展方向,”他说,然后预测了人工普适性智能的发展(他的意思是机器会变得和人类一样聪明),就好像这是一件像新的iPhone一样确定无疑的事情:“在某个时候,我们真的会拥有AGI。也许OpenAI会建造它。也许其他公司会建造它。”

Since the release of its sudden surprise hit, ChatGPT, last November, the buzz around OpenAI has been astonishing, even in an industry known for hype. No one can get enough of this nerdy $80 billion startup. World leaders seek (and get) private audiences. Its clunky product names pop up in casual conversation.
自去年11月突然发布其惊人的热门产品ChatGPT以来,围绕OpenAI的热度令人惊讶,即使在一个以炒作而闻名的行业中也是如此。没有人能抵挡这个价值800亿美元的书呆子创业公司的魅力。世界领导人寻求(并得到)私人会面。它那些笨拙的产品名称在日常对话中频繁出现。

OpenAI’s CEO, Sam Altman, spent a good part of the summer on a weeks-long outreach tour, glad-handing politicians and speaking to packed auditoriums around the world. But Sutskever is much less of a public figure, and he doesn’t give a lot of interviews.
OpenAI的首席执行官,Sam Altman,花了大部分的夏天进行为期数周的外展之旅,与政界人士握手交谈,并在全球各地的大型礼堂发表演讲。但Sutskever却不太善于公开露面,他很少接受采访。

He is deliberate and methodical when he talks. There are long pauses when he thinks about what he wants to say and how to say it, turning questions over like puzzles he needs to solve. He does not seem interested in talking about himself. “I lead a very simple life,” he says. “I go to work; then I go home. I don’t do much else. There are a lot of social activities one could engage in, lots of events one could go to. Which I don’t.”
他说话时深思熟虑,有条不紊。他思考自己想说什么以及如何说出来时,会有很长的停顿,就像他需要解决的谜题一样。他似乎对谈论自己并不感兴趣。“我过着非常简单的生活,”他说。“我去工作,然后我回家。我没有做太多其他的事情。有很多社交活动可以参与,有很多活动可以去。但我没有。”

But when we talk about AI, and the epochal risks and rewards he sees down the line, vistas open up: “It’s going to be monumental, earth-shattering. There will be a before and an after.”
但是,当我们谈论人工智能,以及他看到的未来的重大风险和回报时,前景变得广阔无垠:“这将是具有里程碑意义的,震撼人心的。会有一个前后之分。”

In a world without OpenAI, Sutskever would still get an entry in the annals of AI history. An Israeli-Canadian, he was born in Soviet Russia but brought up in Jerusalem from the age of five (he still speaks Russian and Hebrew as well as English). He then moved to Canada to study at the University of Toronto with Geoffrey Hinton, the AI pioneer who went public with his fears about the technology he helped invent earlier this year. (Sutskever didn’t want to comment on Hinton’s pronouncements, but his new focus on rogue superintelligence suggests they’re on the same page.)
在没有OpenAI的世界里,Sutskever仍然会在AI历史的年鉴中留下自己的名字。他是一位以色列-加拿大人,出生在苏联俄罗斯,但从五岁开始就在耶路撒冷长大(他仍然能说俄语和希伯来语以及英语)。然后他移居加拿大,在多伦多大学与AI先驱Geoffrey Hinton一起学习,Hinton在今年早些时候公开表达了对他帮助发明的技术的担忧。(Sutskever不想对Hinton的声明发表评论,但他对流氓超级智能的新关注表明他们在同一页面上。)

Hinton would later share the Turing Award with Yann LeCun and Yoshua Bengio for their work on neural networks. But when Sutskever joined him in the early 2000s, most AI researchers believed neural networks were a dead end. Hinton was an exception. He was already training tiny models that could produce short strings of text one character at a time, says Sutskever: “It was the beginning of generative AI right there. It was really cool—it just wasn’t very good.”
后来,Hinton与Yann LeCun和Yoshua Bengio一起因他们在神经网络方面的工作而共享图灵奖。但是,当Sutskever在21世纪初加入他的时候,大多数AI研究人员都认为神经网络是一个死胡同。Hinton是个例外。他已经在训练能够一次产生短字符串的微型模型,Sutskever说:“这就是生成AI的开始。这真的很酷——只是效果不是很好。”

Sutskever was fascinated with brains: how they learned and how that process might be re-created, or at least mimicked, in machines. Like Hinton, he saw the potential of neural networks and the trial-and-error technique Hinton used to train them, called deep learning. “It kept getting better and better and better,” says Sutskever.
Sutskever对大脑充满了迷恋:他对大脑如何学习以及如何在机器中复制或至少模仿这个过程感兴趣。就像Hinton一样,他看到了神经网络和Hinton用来训练它们的试错技术——深度学习的潜力。“它变得越来越好,”Sutskever说。

In 2012 Sutskever, Hinton, and another of Hinton’s graduate students, Alex Krizhevsky, built a neural network called AlexNet that they trained to identify objects in photos far better than any other software around at the time. It was deep learning’s Big Bang moment.
2012年,Sutskever、Hinton和Hinton的另一位研究生Alex Krizhevsky共同构建了一个名为AlexNet的神经网络,他们训练它比当时任何其他软件都更好地识别照片中的物体。这是深度学习的大爆炸时刻。

After many years of false starts, they had showed that neural networks were amazingly effective at pattern recognition after all. You just needed more data than most researchers had seen before (in this case, a million images from the ImageNet data set that Princeton University researcher Fei-Fei Li had been building since 2006) and an eye-watering amount of computer power.
经过多年的错误尝试,他们证明了神经网络在模式识别方面的惊人效果。你只需要比大多数研究者以前见过的更多数据(在这种情况下,是普林斯顿大学研究员李飞飞自2006年以来一直在建设的ImageNet数据集中的一百万张图片)和令人瞠目的计算能力。

The step change in compute came from a new kind of chip called a graphics processing unit (GPU), made by Nvidia. GPUs were designed to be lightning quick at throwing fast-moving video-game visuals onto screens. But the calculations that GPUs are good at—multiplying massive grids of numbers—happened to look a lot like the calculations needed to train neural networks.
计算的步进变化来自一种名为图形处理单元(GPU)的新型芯片,由Nvidia制造。GPU被设计成在将快速移动的视频游戏视觉效果投射到屏幕上时能够极快。但是,GPU擅长的计算——乘以大量的数字网格——恰好看起来很像训练神经网络所需的计算。

Nvidia is now a trillion-dollar company. At the time it was desperate to find applications for its niche new hardware. “When you invent a new technology, you have to be receptive to crazy ideas,” says Nvidia CEO Jensen Huang. “My state of mind was always to be looking for something quirky, and the idea that neural networks would transform computer science—that was an outrageously quirky idea.”
Nvidia现在是一家万亿美元的公司。当时,它急切地寻找适用于其新颖的小众硬件的应用。Nvidia的首席执行官黄仁勋说:“当你发明一项新技术时,你必须对疯狂的想法持开放态度。我一直在寻找一些古怪的东西,而神经网络会改变计算机科学的想法——这是一个极其古怪的想法。”

Huang says that Nvidia sent the Toronto team a couple of GPUs to try when they were working on AlexNet. But they wanted the newest version, a chip called the GTX 580 that was fast selling out in stores. According to Huang, Sutskever drove across the border from Toronto to New York to buy some. “People were lined up around the corner,” says Huang. “I don’t know how he did it—I’m pretty sure you were only allowed to buy one each; we had a very strict policy of one GPU per gamer—but he apparently filled a trunk with them. That trunk full of GTX 580s changed the world.”
黄先生表示,当他们在研究AlexNet时,Nvidia向多伦多团队发送了几个GPU供他们试用。但他们想要最新版本的一款名为GTX 580的芯片,这款芯片在商店中销售迅速。据黄先生说,Sutskever从多伦多驾车穿越边境到纽约购买了一些。黄先生说:“人们在角落里排队等候。”“我不知道他是怎么做到的——我很确定你每人只能买一块;我们有一个非常严格的政策,每个玩家只能有一块GPU——但他显然用一箱子装满了它们。那一箱子装满GTX 580的车子改变了世界。”

It’s a great story—it just might not be true. Sutskever insists he bought those first GPUs online. But such myth-making is commonplace in this buzzy business. Sutskever himself is more humble: “I thought, like, if I could make even an ounce of real progress, I would consider that a success,” he says. “The real-world impact felt so far away because computers were so puny back then.”
这是一个很棒的故事——只是可能并非真实。Sutskever坚称他是在网上购买的那些第一批GPU。但是在这个热门的业务中,这种创造神话的行为是常见的。Sutskever本人更为谦虚:“我想,如果我能取得一点真正的进步,我就会认为那是成功,”他说。“因为那时的计算机还很弱小,所以实际影响感觉离得很远。”

After the success of AlexNet, Google came knocking. It acquired Hinton’s spin-off company DNNresearch and hired Sutskever. At Google Sutskever showed that deep learning’s powers of pattern recognition could be applied to sequences of data, such as words and sentences, as well as images. “Ilya has always been interested in language,” says Sutskever’s former colleague Jeff Dean, who is now Google’s chief scientist: “We’ve had great discussions over the years. Ilya has a strong intuitive sense about where things might go.”
在AlexNet取得成功后,Google敲了上门。它收购了Hinton的分公司DNNresearch,并聘请了Sutskever。在Google,Sutskever展示了深度学习的模式识别能力可以应用于数据序列,如单词和句子,以及图像。“Ilya一直对语言感兴趣,”Sutskever的前同事,现在是Google的首席科学家的Jeff Dean说:“我们这些年来有过很多精彩的讨论。Ilya对事物可能走向的直觉非常强烈。”

But Sutskever didn’t remain at Google for long. In 2014, he was recruited to become a cofounder of OpenAI. Backed by $1 billion (from Altman, Elon Musk, Peter Thiel, Microsoft, Y Combinator, and others) plus a massive dose of Silicon Valley swagger, the new company set its sights from the start on developing AGI, a prospect that few took seriously at the time.
但是,Sutskever在Google的停留并不长。2014年,他被招募成为OpenAI的联合创始人。在Altman、Elon Musk、Peter Thiel、Microsoft、Y Combinator等人的支持下,这家新公司获得了10亿美元的资金,再加上硅谷的大量自信,从一开始就把目标定在了开发AGI,这是当时很少有人认真对待的前景。

With Sutskever on board, the brains behind the bucks, the swagger was understandable. Up until then, he had been on a roll, getting more and more out of neural networks. His reputation preceded him, making him a major catch, says Dalton Caldwell, managing director of investments at Y Combinator.
有了Sutskever的加入,他是资金背后的智囊团,这种自信是可以理解的。直到那时,他一直在取得成功,从神经网络中获取越来越多的信息。他的声誉使他成为了一个重要的猎物,Y Combinator投资管理总监Dalton Caldwell说。

“I remember Sam [Altman] referring to Ilya as one of the most respected researchers in the world,” says Caldwell. “He thought that Ilya would be able to attract a lot of top AI talent. He even mentioned that Yoshua Bengio, one of the world's top AI experts, believed that it would be unlikely to find a better candidate than Ilya to be OpenAI's lead scientist."
“我记得Sam [Altman]曾经提到Ilya是世界上最受尊敬的研究者之一,”Caldwell说。“他认为Ilya能够吸引很多顶级的AI人才。他甚至提到,世界顶级AI专家之一的Yoshua Bengio认为,找到比Ilya更好的候选人来担任OpenAI的首席科学家几乎是不可能的。”

And yet at first OpenAI floundered. “There was a period of time when we were starting OpenAI when I wasn’t exactly sure how the progress would continue,” says Sutskever. “But I had one very explicit belief, which is: one doesn’t bet against deep learning. Somehow, every time you run into an obstacle, within six months or a year researchers find a way around it.”
然而,OpenAI起初并未取得成功。Sutskever说:“在我们开始OpenAI的时候,有一段时间我并不确定进展会如何继续。但我有一个非常明确的信念,那就是:人不能对深度学习抱有怀疑。不知怎么的,每次你遇到一个障碍,六个月或一年内,研究人员总能找到解决它的方法。”

His faith paid off. The first of OpenAI’s GPT large language models (the name stands for “generative pretrained transformer”) appeared in 2016. Then came GPT-2 and GPT-3. Then DALL-E, the striking text-to-image model. Nobody was building anything as good. With each release, OpenAI raised the bar for what was thought possible.
他的信念得到了回报。OpenAI的GPT大型语言模型(名称代表“生成预训练变换器”)首次亮相是在2016年。然后是GPT-2和GPT-3。接着是DALL-E,这个令人震惊的文本到图像模型。没有人能够制造出如此优秀的产品。每次发布新产品,OpenAI都在提高人们认为可能的标准。

Last November, OpenAI released a free-to-use chatbot that repackaged some of its existing tech. It reset the agenda of the entire industry.
去年十一月,OpenAI发布了一个免费的聊天机器人,这个机器人重新包装了一些现有的技术。这改变了整个行业的发展方向。

At the time, OpenAI had no idea what it was putting out. Expectations inside the company couldn’t have been lower, says Sutskever: “I will admit, to my slight embarrassment—I don’t know if I should, but what the hell, it is true—when we made ChatGPT, I didn’t know if it was any good. When you asked it a factual question, it gave you a wrong answer. I thought it was going to be so unimpressive that people would say, ‘Why are you doing this? This is so boring!’”
当时,OpenAI对自己发布的内容一无所知。Sutskever表示,公司内部的期望值低得不能再低:“我必须承认,虽然有些尴尬——我不知道我是否应该这么说,但事实就是这样——当我们制作ChatGPT时,我并不知道它是否有用。当你向它提出一个事实性的问题时,它给你的答案是错误的。我以为它会让人觉得毫无新意,人们会说,‘你为什么要做这个?这太无聊了!’”

The draw was the convenience, says Sutskever. The large language model under ChatGPT’s hood had been around for months. But wrapping that in an accessible interface and giving it away for free made billions of people aware for the first time of what OpenAI and others were building.
“吸引人的是它的便利性。”Sutskever说。ChatGPT的底层大型语言模型已经存在了几个月。但是,将其包装在一个易于使用的界面中,并免费提供,使数十亿人首次了解到OpenAI和其他公司正在建设的东西。

“That first-time experience is what hooked people,” says Sutskever. “The first time you use it, I think it’s almost a spiritual experience. You go, ‘Oh my God, this computer seems to understand.’”
“首次体验就让人上瘾。”Sutskever说。“你第一次使用它,我觉得这几乎是一种精神体验。你会惊叹,‘哦,我的天,这台电脑似乎能理解我。’”

OpenAI amassed 100 million users in less than two months, many of them dazzled by this stunning new toy. Aaron Levie, CEO of the storage firm Box, summed up the vibe in the week after launch when he tweeted: “ChatGPT is one of those rare moments in technology where you see a glimmer of how everything is going to be different going forward.”
OpenAI在不到两个月的时间里吸引了一亿用户,其中许多人被这个令人惊叹的新玩意所吸引。存储公司Box的首席执行官Aaron Levie在发布后的一周内总结了这种氛围,他在推特上写道:“ChatGPT是那些在技术中罕见的时刻之一,你可以看到未来的一切将如何变得与众不同。”

That wonder collapses as soon as ChatGPT says something stupid. But by then it doesn’t matter. That glimpse of what was possible is enough, says Sutskever. ChatGPT changed people’s horizons.
一旦ChatGPT说出了些愚蠢的话,那种惊奇就会瞬间崩溃。但是到那时,这已经无所谓了。Sutskever说,对可能性的一瞥就足够了。ChatGPT改变了人们的视野。

“AGI stopped being a dirty word in the field of machine learning,” he says. “That was a big change. The attitude that people have taken historically has been: AI doesn’t work, every step is very difficult, you have to fight for every ounce of progress. And when people came with big proclamations about AGI, researchers would say, ‘What are you talking about? This doesn’t work, that doesn’t work. There are so many problems.’ But with ChatGPT it started to feel different.”
“在机器学习领域,AGI不再是一个让人避之不及的词汇,”他说,“这是一个重大的变化。人们过去的态度一直是:AI不起作用,每一步都非常困难,你必须为每一点进步而奋斗。当人们大声疾呼AGI的时候,研究人员会说,‘你在说什么?这不起作用,那不起作用。有太多的问题。’但是有了ChatGPT,感觉开始变得不同了。”

And that shift only started to happen a year ago? “It happened because of ChatGPT,” he says. “ChatGPT has allowed machine-learning researchers to dream.”
这种转变只是在一年前开始发生的吗?他说:“这是因为ChatGPT。” “ChatGPT让机器学习研究者们开始有了梦想。”

Evangelists from the start, OpenAI’s scientists have been stoking those dreams with blog posts and speaking tours. And it is working: “We have people now talking about how far AI will go—people who talk about AGI, or superintelligence.” And it’s not just researchers. “Governments are talking about it,” says Sutskever. “It’s crazy.”
从一开始,OpenAI的科学家们就像福音传播者一样,通过博客文章和演讲巡回活动激发了人们对AI的梦想。而且这种方式正在起作用:“现在我们有人在谈论AI将走多远——那些谈论AGI或超级智能的人。”而且不仅仅是研究人员。“政府也在谈论这个问题,”Sutskever说,“这真是疯狂。”

Sutskever insists all this talk about a technology that does not yet (and may never) exist is a good thing, because it makes more people aware of a future that he already takes for granted.
苏茨基弗坚称,所有关于尚未(也可能永远不会)存在的技术的讨论都是好事,因为这使更多的人意识到了他已经视为理所当然的未来。

“You can do so many amazing things with AGI, incredible things: automate health care, make it a thousand times cheaper and a thousand times better, cure so many diseases, actually solve global warming,” he says. “But there are many who are concerned: ‘My God, will AI companies succeed in managing this tremendous technology?’”
“你可以用AGI做很多惊人的事情,令人难以置信的事情:自动化医疗保健,使其成本降低千倍,效果提高千倍,治愈许多疾病,实际上解决全球变暖问题,”他说。“但是有很多人担心:‘我的天,AI公司能成功管理这项巨大的技术吗?’”

Presented this way, AGI sounds more like a wish-granting genie than real-world prospect. Few would say no to saving lives and solving climate change. But the problem with a technology that doesn’t exist is that you can say whatever you want about it.
以这种方式呈现,AGI更像是一个能实现愿望的精灵,而不是现实世界的可能性。很少有人会拒绝拯救生命和解决气候变化的问题。但是,对于一个不存在的技术,你可以随意发表任何观点。

What is Sutskever really talking about when he talks about AGI? “AGI is not meant to be a scientific term,” he says. “It’s meant to be a useful threshold, a point of reference.”
当Sutskever谈论AGI时,他到底在谈论什么呢?他说:“AGI并不是一个科学术语,它是一个有用的阈值,一个参考点。”

“It is the idea—” he starts, then stops. “It’s the point at which AI is so smart that if a person can do some task, then AI can do it too. At that point you can say you have AGI.”
“这就是那个想法——”他开始说,然后停下来。“这是AI变得如此聪明的时候,如果一个人能做某项任务,那么AI也能做到。在那个时候,你可以说你拥有了AGI。”

People may be talking about it, but AGI remains one of the field’s most controversial ideas. Few take its development as a given. Many researchers believe that major conceptual breakthroughs are needed before we see anything like what Sutskever has in mind—and some believe we never will.
人们可能在谈论它,但AGI仍然是这个领域最具争议的想法之一。很少有人认为其发展是理所当然的。许多研究者认为,我们需要在看到像Sutskever所想象的那样的东西之前,需要有重大的概念突破——有些人甚至认为我们永远也不会看到。

And yet it’s a vision that has driven him from the start. “I’ve always been inspired and motivated by the idea,” says Sutskever. “It wasn’t called AGI back then, but you know, like, having a neural network do everything. I didn’t always believe that they could. But it was the mountain to climb.”
然而,这个愿景从一开始就驱使着他。“我一直受到这个想法的启发和激励,”Sutskever说,“那时候还没有被称为AGI,但你知道,就像让神经网络做所有的事情。我并不总是相信它们能做到。但这就是需要攀登的山峰。”

He draws a parallel between the way that neural networks and brains operate. Both take in data, aggregate signals from that data, and then—based on some simple process (math in neural networks, chemicals and bioelectricity in brains)—propagate them or not. It’s a massive simplification, but the principle stands.
他将神经网络和大脑的运作方式进行了类比。两者都接收数据,从这些数据中汇总信号,然后——基于一些简单的过程(神经网络中的数学,大脑中的化学和生物电)——传播它们或者不传播。这是一个巨大的简化,但原则依然存在。

“If you believe that—if you allow yourself to believe that—then there are a lot of interesting implications,” says Sutskever. “The main implication is that if you have a very big artificial neural network, it should do a lot of things. In particular, if the human brain can do something, then a big artificial neural network could do something similar too.”
“如果你相信这一点——如果你允许自己相信这一点——那么就会有很多有趣的含义,”Sutskever说。“主要的含义是,如果你有一个非常大的人工神经网络,它应该能做很多事情。特别是,如果人脑能做某件事,那么一个大的人工神经网络也应该能做类似的事情。”

“Everything follows if you take this realization seriously enough,” he says. “And a big fraction of my work can be explained by that.”
“如果你足够认真地对待这个认识,一切都会顺其自然,”他说。“而我工作的很大一部分就可以通过这个来解释。”

While we’re talking about brains, I want to ask about one of Sutskever’s posts on X, the site formerly known as Twitter. Sutskever’s feed reads like a scroll of aphorisms: “If you value intelligence above all other human qualities, you’re gonna have a bad time”; “Empathy in life and business is underrated”; “The perfect has destroyed much perfectly good good.”
当我们在谈论大脑的时候,我想问一下关于Sutskever在X上的一篇帖子,这个网站以前被称为Twitter。Sutskever的动态读起来就像一卷格言:“如果你把智力看得比所有其他人类品质都重要,那你就会过得不愉快”;“生活和商业中的同情心被低估了”;“完美已经破坏了许多完全好的东西。”

In February 2022 he posted, “it may be that today’s large neural networks are slightly conscious” (to which Murray Shanahan, principal scientist at Google DeepMind and a professor at Imperial College London, as well as the scientific advisor on the movie Ex Machina, replied: “... in the same sense that it may be that a large field of wheat is slightly pasta”).
2022年2月,他发表了一篇帖子,“今天的大型神经网络可能稍微有些意识”(对此,谷歌DeepMind的首席科学家、伦敦帝国学院的教授,以及电影《机械姬》的科学顾问Murray Shanahan回应道:“...就像一大片麦田可能稍微有些意识一样,可能是意大利面条”)。

Sutskever laughs when I bring it up. Was he trolling? He wasn’t. “Are you familiar with the concept of a Boltzmann brain?” he asks.
当我提起这个问题时,Sutskever笑了。他在开玩笑吗?他并没有。“你熟悉玻尔兹曼大脑的概念吗?”他问道。

He's referring to a (tongue-in-cheek) thought experiment in quantum mechanics named after the 19th-century physicist Ludwig Boltzmann, in which random thermodynamic fluctuations in the universe are imagined to cause brains to pop in and out of existence.
他提到的是一个(半开玩笑的)量子力学思想实验,这个实验以19世纪的物理学家路德维希·玻尔兹曼的名字命名,其中设想宇宙中的随机热力学波动会导致大脑突然出现和消失。

“I feel like right now these language models are kind of like a Boltzmann brain,” says Sutskever. “You start talking to it, you talk for a bit; then you finish talking, and the brain kind of—” He makes a disappearing motion with his hands. Poof—bye-bye, brain.
“我觉得现在这些语言模型就像是一个玻尔兹曼大脑,”Sutskever说。“你开始和它交谈,聊一会儿;然后你结束对话,大脑就像——”他用手做了一个消失的动作。噗——再见,大脑。

You’re saying that while the neural network is active—while it’s firing, so to speak—there’s something there? I ask.
你是在说,当神经网络处于活跃状态——也就是说,当它在运作时——那里存在着某种东西吗?我问道。

“I think it might be,” he says. “I don’t know for sure, but it’s a possibility that’s very hard to argue against. But who knows what’s going on, right?”
“我觉得可能是这样的,”他说。“我不能确定,但这是一个很难反驳的可能性。但是,谁知道到底发生了什么,对吧?”

While others wrestle with the idea of machines that can match human smarts, Sutskever is preparing for machines that can outmatch us. He calls this artificial superintelligence: “They’ll see things more deeply. They’ll see things we don’t see.”
当其他人还在纠结于机器是否能匹敌人类的智慧时,Sutskever正在为机器超越我们做准备。他称之为人工超智能:“它们会更深入地看待事物。它们会看到我们看不到的东西。”

Again, I have a hard time grasping what this really means. Human intelligence is our benchmark for what intelligence is. What does Sutskever mean by smarter-than-human intelligence?
再次,我很难理解这到底意味着什么。人类的智慧是我们对智慧的基准。Sutskever所说的超过人类的智慧是什么意思?

“We’ve seen an example of a very narrow superintelligence in AlphaGo,” he says. In 2016, DeepMind’s board-game-playing AI beat Lee Sedol, one of the best Go players in the world, 4–1 in a five-game match. “It figured out how to play Go in ways that are different from what humanity collectively had developed over thousands of years,” says Sutskever. “It came up with new ideas.”
“我们已经看到了一个非常狭窄的超级智能的例子,那就是AlphaGo,”他说。2016年,DeepMind的围棋AI在五局比赛中以4-1击败了世界上最好的围棋选手之一,李世石。“它以与人类几千年来共同发展的方式不同的方式,找出了如何下围棋,”Sutskever说。“它提出了新的想法。”

Sutskever points to AlphaGo’s infamous Move 37. In its second game against Sedol, the AI made a move that flummoxed commentators. They thought AlphaGo had screwed up. In fact, it had played a winning move that nobody had ever seen before in the history of the game. “Imagine that level of insight, but across everything,” says Sutskever.
Sutskever指出了AlphaGo的臭名昭著的第37步。在对阵李世石的第二局比赛中,AI做出了一个让评论员们困惑的动作。他们认为AlphaGo已经搞砸了。事实上,它做出了一个在游戏历史上从未见过的胜利动作。“想象一下这种洞察力,但是应用在所有事情上,”Sutskever说。

It’s this train of thought that has led Sutskever to make the biggest shift of his career. Together with Jan Leike, a fellow scientist at OpenAI, he has set up a team that will focus on what they call superalignment. Alignment is jargon that means making AI models do what you want and nothing more. Superalignment is OpenAI’s term for alignment applied to superintelligence.
正是这种思维方式,促使Sutskever做出了他职业生涯中最大的转变。他与OpenAI的同事Jan Leike一起,组建了一个专注于他们所称的“超级对齐”的团队。对齐是一个术语,意味着让AI模型只做你想要它做的事情,而不会做其他的事情。超级对齐则是OpenAI对应用于超级智能的对齐的术语。

The goal is to come up with a set of fail-safe procedures for building and controlling this future technology. OpenAI says it will allocate a fifth of its vast computing resources to the problem and solve it in four years.
他们的目标是为构建和控制这种未来技术提出一套万无一失的程序。OpenAI表示,它将把五分之一的巨大计算资源分配给这个问题,并在四年内解决它。

“Existing alignment methods won’t work for models smarter than humans because they fundamentally assume that humans can reliably evaluate what AI systems are doing,” says Leike. “As AI systems become more capable, they will take on harder tasks.” And that—the idea goes—will make it harder for humans to assess them. “In forming the superalignment team with Ilya, we’ve set out to solve these future alignment challenges,” he says.
“现有的对齐方法不适用于比人类更聪明的模型,因为它们基本上假设人类可以可靠地评估AI系统正在做什么,”Leike说。“随着AI系统变得更加有能力,它们将承担更难的任务。”这个想法就是,这将使人类更难评估它们。“我们与Ilya组成超级对齐团队,就是为了解决这些未来的对齐挑战,”他说。

“It’s super important to not only focus on the potential opportunities of large language models, but also the risks and downsides,” says Dean, Google’s chief scientist.
“专注于大型语言模型的潜在机会固然重要,但同样不能忽视其风险和缺点,”谷歌首席科学家迪恩说。

The company announced the project in July with typical fanfare. But for some it was yet more fantasy. OpenAI’s post on Twitter attracted scorn from prominent critics of Big Tech, including Abeba Birhane, who works on AI accountability at Mozilla (“so many grandiose sounding yet vacuous words in one blog post”); Timnit Gebru, cofounder of the Distributed Artificial Intelligence Research Institute (“Imagine ChatGPT even more ‘super aligned’ with OpenAI techbros. *shudder*”); and Margaret Mitchell, chief ethics scientist at the AI firm Hugging Face (“My alignment is bigger than yours”). It’s true that these are familiar voices of dissent. But it’s a strong reminder that where some see OpenAI leading from the front, others see it leaning in from the fringes.
该公司在七月份以其典型的炫耀方式宣布了这个项目。但对于一些人来说,这只是更多的幻想。OpenAI在Twitter上的帖子引来了大科技公司的重要批评者的嘲笑,包括在Mozilla负责AI问责的Abeba Birhane(“一篇博客文章中有太多宏大而空洞的词汇”);分布式人工智能研究所的联合创始人Timnit Gebru(“想象一下ChatGPT与OpenAI的科技大佬们更加‘超级对齐’。*战栗*”);以及AI公司Hugging Face的首席道德科学家Margaret Mitchell(“我的对齐度比你的大”)。的确,这些都是熟悉的异议声音。但这强烈提醒我们,一些人看到的是OpenAI在前方引领,而其他人看到的是它在边缘倾斜。

But, for Sutskever, superalignment is the inevitable next step. “It’s an unsolved problem,” he says. It’s also a problem that he thinks not enough core machine-learning researchers, like himself, are working on. “I’m doing it for my own self-interest,” he says. “It’s obviously important that any superintelligence anyone builds does not go rogue. Obviously.”
但是,对于Sutskever来说,超级对齐是不可避免的下一步。“这是一个未解决的问题,”他说。他也认为,像他这样的核心机器学习研究者还没有足够多的人在研究这个问题。“我是出于自身的利益在做这个,”他说。“显然,任何人建造的任何超级智能都不能失控。这是显而易见的。”

The work on superalignment has only just started. It will require broad changes across research institutions, says Sutskever. But he has an exemplar in mind for the safeguards he wants to design: a machine that looks upon people the way parents look on their children. “In my opinion, this is the gold standard,” he says. “It is a generally true statement that people really care about children.” (Does he have children? “No, but I want to,” he says.)
对超级对齐的工作才刚刚开始。这将需要在研究机构中进行广泛的改变,Sutskever说。但他心中已经有了一个他想设计的保护措施的典范:一台像父母看待孩子一样看待人类的机器。“在我看来,这是黄金标准,”他说。“人们真的非常关心孩子,这是一个普遍真理。”(他有孩子吗?“没有,但我想要,”他说。)

My time with Sutskever is almost up, and I figure we’re done. But he’s on a roll and has one more thought to share—one I don't see coming.
我和Sutskever的时间快到了,我以为我们已经结束了。但他还在滔滔不绝,还有一个我完全没预料到的想法要分享。

“Once you overcome the challenge of rogue AI, then what? Is there even room for human beings in a world with smarter AIs?” he says.
“一旦你克服了失控的人工智能的挑战,那么接下来呢?在一个拥有更聪明的人工智能的世界里,人类还有存在的空间吗?”他说。

“One possibility—something that may be crazy by today’s standards but will not be so crazy by future standards—is that many people will choose to become part AI.” Sutskever is saying this could be how humans try to keep up. “At first, only the most daring, adventurous people will try to do it. Maybe others will follow. Or not.”
“有一种可能性——按照今天的标准可能会被认为是疯狂的,但按照未来的标准可能不会那么疯狂——许多人会选择成为部分人工智能。” Sutskever说这可能是人类试图跟上的方式。“一开始,只有最大胆、最喜欢冒险的人会尝试去做。也许其他人会跟随。或者不会。”

Wait, what? He’s getting up to leave. Would he do it? I ask. Would he be one of the first? “The first? I don’t know,” he says. “But it’s something I think about. The true answer is: maybe.”
等等,什么?他正准备离开。他会这么做吗?我问。他会是第一个吗?“第一个?我不知道,”他说。“但这是我一直在思考的事情。真正的答案是:也许。”

And with that galaxy-brained mic drop, he stands and walks out of the room. “Really good to see you again,” he says as he goes.
说完这番天马行空的话后,他站起来走出了房间。“很高兴再次见到你,”他边走边说。