No Priors 🎙️113: The Computing Platform Underlying AI, with Jensen Huang, Founder & CEO of NVIDIA (TRANSCRIPT)
EPISODE DESCRIPTION:
So much of the AI conversation today revolves around models and new applications. But this AI revolution would not be possible without one thing – GPUs, Nvidia GPUs.
The Nvidia A100 is the workhorse of today’s AI ecosystem. This week on No Priors, Sarah Guo and Elad Gil sit down with Jensen Huang, the founder and CEO of NVIDIA, at their Santa Clara headquarters. Jensen co-founded the company in 1993 with a goal to create chips that accelerated graphics. Over the past thirty years, NVIDIA has gone far behind gaming and become a $674B behemoth. Jensen talks about the meaning of this broader platform shift for developers, making very long term bets in areas such as climate and biopharma, their next-gen Hopper chip, why and how NVIDIA chooses problems that are unsolvable today, and the source of his iconic leather jackets.
No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode.
Show Links:
- Jensen Huang | NVIDIA
- Nvidia's A100 is the $10,000 chip powering the race for A.I. | CNBC
- Nvidia CEO Jensen Huang: A.I. is at ‘inflection point’ | Fortune
Sign up for new podcasts every week. Email feedback to show@no-priors.com
Show Notes:
[1:26] - The early days when Jensen Co-founded NVIDIA
[4:58] - Why NVIDIA started to expand its aperture to artificial intelligence use cases
[10:42] - The moment in 2012 Jensen realized AI was going to be huge
[13:52] - How we’re in a broader platform shift in computer science
[17:48] - His vision for NVIDIA’s future lines of business
[18:09] - How NVIDIA has two motions: Shipping reliable chips and solving new use cases
[25:41] - Why no one should assume they’re right for the job of CEO and why not every company needs to be architected as the US military
[31:39] - What’s next for NVIDIA’s Hopper
[32:57] - Durability of Transformers
[35:08] - What Jensen is excited about in the future of AI & his advice for founders
-----------------------------------------------------------------------
SARAH:
So much of the conversation in AI today is about new models and new applications, but it's all built on the back of NVIDIA and is specifically Nvidia GPUs. The NVIDIA A100 has been called the workhorse of today's AI ecosystem. And this week on No Priors, we're so excited to have Jensen Huang, the Founder, President, and CEO of NVIDIA.
So, over the last 30 years, he's created a company that began with a goal to accelerate graphics and has transformed the way we compute to become a behemoth as far as I could tell this morning, but that the CEO describes as a company that he's always trying to save. Thank you for doing this with us Jensen.
JENSEN:
Delighted to do it, Sarah.
SARAH:
Why don't we start at the beginning. You worked at LSI and AMD before starting a company. How did that happen?
JENSEN:
They gave me a job. Let's see, I was at Oregon State University and it was a campus company day. And I interviewed at a lot of companies and two companies really, really connected with me. I loved designing chips and designing computers.
And at the time in our lab, in the computer science lab, there was a poster of a 29,032-bit CPU from AMD, and I always thought it'd be cool to build that. On the other hand, there was another company that was a startup at the time built by one of the legends of Silicon Valley, Wilf Corrigan.
And they started a company to design chips using software to design chips, not by hand, but by using programmable logic. And you would describe it in language and it would synthesize it to chips. And of course, I chose to go to AMD. And it turned out I went there to design microprocessors.
And my lab partner, not my lab, but my office mate ended up going to LSI. And she insisted I go there. After I was there and after she went there and the LSI team said, "Hey, we were recruiting this kid from Oregon State and we really wanted him to come work at LSI Logic." and turned out to have been her office mate.
And so, they all reached out to me and I decided to go there, because it was at the beginning of the EDA industry. It was at the beginning of designing chips using computers, and it was probably one of the best things that ever happened to me.
And it was in the beginning of the ability for every company to build their own chips. And it's the reason why I met some really great computer architects. Andy Bechtolsheim was the founder of Sun. I got to work with a bunch of great architects at Silicon Graphics.
And Jon Rubinstein who was at a company called Dana Computer, who became the Vice President of engineering for Apple. And then, of course, the two founders of NVIDIA and Chris Makowski and Curtis Priem, myself. And so, I got a chance to work with some really amazing computer architects and I learned a lot about building computers with chips. And so, that's my early days.
SARAH:
And you were a star at LSI with your co-founders. At what point did you know I have to start a company?
JENSEN:
It wasn't my idea, it was theirs. Chris and Curtis wanted to leave Sun. They had their own reasons. And I was doing really well at LSI Logic and I enjoyed my job. And we had two kids and Lori and I. And just like you, they wouldn't stop hounding me and they said, "Hey, we want to start this company and we really need you to come along."
And I told them that I really needed to have a job and they needed to figure out what to do. At the time, the way of designing computers was rather split between general purpose computing versus using accelerators. And about 99% of the value was believed in general purpose computing and about 1% believed in acceleration.
For 25 years, 99% was right. We decided to start a company on accelerated computing. And at the time, the only thing you could really do with accelerated computing is find applications or find problems that were barely solvable or unsolvable by general purpose computing.
So, that's what we dedicated our company to do, to solve problems that normal computers can't. And if you follow that mission to its limit, it led us to self-driving cars. It led us to robotics. It led us to climate science problems, digital biology, and of course one of the most famous ones is artificial intelligence.
SARAH:
You were working on this huge set of applications before the current wave of artificial intelligence. What was the original technical advantage of NVIDIA and artificial intelligence, and when did you begin to realize that this was going to be an important use case for you guys?
JENSEN:
So, we had expanded the flexibility of our accelerators to be more general purpose. And we invented a new computing model called CUDA. And we're doing those podcasts at 4:00 or something like that in the afternoon. It was like at the lowest point of energy, isn't that right?
ELAD:
Yeah.
JENSEN:
So, we need some-
ELAD:
That's why we need-
JENSEN:
... nerds.
ELAD:
... nerds.
SARAH:
Nerds.
JENSEN:
That's right, we need some nerds.
SARAH:
Thank you nerds.
ELAD:
Now, with gummy clusters as well. So, that's very exciting new technology.
JENSEN:
We need some accelerated computing right now. So, we wanted to make our graphics processors more and more general. And the reason for that in the beginning was because some of the effects that we had to do related to general purpose image processing.
Post effects, you render an image and you do post image effects. Other applications of course. We wanted to bring the scene to life and so we had to do physics processing. And you have to do physics, you have to do particle physics, fluid dynamics, so on and so forth.
And so, we expanded the aperture of our accelerated computing platform to be more and more and more general purpose. The problem with general purposeness is that the more general purpose you are, the less acceleration you get in any particular domain.
And so, you got to find that line really, really carefully. And that's one of the gifts of our company to find that line between, on the one hand, every single generation bringing enormous amounts of acceleration well beyond what CPU could do to the application.
And so, if you become too general purpose, you're like, just like a CPU. How can you accelerate a CPU with a CPU? And so, you to find a way to walk that line. On the other hand, if you don't expand the aperture of the applications that you serve, the R&D dollars that you're able to generate wouldn't be enough to stay ahead of the CPU, which had the largest r and d budget of any chip on the planet.
So, if you think about this problem, it's actually really nearly impossible because you have a small application, let's call it a billion-dollar market at the time. And out of that billion-dollar market, you investing 150 million a year. Out of that $150 million a year, how do you keep up with a few hundred-billion-dollar industry?
It's not even sensible. And so, you have to find that niche very, very carefully where $150 million would accelerate this particular application abnormally and insanely. And then, over time, you could expand your application space so that it goes from a billion dollars to $5 billion to $10 billion, so on so forth without falling off that cliff.
That is the fine line that we walked. And so, we kept expanding the general purposeness and it led us to molecular dynamic simulation, which is what this image seems to look like. And seismic processing was another industry. And slowly by surely, we expanded our aperture.
But one of the things that we did well was to make sure that irrespective of whether somebody used our platform for general purpose computing, accelerated computing, we always maintained the architecture compatibility. And the reason for that is because we wanted a platform that would attract developers.
If every single NVIDIA chip in the world was incompatible, then how would a developer be able to pick one up even if they learned that CUDA was going to be incredible for them. How would they pick that up and say, "I'm going to develop an application, it's going to run on that."
Which chip would they have to go figure it out? And nobody could figure that out. And so, we said, if we believe in an architecture and want this to be a new computing platform, then let's make sure that every one of our chips perform exactly the same way, just like an x86, just like ARM, just like any computing platform.
And so, for the first five, 10 years, we had very few customers for CUDA, but we made every chip CUDA compatible. And you can go back in history and looked at our gross margins. It started out poor and it got worse.
So, we were in a really competitive industry and we were still trying to figure out how to do our job and build cost-effective things. So, it was already challenging as it is. And then, we laid it on top of this architecture that was called CUDA that had no applications that nobody paid for.
ELAD:
Yeah. It's amazing. Because now, when I talk to people in the AI world in terms of one of their reasons that they really love using NVIDIA GPUs is because of CUDA and then because of the ability to scale interconnect. And so, you can really highly parallelize these things as well, which you can't necessarily do with other approaches or architectures that are in the market today.
JENSEN:
And so, this computing platform led is strange in the sense that it performs these miraculous things. And we carried it to the world on the backs of GeForce, which is a gaming card. The first GPU that Geoff Hinton got for his lab. Elad would tell you that Jeff came in and say, "Here's a couple of GPUs, it's called GeForce and you guys should try to use that for DNN." And so, it was a gaming card that he bought.
ELAD:
What applications did you have in mind? Because to your point, you started with gaming, or at least you were very popular with gaming starting in the 90s, when you started the company. And then, I started hearing about NVIDIA GPUs more and more both in the context of cryptocurrencies and mining and then in the context of AI.
And it seemed like those were the two markets where a bunch of people were organically just adopting you. Were you marketing to those communities? Was it just people started realizing that they needed linear algebra?
JENSEN:
That's the beauty of a computing platform, right? In the beginning, you have to target the applications. And in the beginning, we did one of the first applications was in AMD, seismic processing. One of those is particle physics, the other one is image processing, if you will.
And so, inverse physics, if you will. And one particular domain, we just went out to hire to research. We went to scientific computing centers and we said, "What problems are just beyond your reach?" And the list of applications include quantum chemistry and quantum physics and so on and so forth.
ELAD:
What was the moment when you said, "Wow, this AI thing is really important for us?
JENSEN:
It happened around 2012, I guess. And it was because simultaneously, Andrew Ng reached out to Bill Dally, our chief scientist, to work on a way to get the neural network model that they were working on onto GPU so that instead of using thousands of CPU servers, they could use a few GPUs to do training.
So, that was one. Simultaneously, Jeff Hinton reached out to us and we started hearing about that. And the same thing was happening with Yann LeCun and his lab. And so, simultaneously in several different labs, we're starting to feel that there's this neural network emergence and that attracted our attention.
ELAD:
Yeah. I guess, 2012 was also the year when AlexNet came out. So, I felt like that was a year of transition for deep learning in general in terms of really... that was the moment in time at least that I remember thinking, "Wow, this really exciting wave of AI coming." And then, I feel like for 10 years, nothing really happened for startups, but a lot of incumbents started adopting this technology.
JENSEN:
Yeah. We started feeling it. We started hearing about it before that, and then ImageNet, it was the big bang, if you will, got all of our attention.
SARAH:
You talk about early AI labs as pulling this from NVIDIA using gaming cards because you were solving a problem nobody else could solve, and efficiency and scale. Is there a point at which NVIDIA begins to invest in an application because they think it's a growing application? Or is it more, it's a platform and the market will take it from us?
JENSEN:
No. In every single case, when an application finds use, we ask ourselves how can we make it even better? And this time with deep learning, the good insight that we made, it was piecing together observations in a lot of different ways.
But realizing that this isn't just going to be a new algorithm for computer vision, which is really most of the applications in the beginning, but which was going to be very helpful. I mean, just if it was just computer vision, we could have used it for all kinds of interesting applications like self-driving cars and robotics, and we did.
But we observed that this might be a new way of writing software altogether and asking ourselves, what's the implication to chip design, system design interconnect the algorithm, the system software to really reason about not just why is this exciting? Why was it so effective?
Which that alone was plenty miraculous that ImageNet without specifically any human engineered algorithm would reach the level of effectiveness compared to 30 years of computer vision algorithms overnight. It wasn't by a small amount. And so, the first question of course is why is it so effective and was this going to be scalable.
And if it was going to be scalable, what's the implication to the rest of computer science? What problems can't this universal function if you will, that can solve problems of dimensionality extraordinarily high. And yet, you could learn the function using enough data, which at the time we were starting to believe we can get plenty of. And to systematically train this model into existence, because you train them one layer at a time.
ELAD:
I've heard you be very articulate in terms of how you view this as a broader platform shift, just even in terms of how pages are served versus generated or other aspects of that. Could you talk a little bit more about what's really happening right now more broadly in computer science with a shift to AI?
JENSEN:
Yeah. So, you fast forward now a decade. The first five years was about reasoning the impact to computer science altogether. At the same time, we're developing new models of all kinds, right?
And so, CNNs to ResNets to RNNs to LSTMs, to all kinds of new models and scaling them larger and larger, making great strides in perception models particularly. And of course, the Transformer was a big deal. BERT was a big deal. All of you know that story well.
SARAH:
Did you guys see a step change in volume growth with Transformers and BERT and such? Because it feels like having an architecture and an attention mechanism that allowed for scaling of these models really was also a kickstart in the industry.
JENSEN:
Well, the ability for you to learn patterns and relationships from spatial as well as sequential data must be an architecture that's very effective. So, I think on its first principles, you think transformers, it's going to be a big, big deal. Not only that, you could train it in parallel and you can really scale this model up.
And so, that's very, very exciting. I think that when Transformers first came out, we realized that there's a model now that overcame the limitations of RNNs and LSTMs, and we can now learn sequential data in a very large way. So, that was very exciting. BERT was very exciting.
We trained some of the early language models ourselves and we saw very good results. But it wasn't until the combination of reinforcement and learning human feedback wasn't in, and of course, some of the breakthrough work that was done with retrieval models, dialogue managers, that does the guard railing.
It wasn't until some of all of those pieces started to come together that of course that we all enjoy ChatGPT. And Eli, the point that you're trying to make is the observation that computer programming has now been completely disrupted that for the very first time in the history of computing, the language of programming a computer is human.
Any human language. It doesn't even have to be grammatically correct. And it's fairly incredible that anyone can program a computer now. And so, that's a big deal. The fact that you program it differently, it writes different applications. What is the reach of this new computing model? Apparently quite large. And is the reason why ChatGPT is the fastest growing application in history.
SARAH:
We had Alex Graveley, who was the Chief Architect for Copilot on the show as well. Obviously, it's very powerful to have sequential code prediction. But his favorite use cases of Copilot have been people telling him that they don't code, but now they do, which I think is a very democratizing as you said.
JENSEN:
It's quite amazing that you could give ChatGPT a problem to solve, and it reasons through it step by step, but yet it arrives at the wrong answer on the one hand. On the other hand, you could tell it to write a program to solve the same problem, and it writes a program that solves the problem perfectly.
The fact that there's an application that on the one hand reasons and tries to solve a problem and does a fairly good job at it, it's almost there. On the other hand, it can write a program altogether to solve that same problem. You got to really wrap your head around the implication of this.
ELAD:
So, do you hear it's like the future where all is some form of machine sentience?
JENSEN:
First of all, I don't even know what that word means in a technical way. I'm fairly sure that I'm sentient less so today.
ELAD:
So, we have [inaudible 00:17:24].
SARAH:
Keep eating.
JENSEN:
That's why I need nerves to crank me up here.
SARAH:
I'm going to try to.
JENSEN:
Yeah, I know. I know. Today, was a tough day. But I don't know. Do I believe that we now have a software that can reason through a problem for many, many types of problems? Reason through a problem and provide a solution or a program to systematically provide a solution on an ongoing basis? The answer is yeah.
ELAD:
Yeah. And then, as you look forward to that world, how do you think about where you want to take NVIDIA's lines of business. But also, you mentioned in the past that NVIDIA's done things like train models and you've done some really interesting things there.
Is that going to be an increasing part of what you do in the future? Or are you mainly focused on the chip side? Or how do you think about that mix of helping to push forward some research as well as being the underlying platform for the industry?
JENSEN:
Well, we're a computing platform company. And we have to go up the stack as far as we need to so that developers can use it. And so, the question is, what is a developer? And in the beginning, of course, a developer is somebody who controls their own operating system.
And so, in those days, we might only have to go as far up as device drivers or the layers slightly underneath that somehow to enable developers. But for scientific computing and all these different domains, the developer is actually using maybe a solver.
And they need the algorithms of that domain to be somehow expressed in a way that could be accelerated, which is the reason why when we moved into these multi-domain physics problems, we realized that we have to develop the algorithms themselves.
Because the algorithms of solving a problem relates to the computer architecture that's underneath. And if the architecture is CPUs connected through NPI and ethernet or whatever it is, that algorithm is surely very different than thousands of processors that's connected by a fabric inside one GPU and thousands of GPUs inside a data center.
So, obviously, the algorithm has to be reframed and refactored. And so, our company got very good at designing computer algorithms. It could be for particle physics or fluid dynamics. And then, of course, one day, it was related to deep learning and neural networks.
And cuDNN is essentially a domain specific language for accelerated deep learning. And so, we've done that for deep neural nets. We've done that for computer graphics with ray tracing, that's called RTX.
All of these different domain libraries really is about understanding the domain of science and then redesigning algorithms that make them go incredibly fast. Now, in the future, what's a developer? Well, I think in the future, a developer is likely going to be somebody who engages large language models of foundation models.
Now, if somebody could use GPT or open AI's model, I'd really encourage that. And the reason for that is because they do such an incredible job. If somebody could use it through Microsoft, I'd really encourage that. If somebody could use it through Google, I would really encourage that.
But if somebody needs to build a proprietary model for a domain, maybe create a new foundation model, and let's say the domain was proteins, or let's say the domain was chemicals, or let's say the domain was climate science, Multiphysics, that foundation model is pretty nichey.
And it's not a small market obviously, because the field of drug discovery is large, the field of climate science is large, climate tech is large. However, it's not likely to be horizontally useful for every human. And so, we might decide to go do something like a foundation model for 3D graphics, virtual worlds, because they're super important to us.
We might decide to build a foundation model for robotics, because it's at an intersection of the things that we do very well. And even then, we'll probably take it as far up as necessary but no further than that. We're not trying to be an AI model company. We're trying to help industries create AI models. Mostly, we're trying to help developers.
ELAD:
Yeah, that makes a lot of sense. You're basically following your customers up to whatever level they need you to.
JENSEN:
That's right.
ELAD:
And then, you hand it off to them at that proper point.
JENSEN:
We're trying to be as lazy as we can. Do as little as possible as much as necessary. The first principles of computer science we all know well, to reject work as quickly as you can. To defer whatever work you are left for as long as you can until you could be rejected.
And then, whatever remains you have to do, we try to do as little as we can, as much as possible. That's the principles of the company. Laziness, the principles of the company, that's the takeaways. [inaudible 00:21:58].
SARAH:
I'm trying to square that with some of these very long-term commitments that the company makes, right? CUDA is a very long-term bet. And we met a decade ago when NVIDIA was valued at 1,100th of its value today and was facing activist investors and such.
And it was probably like, let's say, a little harder to make long-term bets. How do you balance the pressures of being a large public company and the opportunities of today with architectural commitments or long-term bets and think about that prioritization?
JENSEN:
Investing in the future and being sustainable now are not in conflict with each other. And so, the challenge for all startup CEOs and for all CEOs is to find a way to be able to do what you believe in. The fundamental core belief of the institution and to be able to afford doing it.
That is the purpose of the company. And it's part conviction. It's part skill. Making money is not a matter of conviction. Making money is a matter of skill, and it's a learnable skill. And it took me a long time to learn it, I'll admit that. I've been at this for 30 years.
And well, apparently, for the first 20 years since you went back 10 years. For the first 20 years, I was still trying to figure it out, but it's a skill. Learning how to make money and learning how to run a company efficiently, those are all skills. And the company has to develop the skills.
And the way that we ultimately do it is we ask ourselves, "Do we really believe it or not?" And if we really believe in doing something, then it is the purpose of the enterprise, it's the singular purpose of the institution to go pursue its beliefs.
And the rest of it is up to all of the cleverness of the company and try to do our jobs well and build things that people want to buy, and try to make it as cost effective as possible, make the company as efficient. Those are all skills. The hard part, as it turns out is not the skill part.
It took me a long time, but a lot of companies know how to make money, obviously. So, the fact that they're more than one company that makes money suggests it's not that hard. Somebody else can do it, how hard can it be. And so, singularly advancing a new computing model, we called accelerated computing.
And we believe that someday that on the one hand, accelerated computing can help us solve problems and tackle problems that normal computers can't, and it exposed us to all of these amazing applications like digital biology that I'm excited about today.
Like climate change, that we're excited about. Like robotics and self-driving cars. If not for the fact that we're pursuing applications that were impossible with normal computers, why would we have discovered all of those things? Why would we have discovered artificial intelligence?
Why would we be the workhorse of large language models? Because large language models are barely possible. And if you are doing something that's barely possible, you call us. We're the horse you call to solve those problems. And so, I love that aspect.
I love the fact that we get to discover those future. On the other hand, we deeply believe that someday, everything will be accelerated. And the reason for that is very clearly that the CPU will run its course. And there's a limit to how far you could scale general purpose computing, and you'll always need it.
You'll always need CPUs. But the type of applications that we're all going to run, acceleration is really the best way forward. And at our core, we believe that from day one, 30 years ago, that's the reason why we started the company. And so, it's the true conviction.
SARAH:
You have been enormously vindicated on this 30-year belief. You must have felt that conviction challenge at some point in 30 years of running the company and learning the skills to run the company. What was the nearest death experience or the most concerned where you're like, "Maybe I'm not right." Or has that ever happened?
JENSEN:
The I'm not right for the job?
SARAH:
No, you're not right about accelerated computing and how important it will be.
JENSEN:
The second one is yes. First of all, I don't think anybody should assume that they're right for the job. And so, you should be gut checking on that almost every day.
SARAH:
To be clear, that wasn't the question, but-
JENSEN:
But I'll more than happy to-
SARAH:
... very helpful. Yeah.
JENSEN:
... answer that question. Did I ever believe that it was wrong? No. I believe that accelerated computing is the absolutely the only way to solve problems that are impossible by definition if it's not-
SARAH:
Okay. Axiomatically, yeah.
JENSEN:
Right? And on the other hand, if you can solve problems that are impossible today and someday you need that application to be broad-based, would accelerated computing be the best approach? The answer yes. Yeah.
ELAD:
When do you think the CPU hits its limits? You mentioned that eventually you think everything will move over, or at least big chunks of the future will move over. Is that five years away, 10 years away?
JENSEN:
For certain applications, it happened 12 years ago, right? Jeff Hinton and Yann LeCun and Andrew, right, Andrew Ng, they discovered that 12 years ago. It was the only way forward. And computer graphics, it's the only way forward.
ELAD:
Yeah. Is the way that you organize and run the company changed as AI has gotten more and more prominent? Have you realigned aspects of the business around it? Or how do you think about management in general in this environment where things are changing so rapidly and there's so many exciting things happening in this area?
JENSEN:
You're asking a really good question. And maybe if I just take a step backwards. The company's architecture should not be generic. Every company in the world should not be built like the US military. And in fact, if you look at every company's org chart in the world, they look like the US military.
There's somebody on top, and then it comes down. And yet, the number of direct reports of CEOs are very few. And the direct reports of the people who are just learning how to manage first level managers are very large. It's exactly the opposite of how it should probably be architected.
You would think that the people that report to the CEO requires no management at all. And in fact, it's generally true. My direct reports are sophisticated. They're really talented. They're incredibly good at their job. They're excellent leaders. They have great business acumen. They have excellent vision. They're incredible, every single one of them.
SARAH:
I guess, that means you have more than the management book accept at six or seven or whatever.
JENSEN:
Yeah. I have 40 somewhat direct reports, and no one-on-ones, no career coaching. So, what would you like to do with your life? Those are conversations you have with new college grads and early career. And we love those conversations of course, and helping them shape their career and mentor them, give them access to new experiences.
But at the executive staff level, we're organized so that we can pursue a whole lot of different things at the same time. However, one of the most important things about a software company is you have to understand computer architecture.
And one of the most important thing about computer architecture is you can only afford one. Just as some of the largest companies in the world only have two operating systems, the single largest company on the planet only has two. How is it possible that so many companies have so many different computer architectures, and they have seven or eight or nine instruction sets that they're keeping around?
We have one instruction set. We have one computer architecture, and we're super disciplined about that. And so, where we need to be focused, where we allow for innovation and discovery at the senior level, we allow that. So, I think the company's tapered and organized in a way that is consistent with the nature of our work.
So, that's the most important thing. And that's probably the takeaway for what I've learned building our company is, there is no one generic architecture for every company, it should fit the function of the company, its purpose, and then of course the leadership style of the leaders.
ELAD:
Yeah, I think that's a really important note that most people don't really realize is that a company should almost be a bespoke structure supporting the CEO and their staff and what the company's delivering to customers versus it's always the same thing. And I think-
JENSEN:
Exactly.
ELAD:
... that gets lost a lot.
JENSEN:
Exactly, yeah. There's some particular chief that that you need and chief that that you need and a chief this, that there's some achieves that you do need. But aside from that, you should start from first principles and architects, something that makes sense for the leader and as well as the function.
ELAD:
Yeah. When I was at Google, they had the famous 80/20/10 where it was like 80% is core, 20% is like core adjacent/new stuff, and then 10% was hyper experimental. Do you have any frameworks or ways to think about that stuff?
Or it's just like let's see what organically is used in terms of this generic platform that we've built with CUDA and other things that are built in to help support a lot of use cases. And as they emerge, we say, "Okay, let's go support that new thing."
JENSEN:
I don't have any wise framework like that. There are a couple of things that our company is shaped and structured to do. There's one part, a very large part of our company is designed to build very, very complicated computers perfectly. And so, that is one of its missions.
That architecture, that organization is an invention and refinement organization. We have a whole bunch of skunkworks, if you will. And the reason for that is because we're trying to invent things 10 years out that we're not exactly sure whether there's going to work or not.
And there's a lot of adaptation and a lot of pivoting. And so, our company actually has two different ways of working. One of them is rather organic shapeshifting all the time. If a particular investment's not working out, we give up on it, move the resources somewhere else.
And so, that's the agile part of the company. And then, there's a part of the company that's not rigid but it's really refined. And so, these two systems have to work side by side.
SARAH:
Can you talk a little bit about the H100?
JENSEN:
Mm-hmm.
SARAH:
Next workhorse. And what the most important innovations are and what the design and chip process for that looks like?
JENSEN:
I would say the big breakthrough for Hopper is recognizing that quantization, the numerical quantization, the numerical formats, has a fair amount of innovation and ability to reduce, because it's statistical in the first place. And now, the question is, what models could be created and trained?
And we believe that 8-bit-floating-point rather than, if you look at scientific computing today, 64-bit-floating-point.
And so, just by breaking up 64 into eight, you could increase the performance of an AI supercomputer, but just by a factor of eight by not doing 64-bit. So, that's almost, if you will, a factor of 10 almost in just a couple of generations just by recognizing that 64-bit-flowing-point wasn't necessary.
And so, one of the big things is that. The second thing is Transformer. The transformer engine is so universal and so useful that it's possible for us to design a pipeline that is shaped for learning and inferencing transformers. And so, those are probably the two biggest things.
Otherwise, it's the largest chip the world's ever made, it's the fastest chip the world's ever made, and super energy efficient and uses the fast memories of the world's ever made. And then, we connect a whole bunch of these things together so that it's fast and energy efficient. But those are all brute forcey things. But the big architecture idea is FP8 and transformer engine.
SARAH:
And when you think about then, so that's the big project refinement part of the company, we think about the more agile piece. What's the impossible application you are working on ticket today that's 10 years out you think is likely to be important? I'm sure there are a ton of them.
JENSEN:
There's a whole bunch. We're working on that don't work at the moment, but I've got a lot of confidence it will work. So, for example, autonomous driving is still making progress, but I have every confidence that it will work. I have every confidence that a robotic foundation model will be discovered.
And that through expressing yourself using human language, you could cause a megatronic system of almost different types of limbs and agility to be able to figure out how to bend itself, articulate itself to do a particular task.
SARAH:
What do you think the blockers are to that today?
JENSEN:
Oh, I have no idea, but I can't tell you. I don't know. Yeah, because we have to discover our way there. But one of the things that we do know is that we do know how to learn structure from unstructured information, language, images, and of course the next big thing is video.
And if we could just watch video and learn the structure from the video we're watching, we might be able to learn how we articulate and we might be able to generalize that and be a de-articulation system for robots. And so, I think the road signs, if you will, would suggest that the pieces are coming together.
But when we get there, I have no idea. But I think it's probably less than, my guess is going to be less than 10 years, probably about five years. And I think you're going to see some pretty amazing robots.
SARAH:
It's so exciting.
ELAD:
Yeah, there's some things along those lines too. I guess, Palmy from Google came out recently, that's a step in that direction. And I guess, that's still on the transformer architecture.
And you mentioned the transformer pipeline is baking that into what you all are doing. Are there other new architectures on the AI side that you're watching or especially you think will develop into something especially interesting?
JENSEN:
Well, there's a whole bunch of derivatives of transformers, and they're all just generally called transformers. But that basic architecture is being refined and dealt on the one hand. On the other hand, some of the stuff that we're really excited about that we did a lot of work in, we started with Ian Goodfellow's work on GANs.
And we did some really great work on a style transfer and high-resolution generation of images and which led to a whole bunch of work in variational auto encoders, which then became, if you will, a bit of a cousin of the diffusion models that came out.
And so, that entire path we played a very large role in. And there's a whole bunch of derivative works that's going to come out of that between the ability to learn structure from a giant amount of data, whether it's video or... multi-modality learning is going to be a very big thing, of course.
And then, the next part of it is generating content. And if you can generate images and you can generate 2D and 3D images, why can't you generate proteins and chemicals, and you can generate all kinds of stuff.
SARAH:
There are almost no other entrepreneurs that have gone from three founders to CEO 30 years and 700 billion of market cap. What advice do you have for entrepreneurs that listen to the show?
JENSEN:
It's a really hard job. I don't mean the CEO job. Just building a company is hard. The two of you are associated with a lot of companies being formed from the very beginning. There's nothing easy about building a startup. And I don't even understand that anyone would build a startup twice. It is such an ordeal.
ELAD:
Yeah. I try to talk people out of it, like second time founders. Because I asked, started two companies. And the second time I'm like, "Are you sure you want to do this?"
JENSEN:
Oh, there's no question you shouldn't do it. Yeah, there's no question you shouldn't.
SARAH:
It has to be some forgetting mechanism? Like with having kids, you're like, "Oh, it wasn't bad the first time."
JENSEN:
You got it. You got it. It's exactly right. For example, you have to forget how hard it was. And I don't know how I do it, but I just do. I forget the pain and suffering that goes along with doing something. Once you achieve something, you just move on to the next thing.
And once you achieve that, you move on to the next thing. And it's just like life, you put one step in front of the other. What advice? I'm reluctant to give them any advice. And the reason for that is this, almost any advice will probably discourage you from wanting to do it.
I think ignorance is one of the superpowers of an entrepreneur, and you'll never get it again. You'll never have it again. And so, the thing I really love about our company is we're reinventing ourselves constantly. We're entrepreneurs inside this company.
And all the meetings that I go to are really startup meetings, and they're all painful. They're all painful. Because you're starting something from the ground up again. You have no momentum. You're basically at zero. And it reminds me every single time how painful it is.
But it's also so rewarding when you build something and the people you built it for appreciate it and somehow it made a difference. And then, you combine that skill with some other skills and some other capability, and all of a sudden, you can do something even greater than that.
On the one hand, I would tell them that building a company is extraordinarily rewarding and all the people you get to work with, that's genuinely true. On the other hand, the pain and suffering of doing it is unlike anything you can imagine. And so, you're vulnerable, you're a superhero one day. You were a jerk the next day.
You go through these cycles and somehow you have to look beyond all of that and focus on what you're trying to do. So, I don't know if I gave him any wisdom aside from if you're determined that you want to do it, don't wait too long just go do it.
SARAH:
Before you lose your ignorance?
JENSEN:
Yeah, before you lose your ignorance. If therefore there's one attribute, I would say, you have to be determined enough to stay with your conviction on the one hand. On the other hand, you can't be stubborn, so that you can have agility so that you can continue to learn.
And so, somewhere in that balance of I believe in what I'm doing on the one hand, and I simultaneously believe that I could be wrong, on the other hand, that is weird. And you have to believe both equally hard.
SARAH:
My firm's name Conviction, you can have Agility.
ELAD:
Okay, I'll start that as a candy brand.
JENSEN:
Yeah. And we've seen startup CEOs that are incredibly talented and they're almost right. But they were so determined to be right, they forgot to be agile, to learn along the way and pivot and adapt. And so, I think that's on the one hand. And so, that's one thing to remember.
And then, the other is resilience, which comes along with forgetting. You have to forget the pain, move on. And it's a little bit like coaches saying, "Don't worry about the last point. You just got your face kicked in and you miss a quarter." When I miss a quarter, when you mentioned crypto, my hand started to sweat.
I know. My heart started to beat faster, because I remember missing the quarter. And when we missed a quarter during crypto, we missed it hard. Crypto was hard to predict. And we went from having no supply to too much. Who misses a quarter by $2 billion? That's a big number. Most of the time, you hear CEOs miss it by $15 million, not $2 billion.
ELAD:
I think Sarah had a great point in terms of you built now really one of the marquee companies in the tech world. And you're really pushing forward what is potentially one of the most important ways of all time in technology, which is AI.
10 years from now, 20 years from now, looking back, are there any specific things that you want to accomplish either through the context of the company or more broadly, or other things that you, looking back 20 years from now, you really hope happened?
JENSEN:
That's a good question. And in fact, that's a good way to think. The best way to think about what to do today is to go out into the distance, stand in the future and look back. You guys probably do the same. And so, I'll go out 10 years and look back at what did I wish I had done then, then do it now.
That's the answer. And so, there are a couple of industries we really believe we can make a contribution to. One of them is healthcare and drug discovery. This is a problem that is computationally, numerically, insanely complex. The number of combinations is beyond the number of atoms in the universe.
It's a very large problem space. And we finally have the necessary tools to maybe chip away at that. And at the very minimum, we now have the ability to understand the language of and now potentially the meaning of amino acids and sequences and proteins and chemicals and such.
And so, if you can understand the structure, you can understand the language, you can understand the meaning of the problem space, you might have a chance of solving it. And so, I think one, we're very excited about that. I'm really, really hoping that we go create a foundation model for Multiphysics, for climate science.
So that we can ask questions if these human factors and these human drivers, and we make these impact, what would happen to the earth 10 years, 20, 30 years from now? It's an insanely complicated problem. Computationally, people have estimated it.
It's anywhere from a billion to 10 billion to a hundred billion times more computation than the fastest supercomputer on the planet today. That basically says we'll never get there. On the other hand, with artificial intelligence, we might have a real chance of reducing that computation by a billion times, 10 billion times.
So, I'm hoping that we have the opportunity in our generation to make a huge contribution to these two areas. So, we're doing it. Earth-2 is our climate science system. And Clara is our medical and healthcare system to understand better how to contribute in that space.
ELAD:
Very exciting.
SARAH:
I have one last question that's as important as attacking the most computationally intensive largest search spaces in the world to save humanity and the earth from our audience. Where do the leather jackets come from?
JENSEN:
My wife.
SARAH:
Okay. So, you don't know? We have to ask your wife?
JENSEN:
I have no idea.
SARAH:
Amazing. It remains a mystery.
JENSEN:
My wife, my daughter, they're always hunting for jackets for me. Most of the jackets I have to admit that I have hanging up are too fashion forward for me to carry out. And so, these are more modest ones. But some of them are just, you have to actually be cool to wear them. And so, I just don't want to look out a place-
ELAD:
So, no foundation model-
JENSEN:
Whoever eats this should not wear those jackets.
SARAH:
Well, thank you so much for doing this, Jensen. It's been an inspiring conversation.
JENSEN:
Thank you. Really enjoy it. Keep up the good work.