I think this is an important step, but it skips over that 'fault tolerant routing architecture' means you're spending die space on routes vs transistors. This is exactly analogous to using bits in your storage for error correcting vs storing data.
That said, I think they do a great job of exploiting this technique to create a "larger"[1] chip. And like storage it benefits from every core is the same and you don't need to get to every core directly (pin limiting).
In the early 2000's I was looking at a wafer scale startup that had the same idea but they were applying it to an FPGA architecture rather than a set of tensor units for LLMs. Nearly the exact same pitch, "we don't have to have all of our GLUs[2] work because the built in routing only uses the ones that are qualified." Xilinx was still aggressively suing people who put SERDES ports on FPGAs so they were pin limited overall but the idea is sound.
While I continue to believe that many people are going to collectively lose trillions of dollars ultimately pursuing "AI" at this stage. I appreciate the the amount of money people are willing to put at risk here allow for folks to try these "out of the box" kinds of ideas.
[1] It is physically more cores on a single die but the overall system is likely smaller, given the integration here.
[2] "Generic Logic Unit" which was kind of an extended LUT with some block RAM and register support.
> Xilinx was still aggressively suing people who put SERDES ports on FPGAs
This so isn't important to your overall point, but where would I begin to look into this? Sounds fascinating!
Any thoughts on why they are disabling so many cores in their current product? I did some quick noodling based on the 46/970000 number and the only way I ended up close to 900,000 was by assuming that an entire row or column would be disabled if any core within it was faulty. But doing that gave me a ~6% yield as most trials had active core counts in the high 800,000s
They did mention that they stash extra cores to enable the re-routing. Those extra cores are presumably unused when not routed in.
That was my first thought but based on the rerouting graphic it seems like the extra cores would be one or two rows and columns around the border which would only account for ~4000 cores.
"While I continue to believe that many people are going to collectively lose trillions of dollars ultimately pursuing "AI" at this stage"
Can you please explain more why you think so ?
Thank you.
It's a hype cycle with many of the hypers and deciders having zero idea about what AI actually is and how it works. ChatGPT, while amazing, is at its core a token predictor, it cannot ever get to an AGI level that you'd assume to be competitive to a human, even most animals.
And just as every other hype cycle, this one will crash down hard. The crypto crashes were bad enough but at least gamers got some very cheap GPUs out of all the failed crypto farms back then, but this time so much more money, particularly institutional money, is flowing around AI that we're looking at a repeat of Lehman's once people wake up and realize they've been scammed.
So they massively reduce the area lost to defects per wafer, from 361 to 2.2 square mm. But from the figures in this blog, this is massively outweighed by the fact that they only get 46222 sq mm useable area out of the wafer, as opposed to 56247 that the H100 gets - because they are using a single square die instead of filling the circular wafer with smaller square dies, they lose 10,025 sq mm!
Not sure how that's a win.
Unless the rest of the wafer is useable for some other customer?
It's a win because they have to test one chip, and don't have to spend resources on connecting the chiplets. The latter costs a lot (though it has other advantages). I suspect that a chiplet-based device with total 900k cores would just be not viable due to the size constraints.
If their routing around the defects is automated enough (given the highly regular structure), it may be a massive economy of efforts on testing and packaging the chip.
Is the wafer itself so expensive? I assume they don't pattern the unused area, so the process should be quicker?
> I assume they don't pattern the unused area
I’m out of date on this stuff, so it’s possible things have changed, but I wouldn’t make that assumption. It is (used to be?) standard to pattern the entire wafer, with partially-off-the-wafer dice around the edges of the circle. The reason for this is that etching behavior depends heavily on the surrounding area — the amount of silicon or copper whatever etched in your neighborhood affects the speed of etching for you, which effects line width, and (for a single mask used for the whole wafer) thus either means you need to have more margin on your parameters (equivalent to running on an old process) or have a higher defect right near the edge of the die (which you do anyway, since you can only take “similar neighborhood” so far). This goes as far as, for hyper-optimized things like SRAM arrays, leaving an unused row and column at each border of the array.
> I assume they don't pattern the unused area, so the process should be quicker?
The primary driver of time and cost in the fabrication process is the number of layers for the wafers, not the surface area, since all wafers going through a given process are the same size. So you generally want to maximize the number of devices per wafer, because a large part of your costs will be calculated at the per-wafer level, not a per-device level.
Yes, but isn't a big driver of layer costs the cost of the machines to build those layers?
For patterning, a single iteration could be (example values, no actual values used, probably only ballpark accuracy) on a 300M$ EUV machine with 5-year write off cycle, patterns on average 180 full wafers /hour. Excluding energy usage and service time, each wafer that needs full patterning would cost ~38$. If each wafer only needed half the area patterned, the lithography machine might only spend half its usual time on such a wafer, and that could double the throughput of the EUV machine, halving the write-off based cost component of such a patterning step.
Given that each layer generally consists of multiple patterning steps, a 10-20% reduction in those steps could give a meaningful reduction in time spent in the machines whose time spend on the wafer depends on the used wafer area.
This of course doesn't help reduce time in polishing or etching (and other steps that happen with whole wafers at a time), so it won't be as straightforward as % reduction in wafer area usage == % reduction in cost, but I wouldn't be surprised if it was a meaningful percentage.
> Yes, but isn't a big driver of layer costs the cost of the machines to build those layers?
Let's say the time spent in lithography step is linear the way you're describing. Even with that, the deposition step beforehand is surface area independent and would be applied across the entire wafer, and takes just as long if not longer than the lithography.
Additionally, if you were going to build a fab ground up for some specific purpose, then you might optimize the fab for those specific devices as you lay out. But most of these companies are not doing that and are simply going through TSMC or a similar subcontractor. So you've got an additional question of how far TSMC will go to accommodate customers who only want to use half a wafer, and whether that's the kind of project they could profitably cater to.
Yes, but my understanding is that the wafer is exposed in multiple steps, so there would still be less exposure steps? Probably insignificant compared to all the rest though. (Etching, moving the wafer, etc.)
EDIT: to clarify - I mean the exposure of one single pattern/layer is done in multiple steps. (https://en.wikipedia.org/wiki/Photolithography#Projection)
The number of exposure steps would be unrelated to the (surface area) size of die/device that you're making. In fact, in semiconductor manufacturing you're typically trying to maximize the number of devices per wafer because it costs the same to manufacture 1 device with 10 layers vs 100 devices with 10 layers on the same wafer. This goes so far as to have companies or business units share wafers for prototyping runs so as to minimize cost per device (by maximizing output per wafer).
Also, etching, moving, etc is all done on the entire wafer at the same time generally, via masks and baths. It's less of a pencil/stylus process, and more of a t-shirt silk-screening process.
> This goes so far as to have companies or business units share wafers for prototyping runs so as to minimize cost per device
Can this be done in production? Is there a chance that the portion of the wafer cerebras.ai can't fit their giant square in is being used for production of some other companies chips?
Good question. I think the wafer has a cost per area which is fairly significant, but I don't have any figures. There has historically been a push to utilise them more efficiently, eg by building fabs that can process larger wafers. Although mask exposure would be per processed area, I think that there are also some proportion of processing time which is per wafer, so the unprocessed area would have an opportunity cost relating to that.
AIUI Wafer marginal cost is lower than you'd expect. I had $50k in my head, quick google indicates[1] maybe <$20k at AAPL volumes? Regardless seems like the economics for Cerebras would strongly favor yield over wafer area utilization.
[1] https://www.tomshardware.com/tech-industry/tsmcs-wafer-prici...
There's also no reason they couldn't pattern that area with some other suitable commodity chips. Like how sawmills and butchers put all cuts to use.
Often those areas are used for test chips and structures for the next version. They are effectively free, so you can use them to test out ideas.
They probably pattern at least next nearest neighbors for local uniformity. That’s just litho though. The rest of the process is done all at once on the wafer
Why does their chip have to be rectangular, anyways? Couldn't they cut out a (blocky) circle too?
The cost driver for fabbing out wafers is the number of layers and the number of usable devices per wafer. Higher layer count increases cost and tends to decrease yield, and more robust designs with higher yields increase usable devices per wafer. If circles or other shapes could help with either of those, they would likely be used. Generally the end goal is to have the most usable devices per wafer, so they'll be packed as tightly as possible on the wafer so as to have the highest potential output.
Rather I wonder why do they even need to cut the extra space, instead of putting something there. I suppose that the structure of the device is highly rectangular from the logical PoV, so there's nothing useful to put there. I suspect smaller unrelated chips can be produced on these areas along the way.
I've never cut a wafer, but I assume cutting is hard and single straight lines are the easiest.
I wonder if you could… just not cut the wafer at all??
I suspect this would cause alignment issues since you could literally rotate it into the wrong position when doing soldering. That said, perhaps they could get away with cutting less and using more.
That's the idea in the article. Just one big chip. But the reason why it's normally done is that there is a pretty high defect rate, so cutting if every wafer has 1-2 defects you still get (X-1.5) devices per wafer. In the article thy go into how they avoid this problem (I think its better fault tolerance, at a cost)
The article shows them using a single maximally sized square portion of a circular wafer.
I think the proposal you're responding to is "just use the whole circular wafer without cutting out a square".
Might be jumping in without reading, but the chips you cut out of the wafer have to be delivered to physically different locations.
Normally yes. But they're using a whole wafer for a single chip! So it's actually a good idea.
I guess the issue is how do you design your routing fabric to work in the edge regions.
Actually I wonder how they are exposing this wafer. Normal chips are exposed in a rectangular batch called a reticle. The reticle mask has repeated patterns across it, and it is then exposed repeatedly across the wafer. So either they have to make a reticle mask the full size of the wafer, which sounds expensive, or they somehow have to precisely align reticle exposures so that the joined edges form valid circuits.
Additional wafer area would be a marginal increase in performance (+~20% core core best case) but increases the complexity of their design, and requires they figure out how to package/connect/house/etc. a non-standard shape. A wafer scale chip is already a huge tech risk, why spend more novelty budget on nonessential weirdness?
Why does it have to be a square? There’s no need to worry about interchangeable third-party heat sink compatibility. Is it possible to make it an irregular polygon instead of square?
It’s a win if you can use the wafer as opposed to throwing it away.
A win is a manufacturing process that results in a functioning product. Wafers, etc. aren't so scarce as to demand every mm2 be used on every one every time.
Neat. What about power density?
An H100 has a TDP of 700 watts (for the SXM5 version). With a die size of 814 mm^2 that's 0.86 W/mm^2. If the cerebras chip has the same power density, that means a cerebras TDP of 37.8 kW.
That's a lot. Let's say you cover the whole die area of the chip with water 1 cm deep. How long would it take to boil the water starting from room temperature (20 degrees C)?
amount of water = (die area of 46225 mm^2) * (1 cm deep) * (density of water) = 462 grams
energy needed = (specific heat of water) * (80 kelvin difference) * (462 grams) = 154 kJ
time = 154 kJ / 39.8 kW = 3.9 seconds
This thing will boil (!) a centimeter of water in 4 seconds. A typical consumer water cooler radiator would reduce the temperature of the coolant water by only 10-15 C relative to ambient, and wouldn't like it (I presume) if you pass in boiling water. To use water cooling you'd need some extreme flow rate and a big rack of radiators, right? I don't really know. I'm not even sure if that would work. How do you cool a chip at this power density?
The enthalpy of vaporization of water (at standard pressure) is listed by Wikipedia[1] as 2.257 kJ/g, so boiling 462 grams would require an additional 1.04 MJ, adding 26 seconds. Cerebras claims a "peak sustained system power of 23kW" for the CS-3 16 Rack Unit system[2], so clearly the power density is lower than for an H100.
[1] https://en.wikipedia.org/wiki/Enthalpy_of_vaporization#Other... [2] https://cerebras.ai/product-system/
On a tangent: has anyone built an active cooling system which operates in a partial vacuum? At half atmospheric pressure, water boils at around 80 C, which i believe is roughly the operating temperature for a hard-working chip. You could pump water onto the chip, have it vapourise, taking away all that heat, then take the vapour away and condense it at the fan end.
This is how heat pipes work, i believe, but heat pipes aren't pumped, they rely entirely on heat-driven flow. I would have thought there were pumped heat pipes. Are they called something else?
It's also not a refrigerator, because those use a pump to pressurise the coolant in its gas phase, whereas here you would only be pumping the water.
A Very Fancy cooling engine: https://www.eetimes.com/powering-and-cooling-a-wafer-scale-d...
A good talk on how Cerebras does power & cooling (8min) https://www.youtube.com/watch?v=wSptSOcO6Vw&ab_channel=Appli...
The machine that actually holds one of their wafers is almost as impressive as the chip itself. Tons of water cooling channels and other interesting hardware for cooling.
If rack mounted, you are ending up with something like a reverse power station.
So why not use it as an energy source? Spin a turbine.
If you let the chip actual boil enough water to run a turbine you're going to have a hard time keeping the magic smoke inside. Much better to run at reasonable temps and try to recover energy from the waste heat.
What if you chose a refrigerant with a lower boiling point?
That's basically the principle of binary cycle[1] generators. However for data center waste heat recovery, I'd think you'd want to use a more stable fluid for cooling, and then pump it to a separate closed-loop binary-cycle generator. No reason to make your datacenter cooling system also deal with high pressure fluids, and moving high pressure working fluid from 1000s of chips to a turbine of sufficient size, etc.
There's a bunch of places in Europe that use waste heat from datacenters in district heating systems. Same thing with waste heat from various industrial processes. It's relatively common practice.
If my very stale physics is accurate then even with perfect thermodynamic efficiency you would only recover about a third of the energy that you put into the chips.
1/3 > 0, so even if you don't get a $0 energy bill I'd venture that any company that could get 1/3 of energy bill would be happy
I'm aware of the efficiency losses but I think it would be amusing to use that turbine to help power the machine generating the heat.
Hey, we're building artificial general intelligence, what's a little perpetual motion on the side?
Minor correction, the keynote video says ~20 kW
To summarize: localize defect contamination to a very small unit size, by making the cores tiny and redundant.
Analogous to a conglomerate wrapping each business vertical in a limited liability veil so that lawsuits and bankruptcy do not bring down the whole company. The smaller the subsidiaries, the less defect contamination but also the less scope for frictionless resource and information sharing.
TSMC also have a manufacturing process used by Tesla's Dojo where you can cut up the chips, throw away the defective ones, and then reassemble working ones into a sort of wafer scale device (5x5 chips for Dojo). Seems like a more logical design to me.
I had been under the impression that Nvidia had done something similar here, but they did not talk about deploying the space saving design and instead only talked about the server rack where all of the chips on the mega wafer normally are.
https://www.sportskeeda.com/gaming-tech/what-nvlink72-nvidia...
That shield is just a prop that looks nothing like the real product. The NVL72 rack doesn't use any wafer-scale-like packaging.
It would be nice if they made it real. The cost savings from not needing so much material should be fantastic.
Amazing. I clicked a button in the azure deployment menu today...
This is a strange blog post. Their tables say:
Cerebras yields 46225 * .93 = 43000 square millimeters per wafer
NVIDIA yields 58608 * .92 = 54000 square millimeters per wafer
I don't know if their numbers are correct but it is a strange thing for a startup to brag that it is worse than a big company at something important.
Being within striking distance of SOTA while using orders of magnitude fewer resources is worth bragging about.
Very interesting. Am I correct in saying that fault tolerance here is with respect to 'static' errors that occur during manufacturing and are straightforward to detect before reaching the customer? Or can these failures potentially occur later on (and be tolerated) during the normal life of the chip?
> Take the Nvidia H100 – a massive GPU weighing in at 814mm2. Traditionally this chip would be very difficult to yield economically. But since its cores (SMs) are fault tolerant, a manufacturing defect does not knock out the entire product. The chip physically has 144 SMs but the commercialized product only has 132 SMs active. This means the chip could suffer numerous defects across 12 SMs and still be sold as a flagship part.
Fault tolerance seems to be the wrong term to use here. If I wrote this, I would have written redundant.
Redundant cores lead to a fault tolerant chip.
I have a dumb question. Why isn't silicon sold in cubes instead of cylinders?
The silicon ingots have a rotating production process that results in cylinders, not bricks.
no matter how you orient a circle on a plane, it's the same
> Second, a cluster of defects could overwhelm fault tolerant areas and disable the whole chip.
That’s an interesting point. In architecture class (which was basic and abstract so I’m sure Cerebras is doing something much more clever), we learned that defects cluster, but this is a good thing. A bunch of defects clustering on one core takes out the core, a bunch of defects not clustering could take out… a bunch of cores, maybe rendering the whole chip useless.
I wonder why they don’t like clustering. I could imagine in a network of little cores, maybe enough defects clustered on the network could… sort of overwhelm it, maybe?
Also I wonder how much they benefit from being on one giant wafer. It is definitely cool as hell. But could chiplets eat away at their advantage?
Understanding that there's inherent bias by them being competitors of the other companies, but still this article seems to make some stretches. If you told me you had an 8% core defect rate reduced 100x, I'd assume you got to close to 99% enablement. The table at the end shows... Otherwise.
They also keep flipping between cores, SMs, dies, and maybe other block sizes. At the end of the day I'm not very impressed. They seemingly have marginally better yields despite all that effort.
I think you're missing the point. The comparison is not between 93% and 92%. The comparison is between what they're getting (93%) and what you'd get if you scaled up the usual process to the core size they're using (0%). They are doing something different (namely: a ~whole wafer chip) that isn't possible without massively boosting the intra-chip redundancy. (The usual process stops working once you no longer have any extra dies to discard.)
> Despite having built the world’s largest chip, we enable 93% of our silicon area, which is higher than the leading GPU today.
The important part is building the largest chip. The icing on the top is that the enablement is not lower. Which it would be without the routing-to-spare-cores magic sauce.
And the differing terminology is because they're talking about differing things? You could call an SM a core, but it kind of contains (heterogeneous) cores itself. (I've no idea whether intra-SM cores can be redundant to boost yield.) A die is the part you break off and build a computer out of, it may contain a bunch of cores, a wafer can be broken up into multiple dies but for Cerebras it isn't.
If NVIDIA were to go and build a whole-wafer die, they'd do something similar. But Cerebras did it and got it to work. NVIDIA hasn't gotten into that space yet, so there's no point in building a product that you can't sell to a consumer or even a data center that isn't built around that exact product (or to contain a Balrog).
Looking at the H100 on the left, why is the chip yield (72) based on a circular layout/constraint? Why do they discard all of the other chips that fall outside the circle?
AFAIK all wafer ingots are cylinders, which means the wafers themselves are a circular cross section. So manufacturing is binpacking rectangles in to a circle. Plus different effects/defects in the chips based on the distance from the edge of the wafer.
So I believe its the opposite: why are they representing the larger square and implying lower yield off the wafer in space that doesnt practically exist?
Because the circle is the physical silicon. Any chips that fall outside the circle are only part of a full chip. They will be physically missing half the chip.
That's just the shape of the wafer. I don't know why the diagram continued the grid outside it.
When I was a kid, I used to get intel keychains with a die in acrylic - good job to whoever thought of that to sell the fully defective chips.
wow, fancy with the acrylic. lots of places just place a chip (I'm more familiar with RAM sticks) on a keychain and call it a day.
they're all over eBay, I just checked - the one I was thinking of, that I think I had is going for $150 - the things you get rid of....
Electronic Goldmine sells entire scrapped 200mm wafers for $15 or less
https://theelectronicgoldmine.com/search?options%5Bprefix%5D...
My biggest question is who are the buyers?
mostly 1 ai company in the middle east last I heard
The yield problem is not surprising they found a solution. Maybe they could elaborate more on the power distribution and dissipation problem?
What's yield?
It's the fraction of usable product from a manufacturing process.
When driving a car, to yield means that merging drivers must prepare to stop if necessary to let a driver on another approach proceed.
That's not necessary if you have strong weaponry mounted on your vehicle: research shows that you dont't have to stop if all the other drivers are dead.
this is an important reminder that all digital electronics is really analog but with good correction circuitry.
and run-time cpu and memory error rates are always nonzero too, though orders of magnitude lower than chip yield rates
CPUs may be very digital inside, but DRAM and flash memory are highly analog, especially MLC flash. DDR4 even has a dedicated training mode [1], during which DRAM and the memory controller learn the quirks of particular data lines and adjust to them, in order to communicate reliably.
[1]: https://www.systemverilog.io/design/ddr4-initialization-and-...