This reminds me of the boom and bust oil cycle as outlined in The Prize: The Epic Quest for Oil, Money & Power by Daniel Yergin.
I've been saying this would happen for months. There (was) a giant arbitrage for data centers that already have the infra.
If you could get a hold H100s and had an operational data center you essentially had the keys to an infinate money printer on anything above $3.50/hr.
Of course, because we live in a world of effecient markets that was never going to last forever. But they are still profitible at $2.00 assuming they have cheap electricity/infra/labor.
Problem is - u can find some at $1
The screenshot there is 1xH100 PCIE, for $1.604. Which is likely promotional pricing to get customers onboarded.
With promotional pricing it can be $0 for qualified customers.
Note also, how the author shows screenshots for invites for private alpha access. It can be mutually beneficial for the data center to provide discounted alpha testing access. The developer gets discounted access, the data center gets free/realistic alpha testing workflows.
The PCIE has much lower perf than even a 1x slice of an SXM
The real money is in renting infiniband clusters, not individual gpus/machines
If you look at lambda one click clusters they state $4.49/H100/hr
$2/h rental, not $2 sales price. Pretty misleading.
Some of the Tesla GPUs are almost at this price per unit on eBay now. I've seen them go for under $15 online.
Here's one for ~$18 inc shipping with 6GB DDR5:
https://www.ebay.com/itm/Nvidia-Tesla-K20X-6GB-90Y2351-C7S15...
If we $2 H100 this year or next.
Either AI is super dead, or a new alien GPU rained from the sky
I'm hoping for the first one
Blackwell B100/B200 did kinda rain down, also the AMD MI300X and increased availability of H200.
There's also cheaper NVIDIA L40/L40S if you don't need FP64.
Agreed, and I doubt we’ll see one retail at that price even on the secondhand market anytime soon.
That said, could I see them being offloaded in bulk for pennies on the dollar if the (presumed) AI bubble pops? Quite possibly, if it collapses into a black hole of misery and bad investments. In that case, it’s entirely plausible that some enterprising homelabs could snatch one up for a few grand and experiment with model training on top-shelf (if a generation old) kit. The SXMs are going for ~$26-$40k already, which is cheaper than the (worse performing) H100 Add-In Card when brand new; that’s not the pricing you’d expect from a “red hot” marketplace unless some folk are already cutting their losses and exiting positions.
Regardless, interesting times ahead. We either get AI replacing workers en masse, or a bust of the tech industry not seen since the dot-com bubble. Either way, it feels like we all lose.
2 bucks for a GPU? Maybe a PIC microcontroller.
They don't even have HDMI ports so they are pretty useless, but I'd buy one at $2 as a desk ornament.
GPU display stand:
https://www.reddit.com/r/nvidia/comments/1fw68rl/retiring_a_...
Bruh
> For all the desperate founders rushing to train their models to convince their investors for their next $100 million round.
Has anyone actually trained a model actually worth all this money? Even OpenAI is s struggling to staunch the outflow of cash. Even if you can get a profitable model (for what?) how many billion dollar models does the world support? And everyone is throwing money into the pit and just hoping that there's no technical advance that obsoletes everything from under them, or commiditisation leading to a "good enough" competitor that does it cheaper.
I mean, I get that everyone and/or they investors has got the FOMO for not being the guys holding the AGI demigod at the end of the day. But from a distance it mostly looks like a huge speculative cash bonfire.
> For all the desperate founders rushing to train their models to convince their investors for their next $100 million round.
I would say Meta has (though not a startup) justified the expenditure.
By freely releasing llama they undercut every a huge swath of competition who can get funded during the hype. Then when the hype dies they can pick up what the real size of the market is, with much better margins than if there were a competitive market. Watch as one day they stop releasing free versions and start rent seeking on N+1
> I get that everyone and/or they investors has got the FOMO for not being the guys holding the AGI demigod at the end of the day
Don't underestimate the power of the ego...
Look at their bonfire, we need one like that but bigger and hotter
I spit out my tea when I read your last sentence. You should consider standup comedy.
Isn’t OpenAI profitable if they stop training right at this moment? Just because they’re immediately reinvesting all that cash doesn’t mean they’re not profitable.
And if they stop training right now their "moat" (which I think is only o1 as of today) would last a good 3 to 6 months lol, and then to the Wendy's it is.
This sounds like bad news for the gpu renter farms. Am reading this right?
A good in depth mkt analysis. While it’s not crypto, many of the key points are rinse and repeat of mining - things like insatiable demand and projected ROI. Markets and tech solve high costs all the time. Great point made about the $4/hr number that was most likely a top bullet in a 1000 pitch decks citing NVIDIA. Bagholders could just be all the nations buying all the billionaire’s stories.
There is one big exception in the list of all nations. I don’t know what to make of it. Irony?
Yeah, I did this same kind of math all the time back during the early ASIC mining days except it was accelerated; you had to break even in 9 months or never due to the exponentially growing difficulty.
Last year we reached out to a major GPU vendor for a need to get access to a seven figure dollar amount worth of compute time.
They contacted (and we spoke with) several of the largest partners they had, including education/research institutions and some private firms, and could not find ANYONE that could accommodate our needs.
AWS also did not have the capacity, at least for spot instances since that was the only way we could have afforded it.
We ended up rolling our own solution with (more but lower-end) GPUs we sourced ourselves that actually came out cheaper than renting a dozen "big iron" boxes for six months.
It sounds like currently that capacity might actually be available now, but at the time we could not afford to wait another year to start the job.
If you were able to make do with cheaper GPUs, then you didn't need FP64 so you didn't need H100s in the first place right? Then you made the right choice in buying a drill for your screw work instead of renting a jackhammer even if the jackhammer would've seemed cooler to you at the time.
I was surprised recently when I fired up ollama on my refurbished Thinkpad -- a laptop that doesn't even have a GPU. All of the hype had me convinced that I couldn't run any of this LLM stuff at home!
It's a little bit slower, but while I wait for the text to generate I have another cup of coffee.
Sometimes I even prompt myself to generate some text while I'm waiting.
training is the phase that needs all that compute
This is good to know. I had read somewhere (that was probably on the Internet) that every time I submitted a prompt at the Meta AI web site that I was vaporizing an entire bottle of water, so imagine how thrilled I was to be saving so much water by prompting AI at home! But alas, the water was already vaporized. The climate? Already changed.
Current 1b model will do you no good, just rotate through all the free stuff and it would cover most of you usecases
I will admit that Llama3.1 70B does make my old Thinkpad pretty cranky. But winter is coming, so if I can change the climate of my bedroom while I'm waiting that's always a bonus this time of year.
So, where can a plebian like me buy a (or 10) used H100?
I don't expect them to hit the used market before 2026-2027. Data centers will start replacing H100 with R100 at that time.
TLDR: Don’t buy H100s. The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead.
Is the AI infra bubble already bursting?
I’m hopping more for an open weights AI boom
With cheap compute for everyone to finetune :)
Yes, please only rent instead
- sincerely, all of the cloud providers
No, but the prices will likely converge with MSRP pricing. A lot of datacenter were filled with h100s that cost a premium to get ahold of.
Covered in the article. They are below MSRP essentially
It is not just MSRP, management and operations cost too. The article goes into the details of this.
At $2 per hour, factoring in the overall hardware cost, labor, electricity, and other sunk costs like floor space and bandwidth, how many total hours does it take to break even?
What is the expected hardware operation lifespan in hours of this system?
How much would the hardware cost have to drop for the economic of $2/hour to work?
The details are in the article. They have done the math.
There was no answer to my last question which I think is the most important thing when considering if we are going to have another GFC this year or next year.