Till now, IT leaders have wanted to think about the cyber safety dangers posed by permitting customers to entry massive language fashions (LLMs) like ChatGPT instantly by way of the cloud. The choice has been to make use of open supply LLMs that may be hosted on-premise or accessed by way of a non-public cloud. 

The synthetic intelligence (AI) mannequin must run in memory and, when utilizing graphics processing items (GPUs) for AI acceleration, this implies IT leaders want to think about the prices related to buying banks of GPUs to construct up sufficient reminiscence to carry the whole mannequin.

Nvidia’s high-end AI acceleration GPU, the H100, is configured with 80Gbytes of random-access reminiscence (RAM), and its specification exhibits it’s rated at 350w when it comes to power use.

China’s DeepSeek has been in a position to reveal that its R1 LLM can rival US AI with out the necessity to resort to the most recent GPU {hardware}. It does, nonetheless, profit from GPU-based AI acceleration.

However, deploying a non-public model of DeepSeek nonetheless requires vital {hardware} funding. To run the whole DeepSeek-R1 mannequin, which has 671 billion parameters in-memory, requires 768Gbytes of reminiscence. With Nvidia H100 GPUs, that are configured with 80GBytes of video reminiscence card every, 10 can be required to make sure the whole DeepSeek-R1 mannequin can run in-memory. 

IT leaders could properly be capable of negotiate quantity reductions, however the price of simply the AI acceleration {hardware} to run DeepSeek is round $250,000.

Much less highly effective GPUs can be utilized, which can assist to scale back this determine. However given present GPU costs, a server able to operating the whole 670 billion-parameter DeepSeek-R1 mannequin in-memory goes to price over $100,000.

The server may very well be run on public cloud infrastructure. Azure, for instance, offers entry to the Nvidia H100 with 900 GBytes of reminiscence for $27.167 per hour, which, on paper, ought to simply be capable of run the 671 billion-parameter DeepSeek-R1 mannequin solely in-memory.

If this mannequin is used each working day, and assuming a 35-hour week and 4 weeks a yr of holidays and downtime, the annual Azure invoice can be virtually $46,000 a yr. Once more, this determine may very well be decreased considerably to $16.63 per hour ($23,000) per yr if there’s a three-year dedication.

Much less highly effective GPUs will clearly price much less, but it surely’s the reminiscence prices that make these prohibitive. For example, taking a look at present Google Cloud pricing, the Nvidia T4 GPU is priced at $0.35 per GPU per hour, and is obtainable with as much as 4 GPUs, giving a complete of 64 Gbytes of reminiscence for $1.40 per hour, and 12 can be wanted to suit the DeepSeek-R1 671 billion-parameter mannequin entirely-in reminiscence, which works out at $16.80 per hour. With a three-year dedication, this determine comes all the way down to $7.68, which works out at slightly below $13,000 per yr.

A less expensive strategy

IT leaders can cut back prices additional by avoiding costly GPUs altogether and relying solely on general-purpose central processing items (CPUs). This setup is admittedly solely appropriate when DeepSeek-R1 is used purely for AI inference.

A current tweet from Matthew Carrigan, machine studying engineer at Hugging Face, suggests such a system may very well be constructed utilizing two AMD Epyc server processors and 768 Gbytes of quick reminiscence. The system he introduced in a collection of tweets may very well be put collectively for about $6,000.

Responding to feedback on the setup, Carrigan mentioned he is ready to obtain a processing fee of six to eight tokens per second, relying on the precise processor and reminiscence pace that’s put in. It additionally is dependent upon the size of the pure language question, however his tweet features a video exhibiting near-real-time querying of DeepSeek-R1 on the {hardware} he constructed based mostly on the twin AMD Epyc setup and 768Gbytes of reminiscence.

Carrigan acknowledges that GPUs will win on pace, however they’re costly. In his collection of tweets, he factors out that the quantity of reminiscence put in has a direct affect on efficiency. That is because of the manner DeepSeek “remembers” earlier queries to get to solutions faster. The method known as Key-Value (KV) caching.

“In testing with longer contexts, the KV cache is definitely larger than I realised,” he mentioned, and recommended that the {hardware} configuration would require 1TBytes of reminiscence as a substitute of 76Gbytes, when big volumes of textual content or context is pasted into the DeepSeek-R1 question immediate.

Shopping for a prebuilt Dell, HPE or Lenovo server to do one thing comparable is more likely to be significantly dearer, relying on the processor and reminiscence configurations specified.

A distinct approach to handle reminiscence prices

Among the many approaches that may be taken to scale back reminiscence prices is utilizing a number of tiers of reminiscence managed by a customized chip. That is what California startup SambaNova has performed utilizing its SN40L Reconfigurable Dataflow Unit (RDU) and a proprietary dataflow structure for three-tier reminiscence.

“DeepSeek-R1 is without doubt one of the most superior frontier AI fashions obtainable, however its full potential has been restricted by the inefficiency of GPUs,” mentioned Rodrigo Liang, CEO of SambaNova.

The corporate, which was based in 2017 by a bunch of ex-Solar/Oracle engineers and has an ongoing collaboration with Stanford College’s electrical engineering division, claims the RDU chip collapses the {hardware} necessities to run DeepSeek-R1 effectively from 40 racks down to at least one rack configured with 16 RDUs.

Earlier this month on the Leap 2025 convention in Riyadh, SambaNova signed a deal to introduce Saudi Arabia’s first sovereign LLM-as-a-service cloud platform. Saud AlSheraihi, vice-president of digital options at Saudi Telecom Firm, mentioned: “This collaboration with SambaNova marks a big milestone in our journey to empower Saudi enterprises with sovereign AI capabilities. By providing a safe and scalable inferencing-as-a-service platform, we’re enabling organisations to unlock the total potential of their knowledge whereas sustaining full management.”

This cope with the Saudi Arabian telco supplier illustrates how governments want to think about all choices when constructing out sovereign AI capability. DeepSeek demonstrated that there are various approaches that may be simply as efficient because the tried and examined methodology of deploying immense and dear arrays of GPUs.

And whereas it does certainly run higher, when GPU-accelerated AI {hardware} is current, what SambaNova is claiming is that there’s additionally an alternate approach to obtain the identical efficiency for operating fashions like DeepSeek-R1 on-premise, in-memory, with out the prices of getting to amass GPUs fitted with the reminiscence the mannequin wants.