The all-flash datacentre: Mirage or imminent reality?

Some say disk is dead and we’re on the cusp of the all-flash datacentre. But when will flash capacity and cost hit a point where it is suited to more than just high-tier storage?

Stephen Pritchard

Published: 30 Nov 2023

The idea of datacentres that run only on flash storage has been discussed for at least 10 years.

During that time, some pioneering IT teams have replaced conventional hard disk drives (HDDs) and other storage media such as tape with flash. But others have kept faith with spinning disks, or more often adopted a hybrid approach that mixes flash in its various form factors with HDDs.

Nonetheless, sales of enterprise-grade flash storage continue to grow, with more array suppliers that offer flash as an option, and some only flash.

However, despite improvements in capacity, durability and cost of flash, there is still significant demand for HDDs. In fact, industry experts predict the two technologies will coexist for at least another four to five years in the enterprise, and possibly longer in hyperscale cloud providers and other very heavy users of data, such as media companies.

An all-flash datacentre replaces all HDDs with flash media. This could be with solid-state drives, but it’s now likely to include nonvolatile memory express (NVMe) media or a proprietary format.

Pure Storage, which pioneered the all-flash array, uses its own DirectFlash Modules, for example.

This is distinct from the use of flash storage as a component of an array – typically a cache – used to improve input/output (I/O) across spinning disks, but there’s also a debate in the industry about whether “all flash” should refer to everything in the datacentre, including media used for backups and archives, or just primary storage.

When will the all-flash datacentre arrive?

Pure Storage first put forward the idea of an all-flash datacentre as far back as 2011, when the supplier launched an all-flash array suitable for mission-critical enterprise applications, but even Pure, as the foremost industry proponent of enterprise flash, concedes that adoption has taken time.

“Maybe not in 2012, but certainly by 2015 or 2016, there were people prepared to say, ‘I’m going to go all-flash’,” says Patrick Smith, Pure’s field chief technology officer for EMEA. “But they were talking about their primary datasets and online data. They didn’t consider nearline data requirements and content repositories that just didn’t need that performance.”

What has changed, he says, is that flash can now go head-to-head with HDDs on price.

This is underscored by industry analysts. According to Andy Buss, senior research director for Europe at IDC, this is due to the move from SLC flash to QLC flash.

“IDC has seen the rapid adoption of flash in European datacentres over the last few years, initially in high-end arrays due to the high cost of SLC flash,” he says. “The newest generations of flash storage systems, based on QLC technology, can offer much greater capacity at more attractive price points. So, all-flash arrays have spread into the mid-range and now even to entry-level storage systems.”

“We are at the point where you can get QLC flash-based systems at the price of hard disk systems,” agrees Max Mortillaro, storage analyst at GigaOm. “For me, it’s already here, at least for mainstream workloads.”

However, there are still areas of IT that hold back from all-flash.

So, why not go all-flash?

The initial advantage of flash over HDDs was speed. Flash was adopted in workstations and laptops, and in enterprise servers running performance-critical and especially I/O-dependent applications.

Flash’s performance edge is greatest on random reads and writes. The gap is narrower for sequential read/write operations. A well-configured HDD array with flash-based caching comes close enough to all-flash speeds in real-world environments.

“It does depend what infrastructure you have and what characteristics you are looking for from your storage,” says Roy Illsley, chief analyst of IT operations at Omdia. “That includes performance on read, on writes, capacity. The most appropriate [storage] for your needs could be flash, or just as equally spinning media. All flash datacentres may be a reality where workloads require the strength of flash, but I am not expecting all-flash datacentres to become commonplace.”

According Rainer Kaise, senior manager of business development at Toshiba Electronics Europe – a hard drive manufacturer – 85% of the world’s online media is still stored on HDDs.

The constraint, he suggests, is whether the flash industry can meet demand. “If we want to replace that 85%, we need five, seven or even eight times the [flash] capacity,” he says. “And that’s just the numbers from 2022.”

The price and capacity gap is narrowing, but it will be some time before it closes – and hard drives still have a price advantage, at least for now. Toshiba states that flash is still five to seven times more expensive.

Flash, HDDs and the future

Analysts, however, expect flash capacity to overtake that of HDDs. GigaOm’s Mortillaro points to supplier road maps for 75TB, 250TB and 300TB modules.

This will bring new advantages for flash such as lower maintenance – large datacentres using HDDs need to constantly replace drive modules – and lower power, cooling and rack space requirements.

As flash prices fall, workloads that were not worth moving for performance reasons, such as very large data stores, archives or streaming media, become more viable.

But we are perhaps not quite at that point. Even flash proponents suggest alternatives are needed, at least for now.

“The demise of the spinning hard disk cannot come soon enough,” says Alastair McAulay, a datacentre and cloud strategy expert at PA Consulting. “For the majority of use cases regardless of whether they are housed in laptops or datacentres, they consume more power, produce more heat and are more likely to fail without warning than their solid-state counterparts.

“However, flash storage is not the universal solution for an enterprise’s data storage needs, and there is a compelling case for an alternative.”

According to McAulay, regulations that require longer data retention periods, the need to provide vast datasets for AI, and even the need for immutable copies of data all point to the need for alternatives to flash, such as HDDs, but also tape and optical storage.

One way flash suppliers have narrowed the cost gap is by using flash performance through techniques such as deduplication and compression. But these techniques work less well on data that is already highly optimised or compressed, such as media files or backups.

As a result, many IT teams and suppliers are hedging their bets.

“When it comes to price/performance, high-density SSDs can’t fully replace HDDs, especially with respect to petabyte-scale unstructured data storage across the spectrum of application workloads,” says Paul Speciale, chief marketing officer at Scality, a supplier that sells both media formats.

“QLC flash-based data storage is not needed for all large-scale, mission-critical workloads,” he says. “We recommend and use it for latency-sensitive, read-intensive workloads that justify its higher cost. QLC flash is not an ideal fit today for most other workloads.”