This is my third and last blog on digital storage and memory projections for 2024. The last two articles focused on digital storage and memory devices including magnetic tape, HDDs, SSDs as well as NAND, DRAM and emerging memories. We also covered developments in shared storage and memory networking. This article focuses on developments in digital storage systems and software and their use in various workflows as well as some additional insights on the future of storage devices used in these systems.
2023 followed the second half of 2022 as a depressed period in terms of general digital storage (and memory) demand, which impacted storage systems sales as well as revenue from storage and memory devices. However increasing demand driven by big data analysis applications such AI for data and the changing architecture of digital storage and memory technology promise to revitalize the market for storage systems and software in 2024 and beyond.
This demand for more storage and memory will be key elements in digital transformation. According to a recent NVIDIA blog, McKinsey & Company projects that deep learning algorithms like OpenAI’s ChatGPT, further trained with corporate data, could add the equivalent of $2.5 trillion to $4.4 trillion annually across 63 business use cases.
B. S. Teh, EVP and Chief Commercial Officer at Seagate said that, “In 2024, hyperscale tech giants are expected to accelerate investment into cloud capacity in an effort to support AI program maturation. AI will also drive increased IT spend and data storage demand in the enterprise, as businesses seek a competitive edge through improvements to productivity and efficiency.” He added that, “Hard drive storage will offer mass data storage at less than one-fifth the cost of comparable all-flash solutions on a per bit basis. Relative to data center architectures, the value gap will not come close to closing next year—or over the next decade.”
The recent NVIDIA blog said that “Some NVIDIA experts in AI predict that 2024 will be all about phoning a friend — creating partnerships and collaborations with cloud service providers, data storage and analytical companies, and others with the know-how to handle, fine-tune and deploy big data efficiently.” Digital storage and memory systems and software will be key elements in applying various types of AI to organization’s data.
There is a movement to using more NAND flash memory-based storage for these new workloads as the price of this solid-state storage has declined (although there are projections of significant price hikes in the coming months as storage demand recovers). Steve Leeper, VP Product Marketing for DataDobi observed, “The trajectory of storage technology is also poised for a significant shift as the year 2024 approaches, with declining flash prices driving a broad-scale transition towards all-flash object storage systems. This shift is expected to result in superior system performance, catering adeptly to the voracious data appetites and rapid access demands of AI-driven operations.”
On the other hand, B.S. Teh, from Seagate thinks that, “Data-hungry AI will compel data centers and winning enterprises toward high-density hard drive storage to futureproof their data value by saving raw data sets as well as insights produced by AI and LLM processing. With massive amounts of data being available to users—291ZB to be generated in 2027 (IDC)—the speed at which data is growing will intensify this need.”
Continued use of HDDs is also supported by Giorio Regni, CTO of Scality, “Some all-flash vendors prognosticate the end of spinning disk (HDD) media in the coming years. While flash media and solid-state drives (SSDs) have clear benefits when it comes to latency, are making major strides in density, and the cost per GB is declining, we see HDDs holding a 3-5x density/cost advantage over high-density SSDs through 2028.”
Steve Leeper from DataDobi went on to say that, “According to Gartner’s projections, by 2027, it is expected that no less than 40% of organizations will have implemented data storage management solutions to classify, garner insights, and optimize their data assets, a significant leap from the 15% benchmark set in early 2023. This trend is likely to be propelled by the relentless expansion of data volumes, outpacing the rate at which companies can expand their IT workforce, thus elevating the indispensability of automation for data management at scale.”
There will be growth in both various types of cloud storage as well on-premise storage and this will require data management that spans these data sources. Don Boxley, CEO and Co-Founder of DH2i said, “…as organizations continue to diversify their IT portfolios, the need for solutions that offer cross-platform compatibility and seamless integration will grow in 2024. These solutions will need to support a variety of environments – from cloud services provided by different vendors to on-premises data centers and emerging container technologies.”
Data security is another important factor as ransomware and access threats proliferate. AI will enable more sophisticated attacks but can also provide better defenses. Aron Brand, CTO of CTERA said, “Storage systems will increasingly integrate AI-based defenses as an integral part of their security strategy. These systems will use machine learning to detect and neutralize cyber threats in real time, protecting data from ransomware and ensuring the integrity of stored information. This integration will strengthen cybersecurity and streamline incident response.
Solid-state controller and SSD company, Phison had some interesting things to say about security in solid-state memory and storage in a recent press release. “SSD, GPU, DRAM and other essential data center components will increasingly include device-level cryptographic identification, attestation, and data encryption to help better guard data against attack as AI deployments expose new digital threats.” And, “Ultra-rapid advancements in AI and LLMs will challenge AI infrastructure reliance on GPU and DRAM, resulting in new approaches to architecture that take greater advantage of high-capacity NAND flash. In these systems, PCIe 5.0 NAND flash will gain wider adoption to power applications in production environments at top speed and efficiency, freeing GPU and DRAM to separately run AI inference models, maximizing resource efficiency and productivity.”
An important element in digital storage is archiving. However much of the increasing demand for archiving data stems from the potential for relatively active reuse, rather than just digital data preservation. According to Bhupinder Bhullar, CEO of Swiss Vault, “AI’s mass adoption, driven by Large Language Models (LLM) in 2024, by all organizations (hospitals, government, universities, pharma companies…), will drive the exponential increase in data volumes, especially archive data, in each organization requiring more active access to cold data.”
Keeping data for long periods of time requires careful data management that must last longer than the archivists. Quantum Corporation said that, “This means your organization must have a plan for its long-term management and ongoing platform infrastructure today. How can it be easily curated, accessed, and analyzed in the future when the current experts are no longer present. Plus, how can it be ensured that data is always durably protected? The solution requires implementing an evolutionary storage architecture that can seamlessly adopt and decommission across multiple generations of platforms and technologies.”
Kiyoshi Urabe from IBM DRI Tape Solutions adds that, “Active archive requirements will not shrink, businesses must align data management to monetize outcomes while being fiscally responsible. This is a new focus that will be predominate at all levels of digital data storage management.”
There are also decisions to be made on what to archive on premise and what to put into the cloud. Philip Storey, CEO of XenData said, “In 2023 we saw many customers move their content from the cloud to an on-premises archive that combines a data tape library, disk cache and management software. And for secure remote access, an S3 interface has been easily added, creating a private cloud. We expect these trends to continue in 2024.”
Archives also favor the most cost-effective digital storage (and perhaps the greenest technology that requires less energy for long term data storage). Natlie Kremer Global Prouct and Channel Marketing Manager at Overland-Tanberg said that, “In 2024, the data storage landscape will undergo a transformative shift, with active archive solutions gaining even more prominence. Leveraging the power of LTO Tape technology for frozen data, organizations will adopt a tiered storage approach, seamlessly balancing performance, and cost efficiency. Active archives will continue to integrate Tape libraries as a reliable and cost-effective solution for long-term preservation of infrequently accessed data, ensuring durability and data integrity.” IBM’s enterprise tape is now available with up to 50TB of raw capacity and LTO-9 with 36TB of raw capacity should be available starting in 2024.
However, optical storage may be making a come-back for archival storage. Steve Santamaria, CEO of Folio Photonics says that, “In 2024, there will be a transformation in how we store and archive data with the emergence of optical storage and an alternative to active archiving systems. This trend will be driven by the growing demand for storage solutions that are not only long-lasting and secure but also energy-efficient.”
Active archives can also be an important element in ransomware protections strategies. Tony Mendoza, VP of IT at Spectra Logic said that, “In 2024, continued attacks on critical infrastructure will drive cities and states to rearchitect their approach to ransomware protection, placing equal emphasis on both prevention and recovery plans, and focusing on active archives to facilitate that. The trend will be towards immutable storage which can’t be encrypted, greater focus on air-gapped storage which can’t be reached by electronic viruses, practicing recovery events, and maintaining an accessible and current archive for all data.”
2024 promises a rebound in digital storage and software demand driven by changes in storage and memory technology, the growth of AI and the need for more data to support learning. Data security and cost control will also impact trends for data storage and management to enable digital transformations.
