When you’re dealing with large-scale data systems in today’s digital age of collecting, refining and leveraging data, data integrity is non-negotiable. Processes like data scrubbing help catch and correct errors before they cause real damage, but these processes can also lead to a lot of data movement behind the scenes. That extra load can strain system resources and slow down overall system performance, creating a significant trade-off for those requiring clean data and efficient operations.
Relating to traditional data scrubbing methods, the downside is they unnecessarily consume system resources by transferring large amounts of data from storage to compute nodes which could entail many networks and interfaces. By enabling computation directly on the SSD, data movement and associated overhead can be eliminated. KIOXIA Data Scrubbing Offload technology solves this dilemma by executing the data scrubbing process on the SSD, where the data is already located or can be gathered from peer SSDs. Offloading to storage reduces resource strain while maintaining data integrity.
The KIOXIA RAID Offload technology proposal, and ‘Best of Show’ award winner at FMS 2024, can be leveraged for data scrubbing, erasure coding and data rebuilding to tackle the inefficiencies of unnecessary data movement while helping to preserve the integrity of your precious data. It can work on individual data stripes or across multiple data stripes with the ability to handle small datasets up to massive ones, all while optimizing performance and reducing power consumption. We ran some performance and system utilization tests1 on an initial proof of concept (PoC) using a PCIe® 4.0 configuration2 and obtained the following results when data scrubbing offload was enabled versus a host-based solution without data scrubbing offload:
- With a higher level of parallel computing (3/5/9 SSDs), data scrubbing scales linearly with every added SSD compared to a fixed compute bandwidth with CPU
- DRAM bandwidth savings are proportional to original data transfer
- Even with a limited performing PoC, more data is processed per 1% of the CPU used - in production, usage reductions will continue to decrease
To sum it up, as the data sphere is exploding with the advent of AI and availability of high-capacity SSDs, such as the recent announcement of the KIOXIA LCD Series, with SSD capacities up to 245 TB, the data scrubbing problem will consume significant system resources for throw-away work under traditional data scrubbing methods.
KIOXIA RAID Offload technology can be leveraged for data integrity checks by offloading data scrubbing to SSDs where the data resides. It allows for early error detection and correction, minimizes system resource waste and enhances overall system performance. For those managing large data centers or simply interested in efficient and sustainable data processing, keep an eye on this technology. A technical brief is available here.
FOOTNOTES:
1 Based on testing performed by Kioxia Corporation and completed in July 2024. The testing was demonstrated at FMS 2024. Read and write speed may vary depending on various factors such as host devices, software (drivers, OS, etc.) and read/write conditions.
2 The test configuration included one Dell™ PowerEdge™ R650xs server with one Intel® Xeon® Gold 6338N processor (2-socket, 32 cores). Underlying storage included 3, 5 or 9 KIOXIA CM7-R Series Enterprise SSDs, each with 1.92 terabyte3 capacity. mdRAID4 5 initialized the RAID volume at the time it was created.
3 Definition of capacity: Kioxia Corporation defines a megabyte (MB) as 1,000,000 bytes, a gigabyte (GB) as 1,000,000,000 bytes, a terabyte (TB) as 1,000,000,000,000 bytes and a petabyte (PB) as 1,000,000,000,000,000 bytes. A computer operating system, however, reports storage capacity using powers of 2 for the definition of 1Gbit = 230 bits = 1,073,741,824 bits, 1GB = 230 bytes = 1,073,741,824 bytes, 1TB = 240 bytes = 1,099,511,627,776 bytes and 1PB = 240 bytes = 1,125,899,906,842,624 bytes and therefore shows less storage capacity. Available storage capacity (including examples of various media files) will vary based on file size, formatting, settings, software and operating system, and/or pre-installed software applications, or media content. Actual formatted capacity may vary.
4 mdRAID 5 is Linux® software RAID that enables RAID functionality without a hardware RAID controller.
TRADEMARKS:
Dell and PowerEdge are trademarks of Dell Inc. or its subsidiaries. Intel and Xeon are trademarks of Intel Corporation or its subsidiaries. Linux is a registered trademark of Linus Torvalds in the U.S. and other countries. PCIe is a registered trademark of PCI-SIG. All other company names, product names and service names may be trademarks of third-party companies.
DISCLAIMERS:
KIOXIA America, Inc. may make changes to specifications and product descriptions at any time. The information presented in this blog is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors. Any performance tests and ratings are measured using systems that reflect the approximate performance of KIOXIA America, Inc. products as measured by those tests. In no event will KIOXIA America, Inc. be liable to any person for any direct, indirect, special or other consequential damages arising from the use of any information contained herein, even if KIOXIA America, Inc. are advised of the possibility of such damages.