by Angela Guess
Mary Shacklett recently wrote in TechRepublic, “Largely constructed around systems of record, enterprise storage has revolved around what types of storage media and management systems to buy for files with fixed-length data records, and whether the nature of the data being stored was real-time or near real-time, or batch-populated data that needed to be accessed only occasionally or simply stashed away in data archives. The entry of big data into enterprise data centers and business units changes that — a single big data file can be measured in terabytes or even petabytes. Big data on analytics platforms like Hadoop is processed in parallel, a distinct difference from the sequential processing of transactional data. Unsurprisingly, the storage considerations for big data also change.”
Shacklett goes on, “Nowhere is the change more noticeable than in the data analytics/high-performance computing (HPC) space dominated by Hadoop applications that parallel process petabytes of big data against algorithmic analytics for the purpose of data science and other complex inquiries. For HPC applications, it is difficult to consider concepts like virtualized or cloud-based storage because you need physical processors and storage platforms right there in the data center to directly process and store data and the results of queries. Consequently, the compute- and storage-intensive nature of the work precludes the economics of virtualization or the cloud that data center managers, including storage professionals, have so keenly sought for the past decade. So, too, do the large sizes of single data sets that are characteristic of big data object storage, which uses metadata tags to describe non-traditional data images such as photos, videos, audio records, document images, etc.”
Photo credit: Flickr