Click here to learn more about author Jon Pilkington.
It is the era of Big Data. Businesses are generating data at an ever-increasing rate and with this rising tide of information comes the need for secure storage and extreme organization. The problem is that the traditional method of using Data Warehouses, which structure data into rigid files and formats, can be likened to using a rowboat on the ocean – it just doesn’t work. There is just too much information for IT to be able to process, store and then parse this information into reports for business users in an efficient manner.
The Pros and Cons of a Data Lake
Today, companies are increasingly turning to Data Lakes, which, as its name implies, can store a vast amount of raw data in one large pool (or ocean – depending on the extent of data in your firm). One major benefit of Data Lakes is that they can securely house diverse data repositories, including structured, semi-structured and unstructured files (think PDFs, Excel, JSON, images, etc.). Data Lakes also eliminate barriers to data access, enabling business users to easily acquire the information they need for analysis, reporting and research.
But there is a downside to Data Lakes too as they can become dark and deep. Business users often struggle to find the right data points for analysis amidst this sea of information. It is not as simple as poising a simple query – you may not find all the data you need, you could pull too much extraneous information or when the lake gets too large, you simply can’t find that one small piece of data. And the data challenge extends to the IT team as well – they have the formidable task of ensuring all data remains in compliance with Data Governance and security principles. If business users are fishing in the Data Lake, how can IT best track who is using what information and how it is being manipulated?
New Method of Taming the Data Sea
Self-service data preparation (prep) tools are rescuing business users and IT teams from drowning in data. Data prep enables business users to take raw data – in any format – and combine, blend and manipulate it into the right format for analysis, regardless of whether it’s being used in Excel or a Visual Analytics Platform from IBM Watson, Tableau, Qlik or others. The best part is that instead of waiting around for IT to run a report or provide access to the required data, business users can pull information directly from existing business intelligence reports, PDFs and other semi-structured documents. Users gain fast access to not only the right data, but all the data crucial to getting a holistic view of the business. This means more time can be spent on analysis that influences corporate decision making and enhances operational processes.
For IT, self-service data prep solutions help them serve as a data lifeguard by protecting information and ensuring data usage follows compliance regulations and internal governance policies. IT can monitor Data Lakes with the technology’s built-in governance measures, including:
• Data Masking – Confidential data is reliably obscured with random characters and inherent redaction capabilities.
• Data Retention – Version control tracking and the ability to archive relevant source data provides consistency and helps organizations meet regulatory requirements.
• Auditing and Data Lineage – Users benefit from complete audit logging and reporting of data access, along with the ability to track details of any source document for data reconciliation.
• Role-based Access – Authorized users are given prepared data sets based on specific analyst roles and needs.
To turn Big Data into actionable business insights, information must be organized and securely stored. The traditional Data Warehouse method to accomplish these goals has proven to be unwieldy, and Data Lakes are quickly becoming the preferred storage repository. Self-service data prep tools enable organizations to get maximum ROI from information housed in Data Lakes. The technology provides the ease-of-use and flexibility that business users demand and the governance, automation and scalability needed by IT. As a result, companies are no longer drowning in their data – they are now putting it to work for enhancing decision making processes and operational guidance.