Enterprises are going all in on generative AI (GenAI), with the technology driving a massive 8% increase in worldwide IT spending this year, according to Gartner. But just because businesses are investing in GenAI doesn’t mean they’re broadly implementing it in actual production. Organizations are eager to wield the power of GenAI. However, deploying it safely – getting data governance, quality, cybersecurity, privacy, and compliance right – is proving to be a cumbersome and surprising challenge. 

There are a number of factors that make the safe use of genAI complicated. The biggest is genAI’s considerable reliance on unstructured data.

Unstructured Data Drives GenAI

There are two types of data: structured and unstructured. Structured data is any data that is organized in a traditional row-column database format or has a predefined data model, while unstructured data is all the other data that doesn’t exist in spreadsheets and databases. The latter is typically text-heavy and lacks the structural organization and properties of structured data. It’s also huge: Up to 90% of all enterprise data is unstructured.

GenAI mostly leverages unstructured data, such as the examples cited above. GenAI technologies employ this data to train and fine-tune models as well as to build enterprise AI search capabilities. This causes a problem for organizations as the vast majority of their data management solutions were built for structured data. 

Unstructured data management and handling has simply not seen the same level of attention as its structured data counterpart, with many organizations even struggling to identify all the locations where their unstructured data might live — across shared drives, cloud systems, applications, and so on. Once it is identified, unstructured data requires different, more complex management and specialized techniques in order for data teams to extract meaningful insights and patterns from it – techniques such as natural language processing, text mining, and machine learning.

The Challenge of Governing Unstructured Data

Why is unstructured data so difficult to govern, manage, and secure? Above all is its volume and variety: The massive size and complexity of unstructured data sources – from emails to documents to social media posts to multimedia files – makes it difficult for teams to keep track of and enforce consistent governance and security policies across the organization.

There’s also the issue of uncontrolled access and sharing. Once created, unstructured data proliferates rapidly across various systems, devices, and cloud services as people copy, modify, manipulate, and share the content. As a result, it becomes very easy to lose track of the data’s provenance – its origins and transformations.

Compounding this, unstructured data is highly siloed and defined by ambiguous ownership. It is often created and managed by different departments or individuals within an organization, leading to data silos and ambiguity around data ownership and accountability. While structured data is more likely to have known ownership within an organization due to understood security or cost implications, a company’s unstructured data is often either sequestered for legitimate reasons (e.g., upcoming commentary for an acquisition) or for less desired causes (e.g., political boundaries between divisions).

Finally, the formats of unstructured data are varied. Whereas structured data has collapsed into a small set of universal standards, SQL being a principal one, unstructured content systems have a multitude of formats and legacy patterns. The tools needed to manage these formats in a unified way are unique and require a commitment from the organization to deploy and use them.

Seven Essentials for Managing Unstructured Data

To properly manage unstructured data and effectively use it for GenAI projects, enterprises should focus on these seven essential areas:

1. Discover, catalog, and classify unstructured data: Automatically discover, catalog, and classify files and objects on the fly, which are essential for GenAI projects.
2. Preserve access entitlements of unstructured data: Maintain existing enterprise entitlements at source systems to ensure that only authorized users access relevant data via GenAI prompts. 
3. Trace the lineage of unstructured data: Understand data mapping and flows from source to end results, showing how the data moves from unstructured data systems to vector databases, to LLMs, and finally to endpoints.
4. Curate unstructured data: Automate the labeling or tagging of files to ensure that only relevant data with associated context is fed to GenAI models, thereby providing accurate responses with citations.
5. Sanitize unstructured data: Classify and redact or mask sensitive data from files that GenAI projects use.
6. Focus on the quality of unstructured data: Emphasize the freshness, uniqueness, and relevance of data to prevent unintended data usage in GenAI projects.
7. Secure unstructured prompts and responses with pre-configured policies: Detect, classify, and redact sensitive information on the fly, block toxic content, and enforce compliance with topic and tone guidelines.

Master Unstructured Data Management to Unlock GenAI’s True Potential

Enterprises are eager to harness the power of generative AI, but many underestimate the complexity of managing unstructured data. Unlike structured data, unstructured information presents unique challenges that most organizations are ill-equipped to handle. By recognizing these distinct hurdles and implementing the best practices outlined above, companies can safely deploy genAI across their operations. This strategic approach not only mitigates risks but also positions organizations to fully capitalize on GenAI’s transformative capabilities, unlocking unprecedented value and innovation.