Tim Ward is CEO of CluedIn. He is passionate about making data work for everyone.
As enterprises continue to navigate the complexities of digital transformation, connected data is becoming an increasingly common necessity.
Connected data is when data assets are linked together to form a more holistic view of an entity—like the customer or product—and this view is made accessible across the business. It involves integrating data assets from multiple sources or systems and building the relationships between data points so that they can be used and analyzed more effectively.
The appetite for connected data is fueling a shift from traditional relational databases to interconnected graph-based models. This evolution promises deeper insights and can facilitate a more dynamic and effective utilization of data. It can also support a wide array of data types and structures, providing flexibility and depth in data interaction and analysis.
Challenges Of Traditional Relational Databases
While relational databases have been the backbone of data storage and operations due to their structured and tabular nature, they face limitations in managing the intricate web of data connections today.
These systems, which link data through foreign keys and joins, can be inefficient when scaling complex or extensive interconnected data sets, often making it difficult to discover indirect relationships across large datasets.
Consider a business that wants to find details about a customer named John Smith. The search might return hundreds of entries for “John Smith” stored in the customer table. To narrow down to the correct John Smith, additional queries must be run manually to pull in related data from other tables—such as addresses, transaction histories or interaction logs.
This process is not only time-consuming but also limited because each query can only handle the relationships defined by the table structures. If the data analyst doesn’t specify the right criteria or if the data is fragmented across multiple tables, finding the exact John Smith related to a specific interaction or transaction becomes complex and cumbersome.
The Shift To Graph Databases
A graph-based system, on the other hand, uses nodes and edges to represent and store data, with each node representing an entity (like a customer) and each edge representing a connection or relationship between entities.
In this setup, John Smith would be a node, connected by edges to other nodes representing his interactions, transactions, addresses and perhaps relationships with other customers (like family members or business associates).
When searching for John Smith in a graph-based system, the query can leverage these connections to provide context that significantly narrows down the search. For example, if the query specifies looking for John Smith connected to a particular address or recent transaction, the graph database uses its interconnected data model to quickly traverse the relevant nodes and edges. It can efficiently explore John Smith’s direct relationships (like his transactions) and indirect relationships (like transactions involving his direct contacts) in a single query.
This not only speeds up the data retrieval process but can also enhance the accuracy of the data retrieved.
Graph databases are engineered to map out and manage these complex relationships natively. They are particularly adept at scenarios that demand rich interconnectivity, such as social networks or recommendation systems. This suitability extends to AI and machine learning applications, where dynamic relational algorithms benefit from the inherent flexibility of graph structures.
Challenges With Graph Databases And Future Implications
Transitioning to graph databases involves more than just technical changes. Here are a couple of factors to keep in mind.
• This shift requires significant effort in data normalization and enrichment and demands a shift in mindset from traditional data handling to a more connection-centric approach. This involves moving away from viewing data as isolated units to understanding it as interconnected entities. It means recognizing the relationships and dependencies between different data points and adopting flexible data models that can evolve with the organization’s needs. Some ways of easing the transition include education and training for stakeholders, starting small and scaling gradually, encouraging cross-departmental collaboration and measuring and communicating success.
• The perceived complexity of graph query languages compared to SQL poses an additional learning curve that can deter adoption. Training programs, including specialized training on graph query languages like Cypher, Gremlin or SPARQL as well as hands-on workshops and labs are common ways of upskilling teams to prepare for the change. It typically takes around 12 months for someone to become proficient.
Graph Vs. Relational In AI And Machine Learning
The debate between the appropriateness of graph databases versus relational databases for AI and machine learning highlights the need to choose the right tools for the right job.
For example, graph databases excel in environments where relationships drive functionality, offering advantages in developing custom large language models (LLMs) and other advanced AI-driven applications. Relational databases, on the other hand, are designed to manage structured data with well-defined schemas and are often used for training models on tabular data, such as customer demographics, sales records and sensor readings.
In general, relational databases should be used when the data is highly structured and fits well into a tabular format and when the application involves historical or time-series data analysis. Graph databases offer a better option when:
1. The relationships between data points are complex and central to the application’s functionality.
2. The application involves semantic networks, knowledge graphs or context-rich data for AI models.
3. Flexibility and adaptability to evolving data and relationships are important.
4. There is a need for efficient traversal and querying of interconnected data.
Vision For The Connected Enterprise
The ultimate aspiration of leveraging connected data is to create a “connected enterprise.” This concept envisions an organization where data seamlessly flows across departmental boundaries, becoming a shared asset or “product” that enhances collaboration and strategic decision-making.
By adopting practices that prioritize data connectivity, companies position themselves to better manage their data landscapes and derive significant value from their information assets. The journey toward realizing the potential of connected data and achieving a connected enterprise may be complex, but it is also essential.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
