Software development is increasingly automated. The existence of shortcuts, reference architectures, rapid application development environment design tools and configuration management accelerators have been around for half a century or so, but some of those functionalities are becoming if not obsolete, then potentially superseded by a new age of AI-empowered software coding tools.
The shift to automate many of the roles carried out inside an enterprise’s information technology department is gradual, but systematic. Traditionally, tasks like database administration and running ETL extract, transform and load processes have required specialized skills and a high degree of manual work. But change is afoot.
Data Engineering Robots
While talk of AI replacing developers might make a good soundbite, the practical manifestation of this kind of technology may well be seen amidst the mechanics of working software systems, specifically at the data engineering level. The rise of intelligent coding assistants is reshaping the data engineering discipline; these advanced tools can now connect directly to databases and understand database schemas (the structure of how information is organized inside a database’s columns and rows or across its graph structure) and data types. They can even analyze data samples.
This means they can offer smarter data-driven coding suggestions to software developers and offer a deeper understanding of an application’s codebase within its data environment. All of which, when it works fluidly, makes life easier for developers and for business users who make use of these technology services.
Beyond ‘Simple’ Autocompletion
“A critical feature of modern coding assistants is their ability to connect directly to various database management systems. This capability catapults their utility beyond simple code autocompletion or syntax checking. By accessing database schemas directly, coding assistants give developers and data engineers valuable insights into the structure of data, including the relationships between different tables and data types of specific fields. This, in turn, allows them to better understand the data architecture and allows them to suggest more precise and contextually relevant code snippets,” explained Andrew Filev, CEO of embedded AI coding agent specialist Zencoder.
Filev provides us with a working example.
Imagine a data engineer tasked with writing a pipeline to transform and load sales data into a reporting database. A coding assistant, equipped with access to the relevant database schema, can instantly recommend the optimal way to join relevant tables, select necessary fields and even suggest appropriate filtering conditions based on historical contexts.
Understanding the data model as it does, coding assistants can ultimately alert engineers about potential inefficiencies or mistakes, such as suggesting an index that could improve query performance or warning about joining fields that could lead to unintended cartesian products. As a mathematical aide-mémoire, the cartesian product of two sets A and B is a set containing every possible ordered pair where the first element in that new set is from A and the second element is from B. A straightforward enough rule of mathematics, but one with consequences that we need our data engineering robots to be cognizant of.
Data Samples Provide Insight
“Beyond its structural understanding of information, accessing data samples further enriches a coding assistant’s capabilities. Examining actual data, coding assistants can identify common patterns, anomalies and the distribution of values within the dataset. This provides a more comprehensive level of suggestion that aligns closely with real-world scenarios the software engineer might encounter,” advised Filev.
Let’s look at another example. Take a data engineer developing a transformation script to “cleanse” data (so that it is deduplicated, verified for syntax, or perhaps spellchecked) entered via a web form belonging to an online internet-based application as an example. A coding assistant with access to past data entries can propose data cleansing routines based on frequent issues, such as typical patterns of corruption or common outliers.
If the data shows a recurrent formatting error in dates or numerical entries that contradict expected values, a coding assistant can preemptively highlight these discrepancies, offering constructive advice on correcting them before they propagate through the system.
“Unified access to both software application code and its surrounding data framework (the database schema and its data samples) provides coding assistants with an unprecedented level of contextual awareness. This is fundamental for conducting thorough analysis and validation of codebases, which is crucial in complex data engineering projects,” said Zencoder’s Filev.
This allows coding assistants to trace the flow of data across different components of a system, identifying potential bottlenecks or security vulnerabilities that might not be apparent when considering code in isolation. For instance, a coding assistant might detect a processing routine that redundantly transforms data before a database transaction where the transformation is already handled more efficiently by the database itself.
“Such insights can lead to more efficient code, saving processing time and resources,” clarified,” said Filev. “This contextual intelligence also empowers coding assistants to facilitate onboarding processes for new team members. New engineers can receive contextual queries and explanations regarding how specific operations relate to the broader system, leading to a more rapid and effective integration into the team.”
Beneath The Z Factor
We’ve been deliberately granular here in an attempt to move away from the “AI models will code all our apps” promises and predictions. Although much of Zuckerberg’s prophesizing over the rise of electronic mid-level engineers will likely come true over time, the core of the matter is already playing out at the data coalface.
As the Zencoder team point out from practical experience, coding assistants are dramatically changing how data engineers approach their work by connecting directly to databases to provide smart, context-aware advice that helps streamline workflows and improve code management. They make coding faster and also make it easier to understand and manage complex data systems.
“As these assistants become more advanced, their value will go beyond boosting individual productivity. They’ll create collaborative environments where teams can share insights effortlessly, leading to smarter, data-driven decisions. For data engineering, this shift has the potential to drive real innovation, improving both efficiency and reliability in managing and using data,” concluded Filev, at the close of a deep dive session delivered to uncover the realities behind this subject.
What might matter most in the immediate future is how well organizations are able to forge new partnerships between human data engineers and their software-based coding assistant counterparts. That might sound a little ethereal or insubstantial, but early evidence seems to suggest that it is where working software practices actually consider AI entities as team members that the most progressive work now gets done.
