Sometimes it’s easy to forget that there are a lot of logistics involved in deploying the LLMs and AI agents and engines that are so quickly revolutionizing our world.
In other words, people tend to think of these systems as only creative tools, and put aside all of the technical requirements that go into running any sort of program, including something that utilizes LLMs or neural networks.
I was hearing about this new thing called semantic analytics applications that can help with declarative optimization – making sure that a program runs well in different environments.
As pointed out by people who work on these systems, we’ve seen this kind of thing before, as in the 1970s, when database engineers were evolving data handling practices: you’re looking at how to keep something running well as its environment changes quickly.
As experts often point out, you’re dealing with a whiplash-inducing level of change, all of the time.
Programmers (or maybe more accurately, handlers) are also trying to manage the nuts and bolts of how to build and run systems that are doing cognitive things. (Maybe another way to think of it is like a zookeeper feeding a lion a better steak?)
You have new GPUs, new hardware, new services, new tools, new resources … and all of it becomes overwhelming.
MIT Research Scientist Michael Cafarella quotes:
“Imagine you have some set of concrete goals as a programmer – the programmer also has to make sure that these are very fast, very cheap, and retain high quality.”
“We might be processing 100s, 1000s, or millions of data objects, and so things like runtime and cost are really important.”
“The whole infrastructure is shaking under you every single day.”
“There are a million competing concerns in the programmer’s head.”
“Behind the scenes, (our system) is hypothesizing … or testing hundreds or thousands of ways to implement what you’ve described. After it figures all that out, it tries to choose the implementation that is the cheapest, fastest, and will deliver the highest quality.”
In the recent talk included above, we had a presentation of a multi-modal real estate tool, as an example, and a system called Palimpzest that can streamline operations based on the context. These types of tools can help us to do away with tasks like prompt writing or data labeling, again, as automation engines that change according to other changes.
So, declarative optimization might be instrumental in building the next generation of AI systems. That’s particularly true as we run up against some of the limiting factors (in terms of compute, etc.) that we might be dealing with in the long term.