Software Complexity Metrics meet LLMs

January 21, 2024 · AI

From my PhD days I was always obsessed with Software Complexity Metrics. Back in the day, I even developed and extended some.

I even ran some experiments during my PhD years (large-scale, at least for my size), analysing maven to find out, how and if Domain-specific Languages (DSLs) are used in the Java ecosystem.

When ChatGPT was released, I was impressed on how well it “understood” programs. And as it seemed, it knew a lot about complexity metrics also.

One of the biggest problems, if you want to use those metrics to measure software quality, is that you cannot really find reliable tools to measure it (or each implementation differs), plus in our polyglotism environments, it is almost impossible to find tools that cover all your needs.

Would it be great to have a tool that knows: (a) about software complexity metrics, and (b) all programming languages (or at least the most popular ones)?

There is one, actually many, LLMs (for the following examples I would use chatgpt).

Consider the following program:

From the name we understand that it implements some kind of visitor pattern. Lets try to calculate the Halstead set of metrics for this method:

ChatGPT provided a very thorough analysis, of each metric before it calculated the final value. Pretty impressive, right?

Yes, it is. And it can do that on a variety of languages (I did not even informed this program that this is Java). You will have three issues, if you try to use them:

From time-to-time the LLM fails to find correct/accurate answers.
You cannot easily put a whole program to the LLM’s context for calculation, you have to work your way around.
It is expensive :)

I am pretty sure that time will fix all these three issues, and all tooling around software engineering and its programming, will become AI-first activities.