Static checks for dynamic languages using LLMs

Recently I bumped into pyright, a nice tool, offered freely by Microsoft that performs checks on Python code and warns software engineers for possible bugs.

Dynamic languages are a domain that has haunted me for lots of years and how to improve the software development process around them. But it is not an easy task. Many tools have been around for quite a while, that implemented several strategies for autocompletion and early error detection, like webstorm or vscode extensions. It's like a never-ending story; we want to be less specific when coding but have very sophisticated issue detection at development time (and not at runtime).

So, Pyright is a tool that works in this direction, analyses the code, and spots possible errors. It even has a playground that you can test your use cases and see the results. Curious as always I tested the playground:

It starts with the following snippet:

So, it seems that it works. it successfully detected that the second argument of that add function, is a string, instead of a float. But that is easy because you are using Python’s typing extension. So, what happens if I modify the example a bit, but still give some typing hints:

What is the case here? We declare that the add method should return a float and we also removed the declarations from the type parameters. In addition, we convert the x parameter as float, but not the y.

No error was detected though. It should, but nothing. It seems that it depends on typing a lot and cannot understand more complex use cases.

So, I got the idea to try that in ChatGPT, which is based on a large language model (LLM).

It seem that chatgpt provided a better analysis; it detected that we are passing a float and a string, and this should return a runtime error. It misses the return type though. But catches the runtime error. It is an improvement.

So, the vanilla GPT, with a bit of prompt engineering, does a match better job. Why is that? Mainly, because GPT understand how the program works and it not searching for erroneous patterns. So, maybe, and I say, maybe, pyright (or any next-gen tool) can assist in our quest to detect errors in dynamic languages source code.

There are no significant tooling in this direction (and it is really difficult, I know), but clearly this is the way.