28 December 2023

Copilot experiences

The hype around AI has been huge and full of high expectations

In theory, all you need for programming is a text editor and development tools for your chosen language. In practice, however, we coders have been spoiled over the years with life-saving tools, such as IDEs that are full of different helper tools, language services that provide code suggestions (e.g. Intellisense) and static analysis tools that fix code before it runs (e.g. SonarQube). Indeed, AI-based tools that forge code from scratch are a natural extension of this development.

The hype around AI has been great and also full of expectations. The same has been true for AI assistants in programming and they are predicted to at least change the very nature of programming or, in worst case scenarios, to destroy the entire software development profession. So far, however, nothing has changed dramatically, but AI has already left a lasting mark on the way programmers work.

Currently, AI assistants both write code and act as a substitute for a colleague as a second set of eyes and a rubber duck. As with other any other tool, a full-time programmer doesn't have time to test every new toy, and my own experience is mainly based on Microsoft's GitHub Copilot.

I've been living in the AI assisted world myself since spring and have been using Copilot as a buddy for client projects and other smaller developments. At that time, I also moved to queue up the next iteration of Copilot X, and it was made available when autumn arrived. So I've already had some experience of the benefits and challenges.

Hallucinatory annoyance

I use Copilot mainly to help me write code. When I develop software, Copilot suggests code snippets in Visual Studio that it thinks I need. Unfortunately, the suggestions are repeatedly so badly wrong that my workflow has become: write some code, accept Copilot's suggestion, realize the suggestion is completely wrong and delete it, and then write the code the way I want it. Mainly, the suggestions are a distraction to my own thought process, and do not add anything to the efficiency of writing code.

AIs based on large language models unfortunately suffer from hallucination, i.e. presenting false information as fact. In the case of code, hallucination occurs when an AI confidently presents code proposals where the logic may be correct, but the code contains names of variables or types that do not exist, for example. Unfortunately, this is a problem that I encounter constantly in my own development work.

The most natural explanation for this is that the AI model does not know enough about the topic it is being asked to generate answers about. Copilot has been trained using open source projects on GitHub, a vanishingly small number of which are written in C#. Second, the code generated will be of exactly the quality that the model was trained on, and the quality can be varied in, for example, in hobby projects. It is possible that Copilot works more reliably with other programming languages, but I don't have extensive experience of this myself.

Wow moments

But sometimes Copilot hits the spot and that's when the true potential of AI comes out in the open. For example, Copilot reliably generates lists, quickly learns recurring issues that may come up during refactoring, and sometimes is quicker than thought with code completion, such as with comments.

At its best, AI acts as an extension of the brain, helping you to remember things you might not immediately think of. At its simplest, this reduces the need for Googling, as the AI quickly fills in something that you would have had to dig through the documentation yourself. I've also experimented with AI in programming languages whose syntax I can't remember by heart, and I've noticed how conveniently AI fills in a gap where I know I want a certain kind of functionality, but I don't know exactly how to write it. So as a learning tool, AI works brilliantly.

And you have to admit how cool it is when, with a few words, the AI generates entire methods or classes and code literally appears in front of your eyes. In these situations, thoughts sometimes turn to the question: will we in a near future need a programmers at all?

Towards and beyond the future

AI tools are a strongly dividing topic, and I find myself walking on both sides of the divide in my thinking. For some, AI is a much-needed superpower that allows code to be developed more efficiently and quickly, while for others, it's more of a curiosity as to what the big deal is. I think the differences are explained by how experienced developers use the tools. For an experienced developer, the tools may not be such a big help, but for someone just starting out in their career, AI may help them to raise their level.

In the end, I have to admit that for me, AI tools are currently more of a nuisance than a superpower. But there are also some great moments when Copilot shows its best side. Those great moments make you think that yes, you are dealing with amazing things and AI should still be given a chance.

The benefits of AI depend a lot on the use case, which is another reason not to make a final judgement when we are still in the dawn of AI and the various possibilities of AI are still expanding.

AI tools have a lot of potential, but also a lot of problems, the biggest of which is the human factor. It will be interesting to see if a human programming languages can be excluded from the programming and if AI can learn to generate ones and zeroe through natural language input from humans. We live at a moment of transition, where it is exciting to watch what the future holds and to speculate what our whole industry might look like decades from now.

Originally published here in Finnish. Thanks to Satu Kuusinen for proofreading and editing! Translated into English with a help of DeepL.