To paraphrase a popular saying: AI is like teenage sex. Everyone talks about it. Everyone thinks everyone else is doing it but only few actually do it. When you finally try it, it is not such a big deal after all.
I’ve written code professionally for 25 years but always in the context of a large code base with tools like Visual Studio and JetBrains Rider on Windows. However, I like to try out new things, so when I got a code challenge as part of an application process I entered, I took the opportunity to set up Visual Code on my Mac and to try out GitHub Copilot. What is all the hype about?
GitHub Copilot
GitHub Copilot essentially does two things:
It suggests code to complete what you write. Like IntelliSense on steroids. And it helps you understand your code by explaining issues and how to perform common tasks. GitHub Copilot gets its knowledge from the millions of code lines in GitHub repositories.
The idea is compelling: At your fingertips you can copy-paste context specific boilerplate code without a detour on stack overflow. On the other hand, it has its limitations: If you always follow in someone else’s footstep, you never get ahead. Your code will only be as good as what is already out there.
Like most people who write code, I consider myself to be above average. So I was skeptic. Wouldn’t it be better to use tools and framework that does not make heavy use of boilerplate code? Instead of being a grumpy old man, I decided to get hands on see what it can do.
The job
I tried GitHub Copilot on two projects. One was a code challenge for an interview (to write an ASP.NET Core Web API), the other is to create a website for playing my story game Mars to Stay online (I decided for an ASP.NET Core MVC web app).
The word on the internet is that Copilot can help you write unit tests, help you learn a new technology, and (maybe) help you write code faster. So let’s see how well it did on these.
Write unit tests
Copilot correctly deduces what the code under test does, finds the boundary values of the output, and generates code that verifies this.
But what if you do test driven development? If you write the test first? Can it then generate the content? Let’s try!
Copilot correctly assumes that I want to add a method to generate a dice roll between 1 and 10 when I name a test method Rolld10Test in the DiceRollerTest class. It even corrects the lower case d to an upper case. Not bad!
“Copilot Chat can help you write unit test cases by generating code snippets based on the code open in the editor or the code snippet you highlight in the editor. ”
GitHub Copilot documentation
Learn a new technology
Here I am using the GitHub Copilot chat which is a separate tab in Visual Code once I have installed the extension. I’m asking a basic question to get started. The answer is specific, correct and easy to follow.
The amount of text and the level of details are good. You get something specific right away and you can always ask into the details.
Now let's try a question that is not just a lookup in the documentation. Let's ask it to help me decide which template to use for my project.
Again, a good level of detail in the reply and again very actionable.
“GitHub Copilot Chat is intended to provide you with the most relevant answer to your question. However, it may not always provide the answer you are looking for. Users of Copilot Chat are responsible for reviewing and validating responses generated by the system to ensure they are accurate and appropriate. “
GitHub Copilot documentation
But for Copilot to guide me for technology choices, it needs to be up to date with the latest C#/.NET features. So let's see if it has picked up this neat little detail from 2023: In C# 12 you can initialise an empty list short and sweet with []. Copilot suggests the more verbose new() which was the most brief syntax in earlier versions.
Write code faster
I don’t know about this claim. Writing good code is rarely about typing fast but about writing correct code.
Copilot comes up with good names for local variables and clean syntax for standard constructions. When calling methods that don’t exist yet, suggested names and parameters are ok. However, you as the programmer still needs to at least skim the suggestion and decide whether to accept it or not. I picked up somewhere that only about 20% of suggestions are accepted. If a developer needs to read five times the code they write, you throw the mental load back on the developer. And here is the big downside of a tool like GitHub Copilot: Does it take your mind off trivial things to focus on expressing the domain knowledge? Or does it distract you with suggestions that throws you out of your line of thought?
Here Copilot comes up with the names "Mark Watney" and "Melissa Lewis" out of the blue. They don't appear in the game. I made the characters for Mars to Stay available online, so in principle a LLM could have picked up and trained on this.
Which takes us to the conclusion.
Conclusion
As an experienced software developer with this new tool in my hand: How does it feel? Can I pick up and use this new tool to create better software? Will GitHub Copilot stay in my toolbox?
When GitHub Copilot gets it right, it is super nice. You are in the zone, code flows fast, what you think is what you see. It adds a check for an edge case, a foreach loop, and with nice clean syntax.
When GitHub Copilot gets it wrong, it throws you completely off your train of thought. As if a balalaika orchestra suddenly enters the room.
You never know if you got the happy-go-lucky copilot on a LSD trip or the wise old copilot who reads your mind.
It is like the Mac user experience versus the Windows user experience: When you get a brand new Windows laptop, it wants to show you everything it can. Everything moves and changes. Weather forecast. Stock market news. Notifications that no security threats have been found. To work without distractions, you need to switch off notifications and live widgets. While the Mac just sits there quietly ready from the first boot, helping you with what you need when you need it.
I’m impressed but not convinced. For Greenfield development and learning on my own enough to prototype ideas, GitHub Copilot gives a nice integrated experience over following documentation on Microsoft Learn. However, for production code in a nicely architected backend without a lot of boilerplate code, I prefer a clean code completion that is always correct over a tool that hallucinates at will.
While Copilot Chat can be a powerful tool for generating code, it is important to use it as a tool rather than a replacement for human programming. You should always review and test the code generated by Copilot Chat to ensure that it meets your requirements and is free of errors or security concerns.
GitHub Copilot documentation
Available for new assignments
I’m an experienced C# engineer with deep understanding of complex business domains. I lead teams and design software that delivers on the business goals on time and within budget.
Right now I’m open for opportunities in the Stockholm region.