#aiinsoftwaredevelopment — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #aiinsoftwaredevelopment, aggregated by home.social.
-
I asked for tests. I got tests, plus an implementation I never asked for.
For a recent CSD workshop, I expanded the Tennis Refactoring Kata to include scoring for sets, and gave IntelliJ Junie a tight prompt: build Jest tests for winning a set, and don't write the implementation code.
It did a credible job of generating the test cases themselves. They're simple, clear, and they test what I expected. There's not enough test cases, but that wasn't surprising. It also wrote the implementation code. That isn't what I wanted, but I've yet to find a way to avoid that.
I also tried the same prompt with qwen locally and got test cases that were more idiomatically correct, but it made fatal tennis mistakes (playerOne reaches 6 games before playerTwo has any, so the set is over at 6-0 even though the test case promises 6-2).
It continues to show that the current crop of GenAI tools can't generate code that I would want to rely on.
Try the same experiment on a problem you already know well.
-
I asked for tests. I got tests, plus an implementation I never asked for.
For a recent CSD workshop, I expanded the Tennis Refactoring Kata to include scoring for sets, and gave IntelliJ Junie a tight prompt: build Jest tests for winning a set, and don't write the implementation code.
It did a credible job of generating the test cases themselves. They're simple, clear, and they test what I expected. There's not enough test cases, but that wasn't surprising. It also wrote the implementation code. That isn't what I wanted, but I've yet to find a way to avoid that.
I also tried the same prompt with qwen locally and got test cases that were more idiomatically correct, but it made fatal tennis mistakes (playerOne reaches 6 games before playerTwo has any, so the set is over at 6-0 even though the test case promises 6-2).
It continues to show that the current crop of GenAI tools can't generate code that I would want to rely on.
Try the same experiment on a problem you already know well.
-
We use GenAI everyday, but have you ever stopped to think about how they work? These tools are Mimics. Mimicry isn't the same as intelligence, is it?
To demonstrate what these tools really do, I built a GenAI out of index cards. Five groups, three cards each, one die. A group is called on, they roll, the number picks the card, and their answer is added to the response. In a very simple fashion, we demonstrated how GenAI works. It isn't intelligent. It selects the next most-likely word (or, more accurately, 'token' - a partial word chunk) in the sequence.
Two things fell out of those ten minutes that people don't expect.
The first is that GenAI doesn't learn and doesn't have memory. Nothing from the first round is carried over to the next.
The second is about scale. ChatGPT/Claude/etc has about a trillion times as much data as our human-powered model. So it's not surprising that these tools are better mimics, because they have more parameters. However, they are still not intelligent.
Join me in rolling the die to start a conversation.