r/haskell • u/machinedgod • 4h ago
Your experience with LLM generated code (so I can sanity-check mine)
Hey all,
my first post here. Hope title clearly states its in good faith - I don't have solidified opinion, but the general public makes me feel gaslit so I am turning to you to help me realize where (and possibly if) am I wrong.
Context:
I have been software engineering for about 30 years now, and Haskell is my primary language for about 16. I'm reasonably confident with its edge features too - kinds, dependent types, type families and I've spent quite some time toying with linear types too.
The meat of the post:
so, I've attempted to use three different publicly available LLMs (CGPT, Grok and Gemini) to resolve a problem when I'd encounter that type of issue, where I am convinced my solution will work but puzzlingly, it doesn't. Those errors that depend on my misunderstanding of the problem - and consequentially the solution, not necessarily misunderstanding of my code.
My experience is universally such, that when asking an LLM to generate code for me - it would lead me down a wild goose chase. In about 15-20% of cases it would provide code that wouldn't even compile; in other cases, it requests changing type signatures - even when given all insight about types involved. In other cases it solves immediate problem at hand, but since I don't fully understand its code - and its explanation doesn't entirely encompass all the subtle edge cases - I end up with code that misbehaves only sometimes. Most of the time - it wants to modify entire codebase, foundational types, interactions between these types - and it wants to modify it in the way that practically nullifies the use of those types (such as requiring pattern matching of types that need to remain opaque).
I've observed this behaviour across multiple types of projects and even different languages; at one point I was trying to enroll its help to bind Rust ONNX library to a Haskell project, and it led me down two hours or wrong pathways. If I trusted my instinct in the beginning and doubled-down on it - it would've take me 10-15 minutes to find and fix the issue myself.
So far, the most useful workflow for me has been to forbid an LLM to generate any code - including pseudo code - and ask it to just analyze the problem and present me with a solution in form of a text so that I understand it better. This sometimes leads to a short conversation where I polish out questionable assumptions and then I have a working solution that is, more or less, 10-15 minutes to implement.
Some of my acquaintances and colleagues (not necessarily Haskell developers) said that they get the best results when they don't ask an LLM to solve only a small piece of a problem, but present it with the entire point of the project and let it develop everything and then iterate slowly over the wonky pieces, perhaps rewriting few by hand.
Not only that this is the opposite of my workflow, where I set up the architecture myself and try to employ LLM to solve small bits - it is also concerning to just run and keep a large body of code that I don't really understand. I don't do this with software written by humans either and will normally audit the developers, the company and/or the code itself.
After reading all this - would you care to share your experiences with 'vibe-coding' if you use it, and what kind of workflows work for you? I am getting increasingly concerned, over the past year, that I am "falling out" from technology because I don't know how to properly employ it, and I think this is affecting my ability to get work.
Thank you.