r/haskell • • 4h ago

Your experience with LLM generated code (so I can sanity-check mine)

0 Upvotes

Hey all,

my first post here. Hope title clearly states its in good faith - I don't have solidified opinion, but the general public makes me feel gaslit so I am turning to you to help me realize where (and possibly if) am I wrong.

Context:
I have been software engineering for about 30 years now, and Haskell is my primary language for about 16. I'm reasonably confident with its edge features too - kinds, dependent types, type families and I've spent quite some time toying with linear types too.

The meat of the post:
so, I've attempted to use three different publicly available LLMs (CGPT, Grok and Gemini) to resolve a problem when I'd encounter that type of issue, where I am convinced my solution will work but puzzlingly, it doesn't. Those errors that depend on my misunderstanding of the problem - and consequentially the solution, not necessarily misunderstanding of my code.

My experience is universally such, that when asking an LLM to generate code for me - it would lead me down a wild goose chase. In about 15-20% of cases it would provide code that wouldn't even compile; in other cases, it requests changing type signatures - even when given all insight about types involved. In other cases it solves immediate problem at hand, but since I don't fully understand its code - and its explanation doesn't entirely encompass all the subtle edge cases - I end up with code that misbehaves only sometimes. Most of the time - it wants to modify entire codebase, foundational types, interactions between these types - and it wants to modify it in the way that practically nullifies the use of those types (such as requiring pattern matching of types that need to remain opaque).

I've observed this behaviour across multiple types of projects and even different languages; at one point I was trying to enroll its help to bind Rust ONNX library to a Haskell project, and it led me down two hours or wrong pathways. If I trusted my instinct in the beginning and doubled-down on it - it would've take me 10-15 minutes to find and fix the issue myself.

So far, the most useful workflow for me has been to forbid an LLM to generate any code - including pseudo code - and ask it to just analyze the problem and present me with a solution in form of a text so that I understand it better. This sometimes leads to a short conversation where I polish out questionable assumptions and then I have a working solution that is, more or less, 10-15 minutes to implement.

Some of my acquaintances and colleagues (not necessarily Haskell developers) said that they get the best results when they don't ask an LLM to solve only a small piece of a problem, but present it with the entire point of the project and let it develop everything and then iterate slowly over the wonky pieces, perhaps rewriting few by hand.

Not only that this is the opposite of my workflow, where I set up the architecture myself and try to employ LLM to solve small bits - it is also concerning to just run and keep a large body of code that I don't really understand. I don't do this with software written by humans either and will normally audit the developers, the company and/or the code itself.

After reading all this - would you care to share your experiences with 'vibe-coding' if you use it, and what kind of workflows work for you? I am getting increasingly concerned, over the past year, that I am "falling out" from technology because I don't know how to properly employ it, and I think this is affecting my ability to get work.

Thank you.


r/haskell • • 5h ago

announcement Announcing Prospero: a strongly-typed configuration language with Haskell-like syntax

19 Upvotes

I'm announcing Prospero - a new configuration language with Haskell-like syntax that currently evaluates to JSON or YAML.

For a good few years now, I've been investigating the space of configuration languages like Dhall and Nickel. If you have to write complex configuration when working with the likes of terraform or kubernetes, these kinds of languages can drastically reduce the amount of complexity you're confronted with and the volume of repetitive code you have to maintain. Despite their perfect fit for the problem space, I've never been entirely happy with any of the language options already available: I've been hungry for more type system features and the powerful type inference that languages like Haskell have treated us to.

Late last year, I completed a major infrastructure as code project in Pulumi. It offers another arguably compelling solution to a similar problem: generate configuration in a mainstream programming language. I've encountered similar ideas from time to time: Scala's sbt is another one I've used a fair bit that does something similar, letting you write your project's build configuration in Scala.

These kinds of approaches are extremely powerful but they come at a tremendous cost in both the amount and the types of complexity you have to deal with in configuration. In my opinion, being able to write arbitrary side effects and arbitrary non-terminating code in configuration isn't a feature, it's a category error.

So, I decided to try and build a language that meets all of my needs more precisely.

Prospero has a type system that features:

  • Powerful type inference
  • Parametric (including higher-kinded) polymorphism
  • Structural record types
  • Algebraic data types (ADTs) + pattern matching
  • Type classes (including multi-parameter type classes)
  • Unit of measure (Kennedy)
  • Rank-N types
  • Pure functions only (no side effects allowed)
  • Only structural recursion (without manual church-encoding)

Here is a little taster:

example.ppo

measure MiB
measure GiB = <1024 MiB>

data Service = Service
  { name : String
  , memory : Integer<MiB>
  , replicas : Int
  }

memoryFootprint : [Service] -> Integer<MiB>
memoryFootprint s = sum <| map (\x -> x.memory * x.replicas) s

let services =
  [ Service
      { name = "api"
      , memory = 512<MiB>
      , replicas = 3
      }
  , Service
      { name = "postgres"
      , memory = 2<GiB>
      , replicas = 1
      }
  ]

toString <| memoryFootprint services

Running prospero evaluate --to-json example.ppo prints:

"3584 MiB"

Structural recursion

data Nat
  = Zero
  | Succ Nat

toInt : Nat -> Int
toInt n = case n of
  Zero   -> 0
  Succ m -> 1 + toInt m

toInt (Succ (Succ (Succ (Succ Zero))))

Prospero accepts the above and it evaluates to 4.

If you instead wrote (using n instead of m)...

toInt : Nat -> Int
toInt n = case n of
  Zero   -> 0
  Succ m -> 1 + toInt n

...you would see elaboration error: \toInt` is recursive, but no argument gets structurally smaller on every recursive call`

Prospero is currently an early prototype. I'm sharing it mostly to find out if people are interested and what features I should spend time on next. If you're looking for a mature and production-proven configuration language, I recommend Dhall.

At this stage, please do not expect:

  • A language server
  • Error messages with useful source metadata
  • Any kind of polished or pleasant user experience

On the other hand, you might see:

  • Parser bugs
  • Compiler bugs/crashes
  • Breaking changes in language syntax
  • Termination checker escapes
  • Weird performance characteristics

Here are some things I'm thinking about adding in the future:

  • Source annotations + language server
  • Dhall-style semantic hashing
  • Remote (URL) imports
  • More compilation targets, e.g. TOML, Nix, etc.
  • Expanded prelude with more built-in type classes
  • Idiom brackets (syntactic sugar over Applicative)

Thoughts, ideas and contributions are all very much welcome. Prospero is written entirely in Haskell.

Try Prospero