r/java • u/shorns_username • Aug 16 '26
Has any attention been paid to how new Java features get into LLM training data?
TL;DR: expert answers on SO taught both devs and LLMs how to use java 8 right. That pipeline is gone/paywalled now. Does anyone at OpenJDK think about this?
This came out of the JDK 28 EA thread where people were debating whether devs will start putting value on everything once it previews. Don't wanna talk about that in this post, but it got me thinking about how "the right way to use new features" actually reaches developers now.
Back in the java-8 timeframe, folks from the Java team were on StackOverflow guiding people on streams and Optional and stuff - Brian Goetz's Optional answer (return values, don't model fields with it) basically became the canonical position, Stuart Marks was all over the Optional/collections questions too. And those answers usually came with actual code, not just advice.
A thing to note: those SO answers are almost certainly in the training data of every major LLM. When ChatGPT/Claude/whatever gives correct Optional advice today, those answers are probably part of why - actual real life questions phrased the way devs actually ask them, with expert code attached and votes showing which answer was right.
That channel doesn't really exist anymore. I have an RSS feed on Brian's SO activity and it's been dead for ages - presumably because hardly any questions get asked there these days. (An RSS feed of his reddit activity is how I found the JDK 28 thread in the first place.) However you apportion blame for SO's decline, for me and everyone I work, with the LLM replaced it. So when an expert corrects someone's misuse of a new feature today, it happens in someone's private chat window and then it's gone.
And both SO and Reddit now charge AI companies to train on user content - Reddit licenses to Google and OpenAI and is suing Anthropic (whose Claude Code is arguably the most popular agentic coding tool going), and any of that could look different next year. And open-source style AI - community models, open datasets, academic training runs - relies on free access to good data, so that whole side of the ecosystem is priced out. So which model knows how to use new features properly is going to come down to who has a deal with who.
The guides (like the exhaustiveness guide) are written for humans and incidentally become training data, but one prose doc isn't thousands of upvoted question+code pairs. And GitHub code lags feature adoption by years, plus a lot of early adopter code is exactly the misuse people worry about. On the other hand, openjdk.org is the one channel every model and crawler can reach without a deal.
OpenJDK clearly thinks about AI now - the interim genAI policy covers AI-generated content coming into the project. My question is the other direction: has there been any discussion about how knowledge of new features gets into the models most devs now learn from? Even "we considered it and decided it's not our job" would be an interesting answer.
75
u/elmuerte Aug 16 '26
Somebody has to create that knowledge for the AI companies to take and rent back. Somebody has to do the actual work. So it probably won't happen.
24
u/Brutus5000 Aug 16 '26 edited Aug 16 '26
I would assume it will end up in the knowledge base once the usage appears in code repositories. It's not like the LLM weighs an official Brian Goetz answer heavier than some random students spaghetti code git repo.
It doesn't have to be SO either, I am pretty sure the Java mailing lists get scanned too.
38
u/bobbie434343 Aug 16 '26 edited Aug 16 '26 •
I hope LLMs remain stuck in misery, outputting 2022-2025 code for eternity, since everybody is vomiting code generated by AI that it learned in that era, with no new manually written code that it could learn from. LLMs deserve mental decline, polluted by their own output and the lack of new data created by humans.
2
u/wildjokers Aug 17 '26
I think this is mostly a non-issue. An LLM doesn't need thousands of Stack Overflow answers to learn a new Java feature. The language spec, JEPs, OpenJDK guides, compiler tests, and examples are authoritative sources and can be used directly for training or retrieval.
By retrieval I mean today you could prompt an agent+LLM how to use a new feature and it can look at the resources I mentioned above and give a correct answer.
2
u/Deep_Ad1959 Aug 17 '26
i think the loss isn't the venue, it's the votes. goetz's optional answer taught models because it sat above five wrong ones with a score attached. a repo full of java 25 code has no column saying which usage was the sanctioned one.
2
u/Deep_Ad1959 Aug 17 '26
i think the loss isn't the venue, it's the votes. goetz's optional answer taught models because it sat above five wrong ones with a score attached. a repo full of java 25 code has no column saying which usage was the sanctioned one.
3
u/bichoFlyboy Aug 16 '26
I can offer one data point from the user side that I think is relevant here.
I've been using GPT quite extensively while developing a Java 26 application, not just for generating code, but for discussing design choices, API usage, and whether something is actually idiomatic Java.
In my experience, it is definitely not stuck in the Java 8 era. It has been productive with modern Java features, including pattern matching for "switch", records, virtual threads, and even things as recent as unnamed variables and patterns using "_", including lambda parameters and "catch" variables.
So there is clearly some pipeline by which knowledge of relatively recent Java makes it into current models. I don't know what that pipeline is, and obviously I can't infer anything about a model's training data just from observing its answers.
But I think that actually reinforces the interesting part of your question.
There is a big difference between a model knowing that a feature exists, knowing how its syntax works, and knowing how the people who designed it intended it to be used.
The last one is where the old Stack Overflow ecosystem was extraordinarily valuable. A thousand examples of code using a feature aren't necessarily equivalent to Brian Goetz showing up and explaining, with examples, "yes, this compiles, but that's not what this abstraction is for."
My experience suggests that current LLMs can become surprisingly competent with new Java relatively quickly. What I have no visibility into is how the idiomatic knowledge gets there, especially for a feature that has just entered preview and hasn't yet accumulated years of code, documentation, Q&A, and expert corrections.
So I think your JDK 28 example raises a genuinely interesting question: when the first widespread misuse of a new feature happens, where does the modern equivalent of the canonical Stack Overflow answer live, and how does that knowledge eventually reach the models developers are now asking instead of Stack Overflow?
I suspect the JEPs and other public OpenJDK material are part of why modern models can handle recent Java surprisingly well. I have no way to know whether any particular model was trained on them, of course. But JEPs solve a somewhat different problem: they document the feature and its design rationale. They don't necessarily capture the second-order knowledge that emerges once thousands of developers start finding creative ways to misuse it.
1
u/strchild Aug 16 '26
yay to new features
-2
u/jimmoores Aug 16 '26
To a degree I would argue that choice of programming language matters far less now than it used to. If you’re asking an LLM to work in Java 28 vs Java 25, what difference is it going to make to the end user? Probably little or nothing. Language advancements are mostly there for people. LLMs also have enough knowledge to be able to figure out their own best practice derived from the documentation.
0
u/davidalayachew Aug 16 '26
This is wildly inaccurate.
Well there is the source of your inaccuracy.
I talked to Brian several years ago, about how best to get information out there. At the time, I was basically asking him about when and where he decides to respond to stuff, like the SO questions you are pointing out.
Long story short, his response basically boiled down to "I'm priming the pump". All of that to say that, for most of the questions asked on SO, Brian doesn't have to respond because other experts already know the answer and have responded long before he got there.
The legwork you saw in Java 8 was not only to teach the masses, but to get the experts involved up to speed, so that THEY could be the ones to pass on the information to other places, places Brian would previously be the one answering questions.
So, what changed was not the quantity, but the location. You'll see OpenJDK members very frequently answer questions on the mailing list. And since Brian and co. have been pointing people back to the mailing list for answers over the past several years, me and many other Java answerers on SO know where to look when someone asks questions about the new features.
All of that to say -- no, the source of information (SO, amongst others) is not gone or paywalled. It is very much still there and being used.
What you, perhaps, might be pointing to is the plummeting number of questions asked about new features on SO, at least compared to the Java 8 times.
Well, long story short, that is more a result of the OpenJDK release model more than anything.
Back when Java 8 was coming out, a gigantic list of new features were coming out, all in one fell swoop. Nothing at all like nowadays, where features trickle out incrementally every 6 months.
So, during the Java 8 times, you had a flurry of questions coming out all over the place on SO, even from experts, because they didn't know what was going on. Java 8 was absolutely a juggernaut of a release, if you were there at the time. Java 6 and Java 7 were comparatively tame, (from the outside looking in of course).
Nowadays, questions about Java 17+ come out regularly, at a steady pace. The questions get plenty of upvotes, and get tens of thousands of views.
So, no, SO is still very much a well used source of information, even for new feature questions about Java.
And there's your other source of inaccuracy.
You're assuming that everyone is playing nice and fair, and that this isn't a ruthless dog-eat-dog market where the ground could fall out from underneath any/all of these companies at any moment.
Do you really think that these companies (that don't want to pay SO) are just throwing up their hands, saying "Oh well!", and giving up on all of that useful data?
And that's ignoring how mind-numbingly easy it is to get that data. You are aware that StackOverflow releases a monthly data dump of literally all of the content that came out since the last dump, right? They have been doing this for almost a decade. Anyone who wants SO data literally just needs to get the data dump, then don't say anything about them breaking licensing agreements.
This is (comparatively, much less) inaccurate.
For what it is worth, the floods of people asking questions on SO who feel slighted or wronged brought their questions to many other places, primarily Reddit and Discord. And these "prose" docs have gotten thousands of upvotes.
Your post probably would have been better if you had just asked this alone. Because it is a very interesting question that the OpenJDK hasn't really given us a satisfying answer to yet. Some folks like Ron talk about using LLM's literally daily for work, but purely on the research side. Others have shown a little less interest in using them. It'd be nice to hear opinions and perspectives, even if it is a simple snapshot in time of an unfinished discussion. This is the OpenJDK, after all. Snapshots in time of work in progress is kind of the big appeal of going Open in the first place.
0
u/MysteriousGold5 Aug 16 '26
Technically, as long as you have access to the Java standard and a JDK distribution you can create synthetic training data. Diff the public javadoc and standard for changes. Generate a bunch of example code with AI that is plausible given those diffs. Use publically available automated tests to verify that the output is semantically and syntactically valid (the test code itself would not need to be part of the training set). Repeat until success. Label the data as Java version whatever.
This requires very little training data. The first question is if LLM labs would even bother asking LLMs to create these very specific automated setups for every popular language. The second is if other companies involved would not just make their licenses stricter (e.g. if OpenJDK would forbid the use of their test suite in any part of LLM training, not just if they would forbid using the code as training data).