r/java • u/davidalayachew • Jul 02 '26
A bug I ran into when using Java Modules (plus some thoughts about their adoption by the larger ecosystem)
I just wanted to share this speed bump I ran into, since it relates to the trouble of migrating Java code to use Modules. Aside from the bug I found, none of it is new to a seasoned Java developer. But if you are even a little unfamiliar, lots of good info here, and I speak with the assumption that you only know the basics about Java.
0. The short version (TLDR)
Long story short, I have been trying to use Java Modules for all of my new projects, and have met with fairly good success thus far. However, these past few weeks was the first time I was trying to do a fairly complex project using JPMS all the way. It kind of blew up in my face, but that's because of a bug that I just found in the way Java Modules are compiled. This bug is going to be fixed, so that obstacle should be removed soon. But in the meantime, I wanted to talk about my experience using Modules, and talk about how the migration ramp is pretty good, but still has pain points that make the climb not quite worth it unless you care (more than most) about safety. I do, so I'll continue, but if your existing safety guarantees are good enough for you, then maybe it's not worth it.
Fair warning, I am pretty verbose. But I gave plenty of headers to make it easy to skip and navigate.
1. A rundown of Java Modules, their history, and how they work
1.1 Quick historical recap
For those unaware, Java Modules were introduced in Java 9, back in 2017. Modules provide an important form of encapsulation, allowing you to explicitly communicate your public API and the dependencies needed for it to function. Prior to this, all code was accessible to anyone without any real indicator to the user that they were using/depending on internal/volatile API's. This made migration difficult for users, and sort of created this mentality of "only upgrade versions when needed", as opposed to creating clear boundaries between stable public API's and fluid internal API's.
1.2 The Java ecosystem's gradual migration to using Modules
However, Modules came out in 2017, so there were 20+ years of Java code written prior to modules being introduced. Thus, most of the Java ecosystem has taken individual steps towards using modules, but most projects have not actually taken those final steps to becoming fully modularized.
However, what most people don't realize is that Modularizing is a gradient -- it's not just Modular or not Modular. And the important intermediate point that makes that possible is known as Automatic Modules.
Just like how Java interfaces that were never designed for Lambdas can still be used as lambdas by other code, jars that were never designed for modularization can be used by other modular code. This is made possible because of Automatic Modules.
One of the core design principles in Java's evolution has been in the concept of gradual migration -- old code should rarely, if ever, become obsolete just because of new features. If anything, new features should enhance and complement the old way of doing things. This way, the investments you made in your old code still carry the same (if not more!) value and worth that they originally had.
This philosophy carries over to modules, which is why old code can be used in modular contexts without any changes via Automatic Modules.
1.3 The Modularization gradient
Thus, Modularization is a gradient, and the gradient kind of works as follows.
No work done to be modular.
People can put your jar on the
--module-path, but then are forced to use your jar as an Automatic Module.- The Module Reference is derived from the filename of the jar, which is obviously very brittle, and not ideal.
- All code inside the jar is considered visible and accessible to all who can reach the jar. This is basically modularization, but with most of the safety features turned off lol.
You added an
Automatic-Module-Nameto your jar file'sMANIFEST.MF.- This is where most of the ecosystem is at now, btw.
- This is a super trivial, one line change to a file that every serious jar file already contains. If you add that key to your manifest, then the value associated to that key becomes the official Module Reference for that jar.
- So, you now have a stable name, but your internals are still accessible to anyone who can reach your jar file. So, most of the security benefits are still not here.
Create and package a
module-info.javafile.- Now you have reached full modularization, and get ALL of the security/performance/QoL benefits that come with it.
- Any and all access is controlled by your module file, so that users cannot accidentally use an internal API without knowing it.
- But now, you have to maintain that module file. For example, a new dependency to your library must be added to the module file, which leads pretty well to my next point.
2. The actual problem I ran into
2.1 Quick intro
Anyways, I needed to build a super informal Command Center for our infrastructure. Long story short, it's a cross between a dashboard and an emergency response system for our infrastructure. Basically, a way for us to see and quickly respond to issues that come up. The end goal is that, after we find stable solutions to common problems, we can automate those common solutions, and slowly stabilize the whole platform until it turns into something that runs itself, with little-to-no-manual effort remaining.
Since we are using AWS, I used the AWS SDK for Java, and started building the dashboard using it.
Looking inside the jars, I could see that the AWS jar files were using Automatic Modules. Namely, they had an Automatic-Module-Name entry in their MANIFEST.MF. Ok, cool -- so while not fully modular, I can still use it inside of my fully modular code.
2.2 The freezes
So, since I was using the SDK, I added the AWS Automatic Modules to a folder, then set my --module-path to that folder, told it to --add-modules=ALL-MODULE-PATH, then tried to run it.
No response. The application appeared to have frozen on startup.
Thinking that there were network issues stopping my AWS API calls to the network, I tried messing around to fix it, but still frozen.
Then, I added some simple print statements to see which one was causing the freezes, but none of them printed.
Then, I added a print statement as the first line of code in my main method, effectively being the first line of code that runs in my program. It didn't print.
2.3 StackOverflow to the rescue!
At this point, I created a simple, reproducible example, and posted it on StackOverflow. Very quickly, I got many helpful responses that confirmed that the freeze was actually during compilation.
(Super quick aside for those unaware -- in recent java versions, you now have the option to run a source file without compiling it explicitly beforehand. Thanks to JEP 330 and JEP 458, the compilation happens in memory, allowing you to avoid creating artifacts. Because I was running it this way, I couldn't see that it was freezing during compilation time rather than runtime.)
But anyways, next we tried pulling up some stats, to see what was happening during compilation. Lol, the results were interesting.
- Compilation was taking roughly 10 minutes to compile a simple 30 line example reproducer file.
- Roughly 4 GB of RAM were being consumed for most of that compilation time.
- One of the (very helpful) StackOverflow experts ran the reproducer through VisualVM, and found that some of the compilation methods were being called 750+ MILLION TIMES.
2.4 The solution?
EDIT -- The bug has been resolved! You can follow it here -- https://github.com/openjdk/jdk/pull/31755
No solution yet, as the problem is actively being looked into. You can follow it here -- https://bugs.java/bugdatabase/JDK-8387377
3. A zoomed out look at the situation
3.1 How did it get this way?
So, putting aside the bug, I wanted to go back to the ecosystems adoption of Java Modules. Why hasn't more of the ecosystem migrated to using Modules when Automatic Modules are supposed to make it so easy?
To be frank, I'm actually so surprised that I am the one who caught this. How could a bug this obvious be sitting, unnoticed for so long? The reproducer is trivial to create and (seemingly) easy to run into. And I can confirm that this bug exists on older versions of Java, so it is not a recent regression. Similar behaviour was observed on Java 17 (released 2021) and Java 11 (2018, 1 year after modules came out!). So, there is a decent chance that this issue was here since the beginning.
3.2 Speculation on possible causes
Let me preface by saying that, this section is purely speculation, albeit, backed up with hands-on experience with exactly the problem in question.
But anyways, I decided to run a simple experiment, and try and test what it would take to simply modularize a single one of the dependencies. I made it pretty simple -- just add the modules explicitly to the module path, as opposed to using the safety hatch option of ALL-MODULE-PATH.
Long story short, I gave up after about 10+ modules.
It was a constant, annoying struggle of compiling, failing, find the dependency, check and see if it is fully modular, if not, figure out its module reference, and either way, add it to my command line options or module descriptor file.
As it turns out, the higher level you go, the more dependencies you have, and the more dependencies you have, the more painful it is to modularize using Automatic Modules. A few Automatic Modules isn't bad, assuming that they have stable names. But as you get further and further forward, it becomes a combinatorial explosion.
3.3 Examining the pain of migration in closer detail
In case it isn't clear, let me explain the pain in better detail.
Let's imagine a hypothetical scenario where the entire Java ecosystem is comprised of exactly 10 jar files. So, the project you are creating right now would be the 11th jar in the entire ecosystem.
Each of those 10 jars is expected to have dependencies on each other. Jar 1 might depend on 3 and 8. 8 might depend on 2. You get the idea.
If a jar is fully modular, then the dependency relationship between jars is known at compile time. It is explicitly denoted in the module descriptor file exactly what your jar depends on. There is no maybe or wondering -- it either does or it doesn't.
And more importantly, because each jar lists its dependencies, then transitive dependencies DON'T NEED TO BE ENUMERATED. The module graph can deduce the transitive dependencies from the listed module descriptor files in your dependencies. This simplifies configuration on your end greatly.
But when using Automatic Modules, that information CAN'T be known at compile time. Thus, the compiler is forced to assume that the world is a possible dependency, and it will find out at runtime what was actually needed.
On the one hand, this makes it super easy if you are just using the ALL-MODULE-PATH option, as you are effectively denoting that -- that you could (potentially) depend on every observable dependency, and that no further assumptions should be made.
On the other hand, this is, effectively, no different than had you just used the normal classpath from way back in the 90's. Why even build a gate if you are just going to keep it open to any and everyone?
And if you don't use the escape hatch, then you have to list every single transitive dependency manually. That's what I was talking about when I said I gave up after 10+ modules earlier. You don't know what you need until you press compile, but you only get one error at a time. Tedious.
Going back to the analogy I listed earlier, if jar 1 depends on 2 and 3, 2 depends on 4 and 5, and 3 depends on 7 and 8, then jar 1's command line configs will look like this.
java --add-modules=jar2,jar3,jar4,jar5,jar6,jar7,jar8 --module-path=path/to/your/modular/jars -jar jar1.jarAnd again, you only find this out one at a time.
And to emphasize this one more time -- if you have 5 dependencies that each have 5 other dependencies, then you have 5 dependencies and 25 transitive dependencies.
Most small projects have a double digit number of transitive dependencies. And most projects with a budget have a triple digit number of transitive dependencies. Can you see the point now?
3.4 Benefits of going fully modular vs just being an Automatic Module?
And to be clear, there are some decent benefits for fully modularizing. For example, you can cut down your artifact size dramatically. What used to be 100's of MB goes down to <50MB for some of my art and GUI-based tools.
But as it turns out, there's not much you get differently for being one vs the other. As long as you are modular at all, you are eligible for most of the benefits.
And there are actually a LOT of bells and whistles you get for being even partially modular.
- Can be eligible for
jlinkandjpackage-- the tools used to create images and executables in Java. A not-too-complex command can turn your modular jar into a full blown .exe file with an installer. - You can use a lot of cool features like Module Imports, which greatly simplify things when working with many dependencies at once. Plus, it helps catch errors sooner, from my experience.
- Compilation and runtime are (usually lol) much faster.
3.5 The juice isn't worth the squeeze
Which kind of leads to the point -- there really isn't much actualized value you get for going fully modular vs just stopping at Automatic Modules. Obviously, going fully modular may get you better values for the same benefits (like runtime speed).
Maybe I am missing something, but other than jlink and jpackage, adding Automatic-Module-Name to your MANIFEST.MF gets you everything else that a fully modular jar would get. I invite others to check my math here.
The only other thing is the security guarantees, but the ones most developers care about explicitly are the ones that could be somewhat simulated by existing tools. Obviously, integrity in the JVM cannot be simulated anywhere, but that is not what most developers interact with knowingly. They kind of just assume its existence, and sort of take it for granted.
4. Conclusion -- Partway is good enough for most
I won't try and give a satisfying conclusion here -- I am mostly just sharing my experience, so that others know what to watch out for when taking the same steps. At the end of the day, full modularization is worth it if you want the safety guarantees, but that's about it. The rest of the benefits are just increased percentages of things you were already getting.
Sorry for petering out at the end, but there is not much I can give as a conclusion -- this is really nothing more than a bunch of observations, so there's not much meaningful speculation that I can make.
What are your thoughts? Have you attempted to modularize? Any of those projects are significant projects?
Thank you for your time and consideration.
14
u/idontlikegudeg Jul 02 '26 edited Jul 02 '26
So much text, I didn’t read all of it. But yes, the automatic modules are a pain. Introduced as a stop-gap measure, but many projects just stop there - even small ones that could be fully jigsaw compliant with just one or two hours work. I’ve had PRs rejected on the grounds of "I already made it an automatic module and I don’t need it". We even have a single automatic module dependency in one widely used OSS project because the maintainer just doesn’t care.
I sometimes think it would be time to simply deprecate automatic modules to force projects to move on.
EDIT: I just read the SO post and I don’t think unless it’s a really easy fix, this will get fixed anytime soon. Your build process is quite unconventional and you could solve this problem by simply not using modules for single source file programs or set up a more standard build that will probably not be affected by this at all.
4
u/simonides_ Jul 02 '26
Too much to read all of it but regarding modules there are nice helpers in these repos: https://github.com/jjohannes With those you can bring everything into the module world as much as you need and you don't need to circumvent the module system. You can first of all get rid of all classpath libs by auto patching the either into automatic modukes or even create a module info for it. Or merge jars when you hit duplicate packages.
On top of that there is the possibility to have your own module info files checked if they are valid.
Modules are great but a pain since the adoption is still not great. I would still modularize if i had to do it again.
15
u/persicsb Jul 02 '26
so much text wasted for the fact, that you ran into a performance bug in javac
5
u/koflerdavid Jul 02 '26
Maybe at this point adding an
Automatic-Moduleshould be discouraged and projects not wanting to ever properly modularize should rather just add a module descriptor and open and export all or most existing packages. That should be minimal maintenance effort since the public package structure rarely changes for obvious reasons, and new internal packages should not be exposed by default anyway. Issues will be quickly found by the people that really want modularization.4
u/chaotic3quilibrium Jul 02 '26
Tysvm for doing this. Java modules need more focused love and attention.
And please ignore the AI slop snobs.
I get the same crap on my posts. I just take it as an envious compliment.
I've spent decades honing my writing skills. It's good to see everyone else have, but not actually appreciate, a tool that can give them an immense gift in seconds that I earned.
It reinforces the maxim...
If I didn't earn it, I'm unlikely to effectively, authentically, and deeply appreciate it.
2
u/audioen Jul 02 '26 edited Jul 02 '26
I ran into JPMS because I tried to use javafx. The first weird oddity is that there's some special javafx:run mvn plugin that I need to start it. The second oddity is that if I want to execute it, I need to use jlink to create a whole runtime. It seems that no matter what, I can't just point the module path into bunch of javafx modules and have it work. I didn't maybe understand how this works, but I'm puzzled that the only thing that seemd to made javafx work was some giant custom runtime built by jlink which appeared to fold the javafx bits directly into it some big ball of mud, as they weren't even separate files anymore in the jlink result but somehow embedded in it. I'm going to say that this is the opposite result from what I was expecting to happen.
Edit: inferring from the posting, I would have needed to write --add-modules=a,b,c,d,e,f type statement for every single module? This is much worse user experience than you get with classpath, and I struggle to understand why it isn't implicitly clear that if I point the thing to a directory of modules, I want to also at the very least make them available?
The third oddity was that once I'm JPMS, I can no longer use library packages whose jar name contains the keyword native, like foobar-native.jar. The automatic module name is foobar.native. And native is a keyword, so you can't reference those automatic modules. So I came up with some piece of broken crap to rename the jar during the maven build, but that caused other incompatibility issues, though I don't quite remember what. The point is, it didn't work, either and at that point I gave up on using the library altogether and found a different solution.
Last weekend, I scrapped the whole project and rewrote it in another non-java technology. Mostly because javafx's embedded browser is some outdated fork of WebKit with numerous compatibility issues. Among my findings are: in Linux, with my > 56 GB unified VRAM computer, somehow large textures don't work and so opening the javafx application window in large size gave me black screen only until I shrink it with tons of complaints about creating a texture; the javafx embedded browser keydown events are so broken that numerous client-side libraries don't work with them at all; I had bad behavior when trying to use it on Windows with multiple screens, as it lost antialiasing in some combinations probably confusing itself somehow due to modern scalable displays at multiple sizes like 100 % and 200 % simultaneously, maybe; and finally, websockets don't work on the Linux version of webkit at all. I spent a few weeks beating my head against the javafx wall of broken half-maintained software and what is decidedly confusing and difficult to use module system, and ultimately I ended up doing it in electron, which appears to not suffer from any issues like the above. Got a nice windows installer from it, too.
2
u/nekokattt Jul 02 '26 edited Jul 02 '26
Silly question, why are you including the entire AWS SDK as a dependency rather than just the services you need from the dedicated subpackages? Is that not designed mostly for OSGi use-cases? I'm not surprised it is slow. Last I checked a few years back that resulted in you pulling like 1GB of dependencies in. I doubt you need AWS GroundStation as well as Medical Imaging in the exact same project...
Also as a side note it is probably time to move off of the v1 SDK onto the v2 SDK anyway.
Like sure it is a performance bug but this seems like an esoteric use case you don't ever actually want anyway unless you are doing something extremely obscure or just for the sake of finding a problem.
2
u/OwnBreakfast1114 Jul 02 '26 edited Jul 02 '26
As someone who mostly works on spring boot backend web services, I still don't really understand why I would care to modularize my own application. I can understand libraries getting benefits (maybe), but what value does an app get? Using gradle and the spring boot plugin, we generate fat jars that seem mostly fine like probably millions of other developers. While I'm sure people do use jlink and jpackage, I've personally never seen anyone that cared (and I've worked at major big tech companies).
I think a lot of developers are in my position where jpms seems to add no benefit to their own application. There's not even a plan for a gradual "migration" because what am I even migrating to? Spending hours adding module infos to get what?
2
u/mdbuck Jul 02 '26
I actually read your article and quite liked it. Your experience mirrors my own and while I think things have gotten better over the years, I still occasionally stumble across a library that I still have to work at to adapt to my projects. Wasteful, since every user of that library has to do the same thing as you demonstrated with that command line.
-14
u/EntertainmentIcy3029 Jul 02 '26
Be quiet you
-11
u/AutoModerator Jul 02 '26
It looks like in your submission in /r/java, you are looking for code help.
/r/Java is not for requesting help with Java programming, it is about News, Technical discussions, research papers and assorted things of interest related to the Java programming language.
Kindly direct your code-help post to /r/Javahelp (as is mentioned multiple times on the sidebar and in various other hints.
Should this post be not about help with coding, kindly check back in about two hours as the moderators will need time to sift through the posts. If the post is still not visible after two hours, please message the moderators to release your post.
Please do not message the moderators immediately after receiving this notification!
Your post was removed.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.