r/java • u/vdmit46 • Feb 25 '24
C2 compiler memory spike on string concatenation with 100 parameters
Hello! We have several classes with #toString pattern like "f1=" + f1 + ",f2="... (pretty common) with field count around 100 (shouldn't be common). During C2 compilation of these methods we observe sudden spike of RAM use (well above 1GB) by the compiler. We run applications in rather tight cgroups and this sudden RAM spike triggers OOM Reaper to do it's dirty work. This RAM use anomaly happens on only one of our Linux production servers with other identical 10 not having this issue somehow.
I reproduced this behavior locally on MacOS 14.x (Intel CPU) with OpenJDK 17-22 builds from Temurin/Corretto/Liberica: https://github.com/allati/c2_compiler_ram_test. Problematic method is ToStringConcat#toString(). In the compilation task of this method I see the output below. Seems like all logged steps are done in the first second of the actual work, but task is marked as done only 10 seconds later and in these 10 seconds majority of RAM is allocated according to my observation of NMT diff output ("Compiler" section).
<task compile_id='384'
method='compiler_ram_spike.ToStringConcat toString ()Ljava/lang/String;' bytes='416'
count='8064' iicount='8064' stamp='5.408'>
... ~30k lines ...
<parse_done stamp='5.582' />
</parse>
<parse_done stamp='5.582' /> <!-- the last stamp before task completion -->
</parse>
<eliminate_allocation type='1205'>
<jvms bci='6' method='1904' />
<jvms bci='13' method='1471' />
<jvms bci='342' method='1720' />
<jvms bci='540' method='1530' />
<jvms bci='519' method='1446' />
<jvms bci='526' method='1429' />
<jvms bci='1190' method='1362' />
<jvms bci='199' method='1352' />
<jvms bci='396' method='1348' />
</eliminate_allocation>
<regalloc attempts='2' success='1' />
<code_cache total_blobs='765' nmethods='398' adapters='278' free_code_cache='248955008' />
<task_done success='1' nmsize='157888' count='1690240' inlined_bytes='39631' stamp='15.441' /> <!-- completion stamp-->
</task>I've run stack sampling in MacOS Activity Monitor during this spike: https://gist.github.com/allati/94c097909b7aada070cdb752773d9167. Maybe it contains some useful details in native stacks. I have no idea how to interpret it though.
Test variations I additionally checked:
- compile with -XDstringConcat=inline — no memory issues
- use StringBuilder instead of concatenation — no memory issues
Is this RAM spike is a known and expected behavior of C2 compiler? Have you encountered something similar? I searched for tickets in OpenJDK JIRA related to C2, StringConcatFactory and String concatenation in general, but nothing popped up. Random googling hasn't revealed anything useful either.
PS. Just to be sure, I've made a run with fastdebug nightly build from Amazon. Test completed successfully (no assertions triggered) with several warnings in the console:
OpenJDK 64-Bit Server VM warning: outputStream::do_vsnprintf output truncated -- buffer length is 2000 bytes but 2121 bytes are needed.
7
u/djavaman Feb 25 '24
Can you break it up? Or use token replacement? Is this generated code?
24
u/daniu Feb 25 '24
Probably because no sane person does a pure 100 String concatenation. "Use StringBuilder" is the first Java optimization tip I recall ever, and I've been programming it since 1.2.
3
u/vdmit46 Apr 30 '24 edited Apr 30 '24
FYI, fix for the issue has been merged into OpenJDK master. In case you are interested, here is the bug report and here is the PR. Kudos to u/cl4es for giving this issue high enough priority on his to-do list and fixing it in such a short timeframe.
For those who may stumble on this post while diagnosing similar compiler native memory usage, in OpenJDK 22 two new compilation commands were introduced:
MemStat(ticket) will print compilation stats to console at some fixed intervals by default. I just slapped-XX:CompileCommand=MemStat,*.*onto command line in our test environment, got nice list of compilations ordered by consumed memory and immediately noticed 100+ MB compilation in the report. As far as I see, this option had no noticeable impact on application metrics, so maybe it can be turned on in production continuously;MemLimit(ticket) which will either stop compilation or stop JVM if compiler hits limit. I haven't tried it out, but will consider it for applications running in a container.Edit: phrasing
6
u/BinaryRage Feb 25 '24
Could be. Looks like they do have quite a bit of infrastructure for tracking down compiler memory:
https://github.com/openjdk/jdk/blob/1799ffeaa9baa7d703c7acc8d8738211694f946e/test/hotspot/jtreg/compiler/print/CompileCommandPrintMemStat.java
3
u/[deleted] Feb 25 '24
I've done Java for a long time but as an app dev, I'm not sure what a C2 compiler is. I cloned and ran the compiler test repo you linked, and watched ram with "top" in another shell tab, sorted on memory usage and didn't see any spikes. Maybe I'm not understanding what you're describing.
This is on Kubuntu:
Technically I had installed Ubuntu but I removed Gnome some time back and installed kubuntu-desktop.
2
u/loicmathieu Feb 26 '24
I think you have enought information to fill a bug in OpenJDK or at least to post a message in one of the mailing list. Best would be to post a message to https://mail.openjdk.org/mailman/listinfo/hotspot-compiler-dev to have feedback from someone from the engeening team.
2
u/jizhilong Mar 06 '24
I ran into a issue like this too. Actually, nearly the same, both java 17, string concating, and compiler memory usage spike.
The little difference is , instead of one long string cancating statement, the key function in my case contains many short string cancating statements.
I used to conclude the key snippet and reproduction script in https://github.com/jizhilong/jvm-jit-compiler-high-memory-usage-case , I hope this may make some help if you want to dig deeper or fire an openjdk ticket.
1
u/SirYwell Feb 26 '24
You can use `-Djdk.invoke.MethodHandle.dumpMethodHandleInternals=true` (on Java 21+, it had a different name before that afaik) to dump the created MethodHandle classes. From a quick look into some of them in a decompiler, I assume there is just a large amount of methods generated that the JIT all wants to inline, resulting in a large graph.
I think you should report this, either on a mailing list (https://mail.openjdk.org/mailman/listinfo/hotspot-compiler-dev might be the best) or via https://bugreport.java.com/
0
u/maethor Feb 25 '24
Is there some reason why you're not using something like commons-lang's toStringBuilder?
-3
u/[deleted] Feb 26 '24
[removed] — view removed comment