java - Negative Desired survivor size in GC Logs -

i working on in memory database written in java. have input source feeds system data, extraction/load phase. while running noticed jvm hanging, underlying cause full gc's getting triggered. application freezing, followed outofmemory error (gc overhead limit exceeded)

inspecting gc logs revealed lot of garbage collections both young , old. one thing surprising desired survivor size negative. trying understand why so? has encountered before?

environment:

• windows7 professional service pack1
• dell precision t7500, 24 cores
• 64 bit o/s 192 gb ram

sun jdk version: 1.7.0_13-b20 (64 bit)

jvm settings:

• -xmx100g
• -xms100g
• -verbose:gc
• -xloggc:c:/temp/logs/gc.log
• -xx:+printgctimestamps
• -xx:+printgcdetails
• -xx:+printtenuringdistribution
• -xx:maxpermsize=256m
• -xx:+useparalleloldgc

gc log excerpt(this not full log):

0.666: [gc desired survivor size 178913280 bytes, new threshold 7 (max 15) [psyounggen: 524289k->4431k(30583488k)] 524289k->4503k(100488576k), 0.0052536 secs] [times: user=0.03 sys=0.00, real=0.01 secs]

0.671: [full gc (system) [psyounggen: 4431k->0k(30583488k)] [paroldgen: 72k->4096k(69905088k)] 4503k->4096k(100488576k) [pspermgen: 8852k->8847k(21248k)], 0.0853597 secs] [times: user=0.08 sys=0.00, real=0.09 secs]

12.306: [gc desired survivor size 178913280 bytes, new threshold 7 (max 15) [psyounggen: 26214464k->389676k(30583488k)] 26218560k->393868k(100488576k), 0.1907519 secs] [times: user=0.69 sys=0.69, real=0.19 secs]

19.845: [gc desired survivor size 178913280 bytes, new threshold 7 (max 15) [psyounggen: 26604140k->4369012k(30583488k)] 26608334k->4539738k(100488576k), 2.0671426 secs] [times: user=12.67 sys=10.64, real=2.07 secs]

31.930: [gc desired survivor size 178913280 bytes, new threshold 7 (max 15) [psyounggen: 30583476k->4368998k(30583488k)] 30764447k->5793376k(100488576k), 2.4316614 secs] [times: user=14.32 sys=12.07, real=2.43 secs]

43.950: [gc desired survivor size 178913280 bytes, new threshold 7 (max 15) [psyounggen: 30583462k->4369018k(30583488k)] 32008283k->10474103k(100488576k), 3.0868838 secs] [times: user=31.76 sys=10.64, real=3.09 secs]

57.851: [gc desired survivor size -954466304 bytes, new threshold 6 (max 15) [psyounggen: 30583482k->4369011k(16019904k)] 36688571k->19053916k(85924992k), 5.0616910 secs] [times: user=45.58 sys=21.95, real=5.06 secs]

67.849: [gc desired survivor size -954466304 bytes, new threshold 5 (max 15) [psyounggen: 16019891k->7854065k(23301696k)] 30706222k->22540396k(93206784k), 1.9574226 secs] [times: user=35.38 sys=0.00, real=1.96 secs]

74.940: [gc desired survivor size -954466304 bytes, new threshold 4 (max 15) [psyounggen: 19504945k->11650800k(23301696k)] 34191281k->27450594k(93206784k), 3.2089939 secs] [times: user=54.41 sys=1.51, real=3.21 secs]

83.683: [gc desired survivor size -954466304 bytes, new threshold 3 (max 15) [psyounggen: 23301680k->11650785k(23301696k)] 39101518k->32434323k(93206784k), 4.8105989 secs] [times: user=65.82 sys=10.30, real=4.81 secs]

93.647: [gc desired survivor size -954466304 bytes, new threshold 2 (max 15) [psyounggen: 23301665k->10083445k(23301696k)] 44085203k->35191108k(93206784k), 3.9481522 secs] [times: user=54.77 sys=8.02, real=3.95 secs]

102.406: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 21734325k->10417779k(23301696k)] 46841988k->39667600k(93206784k), 4.7031362 secs] [times: user=63.29 sys=9.50, real=4.70 secs]

112.109: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 22068659k->5501380k(23301696k)] 51318480k->45289996k(93206784k), 5.6499475 secs] [times: user=51.39 sys=23.81, real=5.65 secs]

122.858: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 17152260k->5829244k(23301696k)] 56941300k->51036174k(93206784k), 3.9348524 secs] [times: user=47.49 sys=11.48, real=3.93 secs]

132.599: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 17480124k->4411050k(23301696k)] 62703442k->55124929k(93206784k), 3.2682313 secs] [times: user=35.46 sys=10.86, real=3.27 secs]

141.869: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 16061930k->5983419k(23301696k)] 66775809k->61080627k(93206784k), 3.1660854 secs] [times: user=38.39 sys=9.45, real=3.17 secs]

149.996: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 17634299k->5335607k(23301696k)] 72731507k->65656597k(93206784k), 4.4767380 secs] [times: user=55.41 sys=12.67, real=4.48 secs]

154.473: [full gc [psyounggen: 5335607k->0k(23301696k)] [paroldgen: 60320990k->29187977k(69905088k)] 65656597k->29187977k(93206784k) [pspermgen: 53148k->53088k(106560k)], 55.6080316 secs] [times: user=518.19 sys=3.43, real=55.61 secs]

210.083: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 1942k->96k(23301696k)] 29189919k->29188073k(93206784k), 0.0512343 secs] [times: user=0.50 sys=0.00, real=0.05 secs]

210.134: [full gc [psyounggen: 96k->0k(23301696k)] [paroldgen: 29187977k->29187816k(69905088k)] 29188073k->29187816k(93206784k) [pspermgen: 53088k->53087k(106624k)], 15.5128928 secs] [times: user=236.31 sys=0.19, real=15.51 secs]

239.525: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 11650880k->4622965k(23301696k)] 40838696k->33810782k(93206784k), 1.2739696 secs] [times: user=23.01 sys=0.00, real=1.27 secs]

248.572: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 16273845k->4009690k(23301696k)] 45461663k->37894227k(93206784k), 2.3307283 secs] [times: user=41.84 sys=0.00, real=2.33 secs]

256.854: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 15660570k->5707092k(23301696k)] 49545113k->43643553k(93206784k), 2.5595566 secs] [times: user=46.05 sys=0.00, real=2.56 secs]

265.471: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 17357972k->4996932k(23301696k)] 55294440k->48690372k(93206784k), 2.8941178 secs] [times: user=52.21 sys=0.00, real=2.89 secs]

275.795: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 16647812k->3781542k(23301696k)] 60342280k->52491436k(93206784k), 2.6240427 secs] [times: user=47.18 sys=0.00, real=2.63 secs]

284.083: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 15432422k->5815192k(23301696k)] 64142316k->58360954k(93206784k), 2.3724770 secs] [times: user=42.68 sys=0.00, real=2.37 secs]

291.659: [gc desired survivor size -954466304 bytes, new threshold 1 (max 15) [psyounggen: 17466072k->5357588k(23301696k)] 70011840k->63780322k(93206784k), 3.0063028 secs] [times: user=54.20 sys=0.00, real=3.01 secs]

299.322: [gc desired survivor size -1228210176 bytes, new threshold 2 (max 15) [psyounggen: 17008468k->6341939k(23301696k)] 75431206k->70078817k(93206784k), 4.1999305 secs] [times: user=65.35 sys=5.13, real=4.20 secs]

303.523: [full gc [psyounggen: 6341939k->0k(23301696k)] [paroldgen: 63736877k->38098305k(69905088k)] 70078817k->38098305k(93206784k) [pspermgen: 53125k->53125k(103360k)], 41.2081882 secs]

as can notice @ 57.851 seconds in log above, have negative desired survivor size, not sure why so?

does memory leak?

any appreciated. attaching snapshot of visualvm heap

yes, problem appears have many live objects. "gc overhead limit exceeded" means gc ran, took long time , freed little. there knobs can tweak control tresholds controls that, not improve performance.

i encountered exact same problem when creating memory intensive simulation (also 100 - 120gb max heap). solution move away objects store data, , keep inside primitive int[]. worked ended turning new open source project called banana, supports primitive data structures. banana basing data structures on it's own memory management, dynamic data structures without overhead of objects. go ahead , read project wiki, documents how things working , got sample code , benchmarks.