Caucho Forums  

This forum is permanently closed because of spam. For free community support, please visit Google Groups:


Go Back   Caucho Forums > Resin

Reply
 
Thread Tools Display Modes
  #1  
Old 12-19-2011, 07:34 PM
theBlueSage theBlueSage is offline
Member
 
Join Date: Apr 2011
Posts: 33
Default Memory leak in 4.0.24?

we are seeing some performance degradation in our servers that are running resin-pro-4.0.24. I have attached two images that may help figure out what is going on.
The first Image shows the GC time of 9 servers, the first 5 are running 4.0.24, the last four are running 4.0.6. At around midnight last might there was a wild GC response and GC average went from 2 seconds to over 10 seconds. You can see that that does not happen on the servers that are running 4.0.6 version

The second image shows 3 days worth of data with regard to memory consumption by a single resin (web14 r10880 is the correlated server in the first image) There is a stead decline in the amount of fee memory, which in turn appears to trigger bizarre GC time behavior.






Fixing this is critical to our upgrade path...
Let me know if you need more information

thanks
Reply With Quote
  #2  
Old 12-19-2011, 10:43 PM
ferg ferg is offline
Administrator
 
Join Date: Aug 2009
Posts: 190
Default

The heap dump, or even the top summary of the heap would be very helpful (even more so the changes over time.)

You can use resin's "resinctl dump-heap" or use jmap for a quick summary.
Reply With Quote
  #3  
Old 12-20-2011, 05:55 PM
theBlueSage theBlueSage is offline
Member
 
Join Date: Apr 2011
Posts: 33
Default

Hi Ferg,

Here is the top half (minus the part that refers to our internal code) of the heap dump. The things that appear odd to me are the following as by the time I took the heapdump we had no online users on that particular server.

16.80ý 1336.132M 1336.132M 0.000M 5078528 char[]
13.42% 1067.050M 1067.050M 0.000M 415509 byte[]
7.44% 591.449M 4.003M 587.446M 24983 com.caucho.server.http.HttpResponseStream
6.67% 530.288M 170.706M 359.582M 4474964 java.lang.String
6.62% 526.640M 8.768M 517.872M 24983 com.caucho.server.http.HttpRequest

so I dont see why there should be any resource consumption by the HttpRequest, or HttpResponse servers.

here is the start of the heapdump ...


28.25% 1336.132M 1336.132M 0.000M 5078528 char[]
22.56% 1067.050M 1067.050M 0.000M 415509 byte[]
11.21% 530.288M 170.706M 359.582M 4474964 java.lang.String
3.83% 181.387M 7.984M 173.404M 130802 java.util.HashMap
3.43% 162.469M 40.769M 121.700M 141506 java.util.HashMap$Entry[]
3.02% 142.862M 88.832M 54.031M 1940555 java.util.HashMap$Entry
2.78% 131.514M 2.724M 128.790M 39670 java.util.concurrent.ConcurrentHashMap
2.51% 118.937M 0.187M 118.750M 8179 java.util.HashSet
2.49% 117.676M 28.698M 88.979M 626908 java.util.concurrent.ConcurrentHashMap$Segment
2.11% 99.583M 84.805M 14.778M 415953 java.lang.Object[]
2.00% 94.804M 5.691M 89.113M 39670 java.util.concurrent.ConcurrentHashMap$Segment[]
1.89% 89.192M 31.717M 57.474M 626908 java.util.concurrent.ConcurrentHashMap$HashEntry[]
1.88% 88.816M 35.324M 53.492M 771655 java.util.concurrent.ConcurrentHashMap$HashEntry
1.27% 60.024M 20.132M 39.892M 293188 javax.servlet.http.Cookie
1.19% 56.271M 56.270M 0.002M 2458459 java.lang.Integer
1.00% 47.148M 8.959M 38.189M 234848 java.util.ArrayList

% self self+desc self desc count class
22.28% 591.449M 4.003M 587.446M 24983 com.caucho.server.http.HttpResponseStream
19.84% 526.640M 8.768M 517.872M 24983 com.caucho.server.http.HttpRequest
14.83% 393.655M 5.341M 388.314M 50000 com.caucho.vfs.ReadStream
7.97% 211.495M 2.863M 208.632M 25015 com.caucho.vfs.WriteStream
6.96% 184.828M 184.828M 0.000M 6056448 com.caucho.util.CharSegment
5.10% 135.463M 1.551M 133.912M 16942 com.caucho.db.block.Block
3.32% 88.199M 8.386M 79.813M 274788 com.caucho.util.CharBuffer
2.64% 69.979M 15.512M 54.466M 92419 com.caucho.server.dispatch.Invocation
1.89% 50.078M 5.491M 44.587M 79965 com.caucho.util.HashMapImpl
1.76% 46.749M 46.749M 0.000M 23658 com.caucho.util.CharSegment[]
1.65% 43.749M 1.717M 42.031M 25008 com.caucho.vfs.ByteToChar
1.46% 38.697M 4.384M 34.313M 24983 com.caucho.server.http.HttpResponse
1.24% 33.017M 3.820M 29.196M 25036 com.caucho.vfs.JniSocketImpl
1.20% 31.791M 9.539M 22.252M 50012 com.caucho.util.QDate
0.76% 20.168M 9.431M 10.737M 95083 com.caucho.server.cache.ProxyCacheFilterChain
0.60% 15.858M 0.027M 15.832M 218 com.caucho.util.LruCache
0.43% 11.502M 11.502M 0.000M 218 com.caucho.util.LruCache$CacheItem[]


% self self+desc self desc count class
16.80% 1336.132M 1336.132M 0.000M 5078528 char[]
13.42% 1067.050M 1067.050M 0.000M 415509 byte[]
7.44% 591.449M 4.003M 587.446M 24983 com.caucho.server.http.HttpResponseStream
6.67% 530.288M 170.706M 359.582M 4474964 java.lang.String
6.62% 526.640M 8.768M 517.872M 24983 com.caucho.server.http.HttpRequest
4.95% 393.655M 5.341M 388.314M 50000 com.caucho.vfs.ReadStream
2.66% 211.495M 2.863M 208.632M 25015 com.caucho.vfs.WriteStream
2.32% 184.828M 184.828M 0.000M 6056448 com.caucho.util.CharSegment
2.28% 181.387M 7.984M 173.404M 130802 java.util.HashMap
2.04% 162.469M 40.769M 121.700M 141506 java.util.HashMap$Entry[]
1.82% 144.698M 22.979M 121.719M 70044 com.jnj.mocospace.User
1.80% 142.862M 88.832M 54.031M 1940555 java.util.HashMap$Entry
1.70% 135.463M
Reply With Quote
  #4  
Old 12-20-2011, 05:57 PM
theBlueSage theBlueSage is offline
Member
 
Join Date: Apr 2011
Posts: 33
Default

I had to chop a lot of it off so that I could post it here ... let me know if you need the rest ... and how I would post it if you do
Reply With Quote
  #5  
Old 12-21-2011, 04:45 PM
ferg ferg is offline
Administrator
 
Join Date: Aug 2009
Posts: 190
Default

Can you get a heap dump and trace down the HttpRequest object?

There really should not be anywhere near that many, unless you have a massive number of keepalive and/or comet connections. In other words there should be about as many HttpRequest as TCP established sockets (with about 100-200 extra for pooling, but not 1000s).

Is there any chance that your application is holding onto the HttpRequest?
Reply With Quote
  #6  
Old 12-21-2011, 05:00 PM
theBlueSage theBlueSage is offline
Member
 
Join Date: Apr 2011
Posts: 33
Default

we use a lot of comet and keepalive as we do a lot of AJAX and 'in game' chat, and the application may well be holding on HttpRequest, however the same said application is running on other identical servers that are running resin-pro-4.0.6 and we are not running into this issue ...
Reply With Quote
  #7  
Old 12-21-2011, 07:05 PM
ferg ferg is offline
Administrator
 
Join Date: Aug 2009
Posts: 190
Default

Ok. Do you know what the netstat looks like? It should be in the /resin-admin graphs and the pdf-report, since Resin records it.
Reply With Quote
  #8  
Old 12-21-2011, 08:47 PM
theBlueSage theBlueSage is offline
Member
 
Join Date: Apr 2011
Posts: 33
Default

I dont have the PDF now, but I have included a snapshot of a weeks worth of netstat in a graph. I have included web17 in the graph as it is running resin-pro-4.0.6. web15 and web16 are running resin-pro-4.0.24. They are identical physical machines.
Reply With Quote
  #9  
Old 12-22-2011, 04:28 PM
theBlueSage theBlueSage is offline
Member
 
Join Date: Apr 2011
Posts: 33
Default

I have taken a JVM Heapdump (java.hprof) from the web15 server, and a PDF report from web15 so you can analyze them. Looking at the memory graph from web15 it is already showing signs of repeating the behaviour in the top graph (original post). AS it may well contain information about my company and code in the heap-dump, I'd like to post it to you guys privately, but I dont know how to do that. Any suggestions?
Reply With Quote
  #10  
Old 12-22-2011, 05:29 PM
ferg ferg is offline
Administrator
 
Join Date: Aug 2009
Posts: 190
Default

You can mail it to me (ferg@caucho.com) or to the support email address with your license (which would be better.)
Reply With Quote
Reply

Tags
memory leak, resin-prop-4.0.24, wild gc

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 09:48 PM.


Powered by vBulletin® Version 3.8.6
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.