DWR

ConcurrentModificationException occurs under load in DefaultScriptSessionManager.checkTimeouts()

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Normal Normal
  • Resolution: Fixed
  • Affects Version/s: 2.0.7
  • Fix Version/s: 2.0.9
  • Component/s: Core
  • Documentation Required:
    No
  • Description:
    Hide
    While the changes in 2.0.7 (see DWR-60) helped address the deadlocks we were seeing in 2.0.6 we are now experiencing CME when our server load reaches over 1200 users.

    Here is the stacktrace we are seeing:

    java.util.ConcurrentModificationException
    at java.util.HashMap$AbstractMapIterator.checkConcurrentMod(HashMap.java:122)
    at java.util.HashMap$AbstractMapIterator.makeNext(HashMap.java:127)
    at java.util.HashMap$ValueIterator.next(HashMap.java:212)
    at org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:129)
    at org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109)
    at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:196)
    Show
    While the changes in 2.0.7 (see DWR-60) helped address the deadlocks we were seeing in 2.0.6 we are now experiencing CME when our server load reaches over 1200 users. Here is the stacktrace we are seeing: java.util.ConcurrentModificationException at java.util.HashMap$AbstractMapIterator.checkConcurrentMod(HashMap.java:122) at java.util.HashMap$AbstractMapIterator.makeNext(HashMap.java:127) at java.util.HashMap$ValueIterator.next(HashMap.java:212) at org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:129) at org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109) at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:196)

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Rick Oosterholt added a comment - 21/Dec/11 6:05 AM

I've also experience the CME.
Here's my stack:

ExceptionHandler - Unhandled Exception <java.util.ConcurrentModificationException>java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
   at java.util.HashMap$ValueIterator.next(Unknown Source)
   at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:179)
   at org.directwebremoting.impl.DefaultScriptSessionManager.maybeCheckTimeouts(DefaultScriptSessionManager.java:163)
   at org.directwebremoting.impl.DefaultScriptSessionManager.getScriptSession(DefaultScriptSessionManager.java:50)
   at org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:83)
   at org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305)
   at org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53)
   at org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)
   at org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146)
Show
Rick Oosterholt added a comment - 21/Dec/11 6:05 AM I've also experience the CME. Here's my stack: 
ExceptionHandler - Unhandled Exception <java.util.ConcurrentModificationException>java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
   at java.util.HashMap$ValueIterator.next(Unknown Source)
   at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:179)
   at org.directwebremoting.impl.DefaultScriptSessionManager.maybeCheckTimeouts(DefaultScriptSessionManager.java:163)
   at org.directwebremoting.impl.DefaultScriptSessionManager.getScriptSession(DefaultScriptSessionManager.java:50)
   at org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:83)
   at org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305)
   at org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53)
   at org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)
   at org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146)
Hide
Permalink
Rick Oosterholt added a comment - 21/Dec/11 6:33 AM

I've looked into 2.0.8 (svn) and I think this is fixed there.
Maybe someone can verify that?

Show
Rick Oosterholt added a comment - 21/Dec/11 6:33 AM I've looked into 2.0.8 (svn) and I think this is fixed there. Maybe someone can verify that?
Hide
Permalink
David Marginian added a comment - 21/Dec/11 6:50 AM

No, it does not appear to be fixed in 2.0.8.

Show
David Marginian added a comment - 21/Dec/11 6:50 AM No, it does not appear to be fixed in 2.0.8.
Hide
Permalink
Rick Oosterholt added a comment - 21/Dec/11 7:14 AM

ok, sorry. I see that now.
The problem is when using an iterator of a synchonized map; when using an iterator, the map (even when it is synchronized) should be synchronized while iterating...

Show
Rick Oosterholt added a comment - 21/Dec/11 7:14 AM ok, sorry. I see that now. The problem is when using an iterator of a synchonized map; when using an iterator, the map (even when it is synchronized) should be synchronized while iterating...
Hide
Permalink
David Marginian added a comment - 21/Dec/11 7:20 AM

Yes, I understand the problem. There are also two places where it can occur. We can either synchronize the map OR we can make a copy of the map and iterate the copy. In this case it is probably better to synchronize due to the potential size of the map. Any thoughts?

Show
David Marginian added a comment - 21/Dec/11 7:20 AM Yes, I understand the problem. There are also two places where it can occur. We can either synchronize the map OR we can make a copy of the map and iterate the copy. In this case it is probably better to synchronize due to the potential size of the map. Any thoughts?
Hide
Permalink
Rick Oosterholt added a comment - 21/Dec/11 7:29 AM

copying doesn't sound good to me.
Synchronizing the map is common practice to avoid ConcurrentModificationExceptions while iterating; this would be my preferable solution.

Indeed, there are two places: one in invalidate(..) and one in checkTimeouts() for the 'pageSessionMap' map.

Show
Rick Oosterholt added a comment - 21/Dec/11 7:29 AM copying doesn't sound good to me. Synchronizing the map is common practice to avoid ConcurrentModificationExceptions while iterating; this would be my preferable solution. Indeed, there are two places: one in invalidate(..) and one in checkTimeouts() for the 'pageSessionMap' map.
Hide
Permalink
David Marginian added a comment - 21/Dec/11 7:37 AM

They are both common practice, just depends on the situation. I need to investigate the code a bit to see what would be best in this situation.

Show
David Marginian added a comment - 21/Dec/11 7:37 AM They are both common practice, just depends on the situation. I need to investigate the code a bit to see what would be best in this situation.
Hide
Permalink
David Marginian added a comment - 21/Dec/11 6:30 PM

I have checked a fix in it will be available on our Bamboo CI server in a few minutes.

Show
David Marginian added a comment - 21/Dec/11 6:30 PM I have checked a fix in it will be available on our Bamboo CI server in a few minutes.
Hide
Permalink
Rob Moore added a comment - 22/Dec/11 8:58 AM

This may be old news by now but by the looks of the fix it appears the DWR requires the 1.5 JVM now. No issue here but I had the impression from DWR-60 that it wasn't yet a requirement.

Show
Rob Moore added a comment - 22/Dec/11 8:58 AM This may be old news by now but by the looks of the fix it appears the DWR requires the 1.5 JVM now. No issue here but I had the impression from DWR-60 that it wasn't yet a requirement.
Hide
Permalink
David Marginian added a comment - 22/Dec/11 9:03 AM

No Rob, the fix does not require 1.5. If you are looking at the code you are probably looking at the wrong branch (Trunk uses ConcurrentMap which does require 1.5). The fix was simply to synchronize while iterating over the map.

Show
David Marginian added a comment - 22/Dec/11 9:03 AM No Rob, the fix does not require 1.5. If you are looking at the code you are probably looking at the wrong branch (Trunk uses ConcurrentMap which does require 1.5). The fix was simply to synchronize while iterating over the map.
Hide
Permalink
Rob Moore added a comment - 22/Dec/11 9:19 AM

I was looking at trunk by mistake. Sorry for the noise.

Show
Rob Moore added a comment - 22/Dec/11 9:19 AM I was looking at trunk by mistake. Sorry for the noise.
Hide
Permalink
Rob Moore added a comment - 22/Mar/12 3:01 PM

It looks like we are seeing a similar error in our production environment. This time it looks like the deadlock is occurring in two places invalidate() and setPageForScriptSession() but it's on the same collection apparently so it's possibly triggered elsewhere. I can open another ticket if you'd prefer but thought this would be the place to begin.

DEADLOCK DETECTED:

Name: http-bio-8080-exec-2558

State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2278

Total blocked: 37 Total waited: 82

Stack trace:

org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:136)

org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109)

org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:200)

org.directwebremoting.impl.DefaultScriptSessionManager.getScriptSession(DefaultScriptSessionManager.java:50)

org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:83)

org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305)

org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53)

org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)

org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146)

javax.servlet.http.HttpServlet.service(HttpServlet.java:641)

javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:196)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)

org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)

org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)

org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:217)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:77)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)

org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)

org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:964)

org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)

org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)

  • locked org.apache.tomcat.util.net.SocketWrapper@52515251

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)

java.lang.Thread.run(Thread.java:736)

Name: http-bio-8080-exec-2563

State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2158

Total blocked: 29 Total waited: 79

Stack trace:

org.directwebremoting.impl.DefaultScriptSessionManager.setPageForScriptSession(DefaultScriptSessionManager.java:74)

org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:84)

org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305)

org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53)

org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)

org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146)

javax.servlet.http.HttpServlet.service(HttpServlet.java:641)

javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:196)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)

org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)

org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)

org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:217)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:77)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)

org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)

org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:964)

org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)

org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)

  • locked org.apache.tomcat.util.net.SocketWrapper@45474547

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)

java.lang.Thread.run(Thread.java:736)

Name: http-bio-8080-exec-2562

State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2299

Total blocked: 38 Total waited: 99

Stack trace:

org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:136)

org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109)

org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:200)

org.directwebremoting.impl.DefaultScriptSessionManager.getScriptSession(DefaultScriptSessionManager.java:50)

org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:83)

org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305)

org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53)

org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101)

org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146)

javax.servlet.http.HttpServlet.service(HttpServlet.java:641)

javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:196)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)

org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)

org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95)

org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)

org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:217)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:77)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)

org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)

org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:964)

org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)

org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)

  • locked org.apache.tomcat.util.net.SocketWrapper@1f131f13

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)

java.lang.Thread.run(Thread.java:736)

Show
Rob Moore added a comment - 22/Mar/12 3:01 PM It looks like we are seeing a similar error in our production environment. This time it looks like the deadlock is occurring in two places invalidate() and setPageForScriptSession() but it's on the same collection apparently so it's possibly triggered elsewhere. I can open another ticket if you'd prefer but thought this would be the place to begin. DEADLOCK DETECTED: Name: http-bio-8080-exec-2558 State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2278 Total blocked: 37 Total waited: 82 Stack trace: org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:136) org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109) org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:200) org.directwebremoting.impl.DefaultScriptSessionManager.getScriptSession(DefaultScriptSessionManager.java:50) org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:83) org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305) org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53) org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101) org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146) javax.servlet.http.HttpServlet.service(HttpServlet.java:641) javax.servlet.http.HttpServlet.service(HttpServlet.java:722) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:196) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265) org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107) org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149) org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:217) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:77) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:964) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
  • locked org.apache.tomcat.util.net.SocketWrapper@52515251
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) java.lang.Thread.run(Thread.java:736) Name: http-bio-8080-exec-2563 State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2158 Total blocked: 29 Total waited: 79 Stack trace: org.directwebremoting.impl.DefaultScriptSessionManager.setPageForScriptSession(DefaultScriptSessionManager.java:74) org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:84) org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305) org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53) org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101) org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146) javax.servlet.http.HttpServlet.service(HttpServlet.java:641) javax.servlet.http.HttpServlet.service(HttpServlet.java:722) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:196) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265) org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107) org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149) org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:217) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:77) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:964) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
  • locked org.apache.tomcat.util.net.SocketWrapper@45474547
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) java.lang.Thread.run(Thread.java:736) Name: http-bio-8080-exec-2562 State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2299 Total blocked: 38 Total waited: 99 Stack trace: org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:136) org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109) org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:200) org.directwebremoting.impl.DefaultScriptSessionManager.getScriptSession(DefaultScriptSessionManager.java:50) org.directwebremoting.impl.DefaultWebContext.getScriptSession(DefaultWebContext.java:83) org.directwebremoting.dwrp.BaseCallMarshaller.marshallOutbound(BaseCallMarshaller.java:305) org.directwebremoting.servlet.PlainCallHandler.handle(PlainCallHandler.java:53) org.directwebremoting.servlet.UrlProcessor.handle(UrlProcessor.java:101) org.directwebremoting.servlet.DwrServlet.doPost(DwrServlet.java:146) javax.servlet.http.HttpServlet.service(HttpServlet.java:641) javax.servlet.http.HttpServlet.service(HttpServlet.java:722) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:196) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265) org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107) org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.concurrent.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:95) org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149) org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:217) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:77) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:964) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
  • locked org.apache.tomcat.util.net.SocketWrapper@1f131f13
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) java.lang.Thread.run(Thread.java:736)
Hide
Permalink
Rob Moore added a comment - 22/Mar/12 3:14 PM

I meant to note that it doesn't appear that we're seeing ConcurrentModificationExceptions now. So this probably warrants another ticket.

Show
Rob Moore added a comment - 22/Mar/12 3:14 PM I meant to note that it doesn't appear that we're seeing ConcurrentModificationExceptions now. So this probably warrants another ticket.
Hide
Permalink
Rob Moore added a comment - 22/Mar/12 3:40 PM

Please disregard my last comment. We are experiencing these exceptions as before.

Show
Rob Moore added a comment - 22/Mar/12 3:40 PM Please disregard my last comment. We are experiencing these exceptions as before.
Hide
Permalink
David Marginian added a comment - 22/Mar/12 3:59 PM

"I meant to note that it doesn't appear that we're seeing ConcurrentModificationExceptions now. So this probably warrants another ticket.

Please disregard my last comment. We are experiencing these exceptions as before."

So you say you are seeing both the deadlock and the ConcurrentModificationException now? I see the deadlock I don't see the ConcurrentMod. Same stack trace as original exception?

Show
David Marginian added a comment - 22/Mar/12 3:59 PM "I meant to note that it doesn't appear that we're seeing ConcurrentModificationExceptions now. So this probably warrants another ticket. Please disregard my last comment. We are experiencing these exceptions as before." So you say you are seeing both the deadlock and the ConcurrentModificationException now? I see the deadlock I don't see the ConcurrentMod. Same stack trace as original exception?
Hide
Permalink
Rob Moore added a comment - 22/Mar/12 4:03 PM

We're seeing the exceptions in the log but not in the deadlock dump.

Show
Rob Moore added a comment - 22/Mar/12 4:03 PM We're seeing the exceptions in the log but not in the deadlock dump.
Hide
Permalink
David Marginian added a comment - 22/Mar/12 4:05 PM

Stack trace please I am not sure what you are talking about.

Show
David Marginian added a comment - 22/Mar/12 4:05 PM Stack trace please I am not sure what you are talking about.
Hide
Permalink
David Marginian added a comment - 22/Mar/12 7:57 PM

A brief look at our code and I don't see a deadlock situation here. But I need to look into it more when I have some time. Is that the complete thread dump? The State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2278 is suspicious to me. Is Tomcat under extreme load during this? What is the vendor and version of the jdk you are running?

Show
David Marginian added a comment - 22/Mar/12 7:57 PM A brief look at our code and I don't see a deadlock situation here. But I need to look into it more when I have some time. Is that the complete thread dump? The State: BLOCKED on java.util.Collections$SynchronizedMap@2c722c72 owned by: http-bio-8080-exec-2278 is suspicious to me. Is Tomcat under extreme load during this? What is the vendor and version of the jdk you are running?
Hide
Permalink
Rob Moore added a comment - 23/Mar/12 10:56 AM

The logging message we're seeing doesn't give us a lot to go on since it doesn't provide the full trace:

2012-03-20 16:16:08,132 ERROR org.directwebremoting.dwrp.BaseCallMarshaller --MarshallException: batchId=17 message=java.util.ConcurrentModificationException

The JVM is IBM's 1.6 implementation. Tomcat is under a heavy load when this occurs.

Show
Rob Moore added a comment - 23/Mar/12 10:56 AM The logging message we're seeing doesn't give us a lot to go on since it doesn't provide the full trace: 2012-03-20 16:16:08,132 ERROR org.directwebremoting.dwrp.BaseCallMarshaller --MarshallException: batchId=17 message=java.util.ConcurrentModificationException The JVM is IBM's 1.6 implementation. Tomcat is under a heavy load when this occurs.
Hide
Permalink
David Marginian added a comment - 23/Mar/12 11:12 AM - edited

For the thread dump, are you manually capturing it? Is the "DEADLOCK DETECTED:" string part of the dump or something you added? From my initial analysis invalidate() and setPageForScriptSession() both lock on pageSessionMap but either is holding another lock at the time so we don't have a lock-ordering Deadlock. Have you tried capturing another thread dump a few minutes later? Do you see the same threads blocked?

Show
David Marginian added a comment - 23/Mar/12 11:12 AM - edited For the thread dump, are you manually capturing it? Is the "DEADLOCK DETECTED:" string part of the dump or something you added? From my initial analysis invalidate() and setPageForScriptSession() both lock on pageSessionMap but either is holding another lock at the time so we don't have a lock-ordering Deadlock. Have you tried capturing another thread dump a few minutes later? Do you see the same threads blocked?
Hide
Permalink
Rob Moore added a comment - 23/Mar/12 11:55 AM

We manually captured this dump – I believe it was from the JConsole. I'll have to try repeated dumps in future but I can tell you once we experience this error performance really suffers to the point the application is unusable.

Show
Rob Moore added a comment - 23/Mar/12 11:55 AM We manually captured this dump – I believe it was from the JConsole. I'll have to try repeated dumps in future but I can tell you once we experience this error performance really suffers to the point the application is unusable.
Hide
Permalink
David Marginian added a comment - 23/Mar/12 12:09 PM

Have you looked at our source and come to the same conclusion that I have? I followed all of the threads execution flow based on the provided thread dump and I don't see any of them holding a lock when they enter invalidate or setPageForScriptSession. This leaves me thinking that this is not an issue with DWR. Are you sure your server can handle the load you are pushing to it?

Show
David Marginian added a comment - 23/Mar/12 12:09 PM Have you looked at our source and come to the same conclusion that I have? I followed all of the threads execution flow based on the provided thread dump and I don't see any of them holding a lock when they enter invalidate or setPageForScriptSession. This leaves me thinking that this is not an issue with DWR. Are you sure your server can handle the load you are pushing to it?
Hide
Permalink
Marjan Djokic added a comment - 29/Oct/12 3:23 AM

Hi,
I see this issue is closed but I experience the same problem with version 2.0.9 and under high load (10-15 requests/sec).

Here is part of thread dump log:

"TP-Processor3166" daemon prio=10 tid=0x00002aaabc693000 nid=0x39ef waiting for monitor entry [0x00000000677f7000]      java.lang.Thread.State: BLOCKED (on object monitor)      at org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:136)      - waiting to lock <0x00000006b7a2add8> (a java.util.Collections$SynchronizedMap)      at org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109)      at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:200)

DWR is used to call methods which execution lasts not more than 30sec.

Can somebody confirm that this problem is solved in version 2.0.9?

Show
Marjan Djokic added a comment - 29/Oct/12 3:23 AM Hi, I see this issue is closed but I experience the same problem with version 2.0.9 and under high load (10-15 requests/sec). Here is part of thread dump log: "TP-Processor3166" daemon prio=10 tid=0x00002aaabc693000 nid=0x39ef waiting for monitor entry [0x00000000677f7000]      java.lang.Thread.State: BLOCKED (on object monitor)      at org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:136)      - waiting to lock <0x00000006b7a2add8> (a java.util.Collections$SynchronizedMap)      at org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109)      at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:200) DWR is used to call methods which execution lasts not more than 30sec. Can somebody confirm that this problem is solved in version 2.0.9?
Hide
Permalink
David Marginian added a comment - 29/Oct/12 5:22 AM

Marjan, to avoid confusion this issue is for a Concurrent Modification exception. You appear to be having a separate issue.

Show
David Marginian added a comment - 29/Oct/12 5:22 AM Marjan, to avoid confusion this issue is for a Concurrent Modification exception. You appear to be having a separate issue.

People

Dates

  • Created:
    02/Sep/11 9:00 AM
    Updated:
    29/Oct/12 5:22 AM
    Resolved:
    22/Dec/11 5:53 PM
Atlassian JIRA (v4.1.2#531) | Report a problem | Request a feature | Contact administrators
Powered by a free Atlassian JIRA open source license for DWR. Try JIRA - bug tracking software for your team.