Every few days one of my gemfire servers get shunned and needs to be restarted. The gemfire log only shows this message and not much else. From the logs and thread dumps, my application looks perfectly fine. Can I disable GMS.shun? It has been removed from the jgroups project. Or can it indicate some other bug or configuration problem in my application?
[severe 2013/05/23 09:58:39.981 EST GemFire_ServerSide_Production <CloserThread> tid=0xe238b] Membership service failure: Channel closed: com.gemstone.gemfire.ForcedDisconnectException: This member has been forced out of the distributed system. Please consult GemFire logs to find the reason. (GMS shun)
This is what's in the locator log:
[info 2013/05/23 09:58:10.644 EST <VERIFY_SUSPECT.TimerThread> tid=0x34a2] No suspect verification response received from gemoc011(30843)<v293>:21251 in 5001 milliseconds: declaring it dead
[info 2013/05/23 09:58:10.798 EST <UDP ucast receiver> tid=0x1b] Membership: received new view [gemoc021(32727)<v1>:40020|308] [gemoc021(32727)<v1>:40020/2957, gemoc011(17072)<v3>:40061/39934, gemoc021(29430)<v30>:65465/31761, gemoc011(16558)<v31>:17742/41610, gemoc021(20340)<v292>:18368/59699] crashed mbrs: [gemoc011(30843)<v293>:21251/38294]
[info 2013/05/23 09:58:10.800 EST <View Message Processor> tid=0x39] Member at gemoc011(30843)<v293>:21251/38294 unexpectedly left the distributed cache: departed JGroups view