Differences

This shows you the differences between two versions of the page.

Link to this comparison view

irc:1441144800 [2017/05/27 13:44] (current)
Line 1: Line 1:
 +[03:54:25] *** ChanServ sets mode: +o temporalfox
 +
 +[08:43:21] *** ChanServ sets mode: +o purplefox
 +
 +[09:21:12] *** ChanServ sets mode: +o purplefox
 +
 +[10:13:30] <​purplefox>​ cescoffier: temporalfox:​ morning
 +
 +[10:13:37] <​cescoffier>​ good morning
 +
 +[10:13:38] <​temporalfox>​ hi purplefox
 +
 +[10:13:42] <​temporalfox>​ hi everyone :-)
 +
 +[10:48:30] <​s0va>​ purplefox: i'm debugging strange vertx-hazelcast related memory leak
 +
 +[10:49:42] <​s0va>​ i've wrote an email on mail on this topic, but didn't receive any useful reply
 +
 +[10:50:38] <​s0va>​ here's a situation: the same code works okay on single-node vertx...
 +
 +[10:51:11] <​s0va>​ on clustered version it leaks memory, heap gets filled in production in circa two days.
 +
 +[10:51:19] <​s0va>​ http://​shrani.si/​f/​3m/​A3/​36TPQAHG/​vertx-memleak-heap-compa.jpg
 +
 +[10:56:25] <​s0va>​ here's a yourkit snapshot screenshot
 +
 +[10:56:53] <​s0va>​ here's the deal: when websocket client connects, 3-5 event bus consumers are registered
 +
 +[10:57:11] <​s0va>​ ... when client disconnects,​ consumers are unregistered
 +
 +[10:57:39] <​s0va>​ heap is filled with lots of hazelcast choosable sets instances
 +
 +[10:57:57] <​s0va>​ what am i doing wrong=
 +
 +[11:05:17] <​purplefox>​ s0va: do you have a link to the mail?
 +
 +[11:11:26] <​s0va>​ just a sec
 +
 +[11:12:13] <​s0va>​ https://​groups.google.com/​forum/?​fromgroups#​!searchin/​vertx/​memory$20leak/​vertx/​zzfleuzdCHo/​nnEl0q3PELoJ
 +
 +[11:12:25] <​s0va>​ that was for version 2.1.6
 +
 +[11:12:42] <​s0va>​ as my last resort i ported app to 3.0, but nothing changed
 +
 +[11:13:05] <​s0va>​ i can also provide heap dumps
 +
 +[11:13:09] <​s0va>​ or even code
 +
 +[11:56:27] <​purplefox>​ s0va: it should be fairly straightforward in your profiler to see which object(s) are holding on to the large number of objects in your report
 +
 +[11:57:09] <​purplefox>​ s0va: I'd recommend finding out what is holding on to io.vertx.core.impl.ConcurrentHashSet
 +
 +[11:57:20] <​purplefox>​ and tracing back through references
 +
 +[11:57:34] <​purplefox>​ s0va: that should be pretty easy with any modern profiler
 +
 +[11:58:09] <​purplefox>​ s0va: looks like you're using yourkit.. this is very easy
 +
 +[12:02:02] <​Sticky>​ would have thought it is this one: http://​grepcode.com/​file/​repo1.maven.org/​maven2/​io.vertx/​vertx-hazelcast/​2.1RC3/​org/​vertx/​java/​spi/​cluster/​impl/​hazelcast/​HazelcastAsyncMultiMap.java#​57
 +
 +[12:02:22] <​purplefox>​ is that a guess?
 +
 +[12:02:29] <​Sticky>​ totally
 +
 +[12:02:47] <​purplefox>​ I'd rather see the data from the profiler. That class is used in several places :)
 +
 +[12:03:02] <​purplefox>​ you may well be right, but I don't like guess work
 +
 +[12:03:38] <​Sticky>​ yeah its just based on there being a single instance of HazelcastAsyncMultiMap but a retained set size the same as the possible "​leak"​
 +
 +[12:11:21] <​purplefox>​ Sticky: if that's what's holding on to things then it's likely there are simply a lot of event bus handlers that have been registered by the user
 +
 +[12:15:45] <​Sticky>​ somewhere around 2million
 +
 +[12:18:39] <​purplefox>​ s0va: Sticky: that, in itself is not a huge number, but it depends whether it was desired or not. I don't see any evidence of a leak as such. I suggest s0va looks through your code for places where you register event bus handlers but don't remove them later.
 +
 +[12:21:26] <​purplefox>​ s0va: some basic techniques for tracking down this kind of thing - global search through your code for where you register handlers, and every time you do increment a global counter. when you unregister them decrement your global counter. then have a periodic task that displays the count.
 +
 +[12:25:03] <​purplefox>​ Sticky: having said that I don't see anything that looks like handlers in the report, i guess they could be on other nodes of the cluster
 +
 +[12:25:20] <​purplefox>​ s0va: how many nodes in the cluster here? do your other nodes register a lot of handlers?
 +
 +[12:28:41] <​purplefox>​ s0va: also, one thing to check - may sure you're using the correct version of hazelcast for the vert.x release, using a different version might break the cluster - i know previous versions had bugs in the multimap implementation
 +
 +[12:31:03] <​Sticky>​ oh, hmmm, this is a websocket app, another thing to look at, does what is on the end of that websocket register lots of listners?
 +
 +[12:36:41] <​purplefox>​ Sticky: shouldn'​t do. at least, not _clustered_ handlers
 +
 +[13:02:38] <​s0va>​ purplefox: 10x for hint, will check out
 +
 +[13:02:39] <​s0va>​ purplefox: only 3 nodes
 +
 +[14:02:54] <​s0va>​ purplefox: is this normal: http://​pastebin.com/​TfGhQ69Y
 +
 +[15:49:49] <​aetherius>​ hi. using the HttpClient from vert.x i'm trying to connect to an https host. using ssl set to true and trusting all certificates (generally unsafe, but it's just a toy example) i still get following exception: javax.net.ssl.SSLHandshakeException:​ Failed to create SSL connection. does anyone know how to fix this "​problem"?​ do i have to install specific certificates?​
 +
 +[15:51:43] <​Sticky>​ well debugging ssl is a bitch
 +
 +[15:52:03] <​Sticky>​ that error dosnt give us much to go on
 +
 +[15:52:08] <​aetherius>​ yep :(
 +
 +[15:52:27] <​Sticky>​ try turning on -Djavax.net.debug=all
 +
 +[15:52:50] <​Sticky>​ may help a bit, but yeah, javas ssl warnings suck
 +
 +[15:53:27] <​aetherius>​ hm ok. i found some general information overriding the internal trustmanagers and hence accept all certificates,​ but with no luck/​success
 +
 +[15:55:18] <​Sticky>​ testing with the commandline thing of "​openssl s_client -connect 127.0.0.1:​443"​ can sometimes give more info too
 +
 +[15:56:49] <​aetherius>​ ok. will try that as well .. (the first hint didn't gave much insight ... yet)
 +
 +[16:22:34] <​aetherius>​ hmm is it possible to create exception with a higher level of verbosity?
 +
 +[16:22:42] <​aetherius>​ *exceptions
 +
 +[16:29:12] <​aetherius>​ solution is / was simple, and i didn't thought of that: the port parameter, the first in client.getNow() is important and needs to be set as 443 (https default port) .. well ..
 +
 +[16:29:21] <​aetherius>​ thanks for the help and hints!
 +
 +[16:29:44] <​s0va>​ purplefox: i'm looking at memory snapshots in yourkit
 +
 +[16:30:07] <​s0va>​ purplefox: everything traces back to vertx-hazelcast.
 +
 +[16:30:35] <​s0va>​ ... if i deploy all verticles in the same vertx instance and disable clustering, memory leak does not occur. which is really weird
 +
 +[16:30:49] <​s0va>​ used vertx: 3.0, used hazelcast: 3.5
 +
 +[16:36:11] <​aetherius>​ good bye
 +
 +[16:45:39] <​s0va>​ purplefox: here are two memory snapshots (first one taken shortly after vertx app startup, another after some time)
 +
 +[16:45:42] <​s0va>​ purplefox: https://​www.wetransfer.com/​downloads/​14962e3cd4f7f17529384105862f49eb20150902144322/​4f00d6be4ec7f4b9cf02ebbac709199720150902144322/​24d959
 +
 +[17:27:49] <​melvinross>​ so i've been using vertx without grade or maven for doing test. I'm trying to figure out how to include modules without using either. Do i just need to build the projects as fat jars and put them on the class path?
 +
 +[17:41:28] <​s0va>​ melvinross: yep
 +
 +[17:41:43] <​s0va>​ or put all jars to classpath
 +
 +[17:43:58] <​melvinross>​ thanks
 +
 +[17:56:00] <​melvinross>​ i ended up just putting the project into maven
 +
 +[17:56:42] <​Sticky>​ yeah working without some build or dependency management tool is just a nightmare
 +
 +[18:11:13] <​melvinross>​ it wasn't by choice :-(
 +
 +[18:13:18] <​melvinross>​ i'm fighting a bit of a FUD war
 +
 +[18:36:20] <​melvinross>​ has anyone tried to use the kafka verticle from cyanogen?
 +
 +[19:59:10] <​melvinross>​ i'm attempting to deploy a vertices from JS
 +
 +[19:59:43] <​melvinross>​ vertx.deployVerticle("​com.cyngn.kafka.MessageProducer",​kafkaConfig,​ kafkaComplete);​
 +
 +[20:00:06] <​melvinross>​ but the deployed vertical gets no information when it calls config()
 +
 +[20:00:48] <​melvinross>​ just an empty object, so it looks like the data isn't actually being passed
 +
 +[20:27:52] <​BadApe>​ melvinross, sorry i am using java only, but it is very quiet in here
 +
 +[20:50:04] <​BadApe>​ TSharedRef addr = ISocketSubsystem::​Get(PLATFORM_SOCKETSUBSYSTEM)->​CreateInternetAddr();​ has TSharedRef been deprecated
 +
 +[20:58:53] <​BadApe>​ oh maybe it is ISocketSubsystem that no longer has SetIp
 +
 +[21:03:24] <​BadApe>​ oops wrong place
 +
 +[22:52:30] <​bytor99999>​ melvinross will probably miss this. But in Vert.x 3 you have to put in kafkaConfig,​ a property in the root called "​config"​ then you can put your values under that property.