This version (2017/05/27 13:44) is a draft.
Approvals: 0/1

[03:54:25] * ChanServ sets mode: +o temporalfox [08:43:21] * ChanServ sets mode: +o purplefox

[09:21:12] *** ChanServ sets mode: +o purplefox

[10:13:30] <purplefox> cescoffier: temporalfox: morning

[10:13:37] <cescoffier> good morning

[10:13:38] <temporalfox> hi purplefox

[10:13:42] <temporalfox> hi everyone :-)

[10:48:30] <s0va> purplefox: i'm debugging strange vertx-hazelcast related memory leak

[10:49:42] <s0va> i've wrote an email on mail on this topic, but didn't receive any useful reply

[10:50:38] <s0va> here's a situation: the same code works okay on single-node vertx…

[10:51:11] <s0va> on clustered version it leaks memory, heap gets filled in production in circa two days.

[10:51:19] <s0va> http://shrani.si/f/3m/A3/36TPQAHG/vertx-memleak-heap-compa.jpg

[10:56:25] <s0va> here's a yourkit snapshot screenshot

[10:56:53] <s0va> here's the deal: when websocket client connects, 3-5 event bus consumers are registered

[10:57:11] <s0va> … when client disconnects, consumers are unregistered

[10:57:39] <s0va> heap is filled with lots of hazelcast choosable sets instances

[10:57:57] <s0va> what am i doing wrong=

[11:05:17] <purplefox> s0va: do you have a link to the mail?

[11:11:26] <s0va> just a sec

[11:12:13] <s0va> https://groups.google.com/forum/?fromgroups#!searchin/vertx/memory$20leak/vertx/zzfleuzdCHo/nnEl0q3PELoJ

[11:12:25] <s0va> that was for version 2.1.6

[11:12:42] <s0va> as my last resort i ported app to 3.0, but nothing changed

[11:13:05] <s0va> i can also provide heap dumps

[11:13:09] <s0va> or even code

[11:56:27] <purplefox> s0va: it should be fairly straightforward in your profiler to see which object(s) are holding on to the large number of objects in your report

[11:57:09] <purplefox> s0va: I'd recommend finding out what is holding on to io.vertx.core.impl.ConcurrentHashSet

[11:57:20] <purplefox> and tracing back through references

[11:57:34] <purplefox> s0va: that should be pretty easy with any modern profiler

[11:58:09] <purplefox> s0va: looks like you're using yourkit.. this is very easy

[12:02:02] <Sticky> would have thought it is this one: http://grepcode.com/file/repo1.maven.org/maven2/io.vertx/vertx-hazelcast/2.1RC3/org/vertx/java/spi/cluster/impl/hazelcast/HazelcastAsyncMultiMap.java#57

[12:02:22] <purplefox> is that a guess?

[12:02:29] <Sticky> totally

[12:02:47] <purplefox> I'd rather see the data from the profiler. That class is used in several places :)

[12:03:02] <purplefox> you may well be right, but I don't like guess work

[12:03:38] <Sticky> yeah its just based on there being a single instance of HazelcastAsyncMultiMap but a retained set size the same as the possible “leak”

[12:11:21] <purplefox> Sticky: if that's what's holding on to things then it's likely there are simply a lot of event bus handlers that have been registered by the user

[12:15:45] <Sticky> somewhere around 2million

[12:18:39] <purplefox> s0va: Sticky: that, in itself is not a huge number, but it depends whether it was desired or not. I don't see any evidence of a leak as such. I suggest s0va looks through your code for places where you register event bus handlers but don't remove them later.

[12:21:26] <purplefox> s0va: some basic techniques for tracking down this kind of thing - global search through your code for where you register handlers, and every time you do increment a global counter. when you unregister them decrement your global counter. then have a periodic task that displays the count.

[12:25:03] <purplefox> Sticky: having said that I don't see anything that looks like handlers in the report, i guess they could be on other nodes of the cluster

[12:25:20] <purplefox> s0va: how many nodes in the cluster here? do your other nodes register a lot of handlers?

[12:28:41] <purplefox> s0va: also, one thing to check - may sure you're using the correct version of hazelcast for the vert.x release, using a different version might break the cluster - i know previous versions had bugs in the multimap implementation

[12:31:03] <Sticky> oh, hmmm, this is a websocket app, another thing to look at, does what is on the end of that websocket register lots of listners?

[12:36:41] <purplefox> Sticky: shouldn't do. at least, not _clustered_ handlers

[13:02:38] <s0va> purplefox: 10x for hint, will check out

[13:02:39] <s0va> purplefox: only 3 nodes

[14:02:54] <s0va> purplefox: is this normal: http://pastebin.com/TfGhQ69Y

[15:49:49] <aetherius> hi. using the HttpClient from vert.x i'm trying to connect to an https host. using ssl set to true and trusting all certificates (generally unsafe, but it's just a toy example) i still get following exception: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection. does anyone know how to fix this “problem”? do i have to install specific certificates?

[15:51:43] <Sticky> well debugging ssl is a bitch

[15:52:03] <Sticky> that error dosnt give us much to go on

[15:52:08] <aetherius> yep :(

[15:52:27] <Sticky> try turning on -Djavax.net.debug=all

[15:52:50] <Sticky> may help a bit, but yeah, javas ssl warnings suck

[15:53:27] <aetherius> hm ok. i found some general information overriding the internal trustmanagers and hence accept all certificates, but with no luck/success

[15:55:18] <Sticky> testing with the commandline thing of “openssl s_client -connect 127.0.0.1:443” can sometimes give more info too

[15:56:49] <aetherius> ok. will try that as well .. (the first hint didn't gave much insight … yet)

[16:22:34] <aetherius> hmm is it possible to create exception with a higher level of verbosity?

[16:22:42] <aetherius> *exceptions

[16:29:12] <aetherius> solution is / was simple, and i didn't thought of that: the port parameter, the first in client.getNow() is important and needs to be set as 443 (https default port) .. well ..

[16:29:21] <aetherius> thanks for the help and hints!

[16:29:44] <s0va> purplefox: i'm looking at memory snapshots in yourkit

[16:30:07] <s0va> purplefox: everything traces back to vertx-hazelcast.

[16:30:35] <s0va> … if i deploy all verticles in the same vertx instance and disable clustering, memory leak does not occur. which is really weird

[16:30:49] <s0va> used vertx: 3.0, used hazelcast: 3.5

[16:36:11] <aetherius> good bye

[16:45:39] <s0va> purplefox: here are two memory snapshots (first one taken shortly after vertx app startup, another after some time)

[16:45:42] <s0va> purplefox: https://www.wetransfer.com/downloads/14962e3cd4f7f17529384105862f49eb20150902144322/4f00d6be4ec7f4b9cf02ebbac709199720150902144322/24d959

[17:27:49] <melvinross> so i've been using vertx without grade or maven for doing test. I'm trying to figure out how to include modules without using either. Do i just need to build the projects as fat jars and put them on the class path?

[17:41:28] <s0va> melvinross: yep

[17:41:43] <s0va> or put all jars to classpath

[17:43:58] <melvinross> thanks

[17:56:00] <melvinross> i ended up just putting the project into maven

[17:56:42] <Sticky> yeah working without some build or dependency management tool is just a nightmare

[18:11:13] <melvinross> it wasn't by choice :-(

[18:13:18] <melvinross> i'm fighting a bit of a FUD war

[18:36:20] <melvinross> has anyone tried to use the kafka verticle from cyanogen?

[19:59:10] <melvinross> i'm attempting to deploy a vertices from JS

[19:59:43] <melvinross> vertx.deployVerticle(“com.cyngn.kafka.MessageProducer”,kafkaConfig, kafkaComplete);

[20:00:06] <melvinross> but the deployed vertical gets no information when it calls config()

[20:00:48] <melvinross> just an empty object, so it looks like the data isn't actually being passed

[20:27:52] <BadApe> melvinross, sorry i am using java only, but it is very quiet in here

[20:50:04] <BadApe> TSharedRef addr = ISocketSubsystem::Get(PLATFORM_SOCKETSUBSYSTEM)→CreateInternetAddr(); has TSharedRef been deprecated

[20:58:53] <BadApe> oh maybe it is ISocketSubsystem that no longer has SetIp

[21:03:24] <BadApe> oops wrong place

[22:52:30] <bytor99999> melvinross will probably miss this. But in Vert.x 3 you have to put in kafkaConfig, a property in the root called “config” then you can put your values under that property.