[09:12:46] * ChanServ sets mode: +o temporalfox [12:21:26] * ChanServ sets mode: +o temporalfox

[17:10:25] * ChanServ sets mode: +o temporalfox [22:52:01] * ChanServ sets mode: +o temporalfox

[23:28:14] <temporalfox> hi AlexLehm

[23:41:06] <AlexLehm> Hello temporalfox

[23:41:21] <temporalfox> I've made progress on the netty 4.1.3.Final OOM isue

[23:41:29] <temporalfox> now I know more or less what it is

[23:41:41] <temporalfox> basically it's not a leak

[23:41:59] <temporalfox> The error happening is not a raw out of memory but rather an indication than the VM spend more CPU in the GC that in the application (98%).

[23:42:13] <temporalfox> This happens when weak references are used

[23:42:38] <temporalfox> Netty has a Recycler class that recycles pooled objects

[23:42:58] <temporalfox> normally a thread recycles an object directly into its pool

[23:43:41] <temporalfox> when an object is recycled from another thread, this object is stored in a weakhashmap<Stack, WeakOrderQueue>

[23:44:11] <temporalfox> and later these objects are pulled when the Recycler needs objects (from the thread the recycler belongs to)

[23:44:37] <temporalfox> it turns that the weakhashmap contains references to eventloop thread (VertxThread)

[23:44:54] <temporalfox> and recently a commit changed the behavior of the recycler

[23:45:06] <temporalfox> so more objects are in weakhashmap

[23:45:11] <temporalfox> and thus more threads

[23:45:30] <temporalfox> and it turns that during tests it keeps a lof of thread from being garbaged

[23:45:35] <temporalfox> specially in slow machines

[23:45:46] <temporalfox> where allocation is faster than GC

[23:45:55] <temporalfox> hence the specific GC issue

[23:46:41] <temporalfox> this commit is the change https://github.com/netty/netty/commit/afafadd3d7caf1e4b346da049baab0afeae0a4bc

[23:47:05] <temporalfox> I'm trying to figure out what is the correct thing to do

[23:47:19] <temporalfox> it may simply be a more appropriate GC setting for tetss

[23:47:25] <temporalfox> or change the recycler parameters

[23:47:29] <temporalfox> I don't know yet

[23:48:19] <temporalfox> in the Recycler class there is

[23:48:26] <AlexLehm> is that an issue that mostly happens in tests since there more objects are created and recyled?

[23:48:32] <temporalfox> yes

[23:48:33] <AlexLehm> or would that happen in real applications as well?

[23:48:41] <temporalfox> no I don't think so

[23:48:43] <temporalfox> DELAYED_RECYCLED

[23:48:43] <AlexLehm> ok

[23:48:53] <temporalfox> this is the thread local weak hashmap

[23:50:12] <temporalfox> it's mainly because we create and destroy many Vertx instances

[23:50:14] <temporalfox> in tests

[23:51:13] <temporalfox> and it happens because of the ThreadDeathWatcher

[23:51:32] <temporalfox> that schedule tasks when some thread dies

[23:52:01] <temporalfox> with PoolThreadCache

[23:52:25] <temporalfox> so the ThreadDeathWatcher fast thread local contains a weakhashmap that grows

[23:52:35] <temporalfox> and weak retains the vertx threads

[23:52:56] <temporalfox> with the new change this map reach around 2000 elements

[23:53:02] <temporalfox> and without it remains quite low