This version (2017/05/27 13:44) is a draft.
Approvals: 0/1

[05:41:40] *** ChanServ sets mode: +o purplefox

[12:59:33] <D-Spair> I think that I am going to try expanding the TcpEventBusBridge code to include 2-way capabilities (both a client and server)… The question I have is “How to I ensure single consumer for send() across bridged nodes?”

[13:00:34] <purplefox> temporal_, hey julien, just looking at the latest techempower results

[13:01:46] <purplefox> temporal_, looks like vert.x performance decreased a lot since the last benchmark

[15:11:02] <temporal_> purplefox are you referring to the “plain” benchmark ?

[15:11:25] <temporal_> this one seems significantly slower

[15:11:26] <temporal_> the json seems better

[15:11:35] <temporal_> and the database ones seems better

[15:11:35] <purplefox> yes plaintext

[15:11:51] <purplefox> what's weird is the vertx-web benchmark seems about same performance as the plaintext

[15:12:05] <purplefox> i would expect plaintext to be much faster than vertx-web

[15:12:07] <temporal_> first it was hard to get the numbers

[15:12:12] <temporal_> remember it was disabled

[15:12:16] <purplefox> as vertx-web webserver has quite a lot of overhead

[15:12:20] <temporal_> agreed

[15:12:26] <purplefox> i don't believe these numbers

[15:12:33] <temporal_> the database seems better

[15:12:46] <temporal_> also one of our perf box died

[15:12:56] <temporal_> so we are unnable to run reliable benchmarks internally

[15:13:51] <temporal_> also the vertx web HTTP/1.1 server code has not much evolved between 3.2 and 3.3

[15:14:11] <temporal_> so I'm sceptical

[15:14:25] <temporal_> I would definitely run benchmarks if I could

[15:14:29] <temporal_> I mean my own

[15:14:35] <purplefox> i am very sceptical. my gues is that they ran the vertx-web benchmark twice

[15:14:40] <purplefox> the figures are too similar

[15:15:20] <purplefox> if you look in the round 12 benchmarks vertx raw is WAY faster than vertx-web

[15:15:25] <temporal_> in the round 12 the vertx-web perfs are really low

[15:15:28] <purplefox> which is to be expected

[15:15:32] <temporal_> 2.5%

[15:15:37] <purplefox> yes, vertx-web has a lot of overhead

[15:15:43] <purplefox> compared to raw vert.x

[15:15:45] <temporal_> not that much imho

[15:15:58] <temporal_> it's a factor 10 in round 12

[15:16:24] <purplefox> vertx-web used to do checking if files.exist, file properties etc

[15:16:26] <temporal_> but agreed that the number in 13 of vertx and vertx-web are really close

[15:16:39] <purplefox> plaintext is just hardcoded response from memory

[15:16:46] <purplefox> i'd expect it to be 10 times faster

[15:17:14] <temporal_> if you look also

[15:17:26] <temporal_> in round 12 cpoll_cppsp was almost 100%

[15:17:42] <temporal_> and in round 13 it is 60% of the fastest

[15:18:09] <temporal_> and undertow is fastest than cpoll_cppsp

[15:18:15] <temporal_> and in round 12 it was slowest

[15:18:27] <purplefox> yeah it all looks fishy

[15:18:34] <temporal_> I don't believe that the code of both understow and/or cpoll_cppsp changed between the two rounds

[15:18:38] <purplefox> maybe they screwed up when collating the results

[15:18:40] <temporal_> both are fairly stable

[15:18:51] <temporal_> or maybe they screwed up before :-)

[15:19:14] <purplefox> also the absolute results are all way different for everyone, but i don't think they changed hardware. go figure!

[15:20:04] <purplefox> it's disappointing, i don't think i can trust these benchmarks any more. too many things just don't add up

[15:20:42] <purplefox> i wonder if they're making some beginners error, like not running it on a dedicated network

[15:21:01] <temporal_> benchmarks are good as long as they are favorable for you :-)

[15:21:06] <purplefox> lol, true

[15:21:25] <temporal_> I was also disapointed by the lack of transparency when I asked them what is going on

[15:21:35] <temporal_> what was

[15:21:40] <purplefox> tbh, techempower benchmarks have never represented real world workloads, so they're pure marketing

[15:21:57] <purplefox> which, as you say, is nice when they're on your side

[15:22:27] <temporal_> and also they focus on throughput and not latency

[15:22:32] <temporal_> they give it on the other tab

[15:23:07] <temporal_> but if you look at them, no one is both with the higher throughput and the loweest latency (expected)

[15:23:25] <temporal_> some server favor latency, some favor thoughput

[15:23:40] <temporal_> if you look at plaintext latency

[15:23:52] <temporal_> vertx only has 210ms and vertx-web has 134ms

[15:24:08] <temporal_> so based on these numbers you can deduce that you vertx-web is much better than vertx

[15:24:25] <temporal_> because you have almost same throughput but 1/2 of the response time latency!

[15:24:28] <purplefox> we should get rid of vertx-core and just user vertx-web :)

[15:25:27] <purplefox> temporal_, anyway, how are things going with you? :)

[15:25:33] <temporal_> so the “best” means the server than has the highest concurrency

[15:26:11] <temporal_> purplefox things are doing good

[15:26:25] <temporal_> following a bit what is going on mewbase side

[15:26:25] <temporal_> very interesting

[15:26:53] <purplefox> yeah i am excited about it, it's quite a challenge though

[15:27:07] <purplefox> but i've always wanted to write a database :)

[15:27:35] <temporal_> I will not follow you on this :-)

[15:27:49] <temporal_> do you have access to good hardware for testing ?

[15:28:05] <temporal_> I bet you have :)

[15:28:38] <purplefox> early days, no hardware yet. right now we're just using embedded lmdb for interim storage but we want to write our own

[15:29:00] <purplefox> things have progressed quite a long way past good old btrees in the storage world

[15:29:24] <purplefox> so there's quite a lot to learn to get up to speed

[15:29:25] <temporal_> I'm always facinated by good and smart data structures

[15:30:12] <purplefox> yeah, and these are about as complex data structures that you'll find anywhere i think

[15:30:26] <temporal_> plus you need to think about cache lines, fetches

[15:30:43] <temporal_> think about hardware

[15:31:08] <purplefox> having it work well with storage hardware is probably the hardest thing

[15:31:31] <temporal_> what kind of storage are you going to optimize for ?

[15:31:35] <temporal_> SSD ?

[15:31:37] <purplefox> trying to stick to sequential reads/writes, avoiding random io

[15:31:44] <temporal_> yes

[15:31:53] <temporal_> and flushing is also important

[15:31:58] <temporal_> flush at the right time

[15:32:07] <purplefox> also data integrity is tricky. i.e. ensuring no corruption in case of failure

[15:32:18] <purplefox> yes ssd mainly

[15:33:10] <purplefox> well flushing actually becomes less important in most replicated cloud setups as you generally have a replica so you can turn flushing off (just let the OS do it) and still have a reasonable durability guarantee

[15:33:29] <purplefox> that's what most ppl do with say kafka/cassandra etc

[15:33:41] <temporal_> ok that's good to know

[15:33:56] <temporal_> is there any book you would recommend on the topic ?

[15:34:10] <purplefox> very few of the newer systems actually flush synchronously by default

[15:34:30] <purplefox> i am learning mainly from the web and reading scientific papers

[16:23:48] <D-Spair> When running a clustered Vert.x deployment, how does Vert.x ensure that a message on the event bus from a “send()” is only processed by a single consumer? Does it use the ClusterManager for coordination/concurrency control?

[16:25:12] <D-Spair> For example. if I have 2 nodes in a cluster and each node has a consumer for address 'someaddress' and another verticle “send()“s a message to that address multiple times, I would like either node to be able to answer the request, but not have it answered by noth nodes.

[16:25:18] <D-Spair> s/noth/both/

[17:22:05] <myghty> D-Spair: I think it is removed from the message queue by the first one reading it

[17:22:09] <myghty> at least it looks like it

[17:23:20] <D-Spair> And I assume that the message queue is a distributed data structure managed by the ClusterManager?

[17:26:17] <tsegismont> D-Spair, it selects a registration (stored in the __vertx.subs multimap) and then sends the message to corresponding node