This version (2017/05/27 13:44) is a draft.
Approvals: 0/1

[15:04:26] <purplefox> hi rajith

[15:31:14] <gemmellr> purplefox: I was looking through the changes Hiram had made…but I see you have made some small changes since I started :)

[15:31:40] <purplefox> gemmellr: hi robbie

[15:31:55] <purplefox> gemmellr: yeah, i've basically refactored it into interface and impl

[15:32:11] <purplefox> and hidden stuff we don't need (yet)

[15:32:21] <purplefox> (although it's still available in the impl)

[15:33:11] <gemmellr> purplefox: I had some threading related questions based on the earlier discussion in the mails

[15:34:01] <chirino> purplefox: lol I started hacking in some improvements too.

[15:34:10] <chirino> time to rebase :)

[15:34:58] <gemmellr> purplefox: obviously you want send and recv, and its all to be async….and I beleive you are looking for P2P support, and brokered/routered intermediary support

[15:35:08] <purplefox> what is recv for?

[15:35:24] <purplefox> blocking receive?

[15:35:41] <gemmellr> no, just the handler i mean

[15:35:56] <gemmellr> 'consuming' rather than 'recv'

[15:36:08] <purplefox> yep, so what we want initially is what is in the current api

[15:37:05] <purplefox> i basically hid all the stuff we don't want (yet). The Vert.x philosophy here is less is more. Don't crowd APIs with stuff unless the user has asked for it. if they ask we can expose more stuff (which should be easy as its in the impl)

[15:37:33] <gemmellr> so i was wondering what if any behaviour there was around e.g will there be producers so to speak that are only sending, consumers that are only receiving, or is it a mix..e.g the receving handler then doing sending…

[15:37:56] <gemmellr> yeah, simple is good especially here

[15:38:27] <purplefox> not sure i follow. do producers do other things than send?

[15:39:53] <gemmellr> i basically mean are the connections going to be carrying sending and receiving links, or just 1 or the other, and if so whats the threading going to look like…or eg will the receivers only be listening and taking messages from peers etc etc

[15:39:56] <purplefox> i think we can remove flush too (for now) as i can't think of a good reason why it should be exposed

[15:40:38] <purplefox> i'm not sure the user really cares about links at all

[15:40:45] <purplefox> just sending and receiving stuff

[15:41:00] <gemmellr> sure

[15:41:02] <purplefox> aren't links an implementation detail of the protocl?

[15:41:28] <gemmellr> yes…its just the threading im trying to get a handle of

[15:42:04] <purplefox> not sure i followed what was the question about threading

[15:43:45] <purplefox> i'm not even sure why we need session here

[15:43:47] <gemmellr> er…essentially. are the netclient events (e.g new data arriving) and 'everything else' (e.g new message being sent) processed by the same thread all the time?

[15:45:00] <purplefox> netclient new data for a connection will always be on the same thread

[15:45:38] <purplefox> sending obviously is whatever thread you are calling send() from

[15:46:16] <purplefox> what's the purpose of a session other than something that allows senders and receivers to be created?

[15:46:28] <gemmellr> ok, thats what i thought, hence the questions…the engine is not thread safe, so its going to need locked by each thread

[15:46:45] <purplefox> when you say “the engine” which object are you talking about?

[15:46:58] <chirino> purplefox: it's for flow control

[15:47:20] <gemmellr> first question..what Hiram said ;)

[15:47:21] <chirino> isolates different sender receivers.

[15:47:40] <gemmellr> there are also expiry related things that are session scoped

[15:47:42] <chirino> for example you might want a new session for each request/reply

[15:47:45] <purplefox> chirino: but we could have each receiver/sender have their session, no?

[15:47:54] <gemmellr> transfers ids are session scoped..etc etc

[15:47:55] <chirino> for the best latency

[15:48:07] <chirino> and 1 shared session of bulk async sending

[15:48:15] <gemmellr> purplefox: yes you can

[15:48:24] <purplefox> what would be the disadvantage of each sender/receiver having their own session?

[15:48:39] <chirino> overhead

[15:48:42] <gemmellr> more sessions, more state basically

[15:48:59] <purplefox> what state is maintained in the session?

[15:49:08] <purplefox> message id counter?

[15:49:15] <chirino> data buffers

[15:49:25] <chirino> session has a buffering window

[15:49:29] <gemmellr> id, tracking details for the flow control etc

[15:49:36] <purplefox> buffering for replaying?

[15:49:47] <chirino> if the app does not process fast enough the client will hold on to a window

[15:50:32] <purplefox> but if 10 receivers share a session then one slow receiver can hold up the other ones?

[15:50:40] <chirino> yep

[15:50:52] <purplefox> in which case we would have to have each receiver have their own session anyway

[15:50:56] <purplefox> so there would be no extra overhead

[15:51:06] <chirino> but now your memory my blow up.

[15:51:24] <chirino> with 1 session I can control the memory usage.

[15:51:33] <chirino> better anyways

[15:51:40] <chirino> it's more constant

[15:51:51] <chirino> really depends on how constrained you are.

[15:52:25] <gemmellr> and how you configure the session flow control

[15:52:47] <chirino> or is all that that at the link level?

[15:52:47] <gemmellr> going back a bit…

[15:52:52] <purplefox> thing is we don't know who is creating receivers in vert.x

[15:52:55] <chirino> I know links do it per message

[15:53:02] <purplefox> so we can't just one session for everthing

[15:53:08] <chirino> But I think session also have flow control.

[15:53:13] <purplefox> and they can all configrue flow control diffeerntly

[15:53:29] <purplefox> i think the onus is on the user here not to create too many receivers and set buffers appropriately

[15:53:31] <chirino> yeah protonSession.setIncomingCapacity

[15:53:51] <gemmellr> there is link flow control in terms of messages…and session flow control in terms of transfer frames, which proton gives you a byte 'capacity' for and calculates a window from, if you set a max frame size…if you dont, it uses the maxwindow

[15:54:36] <chirino> protonSession.setOutgoingWindow

[15:54:48] <purplefox> if flow control is done per session, then why is the flow() method on the receiver not the session?

[15:54:48] <gemmellr> incoming is the important one

[15:54:55] <gemmellr> outgoing is essentially ignored

[15:55:15] <chirino> k

[15:55:15] <gemmellr> at a ession level…only link credit is used when deciding on outgoing transfers

[15:55:25] <gemmellr> there are two flow controls

[15:55:32] <gemmellr> link level, and session level

[15:56:05] <gemmellr> consumers give senders link credit…each message uses a credit

[15:56:39] <gemmellr> at the session level, there is a transfer-frame based window…where each message might take multiple transfers

[15:57:10] <gemmellr> thats tehre to avoid e.g a million links with 1 credit each resulting in a million transfers arriving at once on the session…which you might not have ability to handle

[15:57:32] <purplefox> ok but where is the api for that?

[15:57:53] <purplefox> i can't see anything on the session object for that

[15:58:04] <gemmellr> as above, proton (confusing i think) exposes that as a 'session capacity' in bytes

[15:58:16] <purplefox> if it's just an implementation detail why does the user care?

[15:58:36] <purplefox> it's just something that proton handles internally no?

[15:58:52] <gemmellr> it takes that value, and calculates the maximum number of maximum-sized transfer frames it could accomodate….if there is a max-frame-size in use. If tehre isnt, it sets the window to its maximum value rendering it 'off' effectively

[15:58:57] <gemmellr> yes

[15:59:30] <purplefox> ok, so if it's just an implementation detail then why expose a Session object in the api?

[16:00:48] <gemmellr> so you can goven it to some extent

[16:01:04] <gemmellr> it defaults to 1MB…you might want it to be 50MB or 1GB or something

[16:01:49] <purplefox> ok, maybe an advanced use casse

[16:02:10] <gemmellr> pretty much

[16:02:33] <purplefox> i think we can get rid of flush too, as flush can be called internally whenever a message is sent

[16:02:36] <gemmellr> so, going back…the engine isnt thread safe…you can create and manipulate the Message objects independently before they get given to the engine to send, but other than that almost all usage needs to be made thread safe

[16:03:00] <purplefox> which object is “the engine”?

[16:03:05] <purplefox> protonconnection?

[16:03:24] <rajith> purplefox: hey tim .. sorry I didn't see your ping

[16:03:35] <gemmellr> pretty much all of them…

[16:03:52] <purplefox> what i mean is:

[16:04:43] <purplefox> At what level is “the engine” encapsulated? The result of Proton.connection() ?

[16:05:00] <purplefox> i.e. if I create to connections like that they have separate “engines” ?

[16:05:57] <gemmellr> 'engine' = 'all of proton' pretty much, since certain methods will update other objects, but essentially everything revolves around the Transport in the end

[16:05:57] <purplefox> I assume you're not using statics to share a global “engine”

[16:06:26] <purplefox> i think i need more clarification on that

[16:06:31] <gemmellr> no, no statics (that i know of….but i wrote almost none of proton, so.. :P)

[16:07:09] <purplefox> well, if there are no statics then the engine must correspond to some object

[16:07:15] <gemmellr> essentially, each Transport instance and all the objects relating to its use

[16:07:16] <rajith> purplefox: no statics

[16:07:29] <purplefox> ok the transport corresponds to the engine

[16:07:51] <gemmellr> conection, session, links, messages (exception being those created to send, and not yet given to the sender) etc

[16:07:53] <rajith> the transport is what is the engine .. everything revolves around that

[16:08:07] <purplefox> thanks

[16:08:17] <purplefox> so that's ok, as that corresponds to the netsocket

[16:08:30] <purplefox> and we guaranteed that events on the netsocket are executed on the same thraed

[16:08:46] <chirino> purplefopur:

[16:09:39] <chirino> purplefox even

[16:09:51] <gemmellr> purplefox: yes…so all uses by the netsocket are ok in relation to each other…but e.g as you call send on the sender, you need to be sure the netsocket isnt for example processing the transport with newly received data

[16:10:24] <purplefox> ok we can synchronize to protect that

[16:10:52] <purplefox> in most cases even send will be called by the same thread so jvm will use biaised locking to give an overhead approaching zero

[16:11:23] <gemmellr> thats essentially what i was trying to ask originally…is that going to be the same thread, or not :)

[16:11:42] <purplefox> not always the same thread, but we can synchronized so it doesn't matter

[16:12:42] <purplefox> brb need to reboot my machine…

[16:15:38] * ChanServ sets mode: +o purplefox [16:16:47] <chirino> purplefox: hey something to note is that proton connections are are re-connectable. [16:17:17] <chirino> like if the socket socket dies, we can just connect it to a new socket. [16:17:25] <chirino> and it re-covers it's state [16:18:04] <chirino> you don't have to re-open new session/receivers etc. [16:18:09] <purplefox> not sure if i see the value there. if the socket dies in 99% of cases the server has probably died so will lose state when it restarts anyway [16:18:25] <chirino> you can just switch to a new server too [16:18:38] <purplefox> how will that new server have the state? [16:18:52] <chirino> they just re-create the state [16:18:56] <purplefox> from where? [16:19:20] <gemmellr> chirino: well…I believe thats the idea, and i think folks might be doing wwith proton-c…not sure anyone is with proton-j yet [16:19:23] <chirino> it's part of the amqp state recovery process [16:19:46] <chirino> they re-open all previously opened objects etc. [16:20:04] <purplefox> ok but that state needs to be stored somwhere no? so the server cen reload it? [16:20:10] <chirino> no [16:20:35] <purplefox> ok not sure i understand what you mean by “state” then? [16:20:39] <chirino> the new server will just see the client reconnect and replay is open for receivers and senders etc. [16:21:23] <purplefox> but surely the server needs to know what the last message it successfully received was so it can discard those others before it? [16:21:24] <chirino> any in-flight messages will also get re-negociated, and the server will send bad Delivery states that say I don't know about that delivery [16:21:31] <gemmellr> purplefox: I believe the idea is the Transport knows you opened x sessions, y links on it etc and so could 'reopen' them [16:21:35] <purplefox> and didn't that information go down when the last server dies? [16:21:51] <rajith> chirino: purplefox gemmellr I don't think we do the recovery part .. at least I don't see it in the proton-j code … so I'd be very careful in using that [16:22:01] <purplefox> but anyway, i don't care about that. ;) [16:22:02] <gemmellr> chirino: I dont for example know of proton-j supporting the link-recovery (message level) [16:22:11] <purplefox> seems like an advanced feature too [16:22:18] <chirino> rajith: that should be something that proton-j should do tho. [16:22:20] <rajith> gemmellr: no link recovery [16:22:57] <rajith> chirino: agreed but maybe gemmellr is the lucky star to do it for us ;) [16:23:02] <purplefox> btw… why do receivers need a name? [16:23:08] <purplefox> what does that signify? [16:23:25] <gemmellr> purplefox: all links have names,…so you can detach them and re-attach them later…in part, allowing link recovery [16:23:32] <gemmellr> purplefox: but also thigns like durable subscriptions etc [16:24:26] <purplefox> this is the thing about AMQP that kills me, it seems that 99% of the complexity is there to solve the use cases used only be 1% of users ;) [16:24:55] <gemmellr> purplefox: if you arent using those things, essentially a random name, perhaps with some detaisl of what the linkis for (e.g sender → destination) to aid diagnosis etc [16:25:29] <chirino> purplefox: it is a messaging protocol after-all :) If they wanted simple, we'd be using http [16:25:30] <gemmellr> purplefox: yes, its a general purpose protocol and carries baggage as a result [16:25:38] <rajith> chirino: lol :p [16:25:55] <chirino> purplefox: you should have seen AMQP 0.9 ! [16:26:25] <purplefox> i see my requests for stomp than amqp from vert.x users ;) [16:26:31] <purplefox> s/my/more [16:26:41] <chirino> purplefox: have you seen ?? [16:26:56] <chirino> purplefox: yeah that was a good hybrid between the http and messaging styles. [16:27:10] <purplefox> +1 [16:27:32] <purplefox> it solves the 99% case [16:27:50] <purplefox> but anyway, we digress… [16:29:11] <purplefox> chirino: i think what you've done in wraopping the proton stuff is great [16:29:12] <chirino> so I think what we are still missing is best practices for this amqp stuff. [16:29:41] <rajith> purplefox: AMQP is not the easiest to work with and it's complicated … we just need to find ways to reduce the pain for our uses (and on ourselves) [16:29:51] <purplefox> this is where I disagree [16:30:10] <purplefox> messaging is just messaging, and is still messaging whether it's stomp, amqp, jms or whatever [16:30:31] <rajith> u have a point [16:30:34] <purplefox> so the api that the user deals with should be pretty much the same api irrespective of the protocol or implementation [16:30:43] <purplefox> users don't give a shit that it's amqp [16:30:50] <purplefox> they just want to send/receive messages [16:31:06] <rajith> some customers ask why they need to do the whole song and dance when MQTT or stomp is simple …. [16:31:17] <purplefox> problem with most amqp apis I have seen is that they are *amqp* apis not *messaging apis* [16:31:20] <rajith> yep most people want to send and receive messages .. end of story [16:31:28] <purplefox> i.e. for some reason the driver is exposing the protocol details [16:31:30] <chirino> so I think we probably want to get the messenger abstractions on vert.x [16:31:42] <chirino> messenger kinda works that way [16:31:50] <rajith> purplefox: that is why a non JMS client api that is reactive and connection oriented is needed on the java side [16:31:55] <chirino> you don't even worry about setting up connections and stuff I think [16:32:00] <purplefox> there is absolutely no reason why messaging with amqp should be more complex than any other protocol [16:32:26] <rajith> chirino: purplefox messenger has serious flaws :( … but I understand the appeal of it's simplicity [16:32:31] <purplefox> after all when you use socket apis you just send and receive bytes you don't worry about sending FIN and ACK frames annd the like ;) [16:32:54] <rajith> purplefox: lol .. true [16:33:03] <chirino> but the ack patterns are important [16:33:12] <purplefox> yes acks should be exposed [16:33:20] <purplefox> i am not sure about flow control [16:33:23] <chirino> they tell you if you got at least once, at most once, once or only once etc. [16:33:30] <rajith> purplefox: chirino we understand that we have that gap … I guess it's a matter of priority. I think David will address that in time [16:33:41] <purplefox> right, those are business concerns (reliability) [16:33:46] <chirino> flow control is important is more specialized [16:33:52] <purplefox> what's not a business concern is how many tokens you send on flow control [16:34:02] <chirino> it's a tradeoff between memory and performance. [16:34:04] <purplefox> flow control can generall be hidden internally [16:34:15] <purplefox> e.g. send more tokens as messages are acked [16:34:35] <chirino> the question is not how many tokens are sent [16:34:39] <purplefox> link seems to be an implementation detail, so don't care [16:34:41] <chirino> but how large the receive window is [16:34:52] <chirino> and how many of those receive windows you create [16:34:52] <purplefox> yes the window should be configurable [16:35:00] <purplefox> i would say in number of messages [16:35:01] <rajith> chirino: which is what I've done with my API layer .. there reliability modes and then a credit mode of AUTO or EXPLICIT. [16:35:16] <rajith> chirino: auto will auto replenish using a sliding window [16:35:17] <purplefox> but what the user shouldn't be able to do is explicitly send tokens [16:35:28] <purplefox> that's playing too directly with the protocol imho [16:35:30] <chirino> oh don't worry with proton he does not [16:35:47] <chirino> not sure where you thought that was happening [16:36:05] <rajith> yea I'm curious to know too? [16:36:08] <gemmellr> chirino: the presence of the 'flow' call :) [16:36:09] <purplefox> ok why do we have a a flow(int) method in the api? [16:36:09] <chirino> oh perhaps the flow stuff [16:36:32] <chirino> yeah we could auto do that. [16:36:39] <rajith> agreed [16:36:50] <purplefox> right, which is what jms does [16:36:55] <chirino> but once again that's a performance thing that affects sematics [16:37:16] <gemmellr> the thing to note is that proton[-j] is not a client, and was never intended to be…the JMS client for example is, and hides all that…as would any other client written around it, unless the user really wanted control of it [16:37:48] <purplefox> fair point [16:38:13] <gemmellr> for this use it is as you say, just a question of how big to set it, and how often to replenish it as used, since those both affect performance [16:38:18] <rajith> which is why we need something btw JMS and proton engine API [16:38:30] <purplefox> +1 [16:38:48] <purplefox> the JMS 2 api is actually not too bad. e.g. for simple sending of messages is pretty straightforward [16:39:09] <purplefox> i think an amqp client could do something similar [16:39:38] <purplefox> i.e. not require creating of session and producer just to send a message (maybe maintain a default session/producer internally?) [16:40:02] <gemmellr> so, one other thing that hasnt been mentioned yet but is probably important… [16:40:30] <gemmellr> links are to explicit destinations… [16:41:19] <gemmellr> each message carries a 'to' field that need not be the same as the address the link attached to, but the core spec did not really include the cocept of an anonymous producer / 'link that can send anywwhere' [16:41:29] <chirino> purplefox: +1 [16:41:57] <chirino> we could basically squash down sender into connection [16:42:12] <chirino> and have an option to create a receiver from a connection [16:42:20] <chirino> which basically just use the default session [16:42:24] <gemmellr> that has been extended with a connection-capacility used to inducate support for links that are anonymous producers, with the messages carrying 'to' and that being used to determine their destination [16:43:22] <purplefox> yes i think the user just wants to know that whatever is in the to: is where the message goes through [16:43:29] <gemmellr> im guesing with the API suggested so far, you probabyl want that behaviour, or a single link carrying messages for multiple destinations…rahter than the alternatives (link to each destiantion as used…caching etc to make it performant) [16:43:44] <purplefox> i don't think the user needs to know about links at all [16:43:51] <purplefox> what advantage does a link confer? [16:43:55] <gemmellr> s/or a single/of a single/ [16:44:18] <gemmellr> thigns like 'destination exists' 'destination went away' etc [16:44:32] <purplefox> not sure i follow… [16:44:52] <purplefox> i think a user just wants to send a message to an “address” [16:45:03] <chirino> you can try to create a sender to a bad destination ( the sender has the target address) [16:45:18] <gemmellr> for something like JMS, where you often create consumers.recievers from a specific destination…the link represents that consumer/producer [16:45:20] <chirino> open reponse would have an error if its a bad target [16:45:52] <chirino> in case the client wants to check that the target is valid. [16:45:57] <purplefox> is that the only reason a link exists? [16:46:29] <chirino> it's one way it can be used. But you need it to move messages [16:46:29] <gemmellr> but obviously JMS suppots anonymous producers as well…so the client makes use of the 'anonymous relay' stuff when support is signalled, so it can do what you want….single link, messages carry desitnation [16:46:40] <chirino> but you could use an anonymous link for tha ttoo [16:46:50] <chirino> basically where target is null [16:47:04] <gemmellr> purplefox: there are various reasons…im not saying they are good…i did not write the spec :) [16:47:13] <purplefox> to me i think of messages like emails with a to address. as long as it gets to the destination in the to: then it's all implementation details [16:47:14] <gemmellr> chirino: where target *address* is not [16:47:23] <rajith> anon links (or general purpose links) are very useful when message based routing is done … both dispatch and the vertx-amqp-service use that option [16:47:37] <chirino> anonymous link is where target address is null [16:47:47] <gemmellr> purplefox: another reason may be, if all your messages are going to the same place, no need for them to all carry that info… [16:47:57] <purplefox> in vert.x the message will always have a “to:” address [16:48:05] <gemmellr> chirino: yeah., s/not/null/ :) [16:48:19] <rajith> chirino: null? [16:48:28] <chirino> right? not set? [16:48:37] <gemmellr> purplefox: well it will have to if we use a single link, yes…just saying, in the general case… [16:48:45] <gemmellr> chirino: correct [16:48:57] <rajith> chirino: yea [16:49:01] <purplefox> i don't think it's a big deal to compell users to specify a to: [16:49:14] <purplefox> after all we do that in email and I've never seen anyone complain [16:49:17] <gemmellr> to do the anonymous-relay behaviour, where the capability is offered, the Source/Target is set…but the address is not [16:50:12] <gemmellr> purplefox: im not arguing otherwise…jsut discusing general messaging (e.g JMS) semantics and what the protocol does support, for whatever reason… [16:50:13] <purplefox> ok again, seems like an advanced use cases that most users won't care about [16:50:26] <rajith> purplefox: +1 [16:50:30] <purplefox> i think most users are happy saying “send this message to this address” [16:50:32] <chirino> purplefox: some messaging systems wont support that tho [16:50:43] <rajith> purplefox: I handle that underneath [16:50:57] <chirino> they will require the link address to be set [16:52:43] <purplefox> rajith: I had a quick hack this morning on the vert.x event bus and added some embryonic “interceptor” functionality [16:52:51] <rajith> purplefox: that vert'x address could map to a link address that 'it' knows, if not it will be set via the message.address field using a the default link where routing happens elsewhere [16:52:54] <rajith> purplefox: cool [16:53:00] <purplefox> which basically allows all messages to be intercepted [16:53:09] <purplefox> I think this would be a good plug-in point for amqp bridging [16:53:41] <rajith> purplefox: are u envisioning that vertx will do the routing itself … or leverage something like dispatch do it on it's behalf .. Ted is very keen on the latter case [16:53:55] <rajith> purplefox: yes +1 [16:54:14] <purplefox> i am envisioning that the vert.x bridge is very simple and just uses this new proton client to basically forward messages to amqp-land [16:54:26] <purplefox> and in the other direction create consumers [16:54:47] <purplefox> so “amqp-land” could be dispatch router or it could be a broker, it doesn't matter really [16:55:02] <purplefox> so the bridge should actually be very simple [16:55:03] <rajith> purplefox: yes forwarding to the amqp land means you need to know where to forward [16:55:14] <purplefox> where is in the address of the vert.x message [16:55:14] <rajith> purplefox: thats what I meant [16:55:26] <gemmellr> purplefox: rajith: my understanding was 3 uses cases: p2p, dispatch, broker [16:56:26] <purplefox> but the bridge doesn't care about this [16:56:33] <rajith> gemmellr: the endpoint can be another AMQP app … doesn't have to be dispatch or a broker … remember the demo I did? [16:56:36] <purplefox> it's just a host and a port [16:56:44] <rajith> purplefox: ye [16:57:00] <rajith> purplefox: I don't make a distinction btw p2p, dispatch or broker ..etc [16:57:28] <rajith> purplefox: I'm endpoint to receive messages from any AMQP node or forward to any AMQP endpoint [16:57:43] <gemmellr> ah, i missed you swapped to talking about the bridge (ie. the existing stuff rajith has been doing) and not the 'other stuff' :) [16:58:10] <rajith> purplefox: that's all it does … + an address translation and a message translation (json ←→ amqp format) [16:58:29] <rajith> gemmellr: sorry should have been clear [16:58:37] <chirino> purplefox: nudge –> [16:58:58] <chirino> purplefox: if we had that it would be really simple to support AMQP on websockets too. [16:59:06] <rajith> purplefox: if the event-bus is run over AMQP then the bridge is redundent [16:59:16] <purplefox> it's the same [16:59:25] <rajith> purplefox: exactly [16:59:42] <rajith> purplefox: the bridge is pretty much an extension of the event bus [17:00:09] <rajith> purplefox: when we have the event-bus speaking AMQP we don't need the bridge .. as u said it's the same damn thing [17:00:33] <rajith> purplefox: “ extension of the event bus” .. into the AMQP space [17:00:42] <purplefox> no i don't mean that [17:00:59] <purplefox> I've just created some interceptor functionality for the event bus [17:01:08] <purplefox> so basically you can intercept any messages [17:01:18] <purplefox> so the bridge just becomes an interceptor [17:01:45] <rajith> purplefox: gotcha …. instead of having to register handlers … you now allow to intercept .. thats cool [17:01:53] <purplefox> the bridge can just use the vertx-proton client to send/receive to amqp-land [17:01:58] <purplefox> which should be pretty trivial [17:02:05] <rajith> purplefox: +1 [17:02:25] <rajith> purplefox: you have some code on the interceptor ? [17:02:38] <purplefox> take a look in the eb_interceptors branch [17:03:04] <rajith> purplefox: will do [17:03:20] <purplefox> [17:03:27] <gemmellr> purplefox: so you mentioned acknowledgements the other day, do you see that being explicit, or done under the covers where the message is accepted just after the handler runs? (kind of like auto-ack onMessage() in JMS) [17:03:32] <rajith> purplefox: however … I think David has a different idea about the eventbus thing …. or perhaps I missunderstood … gemmellr can confirm. [17:04:01] <rajith> purplefox: I think he envisioned that AMQP is used instead of hazelcast or jgroups to power the event bus [17:04:02] <purplefox> gemmellr: we should expose acks [17:04:06] <gemmellr> rajith: everything I know was mentioned above, lol [17:04:26] <rajith> gemmellr: lol .. just trying to make sure I understand [17:04:47] <purplefox> rajith: it's the same. if you intercept all messages and and route them via amqp that's the same logically as replacing the entire event bus with amqp [17:05:10] <rajith> purplefox: yea you have point! [17:06:08] <rajith> purplefox: l will have a look at the interceptor. [17:06:27] <rajith> purplefox: and give you a shout tomorrow morning. [17:07:43] <purplefox> rajith: here's an embryonic bridge: [17:07:44] <purplefox> [17:07:57] <purplefox> i think you get the idea [17:08:00] <gemmellr> purplefox: so essentially require folks to update the message dispotion and settled, potentially wrapped a bit nicer…in which case, if linking acks and credit, is it then goign to be up to them to ensure the window is big enough for them to receive all they want before acking? [17:08:07] <rajith> purplefox: ….btw how is an interceptor different from handlers? Does an interceptor make sure that the message only gets to the “component” intercepting it? [17:08:15] <rajith> purplefox: and could we do wildcard? [17:08:38] <purplefox> interceptors intercept everythikng [17:09:24] <rajith> purplefox: looking at the test case it's a bit clear now [17:10:06] <rajith> purplefox: so it intercepts everything sent on the “armadillo” name space ? thats the idea? [17:10:16] <purplefox> what code are you looking at? [17:11:08] <purplefox> interceptors intercept everything. full stop [17:11:09] <rajith> purplefox: ok I missunderstood … I misread the damn thing. [17:11:19] <purplefox> take a look at the bridge link above [17:11:26] <purplefox> you can see it checks a prefix [17:11:36] <purplefox> but you can do whatever you want [17:11:44] <purplefox> regexp, prefix, whatever [17:12:03] <purplefox> it was just an illustration [17:12:14] <rajith> purplefox: ok I get the point [17:14:24] <purplefox> rajith: anyway, that's what i was thinking. keep it simple [17:15:30] <rajith> purplefox:I like what u have done .. I think it allows us to do what we want with minimum fuss [17:15:39] <purplefox> we cn [17:15:48] <purplefox> we can always add more features later if need be [17:16:10] <rajith> purplefox: interceptors + bridge will easily do the whole eventbus-over-amqp [17:16:53] <rajith> purplefox: yes .. acks or flow control (which I think shouldn't be exposed) can be built on top of that later [17:16:54] <purplefox> and the nice way of doing it at this level is you can have different event bus addresses mapping to different things [17:17:02] <rajith> purplefox: yes! [17:17:11] <purplefox> e.g. some addresses go to amqp, others go to camel (using the camel bridge) others go to stomp etc [17:17:16] <purplefox> others to mqtt [17:17:25] <rajith> purplefox: we should put together a demo that shows this [17:17:27] <rajith> purplefox: +1 [17:17:32] <purplefox> but the user just uses the same api for everything [17:17:38] <rajith> purplefox: yes! [17:17:47] <purplefox> great, we are on the same page. [17:17:52] <rajith> purplefox: yes :) [17:17:57] <purplefox> now.. are you happy to own this and hack on it? [17:18:04] <rajith> purplefox: yep [17:18:13] <purplefox> cool [17:18:41] <purplefox> and the vertx-proton stuff -i don't know who wants to own that - maybe you, or hiram or robbie, or maybe all of you it's up to you ;) [17:19:10] <rajith> purplefox: I will try to get a rudimentary demo going … I will just use the existing code I've done [17:19:45] <purplefox> by existing code you mean? [17:20:10] <rajith> purplefox: the AMQP code I already have under the vert.x-amqp-service [17:20:20] <rajith> purplefox: so I can quickly put together a demo [17:20:42] <purplefox> i thought that does things a completely different way to how we just discussed? [17:21:25] <rajith> purplefox: yes .. I didn't mean the concepts … but the code etc… so I can quickly clobber something together [17:21:43] <rajith> purplefox: for the same of doing a quick demo/poc [17:22:06] <purplefox> rajith: up to you, but I'd suggest it might be simpler to just use the new client that hiram put together [17:22:25] <purplefox> i'm not sure we really need a demo [17:22:45] <purplefox> but if we can get the basic use case working that's a demo i guess ;) [17:22:53] <rajith> purplefox: I didn't look at the latest thing chirino did .. is it the same link u sent? [17:23:00] <alvaro_sanchez> purplefox:I updated the code to use JDBC client directly: [17:23:15] <rajith> purplefox: thats what I meant :p [17:23:35] <purplefox> here: [17:23:43] <rajith> purplefox: cool [17:39:48] <purplefox> alvaro_sanchez: cool. [17:40:11] <purplefox> i think you can simplify things a bit by moving the repeated getConnection code into its own method [17:40:35] <alvaro_sanchez> yes, I'm on it atm [17:40:43] <purplefox> i also think that the example would be simpler in Java - then you could use vertx-sync. I don't think groovy buys you much here. but that's just my opinion ;) [17:43:48] <alvaro_sanchez> the difference is probably minimal with vertx-sync, maybe a line per method, but if I refactor it, even less [18:01:53] <chirino> purplefox: [18:10:16] <purplefox> chirino: so you put everything back that I took out? ;) [18:10:41] <chirino> you kinda need most of it. [18:11:08] <chirino> going to do the default session / sender stuff next [18:11:26] <chirino> that should simplify it a bit for most users [18:13:34] <purplefox> who needs most of it? [18:13:52] <purplefox> for some advanced use cases fair enough. but for what we're doing here I don't think so [18:15:00] <purplefox> i usually take the approach of “don't add stuff to apis unless there's someone with a real use case” [18:15:21] <purplefox> rather than “chuck everything and the kitchen sink in there just in case someone in the future wants it” ;) [18:15:53] <purplefox> i think sometimes it can cause users freakout and api blindness when they are greeted with a complex api just to do something simple [18:16:23] <purplefox> also, it means a lot more tests to write! ;) [18:29:00] <chirino> purplefox: ok it's in [18:29:14] <chirino> he default session stuff [18:29:25] <chirino> so if you want to talk to AMQ you probably need most of it [18:30:15] <chirino> to do stuff like durable sessions etc. [18:32:04] <purplefox> well you are the amqp expert.. if you think it's really necessary then fair enough, but someone will need to write tests for all that [18:32:33] <purplefox> as long as there is a “shortcut” way to use the api for the simple use case where the user doesn't have to worry about tags and links and all that [18:41:39] <purplefox> chirino: if we make the vertx-proton api an “official” part of the vert.x stack, then it will need an official maintainer. would you be interested in being that person? [18:41:53] <purplefox> then you get commit rights in the repo etc [18:42:04] <chirino> yeah thats fine. [18:42:05] <purplefox> but it also means you have to write tests and fix bugs ;) [18:42:23] <chirino> will try my best. been itching to do a little messaging on the side [18:42:31] <purplefox> awesome. [18:42:41] <chirino> messaging is like a bad habit [18:42:59] <purplefox> what are you mainly working on these days? [18:44:10] <chirino> fuse [18:44:56] <chirino> which means a little camel, karaf, and lots of fabric :) [18:45:06] <purplefox> ok cool. well james and claus are already vert.x component maintainers so you're in good company :) [18:45:11] <purplefox> can u take a look at this: [18:45:23] <purplefox> and this [18:47:52] <purplefox> chirino: i'll move that repo to the vert-x3 umbrella and give you push/pull rights [18:48:15] <purplefox> while hacking it will need to be on a non-master branch as we have a rule under vert-x3 that master branch = production ready [18:48:38] <purplefox> i'll also need to get you voted in as an official committer [18:49:00] <purplefox> but apart from that - hack away! :) [18:49:05] <chirino> can we use the master branch on a fork? [18:49:57] <chirino> or does it have to be a feature branch on official repo? [18:52:30] <chirino> purplefox: you guys got tips for junit tests that want to do assertions in handlers? [18:53:15] <AlexLehm> chirino: you can take a look at vertx-unit [18:53:32] <chirino> AlexLehm: thx will do [18:53:42] <AlexLehm> that is really good for testing vertx async stuff [18:54:04] <purplefox> chirino: we prefer to do feature branches on official repo because we have github hooks to make all the commits go the vertx-commits list then everyone can see what is going on :) [18:54:18] <chirino> I see ok. [19:03:07] <purplefox> if you really prefer to work on a fork it's no big deal [19:03:39] <purplefox> chirino: ok, i have moved the repo here [19:03:45] <purplefox> and the work is on branch “initial-work” [19:03:48] <purplefox> master is empty [19:03:59] <chirino> ok [19:04:00] <purplefox> you should have push/pull rights once you accept the invitation [19:06:13] <purplefox> chirino: rajith is doing the vert.x amqp bridge so he is your first user :) [19:06:21] <chirino> sweet! [19:06:33] <chirino> I <3 smart users [19:06:38] <purplefox> :) [19:10:32] <rajith> chirino: lol [19:11:34] <AlexLehm> purplefox: could we discuss the issue, I am not quite sure what I should add to that to get it merged [19:18:04] <chirino> AlexLehm: thx for the tip vertx-unit rocks [19:27:36] <chirino> rajith: so there's local state that HAS to be set before an the local open [19:28:00] <chirino> perhaps it would be better to put that into like an OpenOption object [19:28:06] <chirino> and pass it to the open call [19:28:53] <chirino> for example calling connection.setContainer does not make sense after [19:29:17] <chirino> purplefox: ^^ [19:29:57] <chirino> but that adds another object which complicates things [19:30:18] <chirino> is there a pattern that vert.x has for these kinds of things? [19:41:25] <rajith> chirino: correct [19:43:37] <rajith> chirino: but as you mention how best to pass that information is something to be discussed. [19:45:15] <rajith> chirino: I haven't taken a closer look at the work you've done yet. Lets discuss further tomorrow. I want robbie to part of it too. [19:47:13] <chirino> k [19:47:27] <rajith> chirino: however the container name is actually interesting … bcos it's not something that an application user will necessarily care about. On the other hand the container-name would be useful in figuring out where a message originated (as in which amqp node) for operational and debug reasons. [19:48:14] <chirino> rajith: it depends on what your talking to also. For example ActiveMQ 5.x rejects AMQP connections that don't set it. [19:48:45] <rajith> chirino: yes I noticed that [19:48:45] <chirino> so it's more than useful, it's required in some cases [19:49:49] <rajith> chirino: u guys use it as part of the identification for a connection [19:49:54] <chirino> yes [19:52:58] <rajith> chirino: I think in the vert.x case we will have to default it to some sort of sensible name : vert.x-<version>-UUID that can be overriden via configuration [19:53:29] <chirino> that sounds reasonable [19:58:10] <chirino> rajith: do we really need to set tags for deliverys? [19:58:25] <chirino> is that mostly just a recovery thing? [19:58:41] <chirino> should we generate those too? [20:00:42] <rajith> chirino: I just you an integer and increment it. My understanding it is that might be useful in recovery and duplicate detection [20:01:01] <rajith> chirino: I just *set* an integer [20:01:28] <chirino> but like for at most once QoS, there is no point [20:01:51] <chirino> i.e. when we send messages pre-settled. [20:09:24] <rajith> chirino: I was looking through the proton code to see if I can give you a better answer … give me a few mins [20:10:19] <chirino> k thx! [20:10:37] <chirino> while your at it, doe we really need to name links? [20:10:48] <chirino> since we don't do recovery [20:11:19] <rajith> chirino: again I think you don't unless you are doing some sort of link based routing [20:11:52] <chirino> link name and target are 2 different things [20:14:06] <rajith> chirino yes they are … I don't mean it in the traditional sense. [20:14:18] <chirino> k [20:14:25] <chirino> I'm not familar with the link routing stuff [20:14:49] <rajith> chirino: I name my links as a way to uniquely identify them on my end (locally) .. for example map a link-name to a vert.x address or some sort of address [20:15:09] <rajith> chirino: I shouldn't have used the word link routing [20:15:35] <rajith> chirino: link routing refers to routing based on the source/target … so my bad for confusing you [20:16:24] <chirino> k! [20:19:36] <rajith> chrino also giving a name to a link is good for debugging (for if u have two different links going to the same source/target) [20:27:25] <chirino> rajith: seems the name field is mandatory [20:27:41] <rajith> chirino so is the delivery-tag [20:27:47] <chirino> but perhaps we should auto gen it to [20:28:33] <rajith> chirino same behavior with proton-C … so perhaps a deliberate choice to force an app to name it … most of my code sets a UUID. [20:28:47] <rajith> chirino: so I see your point about doing an auto-gen if not supplied [20:28:54] <chirino> ya [20:29:19] <chirino> keep in mind this stuff goes over the wire so keeping it small would be good [20:31:47] <rajith> chirino: true … but then again AMQP is a very chatty protocol … ;) [20:31:57] <chirino> :) [20:33:59] <rajith> chirino: I'm sometimes amazed at how much bytes we shuffle over the wire to send and receive a simple message :) [20:34:19] <chirino> yeah [20:34:28] <rajith> chirino: not really the bytes .. but the song and dance the protocol forces us to do [23:55:24] * ChanServ sets mode: +o temporalfox