IRC Logs for #circuits Friday, 2013-01-25

*** Osso has joined #circuits00:40
*** ronny has joined #circuits05:55
ronnyprologic, ok, im starting to see what my problem was now05:56
ronnythe channels where typed events flow from components ticks me totally off05:57
prologichow so?05:57
prologicand remember we majorly changed a lot of that in 2.x05:58
ronnyprologic, im just looking at the examples of circuits-dev freshly cloned05:58
prologicwe removed a lot of crud and useless magic to make the api clean and concise and a lot easier to understand05:58
prologicso what's the problem exactly?05:58
prologicexamples are good to talk to05:58
ronnyprologic, the problem i have is, that practically everything fires events on self06:01
ronnythen they somehow flow to the right thing06:01
prologicnot sure why you're bogged down on that tbh06:02
prologiceverything doesn't fire events on self06:02
prologicit's just the calling mechanism06:03 will always bubble up to the root manager06:03
prologicthe main event loop06:03
prologicall that does is put Foo() in the queue06:03
ronnyand then it chooses the channel and passes it down that way, right?06:04
prologicno there is no choosing of channel06:05
prologicif you wanted it to go to a specific component or channel06:05
prologicyou'd do:06:05, "my_other_channel")06:05
prologicotherwise it defaults to the channel of the current component06:05
ronnyi also really dont see the point of channels06:06
ronnywhat i want is directly connect 2 components that are cooperating endpoints06:07
prologicwhat if you wanted to run two TCPServers in a single process?06:07
prologictheir events would collide without separate channels06:07
prologicchannel (we changed this in 2.x)06:07
prologicare strictly a separation of concerns06:07
prologicnothing more06:07
prologicif you want 2 components that are cooperating endpoints06:08
prologicthen just create two components06:08
prologicdon't worry about setting any channels06:08
prologicthe circuits.bench marking tool does this06:08
prologicone component fires a Hello event06:08
ronnyprologic, no global event quues06:08
prologicthe other receives it and fires a Received event06:08
prologicit does this as fast as possible06:08
prologicwhilst the Monitor component watches and keeps a count and time06:09
prologicyou want separate event queues?06:09
ronnyyeah, i never ever want a global event queue06:09
prologicwe can/do allow for that06:09
prologicby way of bridging06:10
prologiccurrently for subprocesses though06:10
prologicbut we could build a bridge that uses threads and a Queue06:10
ronnyone of the reasons being, that it'll completely fail on top of pypy stm06:10
prologichow so?06:10
ronnyprologic, global event bubble ing is bad if you want to have atomic transactions06:11
prologicno I think you misunderstand06:11
ronnyok, then please explain06:11
prologicevent queieing in circuits is atomic06:11
prologicwe dont' bubble events up per say at all06:11
prologicit's not too different from the tornado or twisted event loops06:12 quite literally puts the Foo event in the root manager's queue (the event loop)06:13
prologicunless you break apart the component graph and run part of it in another thread or process (which we do in tests)06:13
prologicand mehere here has done an incredible amount of work to make all this thread safe06:13
prologicand FYI circuits runs perfectly fine on pypy06:14
prologicand has a 10-15x performance boost :)06:14
ronnyprologic, so no matter what i do, i cant have 2 component graphs of the same form in the same thread without explicitly setting channel names?06:14
prologicyou can ofc06:15
prologicbut the events will collide06:15
ronnyyup, then that is exactly what bugs me off06:15
prologicif you take the examples/ example06:15
prologicand create two instances of the App component06:15
prologicyou'll get two Hello World! resposnes06:15
prologicyou want 2 or more componetns with identical event handlers06:16
prologicin the same process/thread06:16
prologicthis ofc implies an architectural change in circuits.core itself06:16
prologicwhich would greatly impact on performance06:17
prologicbut I do see what you're getting at06:17
prologicthe problem is. to what end?06:17
prologicall our talks have been more or less speculation06:17
ronnyprologic, i want to be able to run events of all graphs in differnt threads at the same time06:17
prologicwhat is your end goal?06:17
prologicif you really wanted that06:18
prologicjust start each component in a separate thread06:18
prologicand ignore component graphs altogether06:18
prologiceach component gets it's own queue btw06:18
ronnyprologic, then i'd have to start a thousand threads in some cases06:18
prologicit's just when you register a component to another component (a manager)06:18
prologicit affects where and how works in that component06:19
prologicwell mabe not a thousand06:19
ronnyso as long as components are singular, its fine?06:19
prologicthe class hiearchy looks like this:06:19
prologic   |-Component06:20
prologicevery component is capable of manging it's own queue06:20
prologicas long as you run it in some manner06:20
prologiceither .start() .start(process=True) or .run()06:20
prologicthe latter blocking06:20
prologicwith a little work/effort we could accomodate your needs easily I think06:21
prologicif all you wanted were separate event queues06:21
prologicwhich every component has06:21
prologicit's just the running manager/component whoose queue gets used in any component graph06:21
prologicchildren put their events in the manager's queue06:22
prologicbut nothing stopping you from having many graphs running simultaneously06:22
ronnyam i understanding it correctly that each running graph needs a own thread?06:23
prologicwell umm06:23
prologicmore precisely you can start any system (component graph) in one of three ways06:23
prologicthreaded, processes or main thread06:23
ronnyso the answer is yes06:24
prologicif what you're after is a central manager to manager the queues of many components/graphs06:24
prologicwe don't currently have that facility06:24
prologicbut it would be trivial to add I think06:24
ronnyhow to avoid collision then?06:24
prologicwell we normally avaoid collision by way of component channels06:25
prologicclass Foo(Component):06:25
prologic   channel = "foo"06:25
prologicfor example06:25
prologic-however- if what you want is a manager that manages the queues of many components/graphs06:25
prologicthen channels are almost irrelevant06:25
prologicexcept in small graphs of components06:26
prologicit's up to you06:26
ronnythats how i would like it06:26
prologicI think it's very feasible what you want06:26
prologicessentially the way I see it is this:06:26
ronnyim thinking of how i'd like component graphs to look and how i'd like smaller graphs to interact06:26
prologicmanager = Manager()06:26
prologicor something that differentiates component registration06:27
prologicif I were to articulate a story on this06:27
prologiccould you review it for it's correctness06:27
prologicand I'll discuss with the other devs and we'll try to develop it in the next sprint06:28
ronnyprologic, i'll also think of making a minimal example of component graphs how i envision them to show the ideas/concepts, will take a while tho06:29
prologicone thing though06:30
prologicyou do realize that such a feature/use-case would be very slow06:30
ronnyprologic, i do not see how it would be slow to have that interaction direct, all events would explicitly have a single target06:34
prologicit would be slower because the manager would be iterating O(n) over all the components/graphs in the system06:35
prologicflushing their queues and processing their events06:35
prologicit will be a lot slower than what the current(default) implementation is now06:35
prologicI would guess 10x slower06:35
prologicbut we can try :)06:36
prologicmaybe we'll do it in a separate branch06:36
ronnysounds like there is a fundamental problem06:36
prologicI'm just saying from a design pov it will be slower06:36
prologicmaybe 10x slower is a slight exaggeration06:37
prologichave you seen pulsar?06:37
prologicit's based on the actor model06:37
prologicand probably achieves what you have in mind somewhat06:37
prologicbut it's god awfully slow compared to circuits06:37
ronnyprologic, havent, let me take a look06:38
prologicthe only way I can see of overcoming performance issues06:38
prologicis by utilziing a thread pool06:38
prologicand trying to process all queues concurrently as much as possible06:38
prologicwhich is also trivial for us to do06:38
ronnyi dont understand why there is a need to have thousands of queues then06:40
ronnyif a message hass a source and a target, processing can be a singular point06:41
ronnyno need to have N when 1 suffices06:41
prologicwell this is our point of confusion then06:41
prologicbecause that's what we have by default06:41
prologicwe keep spinning on this :/06:41
prologicperhaps let me demonstrate by way of example?06:42
prologicsee if I understand you06:42
ronnyprologic, the problem is, that yours has collisions06:42
ronnyoh wow, pulsar is a mess06:43
ronnyi can see why they suck06:44
prologicyeah hang on06:45
prologichold that thought06:45
prologiclet's work with this example06:45
prologicso far so good06:46
prologicas you can see the two handlers collide06:46
prologicnow there are two ways of solving this06:46
prologicthree actually06:46
prologic1) setting s separate channel on App1 (or App2)06:46
prologic2) setting the handlers as filters06:46
ronny ?06:46
prologic3) running App2 in a different thread06:46
prologicyeah ok06:47
prologicwhat does register_namespaced mean and do?06:47
ronnyif subgraphs can have a namespace of their own, collisions disappear06:47
prologiccan I at least demonstrate 1) and 2) though?06:47
prologicso you understand those06:47
prologicyes okay06:48
ronnyyes, please06:48
prologicI'm with you now06:48
prologicok brb06:48
ronnyprologic, if the root manager is the namespace, except for those that are registred as namespace on their own i think the problem is mostly solved06:48
ronnythen there is only communication between namespaces missing06:49
ronnywhich will feel a bit bad as fire(event, channel, namespace)06:49
prologicI don't quite mind what you're proposing06:50
prologicnot only is it simple06:50
prologicit should work perfectly well06:50
prologicbut perhaps it could be even simpler from an API pov06:51
prologic.register(manager, namespace=None)06:51
prologicand06:51, *channels, namespace=None)06:51
prologicand also for the .call and .wait primitives06:51
prologicit adds another/different level of separation which I think is nice06:51
ronnywhy is channels a arglist?06:55
prologicbecause you can fire an event off to multiple channels or components06:57, "app1", "app2")06:57
ronnyprologic, one could fire it at a component of a different graph, right?06:58
prologicby providing an instance07:00, my_other_graph)07:00
ronnyprologic, that gets confusing with the namespaces vs the channel names07:01
prologicyeah I suppose it does07:01
prologicthe other idea I had07:01
ronnyits tricky, i sense various edge-cases i didnt take into account07:01
prologicwas using channels themselves as optional namespaces07:02
prologicchannel = "foo:bar"07:02
prologicany sub-component register to such a parent07:02
prologicwould inherit the namespace07:02
prologicso then you could do:07:02, "app1:")07:02
prologicfor example07:03
prologicor07:03, "app1:some_component")07:03
ronnyprologic, i dont like the implicit namespaces07:04
*** ronny2 has joined #circuits07:06
prologicisn't it explicit?07:09
*** ronny has quit IRC07:09
ronny2prologic, its implied from string contents, and needs substrings07:17
prologicunsure of a better way right now07:18
prologicnamespace= is good07:18
prologicbut like you said confuses things with channels a bit07:18
prologicI prefer channel = "namespace:channel" in some ways07:19
prologicbut if you can come up with a more explicit way great :)07:19
ronny2prologic, i would keep channel nad namespace separate07:35
ronny2back later, got busy07:35
prologicand should we allow the api to fire an event into multiple namespaces07:38
prologicor just one?07:38
prologica nice kwarg convenience would be ns="foo"07:38
ronny2prologic, i'd use longer names, and not sure on one or many08:08
prologicI'm not even sure how that would work tbh08:09
prologicso it's singluar for now08:09, *channels, namespace=None)08:09
ronny2prologic, btw, do you have a socketpool?08:10
prologicsay what?08:10
prologicnever heard of such a thing08:10
prologicthread and process pools sure08:10
ronny2something that keeps connections alife08:10
prologictcp client connections?08:10
ronny2like for http keepalife08:10
prologicI'm not quite sure I follow08:10
prologicwe implement comet-style in circuits.web08:11
prologicas well as websockets08:11
prologicand well you see irclogger_ here?08:11
ronny2prologic, a socketpool is something where to put still alife tcp connections to a certain tcp/ip endpoint and retain them later08:11
prologicit's been connected and running for a month or so now08:11
ronny2so a new http request can reuse a keepalife connection for example08:11
prologicour circuits.web.client can do that anway08:12
ronny2prologic, its totally not the same tho08:13
ronny2back in 10 to explain08:14
prologicsorry just not getting it quite yet :)08:14
ronny2prologic, a socket pool is a pool of sockets, it has an upper bound on how many it can have, and it keeps them so other bits can reuse08:21
ronny2a good example might be the library SocketPool, which is used in restkit (and works with plain, gevent and eventlet)08:21
prologicbbs - feeding Alice08:21
ronny2basic idea is to keep them alive in the pool, and have other components reuse them later08:22
prologicso by implementing a SocketPool component08:24
prologiccan we forgo creation of new sockets?08:25
prologicwe just either rebind or connect them to ned end points?08:25
prologicwe'll build one08:31
prologicshould be fairly easy I think08:31
prologicok I'm done for the night08:40
prologicsee you all tomorrow night08:40
*** Osso has quit IRC09:08
prologicronny2, ping?09:45
prologiccan't sleep :/09:45
prologicif we implemented namespaces09:46
prologicwouldn't it kind of make channels redundant09:46
prologicand assuming that's True09:46
prologicwe could just make channels so-called namespaces09:46
ronny2prologic, im not exactly sure whats the point of channels in the first place09:47
ronny2i really dont understand their point, except resolving the ambiguities that arent there09:47
prologicwell it is kind of like namespacing09:47
ronny2eh shouldn't be there i mean09:48
ronny2the thing is, the way channels are used is a bit weird09:48
prologicwell they're used to separate components that have identical event handlers09:48
prologicnothing more09:48
prologicwhereas I see namespaces as separating an entire component graph09:49
prologicso in theory the two should work together09:49
ronny2hmm, main problem atm is, that i cant make a correct mental map of the problem09:49
prologicI think I can09:49
prologicas you create a component graph09:50
prologicchildren inherit their namespace from their parent09:50
prologicso you could have multiple namespaces within a single system09:50
prologicand multiple channels within a single namespace09:50
ronny2sounds correct09:51
ronny2but i have a nagging feeling something is missing09:51
ronny2i think i would have to understand the message bus again09:53
prologicwell obviously we try to make things as fast as possible09:56
prologicso we walk the tree and cache all handlers we find09:56
prologicand we have logic that makes the cache stale at various points09:56
prologicbut essentially every event handler is stored in a dict mapping of channel -> list of handlers09:57
prologiceach handler stores (by way of the handler decorator) the events it listens to and various other tid bits09:57
prologiceg: like it's priority or whether it's a filter09:57
*** ronny2 has quit IRC11:52
*** ronny2 has joined #circuits12:05
*** ronny2 has quit IRC14:32
*** ronny2 has joined #circuits14:32
*** ronny2 has quit IRC16:02
*** ronny2 has joined #circuits23:18
*** ronny2 has quit IRC23:18
*** ronny has joined #circuits23:19

Generated by 2.11.0 by Marius Gedminas - find it at!