IRC Logs for #circuits Monday, 2014-03-03

*** irclogger_ has joined #circuits00:24
*** realzies has joined #circuits00:55
prologichey I thought about this really great idea to fix/solve the netsplit problems with irclogger06:41
prologicI make irclogger a dstributed bot06:42
prologicthat connects to N servers06:42
prologiclistens to N messages06:42
prologicand records 1 log entry it sees06:42
prologicidentifying each as a unique sha hash06:42
prologicRomster, pdurbin ping?07:06
*** bloody has joined #circuits07:25
prologicbloody morning to you too :)07:27
prologicit's evening here07:27
prologicand I'm about to step out :)07:27
bloodyit's monday here :/07:27
prologichence the bloody morning :)07:27
prologicwhat can we do for you?07:27
bloodyactually i have a problem with circuits, when i'm using the Write Event from
prologicok gotta go07:28
prologicsorry I'll be back in ~3hrs07:28
prologichave to run :)07:28
prologicwill answer your q's then :)07:28
bloodyno problem, i prepare code to show07:28
Romsterprologic, i was thinking of something distributed07:52
bloody examplecode, but after testing it with pickle, it looks more like a problem with msgpack07:53
prologicthis code is based off circuits 2.1.011:09
prologicI'm sorry :)11:09
prologicPlease upgrade to circuits 3.0.0.dev11:09
bloodymeh XD11:09
prologicI'm going to be releasing this soon11:09
prologicpip install -e hg+
bloodysaw it this weekend, that all the sudden there was 3.0.0.dev11:09
prologiccode looks okay though11:10
prologicnicely written too I must say11:10
prologicvery impressed11:10
bloodythe problem with it is that the Write() event gets somehow merged when the Server gets the Read() event11:11
prologicall good :)11:11
prologicwe have historically released very slowly11:11
prologiconce per year or so for the past 10 years11:11
prologicbut I'm going to try to change that in 3.011:11
prologicand do more regular release cycles11:11
prologicdespite the lack of resources, time and devlopers :)11:11
bloodyi upgrade the code to 3.0.0 and see if it happens again11:11
prologicplease do11:12
prologicbecause there are _lots and lots_ of improvements over 2.1.011:12
prologicor 2.x11:12
prologicthe biggest thing you have/should change11:12
bloodyyes and all the example scripts need updates :)11:12
prologicMyEvent -> myevent11:12
prologicor however you want to name it11:12
prologicclass MyEvent -> @handler("MyEvent")11:12
prologicclass my_event -> @handler("my_event")11:12
prologicwe got rid of the implicit CamelCase -> camel_case crap11:12
prologicbecause it confused new users too much11:13
prologicall the examples are all updated :)11:13
prologictrust me :)11:13
prologicI merged the dev branch into the stable branch the other day11:13
bloodyi use circuits since a week and liked the idea :D11:13
prologicso dev and stable are the same right now11:13
prologicjust still working on the actual release11:13
prologicstill a few issues on bitbucket to sort out11:13
prologicI'm glad you do :)11:13
prologicperhaps I might coax you into contributing to our development efforts :)11:14
prologicI like it too :)11:14
prologicI use it literally everywhere11:14
bloodyi can look into the example and fork them on bitbucket if you want11:14
bloodyor provide some new11:14
prologicsince a lot of things I do and write tend to be I/O bound apps with some CPU bound work -- e.g: web apps11:14
prologicall the examples should be up-to-date11:14
prologicnews ones are always welcome :)11:14
prologicupgrade your code, have another play with 3.011:15
prologicand I'll bbs shortly11:15
prologichaven't eaten dinner ;)11:15
bloody now, problem still occurs, when i user another msgpack or another serializer than pickle. Problem happens with itsdangerous too.11:41
prologicwhat's the problem exactly again?12:19
prologicyou're firing two write() events12:20
prologicand you get only a single read() event on the server?12:20
prologicbloody, I think you're trying to create another form of serializing circuits events?12:23
prologicam I right?12:23
prologicbloody, oh hang on12:24
prologicI see what you're doing here12:24
prologicand exactly what's going on12:24
prologicbuffering is occuring here12:24
prologicwhen you fire a write12:24
prologicfollowed by a subsequent write12:24
prologicit buffers these up when they get handled12:24
prologicuntil the socket is read to write12:24
prologicso the sequence will likely be12:24
prologicdata is written to socket client <-> server12:25
prologic_read() <-- server12:25
prologicread() <-- server -- both packets of data12:25
prologicI think this is what'll you end up having to do12:28
prologicunfortunately buffering is kinda necessary12:28
prologic-but- otoh12:28
prologicyou probably should just pad the packets12:28
prologicwith a separate12:28
prologicand parse them back out on the other sdie12:29
prologiccircuits.node and circuits.core.bridge does this12:29
*** zleap has joined #circuits12:30
bloodyok, looking into it12:37
bloodybut yes you got at least my problem here :)12:37
prologicwell it's not exactly a problem as such12:40
prologicbut yes12:40
prologicI think both solutions might work12:40
prologicdefinately one of them will :)12:40
prologicbut I think you could just pad the data12:41
bloodyyield works and looks cleaner than the other one12:41
prologicgreat :)12:41
prologicI thought it might work :)12:41
prologicyield self.call12:41
prologicand yield self.wait12:41
prologicare quite new to circuits12:41
bloodyi'll take a look at the bridge, node stuff12:41
prologicthey provide synchronous primitives on top of circuits message bus12:41
prologicyeah it seems like that's what you're doing anyway12:42
prologicalthough you didn't seem to be seralizing circuits events - just dummy data12:42
prologiccircuits.core.bridge is used to link processes12:42
bloodyyes was for the example here with range()12:42
prologica kind of IPC between circuits systems between parent and child12:42
prologiccircuits.node is designed (though quite immature) to be part of low-level distributed functinoliaty12:42
prologici.e: nodes12:43
prologicso if this is something you're working on12:43
prologicdistributed messages12:43
prologicplease help us make circuits.node better12:43
prologicWARNING: I have deleted the development branch in preparation for upcoming 3.0 release and a new development and release cycle (hopefully monthly) -- we'll no longer develop against a "circuits-dev" branch, but via PR(s) to
pdurbinprologic: yes, sure, make irclogger_ distributed. sounds fun12:55
prologicI think so :)12:57
prologicand what's more, it'll be easy too12:57
prologicrun separate Bot instances on unique channels12:58
prologicand fire all Log events to a "logger" channel12:58
prologicto be collected by all distributed bots/clients12:58
prologicuniquely identify them and write them out to disk12:58
prologiceasy peasy japenesy :)12:58
prologicthat in theory should solve the netsplit problem quite nicely12:59
prologicalthough I have to find out what the IP connection limit is per IP Address on Freenode12:59
prologicand possibly ask for an exception12:59
prologicso I can maintain say at least 5 connected bots working together dinstributively12:59
prologicbloody, I want to go to bed shortly13:02
prologicif you have any other q's/comments13:03
Romsteris bloody aussie?13:04
prologicno idea13:04
bloodyno not yet, testing stuff with the node modules13:04
prologicnot by his hostname :)13:04
Romsterbecasue the word bloody is really an aussie slang word13:04
prologicI'll be here for another few mins or so13:04
Romsterde denmark?13:04
prologicit can also mean bloody13:04
bloodyde germany13:04
prologicas in bloody battle :)13:04
prologicyeah de is Germany :)13:05
bloodybloody, is an old nickname i used years ago in rainbow six 313:05
prologicDenmark is .dk13:05
Romsterbloody, oath :P13:06
Romsterbeen reading the log. after darts.13:06
*** zleap has quit IRC13:19
*** eriknw has joined #circuits13:30
prologichi eriknw13:32
bloodyprologic i changed it into a node setup now, and it works with now too13:48
prologicoh bloody wonderful :013:50
prologicyou mean to say you have a working msgpack based circuits.node?13:50
bloodyyes but13:51
bloodyi show you my server output13:51
prologicwell in any case this is awesome13:52
prologicplease do keep this up :)13:52
prologicthings I want to see in circuits.node are:13:52
prologicother serialization formts13:52
prologice.g: msgpack, bson, etc13:52
prologicother transports13:52
prologiccurrently only supports tcp server<->client13:52
bloody my server output13:52
prologicbe nice if we also had udp13:53
prologicand unix socket13:53
prologicare you not filtering out the I/O events across the node?13:53
prologicyou should or that will happen :)13:53
bloodyi don't know how :( current code13:54
bloodyis this the filter keyword in @handler?13:54
prologicjust had a quick look at circuits.node's code again13:55
prologicit _is_ quite new and immature13:55
prologicneeds some work13:55
prologichave a look at circuits.core.bridge13:55
prologicit has code that filters out the I/O events across the bridge13:55
prologiccircuits.node needs to grow the same thing13:55
prologicor you'll end up with infinite loops of I/O events trying to go across the node connection over and over13:56
prologicyou'll see IgnoreEvents13:56
prologicand IgnoreChannels I think13:56
prologicjust ignore = [13:56
prologica list of events the bridge ignores13:56
prologiclong history short13:57
prologiccircuits.Bridge was written a long time ago13:57
prologicit tried to handle different transports at the time13:57
bloodywhen dinosaurs ...13:57
prologicand support process briding (i.e: IPC)13:57
prologicas well as distributed node communications13:57
prologicbut I was never very happy with the design / implemtnation13:57
prologicso tried to come up with a more flexble one in circuits.node13:58
prologicbut it's incomplete :/13:58
prologicwhen dinasoaurs :)13:58
prologicyes well circuits is over 10 years old now13:58
bloodymy goal here is not really IPC, since server / client would be in a different network13:58
bloodyor even across the internet13:58
bloodywow circuits, really is a dinosaur13:59
prologicyou want distributed comms right?13:59
prologicyeah :)13:59
prologicif you'd like to improve circuits.node13:59
bloodyremote calls like RPC would be ok too, at the moment i'm running a setup where the client pulls stuff from my local database and the server receives the data13:59
prologicI'd really like to see that happen!14:00
prologicI'd suggest you try to drag in circuits.core.Bridge14:00
prologicit's currently designed to be used as a "link" between two points14:00
prologicgiven a socket/transport14:00
prologicso some of the circuits.node event/value synchrnoization could go away and the Bridge could take care of it14:00
prologicdistributed computing is hard to design :)14:01
bloodyi know14:01
prologiccircuits.web has JSON-RPC14:01
prologicand XML-RPC14:01
prologicas well as WebSockets14:01
prologicRPC is nice14:01
prologicbut in theory14:01
prologicif circuits.node as improved to this point14:01
prologicyou could setup N-way node connections to M-points on any given no. of networks14:02
prologicand achieve the same kinds of things you could if circuits was local14:02
prologictransparently transporting the events over the nodes14:02
prologicand transparently fulfilling the sender's promised value(s)14:03
prologicthis at least works quite well for simple cases with circuits.core.bridge and circuits.node14:03
prologicthough tbh I haven't played with either much lately14:03
prologicI used to write a distributed prime number finder as a test bed / use case for circuits.core.bridge (back when circuits.node did not exist)14:04
bloodyand i havn't looked that deep into circuits to know all about that stuff :)14:04
prologicyes well :)14:04
prologicit's quite powerful and flexible14:05
prologicand easy to reason about I think14:05
prologicI'm not happy with some of the latest circuits.core.managers code14:05
prologicbut we'll have to live with that for now14:05
prologicsome of the call/wait and event completion detection stuff got a bit complicated in terms of the code14:05
RomsterO_O oh what could circuits do if say it ran over 100 nodes over the net...14:16
Romsterwork on your AI bot stuff more?14:17
bloodycircuits would turn into agent smith14:21
prologicI might have fixed this bug14:22
prologiciif circuits.node got to a point where event-driven programming was as seamless as it is locally14:23
prologicbut across a distributed cluster14:23
prologicyes I would do more R7D into AI14:23
prologicand perhaps even do my upcoming PhD on it14:23
RomsterphD in what?14:24
prologicComputer Science / Software Engineering14:25
Romsterthought you did all that stuff already14:25
prologicno :)14:26
prologicI've done an undergraduate degree in Software Engineering14:26
prologicnow I want to do a PhD14:27
prologica doctorate14:27
prologicDr. James Mills :)14:27
Romsteri didn't think their were doctors in programming.14:27
Romsterhopping into bed14:28
bloodyhm yes node stuff really needs some work, json in has problems with serialized strings14:29
bloodybye Romster14:29
prologicbloody, agreed14:31
prologicfeel free to completely rewrite circuits.node :)14:32
prologicwell keep the core concepts working14:32
prologicand value/promise sync14:32
bloodyi fork circuits on bitbucket and do my stuff14:34
prologicsweet :)14:42
prologicokay g'night all14:58
prologicbloody, welcome aboard :)14:58
prologichope we can all do lots of fun things14:58
bloodywe'll see :) solved some error bubbling by switching from json to pickle15:03
spaceoneprologic: more stuff will follow, in 4 month i will have finished my apprenticeship - then i can contribute.18:23
spaceoneprologic: question: the event-system of circuits looks a little bit like deferreds - is the idea taken from this? (the only difference is that an event has a name?)18:26
spaceoneand that deferreds can replace the value by another deferred18:27
spaceonethat would maybe be great for circuits?18:27
*** zleap has joined #circuits18:28
*** Osso has joined #circuits18:31
prologicspaceone, no, it's very much unlike deferrads19:56
prologicspaceone, the only similar thing in circuits19:56
prologicare the "feedback events"19:56
prologicsuccess, failure, complete19:56
prologiccircuits uses promises19:56
prologicx =
prologicx is a promise19:57
prologicand spaceone that's great news :)19:57
prologicit is true however that promises are chained19:57
prologicin much the same way as deferreds are19:57
prologicbut they are certainly not the same19:57
*** Osso has quit IRC19:58
prologicin contrast to Twisted for example19:59
prologicTwisted uses callbacks on deferreds20:00
prologicTwisted uses zope.interfaces to define clear interfaces between various components20:00
prologicTwisted achieve a level of loose coupling (but with a higher level of indirection) via zope.interface20:01
prologiccircuits on the other hand20:01
prologicis an event driven system comprised of events and event handlers at it's core20:01
prologicwhich provide loose coupling of components and intrefaces between components20:01
prologicbut there are no dferrads20:02
prologicjust events and promises (or their eventual values)20:02
prologicalso just trying to remember my circuits + twisted integration work at
prologicTwisted's deferrads are not tied into it's reactor20:06
prologicthey are called cooperatively afaik20:06
prologicmuch like python generator based coroutines20:06
prologic(*I think*)20:06
*** zleap has quit IRC23:05

Generated by 2.11.0 by Marius Gedminas - find it at!