IRC Logs for #circuits Monday, 2013-07-29

ninkotechprologic: ?06:45
jgiorgihe appears to be afk :(06:45
jgiorgithe great leader has wandered away lol06:46
ninkotechi really like the style of circuits. it feels very natural...06:46
jgiorgiyeah it took me some getting used to but now i hate programming any other way06:46
ninkotech... but for my project, i will prolly use rpyc, which is closer to my needs, and fast as hell..   i just need to test scalability of it in async mode, there might be some catches06:48
jgiorgiscaling anything always brings out the worst, shows the bugs you never would have imagined lol06:48
ninkotechjgiorgi: i am looking for good way for writing distributed async and synced events06:48
ninkotechto implement sort of distributed k/v store (or filesystem)06:49
jgiorgiafaik there are no synced events in circuits, the call primitive pretends to be synced but in reality it passes back to the main loop and resumes execution later06:50
ninkotechbut there is some timeout, right?06:50
ninkotechthats all i need :D06:50
jgiorgithere can be a timeout06:50
ninkotechsync in a point of time06:50
ninkotechi mean when value comes into my system, i need to spread it between 6 servers and make sure it is there06:50
ninkotechin kind of gossip protocol06:51
ninkotechif you know what i mean06:51
prologicoh hey guys06:51
prologicdidn't notice the activity going on here06:51
ninkotechnodes talking to each other06:51
prologicsorry06:51
ninkotechhi prologic, what was you pinging me about?06:51
ninkotechmy patch?06:51
jgiorgininkotech, the call or wait primitive accepts a timeout argument06:51
jgiorgiby default there is no timeout tho06:51
ninkotechjgiorgi: i love how rpyc transfers data... and how transparent it is... it feels like using python locally06:52
prologicninkotech:  yeah your patch06:53
prologicyou used the wrong repo :)06:53
prologicsorry about that06:53
ninkotechprologic: what about it?06:53
ninkotechlol06:53
ninkotechsorry06:53
ninkotechi would rather use github06:53
prologicplease use https://bitbucket.org/circuits/circuits-dev06:53
ninkotechaha06:53
prologicwell I happen to prefer mercurial06:54
prologicI find it more intuitive to use :)06:54
prologicI haven't found a way to provide git/github repos and an easy way to keep them in sync06:54
prologicsorry :)06:54
prologicyou're the 2nd person that's wanted git/github repo snow06:54
ninkotechprologic: ok, i will try resent merge request today for -dev06:55
prologicnow ninkotech  as far as your project goes06:55
prologicall you're asking is very possible06:55
prologicand really easy to do06:55
prologicwith circuits that is06:55
prologicin the simplest sense you could setup a master server that nodes connect to06:56
prologicand you just broadcast to the clients06:56
prologicusing TCPServer and TCPClient instnaces06:56
prologicor if you want full end-to-end circuits between nodes06:56
prologicuse circuits.node06:56
prologicwhere each machine/server can be connected to each other or to a central controlling node with full event message passing between them06:57
prologicall the primitive mechanisms are there06:57
ninkotechprologic: i can have no spof :)07:03
ninkotechsome example how to use .node would be nice07:04
ninkotechi didnt find much07:04
ninkotechprologic: btw i was using celery up to now to pass messages and it worked great07:04
ninkotechbut i need system without spof, nodes which are trully self-sufficient07:05
ninkotechwho talk to each other07:05
ninkotechand all process the same kind of tasks07:05
prologicspof?07:06
ninkotechsingle point of failure07:06
prologicah ic07:06
prologicsure07:06
prologicif you like I can write up a simple distributed example07:06
prologicwith no spof07:06
ninkotechprologic:  i was not able to come with running (firing) remote events07:06
prologicbut I'd like to know you're going to seriously consider circuits t hough :)07:07
prologicdon't want to spend all this time writing an example and no one uses it :)07:07
ninkotechif that is possible with nodes, i would be gratefull for any example07:07
prologicwell it is possible07:07
prologicand there is a test for it07:07
ninkotechprologic: just simple example would be nice07:07
prologicbut no example at this stage07:07
prologicI can easily write a simple example - that's not hard07:07
ninkotechi can make my own logic using simple example07:07
*** qwebirc77017 has joined #circuits07:08
ninkotechbut that example is really missing in docs07:08
*** qwebirc77017 has quit IRC07:08
prologichttps://bitbucket.org/circuits/circuits-dev/src/e5df864c7a776347d6495f83a3e88f41bce38c12/tests/node/test_node.py?at=default07:08
*** ConflictX has joined #circuits07:08
ConflictXHello :)07:08
prologicthis should be a reasonably simple example07:08
ninkotechhi ConflictX07:08
prologicif there is no example in examples/07:08
prologicthere will be a test07:08
ConflictXI'm trying to find a way to listen to Redis pubsub with Circuits :) anyoone aware of such a way?07:08
ConflictX(im looking for hours) :)07:08
ConflictXHello ninkotech  :)07:09
ConflictX&& prologic :)07:09
prologichello ConflictX07:09
prologicno-one has written Redis + circuits integration (yet)07:09
prologicI could assist you in that if you like07:09
prologicand contribute it to circuits.contrib07:09
prologicshouldn't be too hard07:09
ConflictXi'll be happy to contribute :)07:09
prologicI've not used redid myself personally07:09
prologicbut stick around and I'll help you integrate the two07:10
prologicand please please please please use the dev branch of circuits :)07:10
ninkotechredis works for me without a glitch07:10
ninkotech:D07:10
ninkotechsimple, fast, working solution07:10
prologicthe release on pypi is always out of date :(07:10
ConflictXi'll use the dev branch :)07:10
ninkotech(i store there celery results, and i do millions of them daily)07:10
ConflictXbtw, is there a company standing behind circuits?07:11
prologicno there isn't07:11
ConflictXi see07:11
prologiccircuits is 100% open sourced07:11
prologicno company backing I'm afraid :)07:11
ConflictXcool :)07:11
prologicthat almost changed when koobs and his company were thinking of hiring me and utilizing circuits for distributed computing07:12
prologicbut we haven't progressed further on those discussions of late07:12
ConflictXmaybe i can arrange something, i'll check later :)07:12
prologicalso hi ConflictX  and welcome07:12
prologicI'm always open to opportunities to further my own work(s) :)07:12
ConflictX:) where are you from prologic ? :)07:13
prologiceverything in the OSS world I do (personal and work-related) I use circuits for07:13
prologicmost of the time this means web apps though :/07:13
prologicmost companies you tend to work for don't do very interesting things :)07:13
ConflictXi just need an async, non-blocking framework07:13
prological this boring data, web oriented stuff :)07:13
ConflictXno need web :)07:13
prologicsure07:13
prologicthat's circuits in spades07:13
ConflictXyup :)07:14
prologicbut it also has a very powerful web framework07:14
prologicthat's built on top of the core circuits components07:14
ConflictXim using tornado as a webframework :)07:14
prologicsure :)07:14
ConflictX:)07:14
prologicby built on top btw07:14
prologicbecause circuits is a component architecture above all else07:14
ConflictXcool :)07:14
prologicmost things in circuits eats it's own dog food07:15
ninkotechprologic: i need also ftp server with circuits, or i will need to write one07:15
prologicthings ar built on top of simpler components07:15
prologicso circuits.web is although very powerful and quite nice, it didn't take long to write (years back) and major parts of it rewritten/repalced (recently in dev)07:15
ninkotechas i will use ftp(s) as way for some users to enter data07:15
ConflictXanyways, i need to listen to Redis pubsub channel async07:15
ConflictXim not sure how to start with it...07:15
ConflictXthe code gets blocked when i redis.listen()..07:16
prologicninkotech:  you'll have to implement a basic FTP protocol parser07:16
ConflictXand im not sure how to listen to obejct in circuites07:16
prologicthat's unfortunately one of the only things circuits lacks07:16
prologicprotocol implementations07:16
ninkotechprologic: that should not be that hard07:16
prologicso apologies in advance!07:16
prologicI'm from AU/Brisbane btw07:16
prologicno I thought you'd say that07:16
ConflictXgenerally, if i could listen to an object it will make it a really powerfull framework07:16
ninkotechaustralia or austria?07:16
ConflictXCool :) Me from Israel07:17
ConflictXninkotech? :)07:17
ninkotechwhat is au?07:17
ConflictXwhere are you from mate? :)07:17
prologicAustralia07:17
prologicAustralia/Brisbane07:17
ninkotechic now07:17
ninkotechso we are pretty much on opposite ends of the earth07:18
ninkotechyou must be sitting head down!07:18
prologicConflictX:  umm does redis and it's python bindings/library provide any kind of callback mechanisms?07:18
ninkotechlol07:18
ConflictXi dont think so :(07:18
ConflictXi can show you if you want07:18
prologicConflictX:  so it's a bit like pygame, gtk, inotify, etc07:19
prologicwhere it has it's own event loop?07:19
prologicand it's own set of events07:19
prologicor is there a way to poll redis with some kind of timeout? like a select/poll/epoll style call?07:20
ninkotechthere imho should be a way07:21
ninkotechbut i didnt really dig into that much07:21
prologicbasically you have two ways to integrate external event sources into circuits07:21
prologiccapturing the other system in a separate thread and capturing it's events and poking them into circuits07:22
prologiccircuits/io/notify.py does this07:22
prologicor hooking into some kind of polling/callback mechnaism07:22
prologicall of the pollers in circuits/core/pollers.py do this07:22
ninkotechwhy redis.listen?07:23
prologicas well as an example integrating urwid (nnurses ui library) and circuits in examples/circ.py (curses irc)07:23
ninkotechthats not in my redis library...07:23
prologicanyway chat more when I get home07:24
ninkotechsee u prologic,  and better example would be really nice to have07:24
ninkotech(than the test..)07:24
ninkotechi will read it through though07:24
prologicninkotech:  it's hard to tell what examples should be there07:25
prologicI add to them over time as users find new uses for circuits07:25
prologicthere are tests that cover almost every aspect of circuits and it's functionality across all platforms it's supported on07:26
ninkotechic, tests are a must07:26
prologicbut yeah examples are written and maintained on a use-case basis07:26
ninkotechjust cloned -dev07:26
prologicso I'll write up a simple circuits.node example for you tonight07:26
prologicand chat more then (catching bus now)07:26
ninkotech:D07:26
ninkotechi will resend merge req07:27
ninkotechty07:27
prologicI suggest you check dev before doing that :)07:27
prologicyou'll find the issues you saw are gone :)07:27
ninkotechi will :)07:27
prologiccyas soon07:27
ninkotechi am pretty sure my fascist editor will find few more pep8 issues, lol07:30
ninkotechand for  sure, also some py3 issues07:34
ninkotechi would love to go with py307:34
ConflictXninkotech, which redis lib do you use?07:34
ninkotechi dont really know :)  python-redis package in debian, pip install redis ?07:34
ConflictXyup, i use the same :)07:34
ConflictXlook:07:34
ninkotechConflictX: i was for now just user of redis (more like admin, not developer)  i just used redis via kombu framework07:36
ninkotechkombu is very lovely library for messaging via amqp or other means (includes redis, mongo, ... sql)07:36
ConflictXhttp://pastebin.com/eXKeaxrg07:37
ConflictXcheck this out07:37
ConflictXthe for loop is blocking circuits.. all i need is to listen to the channel.listen() generator07:37
ninkotechwait a moment...07:37
ConflictXok07:38
ConflictXsorry,  it should be: http://pastebin.com/ui1QT6vM07:42
prologicninkotech:  we use shiningpandi ci and regularly test against py26 py27 py32 py33 and pypy07:42
prologici doubt youll find too nany issues ;)07:42
ninkotechmy vim is very touchy...07:43
ninkotechand ninja-ide too07:43
ninkotechbtw try ninja-ide if you didnt, you might like it07:43
prologicalso i dont accept new cod or changes unless it passes flake8 ;)07:43
ninkotech:)  and has docs and tests?07:43
ninkotech:D07:44
prologici us vim with syntadtic07:44
ninkotech:)07:44
prologicdocs and tests are always improving!07:44
ninkotechare you on the bus?07:44
prologicbut yes therss plenty ;)07:44
prologicnearly07:45
ninkotechbtw ConflictX, i was in israel by car... from cz07:45
prologichad to run fadt to ger to the bus stop on timrpe!07:45
prologiclol07:45
ConflictXcool :))07:45
ninkotechiwent even to egypt a bit, and it was hot as today07:45
ninkotechcold, 55 degree celsius wind was refreshing us07:46
ninkotechit was 75+ in the car... lol07:46
ninkotechnever again!07:46
ConflictXlol :)07:46
ConflictX:)07:46
prologici also subscribe to the examples are goid docs (for showing use cases) and keep docs terse and maontainable07:46
ninkotechi was sleeping near dead sea, eaten by mosquitos and flies...07:47
ninkotechand in the morning we went swimming a bit07:47
ConflictXim in the center of IL :) its fairly hot :)07:47
ninkotechabove us, 200 meters under the sea level,  2 army jets were flying07:47
ninkotechthis is not happening often, seeing a plane to fly 200 meters under the sea07:47
ConflictXlol :) yea :)07:47
ninkotechluckily they didnt shoot at us07:48
ConflictXIsrael is nice, im moving to Canada in 10 months :)07:48
ninkotechlol07:48
ConflictXlol07:48
ConflictXnah :07:48
ConflictX:)they wont :007:48
ConflictX:)07:48
ninkotechConflictX: i was target of attack by PKK in turkey07:48
ninkotechthey tried to get us, but turkish army won that round07:48
ConflictXohh07:49
ninkotechshooting bigger caliber07:49
ConflictXsounds bad :(07:49
ninkotechwe were the only non-kurdish turists in area in 7 years07:49
prologicvtw... someone could port this to circuits https://github.com/leporo/tornado-redis/tree/master/tornadoredis08:03
ConflictXthat's an option08:06
prologicit wouldnt he very had08:07
prologicyoure just teplacing tornado with circuits08:07
prologici ports tornados  inotify to circuitts before08:08
prologicnow in circuits.io.notify08:08
ConflictXim  a bit noob im programming :) i wish i could do that08:09
prologicheh08:11
ninkotechConflictX:  did you do python tutorials?08:20
ninkotechif yes, you can do everything you need08:20
ninkotech:D08:20
ninkotechdont be scared08:20
ninkotechits like when you are bullied - show fear and it will consume you08:21
ninkotechevery day do a thing you are scared of - -and in 10 years you will do unimaginable08:21
prologiccouldnt agree more!08:22
prologicive even written my own programming lsnguage in python!08:22
ninkotechsounds fun08:23
ninkotechi have written my own language in assembly, lol08:23
ninkotechfor 65C0208:23
ninkotech6502C i mean08:24
ninkotech8bit...08:24
ninkotechand it was doing fuzzy logic and stuff, really fun08:24
prologiccant day i do much assembly these days ;)08:26
prologichalf eay home ;)08:26
prologicyou happy if i commit a simple node example tonight?08:26
ninkotechif you do, i will sure spend time playing with it08:26
prologiccool08:27
ninkotechwhat i need is this:08:29
ninkotech  one server gets some info (k/v)08:33
ninkotech   he needs to try to pass that info to other 5 servers08:33
ConflictXsorry for the delay, im at work :)08:34
ninkotech   when he passes it to 3 already, he should return 'ok'  back, finishing sync in background08:34
ConflictXi know python, and also know OOP08:34
ninkotechall computers manage some list08:34
ConflictXbut im not taleneted and experienced like you :)08:34
ninkotechand they will share info 'add x to list', remove 'y' from the list08:34
ConflictXit will take a while to convert the tornado-redis :)08:34
ninkotechand when they all know that all otehrs know to have x on list,   its 'included in the list' and syncing is done08:35
ninkotechwhen first server dies, after timeout second server needs to try syncing the change08:35
ninkotechif he does die, next one08:35
ninkotechwhen server comes back up, he needs to sync fast what he needs to know08:36
ninkotechall operations will have timestamp and should be 'synced (merged)' in time08:36
ninkotechwell i am describing it chaotically08:36
ninkotechbasicaly i need to call events on set of servers, in few rounds (to make sure its stored ok and every server knows that all other servers know the change)08:37
ninkotechand i need to make this 30-90 times per second on different groups of 6 servers taken from larger set of servers (different six servers for every k/v)08:38
prologicok08:38
prologicand what would you like to see examples of?08:39
prologicbaeing in mind i cannot do gour work ;)08:39
prologici neither have time nor are you paying me ;)08:39
ninkotech:)08:39
ninkotechremote events and some form of doing it concurently08:40
ninkotechasynchronously08:40
ninkotechwith timeout08:40
prologicok thats easy08:40
prologicyoull have it soon!08:40
ninkotech:D08:40
ninkotechand maybe - i need a way to move larger data blob08:41
ninkotechlike 10-200MB or so08:41
ninkotechi need an efficient way of doing that08:41
prologicid just do it over the same event stream08:41
ninkotechi am not sure how still - just calling it with parameter containing 200MB?08:41
prologicas a spevial event08:42
ninkotechthat would be sweet08:42
prologicno http or ftp overhead08:42
ninkotechif it will work08:42
ninkotechthats what i liked on rpyc08:42
ninkotechit just works very fast -- calling data on remote function is easy as pie08:42
prologicyeah itll work fine08:42
prologicid split up katger chunks though08:42
prologicand do cgunking08:42
ninkotechyes. chunking is ok08:43
prologicor jusy use circuits.web for this ;)08:43
ninkotech(50MB chunk?)  i need to move up to 10GB08:43
ninkotechand my problem is, i need to move it on multiple targets at once :D08:43
ninkotechthinking about ipv6 multicast08:43
ninkotechbut i am not sure how to do that, really08:44
ninkotechand if it would work nicely08:44
ninkotech(under load)08:44
koobsprologic; almost :)08:44
prologicheh08:44
koobsprologic; we had the hiring rug pulled out from under us, sorry for not getting back to you on it08:44
koobsprologic; it wasnt 100% clear at our end either08:45
ConflictXmmm, im trying to figure a way to listen to an object....08:48
ConflictXany object..08:49
ConflictXmaybe i'll make a decorator which puts itin a thread but i guess it will miss the point of using the framework..08:49
ConflictXwhat say you guys?08:50
ConflictXbyw, Hi koobs :)08:50
ninkotechConflictX: looking at it08:53
prologicok well im gling to have dinner and relaxnfor a ehile08:53
prologicthen ill wriye this example08:53
ConflictXthanks ninkotech  :)08:54
ninkotechConflictX: channel.listen() is blocking.  it is generator yielding messages08:55
ninkotechu can find it in /usr/lib/python2.7/dist-packages/redis/client.py08:55
ninkotechdo you really want to listen to channel?08:55
koobsConflictX; hi08:56
ninkotechhttp://stackoverflow.com/questions/7871526/is-non-blocking-redis-pubsub-possible08:56
ninkotechread both top answers08:57
ninkotechso you will need a thread to take care of redis it seems08:57
ninkotechor maybe prologic will think somthing up for you08:58
ninkotechi am still novice in circuits08:58
ninkotechprologic:   i also need to make my remote rpc call 'secure' somehow. thats still open question for me09:00
ConflictXmaybe circuites can provide a decorator to thraead manage sync thinks to make things more simple09:09
ConflictX  i can put my own thread inside the component but i think it will cause alot of problems..09:09
ConflictXmmm09:09
ninkotechprologic: testing -dev version in p3.3 -- examples: circ.py is not ready for 3.309:15
ninkotechechoserver is not echoing back...09:16
ConflictXI have managed to do that in a very ugly way, but it works, im making a sample code :)09:23
ninkotechhello_bridge.py  is running /usr/bin/python -i -- which is not my python interpreter in my virtualenv .... works with python -i hello_bridge.py09:24
ninkotechprologic: chatserver is broken in py3 (str buffer)09:25
ConflictXhttp://pastebin.com/U5ktgtyp09:27
ConflictXit's working async :)09:27
ConflictXbut i dont think it's a clean solution becuase i never did thread programming and im not sure whats the upcomming problems for that09:28
ninkotechConflictX: if you early in the morning eat a big frog, there could be nothing worse during whole day... :D09:28
ConflictX:)09:29
ninkotechthats what i try to do...09:29
ninkotechin work, in relations with others, and with myself09:29
ninkotechfrog = something you are afraid of09:29
ConflictXme too :)09:30
ConflictXthough  i have no idea if it's a good solution :)09:30
ninkotechyou will be top python hacker soon :D09:30
ConflictXwhat do you think about that?09:30
ConflictXbahh :)09:30
ConflictXnahh* :)09:30
ninkotechi will let prologic evaluate it :D09:31
ninkotechit could work09:31
ninkotechthreading always is pain09:31
ninkotechbut it seems that you cant do much else09:31
ninkotechConflictX: you should prolly not only log the text, but use a queue to share it with other threads09:31
ninkotechor some circuits trick09:32
ninkotechqueues are the way to share between threads09:32
ninkotechyou can check if queue is empty or something is in09:32
ninkotech-> efficient work09:32
ConflictXlike ZMQ :)09:33
ninkotechi had much pain with zmq in saltstack09:33
ninkotech:D09:33
ninkotechso i hope it will work better09:33
ninkotechlol09:33
ConflictX:)09:34
ConflictXi cant SIGTERM with that code running09:34
ninkotechnow you know why people hate threads09:34
ninkotechlol09:34
ninkotechit means extra work to do, and extra problems09:35
ninkotechyou need to make sure to end all threads when ending the app09:35
ConflictXmaybe we can simplify with a generator09:35
ninkotechqueue, believe me09:36
ninkotechyou can share queue between threads09:36
ConflictXok09:38
ConflictXi'll check it out :)09:38
ninkotechConflictX:  check this: http://stackoverflow.com/questions/3102163/why-does-using-threading-event-result-in-sigterm-not-being-caught09:45
ConflictXmmm10:20
ConflictXinteresting10:20
ninkotechhttp://www.kamaelia.org/Home.html <- this is also inspiring lib11:08
prologichmm11:28
prologicmissed anything?11:28
prologicninkotech, example is done12:05
*** ircnotifier has quit IRC12:06
*** ircnotifier has joined #circuits12:07
ConflictXhey prologic  :)12:12
*** ircnotifier has quit IRC12:15
*** ircnotifier has joined #circuits12:16
ircnotifier84805a8d87ff by prologic: Code cleanup12:17
ircnotifier9f86093580bd by prologic: Code cleanup12:20
ircnotifiera2f83d776c70 by prologic: Code cleanup12:23
ninkotechre12:45
ninkotechprologic: ?12:45
ninkotechah, ic12:45
ninkotechhow do i pull from your tree into my clone of my fork?12:46
ninkotech:D12:46
ninkotechi guess i will download your branch too12:47
prologichg pull https://bitbucket.org/prologic/circuits-dev12:48
prologiceasy12:48
ninkotechbtw #!/usr/bin/python -i    is not playing nice with virtualenv12:48
ninkotech(where i have different python12:48
ninkotech)12:48
prologicno it does not :)12:50
ninkotechbtw hello node is not executable12:50
prologicI know of no way of doing something like12:50
prologic#!/usr/bin/env python -i12:51
prologicenv doesn't support passing in options to the application afaik12:51
ircnotifier3620603d916e by prologic: Make hello_node.py executable12:52
ninkotechprologic: i tried it too, failed12:52
ninkotechbad luck12:52
prologicyeah12:53
prologicstuck with python -i blah12:53
prologic:)12:53
ninkotechprologic: i might be dumb, but i still do not get the example fully12:54
prologicwell12:54
ninkotechand btw i think your node unittest is wrong - it does too many tests at once12:54
prologicthat's why I coped hello_bridge.py mostly12:54
prologicessentially node and bridge are almost the same in behavior12:54
ninkotechyou should make separate tests for many parts -- one to test each thing12:54
prologicjust one works with processes12:54
prologicand the other works with hosts12:54
ninkotechaha12:55
prologicthey each allow events to be fired remotely or to processes12:55
prologicand receive the value back when done12:55
ninkotechwait... i will play with it for few minutes12:55
prologicall asynchornously12:55
ninkotechhopefully i will get it12:55
ninkotechby reading only its not good12:55
ninkotech:D12:55
prologicone of the reasons the code in circuits/node/*.py is all poorly documented12:55
prologicis because I've been trying to think of ways to make it:12:55
prologica) support different types of transports12:56
prologicand12:56
prologicb) support different types of serialization12:56
prologicbut I haven't really had time to do this  yet12:56
ninkotechif i get into it, i might help you a bit12:56
prologicthat would be sweet12:56
ninkotech:)12:56
ninkotechi know how hard such job is...12:56
prologicit would be rather cool as well to add support for using zmq as a transport12:56
prologicand msgpack as a serialization12:56
ninkotechmsgpack, json  are hot12:57
prologicjust have to code up the abstractiosn12:57
ninkotechand really, rpyc has some own serializer which is really cool!12:57
prologicso transport and serialization are abstract enough to swap them out with other types12:57
ninkotechyou should plug in Serializer12:57
ninkotechso user can choose12:57
ninkotechor write own12:57
prologicwell12:58
ninkotechi would be happy with anything fast which is not pickle12:58
prologicif said code is good enough to reuse12:58
prologicwe'll reuse it12:58
prologicin my experience however a lot of code is not very reusable12:58
prologicsadly12:58
ninkotechmongo has very nice format12:58
prologichowever I did manage to find a lot of code I could ureuse in circuits.web12:58
ninkotechwhich got own library out of mongo system12:58
ninkotechbson <- check it out12:58
ninkotechjson works well, but not on datetimes12:58
prologicyeah12:58
prologicfor now it's basic12:59
ninkotechyaml is generally good, but its not as easy as promised by hand12:59
prologicbut it works and works quite well12:59
ninkotechpickle = security hole12:59
prologiccircuits.node should get you quite far12:59
prologicpickle is not portable12:59
prologicso I chose not to use it12:59
ninkotechxml = sucks, but works12:59
prologicit's only used by the Bridge12:59
prologicfor process-based ipc12:59
ninkotechso, serialization is not yet done?12:59
prologicand Bridge uses a UNIX pipe13:00
prologicfor fast full duplex comms13:00
prologicno it's done13:00
prologiccircuits.node uses json13:00
prologicand tcp for transport13:00
ninkotechok13:00
ninkotechhow to make the communication secure? are there more options?13:00
prologicit's api is quite simple13:00
prologicumm good question13:01
ninkotechssl might be slow13:01
ninkotech(but still might be good for me enough)13:01
ninkotechsome hmac or otr stuff might be nice, or generic message signing13:01
ninkotechi am not quite sure how to make this best13:02
ninkotechfast and secure13:02
ninkotechfast is important, i will make many small connectins13:02
prologichmm13:02
prologicit's not hard to add this13:02
prologicbut it's late and there's not enough time for me to do it properly with tests13:02
prologicbut you could attempt it and send me a pull request13:03
prologichave a look at circuits/web/servers.py13:03
prologicyou just need to modify circuits/node/server.py in a similar way13:03
prologicso you can pass secure, certilf kwargs13:03
prologicand possibly some modifications to circuits/node/client.py13:03
prologicand the Node.add api13:03
ninkotechi think you would be better for this, i am still novice at circuits :D13:04
ninkotechi can wait a few days13:04
prologicok13:04
prologicmy recommendation if you're going to serioualy use circuits for this13:04
prologicimplement your prototyoe first13:04
prologicthen we'll worry about making things faster13:04
prologicoptimze later13:04
ninkotechprototype? what do you mean?13:04
prologicthings that will help make things faster13:04
prologicis if we have a real use-case to work with13:05
ninkotechi will first need to simulate load13:05
prologicand then you and I can perhaps work on abstracting the code in circuits.node so we can swap out transports and serializations13:05
ninkotechand see how it behaves13:05
ninkotechhow much it can transfer, how many updates per second13:05
ninkotechetc13:05
ninkotechwhen it starts to fail delivery13:05
ConflictXprologic, say, did you check my "async" redis solution? http://pastebin.com/U5ktgtyp13:05
ninkotechafter this round i will go directly for simple way to make my unittests pass13:06
ninkotech:D13:06
ninkotechnext round -> security13:06
ConflictXis there more elegant way to do so?13:06
prologicConflictX, lemme see13:06
ConflictXok :) thanks :)13:06
ninkotechConflictX: it tastes sweet when the frog is gone... lol13:06
prologichmm13:07
prologicdoes Channel.listen block?13:07
ninkotechyes13:07
prologicor does it iterate?13:07
ninkotechit yields13:07
prologicdoes it return a generator?13:07
ninkotechbut it blocks13:07
ninkotechits generator13:07
prologicit blocks until an event is avialable?13:07
ninkotechthe method yields coming events13:07
ninkotechyes13:07
prologicok let me ask again13:07
prologicdoes it block indefinately?13:08
prologicor yield None if there's nothing13:08
ninkotechindefinitely13:08
prologiccrap13:08
ConflictXyup, indefinatly13:08
ninkotechyes!13:08
prologicthen yeah13:08
prologicyou're implementation is fine13:08
ConflictXcrap13:08
ninkotechcrazy code, blocking...13:08
prologiccopy the circuits/io/notify.py example13:08
prologicmore or less13:08
prologicbut so far so good on your code13:08
ninkotechbut it can use gevent patch  to make it nonblocking13:09
ninkotechi have heard13:09
prologicyeah no don't worry about gevent patching13:09
ConflictXI dont want to mix too much frameworks :)13:09
prologicjust thread it13:09
prologicit's fine13:09
ConflictXbtw, i'm going to test it for commercial use :)13:09
ninkotechi 'love' threads :)13:09
prologicthreading blocking calls is a prefectly good solution13:09
prologicthen just interate ofer Channel.listen13:09
prologicand inject the events into circuits13:09
ninkotechprologic: i advised him to put coming messages into a queue, which can be shared between threads safely13:10
prologichmm13:10
ninkotechand tested for empty/get message13:10
prologiccircuits is thread safe13:10
prologicit's fine to just fire the events13:10
ninkotechgood idea13:10
prologicbe sure to wrap the data in some nice circuits Event13:10
ninkotechevents are in queues, right? :D13:11
prologicsomething like13:11
prologictry:13:11
ninkotechinstead of try i am now in love with contextmanager stuff13:11
prologic   x = next(self._redis_channel)13:11
prologic   e = RedisEvent(...)13:11
prologic   self.fire(e)13:11
ninkotech(whenever applicable)13:12
prologicyes circuits uses a heapq13:12
prologicso you can also fire high priority events13:12
ninkotechok, i will go test that node stuff13:12
prologicunfortunately the logic is kinda unintuative at the moment13:12
prologicwe may try to reverse and fix this later on13:12
ninkotechbtw instead of python -i   i use ipython :)13:15
ninkotechipython = lovely13:15
prologicsure:)13:16
prologicI've never tried ipython tbh13:16
*** koobs has quit IRC13:18
ninkotechyou need to. its so sweet... i am using it as my main shell for years now13:21
*** koobs has joined #circuits13:21
ninkotech(main shell for my apps)13:22
ninkotechits power at your fingertips, without a mouse...lol13:22
prologicheh13:22
prologicwell I'm off to bed13:22
prologicgood luck!13:22
ninkotechsee you tomorrow13:22
ninkotechbtw some more node docs  would be lovely13:22
ninkotechi will write you tomorrow13:23
prologicyes they would be :)13:23
prologicbut I feel circuits.node needs more work13:23
prologiclook at it's code size13:23
prologicit's nothing :)13:23
prologic294LoC13:23
prologicI want transport and serialziation abstractions before I'm happy with it :)13:24
ninkotechwhat troubles me is that i do not immediatelly understand what is for what there...13:24
ninkotechbut hopefully i will catch the ball13:24
ninkotech:)13:24
prologicwhat are you missing?13:24
ninkotechi dont really know, there is too many words which have unexplained meaning13:24
prologicsuch as?13:25
ninkotechwaht is channel? waht is callEvent?13:25
ninkotechhow differs calling it on app1 and node1 ?13:25
ninkotechhandles, handlers13:25
ninkotechetc13:25
prologicahh13:25
prologicdid you read the tutorial?13:25
prologicit explains most of these13:26
ninkotechyesterday i did...13:26
ninkotech:)13:26
ninkotechbut i forgot most13:26
ninkotechlol13:26
prologicand still not understanding?13:26
prologicoh :013:26
prologicok so quickly13:26
ninkotechits hot here13:26
ninkotech36 degreees C13:26
prologicchannels are used as namespaces - separating components that may fire identically named events13:26
prologici.e: running two ircbots in a single process13:26
prologicchannels avoid event collisions13:26
ninkotechsounds good13:26
prologican event handler - is exactly that13:27
ninkotechyour calls are not really explained here:  y = app1.fire(Remote(Hello(), "app2"))13:27
prologicit is a specially defined method on a component that handles the event it names13:27
ninkotech:D13:27
prologicwhich you can define implicitiy13:27
prologicor explicitiy13:27
ninkotechok sounds good13:27
ninkotechgo to sleep i will read docs again13:27
ninkotech:D13:27
ninkotechhave good night13:27
prologicy = app1.fire(Remote(Hello(), "app2"))13:28
prologicfire a Remote event13:28
prologiccontaining a Hello event inside (wrapped)13:28
prologicto the "app2" node13:28
prologicand assign it's value to y (when received)13:28
ninkotechhow does it know where to find node app2?13:28
prologiccircuits implements promises13:28
ninkotechits a string here!13:28
prologicas opposed to callbacks/errbacks (twisted)13:28
prologicfrom the previous call to13:28
prologicnode1.add("app2", *bind)13:29
prologicyou "name" nodes13:29
prologicmaybe there's a better way in future improvements to circuits.node :)13:29
prologicI dunno13:29
ninkotechi need to spend some time with it instead of chatting13:29
ninkotechdont worry13:29
ninkotech:) sleep well13:29
prologic:)13:29
prologicalso evertyhing are strings in circuits really13:30
prologiccomponents don't know about each other by object references13:30
prologicthat would kind of defeat the point of "loose coupling"13:30
prologicyou fire and forget mostly13:30
ninkotechso i will make few nodes, with apps - fire events and forget nodes/apps?13:31
ninkotechlike with few nodes():13:31
ninkotech      do stuff13:31
ninkotech?13:31
ConflictXnn  :)13:31
prologichttp://shortcircuit.net.au/~prologic/blog/article/2013/02/26/writing-a-simple-irc-logger-bot-and-daemon/13:31
ninkotechah, i will rather read :)13:31
prologicI wrote this blog entry a while back13:31
ninkotechthis is making me feel dumb :D13:31
prologicwalking you through writnig a simple app in circuits13:31
prologicworth a read13:31
ninkotechreading!13:32
ninkotechthanks13:32
ninkotechbtw  multicast over ipv6 -- that might be really funny13:32
ninkotech:D13:32
prologicquite possibly13:33
prologicyes :)13:33
ConflictXprologic, plz read the private :)13:33
ninkotechi would love to have such transport option13:33
ConflictXi think it will make you feel nice :)13:33
prologicnight all13:34
ninkotechgood night!13:34
ConflictX:)13:35
ConflictXi need to learn about iterators...13:35
ninkotechsearch stackoverflow.com  with python iterators13:36
ConflictXi dont like to use try, except in case its not findind an element in a list..13:36
ninkotechsometimes you must close a file handle or somthing, and 'finally' part is really good friend13:36
ConflictXi got a list of dicts, every dict is diffetent, and i need to find specific dict...13:37
ConflictXnothing like xpath for dicts? :)13:37
ninkotechhehe13:37
ninkotech[x for x in list_of_dicts if 'something' in x]13:38
ConflictX:)13:38
ConflictXit will fail if the elemt 'something' not exist :)13:38
ninkotech(if you are searching for key)13:38
ninkotechno this will work13:38
ConflictXyup, im looking for a key13:38
ninkotechthis will give you list of dicts having the key13:38
ConflictXbut it's not the same keys in each dictionary13:38
ninkotechthats ok, try it13:38
ConflictX[{a: 2}, {b: 3}]13:39
ConflictXfinf b :)13:39
ConflictXfind*13:39
ninkotechlod = [{a: 2}, {b: 3}]13:39
ninkotech[x for x in lod if 'a' in x]13:39
ninkotech^^ returns [{'a':2}]13:39
ConflictXmmm13:39
ninkotechyou forgot ' around a, b13:40
ConflictXchecking :)13:40
ninkotechk='a'13:40
ninkotech[x[k] for x in lod if k in x]13:40
ninkotechthis vill return list of values13:40
ninkotech[2]13:40
ninkotechi just love python lists13:41
ninkotech:D13:41
ninkotechother way to not need try:13:41
ConflictX:))13:41
ninkotechvalue=mydictionary.get(k,None)13:41
ninkotechmight just work for you13:42
ninkotech(gives default value of k is not there)13:42
ninkotechif* k is not there13:42
*** ConflictX has quit IRC19:01
*** jgiorgi-home has joined #circuits19:32
*** jgiorgi has quit IRC19:33
*** Osso has joined #circuits19:47
*** Osso has quit IRC20:50
*** jgiorgi has joined #circuits21:04
prologicmorning all21:40
jgiorgimorning23:01
prologicsorry didn't get a chance to do that php stuff23:19
prologicwe had a visitor over for dinner23:19
prologicI'll do it tonight23:19
jgiorgiphp stuff? pgp?23:23
jgiorgilol i would never ask you to touch php :_23:23
jgiorgi:)23:23
prologicpgp23:31
prologicsorry :)23:31

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!