IRC Logs for #circuits Saturday, 2014-03-29

*** robert_ has quit IRC01:37
*** qwebirc19651 has joined #circuits06:46
qwebirc19651_o/06:47
qwebirc19651Hello, is there any good tutorial on circuits ? I need to use a thead pool with for the manager using start() but still the program blocks as soon as the first blocking call is reached.06:49
qwebirc19651I would have expected to have other threads from the pool to continue to work on other events (like a timer or somthing) but it doens't work.06:51
qwebirc19651can somebody help ?06:51
prologicoh hello06:54
prologicdidn't see you there :)06:54
prologicqwebirc19651, can you explain your problem a little more?06:55
prologiccircuits has a Worker component06:55
prologicwhich can either act as a process pool06:55
prologicor thread pool06:55
prologicbut it may not work the way you traditionally think06:55
prologicif you are trying to asynchronously call functions that would otherwise block the event loop06:56
prologicthen yes you should setup a Worker thread pool06:56
prologicand fire task() events at it06:56
ircnotifier32a07eb97c99 by prologic: Added a simple little dummy DNS Client Example that performs simple lookups given any query. Great integration example with dnslib.06:57
*** koobs has quit IRC07:03
qwebirc19651Hi thank you for the answer.07:08
qwebirc19651What I was wondering was about the start() function of the manager component.. I was assuming it was building some kind of a thread pool to process events in parallel but I am not sure anymore.07:10
prologicnoooo!07:11
prologic:007:11
prologic:)07:11
prologicit simply starts the system/app07:11
prologicin a separate thread07:11
prologicwhich is useful for _some_ things07:11
prologicbut not very useful in general07:11
qwebirc19651 ok ok07:11
prologicyou need to have a very good reason to call .start()07:11
qwebirc19651that explains my mistake .07:11
prologice.g: using and playing with circuits interactively on the python shell07:11
prologicso that you can still play and interact on the shell07:12
prologicand discover what circuits is like :)07:12
prologicit can also start the system/app in a subprocess07:12
qwebirc19651ok so the sole option is to use task and worker pool then..07:12
prologicby .start(process=True)07:12
prologicso yeah07:12
prologicnothing to do with threadpools and executing "events" in a thread pool07:12
prologiccircuits is event-driven by design07:12
prologicwell I'm still interested to know more about your so-called "blocking functions"07:13
prologicif you are doing any kind of block I/O07:13
prologicyou really should write a poller for it07:13
prologicif that's impossible you'll have to wrap it up in a thread07:13
qwebirc19651I was reading from a posix queue07:13
prologicor take advantage of the Worker component07:13
prologicwhich can either act as a thread pool or process pool07:13
prologicby firing task() events at it07:13
prologicdefine posix queue?07:14
prologiclike a unix pipe?07:14
prologicoh and btw07:14
prologicare you using circuits 3.0?07:14
prologicpip install circuits==3.0.0.dev07:14
prologicPlease do not use 2.1.007:14
qwebirc19651well I just intalled it yesterday I need to check which version it is ... one moment I have a look07:15
ircnotifier16750b80621d by prologic: Check dns lookup failure against EAI_NODATA too as this can certainly happen on some platforms07:15
prologicalso on a side note07:16
prologicwe'd encourage you to use a real irc client :)07:16
prologicthere are many for all platforms :)07:16
prologicand I'm James btw :)07:16
*** koobs has joined #circuits07:17
qwebirc19651ok it appears to be circuits 2.1.0  ...07:17
prologicyeah remove that and upgrade :)07:18
prologicdid you use pip?07:18
qwebirc19651Oh thank you very much James. and yes I used pip07:18
prologiclong story short, we've historically only released once per year or so over the past 10 years of development due to lack of resources, time, etc07:18
prologicand 3.x has been recommended for over a year now and quite stable as we have over 270 unit tests with something like 80-90% coverage07:19
prologicthe upcoming 3.0 (very soon) we will try to rollout more regular releases07:19
prologicprobably once a month07:19
prologicso if you use pip07:19
prologicpip install circuits==3.0.0.dev07:19
qwebirc19651Thank you  again I will do that , I will install an IRC cilent,  and will come back. you have been very helpfull.07:21
prologichey nps :)07:21
prologicwe're here to help07:21
prologicand ensure you enjoy using circuits07:21
prologic:)07:21
qwebirc19651btw the for reading the posix queue I iam using  :  posix_ipc07:22
prologicahh ic07:24
prologicumm07:24
prologicwhy? :)07:24
prologicafaik they're not very asynchrnoousy :/07:24
prologicnot sure you can even poll them07:24
prologicare you trying to communicate between processes in python?07:25
prologiccircuits has builtin IPC comms07:25
prologichttps://pypi.python.org/pypi/posix_ipc07:25
prologicso you're using this I guess?07:25
qwebirc19651yes,07:26
prologichttp://semanchuk.com/philip/posix_ipc/#message_queue07:26
prologicyeah :/07:26
qwebirc19651well, to make a short story, I am not that familiar with python ... I am a C++ dev and normally I woul have use Boost ASIO07:26
prologicyou're going to have to wrap this in a thread if you want to use it in circuits07:26
prologicit has no poll/async support07:27
prologicBoost ASIO hmm07:27
prologicoh ic07:27
prologicin that case forget about posix_ipc07:27
qwebirc19651yes that was my plan, to use a thread ( one from the pool)07:27
prologicjust use circuits as-is07:28
prologicit has a lot of I/O and low-level networking components built right in07:28
qwebirc19651ok I understand07:28
prologicit all depends on the kinds of tasks you're trying to solve07:28
prologicif you are trying to do CPU-bound work in Python07:28
prologicit's best to use processes07:28
prologic(threads suck in Python)07:28
prologicso use the circuits Worker() component07:28
prologicotherwise07:28
prologiccircuits is a full Asynchornous I/O event-driven framework07:28
qwebirc19651well here again , I was thing to python to prototype( for the time being)07:29
prologicwith a very nice component architecture to help deal with separation of concenrs, etc07:29
prologicwell obviously developing in Python is nothing alike C++ :)07:29
qwebirc19651I wasn't sure I could get that much from circuits.. but with what I learned now I will give it another  go.. deeper.07:30
prologicPython is of course quite capable of production quality stuff - you don't have to resort to just using it for prototyping :)07:30
prologicyou'll be (hopefully) pleasantly surprised really07:30
prologicwhat docs have you been reading so far?07:30
prologicDocs: https://circuits.readthedocs.org/07:31
qwebirc19651yes I start feeling it for some time now.07:31
prologicCode: https://bitbucket.org/circuits/circuits07:31
qwebirc19651Thank you again see you07:32
prologicjust reading more on the Boost ASIO docs07:32
prologiccircuits can do all of what it does at the surface07:32
prologicand probably a hell of a lot more tbh07:32
qwebirc19651ok I understand. thks,07:34
*** leuben has joined #circuits08:04
*** leuben has quit IRC08:05
*** leuben has joined #circuits08:06
prologichi leuben08:35
leubenhi08:37
leubenI connected earlier regarding thread pool08:38
leubenI was qwebirc1965108:40
prologiccool08:40
prologichi :)08:40
prologicand welcome again08:40
leubenWell,  I had  a look at the doc again, and  ( I don't know if this is because I am not familiar enough with python) I feel like a little tuto around task and worker will not do any harm. ;-)08:43
prologicthis is true08:44
prologicthere isn't a lot of docs around Worker and task event(s)08:44
prologicbut to be honest there really isn't a lot to document08:44
prologicbecause it simply wraps Python's std. lib multiprocessing library08:44
prologicyou use to do either CPU bound work08:44
leubenah ic08:44
prologicor potentially blocking I/O that you cannot turn into proper async I/O08:45
prologicthe important bits of circuits are obviously all well documented (we hope)08:45
prologicwith loads of examples08:45
prologicas I mentioned to robert_ a couple of dayas ago08:45
prologiccircuits' API looks something like this:08:45
prologicfire()08:45
prologiccall()08:45
prologicwait()08:45
prologicstart()08:46
prologicstop()08:46
prologicregister()08:46
prologicunregister()08:46
prologicand that's about it08:46
prologicand you construct your appilcation/system in a loosely coupled way by using Component08:46
prologicand define your events using Event08:46
prologicoh I forgot run()08:46
prologic:)08:46
prologicd'uh08:46
leubenthat is true,  but for the time being I wanted to keep the "not so async" posix queue.08:47
leubenthough I clearly see I could do it better using all the cicuits features. but I need to go in two step regarding it.08:49
prologicwell08:51
prologicobviously you can use posix_ipc08:51
prologicbut I'm not sure why you're using it tbh08:51
prologicif you do use it08:51
prologicwrap it in a Thread08:51
prologicfrom threading import Thread08:51
prologicor fire task() events at a Worker as I mentioned before08:51
leubenyes I am trying. I just need to spend a bit more time around python  ways to do things...08:56
leubenI will come back when I feel more confident if I still have questions.08:57
leubenthank you08:57
leubenbye09:05
*** qwebirc19651 has quit IRC09:05
*** leuben has left #circuits ("Leaving")09:05
ircnotifierbedd899b248c by prologic: Improve tests.tools.test_tools so it doesn't randomly fail09:24
*** circuits_github has joined #circuits09:25
circuits_github[circuits] therealprologic pushed 6 new commits to master: http://git.io/tqSg_w09:25
circuits_githubcircuits/master 6983048 James Mills: Added a simple little dummy DNS Server Example that responds with an A record given any query of any type. Great integration example with dnslib09:25
circuits_githubcircuits/master b639e94 James Mills: Removed sync Fabric Task. We'll find another way to synchronize the Github mirror. Perhaps a hook?09:25
circuits_githubcircuits/master 62104ff James Mills: Updated: tox 1.7.0 -> 1.7.1, Fabric 1.8.2 -> 1.8.309:25
*** circuits_github has left #circuits ()09:25
*** irclogger has joined #circuits14:58

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!