IRC Logs for #circuits Monday, 2013-12-16

*** ircnotifier has quit IRC04:00
*** ircnotifier has joined #circuits04:00
*** SX has joined #circuits07:44
SXhi guys! may i ask if circuits is suitable for async tcp <-> serial (with multiple tcp connection and opened serial ports) server?07:53
prologicSX, hi08:36
prologicSX, in short - yes08:36
*** Osso has quit IRC08:49
SXcould you please tell me09:10
SXwhy my simple event handler with yield and print from collection works so slow?09:11
SXfor data in self.gen:09:11
SX            yield print(data)09:11
SXself.gen = (i for i in range(0, 500))09:12
prologicwhat are you doing?09:14
SXderived from component, made start handler, where i call fire(PrintEvent())09:18
SXdef print_event1(self):09:18
SX    for data in self.gen:09:18
SX        yield print(data)09:18
SXor is it wrong to use yield in handlers?09:22
prologicyeah i don't think using yield dierctly is supported09:23
prologicyou need to use call/wait instead09:23
prologicwhat you're in fact doing there09:23
prologicis causing the event loop to pauase09:23
prologicI mean I don't really understand what you're doing per say really09:24
prologicseems like you're doing async I/O?09:24
prologicthere's for this09:24
prologicfrom import stdout09:24"HelloWorld"), stdout)09:24
prologicbut your above sample needs to be:09:24
prologicfor data in self.gen:09:24
prologic    yield
prologicbut I still question why :)09:25
prologicit's not natural to do this in an event handler09:25
SXjust doing some experiments, thats all =)09:26
prologicall that's alright then :)09:26
prologicexperimenta are always okay :)09:26
prologicbut yeah09:26
prologiccircuits provides 3 basic primitives09:26
prologiccall, wait and fire09:27
prologicfire - fire and forget the given event09:27
prologiccall - synchronously call the given event09:27
prologicwait - synchronous wait for the given named event09:27
prologicNB: it's all still asynchronouns underneath09:27
prologiccall/wait create lightweight coroutines with the circuits event loop called tasks09:27
SXso if i use call it would pause my function till event got success status?09:32
prologicyes (more or less)09:36
prologicuntil the event was complete09:36
prologicit's internal to circuits09:36
SXmay I ask one more question? how to use 'wait'?10:04
prologicyou may ask as many as you want :)10:06
prologicso wait works by waiting for a named event10:06
prologicx =
prologicyield self.wait("foo")10:06
prologicno problems :)10:10
prologicbtw are you using circuits-dev?10:10
prologicI assume you are :)10:10
prologiccall/wait don't exist in 2.1.0 :)10:10
SXthat's strange10:14
SXpip's telling me i got ver. 2.1.010:14
prologicinstall dev :)10:14
prologicwe release very slowly10:14
prologicbut that's about to change with the upcoming 3.010:15
prologicpip install -e hg+
prologicpip uninstall circuits10:15
SXbut waitEvent does work10:15
prologicmaybe we did add it in 2.1 :)10:16
prologicI forget :)10:16
prologicall good10:16
prologicbut I recommend you use dev anyway10:16
*** SimpleX91 has joined #circuits10:38
*** SX has quit IRC10:38
*** koobs has quit IRC11:45
prologicSimpleX91, you SX or were?12:33
prologicahh cool13:20
prologicso what you going to use circuits for?13:20
*** litzomatic has joined #circuits13:20
prologicor just evaluating?13:20
prologichello litzomatic13:20
litzomaticHello :)13:20
litzomaticI'm looking for more docs on workers, is there any?13:20
SimpleX91async tcp <-> serial (with multiple tcp connection and opened serial ports) server13:21
prologicWorker(s) just wrap multiprocessing13:21
prologicI could write some more docs if you really need them :)13:22
prologicdocstring docs okay?13:22
prologicSimpleX91, ahh nice :)13:22
prologicSimpleX91, you the bloke from SO? :)13:22
prologicI hope you're using
litzomaticJust interested in the architecture and how they work with the Pool.13:22
prologiclet me have a look at the code again :)13:23
prologicit's pretty well documented from an API perspective already13:24
prologicwhat do you want to know specifically?13:24
litzomaticJust trying to gain a high level understanding of how it works.  Perhaps there is some code that uses it.13:25
*** SimpleX91 has quit IRC13:26
prologicokay I'll have to develop some examples that uses Worker then13:27
prologicI'll get on to that, but it won't happen by tonight :) it's quite late!13:27
litzomaticah its using multiprocessing.Pool13:27
prologicbut a high level overview of circuits I can give13:27
prologicyes :)13:27
prologicand using apply_async on the pool13:27
litzomaticprobably can just read the python docs :)13:27
prologicbasically taking a task() event13:27
prologicand throwing it as the pool13:27
prologicand passing the result (when it's completed) back to the circuits message bus13:28
prologicthus all working seamlessly with the rest of your circuits ap13:28
litzomaticYeah, I'm thinking if I had a long lived and multi step thing a web handler needed to do (i.e. a checkout)13:29
prologicthis is absolutely the typical use case for soemthing like this :)13:29
litzomaticI'd want to do it with a worker instead of coding in an event based way13:29
prologicto wrap potentially CPU-bound oepratins13:29
prologicso as to not tie up the main even tloop13:29
prologicyou would basically do something like:13:29
prologicsay you had a request handler such as:13:30
prologicclass Payment(Controller):13:30
prologic   def POST(self, *args, **kwargs):13:30
litzomaticWill I have to spin up a new process each time or is there a way to keep a process "alive"13:30
litzomatici.e. I'm guessing with a pool13:30
prologic        yield, *args, **kwargs), "worker")13:30
prologicit starts the pool automatically for you13:30
prologicand the pool is kept around indefinately13:31
prologicI think it starts up the pool on started() event13:31
prologicwhen your app/system starts13:31
prologichrmm nope13:31
prologicstarts up the pool when you create a Worker instnace :)13:31
prologicshould probably change that to listen to the started() event13:32
prologicohw ell :)13:32
prologicit won't change the API at all13:32
litzomatichmm how does this line work: self.workers = workers or (cpu_count() if process else DEFAULT_WORKERS)13:32
litzomaticI'm wonder how I configurae the number of processes13:33
litzomaticand is there only one pool for an application?13:33
prologicyes there is only one pool13:33
prologicit only makes sense to have one pool13:33
prologic(I think)13:33
prologicfor a process pool of 4 processes13:33
prologicso basically this wraps either a process pool13:34
prologicor a thread pool13:34
prologicusing the multiprocessing library as an abstraction point13:34
prologicso you can do:13:34
litzomaticwell I can make up a situation where I'd want a pool and queue perhaps..13:34
prologicw = Worker(process=False, workers=4)13:35
prologicwhich will start up a thread pool of 4 threads13:35
litzomaticsay the payment API only wants us to process at most 4 payments simultaneously.13:35
prologicwhich is actually the default13:35
prologicI think the default should be process=True13:35
prologic*sigh* :)13:35
prologicyeah it's quite configurable13:36
prologicbut you'd made me unsure of the defaults now13:36
prologicactually now that I think about it13:36
litzomaticso it uses cpu_count?13:36
prologicthere is no reason whatsoever you coudn't start another Worker instance13:36
prologicand create a 2nd pool13:36
prologicyou would just do:13:36
prologicw2 = Worker(channel="my_other_worker_pool")13:36
prologicso it will use cpu_count()13:37
prologiciif process=True13:37
prologicand workers is None13:37
prologicotherwise it will use workers13:37
litzomaticyeah, just so I could limit the number of connection per application server.13:37
litzomaticwith the number of processes that are allowed.13:37
litzomaticSay I want 10 max connection per application server to a database.13:37
prologicjust pass workers=413:37
litzomaticwhat if I want a process that manages it's own pool?13:38
litzomatichmm, well say session in sqlalchemy13:39
prologicI think what you're asking13:39
prologicis a separate child process13:39
litzomaticit manages its own pool13:39
prologiclinked to the parent via some kind of message/event bridge13:39
prologicand the child process has it's own worker pool13:40
litzomaticyeah a worker that is just a process, no pool.13:40
prologicoh just a single process?13:40
prologicwe used to have a wrapper around multiprocessing.Process a while ago13:40
prologicbut we removed it13:40
prologicwe found no use-case for it13:40
prologicbut you could just start a component or part of your systme as a process13:41
prologic(FOo() + Bar()).start(process=True)13:41
litzomaticwell the architectur3e would be event loop -> processmanager -> pool of processes13:41
prologicdon't see why you coudln't architect it that way13:41
prologiccircuits is just a nice framework that provides you tools to help you loosely decouple your application13:42
prologicinto reusable components13:42
prologicwith loose dependneices on each other13:42
prologicand gives you great inversion of control and injection points13:42
prologicplus it also has a very powerful message bus13:42
prologicso do whatever you want :)13:42
*** litzomatic has quit IRC13:45
*** litzomatic has joined #circuits13:47
litzomaticThe only downside compared to CherryPy I'm seeing is the implementation details are more complex to understand.  Not that CherryPy didn't make up a lot of hard to understand architecture in places... Circuits seems to have better architecture, but tougher implementation :)13:48
prologicwhy do you say that?13:49
litzomaticCherryPy's implementation details are pretty straight forward when I read through the code.  This is a bit more difficult, probably because I do not understand event loops as well as concepts like threads.13:51
prologicwell that's the thing13:52
prologicit's a completely different way of doing things13:52
litzomaticCherryPy did add some interesting abstractions, that added complexity though, like the Proxy classes to serving.13:52
prologicthreaded programming is argulely just as hard as async programming13:52
prologicbecuase you have to worry about semaphores, locks, etc13:52
prologicwhereas async is quite hard too13:52
prologicbecause you have to worry about events and flow13:52
prologichowever it's always been my opinino that async is far better than threading13:53
prologicthreading is (at least in Python) utterely useless13:53
prologicexcept for wrapping blocking I/O that doesn't have support for async13:53
prologicalso I do not understand locks, sempahores, etc13:53
prologicand it's really really hard to get right13:53
prologicit takes much an expert to be good at threaded programming13:53
prologicso in circuits13:53
prologiceverything is a message/event13:53
prologicand all components communicate with one another13:54
prologicdoing a part of the whole13:54
prologicthe motiviating princinple behind circuits is this:13:54
prologicWhen you see an airplane in the sky what do you see?13:54
prologicSome say: "I see a plane"13:54
prologicWhilst my late Professor Geoff Dromey once said:13:54
prologic"I don't see a plane! I see a millions components flying in coordination!"13:55
litzomaticYeah, I think you guys are doing a good job of the async abstractions, just got to wrap my head around async implementation.  Like what is a BasePoller?13:55
prologicyou needn't worry yourself too much about things13:56
prologicwhat are you getting confused about?13:56
prologicusing circuits in an app13:56
prologicor circuits' internals?13:56
prologicBasePoller is just typical OOP class stuff13:57
prologicwhere we define a common base for all I/O polers13:57
prologicit's never directly used and should never be13:57
litzomaticYeah, engineers see the pieces because they understand them :)13:57
prologicSelect, Poll, EPoll, KQueue are the pollers to use13:57
prologicalso it's worth noting (if I didn't before)13:57
litzomaticIt's my vice :)  My question mainly is what the heck is a poller at all :)13:57
prologiccircuits as a bonus _has_ Async I/O components13:58
prologicit isn't it's key feature however13:58
prologicremmeber that :)13:58
prologiccircuits can take in many many event sources13:58
prologicwe used to have drivers/pollers for gtk, pygame13:58
prologicwhich I'll try to bring back at some point13:58
prologicand we have drivers/pollers for pyinotify13:58
prologicand pyserial13:58
litzomaticEvent loops are implemented using polling?13:58
prologicand there's an example in examples/circ.py13:58
prologicwhich wraps urwid13:58
prologicand creates a very simple curses irc licent13:59
prologicand eats urwid's events13:59
prologicthere is only one event loop13:59
prologicit's implemented in circuits/core/manager.py13:59
prologicpollers just participate in polling for events from event sources13:59
prologicthe main event loop sends out a generate_events() event13:59
prologicevery now and again13:59
prologicasking anyone participating in event generation14:00
prologicto get some new events and return them14:00
prologicif not stfu and go to sleep :)14:00
prologicthe simplest way to understand a poller14:00
prologicis to look at
prologicbad example14:01
prologicthat reuses a real poller :)14:01
prologichang on14:01
prologichave a look at circuits/core/timers.py14:01
prologicthat is the essence of a poller14:01
prologicor event source14:01
prologica timer is also an event source/poller14:01
litzomaticwhat am I looking for :) ?14:05
prologicthe generate_events handler14:05
litzomaticSo if I am understanding right.14:07
litzomaticThe event loop calls "generate_events" every so often asking if the timer is done?14:07
prologicpretty much14:08
prologicbut it's a little smarter than this14:09
prologicthe main event loop sleep until there is something to do14:09
prologicso that it eats very little cpu14:09
prologicbut still responds almost instnatly14:09
prologicso for example if you had a look at my irclogger bot's source code14:09
prologicI setup a Timer() instance there14:10
prologicthat triggers once per day14:10
prologicusing a datetime as the time interval14:10
litzomatichmm, how does it know there is something to do :)14:10
prologicthe main event loop would in fact wake up on eactly that time14:10
prologicbut since it's also bssy doing other things14:10
prologiclike being connected to irc14:10
prologicand logging stuff14:10
prologicit's awake a lot of the time anway14:10
prologicgenerate_events() tells it there is something to do14:10
prologicif no component responds14:11
prologicthen it defaults to it's fallback14:11
prologicwhich is to sleep for a very very long time14:11
prologicunless something changes to break this14:11
prologici.e: fire was called14:11
litzomaticso does the timer return to the event loop the amount of time left or just when the timer is done?14:12
litzomaticI think this helps me get a start on gaining a deeper grasp on event loops.14:14
litzomaticand the current helping architecture in circuits.  I'm liking what I see so far.14:14
litzomaticThe real test for me once, I get some stuff working will be, can I implement my business logic easily :)14:14
prologicthe amount of time left14:17
prologicit modifes the generate_event() event that it's sent14:17
prologicall participants modify it at some point14:17
prologicdepending on whether they want to reduce the amount of time left14:17
litzomaticThanks, again, I'm sure I will be back with more questions.14:17
prologicor reset it to 014:17
prologicoh you're off? :)14:18
prologicI think you'll be pleasantly surprised tb :)14:18
prologicyou'll find writing large complex apps quite easy in circuits14:18
prologicif you follow the circuits way :)14:19
prologiccomponents, loose coupling14:19
litzomaticYeah, you've given me more then enough direction for now :) got some understanding to gain.14:19
*** litzomatic has quit IRC14:23
*** LeuX_ has joined #circuits15:13
*** LeuX has quit IRC15:14
*** realz has quit IRC15:14
*** realz has joined #circuits15:15
*** realz has quit IRC15:15
*** realz has joined #circuits15:15
*** An_Ony_M1ose has joined #circuits15:49
*** space has joined #circuits15:51
*** prologic_ has joined #circuits15:51
*** prologic_ has quit IRC15:51
*** prologic_ has joined #circuits15:51
*** An_Ony_Moose has quit IRC15:54
*** prologic has quit IRC15:57
*** spaceone has quit IRC15:57
*** pdurbin has quit IRC16:18
*** prologic has joined #circuits16:21
*** pdurbin has joined #circuits16:30
*** kanycma has joined #circuits17:19
*** kanycma has quit IRC17:19
*** mscout has joined #circuits18:49
*** mscout has quit IRC18:49
*** Osso has joined #circuits19:11

Generated by 2.11.0 by Marius Gedminas - find it at!