IRC Logs for #circuits-dev Thursday, 2013-01-10

*** Osso has joined #circuits-dev00:53
prologicThu Jan 10 18:51:1600:54
prologic$ circuits.bench -e 100000000:54
prologicSetting up Speed Test...00:54
prologicSetting up Sender...00:54
prologicSetting up Receiver...00:54
prologicTotal Events: 1000006 (264453/s after 3.78s)00:54
prologicpypy is fast!00:55
prologicwe agree we need to optimize the locking that occurs?01:00
prologicor that's needed rather01:00
prologicI think mehere's original implementation was the correct one01:00
prologicjust needs to be optimized a bit01:01
OssoI'll take some time today to profile just adding a simple lock in fire01:08
OssoTotal Events: 1968110 (45352/s after 43.40s)01:32
OssoTotal Events: 818317 (41576/s after 19.68s)01:39
Ossowith lock01:39
Ossohum I am getting variable result01:41
OssoTotal Events: 3097996 (45153/s after 68.61s)01:41
Ossostill with lock01:41
Ossorevision 3272 is01:45
OssoTotal Events: 1560569 (41382/s after 37.71s)01:45
OssoTotal Events: 1687354 (41480/s after 40.68s)01:45
meherePlease note that in my implementation, the lock in _fire is only established if the event isn't from the same thread -- very rarely.01:46
Ossoyeah we are arguing all the time about the lock but it does change the benchmark01:47
Ossosomething else was making 3272 slow01:47
Ossoit does not*01:47
Ossoand slower*01:48
mehereAnd think of the advantage: the lock made it work reliably ... ;-)01:48
Ossoanyone wants to dig what else we optimised ?01:56
prologicwell now that you ask :)02:01
prologicthe whole, self.wait stuff could be optimized02:02
prologicwe add an overhead of an additinoal function call per call02:02
prologicjust so we get the root component02:02
prologicinstead we could do a bit more here02:02
prologicwhen we register a component02:02
prologicwe could dynamically bind .fire .wait and .call methods to the child component02:02
prologicthat point to the root02:02
prologicit would reduce a whole layer of function calls02:03
Ossothat's true02:15
OssoI can see a few elements02:20
Ossothe removal of ticks02:20
prologicI thought we already removed that?02:26
prologicI swear we have02:26
prologicI don't see ticks anywhere in tip02:26
OssoI mean02:26
OssoI was comparing the 3272 speed difference02:26
Ossowe removed ticks, the check for the FallbackGenerator02:26
Ossobut there's still a speed difference02:27
Ossowhen removing those 2 in 327202:27
prologicbecause locking is expensive?02:29
prologicor simply function call overheads?02:29
Ossonope not locking either I removed that too02:29
OssoI removed the call to reduce_time_left too02:30
Ossowhy are eating exeptions from tick() now (at the end of
prologicI added that I believe02:54
prologicto prevent some stupid None exceptions thrown during shutdown02:54
prologicno idea how you're suppose to solve that02:55
prologicso have you pushed these changes yet?02:55
Ossooh no I can't push those02:56
prologicstill experimenting?02:56
OssoI took 3272 and I ripped off the stuff to make it fast02:57
prologicbut still at 0 cpu cycles when idle?02:57
Ossonot meant to work safely just to benchmark02:57
prologici see02:57
prologicgotcha :02:57
prologicon another topic - worker components02:58
prologicstill playing with this obviously02:58
Ossosee what is the optimisation we did that improved speed02:58
prologiclittle unsure of things at present02:58
prologicsince rewriting Worker to be a wrapper around multiprocessing.Process02:58
Ossoto see if we can just revert my changes or we just add back a lock on top02:58
prologicit works really well except that @future doesn't work anymore02:58
prologicspeaking to some colleagues at work and doing some experiments with threads and threadpools02:59
prologicwe found that (obvious) python does not do preempptive multirehading02:59
prologicso doing cpu-bound work in any thread is a stupid idea02:59
prologicwhich leads me to believe that Worker component should do "work" in a sub process02:59
prologicbut how to solve @future not working03:00
prologicserialization problems apparently03:00
Ossocertainly but threading is still useful for wrapping synchronous stuff03:00
Ossoso a Worker with threads makes sense03:00
prologicyes whilst I realize that03:00
prologicwe can still do that obviously03:00
prologicbut it's not good for doing "work"03:01
Ossogoing to eat, bb in 1 hour03:01
Ossoyeah I agree03:01
prologicwhich is what Worker was always designed to do (or at least it was supposed to)03:01
prologicbut I think it's implementation wasn't quite there03:01
prologicI see what you're saying ...03:01
prologiccomment when you're back03:02
prologicI understand what you're saying to mean that we should combine Threading and Processing into Worker03:02
prologicallowing @future to work as expected - but only with a Threaded Worker03:02
prologicI think this could work03:02
prologiclet me know what you think03:02
prologicand then Pool could just manage multiple instances of Worker03:03
prologicThe API would/should not change either03:03
prologicwould just hopefully be a better implementation and actually with working multiprocessing03:03
*** Osso has quit IRC04:17
*** Osso has joined #circuits-dev04:44
Ossoit would be nice to have future work with a different process too04:45
Ossocurrently it seems that it is the same exact handling for process and threads in both worker and future04:50
Ossoso it is good to keep them together04:50
prologicI don't think I can easily make @future work in a process04:52
prologicbecause of pickling issues04:52
prologicI just pushed some chnages to Worker04:53
prologicbut I can't make it sleep properly04:53
Ossocan the process worker rely on circuits.node to handle the pickling ?04:59
prologicI don't think it's a problem pickling the event object or value05:01
prologicbut rather the method being executed05:01
prologicThe only solution I can think of05:01
prologicis to grok the component graph for any handlers marked with @future(process=True)05:02
prologicand start their components in a separate process05:02
prologicor something05:02
prologicor just figure out what's not picling05:02
Ossoin theory it should be simple05:03
Ossono ?05:03
prologicin theory :)05:04
OssoI mean we have our event05:04
prologicbut I can't even make this Worker component work proeprly05:04
Ossowe have our worker for the event05:04
prologicwithout chewing up 100% cpu05:04
prologicwhile yielding for a result05:04
Ossoit's not like we need all the events to go through like node05:05
Ossowe just need to bind to our one event05:05
prologichtis baffles me a bit right now05:10
prologicmaybe because it's getting late05:10
prologicthere are two issues I'm finding hard to solve05:11
prologica) keeping results in order for the right value05:11
Ossoall the code is on dev ?05:11
prologic2) allowing Task to be fired multiple time whilst 1)05:11
prologic3) keeping cpu usage down when waiting for a result to come back05:11
prologicit is05:11
prologicbasically there's circuits/core/workers.py05:12
prologicand in the top-level05:12
Ossoso you have 4 problems a) 2) and 3) ?05:12
prologicboth thread and process mode work05:12
prologicreally two05:12
prologichandling multiple Task events and keeping results in order for the right value05:12
prologicAnd Keeping CPU usage down while waiting for resutls05:13
Ossokeeping CPU usage down seems easy05:13
prologicyou'd think so :)05:13
prologicbut apparently I haven't quite gotten my head around how to do it properly05:13
prologicI did try :)05:13
prologicthe whole needs_resume stuff05:13
prologicsetup a way to interrupt the Queue.get(...) calls05:13
Ossowhere this Queue handled ?05:16
OssoI don't see it in worker05:16
Ossooh wait05:16
OssoI was in the wrong revision05:17
OssoI see how it works05:23
OssoI think multiprocessing is actually acting against us for futures05:24
prologicwhy's that?05:25
Ossoif we just fork05:25
Ossothe child runs the function05:25
Ossothere's nothing to pickle nothing to wait for05:25
Ossoin the child at least05:25
prologichmm I see what you're saying05:26
Ossono maybe it is just how we are using process05:27
Ossoif we build the target on the fly05:27
Ossodef target(): f(*args, **kwargs)05:27
Ossoyou don't need self.queue05:28
prologichowever you are spawning a new Thread/Process each time you handle a Task event05:28
prologicthat can get rather expensive05:28
Ossodamn you are right05:29
Ossothat is not a problem for future because it only has one function05:32
Ossobut for a generic worker it is05:32
OssoDid you check apply_async ?05:34
Ossofrom Pool()05:34
Ossoit cuts down a bit of our code but it'll behave exactly like your code05:37
OssoI think I can solve your order problem05:39
Ossoactually how does your future work ?05:40
Ossoyou removed the inside05:40
Ossoso it is yielding before it gets the result05:40
prologicjust discovered it's trying to pickle the damn component where the @future is defined05:41
prologicthat's a bug05:41
prologicas part of it's args05:41
prologicignore @future I think for now05:42
Ossoremoved in b0559e01f46005:42
prologicit's Worker that doesn't quite work quite right05:42
prologicI didn't mean to remove that05:43
prologicmy bad05:43
Ossowell if there's a problem with worker then05:46
Ossowe can use apply_async05:46
Ossoit'll most likely fix the bug05:47
Ossowant me to try ?05:47
*** mehere has joined #circuits-dev05:47
prologicOsso, please05:47
Ossook should be quite fast05:48
Ossocan you test this commit 7bdd4b3130dd2cf84e3f437b85955a9d86fdf4f3 ?05:57
prologicstil same behavior as what I had?06:03
prologicorder is not maintained06:03
prologicand still high cpu06:04
Ossoyes yes06:04
Ossobut it is not pickling the component anymore06:04
prologicyeah I know :)06:05
prologicbut that's because I fook'd something up in circuits/core/ a while back I think06:05
prologicI was passing self as an argument06:05
prologicno idea why06:05
Ossonow my question is06:06
Ossowhere do you get the wrong order ?06:06
prologicso am I to think that you're thinking Worker should just start up a threadpool or processpool upon init?06:06
prologicand just processing Task events?06:06
Ossoyes I think so06:06
prologicI apologize06:08
prologicyou're apply_async with multiprocessing Pool06:08
prologicworks perfectly06:08
prologicjust have to solve the 100% cpu problem06:08
Ossothe 100% is because of tasks06:09
Ossothe kind of manager._tasks06:09
prologicthat's better06:14
prologicwe let it sleep between tasks?06:14
prologicor between ticks if we have tasks06:15
Ossobetween ticks06:15
prologicok so I like how you've wrapped multiprocessing.Pool there06:16
prologicwhat about a thread pool?06:16
Ossois there one ?06:17
prologicnot one like multiprocessing.Pool's API that I can fine06:18
Ossoit is a bit shady but exists06:20
prologicor we could just not worry about multithreading?06:20
prologicit's not preemptyive in python anyway06:20
prologichold on06:22
prologiccan't we just use that?06:22
prologicand use the same api?06:22
prologicyou're brilliant :)06:25
prologicI was just donig that06:25
prologicawesome stuff06:26
prologicwe should merge this into default06:26
prologiccan you?06:26
prologicI think I just did06:28
Ossook I let you push then06:31
OssoI closed the other 2 heads06:31
prologiccleaned it up06:32
prologicI love this :)06:32
Ossowe can use the same approach for our pool class06:33
prologicwell actually06:33
prologicwe don't need one anymore06:33
prologichave a look at what I've done06:33
Ossothat's true06:35
prologicand test_worker_process06:35
prologicboth pass06:35
prologicnice :)06:35
prologictest_future doesn't though :/06:35
prologiclooking into it06:35
prologicwe still have issues with @future06:36
Ossomaybe pool.close() or pool.join()?06:39
prologicwhat's that got to do with it?06:40
prologicI'm finding that it's event trying to pickle things when using ThreadPool06:40
prologicfutures aren't working at all now06:40
prologicbut Worker works nicely06:41
Ossoah ok06:43
prologicI have to go to bed06:43
prologicbut I do very much like the new Worker Component design06:43
prologicsame API - better design06:44
prologicI wouldn't mind at some point also adding Map event as well translating that into map_async call06:44
Ossogn !06:47
*** Osso has quit IRC07:42
prologicMust have been thinking this while dreaming in sleep last night12:54
prologicBut if we can't make @future work with process workers12:54
prologicLet's just remove @future altogether12:55
prologicAllowing the user to be more explicit about what they're doing12:55
prologicThey can achieve the same functionality or similar by breaking their event handler up into small functional bits12:55
prologicUtilizing, ...))12:55
prologicas well as self.wait and/or self.call12:56

Generated by 2.11.0 by Marius Gedminas - find it at!