IRC Logs for #circuits-dev Thursday, 2013-04-11

*** Osso has joined #circuits-dev00:07
prologicheya Osso00:41
Ossowe finally catch each other00:42
Ossohave you seen the video ?00:48
prologicwatched it this morning01:08
prologicsend Guido an email01:08
prologicmy thoughts are:01:08
prologic1) nothing to worry about01:08
prologic2) we will continue as we always have01:08
prologic3) when we drop 2.x support we'll replace some parts of circuits.core.pollers and circuits.core.timers with tulip01:09
prologicthe rest of it should remain the same01:09
prologicasync in circuits is just a set of components afterall01:09
Ossooh yeah it's not worrying01:19
Ossohe did not mention us :(01:20
Ossobut we did use the yield pretty much like he explains01:20
Ossoyou would in py201:20
prologicwe did and do :)01:24
prologicand I'm quite proud of that01:24
prologicthe only thing I think we are missing01:24
prologicis this whole notation of calling a function01:24
prologica blocking function01:24
prologicin an executor / thread pool01:24
prologicwe need to add this01:24
prologicso we can do things like (example):01:25
prologicpromise = self.apply(function)01:25
prologicyield promise.wait()01:25
prologicor something01:25
Ossobut we already have self.call01:30
Ossowhich gives a result not a promise01:30
Ossoin the case you actually want to wait01:30
prologicoh yes we do01:36
prologicbut we have no ability to specifically call a function asynchronsouly01:36
prologiconly events :)01:36
prologicwhich themselves may block01:36
prologicbut we provide yield, call and wait01:36
prologicsay you want to do in your handler01:37
prologicaddr = sock.getaddrinfo()01:37
prologicthis is a blocking call (Guido talked about this)01:37
prologicinstead we could do:01:37
prologicaddr = yield self.apply(sock.getaddr)01:38
prologicsynchronous (like call/wait) but does not block01:38
prologicand the API for apply would look like:01:38
prologicdef apply(self, f, *args, **kwargs)01:38
prologicidentical to taht of the apply builtin in Python01:38
prologicApply the function f with optional *args and **kwargs in a thread pool returning a Promise01:39
Ossothat's our futures right ?01:46
Ossoit used to run in thread pool01:46
Ossobut it's hard to get it right01:46
Ossothe worker does most of the hard work there now01:47
prologicwell I agree02:18
prologicthe Worker compnoent we have is great02:18
prologicgreat for doing CPU bound work in separate processes02:18
prologicbut we have nothing to do I/O blocking work in a thread pool02:19
prologicwe can still use Worker (which uses multiprocessing.pool.ThreadPool)02:19
prologichowever it uses a Pipe + Queue internally and serialized everything costing some performance02:19
prologicrather than just straight threadpool02:19
prologicin that case I would recommend we make circuits.core.workers.Worker multiprocessing only02:20
prologicand pull in and build a apply() API02:20
Ossouhm I see02:49
Ossoyou have a point02:49
prologicI do :)02:51
prologicand because thread inherently used shared memory02:52
prologicthey can easily do blocking I/O02:52
prologic-however- not CPU bound work02:52
prologicany CPU bound work blocks all other threads in python02:52
prologicso My proposal is to build02:52
prologicself.apply(f, *args, **kwargs) - identical to that of builtin apply and returns a Promise similar to how call/wait works right now.02:52
prologicand modify Worker to only use processes02:53
prologicthat way we have both:02:53
prologica) a way to do blocking I/O in a synchronous manner02:53
prologicb) a way to do blocking CPU work02:53
*** logix812 has joined #circuits-dev07:07
prologichey logix81207:12
prologiccaught me as I was just about to go to bed :)07:12
logix812And here I am!07:12
prologicbut I'll stay a few mins if you're free07:12
prologicbeen working on this all night07:13
logix812What's your day job have you do?07:13
prologicright now I'm working on building a climate model visualization tool07:14
prologicfor the average John Doe07:14
prologicand decision makers, etc07:14
logix812That's very cool07:14
prologicthe code I pasted above deals with clipping a raster data source by a bounding box07:14
prologicand we have to work with 5km, 1km and 250m resolution data07:15
prologicat least at 250m it only takes ~5s to clip the area I want07:15
prologiccircuits :)07:15
prologicI'll hopefully be presenting a talk at PyCon AU this year in July07:15
prologic*fingers crossed* the proposal gets accepted :)07:16
prologicahh sweet haha you've been working on this I see?07:19
prologichaving thought about this a lot over the last few weeks07:20
prologicI realized that your original coroutine style multipart parser _could_ work07:20
prologicwith just a few modifications07:20
prologicyou would have to let circuits do the coroutine07:20
prologicit would look something like:07:21
prologicresult = yield parse_multipart(request)07:21
prologicwhere parse_multi_part is:07:21
prologicdef parse_multi_part(request):07:21
prologic   while True:07:21
prologic      ... do a bit ...07:21
prologic      yield07:21
prologicor such07:21
prologicthis pattern would in theory work with the rest of circuits and continue the event handler / coroutine when the parsing has finished07:22
prologicwell I'm in two minds about CPu bound work07:22
prologicon one hand we can do coroutine style in the main process07:22
prologicor pass it off to a worker process pool (which we haven't even tried seriously with circuits.web)07:23
prologicI suspect serialization will be expensive and cost raw throughput07:23
prologicin the long run I think it's easier/better if parse_multipart was just a function07:24
prologicand if you wanted to scale just start your circuits.web app in process mode per cpu/core07:24
prologicsharing the listening socket07:24
prologictested on Linux and BSD - scales almost linearly per CPU/core07:24
logix812Well it's not a big deal to just make it a function.07:25
logix812Ok, I will make that my goal re:functionize it07:27
prologicthen we can rip out the crappier implemetnation we're using07:27
prologicmaybe your impl. might make it in time for 2.1.0 :)07:27
prologicstill have a few chores and bugs to resolve07:28
logix812ooo deadlines are good motivators for me!07:28
logix812when were you planning a 2.1.0 lock?07:28
prologicas indicated there :)07:28
prologic2.1.0 is slated for whenever we finish the remaining chores/bugs07:28
prologicaside from replacing multipart parser, it's in feature freeze07:29
prologiclooking like Map at this rate07:29
prologicuggh :)07:29
prologicwe lost a developer recently (was an ex-Java dev - went back to Java)07:29
prologicyou're not interested in contributing to circuits are you? :)07:29
prologicother than multipart parser :)07:30
prologicapparently it is :)07:32
prologicI could never07:32
prologicI would die07:32
prologicbut to each their own07:32
logix812evidently, it's now <oneperson, somewhere>07:32
prologiche did contribute A LOT to circuits07:32
prologicall the locking (mostly to help with tests) and FallBackGenerator and poller component improvements07:33
logix812I don't think I would have the time to be a full contributor. My days get pretty packed quickily and I would feel bad if I could not devote enough attention07:33
prologicworth a shot :)07:34
prologicI myself don't have much time either07:34
prologic3 month old, family, work, etc :)07:34
prologicyou know usual stuff07:34
logix8127 mo old over here07:34
prologicalrighty I'm off to bed07:38
prologicenough racking my brains for one night07:39
*** Osso has quit IRC08:08
*** logix812 has quit IRC18:26

Generated by 2.11.0 by Marius Gedminas - find it at!