IRC Logs for #circuits-dev Monday, 2013-02-04

*** Osso has joined #circuits-dev00:39
prologicOsso, hey :)02:52
prologicCheck this out:02:52
prologichttp://code.activestate.com/recipes/578447-attempt-to-estimate-a-size-of-python-object/02:52
mehereLooked at it, just don't quite get your point, though02:53
prologicahh02:53
prologicok02:53
prologicfirst I wanted you to at least have a look02:53
prologicnext my point02:53
prologicI think we could use this somehow in some way02:53
prologicto optimze memory usage of circuits02:53
prologicWHat do you think?02:53
prologiceg: optimize the memory of event objects, etc02:53
mehereAlthough estimates from pondering are always wrong when optimizing performance and memory usage, I'd first of all remove the inheritance relationship between Component and Manager. By making every Component a potential manager (but actually having only one, the root) components get far too many attributes.02:58
mehereAren't there any clever tools for getting statistics about memory usage for python?02:58
mehereThere are for Java...02:58
mehereLots, actually.02:58
OssoI see03:00
mehereSeems to be a common question http://stackoverflow.com/questions/110259/python-memory-profiler03:00
OssoI can see the point in splitting the manager and the components03:01
OssoI think it is quite obvious we should optimise events size03:02
mehereNot necessarily, events have short lifetime, so who really cares?03:03
Ossoyou can see it in terms of throughput03:03
Ossoif they are twice as small you get to copy them twice as fast03:04
mehereSmaller objects being handled faster by Python than larger ones?03:04
mehereWhere do we copy events?03:04
Ossofrom a purely cpu processing point of view03:05
Ossoyou are limited by the time it takes to fill up the memory with stuff03:05
mehereBut this aims at performance, not memory usage, right?03:06
Ossoyeah03:06
OssoI don't think we have enough events in the queue to take up that much space03:06
prologicwell03:07
prologicmemory usage and performance go hand in hand really03:07
prologicif we optimze the memory footprint of events03:07
prologicthings will get faster03:08
prologicI'm not too overyly concerned about the footprint of components though03:08
prologicreason being is that it's convenient to be able to just "run" a component03:08
prologicwithout having to worry about the boiler plate of a Schedule object03:08
prologicand adding componnets to it, etc03:08
prologicOsso, well that's not strictly true03:09
prologicthink of a high concurrent web app03:09
prologicserving up thousands of requests03:09
prologicat any time there could be hundreds if not thousands of events in the queue03:09
prologicso memory usage will spike during high load03:09
prologic1/4 or having this would be good03:09
prologicihmo03:09
Ossoeven if we have 200003:10
Ossoif we have a 2KB overhead03:10
Ossoit's like 4MB?03:11
mehereThe *real* problem with trying to build a high performance web server with circuits is that Python cannot make use of more than one CPU.03:12
prologicwell we can start multiple processes though03:12
prologicand share a listening socket03:12
prologicin the past we've had circuits.web bench at 7k req/s at a concurrency of 100003:13
prologicwould need to re-bench this to see if we'd made any difference of late'03:13
Ossoin the case of a webapp03:15
Ossowouldn't you have a reverse proxy in front already ?03:15
mehereHonestly, I'd never do such a multi-process design if I could avoid it. Think about the resources that you cannot share between process (e.g. file cache).03:15
prologicyeah ofc03:18
prologicreverse proxying is the preferred way i think03:18
prologicregardless of amy real applications, could we not benefit from small optimizations in evnts objevts?03:19
mehereFrom what I have seen when profiling the bench, I'd say no. But maybe running this with http://packages.python.org/line_profiler/ would provide more details.03:24
prologicwell anyway im off to bed ;)03:28
prologichave a nice day guts!03:28
prologichmm07:19
prologic1.30am feed07:19
prologic:)07:19
prologichey guys07:19
prologicwhat are you both using circuits for again?07:19
prologicdo you want to add yourself to the docs/source/Users.rst page?07:19
*** Osso has quit IRC07:45
prologicmehere: ping?16:29
prologicHow do you guys feel about implementing the observer pattern for circuits?21:53
prologicUse case (have wanted it a few times):21:53
prologicReference some value that might get changed by an event or some other process21:53
prologicand automatically have it updated21:53
prologicWe'd use Python Data Descriptors for convenience and the nice implicitness it gives you21:54
prologicAlso 2nd topic (circuits.web and uploading files)21:54
prologicQ: Do we support streamed uploading of files right now?21:54
prologicI don't think we do. If not we should probably think about doing this.21:54
*** anunnaki has joined #circuits-dev23:44
prologichi an23:50
prologichi anunnaki23:50
anunnakihello23:55

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!