IRC Logs for #circuits-dev Monday, 2013-01-21

prologicmehere, thinking about the problem a bit more00:07
prologiccan we put locks around changing the structure of a system from other threads?00:07
prologicsort of like a gil of gils00:07
mehereThe problem with performance is that I forgot to reset _cache_needs_refresh, which causes handlers to be recalculated on every dispatch. BUT if00:10
mehereI fix this, udp tests fail randomly again.00:10
mehereSo there is still some problem left with the concurrent change of component structure done by udp tests.00:11
mehereI look at it further.00:11
mehereMaybe it is also the stop(). Called from another thread, it calls tick() and then _fire() which means00:12
meherethat two threads may be running _dispatch for the same component tree concurrently.00:12
mehere_dispatch has never been designed for that, however,00:13
mehereI'll head for another way to do stop() first. Two threads on _dispatch() is really ugly.00:14
prologicthanks for the effort btw00:14
prologicthe project needs a multithreading guru like you :)00:15
mehereWell, by now I have invested so much time in my project built on top of circuits that I cannot drop it ;-)00:16
prologicahh :)00:16
prologicwell that's a good thing I guess00:16
prologicall my projects also use circuits00:17
prologicand I may use it at work as well in some tools00:17
prologicyou're familiar with circuits.bench right?00:17
prologiccircutis.bench -e 10000000:17
prologicto guage performance00:17
mehereI like the event driven approach. I'm still pondering about its difficulties and why people aren't adapting it on a larger scale...00:18
mehereYes, I know about circuits.bench. Actually I have started00:18
meherelooking into performance as well.00:19
mehereI use it in combination with kcachegrind00:19
meherewhich shows you very nicely where time is spent. Can I send you a picture?00:19
mehereThere seems to be no codepad for pictures. Well have a look at
mehereIt shows how time is spent in tick *after* fixing the currently existing bug with _cache_needs_refresh. But I'm not going to check in the fix until I get tests running again.00:30
prologicyeah I like the event-driven approach as well00:33
prologicwhich works nicely with async00:33
prologicand concurrency00:33
prologicI believe programmers find it difficult and there is little adoption00:33
prologicbecause it's a hard paradigm to get your head around00:34
prologicthe whole concept of loose coupling and message passing is hard to cope with at a large scale for "some"00:34
prologicbut if you treat everything as a small unit of work (a Component)00:34
prologicit really isn't that hard00:34
*** Osso has joined #circuits-dev01:32
*** Osso has quit IRC01:34
*** Osso has joined #circuits-dev01:36
prologichey Osso02:19
prologic - Asynchronous Functional Decorators02:19
prologic - JavaScript/CSS Minifier Component02:19
prologicWhat do you think?02:19
Ossofor me ?02:21
OssoI think a javascript or css minified would require an external lib02:24
Ossoso one more dependency02:24
mehereI'd like to keep circuits small and based on standard python. More sophisticated web components should go to some extension pack. And my experience is, once you start with JavaScript, you begin to pull in small parts from libraries, and eventually there's the basic choice to go for jQuery or some other framework. And you small parts don't go well with the chosen big library and everything becomes rather ugly. So if you want a web02:41
mehere toolkit, make the "base library" decision first and add components, but do it in some extension pack to circuits.02:41
prologicok agreed :)02:45
prologicok back03:20
prologicI've got a few hours before I feed Alice and go to bed03:21
prologicperhaps we should concentrate on apis and functionality that make extensions easier to work with?03:21
prologickind of what I've been doing with .handles(*names) and .handlers()03:22
prologicwhich work on both instances and classes of components03:22
prologicI think the only thing we need now (ihmo) is a good plugin/extension loading mechanism03:22
mehereWhat's the use case?03:24
prologicso you can pragmatically load components given a valid entrypoint and package03:28
prologicand determine if it's the component with the "right interface" you want03:28
mehereWhile, in general, I agree that dynamic loading is a feature, why would I want to write a circuits based application that finds out about the components that it needs at runtime *and* cannot rely on the python interpreter to find them. Probably haven't understood what you are aiming at...03:31
prologicand I'm really glad you asked about the use case03:31
prologicbecause I just tried a simple example03:31
prologicAnd it totally doesn't work as I'd expect it to03:32
prologiccertainly not for subclassed components03:32
prologicas is the case for TCPServer03:32
prologic  |-TCPServer03:32
prologicso that sucks :/03:32
prologicMaybe it's a case of; the programmer should know what they're doing?03:33
prologicAnd we already have dynamic loading anyway via python import statement03:34
prologicand who cares what the interface is - the programmer should have known this at design/implementation time?03:34
mehereAs far as I can see, the problem is that you haven't got a poller yet. The poller should be created automatically, when a socket is registered. I can see the code for both Client and Server. So it's probably a bug. Could you make a test case from that?03:40
prologichang on03:40
prologicyou've lost me there - slight tangent?03:41
prologicwhat are we talking about now?03:41
mehereWell the problem is03:41
prologicI thought we were talking about dynamic loading extensions/plugins and a way to query component interfaces03:41
mehere'NoneType' object has no attribute 'discard'03:41
meherewhen you access _poller, right?03:41
prologicyes ok that's new03:42
mehereYeah, but what else does your example show except for an error?03:42
prologicI didn't notice that :)03:42
prologicI actually meant it showed a lack of usefulness in the server.handlers() call03:42
prologicwhich didn't return a list of inherited handlers03:42
prologicbecause TCpServer subclasses Server03:42
prologicthat error btw might have been a boo boo on my part03:43
prologicI might have screwed something up in the setup03:43
prologicI think I know what's going on here03:45
prologicfor some reason I/we removed the automatic poller registration on startup03:45
prologicgah that was stupid03:45
mehereIt should get created automatically03:45
meherewhen you register a socket.03:46
mehereDon't need it on startup03:46
meheremaybe the application doesn't need a poller at all03:46
mehereBTW just pushed the fix for the performance problem with better handling of stop() (avoiding concurrent access to _dispatcher()), Please try out.03:47
prologicso for when you get back03:48
prologicwe should locate and/or create a poller during init?03:48
prologicas opposed to during registration or startup?03:48
prologic -- Example03:49
prologic - Diff03:49
prologicThat works.03:49
*** Osso has quit IRC04:10
mehereThe patch is OK. The reason why you cannot create the poller in __init__ is that you don't have a connection to the component hierarchy yet. So you cannot check whether there already is a poller (and There can only be one).04:36
mehereWhat do those file list show me?04:36
prologicjust muchking around with loc counts04:37
prologicloc of circuits itself04:37
prologicand loc of the test suite04:37
prologicok with the fix above?04:37
mehereYeah, fix it.04:37
prologicdo you have an example that uses your new websockets component btw?04:40
prologiccode looks interesting04:40
prologicin theory I think you've implemented it such that it becomes almost like a real socket right?04:40
mehereYour new test case fails here with global name "'CERT_NONE' is not defined", maybe because my python 2.6 doesn't have SSL compiled in.04:42
mehereI use WebSockets in my Portal
mehere(currently only in github HEAD)04:44
prologicmehere, can you paste me the error?04:44
prologicmaybe I need to do a pytest.importorfail(...)04:44
mehere============================= ERRORS =============================04:45
mehereFile "/home/mnl/devel/circuits-dev/circuits/net/", line 51604:45
mehereNameError: global name 'CERT_NONE' is not defined04:45
meheredef test_secure_server():04:45
mehere        from circuits import Debugger04:45
*** mehere has quit IRC04:45
*** mehere has joined #circuits-dev04:46
prologicooops :)04:46
mehereBTW the other SSL test is nicely skipped by pytest04:46
mehere"collected 232 items / 1 errors / 1 skipped"04:46
prologicI believe04:47
prologicyeah I did the same thing04:47
*** Osso has joined #circuits-dev06:05
prologichey Osso06:09
prologicyou read the logs right?06:09
Ossoactually at first glance I don't see why the "registered" is not run06:10
prologicit's ok I fixed that small issue06:10
prologicthe registered handler won't get run unless the system has already started06:10
prologicsimple fix though06:10
prologic@handler("register", "start")06:10
prologicdef _on_registered_or_started(...)06:10
prologicall tests passing here thanks to mehere06:11
prologicfull performance back too06:11
OssoI see06:11
prologicI wrote a few more unit tests and closed off those stories06:11
prologicbeen contemplating the db-api wrapper story while waiting for Alice to wake up for her feed06:11
prologicit's not looking good06:12
prologicalso mehere had some good points about dynamic extension/plugin loading06:12
prologicperhaps it's not necessary really06:12
prologicthere are many such facilities out there that will/should just work06:12
prologicor you could just "from blah import Foo"06:12
prologicbut is .handlers() and .handles(*names) useful still?06:13
prologicAlso congrats on our performance so far :)06:13
prologicVelocity of 8!06:13
Ossoyeah maybe we don't need dynamic loading for plugins06:14
Ossobut how is it related to the db wrapper ?06:14
prologicno sorry06:15
prologictalking multiple topics here :)06:15
prologicmy bad06:15
prologicI think I have to feed Alice shortly, then bed06:15
Ossoyeah .handlers is still useful06:16
prologicand obviously .handles(*names)06:16
prologicin which case I should fix these06:16
prologicto take into account subclassed components06:17
prologic>>> from import TCPServer06:19
prologic>>> x = TCPServer(0)06:19
prologic>>> x.handlers()06:19
prologic>>> x.__class__.__bases__[0].handlers()06:19
prologic['stopped', 'read_value_changed', 'broadcast', 'close', '_disconnect', 'registered', 'started', '_read', 'write', 'parse_bind_parameter', '_write']06:19
prologicI think I'd kind of expect the first x.handlers() to return a merge list of thw teo06:19
prologicok I'm off to bed :)07:32
*** Osso has quit IRC07:48
*** welder-in-a-chai has joined #circuits-dev07:48
*** marky1991 has joined #circuits-dev09:11
*** Osso has joined #circuits-dev09:41
*** welder-in-a-chai has quit IRC12:02
marky1991hmm, so the test_logger tests pass now, but the test_directory test in test_static still fails.12:34
marky1991I need to ask about the logger part. I've got a fix, but I'm not sure if I'm working around a problem in the code or if my setup was broken.12:35
*** Osso has quit IRC13:49
prologicmarky1991, what do you mean?14:38
marky1991hold on. Be back in a second14:39
marky1991er, 10 mimnutes or so14:39
marky1991On my system, the hosts file includes both the standard loopback address,, and a hostname pointing to
marky1991described here:
marky1991what ended up happening was the requests were getting the as their address, and the checks were checking for the standard
marky1991i "fixed" it by renaming the hostname to
prologicfor which tests?14:49
prologicoh i see14:49
marky1991I still don't entirely understand what the system is doing with
prologicprobably better to rewrite the test slightly14:49
prologicto better take into account different platforms and behaviors14:50
prologicie: get the system hostname14:50
prologicand do some string formatting to assert stuff14:50
marky1991but that doens't quite work14:50
marky1991system.gethostname returns
marky1991it'll be the same14:50
prologicbreakfast - bbs14:51
marky1991I don't understand. How do we use the system's hostname? The test is doing the right thing; it's in the logger that the wrong thing is done.15:06
marky1991What ends up happening is the logger returns, which is we get the hostname attached to that, it's localhost, which isn't the same thing as socket.gethostname()15:06
marky1991*which if15:07
marky1991i.e. the logger returns "", socket.gethostbyaddr("") returns "localhost", socket.gethostname() returns "mark-Virtualbox", and socket.gethostbyname(socket.gethostname()) returns ""15:10
marky1991the problem is request.remote.ip15:20
marky1991that's returning instead if
marky1991i don't know why15:20
prologicI don't think there's a problem with that code Mark15:20
prologicthat code was directly borrowed from cherrypy15:20
prologicit's the remote hostname or ip15:20
prologicie: the client15:20
marky1991request.remote is the client?15:20
marky1991the docs on request objects aren't that great15:21
prologicthey're not :)15:21
prologicthey kind of suck15:21
prologicbut they work15:21
prologicit was sort of a hodge posh of stuff I wrote myself15:21
prologicand stuff I borrowed from cherrypy15:21
marky1991request objects are from the standard library, aren't they?15:22
marky1991i see.15:22
marky1991haven';t even looked at the docs then15:22
marky1991i thought urllib generated them15:22
prologicrequest.remote is just a simple object15:22
prologicthat contains the remote client's ip address or hostname15:22
marky1991yes, i saw that part15:22
marky1991okie doke15:22
marky1991i'll look at that then and see what's going on15:23
prologicihmo logger should be doing the right thing15:23
prologicI'd be very surprised if it's not15:23
marky1991the problem isn't logger per se15:23
prologicit's basically apache-style logging15:23
marky1991it's recieving the wrong data from remote.ip15:23
marky1991it's almost impossible for logger to be wrong15:23
marky1991considering how simple it is15:23
prologicmy point exactly :)15:23
prologicif it is indeed logging what you think is the wrong thing15:23
prologicbest look into why and how15:24
prologicand it's possible it's a system configuration issue anyway15:24
marky1991yeah, going to go look at now15:24
prologicI'm not really sure how many folks really use the Logger component anyway15:24
prologicit would be removed it's so simple to write15:24
marky1991it could be, but if it is, it's either distirbitun wide, or specifici to the fact that I'm unde a VM15:24
marky1991I really should boot into linux, but then I have to deal with grub15:25
marky1991anyway, off to go see wrappers.py15:25
prologicI'll be around15:25
prologica lot of our tests are fialing on OS X17:04
prologicthey all passed at revision 319417:04
prologiclong gone :)17:04
prologicI'll look into it17:04
marky1991what version of python?17:04
prologicsomeone remind me17:04
prologicI need to setup buildbot or something17:05
marky1991that would be cool17:05
prologicand a ci server17:05
prologicso we can continually test every revision against multiple platforms and versions of python17:05
prologicon my MacBook Air it's 2.7 I beleive17:05
prologicinstalled from source via brew17:05
prologicI might buy a Mac mini just for the purpose I think17:05
marky1991I've never actually used that sort of thing, but that'd be something I could register my machine with, correct?17:06
prologicand set this all up at home soon-ish17:06
prologicyour machine would be a build slave17:06
marky1991that'd be great17:06
prologicjust keep poking/proding me to set this up17:06
prologicit requires infrastructure unfortuantely17:06
prologicI don't believe there is a cloud service that provides such functionality out of the box17:07
prologicif there is let me know :)17:07
prologicI registered a while ago17:07
prologicwith the intent of creating such a service17:07
prologicsimilar to readthedocs.org17:07
marky1991what does python use?17:07
prologicthey have their own17:07
prologicusing buildbot17:07
prologicand various slaves17:07
prologicthere is (afaik) no nice CI for Python though17:07
prologicthe ones that exist:17:08
prologicand Atlassian's Bamboo17:08
prologicare all written in Java iirc17:08
prologicit would be really cool if we wrote such a CI using circuits and circuits.web :)17:08
prologic"eat your own dog food"17:08
marky1991printer is misbehaving17:09
*** marky1991 has quit IRC20:12
*** marky1991 has joined #circuits-dev20:12
prologicmehere, hey did you write all the effects/causes and alert_doen stuff?22:27
prologicwas just fiddling around with performance22:28
prologicand found that the entire def _alertDone(...) call costs about 10% in performance22:28
prologicdo we care?22:28

Generated by 2.11.0 by Marius Gedminas - find it at!