IRC Logs for #circuits Thursday, 2013-03-21

*** Osso has joined #circuits00:29
*** ronny_ has quit IRC04:25
*** ronny has joined #circuits04:27
*** ronny has quit IRC04:27
*** ronny has joined #circuits04:27
*** spaceone has quit IRC04:52
*** ronny has quit IRC05:51
*** ronny has joined #circuits06:06
*** spaceone has joined #circuits06:52
*** Osso has quit IRC07:26
*** Osso has joined #circuits07:45
*** koobs_ has joined #circuits07:54
*** Osso has quit IRC10:11
jgiorgiprologic: any word on fixing our issue with multiple filters? i'm developing even more lol10:11
*** ronny has quit IRC13:55
*** Osso has joined #circuits14:34
*** Osso has quit IRC14:37
*** bleep has joined #circuits15:09
*** FSX has quit IRC15:11
jgiorgiwas any work ever done on a wrapper for database queries? i'm wrapping mongodb and figured i follow a similiar format if one existed15:11
*** frank-san has joined #circuits15:13
prologicjgiorgi:  will look into that issue now15:16
prologicjgiorgi:  show us how you wrap/use mongodb in your circuits.web apps?15:16
jgiorgiright now i basically dont15:17
jgiorgii initiate a connection, slap it on an environment object i pass around and let everything just access the connection directly15:17
jgiorgimy idea is to follow the format for mongodb almost to the letter, wrap the output in a simple dictionary based wrapper (almost exactly the same as PyMongo does already)15:19
jgiorgithen when (if) i wrap another db i'll wrap it with the same api15:19
prologicthat's what I do with sqlalchemy as well15:22
prologicof course long running transactions will block the app15:22
prologicbut right now I don't care for most of my apps15:22
prologicwe do plan to create async wrappers around dbapi, mongo and a few others15:22
jgiorgii still have an async Mongo driver in the works but it isnt high priority15:22
prologicwell there is motor (seen it?)15:23
prologicI'd like to just port that to circuits15:23
prologicalso with your issue15:23
prologicI could not reproduce it here with this test case15:23
jgiorgii'm hoping to mimic the API of PyMongo just keep the network blocking minimal15:23
prologicCan you study your code again and see what I've gone wrong?15:23
prologicideally we need complete async db support for a no. of database engines15:23
prologicour first pass at this will be to wrap dbapi (using threads) and make db access through standard dbapi asynchronous15:24
jgiorgihrmm that's an idea, mongo's protocol was so simple i was essentially just writing a protocol driver15:24
jgiorgibut for multi-node db's it gets complex15:25
prologicno no  :)15:26
prologicno need to reinvent wheels15:26
prologicjust wrap existing libraries using threads if they don't support async callbacks15:26
prologicotherwise use the async callbacks provided by some db libs15:26
prologicit should be fairly trivial15:26
prologicthis is next on my radar15:26
prologicanyway could you look at the paste above15:26
prologicand tell me what I've done wrong?15:26
prologicthis doesn't reproduce your errors/issue15:26
jgiorgii'm looking at it, adding in one step of complexity until i reproduce the issue15:27
prologicI though it was as simple as that15:29
prologicbut obviously not :)15:29
spaceoneprologic: you can remove circ-exp15:30
jgiorgiprologic: almost that simple, the problem occurs when streaming, Component based returns work15:37
prologicspaceone:  ok, gn815:53
prologicjgiorgi:  ok lemme see...15:54
prologicok got it15:58
prologiclet's see if we can fix this15:58
jgiorgiim here if you need anything, working on a Results wrapper16:00
prologica Results wrapper?16:04
prologicI'm not entirely sure how to solve this16:04
jgiorgii couldnt even begin to find the area of the code causing it16:14
jgiorgiI'm writing a thin wrapper for results retrieved by a PyMongo Cursor object, i'm hoping to add a bit of functionality (such as being able to iterate more than once) and create something that i could wrap other database results in16:16
jgiorgisort of an ORM wrapper for no-sql databases16:16
prologicI'll tell you16:17
prologicif you can help16:17
prologicI have to work (at work)16:17
prologicI'm really sorry about this16:17
jgiorgilol ok16:17
prologicwe f'd up streaming somehow16:17
prologicnothing I think that's intuitive works for me :/16:17
prologicso sorry :)16:17
prologicit's all in circuits/web/http.py16:17
prologicPlease refer to the dev repo though when debugging this16:17
jgiorgiit's not really holding me up anywhere, just keeping me from testing frankencircuits16:17
jgiorgii'll take a look at the code and see if i can think of something16:18
prologichave a look at my revised test cases16:19
prologicthere are two problems I've identified16:19
prologicThe exact problem is in two parts16:19
prologichandling of streaming in general (I can't get a simplest thing to work)16:20
prologicand handling of request_success16:20
prologicThe problem as I see it is this:16:20
prologicYou have two request handlers that manipulate the request/resonse objects in some way16:20
prologicthey just let the request pass through16:20
prologici.e: return None16:20
prologicProblem is, the request_success handler treats this as a 404 NotFound16:20
prologicbecause of the None return16:21
prologicAnd so therein lies the problem I believe16:21
prologicIf we fix that, the other issue should go away (I think)16:21
jgiorgicould we just change the behavior to ignore another boolean False, non-None return value16:32
jgiorgiie None == 404 but 0 (which is still boolean false and shouldn't filter/stop the event) would mean nothing to do here, move along16:33
jgiorgiwait, if stream is True we shouldn't treat none as 404, if stream is set to True we should expect Stream events16:34
prologicahh yes16:39
prologicbut you don't set to True until later16:39
prologicso the sequence is16:40
jgiorgii though the _success event was fired after all handlers?16:40
prologicI think you're idea of treating False as "do nothing" might work16:40
prologicbut I'm not sure...16:40
prologicI'd have to play around with it16:41
prologicleave it with me?16:41
prologicI'll talk to Osso about it today/tonight sometime16:41
prologicand we'll come up with a plan to fix that16:41
prologicand also we'll vastly improve streaming in 2.1.116:41
jgiorgidoes circuits treat return False, return None and return 0 as don't interrupt this event16:41
jgiorgior with filter=True does any non-None result interrupt the event16:41
prologicif filter=True16:42
prologicany true-ish return will filter the event16:42
prologicpreventing other event handlers from running for that event16:42
prologicbut that's moot here16:42
prologicit's the handling of request_success here that's the problem I think16:42
jgiorgiok, well i'm playing around with it, if i find something that works great, if not i'll leave it to you guys16:42
prologici.e: intercepting but not filtering a request16:42
prologickk nps16:42
prologicdo make sure you run all the web tests though if you find something that works for you16:43
prologici.e: ensure you don't break other behaviors/functionality16:43
prologicpy.test -x -s tests/web/16:43
jgiorgiwell after correcting None handling i found a problem in stream, no headers are sent at all17:14
jgiorgijust data17:14
jgiorgiso the browser is getting data and not knowing what to do with it17:14
jgiorgisolved that problem and found another, sometimes i do want to interrupt a Request event, so i return 1 to prevent further propagation, circuits.web attempts to use this as data for the client17:26
jgiorgino idea how to fix that one without breaking nearly everything17:26
prologicwith the 1st problem17:50
prologicyou need to not set Content-Length in circuits/web/ if is True17:50
prologicI hope you did that17:50
prologicthe 2nd problem might not be a problem at all17:50
prologicif you want to filter a request17:50
prologicand stop all further processing17:50
prologicyou really ought to do:17:50
prologicreturn response17:50
prologicwhich results in a true-ish object17:51
jgiorgithe problem is at the stage i want to stop the request i dont know the response yet17:53
prologicif you stop the request processing17:53
prologicwithout returning some kind of response to request_filtered17:53
prologichow will you get anything to the client?17:53
jgiorgiby firing Stream or Response17:54
jgiorgii essentially want to 1) stop the request processing and 2) prevent circuits.web from sending a 404 (ie wait)17:54
prologicI think this might be an edge case we can't handle per say17:54
prologicI think you should instead of something like:17:54
prologic@handler("request", priority=1.0)17:54
prologicdef _on_request(self, request, response):17:54
prologic   yield, response))17:55
prologic   yield response17:55
prologicthat might work I thnk17:55
jgiorgithat might work, not my preferred method but it might work17:55
jgiorgibut if i did it that way i wouldn't have to worry about stream at all17:56
jgiorgiis call truly sync or is does it just appear to by sync17:58
jgiorgi... that worked really well actually17:59
jgiorgibut i still have a 404 in my Debugger output17:59
jgiorginvm, that's a bad stylesheet reference18:00
jgiorgiprologic: new problem, if my filter is a generator (even one that yields None) execution stops so now when i do intend to proceed i dont18:13
prologicso that worked for you?19:44
prologicgood :)19:44
prologicyes I think I'm going to (this weekend( change a few things19:44
prologicI'll deprecate in the upcoming version19:44
prologicand make obsolete in 2.219:45
prologicI want to make the following changes:19:45
prologica) Get rid of filter=True19:45
prologicand replace with:19:45
prologicdef _on_foo(self, event, *args, **kwargs):19:45
prologic   event.stop()19:45
prologicAnd add in a new singleton value called noop19:45
prologicso that you can do things like:19:45
prologicreturn noop19:45
prologicfrom an event handler19:45
jgiorgiprologic: interesting, well it sort of worked19:48
jgiorgisince i'm yielding in the filter now i'm returning a bool(True) (a generator)19:48
prologichat's not working for you?19:48
jgiorgiso even when i want to continue i dont19:48
prologicso the request_success is still not handling that properly?19:48
prologiccould you write another Root component/controller based on my recent code pad paste19:49
prologicthat demonstrate your 3rd use case?19:49
prologicthe whole request/filtering/streaming thing19:49
prologicI will solve all three cases this weekend19:49
jgiorgiok, well i'm not using stream anymore19:49
prologicthis one19:50
prologicplease add in your 3rd case19:50
jgiorgijust either yielding a result (which works) or yielding None (which stops the event when it shouldnt)19:50
prologicas another (commented out) Root19:50
prologicyeah add that in19:50
prologicstreaming should only be used19:50
prologicand only works (currently)19:50
prologicin a request handler that does:19:50
prologicdef index(...):19:50
prologic = True19:50
prologic   return self.response19:51
prologicand then later on there are Stream events with the original request/response objects plus a data packet19:51
prologicterminating with Stream(request, response, None)19:51
prologicfor what you want you'll need to take advantage of call/wait IHMO19:51
prologicbut add it into that codepad19:51
prologicso I can make all three work19:51
prologiccall/wait is very useful when you want to fire and not forget19:52
prologicand yes call/wait fake sync19:52
prologicit's still async underneath19:52
prologicdid you see ?19:52
jgiorgithat doesnt work, simple use case for our generator issue19:57
jgiorgino i havent looked at that but i am right now19:57
prologicso basically it doesn't work (although should) because?19:59
jgiorgithe filter is a generator19:59
prologicoooh hang on20:00
prologicthat's not what we talked about20:00
prologicthat won't work anyway20:00
prologicbecause Filter2 yields None20:00
prologicyou want that the other way around20:00
jgiorgiso you told me earlier to yield then yield response20:00
prologicI don't see a in there though20:01
jgiorgiit isn't in the example, but in the real code it's20:01
jgiorgiif foo: yield; yield response20:01
jgiorgielse: yield None20:01
prologicyou need to do:20:03
prologicwhere Foo is some event you want to fire20:03
jgiorgiyes i do that (didnt do it in the example)20:03
prologicand then yield it as a response20:03
prologicdon't worry about yield None20:03
jgiorgiok... i don't explicitly have to yield None20:03
prologiclet me show you with an example20:03
jgiorgibut whether i do it or not python does20:04
prologicyou shouldn't yield None20:04
jgiorgiok hold on20:04, *channels)20:04
prologicwill fire the event on *channels and block and wait for it to complete20:04
prologicbefore returning control to your handler at the next expression/statement20:04
jgiorgididn't yield None20:06
jgiorgisame result because a yield statement anywhere turns the method into a generator20:06
jgiorgiwhich, explicitly or not, will yield None if nothing else20:06
prologicyeah no that's ok20:07
prologicI'll fix all this on the weekend20:07
prologicso it works as expected20:07
jgiorgithat's the last bug holding me up then, i seem to be good at finding these fringe cases with bugs lol20:08
prologicI expect more something like this:20:10
prologic*eh* they're edge cases we did not think of :)20:10
prologicit's good you've found them20:11
jgiorgiwait why would we be expecting exceptions?20:11
jgiorgii would expect that circuits realizes the generator didnt yield anything useful and continue execution20:12
jgiorgisame as if i return None in a normal method20:12
prologicno I mean20:13
prologicthis is the kind of code I'm expecting20:13
prologicand that's the current bug/output20:13
prologicwhich I'll fix on the weekend20:13
jgiorgiah ok\20:15
prologicI got an interesting project for you20:20
prologicif you're interested :)20:20
prologicHow would you like to port:
prologicover to circuits.web20:20
prologicso we can put up a circuits user map on the website20:20
jgiorgithat would be interesting20:22
prologicup for it?20:24
prologicshouldn't take too long20:24
prologicjust have to swap out the php bits for circuits.web20:24
jgiorgii think i can manage that20:32
jgiorgijust want me to read the log file occasionally or bind this in the db?20:32
prologicport it as-is :)20:33
jgiorgihrmm, not a fan of port other peoples code, not my strong suit20:47
jgiorgibut i'll take a whack at it20:47
jgiorgiif it's SQL based i'm porting it to Mongo20:47
prologicI'd rather you didn't :)20:59
prologicI want to run this as simply as possible20:59
prologicif you want to swap out the db20:59
prologicthen use pickle or json20:59
prologicand just write to a file20:59
prologicin fact use durus if you want20:59
jgiorgiyeah that'd be better, i'll flat-file it if there's any db work21:02
prologicuse either of those two libraries for db21:02
prologicI don't expect this to be heavy traffic21:02
prologicso concurrent db writes won't be very high21:02
prologicor dobbin or durus21:02
prologicor just pickle/json to a flat file21:02
prologicbut if you do pickle/json to a flat file keep is simple and short21:03
prologichmm dobbin seems nicer ihmo21:06
prologicand quite fast enough21:06
jgiorgiwell i'm not in the mood to learn a new db21:06
jgiorgii'll just pickle it21:06
jgiorgidiscovered a workaround for our generator issue22:44
jgiorgii made a generator outside the filter which grabbed the response, so the filter is still a normal method with a normal return22:45
jgiorgikinda messy but it works :D22:45
*** ronny has joined #circuits23:06
*** koobs` has joined #circuits23:21
*** koobs has joined #circuits23:22
*** koobs_ has quit IRC23:23

Generated by 2.11.0 by Marius Gedminas - find it at!