IRC Logs for #circuits Monday, 2017-01-23

prologicoh believe me00:00
apollo13wait what?00:00
prologicwe've had lots of users make this mistake over and over :)00:00
apollo13how can you race between registration and __init__?00:00
prologicoverride __init__ on a subclass of Component and wonder why things don't work :)00:00
apollo13init is done before you can even call register00:00
prologicso we created an API for that; we call .init() if it exists after component creation/registration00:00
prologicmetaclasses :)00:00
apollo13oh boy00:01
prologicautomagically decorate methods as @handler(s)00:01
prologicComponent that is00:01
apollo13well that one I found00:01
prologicBaseComponent doesn't; more explicit00:01
apollo13yeah, I've read :)00:01
prologicconvenience is nice sometimes :)00:01
apollo13sure, that handler registration is nice00:02
apollo13but I don't get the __init__ stuff yet, it probably sacrifices too much00:03
apollo13ie all those getattr calls where you just think "WTF"00:03
prologicyeah maybe00:04
apollo13at least it certainly confuses me ;)00:04
apollo13I agree that super() etc can be hard to grok, but those are python things that people need to know00:04
prologicof course00:05
prologicbut people forget more often than it's worth :)00:05
apollo13hihi, look what I found: channels = event.channels or (getattr(self, "channel", "*"),) or ("*",)00:05
prologiclet me see if I can remember why the property stuff is so hard00:06
prologicso this says what the expression does :)00:06
prologicevent.channels -> component.channels (if not falsey) otherwise global "*"00:07
apollo13yeah, took me a second glance though since I couldn't figure out if you were trying to do something tricky given that the last or seems useless00:08
apollo13anyways, I'll try if I get something to reproduce but so far nothing :(00:08
prologicso I removed one of the if getattr(self, "_uri", None) is None:00:09
prologicchecks in HTTP00:10
prologicand so far it works just fine00:10
prologicso maybe whatever problem we had is gone? :) lol gawd only knows00:10
apollo13well it worked fine before too :(00:10
apollo13it's not gone since it showed up on the recent build and I got it once locally00:10
apollo13I also think that running pytest with -v makes the error disappear00:10
apollo13as if adding the Debugger causes issues00:10
apollo13or rather fixes a race condition00:11
apollo13that said, while I do have you here00:11
apollo13assume that I have a manager running in an extra thread00:11
apollo13no I do .fire(some_event) to that manager from the main thread00:11
apollo13if the other manager is in select() with a timeout of -1, will it ever wake up?00:12
prologicit should - yes00:12
prologicI say it *should* but I'm not 100% sure00:12
apollo13what would cause it to wake?00:12
prologicthere's suppose to be a way to wakeup the event loop from heavy sleep00:13
prologiceven with I/O pollers00:13
prologicgood question00:13 is a system call, the gil is released, so unless you send a signal or touch a socket you lost00:13
prologicso maybe that only works for the FallBackGenerator?00:13
apollo13you probably could send something via the control_con00:13
prologicdon't we have the control socket for that?00:13
prologicyeah I thought that's what it's purpose was for00:14
prologicwe don't wakeup via the control socket I presume?00:14
apollo13I didn't yet fully follow it00:14
apollo13still trying to grasp stuff00:14
apollo13prologic: anyways, the only code where I find _ctrl_* to be used is in the pollers, and the resume() method doesn't seem to get used anywhere00:16
apollo13so I guess I have to test that tomorrow00:17
prologicthat's sad :)00:17
prologicI was looking at this the other day too00:17
prologicand it seems whoever wrote that forgot to actually use it? :)00:18
apollo13well unless00:18
apollo13reduce_time_left in has something which seems like calling resume00:18
apollo13which might call that (tm)00:19
prologicit does but that code is kind of hard to follow00:19
apollo13yeah, but fire from another thread sets reduce_time_left to zero iirc00:19
prologicfor example I don't understand why we dynamically add the _on_generate_events from the FallbackGenerator00:19
prologicand not just register the FallbackGenerator in the first place and take advantage of event handler priority and event.stop()00:19
apollo13git grep Fallback00:20
apollo13no results00:20
prologicso if you have other I/O pollers in the mix they take over and stop the event from FallbackGenerator (which is kind of badly named IHMO)00:20
prologicsorry bad typing :)00:20
apollo13na, /me is to stupid to use case insensitive grepping00:21
prologicgrin -i fallback00:22
apollo13anyways, instead of this getattr magic in core/ to find resume, can't we just make the resume method a proper handler?00:22
prologicif removing a bunch of this useless hasattr/getattr stuff works I'll put up a PR00:22
prologicapollo13: where would we put .resume()?00:23
apollo13it is already there on the pollers00:23
apollo13ah stuff clears up00:24
apollo13I guess core/ has generate_events as handler in reduce_time_left00:25
apollo13which is usually only on pollers00:25
apollo13and from the generate_events handler it searches up to the poller and then back to resume00:26
prologicthat sort of makes sense00:26
apollo13but, that call in is guarded by an if isinstance(handling, generate_events):00:26
apollo13so if handling is not generate_events, when another thread fires an event, reduce_time_left is not called00:27
apollo13ergo the control socket is not woken, which leaves the thread in select00:27
apollo13now I have to find out when handling (which = _currently_handling) can be != generate_events00:27
apollo13which can happen if I look at the _dispatcher00:28
apollo13I'll see tomorrow if I can write a testcase to exploit that00:28
apollo13I gurantee you it will be pretty hacky00:28
prologicwell it sounds lke we don't have a test case for that anyway00:28
prologicso we *sould* writ eone :)00:29
apollo13mhm, there might be one00:29
prologicthen make this whole FallBackGenerator and resume stuff better :)00:29
apollo13or rather something like this might be happening:00:29
apollo13if a different thread calls fire and a normal event is currently handled00:29
apollo13then we are not in a select loop00:29
apollo13ah yes, cause _generate_events is called from the generate_events handler00:30
prologicbut we should wakeup any sleeping I/O pollers anyway no?00:30
apollo13there can be just one poller, no?00:30
prologicif another thread injects an event we expect it to react00:30
prologicnot wait for an I/O event before so00:31
apollo13so if another threads injects an event two things can happen00:31
apollo13a) if we are currently in a call, this means we are handling the generate_events event00:31
apollo13in this case core/ calls reduce_time_left on the event00:32
apollo13which calls resume00:32
apollo13which breaks the select00:32
apollo13b) if we are currently handling any other event in the thread, we are not blocking00:32
apollo13and once that event is handled, new events will hopefully get handled00:32
apollo13b) is up for investigation, but I think the flow from a) is correct00:33
prologicgood find!00:33
apollo13sooo, stupid question -- where exactly is the main eventloop started00:34
apollo13because if the manager/root_component knew about the poller instance in the first place, we could get rid of that getattr magic to find resume and just call that directly00:35
GitHub43[circuits] prologic created simplify_HTTP_properties (+1 new commit):
GitHub43circuits/simplify_HTTP_properties 5eaedf0 James Mills: [web]: Simplify HTTP properties by removing useless hasattr/getattr checks00:35
GitHub62[circuits] prologic opened pull request #211: [web]: Simplifies circuits.web.HTTP properties by removing useless hasattr/getattr checks (master...simplify_HTTP_properties)
prologic#211 is up00:35
apollo13oh dandy, pollers are create on demand00:36
prologicso when you call .run() on *any* component in a component graph00:36
prologicthat's what starts the whole system up00:36
prologicand yes00:36
prologicpollers are created on demand00:36
apollo13yeah with _on_registered_or_started00:36
prologicbut only if none are already found00:36
apollo13but there can be only one00:37
prologicso you can explicitly register one up front00:37
apollo13so that one could register on the root component00:37
prologicit shoud be possible to track/know what the Poller being used is00:37
apollo13that would be good I think00:37
prologicor just search for it and memoize it00:37
apollo13is there something like a global event?00:38
apollo13ie so I can add a global handler for registered and then attach the poller to the root component?00:38
kdbUnknown Command: handler()00:40
prologicdef on_all_events(self, event, *args, **kwargs):00:40
prologic   if isinstance(event, registered):00:40
prologic        ...00:40
apollo13ah didn't think that handler doesn't need a class or anything :D00:41
prologicit's kind of tricky though00:41
prologicyou can't do for instnace00:41
prologic@handler("registered", channel="*")00:41
kdbUnknown Command: handler("registered",00:42
prologicat least I don't think it works as well as it should :)00:42
prologicbut @handler() is a special case00:42
apollo13ok, left a comment on the PR00:43
apollo13it is already 01:40 here, time for bed00:43
apollo13good night!00:43
apollo13(and thanks for your time)00:43
GitHub151[circuits] prologic force-pushed simplify_HTTP_properties from 5eaedf0 to df4d609:
GitHub151circuits/simplify_HTTP_properties df4d609 James Mills: [web]: Simplify HTTP properties by removing useless hasattr/getattr checks00:46
prologicyeah g'night :)00:46
prologicI realize now you're actually in EU00:46
prologicwhere I ws last week :)00:46
prologicnow I'm back in the US00:46
apollo13hehe, where did you go to?00:46
prologicDublin for a week00:47
apollo13one last question, was just reading core/handlers.py00:48
apollo13f.event = getattr(f, "event", bool(args and args[0] == "event")) <-- what is that trying to achieve?00:48
apollo13note done if the handler wants to get the event as argument=?00:48
prologicpretty much00:49
prologicand it's not so nice00:49
prologicif we can improve that let's :)00:49
apollo13no idea00:49
apollo13seems pretty okay imo00:49
prologicdef some_handler(self, event, *args, **kwargs): vs def some_other_handler(self, *args, **kwargs):00:49
prologicyeah it was the decision/design at the time00:49
prologicit works00:49
prologic(tm) :)00:50
apollo13certainly does, though stuff like im_self and argspecs might be optimization barriers for pypy00:50
apollo13the latter probably because of the use of inspect or similar stuff (maybe)00:50
apollo13but that is only on registration, so not that often executed00:51
apollo13pypy is amazing with all it's JIT stuff, just make sure you never touch traceback frames :D00:51
prologichaha it is :)00:52
prologicwe get ~10-20x perf out of pypy00:52
prologicon decent newish hw nearly 500M events/s00:52
apollo13we beeing facebook?00:52
prologicsorry that was wrong00:52
prologic500K event/s00:52
prologic0.5M :)00:53
prologicwe as in circuits :)00:53
prologicgo to bed :)00:53
apollo13oh right, good night :D00:54
*** travis-ci has joined #circuits01:03
travis-cicircuits/circuits#616 (simplify_HTTP_properties - df4d609 : James Mills): The build passed.01:03
travis-ciChange view :
travis-ciBuild details :
*** travis-ci has left #circuits ()01:03
*** ke4roh_ has joined #circuits02:27
GitHub189[circuits] prologic created Issue#133 (+1 new commit):
GitHub189circuits/Issue#133 5772548 James Mills: [web]: Improve the API for session management and add expire support....06:07
GitHub66[circuits] prologic opened pull request #212: [web]: Improves the API for session management and adds expire support. (master...Issue#133)
*** travis-ci has joined #circuits06:18
travis-cicircuits/circuits#618 (Issue#133 - 5772548 : James Mills): The build failed.06:18
travis-ciChange view :
travis-ciBuild details :
*** travis-ci has left #circuits ()06:18
GitHub185[circuits] prologic created process_tasks_faster (+1 new commit):
GitHub185circuits/process_tasks_faster be1a2b9 James Mills: [core]: Improve performance of coroutines/tasks and fix event.stop()...07:23
GitHub186[circuits] prologic opened pull request #213: [core]: Improves performance of coroutines/tasks and fixes event.stop() (master...process_tasks_faster)
GitHub76[circuits] prologic force-pushed Issue#133 from 5772548 to 1b72fad:
GitHub76circuits/Issue#133 1b72fad James Mills: [web]: Improve the API for session management and add expire support....07:25
*** travis-ci has joined #circuits07:33
travis-cicircuits/circuits#620 (process_tasks_faster - be1a2b9 : James Mills): The build failed.07:33
travis-ciChange view :
travis-ciBuild details :
*** travis-ci has left #circuits ()07:33
*** travis-ci has joined #circuits07:45
travis-cicircuits/circuits#622 (Issue#133 - 1b72fad : James Mills): The build was fixed.07:45
travis-ciChange view :
travis-ciBuild details :
*** travis-ci has left #circuits ()07:45
*** ke4roh_ has quit IRC12:21
*** sphalerite has joined #circuits13:27
*** ke4roh_ has joined #circuits13:30
*** sphalerite has quit IRC13:37
*** sphalerite has joined #circuits13:39
*** sphalerite has quit IRC14:37
GitHub101[circuits] prologic closed pull request #211: [web]: Simplifies circuits.web.HTTP properties by removing useless hasattr/getattr checks (master...simplify_HTTP_properties)
GitHub85[circuits] prologic pushed 1 new commit to master:
GitHub85circuits/master 25442fc James Mills: [web]: Simplify HTTP properties by removing useless hasattr/getattr checks (#211)16:11
*** travis-ci has joined #circuits17:03
travis-cicircuits/circuits#624 (master - 25442fc : James Mills): The build passed.17:03
travis-ciChange view :
travis-ciBuild details :
*** travis-ci has left #circuits ()17:03
*** ke4roh__ has joined #circuits18:25
*** ke4roh_ has quit IRC18:27
*** sphalerite has joined #circuits18:35
*** ke4roh__ has quit IRC19:06
*** ke4roh__ has joined #circuits19:44
*** ke4roh__ has quit IRC21:48
GitHub176[circuits] spaceone created starttls (+1 new commit):
GitHub176circuits/starttls d889942 SpaceOne: Implement STARTTLS event; Fix #113...23:54
GitHub54[circuits] spaceone opened pull request #214: Implement STARTTLS event; Fix #113 (master...starttls)

Generated by 2.14.0 by Marius Gedminas - find it at!