IRC Logs for #circuits Sunday, 2017-01-22

prologicyeah :)00:09
prologicI gotta spin me up Prometheus then :)00:09
prologicpetty sure there are decent docker images of00:09
apollo13prologic: it is a single go binary iirc00:12
apollo13prologic: around, I got a very very stupid question :D00:14
prologicapollo13: shoot?00:37
*** romster_ has joined #circuits06:02
*** romster_ has quit IRC06:07
*** romster_ has joined #circuits08:53
*** romster_ has quit IRC08:59
*** romster_ has joined #circuits09:15
*** romster_ has quit IRC09:22
*** Workster has quit IRC11:03
*** laftdyidamupnvec has joined #circuits11:03
*** Workster has joined #circuits11:03
GitHub96[circuits] apollo13 opened pull request #209: [WIP] Issue198 (master...issue198)
*** pdurbin has joined #circuits14:08
apollo13^ wooot :D looks as  if I actually get pypy fixed14:17
apollo13prologic: do you think it would be okay to have a global variable in circuis (or just on the root components) which disables/enables event priorities?14:17
spaceoneapollo13: why? will this improove performance or fix the problem with the thread locking?14:18
apollo13spaceone: thread locking is fixed by I think14:18
apollo13it should (tm) increase performance since ordering in events __cannot__ be faster than not ordering14:19
spaceoneyes, of course14:21
spaceoneso you want this to increase the performance?14:21
apollo13partially, since if we always use a dequeue + then resort into heap queue on flushing it is more work14:22
apollo13so in general, my code, while correct will be slower14:22
apollo13not by much I guess14:22
apollo13and dunno how to measure :D14:22
apollo13spaceone: always uses a deque now -- if that is fast enough, that would be cool :)14:25
spaceoneapollo13: would you think a dict with unicode -> bytes is not a nice API?15:46
apollo13spaceone: mhm, I need more context15:46
apollo13what are you trying to do?15:46
spaceoneapollo13: a dictionary for http header values15:47
apollo13what are the keys and what are the values?15:47
apollo13also what values? do you mean literally header values or header names?15:48
spaceonefor example: {u'Content-Type': b'text/plain; encoding=UTF-8'}15:48
apollo13mhm header names are required to be ascii I assume?15:48
spaceoneplus there is some mime-encoding15:48
apollo13okay, but handling that as unicode as long as possible makes sense15:49
apollo13what about the values? that example seems as if it could be unicode too15:49
spaceonewell, my API should offer everything as unicode15:49
spaceoneno, these can't be unicode15:49
spaceonebecause if it is unicode the pprotocol-element information gets lost15:50
apollo13what is the encoding of those then?15:50
spaceonein the end it is always latin-1 plus mime encoding15:50
apollo13then you should be passing unicode, no?15:51
spaceoneif you e.g. want → as value (which is not part of latin-1) you need to mime encode it15:51
apollo13then why would the value be bytes?15:51
apollo13work with → till you need to encode for the wire15:51
spaceonebecause if it where unicode and the value would be, e.g: u'foo, bar→baz; blah=blub' → how do i know if the , ; = is part of the strin or part of the value15:52
spaceoneI could of course say i parse this directly and then it wouldn't be bytes/unicode at all15:52
spaceonebut people tend to write headers['Content-Type'] = 'application/json'15:53
spaceonei could also support both, unicode and string?15:54
apollo13mhm, I do not get the problem with foo, bar→baz; blah=blub15:54
spaceoneheaders['Content-Type'] = ContentType('text/html', encoding='UTF-8', foobar='→')15:54
apollo13ah I see what you are getting it15:55
apollo13basically if you wanna use ; as value and not as field separator you need to mime encode it15:55
apollo13or did I miss somethng15:55
apollo13imo I'd still accept unicode and then just latin-1 encode it15:56
apollo13and late the user take care of quoting15:56
apollo13or support classes like ContentType where needed15:56
spaceonewell, this leads to unicodeEncodeError if it contains →15:57
spaceonei already have these classes15:57
apollo13hence: "<apollo13> and late the user take care of quoting"15:57
spaceonethe user should not know about quoting at all15:57
spaceonehe wants a API where he can enter his strings15:57
apollo13 this is what django does15:58
spaceoneand don't care about the underlying protocol syntax15:58
spaceonejeah but this leads to incorrect implemented RFCs and security holes15:58
apollo13security holes? tell me more :D15:58
spaceonei read all those code some years ago15:59
apollo13well, if you find a security issue there, I'd be glad to hear about them16:01
spaceonethe web is full of security issues.16:01
spaceoneone example i could give you about this:16:01
spaceoneif the user needs to care about percent encoding in a URI16:01
spaceone. is different from %2C16:02
spaceonehe needs to be aware of this16:02
apollo13yeah depends on the context etc16:02
spaceonebut this is an actual security issue which raise up in a lot of things16:03
spaceoneeven in apache httpd16:03
apollo13yes, but hard to find usually -- hence me asking if you see an actual security issue there that I can fix16:03
spaceoneah, no i don't see one concrete issue. i just want my project to be designed that these issues can't ever happen16:04
apollo13http is hard16:04
spaceonehm, not really16:04
apollo13and with all those browsers doing different things it is not getting easier :D16:04
spaceonea lot of broken stuff16:04
spaceonehm. if i port httoop to python3 i drop support for byte-strings in the API. when i started the project i left them there for python2 because i wanted users to be abled to write e-g- headers['Content-Type'] = 'foo/bar' (both bytestrings). now they need to prefix all strings with u''. i didn't want to make type-checks everywhere16:09
spaceonein python3 i like that everything throws a TypeError if unicode and bytes gets mixed16:09
spaceonei which this was in python216:10
spaceonebut i dislike that they changed the parameter of bytes() (in py2 there is no encoding param :( and the name 'unicode' is not aliased to str in python3 :(16:10
apollo13spaceone: well, partially there is six -- but yes, the missing encoding is annoying16:13
spaceonesix is not part of the stdlib16:14
apollo13it is a single file, literally every third project vendors it16:14
spaceoneand i hate to import things which the language itself should provide16:14
apollo13ACTION shrugs16:14
apollo13I wish I had your problems ;)16:14
apollo13"<spaceone> in python3 i like that everything throws a TypeError if unicode and bytes gets mixed"16:16
apollo13and that is a good thing16:16
apollo13in python 2 it worked, till it did not…16:16
apollo13also from __future__ import unicode_literals ;)16:16
apollo13though that makes the stdlib kinda annoying on py2 (at least sometimes) :D16:16
apollo13best thing: os.walk, pass in a unicode dirname and you get unicode back16:17
apollo13at least till python cannot decode the filename in which case it will give you bytes16:17
spaceoneah nice. but it also causes problems16:17
apollo13sure it does :)16:17
spaceonei started to add everywhere unicode_literals16:17
spaceoneand got some other errors :/16:17
apollo13sure, that is to be expected16:18
apollo13especially if you use urlparse and other stuff which used to expect bytes in py216:18
spaceonei will begin to start writing everything py3 ready16:19
spaceonebut nearly everywhere i still use python216:19
spaceoneand i like python216:19
spaceonethe greatest reason i hate python3 is because print is a function now, which costs me so much time in debugging things16:20
apollo13hehe, but it made sense to change that16:20
apollo13from teaching I know how confusing that can be16:20
apollo13ie matlab: hold on == same as hold('on')16:21
apollo13try explaining that to students16:21
spaceonebut can't there be a statement like echo or debug or something16:21
apollo13ie if a function takes one (string argument) you can use the first form16:21
spaceoneor at least work in an interactive python shell16:21
apollo13in the shell I usually just to "x"16:21
apollo13ie no print around it16:21
spaceonejeah, but try repr(b'')16:22
apollo13In [6]: b''16:22
apollo13Out[6]: ''16:22
apollo13In [7]: print(b'')16:22
apollo13I like that more than print ;)16:22
spaceonei was sure that it showed me a list of ints16:23
spaceonebut it doesn't16:23
apollo13a bytearray might return ints16:23
prologicapollo13: root component is a good place to store state of the running component graph17:23
apollo13prologic: ok, but aside from the len() stupidity, my approach looks okay?17:33
prologicI think so17:36
prologicthe tests and benchmarks will tell :)17:37
apollo13we have benchmarks?17:37
prologicwe do :)17:40
prologic$ circuits.bench -e 10000017:41
prologicSetting up Speed Test...17:41
prologicSetting up Sender...17:41
prologicSetting up Receiver...17:41
prologicTotal Events: 100006 (44106/s after 2.27s)17:41
prologic$ circuits.bench -m latency -t 1017:42
prologicSetting up Latency Test...17:42
prologicSetting up Sender...17:42
prologicSetting up Receiver...17:42
prologicLatency: 57.935714722 us17:42
prologicLatency: 241.994857788 us17:42
prologicLatency: 135.183334351 us17:42
prologicLatency: 50.067901611 us17:42
prologicLatency: 51.975250244 us17:42
prologicLatency: 97.990036011 us17:42
prologicLatency: 74.863433838 us17:42
prologicLatency: 92.029571533 us17:42
prologicLatency: 54.836273193 us17:42
prologicLatency: 79.154968262 us17:42
prologicLatency: 236.988067627 us17:42
prologicTotal Events: 28 (3/s after 11.04s)17:42
prologicthis is on my shitty ~2015 Macbook with no power in it whatsoever :)17:42
apollo13prologic: mhm, if you do not mind it would be great if you could test my PR and master -- thermal control on my laptop currently does not give me __any__ consistent results17:45
prologicsure one sec17:47
apollo13prologic: so basically the same18:18
apollo13which is good :)18:18
prologicpretty much18:18
apollo13ok, will rewrite it according to your and spaceone's suggestions18:20
prologicyou can also circuits.bench --profile --events=N18:21
prologicif you care about optimizing circuits' core :)18:21
apollo13yeah, I first have to turn off thermal control18:22
apollo13or find an old laptop/pc18:22
apollo13cpu-frequency scaling kills me here18:22
*** robert_ has quit IRC18:22
apollo13and well, performance is not going to be that important for me18:25
prologicwhy's that?18:26
*** robert_ has joined #circuits18:27
*** robert_ has quit IRC18:27
*** robert_ has joined #circuits18:27
apollo13prologic: cause fast enough is fast enough1118:37
prologicwell sure18:38
prologicbut we can always improve :)18:38
apollo13or differently put: if I needed performance I wouldn't be using python :þ18:38
prologicGoland perhaps?18:43
apollo13that or rust, rust looks nice18:50
apollo13also elixir18:50
prologicthose langs are too complicated to learn / be proficient in IHMO :)19:49
prologicGolang has the benefits of being simple enough with a good balance of features and raw performance19:50
prologicit's kind of like a faster Python (it feels)19:50
apollo13I do not find rust that bad19:50
prologicmaybe their docs/tutorial just suck then :)19:51
apollo13well, I do like C, so…19:51
prologicremember I'm legally blind so when reading through Rust docs I kind of went "meh, this looks too hard with too many things to know"19:52
prologicyeah Go is just a better C in many repsects19:52
prologicbut importantly (what every new language should have); Go has an awesome ecosystem19:52
apollo13prologic: I hope that is not overkill :D21:26
apollo13 haha, perfect :D21:31
apollo13I am not going to pretend to understand that test or circuits.web21:35
apollo13if ready triggered, the http component did set ._uri and as such base, is there any way that ready is on the wrong channel or something?21:41
apollo13also works locally, I am pretty sure my PR is not at fault21:44
apollo13prologic: also if you get pypy running on your mac:
prologicI've been looking at the Tick-stack on/off today (plus spending time with my family)21:46
prologicbut I'll take a look at this PR tonight and see what's up with pypy on OS X21:47
apollo13no rush, go back to your family :D21:47
prologicI'd love it if we could build generic counters/metrics into circuits; but finding it hard to find a standard-ish way in OSS :)21:47
apollo13tomorrow is a new week ;)21:47
prologicTelegraf (Influxdata/TICK) supports quite a lot of plugins and collectors21:48
prologicback to work for me :)21:48
prologicapollo13: PR 209 looks good to me21:58
prologicmy only concern is design re dispatching21:58
prologicit seems we do something like self._queue.dispatchEvent(self._dispatcher)21:58
prologicseems a bit weird :)21:58
apollo13yeah, but this was the only nice way I were able to hide the flush_batch stuff in the eventqueue21:59
apollo13if I wouldn't do that I would have half private vars there all over the place21:59
apollo13or calls to ._queue._queue/_priorty_queue21:59
prologicactually this isn't that bad22:00
prologicdispatcher here is like a "dispatching function"22:00
apollo13I even tried context managers ala: with self._queue as flushable_events: for event in flushable_events: self._dipsatch(…)22:00
apollo13but that requires a lot more programming effort for not much gain22:00
prologicall tests are passing with this right?22:01
prologicall bar that one test that seems to be broken before?22:01
apollo13I ran it like 10 times locally :D22:01
apollo13lovely I cannot merge it anyways since travis has to pass, which might get a bit fun :D22:05
apollo13worst case I'll do a manual merge once spaceone gives his okay22:06
apollo13mhm, something is weird, just got that error on from travis locally22:09
apollo13seems to be a thing, but not happening often22:09
apollo13which makes me wonder, MakeQuiet stops the ready event, doesn't that prevent the http ready handler to fire?22:10
apollo13yeah there is a component in there named MakeQuiet which does event.stop for ready events23:50
apollo13I have no idea what it is supposed to do23:50
apollo13and the circuits.web.http.HTTP component listens to ready with priority 1, so it should (tm) before that, but what do I know :D23:51
prologicno you are right23:54
prologicit should have no effect23:54
prologicbecause event priorities23:54
apollo13why is it there at all btw?23:55
prologicbecause the ready event on circuits.web.servers.BaseServer is noisy :)23:55
prologicspin up examples/web/ and you'll see23:55
apollo13but doesn't py.test hide stdout/stderr unless an error occurs23:56
prologicactually you are right it does23:56
prologicso I forget why we added this now :)23:56
prologicbut that component cannot possibly interfere23:56
prologiculess we have a bug in event priority handling23:56
apollo13which might be due to my commit23:56
prologicif you call event.stop() in a lower priority handler23:57
apollo13but I cannot reliably repro locally23:57
prologicit should only prevent propagation from that handler onwards23:57
apollo13yeah, also web.http.HTTP is kinda weird, there is a lot of getattr() in there (ie in the base and uri properties) for stuff that is certainly set in __init__23:57
apollo13so maybe I missed something23:58
prologicthere's a good reason for doing this23:59
prologicbut maybe it's a work-around of an actual problem23:59
prologicbecause when you override __init__23:59
apollo13and not call the parent init? well then your problem XD23:59
prologicthe component registration looks at properties too (for some reason)23:59
prologicand those may not be set yet (race between registration and __init__)23:59

Generated by 2.14.0 by Marius Gedminas - find it at!