IRC Logs for #circuits Monday, 2016-10-03

*** ke4roh has joined #circuits01:34
*** robert_|disconne has joined #circuits01:46
*** robert_|disconne has quit IRC01:47
*** robert_|disconne has joined #circuits01:47
*** kxyggvorsaokiddu has joined #circuits02:17
*** Workster has quit IRC02:17
*** Workster has joined #circuits02:18
*** braoru has joined #circuits05:55
*** Workster has quit IRC07:35
*** Coldblackice has joined #circuits08:51
*** Coldblackice has quit IRC09:02
braoruHello, is there a way with JSON controller to easily manipulate json data post/put ?09:11
braorucontroler seem to not react when I set -H "Content-Type: application/json" on my curl09:12
braoruI can see in the request object a bytesIO object .. but seem empty09:42
braorubut in the logger data are loggged ./09:42
braoruseem to be no way to get back the body of a post .. that seem very wierd...10:04
Romsterbraoru, the developer of circuits just went to bed, but hang around and he'll read it later. in the mean time the documentation might have the answer. i have not looked at circuits in some time.10:31
braoruno problem10:49
braoruI'm strill trying to explore the code of circuits .)10:50
braoru200% on the fact it's a problem between my seat and my keyboard :)10:50
braoruspotted10:59
braorubut still wierd stuff a decoding :/11:08
RomsterPEBKAC i get it too.12:18
Romsterproblem exists between keyboard and chair12:18
Romstersometimes when you've been looing at it too long, taking a break and its right under your nose.12:19
rioti have a minimized websocket demonstrator. Could serve as example. No dependencies except HTML5 and Circuits, three files (index.js/html + websockettester.py)12:21
riotthing is, it shows signs of breaking up and not catching multiple pieces of large messages. Esp. in regards to streaming multimedia content, i'd like to know how to deal with these kind of things?12:22
riot(I'd also like to know how this is handled in HTML5, esp. for high-rate things like video-streaming)12:24
Romsteri'm not sure how html5 video/audio handles that12:24
rioti don't expect that here ;)12:26
riotbut right now, i'm just sending big stringified json objects, as soon as i hit a limit of a few kb, the messages seem to get split up and i don't know how to get them together again.12:27
rioti understand, that streaming is a complicated matter, but i'd already be very happy, if i could get 150kb objects from the frontend safely and in a sane way.12:27
Romsterso something is fragmenting it up and you need a way to assemble it again12:29
Romsterdoes it split at a valid json point or does the json get get truncated at any part of the string and not get closing } braces?12:30
Romsterjust get*12:30
Romsteri've studied tcp/udp protocols but i haven't got that involved in hls and such that twitch uses.12:31
rioti only see byteobjects here12:34
riotseems to end inmidst of the data12:35
riotso i could in theory aggregate until the parser says: YES, this is json - but that would be kinda madness12:35
riotand as far as i understand the docs about the websocketcodec/protocol etc., the framing should be handled automatically.12:36
Romsteryou should be able to open a tcp connection and send as much data indefinitely. until such times as too much packet loss time out12:36
riot(where again i ask: how about non-finite data?)12:36
Romsterthe good thing with json is if it's invalid it gets silently ignored.12:36
riotthing is, pretty much that happens, when i do that: once, the receive queue in the dispatcher is full, the websocket disconnects12:36
Romsterthe bad thing is it gets ignored.12:37
riotthe bad thing with that is, checking takes time, whereas a clear continuation message-flag or similar prepared by the protocol, does not12:37
riotand how is inifinite data framed? E.g. i have a video-stream, how do i frame it over a websocket?12:37
Romsterusually it's chunked up at some set size, and being video if some chunk size is lost in transmission the reciver can ask for that sequence number again... and if it the clients end buffer is near empty and their is no time to get it in time it gets dropped/ignored. (or the server marks that as invalid after a set ie 404 that chunk). in which case the server keeps sending out new chunks and the clients player skips that as a hicup pause12:41
Romsterafter a set time it gets deleted ie 404*12:42
Romsterif the delay gets too bad though the socket gets closed.12:42
Romsterdepends on the video streaming protocol being used etc...12:43
braoruRomster, :-)12:45
braoruRomster, after the lunch break every went better :)12:45
Romsterhehe sometimes helps.12:46
braoruthen I have a json interface .. now I have to make it able to handler more than 1 request at time :)12:55
riotRomster: thanks for those insights. i assume the same procedure is possible via websocket only13:07
riotquestion still remains: how do i get the json chunks reunited properly.. hmm13:08
braorudid I understand something wrong but fire should be asynchronous no ?13:15
Romsterwell not sure exactly how websocket works.13:22
Romstersend a well formatted chunk of json make the server reply saying it got that hash chunk correctly? send another...13:23
riotthere is no response13:27
riotit just gets sent. One of the benefits of websocket. If you want to reply, send a reply ;) I built an echo-server to test this.13:28
Romsterhow do you know it got sent and parsed? also is this over tcp?13:28
riotafter 4kB of data, the string gets framed and not correctly deframed again13:28
riotthis is over tcp. I observe the read events, the websocketdispatcher emits and print the event data.13:28
Romsterbug?13:28
rioti'm now assuming as much. Haven't really dug the source of the protocol/codec, yet, though.13:29
rioti think, i'll first publish the demonstrator/test-utility - useful all in itself13:29
Romstertcp is meant to guarantee no data is corrupted or missing13:29
riotyeah, the data arrives - pretty sure of that.13:29
riotit just seems to be not correctly assembled.13:30
Romstertcp will break it apart but then it gets reassembled at the end13:30
Romsterfor the program13:30
riotand as far as i understand, http does break it again, websocket protocol should handle that, though13:30
riot(oh my. i assume the protocol has to adapt to tcp to be of any efficiency)13:31
riotif you start splitting just behind the tcp-framesize - bad things will happen13:31
Romsterhmm13:31
riotinteresting topic, no? I'd rather prefer it working ;)13:31
Romsterit just pads zeros to make up the frame size.13:31
Romsterpayload doesn't have to take the entire tcp packet.13:32
riotinsofar, i'd scrap the padding ,)13:32
riot(ie waste of time)13:32
Romsterbut yes if you split it at the right size it'll be more efficient13:32
riotcould this be faulty? https://github.com/circuits/circuits/blob/master/circuits/protocols/websocket.py#L8913:34
riota lot of stuff going on in that single method13:34
Romsterpayload_bytes = 2 if payload_length == 126 else 813:36
Romsterweirdness13:36
riotACTION studies https://tools.ietf.org/html/rfc645513:36
Romsterit has logic for pending payload13:37
riotthat is weird, i only encounter problems after 4k13:39
rioteverything smaller than that is totally working - thats why i was able to work with this bug remaining hidden for quite some time.13:39
riot126 bytes?! I'd totally have noticed.13:39
Romsterit seems it reads that and looks for mask opcodes etc... before the payload. but it has a continue bit for more payload on additional data coming in13:41
Romsteror ow i read it...13:41
Romsterthere must be a buffer somewhere else your hitting at 4KiB ?13:41
rioti don't see where, the demonstrator is really tiny, barebones. Frontend uses html5 only. Backend uses circuits only13:42
rioti'm doing this on localhost13:42
Romsterif len(data) - offset < payload_length:13:46
Romsterbreak13:46
Romsterhmm nevermind that seems ok13:47
riothttps://github.com/ri0t/websocketdebugger13:49
braoruIf I want my webserver (build with circuit) to accept concurrent connection, I must use worker ?14:05
braoruwill see tomorrow then ..14:25
*** braoru has quit IRC14:25
*** Coldblackice has joined #circuits16:06
*** Coldblackice has quit IRC16:07
*** Coldblackice has joined #circuits16:09
*** Coldblackice has quit IRC16:24
*** Workster has joined #circuits22:13

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!