IRC Logs for #circuits Thursday, 2014-09-25

*** robert_|disconne has quit IRC00:15
*** robert_|disconne has joined #circuits00:15
*** Workster has joined #circuits04:21
prologiceven'n all09:07
*** koobs has quit IRC11:14
*** koobs has joined #circuits11:40
Yamahi15:05
cfarmerprologic: any idea why i might be getting bad pickle data coming through a bridge? looks like _on_read messages are getting split into two?!16:29
prologiccfarmer, what is it that's trying to be pickled?21:02
cfarmerprologic: just a dict21:02
prologicpickling and multiprocessing can be problematic depending on what you're doing21:02
prologice.g: you can't pickle functions, classes, etc21:02
prologicreally only basic stuff21:02
cfarmeryeh. i’m making sure to stick to basic python types… in this case, just a dict, here’s some error output if that helps: http://circuits.codepad.org/uXPe9I4F21:04
*** Osso has joined #circuits21:05
cfarmerprologic: its just a dict created from a twitter json feed21:05
cfarmerjson.loads21:05
cfarmerprologic: I should mention, *some* of the messages so through fine, and others don’t… which are obviously the hardest ‘bugs’ to find :-p21:05
prologic*nods*21:19
prologiclet me know what you find21:19
prologicEOFError: Ran out of input21:19
prologicwtf? :)21:19
prologicand I'm not sure about the PickleError21:20
prologichard to tell what wasn't picklable21:20
cfarmerIt looks like the dict is being split into two pieces, and since that doesn’t make sense, we get the pickle error21:20
cfarmeri’m thinking it is an issue with the socket… but not sure why it is sending things in two pieces… apparently socket.recv isn’t guaranteed to work all of the time, so haven’t checked that yet, but that is a possible source? possibly something to do with BUFSIZE? Maybe 4KB isn’t big enough?21:25
cfarmeri’ll try changing that...21:25
cfarmerI’m looking in circuits.net.socket around line 47 (on github mirror)21:26
cfarmerprologic: eureka! if i go in and change the bufsize of the bridge’s parent socket, i no longer get the message truncation (i guess twitter json is too big?!)21:35
cfarmerprocess, bridge = streamer.start(process=True, link=component)21:36
cfarmerbridge._socket._bufsize = 500021:36
cfarmerprologic: so i am currently streaming all tweets for the nyc area to my console, in a completely non-blocking and event-driven way :-) so cool!21:39
cfarmerprologic: are their any serious side-effects to changing the buffer size on a socket? even with the lager bufsize, I still get some split messages22:02
Yamagn822:06

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!