IRC Logs for #circuits Friday, 2014-02-14

*** Osso has quit IRC00:10
*** DrSAR has joined #circuits04:06
DrSARHiya. I just posted about my problem in google groups: "What is the required structure for a UPnP listener?"04:10
DrSARCan anyone help me with that?04:10
prologicI'f say if you're building a UPnP server/listeniner04:14
prologicfollow _any_ of the server-like examples in examples/04:14
prologicDid you clone the circuits-dev repo and run python develop/install?04:14
prologicIf so you should have easy access to all examples :)04:14
DrSARI did build and install04:15
prologicI was more referring to whether you had a local copy of the repo :)04:16
prologicbuild + install works04:16
prologicjust as well as "develop" (i.e: in edit mode)04:16
DrSARyep. I am currently looking at */examples. any in particular that might work? chatserver, echoserver echoserverunic nodeserver04:17
prologicthey're all good :)04:17
prologicthe thing here is that you _want_ the UPnP protocol04:17
DrSAR:-) I'm sure. Let me have a look04:17
prologicMichasel Lipp (one of our ex developers) wrote such componnets04:19
prologicyou may want to look at this04:19
prologicand/or bororw/improve upon it04:19
prologicperhaps even contributing appropriate components into
prologicunfortunately whilst circuits is a really nice framework04:19
prologicwe _do not_ have very many protocols implemented as we'd lik04:19
prologicthat being said...04:20
prologicThere is also: circuits.twisted04:20
prologicYou _could_ in theory use this to utilize any Twisted protocols and libraries on top of the circuits framework04:21
prologici.e: all the benefits of reusable Twisted protocols and libraries -- Without the Twisted nightmare/mess and all the elegance of circuits04:21
prologicI have not obviously officially released circuits.twisted (yet) -- so use at your own risk :)04:21
prologiccontributions/fixes welcome!04:21
DrSARI believe my requirements are actually fairly basic and I am trying to keep external dependencies limited. So I'm taking small steps (and I did mention that I'm just as clueless about UPnP as I am about frameworks, no?)04:22
prologicI think so :)04:23
prologicI have to admit I mostly ignored most of your original post04:23
prologicmostly because I knew you'd be trying this with 2.1.004:23
prologicwe haven't as yet found the time to release 3.0 properly04:23
prologiceven though it's arguably mostly ready04:24
prologici.e: we do continuous integration/testing per commit04:24
prologicit's kind of very cortical that circuits works flawlessly04:24
prologiceven if _some_ components have defects (only one -- the web sockets client)04:24
prologicI'm James Mills - the original author and primary maintainer :)04:25
DrSARI tried my code with 3.0.0 but the result is the same.04:25
prologicPost your code up to circuits.codepad.org04:25
prologicLet's see what you're doing here :)04:25
DrSARThis didn't surprise me because it felt like it was doing what it's supposed to04:26
prologicAlso it would be useful to post your "debug output" as well -- the trace of event flow04:26
prologicadmittedly it takes some getting used to -- circuits -- and event driven programming04:26
prologic-but- circuits tries to make this as nice and as elegant as possible04:26
prologicbrb - please post some stuff on circuits.codepad.org04:27
prologicI'll be back in a few mins04:27
DrSARmy code
DrSARthe output from a little ipython session
prologicSo what are you expecting to happen?04:37
prologicAnd what is not working for you?04:38
prologicI notice your Root component is not bound to the "web" channel04:38
prologicfor which all circuits.web components send their events to by default04:38
prologicSo the "request" event handler will never run here04:38
DrSARyou see, that's the sort of thing that doesn't quite occur to me.04:39
DrSARhere is what I expect:04:40
DrSARin method start, I initially let the zoneplayer know that I would like to be notified of interesting events.04:40
DrSARI do this by sending SUBSCRIBE to it and giving it my own IP and a port to connect to.04:41
DrSARwhat I was hoping I could do then is to start the server and wait for anything that comes back.04:41
DrSARhence: self.server = circuits.web.BaseServer((ip, port)) + Root() + Debugger()04:41
prologicall that is totally fine04:42
DrSARI suspect that my Root (just as you said) is somehow not equipped to deal with the connection coming back04:42
prologichowever look at your debug output more closely04:42
DrSARfrom my zoneplayer.04:42
prologicbecause it's channel is not set04:42
prologicall circuits.web events get sent to the "web" channel04:43
prologicyou just need:04:43
prologicclass Root(Component):04:43
prologic    channel = "web"04:43
prologicand your request event handler will then run on inbound requests04:43
DrSARI see.04:43
prologicnow here's another point as well04:43
prologicDid you know circuits has builtin events for many useful things?04:43
prologice.g: started04:43
DrSARin the debug I am noticing04:43
DrSAR<_read[web] (<socket._socketobject object at 0x10dd14ad0> )> <connect[web] (<socket._socketobject object at 0x10dd14bb0>, '', 3346 )> <_read[web] (<socket._socketobject object at 0x10dd14bb0> )> <read[web] (<socket._socketobject object at 0x10dd14bb0>, 'NOTIFY / HTTP/1.1\r\nHOST: 192.168.04:43
prologicyou could replace your Events class with something like04:44
prologicclass App(Component):04:44
prologic   def started(self, *args):04:44
prologic      ...04:44
prologicso see in the output above04:44
prologic<_read[web] (...)>04:44
prologicthe first bit i the event name04:45
prologicthe bit in square brackets is the channel04:45
prologiccircuits has this notation of events and channels04:45
prologicused as separation of concerns04:45
prologicor running up multiple identical components simultaneously without colliding with each other04:45
prologice.g: two servers, two clients04:45
DrSARso would I need methods named 'read' to intercept this?04:46
DrSARcoming back to setting the channel to 'web'04:48
DrSARHow do I then capture this?04:49
DrSARI have a method requests in Root which should be given this incoming request04:49
DrSARyet only two params (request, response) are given to it.04:52
DrSARwhere in request can I extract the body?04:52
prologicsorry for the afk04:53
prologicI wrote an improved version of your code04:53
prologic(not tested obviously)04:53
prologicbut it should work nonetheless (hopefully)04:53
prologicI'll let you look at it then ask more questions :)04:54
DrSARlet me try04:54
prologicI'm just utilizing more of circuits here04:54
prologicand the full circuits.web stack04:54
prologicso you don't need to worry about the request body per say04:54
prologicTo answer some of your earlier questions (for completeness)04:57
prologicTypically you don't intercept _read events -- they're prefixed with an _ to indicate their privateness (convention)04:57
prologic_read and _write are poller events04:57
prologicfrom circuits.core.pollers04:57
prologici.e: Select, Poll, EPoll, KQueue04:57
prologicAlso,, not using circuits.web.Server with circuits.web.Controller you are putting yourself quite low in the circuits.web stack04:58
DrSARholy cow.04:58
DrSARit seems to do something04:58
prologicmeaning you have to take care of the request parameters and body yourself04:58
prologicyeah :)04:58
prologicI thought it *might* work04:58
prologicyou'd think the guy that wrote circuits can write circuits code :)04:58
prologic10th year of development this year ;)04:58
DrSARbut don't run away yet04:58
prologicSo that I provided you should be the basis of what you need so far04:59
DrSARI still have some question if that's OK04:59
prologicat least it'll get something coming in and out :)04:59
prologicyeah sure fire away!04:59
prologicI'm heading home in 30-45mins04:59
prologicbut I'm always here - just not always at my keyboard -- others here will also chime in and help out when/if they can04:59
prologicespecially jgori if you ever see him in here05:00
prologicand Osso (our other dev)05:00
DrSARWhen I switch the Debugger off (I just comment out the registration), I only get this output:05:00
DrSARUsing circuits-dev serial circuits.web/ ready! Listening on:
DrSARbefore, in the debugger I would see this huge XML file that comes with the NOTIFY request05:02
DrSARHow can I get it?05:02
prologicoh :)05:07
prologicit's probably in self.request.body then05:08
DrSARwhen I switch the debugger back on I see it is associated with read[web]05:08
DrSARthe request[web]05:09
DrSARand index[/] seem to be empty05:09
DrSAR(this is where I'm probably misunderstanding how the zoneplayer wants to communicate with me)05:09
prologicshow me that debug output again05:10
prologicof the inbound request to your app?05:10
prologicfrom zoneplayer -> your circuits.web app05:10
DrSARyou're good!05:11
prologicI should hope so :)05:11
prologicI'm still curious to see that debug output again05:11
prologicof the inbound request and it's headers05:11
prologicand the body05:11
prologiccurious it would throw you XML - but not surprising05:11
prologicprobably some home-grown non-standard crap :)05:11
prologicwho makes these zone players anyway?05:12
prologic <-- these the ones?05:12
DrSARthat is it.05:13
DrSARthey sounded reasonably convinced that their UPnP implementation is all to standard05:13
prologicyeah okay cool05:15
prologicjust as I thought05:15
prologicyeah perhaps it is :)05:15
prologicjust nonstandard web stuff05:15
prologici.e: an XML body you have to deal with05:15
prologicin some way05:15
prologicokay this is great :)05:15
prologicyou can interact with your Sonos speakers now :)05:15
prologicas a useful exercise you could:05:16
DrSARwell, you have helped me an awful lot tonight.05:16
prologicwrite a protocol components that handles such requests05:16
DrSARthis has saved me days (and cost you an hour or so)05:16
prologicand puts them into a Worker pool05:16
DrSARthank you very much05:16
prologicfor processing05:16
prologici;e using lxml to parse them05:16
prologicthen hand them back to your original request handler as an lxml element instance05:16
prologicfor better handling and less dealing with parsing/xml in your actual app code05:16
prologicthe idea behind any event driven programming approach05:17
prologicand circuits too05:17
prologicis to do "work" (i.e: CPU bound work) in separate processes05:17
prologicoutside of the main Async / I/O / Event Loop05:17
prologicalso you can combine with nicely with the call/wait primitives05:17
prologicsee examples/primitives/05:18
DrSARthat reminds me.05:19
DrSARmy former implementation somehow rocketed the CPU usage up to 95%/05:19
DrSAR(after the first REQUEST came in)05:19
DrSARthis is not happening anymore05:19
prologicthat's why you should use :)05:20
prologic2.1.0 is okay but it's broken in a few fundamental ways05:20
prologicnamely idling in 3.0 was fixed05:21
DrSARwhen is the plan to release the next one?05:21
prologicor at least rewritten to not consume any cpu time05:21
prologiclast month? :)05:21
prologiclack of time/energy tbh05:21
prologicneed more developers like yourself to get enthustiatic05:21
prologicand help out more :)05:21
prologicit's a distributed effort05:21
prologiceven out other dev Osso has had little time to devote too05:22
DrSARat this stage I am adding more drag than anything.05:22
DrSAROK, this is great. I'll run with this.05:27
*** Myke_ has joined #circuits05:30
prologicNo problems!05:33
prologicIt would be good to get some more interest in the development and maintenance of circuits05:33
prologicSo if/when you fell up to it let us know!05:33
prologicWe deparatetly need some help getting 3.0 released proper :)05:33
DrSARI'll consider it. Tbh, I will have quite a bit of learning to do, before I can be useful.05:37
*** Workster has joined #circuits05:37
prologicOkay I"m outta here -- heading home05:38
Worksterhi DrSAR prologic when i have time i'll throw some money your way on help on getting up and running on circuits and sqlite3 and i'll probably need a hand on scrapy once i re-learn enough i might be able to help more on python05:39
DrSARTHanks - good night05:39
Workstertake care james05:39
prologicWorkster: you should consider using spyda instead05:39
prologicfor all your web scraping needs05:39
Worksterhmm i'll look into that05:39
prologicI'll bbl - heading home :)05:39
Worksterdid not realize you wrote a spider05:41
WorksterDrSAR, is that just for yourself or for a greater community?05:41
Worksterbtw my normal nick is Romster when I am home.05:42
DrSARWorkster, it's for personal use but when I was looking for a solution05:44
DrSARI discovered that I should contribute to
DrSARso it's now an open issue there that I'm trying to fill...05:44
Worksterah cool it's great to give back where possible.05:45
Worksteri maintain packages for #crux linux that prologic also uses crux. And i help out other projects but i'm not great at programming  can do small patches and go though autohell compiling though.05:46
Worksterif any thing could be made as a module to circuits though that would be awesome.05:46
Worksteri intend to use circuits for versionsort.com05:47
Workster45 minutes and i'll be home so i can mess with spyda05:48
Worksterwork is been slow today 4 jobs can't do any waiting on parts.05:49
WorksterIT computer phone repairs.05:49
DrSARactually - I have to go and get some work done myself. (kids are in bed, second shift now...)05:50
Workstermay see you around some other time then05:50
DrSARyes. good chatting, though briefly, see ya.05:51
*** Workster has quit IRC06:12
prologicehlo :) I'm on the bus :)06:13
prologicOK no one is around :)06:21
prologicjust read the backlog!06:22
*** Osso has joined #circuits07:22
*** Osso has quit IRC07:37
*** Romster has joined #circuits07:47
Romsteris around07:48
Romsteras soon as i go get some milk and something for dinner i'll recompile mu kernel for docker and will wanna test out spyda and database it.07:49
Romsterand figure out how to database my filter-tool07:50
*** koobs has joined #circuits08:00
koobsprologic: oops sorry! i migrated to weechat recently and i must have forgot autojoin :)08:01
prologickoobs, oh hiya :)08:22
prologicgood of you to rejoin us :)08:22
prologicI might switch to this so-called X-Chat replacement one day :)08:22
prologic*meh* :)08:22
prologicmaybe I'll just try using circ08:22
prologiccircuits irc client :)08:22
*** ircnotifier has joined #circuits08:36
koobsprologic: weechat is pretty epic :)08:53
koobsprologic: bbs, dinner08:53
koobsbtw, whens the next circuits release?08:53
Romsteri migrated from x-chat to hexchat09:17
Romstersometimes use irssi09:19
Romsterprologic, if i'm gonna use spyda i'll need to be able to database the results.09:21
Romsterthen i'l use another process to read the database and fetch version sort the urls09:22
*** Myke_ has quit IRC09:28
*** Romster has quit IRC10:04
*** Romster has joined #circuits10:17
*** Romster has quit IRC10:30
*** Romster has joined #circuits10:31
*** ninkotech has joined #circuits10:37
prologickoobs, when I can afford time to do it :)11:23
prologicsomeone pay me $2000 so I can take 2 weeks off work to revampt circuits 3.0 and push it out :)11:24
prologic<Romster> prologic, if i'm gonna use spyda i'll need to be able to database the results. <-- spyda's crawl outputs links it's found to stdout, so it's easy to store that. then there's extract for extracting information which you can tell it to output as json to files. spyda is a set of CLI tools and a set of library functions11:25
prologicspyda is used quite estensively at Griffith ( University to harvest data from websites and other institutions11:25
*** ircnotifier has quit IRC11:26
*** ircnotifier has joined #circuits11:27
Romsteri need that ups i don't know what the power company or line distribution is doing but i've never had this many brownouts ever in my life in a short time span.11:36
Romsterok so far i have a fetch url and a version sorter python script i pipe one to the other.11:37
Romsterso the ultimate goal is to spider mirror and main project sites. store results. check at set intervals for new files index new files.11:38
Romsterversin sort store inthe maintable for page searching.11:38
Romsterafter than i can expand it more to test hashes of files.11:38
prologicsounds good11:47
Romsterexcept a) frequent power outages lately until i get the bigger UPS next week. and b) i suck at python still.11:50
Romsterstill messing with sqlalchemy11:50
prologicmake it easier on yourself11:51
prologicforget about databases11:52
prologicand sql11:52
prologicand sqlalchemy11:52
prologicjust deal with the data11:52
prologicand the desired behavior11:52
prologickeep it simple :)11:52
prologicstore data as text files11:52
prologicjson files11:52
prologiccsv files11:52
prologicetc etc11:52
prologicrenders a 500GB dataset11:52
prologicwhich takes 2 weeks for the dataset to compute and generate11:52
prologicbut the website is snappy and quick11:53
prologicno databases whatsoever11:53
prologicall files (json files) on the local file system11:53
prologichmm this one works11:54
prologicthe production version seems to be down for some reason11:54
Romsteri'll see how i go12:14
Romsteri know SQL but then when i get to python i get stuck...12:15
prologicit's just like Bash :)12:18
prologicif ...:12:18
prologic   ...12:18
prologic   ...12:18
prologicfor x in xs:12:18
prologic   ...12:18
prologicdef foo(...):12:18
prologic   ...12:18
prologicclass Foo:12:18
prologic   ...12:18
prologicthere's only 23 keywords in Python12:19
Romsterso why do i get so stuck... i pretty much know bash inside and out.13:18
prologicbecause you're not writing or designing a batch process13:21
prologicyou're trying to design an interactive system13:21
prologicwith mayn complex parts13:21
prologicconcentrate on one at a time13:21
*** Romster has quit IRC14:01
*** Romster has joined #circuits14:03
*** Osso has joined #circuits18:16
*** zleap has joined #circuits19:12
*** zleap has quit IRC23:36

Generated by 2.11.0 by Marius Gedminas - find it at!