IRC Logs for #circuits Sunday, 2014-04-20

*** Shoggoth has joined #circuits02:47
kdb_Hello shoggoth02:47
kdb__Hi shoggoth02:47
Shoggothhi02:49
Shoggothwas just searching though the channel list and saw you... thought I'd take a look :)02:49
Romsterno problem, python framework there is ##electronics if your after that02:50
Shoggothnope... I'm using xchat and was filtering on python... saw this and was curious... had to actually join to see the whole url in topic :)02:53
Romsterahi've sinced moved to hexchat02:53
Shoggothanyhow... I've bookmarked the url... will take a look when I've got some time02:54
Romsterok02:54
Romsteri'm a aussie so is prologic02:54
Romstersee your on bigpond02:54
Shoggothwell you can't help bad luck ;)02:57
Shoggothbigpond that is02:57
Romsteri'm on eftel good service02:58
Shoggothheh... I'm 6km from my phone exchange... so it's cable or dialup03:07
Romsteroh so waiting for NBN03:07
Romsteradsl2+ uplods speeds are just too slow for uploading to servers03:08
Romsterlike i need sdsl but not provided or far too expensive03:09
Shoggothwaiting for NBN? ...NBN is dead my friend03:10
prologichi Shoggoth03:12
prologicwelcome to circuits :)03:12
prologicThe awesomeest python framework there ever was and is :)03:12
Shoggothty03:13
prologicif you have any questions feel free03:14
ShoggothI probably will in a few weeks.... for the moment I've just bookmarked it and I will take a look time permitting....03:19
prologic*nods* nps03:22
Romsteryeah i don't hold much hope o nNBN damn tony abbot03:26
Romsteron NBN03:26
Shoggothit's dead.... and tbh as much as Abbot shit's me... it's Turnball that did it in.... I used to work at one of his companies in the 90s.... dishonest bastard03:29
Romsterdamn it03:30
prologicsweet03:42
prologicI just got my Tiny WIFI Shield working03:42
prologichttps://tiny-circuits.com/shop/tinyshield-wifi/03:42
Romstercool, and i'm thinking maybe it's best i use the spydy library and make my own program to do what i require03:43
Romsteris this for your chicken pen?03:45
Romsteris that all the files in /usr/lib/python2.7/site-packages/spyda/ prologic04:31
Romster/usr/lib/python2.7/site-packages/url.py might be part of it04:32
Romsterthat last one url.py is probably another project. still missing some files for the actual crawler...04:35
prologicsorry what?04:44
prologicwhat are you doing?04:44
prologicjust pip install spyda04:44
prologicalso please have some patience04:44
Romstermax_depth in crawler.py where eon earth is it actually defined?04:44
prologicmy grandpa is dying this weekend04:45
prologicso I haven't been able to fix those bugs yet04:45
Romsteroh04:45
prologicbeen having very little sleep last few days04:45
Romsterdamn i'll leave you be this weekend then.04:45
prologichehe04:45
prologicit's okay :)04:45
prologicjust haven't really been motiviatd enough04:45
Romsteryou never said anything until now.04:45
prologicI'll get to those bugs soon04:45
prologicI was hoping I didn't have to :)04:46
prologickinda a family/private thing04:46
Romsteri'm hunting around where the depth loop is so i can take a look04:46
prologicok04:46
Romsterand understand it better04:46
prologicso it's in spyda/crawl.py04:46
Romsteroh crawl not cralwer04:46
prologichttps://bitbucket.org/prologic/spyda/src/97018bba6b0d9373559b0e5e6a419398f3872ea6/spyda/crawler.py?at=default#cl-3004:47
prologicthe whole crawl() function is htere04:47
Romsterah that's easier to find.04:47
prologichttps://bitbucket.org/prologic/spyda/src/97018bba6b0d9373559b0e5e6a419398f3872ea6/spyda/crawler.py?at=default#cl-10104:48
Romsterjust want to study it see what todo04:48
prologicthis is the test for max_depth04:48
prologicso the logic is clearly wrong there04:48
Romsteri was poking around in my /usr/lib/python2.7/site-packages/04:48
prologicso...04:48
prologicif you want to have a go at fixing this yourself04:48
prologicand submitting a pull request04:48
prologichere are my thoughts04:48
Romsteri went past that function -_-04:48
prologicyou'd have to figure out a way of keeping a tree of urls04:49
prologici.e: a graph04:49
prologicso imagine urls a, b, c, d, e04:49
prologicyou could have a tree of visits such as:04:49
prologica04:49
prologic|-b04:49
prologic|-c04:49
prologic  |-d04:49
Romsteryeah04:49
prologic    |-e04:49
prologicso max_depth there is 304:49
Romsterof urls deep.04:49
Romsterhow it works now makes no sense to me.04:50
Romsterthat's how i expceted it ^04:50
Romsterexpected*04:50
Romsterand sorry about your grandpa i'll mess around the code see what i can come up with.04:50
prologicwell04:51
prologicI don't think it's hard to keep a graph/tree of urls visistied04:51
prologicyou could use nested dicts for example04:51
Romsterand at the end list them all or stop at the level you asked at.04:52
prologicmaybe04:52
Romsterand this is the same area i'd need to do a mime type check too.04:52
prologic*nods*04:52
prologicbut I'd do a head() first04:52
prologicand only do a get() if it's a Content-type we care about04:53
Romsterright that way avoids needlessly downloading huge amounts of data.04:53
Romsteri'll see what i can do, go take care of grandpa04:54
Romsteri kinda got this working as i'm after but these few changes will be the icing on the cake.04:54
Romsterbut i'm still impressed with your work.04:54
Romstermaybe after all the stuff i need is added 0.0.3 could be released.04:55
prologicwell if you can work around it for now05:03
prologicI'll fix those said bugs and push out 0.0.3 soon05:03
Romsterok05:03
Romsteri'll look into the nested dictionary part i gotta try and learn python better again.05:04
Romsteryeah the while loop is how i found out it's behaving, depth on like yacy spider is actually n levels deep in a url tree.05:05
Romsterthat's what confused me so much05:05
*** Shoggoth has quit IRC05:09
prologicedk, ping?05:14
*** jpenny_ has left #circuits ("Leaving")12:41
*** ninkotech has quit IRC13:14
*** ninkotech has joined #circuits13:14
kdb_Heya ninkotech13:14
kdb__Hi ninkotech13:14
*** robert_ has quit IRC22:21

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!