IRC Logs for #circuits Saturday, 2014-04-19

prologicpdurbin, of course you realize it's just an easy thing to spin up and test things with :)00:03
prologice.g: the Tutum.co service offerings are actually quite good ihmo00:03
prologicand they gave me $25 credit to have  aplay00:03
prologicI've been providing them useful feedback ,etc00:03
*** bleep has joined #circuits00:15
kdb_Howdy bleep00:15
kdbHello bleep00:15
*** FSX has quit IRC00:19
RomsterCrawling http://github.com/chrippa/livestreamer05:11
RomsterERROR: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed05:11
Romsterprologic, ok just found out it hates certificates.05:11
Romsterprologic, i got some progress and a interesting discovery05:43
Romsterhttp://pastebin.com/Scw1nRnQ05:45
Romster--max_depth does not behave as i expected05:57
Romsterit's actually like while or for loop maximum count05:58
Romsterwhat i need todo is set an actual depth of number of directories/links deep from the base url i start crawling from.05:59
*** Osso has quit IRC06:21
*** Osso has joined #circuits06:27
kdb_Hey osso06:27
kdbHey osso06:27
*** Osso has left #circuits ()06:28
prologicrealzies, re max-depth08:28
prologicerr08:28
prologicRomster, ping08:28
prologicre max-depth08:28
prologicisn't that exactly what it does?08:28
Romstermy vision of max depth is directories deep in the hatchery from the starting crawl08:36
Romstersee the paste test it out the more depth you give the more them version directories are crawed. being that it's 1 directory deep a spider should crawl 1 level of urls/directories in total.08:37
Romsteras it is now max depth is like a while loop only doing a number of iterations over the directories i'm looking at.08:38
Romsterhttp://foo.com/a http://foo.com/b http://foo.com/c i have those that's 1 level deep from http://foo.com08:39
Romsternow if set max depth to 1 it'll only hit a and miss going into b and c08:39
Romsteri level deep is just that one directory or url to another page or site.08:40
Romsterprologic, ^08:40
prologicI think you're gettinc confused with08:48
prologicdirectories08:48
prologicwhich aren't actually diretories as such08:49
prologicthye represenations of such08:49
prologicand urls and resources and urls that point to other resources08:49
prologicthe way max-depth works in spyda is as you'd expect08:49
prologicyou start from your starting url08:49
prologiceverytime we follow a new url08:49
prologicwe increment the depth by 108:49
prologicbut I think there might be a flaw09:06
prologicsuch that it doesn't keep a track of how far away from the original starting url it's followed09:06
Romstersigh i give up explaining09:41
Romsterwhat i need is to beable to stop at 1 or 2 levels deep not literations09:42
Romstersay there us a url of http://foo.com/downloads.html parse that then follow say http://downlaods.foo.com/{a,b,c,d} each directory on the url or url to another site has a bunch of files in it. to me that is following a tree09:57
Romsterto add to that directories and other html pages as in the path of the url10:14
*** edk has quit IRC10:21
*** Romster has quit IRC10:25
*** Romster has joined #circuits10:39
kdb_Yo romster10:39
kdbHowdy romster10:39
*** edk has joined #circuits11:08
kdb_Hi edk11:08
kdbYo edk11:08
*** kdb__ has joined #circuits12:07
kdb_Howdy kdb__12:07
*** kdb has quit IRC12:14
*** irclogger_ has joined #circuits14:42
kdb_Hello irclogger_14:42
kdb__Heya irclogger_14:42
*** Guest13280 has joined #circuits14:44
kdb_Yo guest1328014:44
kdb__Hi guest1328014:44
*** c45y has quit IRC18:54
*** c45y has joined #circuits18:56
kdb_Hello c45y18:56
kdb__Hey c45y18:56
*** bleep has quit IRC19:39
*** FSX has joined #circuits19:39
kdb_Howdy fsx19:39
kdb__Hey fsx19:39
FSXtwo bots?19:45
*** FSX has quit IRC20:55
*** FSX has joined #circuits21:10
kdb_Welcome back fsx :)21:10
kdb__Welcome back fsx :)21:10
FSXServer doesn't seem to be stable.21:10
Guest13280FSX, eh?23:40
*** prologic has quit IRC23:40
*** prologic has joined #circuits23:40
kdb_Yo prologic23:40
kdb__Hi prologic23:40
prologicFSX, eh?23:40

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!