IRC Logs for #circuits Friday, 2014-04-18

circuits-ciProject spyda » pypy,Sliave build #2: STILL FAILING in 30 sec: http://localhost:49155/job/spyda/TOXENV=pypy,label=Sliave/2/00:07
circuits-ci* James Mills <>: Automated merge with
circuits-ci* James Mills <>: Added --cov options to tox00:07
circuits-ciProject pymills » py27,Sliave build #2: STILL FAILING in 1 min 2 sec: http://localhost:49155/job/pymills/TOXENV=py27,label=Sliave/2/00:07
circuits-ciJames Mills <>: Add --cov options to tox00:07
circuits-ciProject spyda » py26,Sliave build #2: STILL FAILING in 1 min 0 sec: http://localhost:49155/job/spyda/TOXENV=py26,label=Sliave/2/00:07
circuits-ci* James Mills <>: Automated merge with
circuits-ci* James Mills <>: Added --cov options to tox00:07
circuits-ciProject spyda » py27,Sliave build #2: STILL FAILING in 1 min 0 sec: http://localhost:49155/job/spyda/TOXENV=py27,label=Sliave/2/00:07
circuits-ci* James Mills <>: Automated merge with
circuits-ci* James Mills <>: Added --cov options to tox00:07
circuits-ciProject pymills » py26,Sliave build #2: STILL FAILING in 1 min 6 sec: http://localhost:49155/job/pymills/TOXENV=py26,label=Sliave/2/00:07
circuits-ciJames Mills <>: Add --cov options to tox00:07
circuits-ciProject mio-lang » py27,Sliave build #2: STILL FAILING in 1 min 58 sec: http://localhost:49155/job/mio-lang/TOXENV=py27,label=Sliave/2/00:08
circuits-ciJames Mills <>: Added --cov options to tox00:08
Romster54 urls in 1640.16s (0/s) CPU: 1190.69s00:37
*** bleep has quit IRC01:16
*** circuits-ci has quit IRC02:19
*** circuits-ci has joined #circuits02:19
kdbWelcome back circuits-ci :)02:19
*** koobs has quit IRC02:25
*** circuits-ci has quit IRC02:31
*** circuits-ci has joined #circuits02:32
circuits-ciProject pymills » py26,Slave build #4: FAILURE in 51 sec: http://localhost:49155/job/pymills/TOXENV=py26,label=Slave/4/02:43
circuits-ciProject pymills » py27,Slave build #4: FAILURE in 1 min 35 sec: http://localhost:49155/job/pymills/TOXENV=py27,label=Slave/4/02:43
circuits-ciProject spyda » pypy,Slave build #4: FAILURE in 1 min 43 sec: http://localhost:49155/job/spyda/TOXENV=pypy,label=Slave/4/02:44
circuits-ciProject spyda » py27,Slave build #4: FAILURE in 1 min 58 sec: http://localhost:49155/job/spyda/TOXENV=py27,label=Slave/4/02:45
circuits-ciProject spyda » py26,Slave build #4: FAILURE in 3 min 59 sec: http://localhost:49155/job/spyda/TOXENV=py26,label=Slave/4/02:47
circuits-ciYippee, build fixed!02:47
circuits-ciProject pymills » py26,Slave build #5: FIXED in 1 min 23 sec: http://localhost:49155/job/pymills/TOXENV=py26,label=Slave/5/02:47
circuits-ciJames Mills <>: Added coverage configuration02:47
circuits-ciYippee, build fixed!02:49
circuits-ciProject pymills » py27,Slave build #5: FIXED in 4 min 4 sec: http://localhost:49155/job/pymills/TOXENV=py27,label=Slave/5/02:49
circuits-ciJames Mills <>: Added coverage configuration02:49
circuits-ciProject mio-lang » Slave,py27 build #3: FAILURE in 2 min 3 sec: http://localhost:49155/job/mio-lang/Slave=Slave,TOXENV=py27/3/02:53
*** koobs has joined #circuits02:55
kdbHey koobs02:55
circuits-ciProject mio-lang » Slave,py26 build #3: FAILURE in 5 min 8 sec: http://localhost:49155/job/mio-lang/Slave=Slave,TOXENV=py26/3/02:57
circuits-ciProject spyda » py27,Slave build #5: STILL FAILING in 54 sec: http://localhost:49155/job/spyda/TOXENV=py27,label=Slave/5/03:00
circuits-ciProject spyda » py26,Slave build #5: STILL FAILING in 2 min 38 sec: http://localhost:49155/job/spyda/TOXENV=py26,label=Slave/5/03:02
Romsterprologic, damn i really do need that bug for using files for white lists, i'm gonna be making a massive list of white lists for soruceforge to crawl03:03
Romsterand hit up every project page that is listed in Pkgfile's03:03
Romsterthis is gonna be more effort than i first thought.03:04
prologicIsn't there an easier way to do this? :)03:07
prologicI will fix that bug soon03:07
Romsteri'm trying to find it.03:09
Romsteri need to resttrict my searches more, i don't want to spider the entire sourceforge url.03:10
Romsterwhitelisting only lets it follow those paths correct?03:10
Romsterand restricting mime types is fine not sure what mime types php and other such site site spit out for dynamic content is it still texx/html ?03:14
prologicit's blacklist then whitelist03:14
prologicso if something is blacklsited03:14
prologicit'll check if it's whitelisted03:15
prologicwhitelisting (with no blacklist) has no effect03:15
prologicmy plan was to do a head on the resource03:15
Romsterright it's like deny or allow in iptables for blacklist and whitelist03:15
prologiccheck it's content-type03:15
prologicif it's not something that may contain uris03:15
prologictext/html, text/xml03:15
prologicthen we skip it03:15
Romsterall i am interesed in is finding the download file listings off each project site.03:16
Romsterof interested directories/project names.03:16
Romsterand not crawling needlessly03:16
Romsterand spit out all the files found.03:16
Romsterthen i can use my tools to parse it.03:17
prologicWhat do you think about my offering hsoted jenkins03:17
prologicat $1/month03:17
circuits-ciProject spyda » py27,Slave build #6: STILL FAILING in 1 min 33 sec: http://localhost:49155/job/spyda/TOXENV=py27,label=Slave/6/03:17
Romsteri might have to use the crawl library directly to get exactly what i want but the crawl command will do for now.03:17
prologicthey are identical in terms of behavior03:18
Romsterjenkins is a build bot right for projects?03:18
circuits-ciProject spyda » py26,Slave build #6: STILL FAILING in 2 min 51 sec: http://localhost:49155/job/spyda/TOXENV=py26,label=Slave/6/03:19
prologica continuous integration server03:23
prologic@spell continuous03:23
kdbcontinuous is spelled correctly.03:23
Romsteri could use that for crux builds but then i got a local mini distcc cluster already03:23
prologicyou can still use a dictcc cluster with jenkins03:23
prologicI'm going to do this myself soon03:23
prologictime permitting03:23
*** jpenny_ has joined #circuits03:24
kdbYo jpenny_03:24
prologici.e: I'm planning on having a:03:24
Romsteri might try out icecream i think it has advantages over distcc03:24
prologicwhere I continuously publish crux packages to03:24
prologicso if you'd like to help with said efforts03:24
Romsternot much point in me doing the same as you are doing. even though that is one of my many goals.03:24
prologicthen just wait for me to complete it :)03:24
prologicI basically want full crux binary package support03:25
prologicwith continuous builds03:25
Romsterwhat i'm doing is continuously checking file versions.03:25
prologicso that my docker containers can easily just pkgadd anything they might want/need03:25
Romsterso ports can be updated more regularly with very little effort.03:25
Romsterall that needs to be done then is fix build failures and test new versions03:26
Romsteras well as me generating meta4 files for each new release.03:26
*** jpenny has quit IRC03:27
Romsterso that's my first goal sort version numbers and you've seen me do this already. second goal fetch all project related files on a regular bases03:28
Romsterand your spider is ideal for this.03:29
Romsteri'd like to set it up as a cron job when it's ready03:29
Romsteror even better have circuits fire off events to spider said urls.03:29
Romsteri don't want to get too far ahead and i know mirrorbrains does some of what i want now.03:30
Romsterthere is no decent version sorting library out there, yet many ask for how to do it. so i'm making one.03:31
Romsterfor my needs and others.03:31
Romsterhopefully its not too difficult for me to do this.03:32
Romsteri'm no programmer.03:32
Romsteryou have a few things to consider when offering binary ports for crux.03:36
Romsterconsidering we don't have strict dependencies for versions.03:36
RomsterABI changes and such03:36
Romstermy idea was something like how git tells the server what files i have and what it has and it makes a delta of that and sends it to me.03:37
Romstermy idea is to have a crux script send the installed package list and versions setup an environment for that, add in programs they requested send built packages to them and make a hash of each built package and a table so it'll eventually know what combinations of versions of packages allow a already built package to run.03:38
circuits-ciYippee, build fixed!03:47
circuits-ciProject spyda » py27,Slave build #7: FIXED in 1 min 45 sec: http://localhost:49155/job/spyda/TOXENV=py27,label=Slave/7/03:47
circuits-ci* James Mills <>: Fixed all tests. Ported tests to circuits 3.003:47
circuits-ci* James Mills <>: Updated to circuits 3.x03:47
circuits-ci* James Mills <>: Fixed circuits 3.0 compatibility for tests03:47
circuits-ciYippee, build fixed!03:48
circuits-ciProject spyda » py26,Slave build #7: FIXED in 3 min 15 sec: http://localhost:49155/job/spyda/TOXENV=py26,label=Slave/7/03:48
circuits-ci* James Mills <>: Fixed all tests. Ported tests to circuits 3.003:48
circuits-ci* James Mills <>: Updated to circuits 3.x03:48
circuits-ci* James Mills <>: Fixed circuits 3.0 compatibility for tests03:48
circuits-ciProject mio-lang » Slave,py27 build #4: STILL FAILING in 2 min 0 sec: http://localhost:49155/job/mio-lang/Slave=Slave,TOXENV=py27/4/04:06
circuits-ci* James Mills <>: mio's development only supports Python 2.7 at this time04:06
circuits-ci* James Mills <>: Added Python 2.6 to tox configuration04:06
circuits-ciProject mio-lang » Slave,py27 build #5: STILL FAILING in 1 min 58 sec: http://localhost:49155/job/mio-lang/Slave=Slave,TOXENV=py27/5/04:15
circuits-ciJames Mills <>: Fixed Traits test04:15
Romsterprologic, liek ic an do this in a few seconds04:38
Romstercrawl -v -b ".*" -w "^http\:\/\/clementine-player\.googlecode\.com\/files\/.*$" -w "^http\:\/\/alleg\.sourceforge\.net.*$" -p "^.*\.tar\.*$" > clementine-urls.log 2> log04:39
Romsterctrl+c it after a few seconds and04:39
Romsteregrep "\.tar\." log |cut -d' ' -f404:39
Romsterwhat i'm after.04:39
Romsteri was hoping crawl depth would be like 2 or 3 levels deep to find the files i'm interested in04:40
Romsterit definitely needs a mime restriction it was downloading exe files -_-04:41
prologicso add one :)05:08
prologicyou're welcome to contribute05:08
prologicso here's what I was thinking05:08
prologicbefore the get() call05:08
prologicdo a head() on the url05:08
prologicresponse = head(...)05:09
prologicif response.headers["Content-Type"] in allowed_content_types::05:09
prologic    ...05:09
prologicsomething like this05:09
Romsterhmm i'll have to read though the source and figure it out.05:16
Romsteralso a depth of 1 gives me no output but in the verbose mode i do see what i want05:19
Romsterfrom starting at
Romsterfiles/ is one level deep no?05:19
prologicwhat's your -p/--pattern?07:09
*** circuits-ci has quit IRC07:12
*** circuits-ci has joined #circuits07:12
kdbWelcome back circuits-ci :)07:12
Romster-p "^.*\.tar\.*$"07:41
*** FSX has joined #circuits08:18
kdbHi fsx08:18
prologicRomster, hmmm08:37
prologicso what you're saying is it found the urls you're after in the verbose log08:37
prologicbut the pattern didn't match it?08:37
prologicand didn't spit it out as output?08:37
circuits-ciProject mio-lang » Slave,py27 build #6: STILL FAILING in 1 min 54 sec: http://localhost:49155/job/mio-lang/Slave=Slave,TOXENV=py27/6/08:43
circuits-ciJames Mills <>: Skip broken completer tests for now08:43
Romsterprologic, it spits it out when i don't restrict the mad depth and i let it download every darn file it finds thats not a tar it'll download .md5sums .sha256sum and now i see it'll download all the .exe files and then after many minutes later then spit out the tarball names i am after.08:48
Romsterif i set it to depth of 1 then i don't get the results no matter how long i wait for it to finish.08:49
Romsterwhile it still downloads exe files and then spits out an exception for that file.08:49
circuits-ciStarting build #7 for job mio-lang » Slave,py27 (previous build: STILL FAILING)09:02
circuits-ciYippee, build fixed!09:03
circuits-ciProject mio-lang » Slave,py27 build #7: FIXED in 2 min 7 sec: http://localhost:49155/job/mio-lang/Slave=Slave,TOXENV=py27/7/09:03
circuits-ciJames Mills <>: Skip broken completer tests for now09:03
*** kdb_ has joined #circuits09:06
kdbHi kdb_09:06
prologickdb_, load drone09:06
kdb_Loaded plugin: Drone09:06
kdbUnknown Command: _,09:06
prologickdb_, ircinfo09:06
kdb_I am kdb_!kdb@11b732643c1f on the Unknown IRC Network. Connected to Running version ircd-seven-1.1.309:06
kdbUnknown Command: _,09:06
ircnotifier231068032a4c by prologic: Updated Drone Plugin to update it's nick in more places09:09
*** kdb has quit IRC09:19
kdb_Uptime: 0+0:14:0 (CPU: 1.16s 0.14%)09:20
*** kdb_ has quit IRC09:20
*** circuits-ci has quit IRC09:21
*** kdb has joined #circuits09:22
*** kdb_ has joined #circuits09:25
kdbHi kdb_09:25
d56da9b0b231IRC: Online09:26
d56da9b0b231Web: Online09:26
kdbIRC: Online09:26
kdbWeb: Online09:26
prologic@timer 10 @timer 10 Hello!09:26
d56da9b0b231Timer set for 10s09:26
kdbTimer set for 10s09:26
d56da9b0b231@timer 10 Hello!09:26
kdb@timer 10 Hello!09:26
kdbTimer set for 10s09:26
d56da9b0b231Timer set for 10s09:26
prologicthat was slightly exciting09:27
pdurbin@timer 10 good morning12:02
d56da9b0b231Timer set for 10s12:02
kdbTimer set for 10s12:02
d56da9b0b231good morning12:02
kdbgood morning12:02
pdurbingood morning12:02
prologictried this one?12:12
prologic@timer 10 @timer 10 Hello World!12:12
d56da9b0b231Timer set for 10s12:12
kdbTimer set for 10s12:12
d56da9b0b231@timer 10 Hello World!12:12
kdb@timer 10 Hello World!12:12
kdbTimer set for 10s12:12
d56da9b0b231Timer set for 10s12:12
kdbHello World!12:12
d56da9b0b231Hello World!12:12
d56da9b0b231Uptime: 0+2:47:43 (CPU: 9.64s 0.10%)12:13
kdbUptime: 0+2:50:57 (CPU: 10.88s 0.11%)12:13
ircnotifier0c67244c6545 by prologic: Change default port in sample config back to 800012:20
*** kdb_ has joined #circuits12:32
kdbWelcome back kdb_ :)12:32
kdb_Uptime: 0+0:3:11 (CPU: 0.69s 0.36%)12:35
kdbUptime: 0+3:12:42 (CPU: 11.96s 0.10%)12:35
*** kdb_ has quit IRC12:35
*** kdb_ has joined #circuits12:35
pdurbinso many bots12:47
pdurbinwe could race 'em!12:47
*** eyecue has joined #circuits13:28
kdb_Howdy eyecue13:28
kdbHey eyecue13:28

Generated by 2.11.0 by Marius Gedminas - find it at!