| prologic | @load 9013e13aeccd3b43166a179539343ee59d8ce791 | 00:00 |
|---|---|---|
| kdb | Could not load plugin: 9013e13aeccd3b43166a179539343ee59d8ce791 Error: Attempted relative import in non-package | 00:00 |
| prologic | @rplugins add http://codepad.org/fiRk3JsS/raw.py | 00:01 |
| kdb | Remote Plugin http://codepad.org/fiRk3JsS/raw.py (0b22ae1245d6e6398af56620ec8ff98f689c3dfe) pending authorization. | 00:01 |
| prologic | @load 0b22ae1245d6e6398af56620ec8ff98f689c3dfe | 00:02 |
| kdb | Loaded plugin: Hello | 00:02 |
| prologic | @hello | 00:02 |
| kdb | Hello World! | 00:02 |
| kdb | Hello World! | 00:02 |
| prologic | @unload 0b22ae1245d6e6398af56620ec8ff98f689c3dfe | 00:02 |
| kdb | Plugin 0b22ae1245d6e6398af56620ec8ff98f689c3dfe is not loaded! | 00:02 |
| prologic | @hello | 00:02 |
| kdb | Hello World! | 00:02 |
| kdb | Hello World! | 00:02 |
| prologic | guess I’ll have to fix that inconsitent internal state | 00:02 |
| *** qwebirc72978 has joined #circuits | 00:07 | |
| kdb | Heya qwebirc72978 | 00:07 |
| qwebirc72978 | prologic, aqre you on? | 00:08 |
| *** qwebirc72978 has quit IRC | 00:09 | |
| *** marky1991 has joined #circuits | 00:09 | |
| kdb | Hello marky1991 | 00:09 |
| *** marky1991 has quit IRC | 00:09 | |
| *** marky1991 has joined #circuits | 00:09 | |
| *** marky1991 has quit IRC | 00:09 | |
| *** marky1991 has joined #circuits | 00:09 | |
| marky1991 | brb | 00:10 |
| prologic | I am | 00:10 |
| prologic | always online :) | 00:10 |
| prologic | @uptime | 00:11 |
| kdb | Uptime: 0+3:9:56 (CPU: 11.78s 0.10%) | 00:11 |
| prologic | c45y: ping? | 02:09 |
| c45y | sorry | 02:55 |
| c45y | only just got a chance to sit down at my desk | 02:55 |
| c45y | And I have 1 hour before next meeeting | 02:55 |
| c45y | whats up? | 02:55 |
| prologic | c45y: oh :) | 03:21 |
| prologic | just wondering where you’re been all day | 03:21 |
| prologic | busy i see :) | 03:21 |
| prologic | was going to see if you wanted to have lunch, etc | 03:21 |
| prologic | but I just got back from that | 03:21 |
| prologic | also I’ve implemented remote plugins as I said | 03:22 |
| c45y | I just got stolen by a contractor | 03:35 |
| c45y | I've managed to have 10 minutes of my lunch break | 03:36 |
| c45y | and have a meeting in 15 minutes | 03:36 |
| c45y | IM HUNGRY | 03:36 |
| c45y | go away life | 03:36 |
| prologic | heh | 03:38 |
| prologic | you _are_ entitled to your lunch break | 03:38 |
| prologic | :) | 03:38 |
| c45y | I am aware | 03:54 |
| c45y | however, I also need to attend these meetings | 03:54 |
| c45y | And I cannot do both at the same time | 03:55 |
| c45y | I'll just leave a bit early | 03:55 |
| prologic | @eval 60*15 | 03:59 |
| prologic | @uptime | 03:59 |
| prologic | wt | 03:59 |
| prologic | f | 03:59 |
| prologic | kdb: uptime | 03:59 |
| kdb | 900 | 04:01 |
| kdb | Uptime: 0+6:59:46 (CPU: 25.38s 0.10%) | 04:01 |
| kdb | Uptime: 0+6:59:46 (CPU: 25.38s 0.10%) | 04:01 |
| prologic | oh geez christ | 04:02 |
| prologic | wtf is with FreeNode | 04:02 |
| prologic | @commands | 04:12 |
| kdb | All available commands: load rehash unload unregister reload plugins unregister google uptime errors unregister graph inspect mstats version nstats events cstats unregister info commands help unregister rplugins unregister status unregister rlog unregister channels part join unregister host resolve unregister weather unregister timer unregister sum eval unregister beat itime status quit unregister die ircinfo jump nick unregister say hello | 04:12 |
| ircnotifier | d8ad38ef8f21 by prologic: Remove the implicit default unregister event handler from all components. | 04:27 |
| ircnotifier | 5631bfb9e442 by prologic: Remove the unregister() event class as well | 04:27 |
| ircnotifier | 67aa7ff70c50 by prologic: Fix broken tests.tools.test_tools | 04:27 |
| ircnotifier | d12b50a40744 by prologic: Code cleanup | 04:27 |
| ircnotifier | 98b94dda6281 by prologic: Fixed formatting of errors in Debugger for the error type | 04:27 |
| ircnotifier | daa5acc26cc6 by prologic: Updated Developer Docs. Fixes Issue #95 | 04:34 |
| ircnotifier | 0449eec7ea61 by prologic: Modified the circuits.web Logger to use the response_success event. Fixes Issue #94 | 04:34 |
| ircnotifier | fccf4d76fb46 by prologic: Documented Development Standards we use and should be using | 04:34 |
| ircnotifier | eb2762293b52 by prologic: Merged with fccf4d76fb46 | 04:34 |
| ircnotifier | 792bea1482f2 by prologic: Merged with 0449eec7ea61 | 04:34 |
| *** circuits_github has joined #circuits | 04:35 | |
| circuits_github | [circuits] therealprologic created Issue94 at 45fe299 (+0 new commits): http://git.io/vRbXPQ | 04:35 |
| *** circuits_github has left #circuits () | 04:35 | |
| kdb | Yo circuits_github | 04:35 |
| prologic | marky1991: you pinged me earlier? | 05:47 |
| prologic | what a day | 08:18 |
| prologic | c45y, you around? | 08:18 |
| Romster | prologic, http://pastebin.com/j6ueB0K2 now that's what i wanted. | 08:30 |
| Romster | seems it /does/ take forever to get to the directories i am interested in. is there any way to restrict the indexing more to speed it up? | 08:31 |
| prologic | not really | 08:35 |
| prologic | only with blacklist/whitelist of urls to follow | 08:35 |
| prologic | problem is it still has to a) parse the document and b) decide what to do | 08:35 |
| prologic | the real problem is: | 08:35 |
| prologic | no async i/o or multiprocessing | 08:35 |
| prologic | I did have a sort of half working circuits prototype working | 08:35 |
| prologic | but it wasn't quite prefect so I abanonded | 08:36 |
| prologic | but it was orders of matnitude faster | 08:36 |
| prologic | perhaps I'll try to write a new crawler in circuits again when I have the energy/time | 08:36 |
| Romster | http://pastebin.com/ibU0xgbv with --head 1 | 08:37 |
| prologic | or you could have a go at writing one in circuits | 08:38 |
| prologic | you'd need to use circuits.web.client | 08:38 |
| Romster | see what i'm getting at, i'll be constantly polling sites of files for Pkgfile projects. auto download the files test the sums/signatures off the site and build ports in docker when i get that far. | 08:38 |
| prologic | and circuits.Worker for parsing | 08:38 |
| prologic | it's not all that hard | 08:38 |
| prologic | ahh | 08:38 |
| prologic | yes well that's one thing circuits is good at really | 08:38 |
| prologic | if you want to setup a daemon that pols things on regular intervals | 08:38 |
| prologic | do some "work" | 08:38 |
| prologic | and so forth | 08:38 |
| Romster | and i'm going to collect urls that pass off mirrors to add to meta4 files. | 08:38 |
| prologic | look at the way kdb's redesign has taken place | 08:39 |
| Romster | yep there will be lots of opportunity for events in what i am doing. | 08:39 |
| prologic | almost every aspect of it is completely asynchronous and event-driven and extensible via events | 08:39 |
| prologic | speaking of which | 08:39 |
| Romster | and i like your event circuits framework | 08:39 |
| prologic | you can now add plugins to kdb remotely | 08:40 |
| Romster | saw that. | 08:40 |
| prologic | via any gist or raw pastebin or codepad like service | 08:40 |
| prologic | rplugins add <url> | 08:40 |
| prologic | of course it requires authorization | 08:40 |
| Romster | ya know i don't think any bot or service has ever had that feature. | 08:40 |
| prologic | of which I only know the authorization password at this time :) | 08:40 |
| prologic | you should try it out | 08:40 |
| prologic | write a quick plugin to see what it's like :) | 08:41 |
| Romster | but you'd got to be super careful on any possible bugs to get around the security. | 08:41 |
| Romster | at some point i want to setup kdb as well to notify when new versions are out and success failure of building the Pkgfiles | 08:41 |
| Romster | also with this filter-tool i can also keep like the last 3 versions of files to clean up | 08:42 |
| Romster | still need to figure out different major versons like keeping 3.1.2 and 4.1.4 | 08:43 |
| prologic | you already can | 08:44 |
| prologic | docker run -i -t prologic/kdb | 08:44 |
| Romster | well head 3 will show the last 3 lines i could add in a major 3 or something to keep the last 3 1.2.3 2.3.4 and 3.4.5 | 08:45 |
| Romster | yeah i know kdb is ready | 08:45 |
| Romster | when i get that far that i have a working searchable database i want to docer it all and put it up for versionsort.com | 08:46 |
| Romster | that is my goal | 08:46 |
| Romster | i don't beleave anyone has done exactly what i am doing. mirror brains comes close to the file distrobution meta4 generation, last version stuff exists for windows but not for linux | 08:46 |
| Romster | rss feeds for packagers to listen to there project versions. | 08:47 |
| Romster | i know there is mailing lists and everything but not one central site that offers this. | 08:48 |
| Romster | so here i am hoping to make a mark on the net. | 08:48 |
| Romster | and help everyone out. | 08:48 |
| Romster | i could have circuits fire off that spider to do its thing and return the results as i'm doing other stuff and stagger checking all the projects so i'm not running it all in one hit but spread out over a 24 hour day. | 08:50 |
| Romster | and make the filter-tool work with circuits as a module. | 08:51 |
| Romster | real question is am i crazy doing something this big | 08:52 |
| Romster | i've been dabbling with all the possibilities and what is worth doing for a number of years | 08:52 |
| Romster | i might mess with pastebin on kdb later on. | 08:56 |
| Romster | right now i'm in to getting my pet project functional | 08:56 |
| Romster | prologic, i can issue multiple -p options to crawl? or can i add the exta regexs by (a|b) | 10:14 |
| prologic | Romster, yes | 10:31 |
| Romster | i just used .*tar.* for now instead of messing with regex on .tar.gz .tar.Z .tgz .tar.bz2 .tbz2 .tbz .tar.lzma .tar.xz .zip .bin .rpm .deb | 10:32 |
| Romster | trying another site see how this goes. | 10:33 |
| prologic | @plugins | 10:35 |
| kdb | Plugins: channels google stats help xmlrpc core ctcp greeting rmessage broadcast dnstools weather swatch web eval timers rplugins irc hello | 10:35 |
| prologic | @hello | 10:36 |
| kdb | Hello World! | 10:36 |
| kdb | Hello World! | 10:36 |
| Romster | swatch time? | 10:36 |
| Romster | if there is any way to speed up crawl be even having 4 threads doing different queries to a server over a concurrent tcp connection, that would be nice. | 10:38 |
| Romster | never thought this spider be this slow. | 10:38 |
| Romster | like it's nearly hit 2000 queries | 10:38 |
| Romster | i guess what i really need todo is specify what i am looking for on the first level so it can transverse those only, that would be tons faster. | 10:40 |
| Romster | not just the actual results i want to find but a regex of directories/urls to whitelist. | 10:41 |
| Romster | and i see how you do it blacklist everything only allow what to look for. hmm i think i can speed this up | 10:45 |
| prologic | yeah sorry about that | 10:45 |
| prologic | it _is_ slow | 10:45 |
| prologic | th only sensible way to fix it is like I said | 10:45 |
| prologic | async + multiprocessing | 10:45 |
| prologic | but it's design does work | 10:45 |
| Romster | well i see a possible speed up with just making the whitelist instead of just whitelisting for files but the actual directories i am interested | 10:46 |
| Romster | -w "^http\:\/\/gstreamer\.freedesktop\.org/src/.*/[0-9.]*.*$" | 10:54 |
| Romster | would that work? | 10:54 |
| Romster | looks like it will | 10:55 |
| prologic | glad you're finally finding it useful | 10:57 |
| prologic | it's been used to traverse and crawl many 10's of thousands of links where I work | 10:57 |
| prologic | which we then use to ingest as data sources | 10:57 |
| Romster | i've messed with xml2 and curl and tidy and heaps of stuff but noting was 100% acurate. | 10:58 |
| Romster | i know how difficult a spider is | 10:58 |
| Romster | i could so use this in my project. | 10:58 |
| Romster | you do need to make a exception for the times where beatufalsoup fails to return a result so it, just fail logs to stderr that a url failed to parse. than this excemption i keep seeing | 10:59 |
| Romster | File "/usr/lib/python2.7/site-packages/spyda/crawler.py", line 114, in crawl | 10:59 |
| Romster | cannot concatenate 'str' and 'NoneType' objects | 11:00 |
| Romster | some sort of check and stderr the url that failed to parse. | 11:00 |
| prologic | you can ignore the errors | 11:01 |
| prologic | I believe found links go to stdout | 11:01 |
| prologic | whilst errors to stderr | 11:01 |
| prologic | so you can filter the stderr to /dev/null if you wish | 11:02 |
| prologic | otherwise you can also use spyda as a library | 11:02 |
| prologic | from spyder.crawler import crawl | 11:02 |
| Romster | yeah it does it just looks ugle how it spews this out to the terminal when you pipe stdout to a file | 11:02 |
| prologic | results = crawl(...) | 11:02 |
| Romster | oh really hmm | 11:02 |
| prologic | yes really | 11:03 |
| prologic | don't forget there is also extract | 11:03 |
| Romster | i could modify lsurl to do extra processing. | 11:03 |
| prologic | used for extraction bits of data out of html/xml documents | 11:03 |
| Romster | uh not looked at them other 2 commands | 11:03 |
| prologic | using css selectors | 11:03 |
| prologic | also a cli tool and library | 11:03 |
| prologic | and match | 11:03 |
| prologic | but match is used to match a set of data with another set of data using difflib algos | 11:03 |
| prologic | for close matching | 11:03 |
| Romster | isn't the css selectors the dom object | 11:03 |
| prologic | probably not useful for you | 11:03 |
| prologic | or maybe it is? for matching versions? :) | 11:03 |
| prologic | extract uses lxml + cssselect | 11:04 |
| prologic | so you can use css seelctors to actually extract parts of the doumcne tout | 11:04 |
| Romster | i was already dicking around with lxml2 for ages that is a great tool but i never used cssselect | 11:04 |
| prologic | spyda == crawl, extract, match | 11:05 |
| Romster | i'll keep that in mind i might find a use for that too | 11:05 |
| prologic | it's 3 tools/libs in one | 11:05 |
| Romster | nice at some point i want version sort as a lib and a tool and i'd have a final top level tool that does nearly everything | 11:06 |
| Romster | and the sorting filtering parts in there own libraries | 11:07 |
| Romster | as i can see some of the libraries of these tools will find there way into other projects | 11:08 |
| prologic | hmm so much reading to do | 11:11 |
| prologic | in real-time collaboration web based systems | 11:12 |
| prologic | kdb, read all this crap for me | 11:14 |
| kdb | Unknown Command: read | 11:14 |
| prologic | damnit | 11:14 |
| prologic | kdb learn to read | 11:14 |
| kdb | Unknown Command: learn | 11:14 |
| prologic | arggg | 11:14 |
| Romster | hmm the spider isn't hitting all them fancy index urls? | 11:26 |
| Romster | NameLast modifiedSizeDescription | 11:26 |
| Romster | hehe kdb | 11:27 |
| Romster | crawl -b ".*" -w "^http\:\/\/gstreamer\.freedesktop\.org/src/.*$" -w ".*/.*tar.*$" http://gstreamer.freedesktop.org/src/ -p "^.*tar.*$" > urls_gstreamer_new | 11:27 |
| pdurbin | prologic: bot goes offline silently · Issue #10 · shaunagm/oh-irc-bot - https://github.com/shaunagm/oh-irc-bot/issues/10#issuecomment-40470999 | 11:27 |
| Romster | i run that for a bit then ctrl^c | 11:27 |
| Romster | and it takes forever to look though http://gstreamer.freedesktop.org/src/ | 11:28 |
| Romster | and it's only like 2 levels deep hmm | 11:28 |
| Romster | http\:\/\/gstreamer\.freedesktop\.org/src/ from tat base url i'm only looking at most 2 levels deep right? i'm ignoring the .*/.*/pre directories for this example | 11:31 |
| Romster | i don't think it works as i expect | 11:32 |
| prologic | pdurbin, thanks :) | 11:33 |
| Romster | i mean http://gstreamer.freedesktop.org/src/ i'm looking one level deep 2 tops and yet at that low depth i get no results. | 11:42 |
| Romster | takes forever to resolve what i'm after. what is it possibly following? | 11:42 |
| Romster | or is it even downloading every tar file to try and follow that? | 11:43 |
| Romster | i added -b "/?C.*" after the -w to block out the fancy indexing urls. | 11:43 |
| Romster | this algorithm is really inefficient. | 11:44 |
| Romster | i can understand taking this long on 10,000 of thousands of links but this is about max of 200 links at a depth of 2. | 11:44 |
| Romster | prologic, look at this less than 1 second http://pastebin.com/pySPyYCe | 11:52 |
| Romster | crawl -b ".*" -w "^http\:\/\/gstreamer\.freedesktop\.org/src/gst-libav/.*$" -b "/?C.*" http://gstreamer.freedesktop.org/src/gst-libav/ -p "^.*tar.*$" > urls_gstreamer_new | 11:52 |
| Romster | still running minutes later | 11:52 |
| Romster | even if i set a depth to 1 it wont show results. | 11:52 |
| Romster | or 2 for that matter | 11:52 |
| Romster | i still think there is some bug in the algorithm | 11:53 |
| Romster | or i'm using it wrong. | 11:53 |
| Romster | i have a feeling the crawling is starting at http://gstreamer.freedesktop.org and not at http://gstreamer.freedesktop.org/src/gst-libav/ | 11:55 |
| Romster | looks at -v | 12:04 |
| Romster | does spider it all fast then it hits | 12:04 |
| Romster | (P): http://gstreamer.freedesktop.org/src/gst-libav/pre/ | 12:04 |
| Romster | (W): http://gstreamer.freedesktop.org/src/gst-libav/pre/ | 12:04 |
| Romster | 200 OK application/x-md5 60 0 http://gstreamer.freedesktop.org/src/gst-libav/gst-libav-0.11.90.tar.bz2.md5 | 12:05 |
| Romster | ah this is it it's reading the md5 file so far... | 12:05 |
| Romster | mime type application/x-md5 | 12:06 |
| Romster | wonder what else it will try to read. | 12:06 |
| Romster | and the huge wait is probably from BeautifulSoup timing out? | 12:07 |
| pdurbin | prologic: thanks for the comments | 12:14 |
| prologic | heh nps ;) | 12:14 |
| Romster | prologic, how do i restrict the mime type of what the spider parses? | 12:14 |
| Romster | it's skipping the tar archives but it's going over gst-libav-0.11.90.tar.bz2.md5 and gst-libav-1.0.0.tar.xz.sha256sum | 12:15 |
| Romster | so like 2 seconds i got what i want with -v but then it sits there trying all them files. | 12:16 |
| Romster | adding a -b "\.md5$" -b "\.sha256sum$" does not skip them either. | 12:17 |
| Romster | my guess is you decided to just skip archives when spidering | 12:17 |
| Romster | than to only follow directories and html xml php other dynamic pages. | 12:18 |
| prologic | no support for that | 12:20 |
| Romster | could something be done please if it's not too much effort it's a serious slow down. | 12:21 |
| Romster | i installed with pip i got lazy any easy way to update to a dev release or i could just patch if you pastebin what i need to change. | 12:22 |
| Romster | but your highly busy so maybe i should try and see if i can implement this myself | 12:22 |
| prologic | umm | 12:30 |
| prologic | I'm not sure what you want yet | 12:30 |
| Romster | it's trying to index non html files that contain sums of files. how do i prevent this. | 12:30 |
| Romster | that's what is causing the massive amount of excepts and slow downs. | 12:31 |
| Romster | i'm surprised you never noticed this yourself on your live searches. | 12:32 |
| Romster | i'm only after looking though text/html | 12:32 |
| Romster | prologic, http://pastebin.com/2EyKeYfb | 12:34 |
| Romster | two problems 1) avoid looking though index sorting ?C=S;O=A stuff 2) avoid looking at mime types that do not contain html/xml/xhtml etc. | 12:35 |
| prologic | http://mobwrite3.appspot.com/static/demos/editor.html | 12:40 |
| Romster | 200 OK text/html;charset=UTF-8 758 5 http://gstreamer.freedesktop.org/src/gst-libav/pre/?C=D;O=D | 12:40 |
| prologic | someone test this with me :) | 12:40 |
| Romster | yeah it's hammering all them too. | 12:41 |
| Romster | ok so what do i do on that page prologic | 12:41 |
| Romster | add something and others edit too? | 12:41 |
| prologic | hehe | 12:42 |
| prologic | pretty cool | 12:42 |
| Romster | that is neat | 12:43 |
| prologic | hmm | 12:43 |
| prologic | oh is it trying to parse links from binary files | 12:43 |
| prologic | or other non-html files? | 12:43 |
| Romster | pretty much but the sha256sum and md5 sum files. | 12:59 |
| Romster | and it takes like 15 seconds on each file. | 12:59 |
| Romster | now add in like 50 of these files it's minutes later. | 12:59 |
| prologic | file an issue | 13:09 |
| prologic | I'll see about fixing it soon | 13:09 |
| prologic | on that note though I'm off to bed | 13:09 |
| prologic | Romster, ping? | 13:15 |
| prologic | http://arrow.shortcircuit.net.au:9000/?editor | 13:15 |
| prologic | try this out | 13:15 |
| prologic | runnong on my desktop here :) | 13:15 |
| *** kdb has quit IRC | 13:28 | |
| Romster | thanks prologic | 13:31 |
| *** kdb has joined #circuits | 13:58 | |
| *** kdb has quit IRC | 13:58 | |
| *** kdb has joined #circuits | 14:04 | |
| *** kdb has quit IRC | 14:04 | |
| *** kdb has joined #circuits | 14:11 | |
| *** kdb has quit IRC | 14:13 | |
| *** kdb has joined #circuits | 14:14 | |
| ircnotifier | 55f0034c3a09 by prologic: Removed erroneous file | 14:14 |
| ircnotifier | ce40e4b731ee by prologic: Updated requirements and Dockerfile | 14:14 |
| ircnotifier | 7fb26d575e65 by prologic: Fixed missing html2text and aspell-en dependency. | 14:14 |
| *** kdb has quit IRC | 14:14 | |
| ircnotifier | e884999afca4 by prologic: Re-fix sample config with rplugins password | 14:16 |
| *** kdb has joined #circuits | 14:17 | |
| ircnotifier | dc729561d25d by prologic: Updated README | 14:44 |
| ircnotifier | 317b71e00498 by prologic: Preparing 0.1.0 Release | 14:44 |
| ircnotifier | 0a25b371e79c by prologic: Added tag 0.1.0 for changeset 317b71e00498 | 14:44 |
| ircnotifier | 7702f2cabfaa by prologic: Updated API Docs | 14:44 |
| ircnotifier | 1079b8a51f85 by prologic: Fixed version | 14:44 |
| ircnotifier | c16fe95fc0b1 by prologic: Try to fix publishing docs | 14:44 |
| ircnotifier | 84938e6da246 by prologic: Added wheel to dev requirements | 14:44 |
| ircnotifier | 2bdd954b9aa1 by prologic: Removed tag 0.1.0 | 14:44 |
| ircnotifier | b1845cc3bc20 by prologic: Added tag 1.0.0 for changeset 2bdd954b9aa1 | 14:44 |
| ircnotifier | 3455010e7371 by prologic: Preparing 1.0.1 Development | 14:44 |
| ircnotifier | 981db59a5d19 by prologic: Fixed link to Docker | 14:46 |
| *** Romster has quit IRC | 15:28 | |
| *** Romster has joined #circuits | 15:29 | |
| kdb | Hi romster | 15:29 |
| *** tpelka has joined #circuits | 16:35 | |
| kdb | Hi tpelka | 16:35 |
| *** jpenny has joined #circuits | 17:15 | |
| kdb | Yo jpenny | 17:15 |
| jpenny | Hi | 17:15 |
| jpenny | Can someone tell me how to integrate a Timer into a circuits.web application? | 17:18 |
| tpelka | Hi all, is it possible to detect timeout in IRC plugin? | 17:31 |
| *** jpenny has quit IRC | 18:08 | |
| *** jpenny has joined #circuits | 18:09 | |
| prologic | hi jpenny | 19:59 |
| prologic | hang around we'll be with you soon | 19:59 |
| prologic | just having breakfast and getting ready to catch the bus to work :) | 19:59 |
| prologic | hi tpelka | 20:00 |
| prologic | tpelka, what sort of timeout? | 20:00 |
| prologic | circuits has timers | 20:00 |
| prologic | not sure what you mean exactly :) | 20:00 |
| prologic | jpenny, what are you hoping the Timer() does? | 20:00 |
| prologic | by it's very nature Timer() and a circuits.web Request/Response don't go well together | 20:01 |
| prologic | Request/response is synchronous | 20:01 |
| prologic | whilst Timer() is completely asynchrnous | 20:01 |
| prologic | you'd have to use WebSockets | 20:01 |
| prologic | or hold the Response open | 20:01 |
| prologic | but in any case it's easy in general | 20:01 |
| prologic | Timer(event, channel).register(self) | 20:02 |
| prologic | you can attach it to the Controller you're in for instnace - doesn't matter | 20:02 |
| prologic | err wait | 20:02 |
| prologic | http://circuits.readthedocs.org/en/latest/api/circuits.core.timers.html#circuits.core.timers.Timer | 20:02 |
| prologic | Timer(interval, event, *channels, **kwargs).register(...) | 20:02 |
| jpenny | I really want an SSE server (not websockets). But I need a periodic timeout event. | 20:03 |
| jpenny | OK, thanks for the above, will look. | 20:03 |
| prologic | an SSE? | 20:03 |
| jpenny | Evensource, aka server sent events | 20:03 |
| prologic | hmm i see | 20:03 |
| prologic | yes so as I said | 20:03 |
| prologic | you'll have to hold the response open | 20:03 |
| prologic | self.response.stream = True | 20:03 |
| prologic | it's all possible | 20:04 |
| prologic | but you'll have to design how that all works :) | 20:04 |
| prologic | there is an example of this | 20:04 |
| jpenny | OK, thanks | 20:04 |
| prologic | examples/web/terminal/ | 20:04 |
| jpenny | OK | 20:04 |
| tpelka | prologic, I mean for IRC, when user disconnected by timeout | 20:05 |
| jpenny | BTW, there is a lot of breakage in the examples. I suspect this is mostly redesign. But, more seriously, the telnet tutorial does not run. | 20:07 |
| prologic | tpelka, there's no difference | 20:09 |
| prologic | tpelka, you'll get a disconnected() event | 20:09 |
| prologic | tpelka, or an error() event | 20:10 |
| prologic | either way you'll get a disconnect() event | 20:10 |
| prologic | See: https://bitbucket.org/prologic/kdb/src/tip/kdb/bot.py | 20:10 |
| prologic | See: https://bitbucket.org/prologic/kdb/src/tip/kdb/core.py | 20:10 |
| prologic | jpenny, url? | 20:11 |
| prologic | just checking to make sure you're using circuits 3.0 | 20:11 |
| prologic | and not 2.1.0 | 20:11 |
| prologic | in fact I seem to recall someone here the other day was saying they were using 2.1.0 :( | 20:11 |
| prologic | please ensure you: | 20:11 |
| prologic | pip install circuits==3.0.0.dev | 20:11 |
| prologic | ideally (since 3.0 isn't out yet) | 20:11 |
| prologic | pip install hg+https://bitbucket.org/circuits/circuits#egg=circuits | 20:11 |
| prologic | https://bitbucket.org/circuits/circuits is the latest stable upcoming 3.0 codebase | 20:12 |
| prologic | If you are using 2.1.0 from PyPi for whatever reason, it is not supported and is far too old | 20:12 |
| jpenny | checking... | 20:12 |
| jpenny | OK, I think I installed from pyPi, reinstalling..... | 20:13 |
| prologic | Yeah :) | 20:14 |
| prologic | Sorry about that! | 20:14 |
| tpelka | prologic, diconnect make sense | 20:14 |
| jpenny | No problem. | 20:14 |
| prologic | even though I put circuits 3.0.0.dev up on PyPi | 20:14 |
| prologic | for some reason pip install circuits | 20:14 |
| prologic | still gets you 2.1.0 | 20:14 |
| prologic | :( | 20:14 |
| prologic | so yeah | 20:14 |
| prologic | best if you install the dev version from our repo | 20:14 |
| prologic | tpelka, disconnect() - yeah :) | 20:14 |
| prologic | tpelka, watch the events via Debugger() | 20:14 |
| prologic | it's truly entertaining :) | 20:15 |
| prologic | http://circuits.readthedocs.org/en/latest/tutorials/telnet/index.html | 20:15 |
| prologic | Also the telnet tutorial _does_ and _should_ work | 20:15 |
| prologic | But not again 2.1.0 :) | 20:15 |
| prologic | ok bbs (at work) gotta catch my bs | 20:16 |
| jpenny | thanks | 20:17 |
| jpenny | Sorry to bother you with bad version. | 20:22 |
| prologic | ®uptime | 20:31 |
| spaceone | @ | 20:31 |
| kdb | Unknown Command: | 20:31 |
| prologic | jppwnny its no problems at all! | 20:32 |
| prologic | damn tablet! | 20:32 |
| prologic | @uptime | 20:53 |
| kdb | Uptime: 0+6:36:37 (CPU: 24.46s 0.10%) | 20:53 |
| prologic | @errors | 20:54 |
| kdb | No errors | 20:54 |
| prologic | @cstats | 20:54 |
| kdb | Command Stats: ~=0 Total: 3 Top 5: uptime:1 errors:1 cstats:1 | 20:54 |
| prologic | I should write a swarm plugin to swarm across the network but at most one instance ever responds | 20:57 |
| *** tpelka has quit IRC | 21:14 | |
| prologic | morn’n all | 21:44 |
| prologic | @status | 21:47 |
| kdb | IRC: Online | 21:47 |
| kdb | Web: Online | 21:47 |
| *** kdb_ has joined #circuits | 22:16 | |
| kdb | Yo kdb_ | 22:16 |
| *** kdb_ has quit IRC | 22:17 | |
| *** kdb has quit IRC | 22:22 | |
| *** kdb has joined #circuits | 22:23 | |
| prologic | @status | 22:24 |
| kdb | IRC: Online | 22:24 |
| kdb | Web: Online | 22:24 |
| ircnotifier | e0589ccbef27 by prologic: Fixed web plugin | 22:24 |
| ircnotifier | e118fb32bcf6 by prologic: Updated Docker run command in README | 22:24 |
| *** kdb has quit IRC | 22:26 | |
| *** kdb has joined #circuits | 22:28 | |
| *** kdb has quit IRC | 22:31 | |
| *** kdb has joined #circuits | 22:32 | |
| *** kdb has quit IRC | 22:32 | |
| *** kdb has joined #circuits | 22:35 | |
| *** kdb has quit IRC | 22:36 | |
| *** kdb has joined #circuits | 22:37 | |
| *** kdb has quit IRC | 22:37 | |
| *** kdb has joined #circuits | 22:41 | |
| *** kdb has quit IRC | 22:42 | |
| *** kdb has joined #circuits | 22:42 | |
| *** kdb has quit IRC | 22:43 | |
| *** kdb has joined #circuits | 22:48 | |
| prologic | @status | 22:49 |
| kdb | IRC: Online | 22:49 |
| kdb | Web: Online | 22:49 |
| prologic | @uptime | 22:49 |
| kdb | Uptime: 0+0:1:48 (CPU: 0.50s 0.46%) | 22:49 |
| c45y | @load sys | 22:50 |
| kdb | Could not load plugin: sys Error: reload() argument must be module | 22:50 |
| prologic | hehe | 22:52 |
| prologic | you’ve been through that :) | 22:52 |
| *** kdb has quit IRC | 23:00 | |
| ircnotifier | 5aeed79c2298 by prologic: Fixed Web Plugin | 23:00 |
| ircnotifier | eaa136aecc2e by prologic: Preparing 1.0.1 Release | 23:00 |
| ircnotifier | 1d4bee8f57fa by prologic: Added tag 1.0.1 for changeset eaa136aecc2e | 23:00 |
| ircnotifier | 4b5c69b4115d by prologic: Preparing 1.0.2 Development | 23:00 |
| *** kdb has joined #circuits | 23:01 | |
| prologic | $version | 23:01 |
| prologic | @version | 23:01 |
| kdb | kdb [ Knowledge Database Bot ] v1.0.1 by James Mills - CopyRight (C) 2004-2014 by James Mills | 23:01 |
| c45y | should call it application architecture support bot | 23:02 |
| c45y | wait, thats not a.s.s. | 23:02 |
| c45y | aasb? | 23:02 |
| c45y | I give up making jokes | 23:02 |
| prologic | @nick aasb | 23:03 |
| prologic | :) | 23:03 |
| prologic | done | 23:03 |
| c45y | @nick ProServ | 23:03 |
| c45y | :( | 23:03 |
| prologic | err | 23:03 |
| prologic | aasb: uptime | 23:03 |
| aasb | Uptime: 0+0:2:42 (CPU: 0.49s 0.30%) | 23:03 |
| prologic | no it does know who it is | 23:03 |
| prologic | @ircinfo | 23:03 |
| aasb | I am aasb!kdb@bbc98bb750b6 on the Unknown IRC Network. Connected to sendak.freenode.net Running version ircd-seven-1.1.3 | 23:03 |
| c45y | its prohibited by freenode | 23:03 |
| prologic | yeah :) | 23:03 |
| c45y | @nick drugsbot | 23:04 |
| prologic | I was like wait a sec | 23:04 |
| prologic | another bug | 23:04 |
| prologic | that’s impossible :) | 23:04 |
| c45y | hmm, drugs is only disabled for channels then | 23:04 |
| prologic | heh | 23:04 |
| prologic | don’t get it Klined ;) | 23:04 |
| c45y | @nick beginnerlogic | 23:04 |
| c45y | im trying! | 23:04 |
| prologic | awww | 23:04 |
| prologic | can I at leave move it off my home dsl connection? :) | 23:04 |
| prologic | if you’re “going to try” :) | 23:04 |
| c45y | nah I'm done | 23:04 |
| prologic | lol | 23:05 |
| prologic | try to break it instead :) | 23:05 |
| prologic | it has irc, xmlrpc and web interfaces | 23:05 |
| prologic | :) | 23:05 |
| c45y | some other time, sure | 23:05 |
| prologic | haha | 23:05 |
| prologic | chicken :) | 23:05 |
| c45y | I only have half a day to get something done | 23:05 |
| prologic | just kidding :) | 23:05 |
| c45y | heading out to qut today | 23:05 |
| prologic | half a day is that all? | 23:05 |
| prologic | hmm | 23:05 |
| prologic | so thatls umm | 23:05 |
| prologic | 20 lines of new code? :) | 23:05 |
| c45y | leaving at 12 for a meeting with AAF manager | 23:05 |
| c45y | should be interesting | 23:06 |
| prologic | yeah that’s gonna be a big feature | 23:06 |
| c45y | "We stopped using AAF because it sucks" | 23:06 |
| prologic | hahaha | 23:06 |
| c45y | technically speaking | 23:06 |
| prologic | of course | 23:06 |
| prologic | yeah be sure to mention that their API(s) suck | 23:06 |
| prologic | and that it should be a lot easier than it is | 23:06 |
| c45y | I can't decide if its a waste of time or a good idea | 23:06 |
| prologic | :) | 23:06 |
| c45y | Yep for sure | 23:06 |
| prologic | or better yet | 23:06 |
| c45y | pingfederate adapters are so much easier | 23:06 |
| prologic | convince the guy thtat they should implement OpenID | 23:06 |
| prologic | or something more open and standard | 23:06 |
| c45y | saml is the standard | 23:07 |
| prologic | instead of this Shibboleth crap | 23:07 |
| prologic | or SAML | 23:07 |
| c45y | I have no problem with that | 23:07 |
| c45y | but their setup is ass | 23:07 |
| prologic | OpenID would be better | 23:07 |
| prologic | more libraries and tools for it | 23:07 |
| c45y | I like saml better | 23:07 |
| c45y | openid is going to die before too long | 23:07 |
| prologic | realy? | 23:07 |
| c45y | thats my view anyway | 23:07 |
| prologic | @google python saml | 23:07 |
| beginnerlogic | Total results: 11900 | 23:07 |
| beginnerlogic | Top 4 hits: | 23:07 |
| beginnerlogic | 1. https://github.com/onelogin/python-saml | 23:07 |
| beginnerlogic | 2. https://github.com/concordusapps/python-saml | 23:07 |
| beginnerlogic | 3. https://wiki.python.org/moin/SAML | 23:07 |
| beginnerlogic | 4. http://stackoverflow.com/questions/5056475/python-library-for-implementing-saml2-based-service-provider-and-identity-provid | 23:07 |
| beginnerlogic | For more results, see: http://www.google.com/search?oe=utf8&ie=utf8&source=uds&start=0&hl=en&q=python+saml | 23:07 |
| c45y | saml is not as easy as openid | 23:08 |
| c45y | but has more features imo | 23:08 |
| prologic | hmmm | 23:08 |
| c45y | @google saml vs openid | 23:09 |
| beginnerlogic | Total results: 3440 | 23:09 |
| beginnerlogic | Top 4 hits: | 23:09 |
| beginnerlogic | 1. http://stackoverflow.com/questions/7699200/what-is-the-difference-between-openid-and-saml | 23:09 |
| beginnerlogic | 2. http://www.softwaresecured.com/2013/07/16/federated-identities-openid-vs-saml-vs-oauth/ | 23:09 |
| beginnerlogic | 3. http://www.mutuallyhuman.com/blog/2013/05/09/choosing-an-sso-strategy-saml-vs-oauth2/ | 23:09 |
| beginnerlogic | 4. http://architects.dzone.com/articles/saml-versus-oauth-which-one | 23:09 |
| beginnerlogic | For more results, see: http://www.google.com/search?oe=utf8&ie=utf8&source=uds&start=0&hl=en&q=saml+vs+openid | 23:09 |
| prologic | hard to say I guess | 23:09 |
| prologic | dunno about the state of open auth/id (not openid specifically) | 23:09 |
| c45y | oauth then | 23:09 |
| c45y | openid is the wrong word to be using sorry | 23:09 |
| prologic | from what I’m reading I don’t get the impression OpenID is going way anytime soon | 23:11 |
| prologic | they are two perfectly valid and good approahces to the problem | 23:11 |
| prologic | So you’re correcting your statement to | 23:11 |
| c45y | the problem is not big enough to need 2 approaches | 23:11 |
| prologic | “OAuth is dying”? | 23:11 |
| prologic | Oh no I disagree | 23:12 |
| prologic | I think it is a) big enough | 23:12 |
| c45y | I'm not sure what im saying anymore | 23:12 |
| prologic | and b) more solutions are always a good thing | 23:12 |
| prologic | hehe | 23:12 |
| prologic | I’ll have to dig into SAML a bit more though ymyself | 23:12 |
| prologic | don’t know terribly much about it | 23:12 |
| prologic | I know more about OpenID | 23:12 |
| c45y | its nearly the same as openid/oauth | 23:12 |
| c45y | just single logout, and the trust is set up before the connection | 23:13 |
| c45y | I trust griffith SSO, so I know to believe things it tells me | 23:13 |
| c45y | if it was openid I couldn't really verify what it was saying was correct | 23:13 |
| c45y | so I would have to verify details on my own | 23:13 |
| prologic | well yes I agree | 23:15 |
| prologic | SAML is a bit different in that respect | 23:15 |
| prologic | where you can verify other emtadata about the authenticateee | 23:15 |
| c45y | best way to describe it to me is it is 'enterprise' | 23:15 |
| prologic | whereas not so easy to do with OpenID | 23:15 |
| prologic | the information could be false | 23:15 |
| prologic | hehe | 23:15 |
| prologic | yeah funny though the whole term “enterpsie” means absolutely nothing to me | 23:16 |
| prologic | it’s become so over used | 23:16 |
| prologic | like “cloud" | 23:16 |
| prologic | grlorrifed remote data storage | 23:16 |
| c45y | It has single logout, inherent trust and support by major idp's | 23:16 |
| *** kdb has joined #circuits | 23:38 | |
| beginnerlogic | Hello kdb | 23:38 |
| prologic | c45y: there ya go :) | 23:39 |
| prologic | I’ve spun up another kdb on my SG1 server :) | 23:39 |
| *** jpenny has quit IRC | 23:50 | |
| *** jpenny has joined #circuits | 23:52 | |
| beginnerlogic | Yo jpenny | 23:52 |
| prologic | hi again jpenny | 23:56 |
| prologic | kdb: nick kdb2 | 23:56 |
| prologic | beginnerlogic: nick kdb | 23:56 |
| prologic | @status | 23:56 |
| kdb | IRC: Online | 23:56 |
| kdb | Web: Online | 23:56 |
| prologic | kdb2: load broadcast | 23:57 |
| kdb | Unknown Command: 2: | 23:57 |
| kdb2 | Loaded plugin: Broadcast | 23:57 |
| prologic | gotta fix that :) | 23:57 |
| prologic | @status | 23:57 |
| kdb2 | IRC: Online | 23:57 |
| kdb | IRC: Online | 23:57 |
| kdb | Web: Online | 23:57 |
| prologic | @uptime | 23:57 |
| kdb2 | Unknown Command: uptime | 23:57 |
| kdb | Uptime: 0+0:57:10 (CPU: 3.96s 0.12%) | 23:57 |
| prologic | kdb2: load stats | 23:58 |
| kdb | Unknown Command: 2: | 23:58 |
| kdb2 | Loaded plugin: Stats | 23:58 |
| prologic | @uptime | 23:58 |
| kdb | Uptime: 0+0:57:28 (CPU: 3.99s 0.12%) | 23:58 |
| kdb2 | Uptime: 0+0:0:2 (CPU: 3.05s 112.31%) | 23:58 |
| prologic | @uptime | 23:58 |
| kdb | Uptime: 0+0:57:51 (CPU: 4.02s 0.12%) | 23:58 |
| kdb2 | Uptime: 0+0:0:25 (CPU: 3.09s 11.99%) | 23:58 |
Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!