IRC Logs for #circuits Wednesday, 2013-03-27

*** Osso has joined #circuits01:13
*** ronny_ has quit IRC03:17
*** ronny_ has joined #circuits03:26
*** ronny_ has quit IRC03:34
*** Osso has quit IRC05:57
*** Osso has joined #circuits09:43
*** Osso has quit IRC10:22
prologicmorning all13:30
*** ronny has joined #circuits13:51
*** ronny has quit IRC13:51
*** ronny has joined #circuits13:51
spaceoneprologic: mornin13:58
jgiorgiim quickly remembering why i hate sql15:07
*** irclogger_ has quit IRC15:23
*** irclogger_ has joined #circuits15:23
jgiorgiprologic: any particular databases you use? i'm writing data wrappers15:24
prologicI try to avoid them all :)15:29
jgiorgifair enough, i find them useful but overused and missused by most15:32
jgiorgii'm writing a wrapper to use use flat files and pickle too so databases can be avoided for most tasks15:34
prologicdatabases are a complicated topic16:03
prologicI would honestly rather you focused your efforts16:03
prologicon warpping:16:03
prologicto start with16:03
prologicif we had component wrappers for thse16:03
prologicthat would be a nice start16:04
jgiorgisorry no, looked at and hate all of the above16:04
prologicyes but wrapping is less work :)16:04
prologicbut if you want to write lots of code16:04
prologicthat's ok :)16:04
jgiorgiim just wrapping the basics so i can start reusing some old components that need a data layer16:04
prologicI'm not saying they're particularly good16:04
jgiorgibasically im just attempting to wrap find, insert/update, and delete, everything else can wait (or be done directly with the underlying driver) i just want to bring a few components out of the closet and convert them to use a universal data api16:06
jgiorgialthough wrapping sqlite is essentially wrapping dbapi16:07
prologicsee here's the thing16:22
prologicif we wrapped the DBAPI-spec16:22
prologicfor any SQL database that supported it16:22
prologicthen you'd be able to build on top of this if you wanted to16:22
prologicwe actually have a story for this16:23
prologicwrapping dbapi16:23
prologicso I do want it done :)16:23
prologicyou can even copy (more or less) the twisted implemtnation16:23
jgiorgiessentially that's what im doing on a limited scope with my sqlite wrapper16:23
jgiorgiafter that it's just a couple minor changes for any DB backend16:23
jgiorgiim basing everything on my mongo wrapper (which is an incredibly thin wrapper) so everything is going to have a mongo-ish api16:24
prologichard to judge by just talking about it16:25
prologicso I'll see what you come up with16:25
prologicand add my bits - if that's ok16:25
prologicif it's good enough and generic enough16:25
jgiorgicheck out contrib-unstable/data16:25
prologicI'll whack it into
prologictonight I will :)16:26
prologicgot a long weekend coming up16:26
prologicyay :)16:26
jgiorgii've got an untested mongo api and working on an sqlite api16:26
jgiorgiawesome, enjoy it man16:26
prologiccool sounds good16:27
prologici do like mongo btw16:27
prologicbut haven't used it in anything yet16:27
jgiorgime too, that's why i kinda followed their api16:27
jgiorgieverything is experimental atm so i welcome notes and tweaks, i just want something that's functional enough that some components can use it for data storage, advanced stuff can be added later if it's needed16:28
prologicTrying to write a web crawler using circuits16:55
prologicNot having much luck at the moment16:55
prologicdue to certain limitations16:56
prologicMy goals are:16:56
prologic1) High I/O utilization16:56
prologic2) High CPU utiliization16:56
prologici.e: I want to crawl as fast as possible16:56
prologicboth in terms of asynchronously fetching resources16:56
prologicand parsing said resource documents for links16:56
prologicI think we have basically one limitation here16:56
prologicand that is our TCPClient (or any Client socket)16:57
prologicdoes not support multiple outgoing connections at the same time16:57
prologicI think this is vital16:57
*** ronny has quit IRC17:00
jgiorgiprologic: i have an old bot that i used, i just used multiple TCPClients19:25
jgiorgiiirc i had fairly high i/o and i believe cpu was my limiting factor19:26
prologicspawning multiple TCPClients is fine19:46
prologicbut expensive19:46
prologicI want to improve the internals of TCPClient19:46
prologicto support multiple outgoing connections at once19:46
jgiorgihrmm, that would be interesting20:03
prologicyes yes it would :)20:27
prologicit would make the API of Client/Server almost identical20:27
jgiorgialways good20:38

Generated by 2.11.0 by Marius Gedminas - find it at!