IRC Logs for #circuits-dev Tuesday, 2013-04-02

*** Osso has joined #circuits-dev00:29
prologichey Osso00:30
prologicYou got some time to look at something?00:30
*** Osso has quit IRC02:49
*** Osso has joined #circuits-dev02:57
Ossomaybe04:03
prologicwith the latest circuits-dev04:09
prologiccan you have a look at https://bitbucket.org/prologic/spyda04:09
prologicspecifically circuits-crawler.py ?04:09
*** Ossoleil has joined #circuits-dev06:03
*** Osso has quit IRC06:03
prologicOsso, not sure if you've had a chance to look at circuits-crawler.py yet06:32
prologicbut I'm trying to create a web crawler (based on work I've already done for a project at work)06:32
prologicbut I want it to be very fast06:32
Ossooh nope I haven't06:32
prologicso I'm trying to combine circuits I/O06:32
prologicwith circuits Worker(s)06:32
prologicbut I must sleep now/soon :)06:32
Ossosounds good06:32
prologicbeen up configuring my new server06:32
prologicmigrating my old server to a Quad Core Xeon 8GB RAM06:33
Ossonice!06:33
prologicI'll setup nginx I think06:34
prologicwith reverse proxy06:34
Ossoyay!06:34
Ossoso is circuits-crawler.py bugging somewhere?06:34
prologicand give you guys shell access fo run up reader06:34
prologicno, it's not bugging per say06:34
prologicthe version there along with circuits-dev (tip)06:34
prologicworks as expected06:34
prologicbut I'm trying to find ways to optimize it even further06:34
prologicright now it's on par with the serial version in spyda06:35
Ossoah06:35
prologichg clone https://bitbucket.org/prologic/spyda && cd spyda && python setup.py develop06:35
prologiccrawl --help06:35
prologic./circuits-crawler.py --help06:35
prologicthe two perform roughly the same right now06:35
prologicI found 1 thing today:06:35
prologiccheck_singleton is expensive06:36
prologicand I have a sneaking suspicion that processTasks is also expensive06:36
prologicit seems the more tasks we have the slower06:36
prologicso things like call/wait aren't as quick as you'd expect06:36
prologicbasically this is a real-world task I'm trying to solve and optimize06:37
prologicso it'd be good if we could improve the circuits framework as a result06:37
prologicgreatly appreciate any input/insights you might have :)06:37
prologicbut hopefully out of it we'll have a nice (stripped down) simpler version in examples/crawler.py :)06:38
Ossowell it is run after each event so I can see it slowing down stuff06:38
*** Ossoleil has joined #circuits-dev06:43
prologicyeah I don't think that's the problem though06:46
prologicOssoleil, there?06:46
prologicyeah I don't think that's the problem though06:48
Ossouhm06:49
Ossowell06:49
prologicI've noticed that as you have call/wait tasks06:49
prologicthat it takes a while for them to finish06:49
Ossoah ok something else on top?06:49
prologicmy subjective analysis06:49
Ossook06:49
prologicyeah it seems that way06:49
Ossoneeds investigating indeed06:49
prologicI could be wrong though06:50
prologicalso I did improve the web client as you suggested06:52
prologicso you can now just do:06:52
prologicresponse = self.fire(Request("GET", "http://www.google.com/"))06:52
prologichas to still be inside a component though06:52
prologicoutside you get a generator back and have to iterate over it by hand06:52
*** Ossoleil has joined #circuits-dev06:53
prologicI must go get some sleep now06:56
prologicor I'll be dead at work :)06:56
prologicwill you have a look at it over the next few days? (time permitting?)06:56
Ossook good night!07:03
OssoI hope07:03
Ossobut wednesday is booked for me07:04
OssoI may not have time today07:04
Ossothursday might be a good day07:04
prologic<  prologic: are you online?07:06
prologic<  i just want to know why you also add the third argument peer_cert to the Request event07:06
prologic<  and especially why you set in your default dispatcher in the request handler: event.peer_cert = peer_cert07:06
prologic<  instead you could this in the HTTP _on_read: req = Request(req,resp); if hadattr(sock, getpeer): req.peer_cert = sock.getpeer()...07:06
prologic<  am i right?07:06
prologic<  also who need it?07:06
prologic<  if someone need it he could also do request.sock.getpeercert07:06
prologic<  needs*07:06
prologicI kinda agree with spaceone on this one07:07
prologicjust noticed this privmsg07:07
prologicOsso, nps :)07:07
prologicg'night!07:07
*** Osso has joined #circuits-dev07:29
*** Osso has quit IRC07:38
*** Osso has joined #circuits-dev07:45
*** Osso has quit IRC08:52
*** circuits has joined #circuits-dev18:11
*** irclogger_ has joined #circuits-dev18:52
circuitsad5462517eac by prologic: Re-added circuits.web ready message to BaseServer23:34
circuitsa9b8ea805c2f by prologic: Added very simple (basic) CRUD demo23:34
circuits3cec8c56009c by prologic: Automated merge with https://bitbucket.org/prologic/circuits-dev/23:34
*** Osso has joined #circuits-dev23:54

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!