IRC Logs for #circuits Friday, 2013-12-13

*** ninkotech__ has quit IRC00:01
*** ninkotech__ has joined #circuits00:02
*** ninkotech__ has quit IRC00:13
*** ninkotech__ has joined #circuits00:16
*** ninkotech__ has quit IRC00:22
*** ninkotech__ has joined #circuits00:24
*** ninkotech__ has quit IRC00:30
*** ninkotech__ has joined #circuits00:35
*** ninkotech_ has joined #circuits00:41
*** ninkotech__ has quit IRC00:42
*** ninkotech_ has quit IRC00:48
*** ninkotech_ has joined #circuits00:49
*** ninkotech_ has quit IRC01:00
*** ninkotech_ has joined #circuits01:00
*** ninkotech_ has quit IRC01:08
*** ninkotech_ has joined #circuits01:09
*** ninkotech_ has quit IRC01:14
*** ninkotech_ has joined #circuits01:15
*** ninkotech_ has quit IRC01:25
*** ninkotech_ has joined #circuits01:27
*** ninkotech_ has quit IRC01:33
*** ninkotech_ has joined #circuits01:45
*** ninkotech_ has quit IRC01:49
*** ninkotech_ has joined #circuits01:51
*** ninkotech_ has quit IRC01:55
*** ninkotech__ has joined #circuits02:01
*** ninkotech__ has quit IRC02:07
*** ninkotech__ has joined #circuits02:12
*** ninkotech__ has quit IRC02:12
*** ninkotech__ has joined #circuits02:12
*** ninkotech__ has quit IRC02:21
*** ninkotech__ has joined #circuits02:28
*** ninkotech__ has quit IRC02:32
*** ninkotech__ has joined #circuits08:29
*** ninkotech__ has quit IRC08:38
*** ninkotech__ has joined #circuits08:39
*** ninkotech__ has quit IRC08:50
*** ninkotech__ has joined #circuits08:52
*** ninkotech__ has quit IRC08:58
*** ninkotech has joined #circuits09:00
*** ninkotech has quit IRC09:09
*** ninkotech has joined #circuits09:16
*** ninkotech has quit IRC09:25
*** ninkotech has joined #circuits09:52
*** ninkotech has quit IRC10:00
*** ninkotech has joined #circuits11:15
*** litzomatic has joined #circuits12:35
litzomaticHow are ya? So what time is it there? :)12:38
prologicFri Dec 13 22:39:37 EST 201312:39
litzomaticheh. 6:30AM :)12:39
prologicSo you want to know why you should devote yourself to circutis? :)12:40
prologicheh good timezone differences :)12:40
litzomaticMainly need convincing of the event based model for the app server over the thread based model.12:40
litzomaticSeems like you need many more abstractions for the event loop, is this correct?12:41
prologicOne thing I was saying in my emails12:42
prologicis that circuits.web has very ltitle hand-written code by us12:42
prologicand it's only 6k loc12:42
prologicmost of it's modules/code borrowed and wrapped in componnets12:42
prologicour event loop is msotly what you'd expect of most event looop12:43
prologicnot too many surprises there really12:43
prologicthe key thing that sets circuits apart though is int he architecture12:43
prologiccircuits gives you the tools to decouple your code12:43
prologicfor horizontally scable applications12:43
prologicby providing registerable components12:44
prologicand a powerful message bus12:44
prologicthe async I/O components are just a bonus12:44
prologicand we implement and well test all our core components12:44
litzomaticSo if I want to hook up something like SQLAlchemy, how do I handle what they call sessions in the event loop?12:44
prologicwe even now have a twisted itnegration set of components12:44
prologicthat lets us run twisted apps on top of circuits12:44
prologicand integrate twisted protocols into circuits apps12:44
prologicwe plan to releas this along with 3.012:44
prologicwell sessions and event loops don't really mix12:45
litzomaticYeah, that looks really cool.  I don't have a good grasp of the architecture yet though, what's the best way to learn that?12:45
prologicSA uses a threading model unfortuantely12:45
prologicand most databases do not provide callback mechanisms12:45
prologicor asynchronicity12:45
prologicso sadly it has to be faked with threads12:45
prologictwisted does this with it's dbapi wrappers12:45
prologicwe don't yet have such wrapper components12:45
litzomaticYeah, I've looked at those at one point.12:46
prologicfrom a preference, I tend not to use dbs anyway12:46
litzomaticWhat do you end up using?12:46
prologicthe architecture is really simple12:46
prologicyou have one thing (provided in two ways for flexibility)12:46
prologicComponent (and BaseCompoennt)12:46
prologicthe API on this is:12:46
prologicthat's it really12:47
prologicthen there are the core primitives12:47
prologicthe message bus12:47
prologicevery component in the system has:12:47, *channels)12:47, *channels)12:47
prologic.wait(event, *channels)12:47
litzomaticSo this register / unregister just makes it callable from the event loop?12:47
prologicthen of course at your top-level component12:47
prologicthere is12:47
prologic.start()  # thread mode12:47  # main thread (typical use case)12:48
prologicand .stop()12:48
prologic.register quite litereally registers the component to the top-level manager12:48
prologicwhatever that happens to be12:48
prologicand register all it's event handlers12:48
prologicand rebuilds the event handler cache cfor fast O(1) lookups12:48
prologicwell it's almost O(1)12:49
prologicthere is a little bit of bookkeeping and overhead to provide the expressivenes, power and flexibilty12:49
prologicunlike things like zope.interfaces or twisted however12:49
prologiccircuits components are12:49
prologicand easy to follow the event flow12:50
prologicalso very dynamic12:50
litzomaticYeah, so the more stuff I have registered, the more stuff the event loop needs to check, which is why we only want to register what we need?  Do you guys register once at the start up or is there advantages to unregister and re-registering?12:50
prologice.g: hot loading of components are run-time is possible12:50
prologicregistering more components doesn't slow down the event loop12:50
prologicit affects the perforamnce none at all12:51
prologicwe go to great lengths to ensure good performance overall12:51
prologicwith high concurrency loads12:51
prologicwe don't hit high grunt/raw benchmarks per say12:51
prologicbut we can handle large concurrenct loads12:51
prologicespecially if you ensure you do your CPU boudn work in a Worker (wraps multiprocessing)12:52
litzomaticAh, so is there a point to unregister and re-register?  Just less memory overhead?12:52
prologicdepends on your applicatin12:52
prologicfor kdb ( [outdated]12:52
prologican irc bot very pluggable12:52
prologicit lets you load/unload plugins at runtime12:53
prologicwithout ever terminating the bot12:53
prologicit was developed to demonstrate this ability really12:53
litzomaticAh, so it is good for updating the server with new functinality or disabling functionality?12:53
prologicthat's one use case sure12:53
prologicyou could hot load new components into youru webapp without ever brining it down12:54
prologicI should demo this too one day12:54
prologictypically I don't do it much though12:54
prologicanyway with respect to CherryPy re circuits.web12:54
prologicthere is one thing we did that CherryPy doesn't do12:54
prologicand that is we combined both object (we call it event) dispatching12:55
prologicwith method (HTTP verb) dispatching12:55
prologicin a single default dispatcher12:55
prologicbut the dispatcher can be unregistered12:55
prologicand your own used12:55
litzomaticyeah, I never did like having to either do if conditions or switch dispatchers.12:55
prologicwe did once provide a RoutesDispatcher component12:55
prologicbut sadly the guy that wrote it did not write unit tests for it12:55
prologicso I deleted it12:55
prologicso yeah in circuits.web12:56
prologicurls are mapped quite ltierally to events and channels12:56
prologicso we build upon other lower-level componets in the framework12:56
litzomaticIn regard to routes... There are two ways to program: One there is no obvious bugs and one there is obviously no bugs, and routes is the former...12:56
prologicwhich is why the web framework is so small12:56
prologicbut yet quite feature rich12:56
prologicpersonally I don't like regex based routing12:57
prologicso I never bothred with it12:57
litzomaticOn the same page :)12:57
prologicbut feel free to ressurect it if you want :)12:57
*** ninkotech has quit IRC12:58
prologicThis is the component graph of:12:58
prologica cms/wiki writtein in circuits.web12:58
prologicI use it everywhere for all my websites12:58
litzomaticSo, would you say that circuits is better for a certain niche of application then a thread based option?12:58
prologicin terms of performance and I/O12:59
prologicbecause asynchornous I/O is far better than threaded I/O12:59
prologicbecause of the Python GIL mostly12:59
prologicbut also that multiplexing your I/O is the best strategy12:59
prologicthe only way to improve this even further12:59
prologicis to split up your backend/webapp into multiple processes12:59
*** ninkotech has joined #circuits13:00
prologiceach listening/sharing the same listening socket13:00
prologicor laod balanacing across multiple worker backends13:00
litzomaticLike massive amounts of requests but very little processing.13:00
prologicso in short - yes13:00
litzomaticActually I/O for threads in Python is just fine, concurrent processing in threads however does not work.13:00
prologicyou pass your cpu bound work/processing to Worker components13:00
prologicand let circuits handle the return responses13:00
prologiccircuits very pwoerful message bus lets you do all this seamlessly13:01
prologicfire job() event to a Worker queue13:01
prologicwait for it's completion13:01
prologicreturn the response13:01
prologicall async13:01
prologicdoes not tie up the webapp/backend at all13:01
prologicyeah the troulbe is however that I/O still requires some level of processing too :)13:02
prologicso threading I/O in Python just sucks13:02
prologicbut it's "okay" for more things13:02
prologicbut yes, CPU bound work in Python just doens't work period13:02
prologicmultiprocessing ftw13:02
litzomaticwell, when doing the I/O usually something outside of Python is doing the CPU work or its not even on the server when its over the network.13:03
prologicso yeah there are many advantages to using circuits for application development13:03
prologicperformanece, scalability13:03
litzomaticC libs can drop the GIL.13:03
prologicflexibilty, expressiveness13:03
prologicand a good solid framework :)13:03
prologicI mean things like string parsing, etc costs spu time13:03
litzomaticYeah, so the niche really is if you have low CPU overhead but tons of requests.13:05
litzomaticYeah, so the niche really is if you have low CPU overhead but tons of concurrent requests.13:05
prologicwell I woudln't say that really13:06
prologicI'd say the niche is:13:06
prologicif you want a horizontally scalable architecture13:06
prologicthat lets you handle rising complexity in your application requirements over time13:06
prologicthe circuits component library is just there to help13:06
prologic:)13:06,,, circuits.web, circuits.node, circuits.tools13:07
prologicit is a *framework* albiet a lightweight one13:07
litzomaticI am trying to think of a complex enough application where Python would come in handy.13:07
prologicwith no external dependenices and pure python13:07
litzomaticOnce it gets  fairly complex seems like event vs thread might be a bit moot.13:07
prologicis also written in circutis.web13:08
prologiccomplexity has nothing to do with performance though13:08
prologicor async vs. threads13:08
prologicI mean complexity in terms of maintainence overheads13:08
prologicas I said ealier13:08
litzomaticmaintenence of the code?13:08
prologiccircuits gives you tools to decouple your software into reusable scalable components13:08
prologicif designed correctly (most components in circuits are)13:09
prologicyou can simply rip one component our13:09
prologicand replace it with another that listens to the same events13:09
prologiccomponents are highly reusable13:09
litzomaticAh, so I know what we mean when we say maitencence (but it is the wrong word haha), its easier to iteratively improve and make reusable components.13:09
prologicand extensible without subclassing13:09
prologicso you avoid OOP MI problems13:09
prologicfor example see the graph I showed you above13:10
prologicMultiple Inheritence13:10
prologicis quite literally made up of those components13:10
prologiceach doing one part of the overall system13:10
prologicthere is a Caching component13:10
prologicLogging component13:10
prologicEnvironment component13:10
prologicetc etc13:10
prologicThe best example(s) of this are in the examples/ dir13:11
litzomaticI think I get that part now, I don't have to remember I need to mixin 4 classes.13:11
prologicand a blog tutorial/article I wrote:13:11
litzomaticFor every object.13:11
litzomaticMore like, does this app need it, and now I can use it from any handler.13:12
prologicyou don't bother with classes/subclasses anymore with circuits13:12
prologicyou just register the component syou need somewhere13:12
prologicand let the rest get taken care of13:12
prologicsome config13:12
prologicmost if not all compnoents in circuits13:13
prologicare in fact made up of smaller ones13:13
prologicbut not using subclassing13:13
prologicby registering them13:13
prologichorizontal scalable architecture13:13
litzomaticInteresting. So how does this work if two have the same dependency?13:13
prologicwe provide separation of concerns via channels13:14
prologicif two or more components are regsitered with the same event handlers13:14
prologicyou can separate them out into logical groups13:14
prologici.e: run two irc bots at once in a single process13:14
prologicon separate channels13:14
prologicrun two or more web servers apps on a single process13:14
prologice.g: a https version and a http version13:14
prologicor a frontend app and a backend restful api app13:14
prologicthe psosiblility are endless13:14
litzomaticCool, so how do I take avantage of multiple processes with circuits?13:15
prologicwell in a number of ways13:15
prologica) start a copy of your component graph in "process" mode13:15
prologica1) optionally linking to a parent process13:16
prologicb) using the Worker component that processes job() events in it's queue and returns resutls once completed13:16
litzomaticSo its like have one event loop as a load balancer to many event loops?13:16
prologicsort of like celery, etc and other job queues13:16
prologicwell hmm13:16
prologictypical use case is this (so far):13:16
prologic(speaking a web app here):13:16
prologicfrom circuits.web import Controller, Server13:17
prologicclass Root(Controller):13:17
prologic    def index(self):13:17
prologic        return "Hello World!"13:17
prologicapp = Server(("", 8000)) + Root()13:17
prologicsay if you had a 4 core server13:17
prologiceach process would share the same listneing socket13:17
prologicand load balance requests across the proceses13:18
litzomaticSo I don't need to run this behind apache either.  Like I need to do with cherrypy :)13:18
prologicin fact13:18
prologicfor the most part13:18
prologicI mod_proxy/proxy from apache to circuits.web13:18
prologicor whatever the frontend web server is on my servers13:19
prologicnormally I use lighttpd on my production systems (outside of work)13:19
prologicat work we use centos + apache13:19
prologicbut - yes -13:19
prologicJustin (a developer who developers commercial web apps for clients)13:19
prologichs runs plain ol circuits.web13:19
prologicnothing in front13:19
litzomaticWhy even use one? For their feature set?  (like mod_pagespeed?)13:19
prologicin general we don't recommend you run circuits.web buitl apps via mod_wsgi anyway13:19
prologicbecause you loose the advantage of a running system13:20
prologiceven though circuits.web _does_ provide a set of WSGI components for iteropability with WSGI13:20
prologice.g: Gateway and Application13:20
prologicwe rearly ever use Application13:20
prologicbut is't there for completness13:20
prologictbh there is not much need to use Apache in front13:21
prologicexcept maybe for static file srving13:21
prologicbut even then I find circuits.web "fast enough"13:21
prologicso *meh* :)13:21
prologicwe plan to implement send_file into circuits soon too13:21
litzomaticYeah. Doesn't everyone server statics from the "cloud" :)13:21
prologiczero copy socket writing of files13:21
litzomaticwhat's that mean13:22
prologicfast static file serving for circutis.web13:23
prologicmaybe even as fast as apache :)13:23
prologicwe stream everything in circuits.web anyway13:23
prologicso again everything is async13:23
litzomaticAh cool.  For when I need to keep things under a more dynamic setting where a CDN is not appropriate.13:23
prologicand can handle large concurrenct loads13:23
litzomaticThis is a very good use case for ciruits.13:23
prologichave you installed dev?13:24
prologicpip install -e hg+
litzomaticNo, I tried to it wasn't working, let me try again.13:24
prologic>>> from circuits import Manager13:24
prologic>>> m = Manager()13:24
prologic>>> from circuits import Event13:24
prologic>>> m.start()13:24
litzomaticIs ciruits going to support Python3?13:24
prologicit already does13:25
prologichas done since 2.1.dev13:25
litzomaticso I should be able to just pip3 install it as well? same as above?13:25
prologicpip install -e hg+
prologicshould do :)13:26
prologicit's regularly testes against CI13:26
litzomaticYeah, many apps are not doing shared codebases.13:26
litzomaticCherryPy got it right.13:26
litzomaticor I should say packages :)13:27
prologicyeah we started the Python 3 shared codebase work quite a few years ago13:27
prologiclike I said we release very slowly13:27
prologicbut that's about to change13:27
prologicwith 3.0 we will release as often as we can13:27
litzomaticI saw you mentioned "scrum" :)13:28
litzomaticPersonally, if you aren't working for micro-managers, I think spiral is the best :)13:28
prologic*nods* :)13:28
prologicwe cater for all types of devlopments styles arella13:28
prologicwe do this in our spare time13:29
litzomaticAh, spare time... you have that in Australia?13:29
prologicwitho ur current government that might no longer ba an option13:29
prologicbut I find the time between work and family13:29
prologicI'm also designing a new programming lanauge too13:30
litzomaticHopefully better then Scala :)13:30
prologicif you're curious :)13:30
litzomaticYeah, language design in computer science has got to be one of the toughest things ever.  Most just give up and design it for the machine :)13:31
litzomaticThis way the compiler is easier to write.13:31
prologicso have I convinced you somewhat? :)13:32
prologicor have I failed miserably :)13:32
litzomaticLooks better then NodeJS ;)  You have convinced me of the architecture benefits.  I do not like having to miss out on SQLAlchemy though :(13:33
litzomaticDo you just use something like MongoDB?13:33
prologicyou don't have to13:33
prologicyou can just use SA as-is13:33
prologicand then later on wrap it nicely in components13:33
prologicyou could be the one to write db componetns for circuits13:33
prologicand contribute them to circuits.app13:33
prologicwe just never got around to it13:33
prologicbecause most databases you can just use as-is13:34
prologicsame with things like jinja2, etc13:34
prologicjust use it :)13:34
prologicyou are afterall writing regular Python13:34
prologicbut just ensuring you decouple your application design into components13:34
prologicinto reusable components13:34
prologicand taking advantage of the powerful message bus where you can/should13:34
prologicand the async I/O benefits13:34
litzomaticYeah, so with 4 servers, the way SQLAlchemy sessions work, I will only need 4 open connections?13:35
prologictbh I'm not sure :)13:36
prologicI dno't use SA all that much (if at all)13:36
prologicI'd imagine you'd get a new session per process anyway13:36
litzomaticYeah, it's great if you want to use a relational DB.  SQLAlchemy has a high level session manager and session object that maintains the connection.  This way you do not have to reconnect to the database over and over again.13:37
litzomaticIt even has several formula for maintaining that open connection robustly.13:37
prologicI'm aware :)13:37
prologicI _have_ used it once or twice13:37
prologicbut in nothing serious13:37
prologic does not use a DB at all13:38
litzomaticJust curious what you use for distributed ACID storage?13:38
prologicit's all just files on the file system13:38
prologicwith O(1) lookups of all of the data13:38
prologicfor a lot of our projects at work we use zodb :)13:38
prologicand in other projects I have just used straight sql13:38
litzomaticThat link is not working well for me13:39
prologicI think our cloud is down :/13:39
litzomaticYeah before it sept out some raw html, then it tried to download something, now it is not serving anything very fast.13:39
prologicanother problem to fix on monday13:40
prologicstupid university provided cloud(s) :/13:40
litzomaticOh strange.. I have something!13:40
prologicyeah I recall they're doing upgrades on the openstack infrastructure this weekend13:40
litzomaticits working now...13:40
prologicuseless pile of poo :(13:40
litzomaticMaybe they do there updates only when you give people links :)13:41
prologicgood lord13:41
prologicit works now13:41
prologicunrelaible data storage13:41
prologicall NFS backed :(13:41
prologicanyway dude13:41
prologicdon't fork cherrypy or circuits13:42
prologiccome help us improve circuits.web :)13:42
prologicand write some nice SA/dbapi wrapper components13:42
prologicwhich everyone can benefit from :)13:42
litzomaticSeems worthwhile, BUT13:42
litzomaticI love love love love mod_pagespeed13:42
prologicso? :)13:43
prologicbuild that too in circuits :)13:43
prologicPageSpeed component13:43
prologicthat times request to response13:43
prologiceasy as :)13:43
litzomaticHaha, mod_pagespeed is insane.  Instead of having to basicly compile all your tiny images, js, css before deployment, it does it on the fly for you.13:44
litzomaticProbably could figure out how to hook it up though13:44
litzomaticAfterall it's probably C code.13:44
prologicit compresses js/css down for you?13:46
prologicwe have plans to write such a component for circuits.web13:46
prologicthat automatically do this for any web app13:46
litzomaticYeah, but you don't have to do it as part of your deployment process.13:46
prologicyeah no I menat at runtime13:46
prologicI do this as part of deployment now for many proejcts13:46
prologicbut we were going to write a dynamic compnoent tha tdoes it live13:46
litzomaticYep, and it base64 encodes images into the CSS.13:46
prologicand then caches it13:46
prologicnot hard to write ihmo13:47
litzomaticYeah, and maybe thats all I need, it does some other stuff maybe I don't really care about.13:47
prologicI think I'm off to bed13:48
prologicFri Dec 13 23:48:51 EST 201313:48
litzomaticYeah, thanks for the talk13:48
prologicno problems!13:49
prologicit was valuable for me too13:49
litzomaticI'll definitely play with this.13:49
litzomaticI would like to simplify my stack for sure....13:49
litzomaticand this seems like a good base.13:49
prologiclet me know if anything pisses you off too much13:49
prologicwe'll fix that :)13:49
litzomatichaha ;)13:49
litzomaticGood night.13:49
*** litzomatic has quit IRC13:51
*** ninkotech_ has joined #circuits15:26
*** ninkotech has quit IRC15:30
*** ninkotech_ has quit IRC16:37
*** sz0ka has quit IRC16:37
*** pdurbin has quit IRC16:37
*** ircnotifier has quit IRC16:37
*** realzies has quit IRC16:37
*** koobs has quit IRC16:37
*** anunnaki has quit IRC16:37
*** ninkotech_ has joined #circuits16:38
*** anunnaki has joined #circuits16:38
*** sz0ka has joined #circuits16:38
*** prologic has joined #circuits16:38
*** pdurbin has joined #circuits16:38
*** ircnotifier has joined #circuits16:38
*** c45y has joined #circuits16:38
*** FSX has joined #circuits16:38
*** realzies has joined #circuits16:38
*** koobs has joined #circuits16:38
*** An_Ony_Moose has joined #circuits16:38
*** LeuX has joined #circuits16:38
*** spaceone has joined #circuits16:38
*** ChanServ has joined #circuits16:38
*** ninkotech_ has quit IRC16:43
*** sz0ka has quit IRC16:43
*** LeuX has quit IRC16:43
*** pdurbin has quit IRC16:43
*** ircnotifier has quit IRC16:43
*** FSX has quit IRC16:43
*** realzies has quit IRC16:43
*** koobs has quit IRC16:43
*** prologic has quit IRC16:43
*** An_Ony_Moose has quit IRC16:43
*** spaceone has quit IRC16:43
*** ChanServ has quit IRC16:43
*** anunnaki has quit IRC16:43
*** c45y has quit IRC16:43
*** c45y_ has joined #circuits16:54
*** anunnaki_ has joined #circuits16:54
*** ninkotech_ has joined #circuits16:54
*** sz0ka has joined #circuits16:54
*** prologic has joined #circuits16:54
*** pdurbin has joined #circuits16:54
*** ircnotifier has joined #circuits16:54
*** FSX has joined #circuits16:54
*** realzies has joined #circuits16:54
*** koobs has joined #circuits16:54
*** An_Ony_Moose has joined #circuits16:54
*** LeuX has joined #circuits16:54
*** spaceone has joined #circuits16:54
*** ChanServ has joined #circuits16:54
spaceoneprologic: hi, on?19:41
*** ninkotech has joined #circuits20:52
prologicspaceone, hi21:43
spaceoneprologic: what do you think about twisted?22:26
prologicWhat do I think about it?23:34
prologicI think it's fine if you like that sort of thing23:34
prologicwhich I don't (zope.interfaces)23:34
prologicthe level of indirection is quite hihgh23:34
prologicand twisted is a complex framework with many concepts23:34
prologicdeferreds are not a natural thing for most programmers that are new to async23:35
prologicIHMO it's better to use promises23:35
prologicwhich circuits implements23:35
spaceoneprologic: jeah23:36
spaceonei don't really like twisted23:37
spaceonebut i have to use it at work now23:37
spaceonetwisted isn't much generic23:37
spaceonei like circuits more23:37

Generated by 2.11.0 by Marius Gedminas - find it at!