IRC Logs for #circuits Sunday, 2014-11-16

LyndsySimonprologic: thanks! I finally got home, but still have to put both my girls to bed and get settled in before hacking on anything.01:55
prologicnps :)02:04
LyndsySimonIt's apparent to me that circuits is an implementation of a dramatically different way of thinking about software architecture. With that in mind, I'm trying to nail down exactly what to call that paradigm so I can quickly develop a mental model to fit everything into as I learn. I'm currently reading the Wikipedia article for "Flow-based Programming". It02:15
LyndsySimonseems to ring true with what I've read so far of circuit's docs. Am I on the right track?02:15
LyndsySimonArticle link: http://en.wikipedia.org/wiki/Flow-based_programming02:15
prologichmm02:20
prologicI believe I answered that last night02:20
prologicComponent Architecture02:20
prologiceverything is a loosely coupled component02:20
prologicyou compose components of simpler ones02:20
prologichere's an old visual representation of sahriswiki02:20
prologichttp://shortcircuit.net.au/~prologic/tmp/Manager.png02:20
prologiccircuits.tools.graph() is able to visually spit out a graph and image of what your system looks like02:21
prologicyeah02:21
prologicflow based programming02:21
prologicevent-driven programming02:21
prologiccomponent architecture02:22
prologicthey all have overlapping ideas02:22
prologiccircuits has a powerful message bus as such02:22
prologicto facilitate event-driven programming02:22
prologichas async I/O components (but they are exactly that -- components)02:22
prologicand implements a component architecture such that each component never needs to know about the other02:22
prologicthis leads to nice things like02:23
prologiccomposable software02:23
prologicloosely coupled functionality02:23
prologicand live/hot reloading of code/components02:23
LyndsySimonFWIW, this all seems to be the natural end result of some problems I've been wrangling with at work - mainly, how as applications grow larger and more developers contribute, abstractions start to break down and things start to be more and more tightly coupled. Models start knowing more about the objects with which they interact, etc.02:28
prologic*nods*02:30
prologicit's the same old story02:30
prologicat my new job we have the same thing02:30
prologiclarge 180k SLOC codebase, monolithic02:30
prologiclots of tightly coupled code02:30
prologicit's going to be a long haul to get that into a state of better maintainability -- despite it being written in Python :)02:30
prologiccircuits itself as a framework is directly inspired by the work of my late Prof. Geoff Dromey and his work in Genetic Software Engineering and specifically Behaviors Trees as a design pattern02:31
prologicone of the largest problems we face is software maintainability in large complex systems :)02:32
prologicZope and the ZCA is another approach02:32
prologiccircuits and it's Component(s) is another :)02:32
prologicIHMO there is no sliver bullet hwoever02:32
prologic(not yet) :)02:32
LyndsySimonIs there a paper there I could read and share?02:32
prologichmm02:33
prologicperhaps lots of papers :)02:33
prologichttp://shortcircuit.net.au/~prologic/tmp/Manager.png02:33
prologicstart here :)02:33
LyndsySimonThat diagram? Hmm. OK. At first glance, it seems to be very similar to what I saw in Sahriswiki's main.py02:35
LyndsySimonLOL. Nevermind. I see above where you said that.02:36
LyndsySimonMy head is kinda swimming at the moment, I'm a little overwhelmed wrapping my head around all this new stuff. That's a good thing, though, and I'll try to stop bothering you until it coalesces into something a little more solid.02:37
prologicit's ok I don't mind :)02:38
prologicdespite my first time presenting at PyConAU this year (2014)02:38
prologicthe slides I did are somewhat pretty good :)02:39
prologichttp://pycon.circuitsframework.com/#102:39
prologicgive a nice high level overview of what circuits is all about02:39
LyndsySimonIt's crazy to me that this has been under development for a decade, only has ~30 contributors, and I'd not heard of it before yesterday. Has it ever been presented at PyCon US?02:50
prologicno :/02:50
prologicI could probably not justify the cost myself persnoally02:51
prologicunless funded02:51
prologicI'm from AU you see :)02:51
prologicI could easily justify the cost going to PyConAU this year02:51
prologicsince it was held in my city (Brisbane)02:51
LyndsySimonI think I'm going to write my give a presentation on component architecture using circuits at work as soon as I feel I'm competent to do so. After that, there are several local dev groups I could give it to, and ultimately it would be a great one to give at various conferences here at the States.02:58
LyndsySimonI feel like once I'm able to demonstrate this, the cofounder (sorta/kinda CTO) of my employer will embrace it immediately. I want to help.02:59
prologicif you managed to accomplish that03:00
prologicwe'd greatly appreciate it :)03:00
prologicgive the project a bit more of a boost :) for sure03:00
prologicI think the biggest help we can get right now is more exposure03:01
prologicand more development work03:01
prologicmostly in the integration side of things I think03:01
prologicI also wrote (still unreleased) circuits+twisted integration03:01
LyndsySimonI've been around long enough to know better than to promise - but I have to say, I've not been this engaged in an idea since I first really grokked MVC.03:01
prologici.e: run twisted apps and protocols on top of circuits03:01
prologichttps://bitbucket.org/circuits/twistedintegration03:02
prologicyeah03:02
prologicI can understand that03:02
prologicMVC is great but doesn't solve all problems03:02
prologicand you can model just about anything on top of components IHMO03:02
prologicjust a matter of design and orchestration of events (flow-based programming)03:03
*** cfarmer has joined #circuits03:04
cfarmerHi gang, I'm still having some trouble with interprocess communication via a Bridge (when starting a Component via this.start(process=True, link=other)). I think it has something to do with the hardcoded BUFSIZE = 4096 in circuits.net.sockets (line 47). Is there a clever way to update this so that larger messages will make it through without one of these errors: UnpicklingError('unpickling stack underflow',) or UnpicklingE03:09
cfarmerrror('could not find MARK',) or EOFError()?03:09
prologicoh03:11
prologicthat old beast03:11
prologicwow03:11
prologicpickling is a pain :)03:11
prologichi cfarmer03:11
prologicclever way hmm03:12
prologicdo you have any ideas? :)03:12
prologicI'm open to suggestions03:12
prologicI find pickling to be quite poor for IPC or Inter-Node-Communications :/03:13
cfarmerwell, since i have control at both ends, i could really just inspect the message before sending, update the buffersize, and then grab it at the other end right?03:13
cfarmerprobably the more 'useful' way would be to send along the message size first as a 4 byte packet?03:13
prologicahh03:14
prologicnow we're talking :)03:14
prologicI knew we'd come up with something03:14
cfarmeri don't know a lot about this stuff i'm afraid, so i'm just kinda googling around for ideas :-p03:14
prologicI've often thought we should design a better protocol03:14
prologicyeah no I kind of liek it03:14
prologiclet's do that03:14
prologicit won't break anything either03:14
cfarmercool!03:14
prologicsince this is jsut the Bridge03:14
prologicmake it happen and I'll merge it in :)03:15
prologicit'll prevent pickling problems for sure I think03:15
prologicXXXX<packet>03:15
prologicthat way if it breaks over two packets03:15
prologicwe know how much to read in before unpickling03:15
cfarmeri'll try it out... but i'm not super confident in my skills in this area, so i'll need your careful review :-)03:16
prologicand we can get rid of the silly separator at the same time03:16
cfarmeryeh, that's the idea03:16
prologic*nods*03:16
prologicyou should need only modify circuits/core/bridge.py for this03:16
prologicI expect03:16
cfarmerok, i'll try to have something for next week03:16
prologicawesome03:17
prologicdo you want to write up an issue for this03:17
prologicfor tracking?03:17
prologicI released circuits 3.1.0 the other week03:17
prologicand the lack of issues meant the ChangeLog was kind of useless at best :)03:17
prologicSee: http://circuits.readthedocs.org/en/latest/changes.html03:17
prologicSee all the unlinked things that got fied/improved03:18
prologicI use releases: http://releases.readthedocs.org/en/latest/ to manage the ChangeLog03:18
cfarmeryup, i'll submit an issue and then a pull request when that's ready03:18
prologicso the workflow needs03:18
prologicIssue -> Fork/Branch -> Pull Request03:18
prologicand the commits should add the issue to CHANGES.rst03:19
prologicjust so release time we can release in ~5mins instead of several hours :)03:19
prologicsweet03:19
cfarmeras in, manually add an entry to CHANGES.rst?03:19
prologic*nods*03:20
cfarmersure, i'll be sure to do that :-) thanks for confirming my thoughts, if i can get this working nicely, then i'll really be off to the races!03:20
prologichttps://bitbucket.org/circuits/circuits/raw/e22bc60e09845bfa0634ac1dcd240ecc01bf8fc0/CHANGES.rst03:20
prologicpretty much is just03:21
prologic:bug:`xxx` ...03:21
prologicor :feature:`xxx` ...03:21
prologicwhen we're ready to cut a release03:21
prologicwe put :release:`x.y.z` on top03:21
prologicand push to pypi and send an email ann :)03:21
prologicwell I think you're work with the circuits Bridge has improved the way it works quite considerably :) so thanks!03:22
prologicI may yet improve circuits.node in coming releases03:22
prologicdue in part to a new proejct and use-case03:22
prologicautodock: https://bitbucket.org/prologic/autodock03:23
cfarmerah yes, nothing like a new usecase to stir things up ;-)03:23
prologicand see example plugin: https://bitbucket.org/prologic/autodock-logger03:23
prologicyeah :)03:23
prologicI wrote a whole new Node for autodock03:23
prologichttps://bitbucket.org/prologic/autodock/src/ad0b3eb4ee1c33c16cfab21b704d7fdea22c37b5/autodock/node.py?at=default03:23
prologicso far it's been good03:24
prologicit's made writing a distributed pluggable system quite easy03:24
cfarmernice03:24
prologicsee: https://bitbucket.org/prologic/autodock-logger/src/3ce8e31346433a831691e8c72b5f5a6f81cc98a6/autodock-logger?at=default03:24
prologicwell that just subclasses Plugin for convenience03:24
prologicall the magic is in there03:24
prologichttps://bitbucket.org/prologic/autodock/src/ad0b3eb4ee1c33c16cfab21b704d7fdea22c37b5/autodock/plugin.py?at=default03:25
cfarmeroooh so clean and simple!03:25
prologicyeah03:25
prologicthe transport is UDP based atm03:25
prologicand the serialization is JSON based atm03:25
prologicbut I'll have another crack at abstracting this out a bit more :)03:25
cfarmeranything that makes Docker easier to use is AOK in my book03:26
prologictrouble's always been keeping track of values, errors and feedback with any kind of ipc or node03:26
prologicyou improve some of the error stuff in the bridge right?03:27
prologicso exceptions raised in the subprocess are sent to the parent?03:27
prologichehe03:27
prologicyeah the plan with autodock is exactly that03:27
prologicor think of it as a mini-PaaS03:27
prologicautodock + autodock-hipache03:27
prologicand you have yourself virtual hosting for web containers03:28
prologicwell autodock + autodock-hipache + hipache03:28
cfarmeractually, the exception stuff was mikus i think03:29
cfarmerthat i have been using it... actually, i may have found a bug with that stuff :-p have to test to make sure i can reproduce it03:30
prologicoh that's right03:30
cfarmerwell, its getting late for me, so i think for now it's time for some desert and sleep! cheers! i'll be back soon03:31
cfarmerdessert!03:31
prologicnps03:31
prologic:)03:31
*** cfarmer has quit IRC03:37
*** cfarmer has joined #circuits04:38
*** cfarmer has quit IRC04:43
*** Romster has quit IRC08:33
*** Romster has joined #circuits10:52
*** cfarmer has joined #circuits15:35
*** cfarmer has quit IRC15:41
*** Osso has quit IRC17:31
*** Osso has joined #circuits17:33
*** Osso has quit IRC17:41
*** Osso has joined #circuits20:20
*** sloria has joined #circuits21:49
*** sloria has quit IRC22:00
*** sloria has joined #circuits22:01
*** Workster has joined #circuits22:15
*** sloria has quit IRC22:24
*** sloria has joined #circuits22:26
*** Workster_ has joined #circuits22:31
*** Workster___ has joined #circuits22:33
*** Workster has quit IRC22:35
*** Workster has joined #circuits22:36
*** Workster_ has quit IRC22:37

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!