IRC Logs for #cmvt Monday, 2013-05-13

prologicDanielBaird:  Good Morning :)00:06
prologicYou here? Or ill? :)00:06
DanielBairdhi, really here00:06
DanielBairdhave you got a list of fields that we will eventually graph on the parallel coords graph?00:20
DanielBairdi guess it's every bioclim variable00:20
DanielBairdis it *just* the bioclim vars?00:20
prologicI haven't gotten to this y et00:21
prologicI was sick all last week :)00:21
prologicdid more work on Friday on deployment side of things00:21
prologicalmost complete with this00:21
DanielBairdyeah no worries00:21
prologicworking on actual ccav deployment now that all the required infrastructure is in place and working perfectly00:21
prologiccreating apache vhost stuff now00:21
prologicwith mod_proxy setup00:22
DanielBairdvagrant can automatically run ansible playbooks, once it's working i'll be able to wire it in to the vagrantfile to all happen automatically00:22
prologicshould hopefully have a full deploy of the entire app running on nectar soon00:22
DanielBairdi'm under now the impression that this whole thing will be swallowed into a plone system, though00:23
DanielBaird* now under00:23
prologicmy plan is this btw00:29
prologicansible (all done) will build the infrastructure00:29
prologicand configure everything00:29
prologicwhilst fabric will do the actual dpeloyments00:29
prologicso we'd do:00:29
prologicfab deploy:user=foo,host=bar00:30
prologicI'm going to write it in such a way that it (by default) assumes our setup (configured by ansible)00:30
prologicthat is:00:30
prologica /webapps dir00:30
prologicwith /webapps/conf00:30
prologicand /webapps/logs00:30
prologicsymlinks to their real apache locations00:31
prologicwe'll be ssh'ing in as "webdevs" (shared user)00:31
prologicapps will run as webapps (which both apache and webdevs have full read/write access to it's files/dirs)00:31
prologicDanielBaird: i'm under now the impression that this whole thing will be swallowed into a plone system, though00:32
prologic[10:23am] DanielBaird: * now under00:32
prologicHow are you under this impression again?00:32
prologicI'm not under that impression00:32
prologicI've been told (and told again)00:32
prologicthis will run separately00:32
prologicunless someone's telling me fibs00:32
DanielBairdyeah i know, but each time i talk to andrew or um, mark? they talk about plone.  In fact when mark heard i was using require js to handle module loading, he asked for a meeting where he described how plone could already handle loading in order or something.  I think he didn't want us to spend any extra time on setting up require00:35
prologicok brb00:41
prologicI'm going to talk to people00:41
prologicthis is ridicouousl00:41
DanielBairdgood luck00:43
prologicI'm back02:27
prologicwhat a discussion :)02:28
prologicYou my friend are confused :)02:28
prologicSeems you keep getting confused between the Biodiversity VCL project02:28
prologicand the Climate Hub Visualization Tool that we're working02:28
prologicwe should probably Skype if you're here?02:28
prologicAlthough I want to get some lunch first02:29
*** DanielBaird has quit IRC02:42
*** DanielBaird has joined #cmvt02:47
*** DanielBaird has quit IRC02:51
*** DanielBaird has joined #cmvt02:52
prologichey Dan03:01
prologicdid you get what I said03:01
prologicor did you miss it in the netsplit?03:01
DanielBairdsorry back.03:05
DanielBairdi said "good luck :S", then you said "hey Dan"03:05
DanielBairdso i think i missed it03:05
DanielBairdi can check logs..03:05
DanielBairdif you're disinclined to re-rtype stuff03:06
prologicCheckt eh logs then03:06
prologicthe irc logger  but stayed with me on the same server during the split03:06
DanielBairdup to day03:06
DanielBaird* date03:06
prologicso all-in-all03:07
prologicignore and forget everything about Plone03:07
DanielBairdso, i wonder why mark wanted to tell me all about plone them03:07
DanielBaird* then03:07
prologicI had a big discussion with Mark about all the design and architectural aspects of what we're doing03:08
prologicand how it _can_ play nicely with Plone03:08
prologicor any other content management system for that matter03:08
prologicso our original plan is still sound and perfectly fine03:08
DanielBairdahh so maybe he was _offering_ plone rather than _suggesting_ plone03:08
prologicrequireJS I'd still like us to use on the Javascript side03:08
DanielBairdeverything's using require right now03:09
DanielBairdi haven't tried the r.js compiler, i presume it'll work fine03:09
prologicyeah it should03:09
DanielBairdanyway i will officially stop worrying about deployment now03:09
prologicand hopefully no changes are required when we go to deploy in "production" mode03:10
prologicyeah just concentrate on the js/css side of things ):03:10
prologicand talking to my non-existent API :)03:10
DanielBairdactually i guess i can add the r.js into fabric at any time.03:10
DanielBairdmy two minute dev cycle is edit cmd-tab fab cmd-tab reload03:11
DanielBairdso require can compile as part of the frontend build03:11
prologicyes well03:11
prologicit would be good if you could build that into the frontend build tasks soon03:11
prologicso we can actually test it all out in production mode03:11
DanielBairdyeah i'll hav a go next time i'm sick of slickgrid and it's 90's style pixel sizing03:12
DanielBairdalso, i'll wire your playbooks into the vagrantfile.03:12
DanielBairdshould the playbooks be inside the repo?03:13
prologicno I've kept them separate03:13
prologicall your Vagrantfile needs to do in it's provisioning script is:03:13
prologica) clone the playbooks repo03:13
prologicb) install ansible03:13
prologicc) run the playbooks against localhost03:13
prologici.e: run it all inside the vm itself03:13
prologicnormally however I would and (Michael Pond out sysadmin) would/will run the playbooks against an external host (the production server - which isn't ready yet)03:14
prologicbut for the sake of testing via Vagrant03:14
prologicyou could run it all up locally on the vm itself03:14
prologicagainst localhost03:14
prologicmake sense?03:14
DanielBairdi could do it that way, but vagrant let's me:03:15
DanielBaird  config.vm.provision :ansible do |ansible|03:15
DanielBaird    ansible.playbook = "playbook.yml"03:15
DanielBaird    ansible.inventory_file = "ansible_hosts"03:15
DanielBaird  end03:15
prologicvagrant integrates with ansible?03:15
prologicyou're f'n kidding? :)03:15
DanielBairdvagrant is awesome, even though it's not written in python03:16
prologicWoW! :)03:16
prologicI'm impressed they integrate with a configuration management tool03:16
prologiceven more impressed Ansible is supported03:16
prologicobviously my choice of Ansible was well founded :)03:16
prologicanyway that's good for you :)03:16
DanielBairdyeah they do puppet and chef, which are pretty popular and get talked about a lot.. i didn't notice the ansible intrgration for a while03:17
prologicahh nice03:17
prologicyeah puppet/chef are a bit more popular03:17
prologicbut often more popular != better03:17
prologicit usually means more painful in some way03:17
prologicbecause of Enterprise crap :)03:17
prologicif you're working in an Organization with a wide variety of tools, languages, with very little good working processes for managing things03:18
prologicChef/Puppet doesn't really work quite so well :)03:18
prologiclike we use Puppet here on our internal servers03:19
prologicbut we're going to be deploying to either the NECTAR cloud03:19
prologicor the AWS cloud03:19
DanielBairdare they all that different from ansible?  Chef Solo or whatever the non-central-server one is, is pretty similar i think03:19
prologicwhich we have no processes for03:19
prologicand I wasn't going to wait for any of our sysadmin team(s) to build me vms03:19
prologicand get all the required software installed, etc03:19
prologicI'd rather do that myself03:19
prologicand "get it right" :)03:19
DanielBairdyes there's something to be said for pushing changes rahter than chef's pull changes strategy03:19
prologicwell yeah03:20
prologicthat's the other reason I chose Ansible03:20
DanielBairdi don't know how puppet works, but i think chef has a client on the remote node03:20
prologicbecause of it's push nature03:20
prologicand it scales a lot better than Chef/Puppet03:20
prologicboth Chef/Puppet require clients on the target nodes03:20
prologicAnsible does not03:20
prologicbig plus ihmo03:20
DanielBairdgenerally i'd imagine the simpler the node setup, the better03:20
prologicsimpler tools often end up being the best tools03:21
prologicbecause of reduced complexity03:21
prologicand reduced learning curve03:21
prologicback to work on deploy fabric tasks03:21
DanielBairdwe can revisit the issue once we're deploying thousands of new ccav nodes a day.. we'll have some experience under out belts by then.03:21
prologicalso btw03:21
prologicassume you are not to worry about deployment with your frontend fabric tasks03:22
prologicI'll write a set of deploy taks03:22
prologicthat take the results of earlier builds03:22
prologicso it might look something like03:22
prologicfab build deploy03:22
prologicI'll work out the finer details by the end of the day hopefully03:22
DanielBairdnp, i'll maintain to do the compilation etc03:23
DanielBairdstill building into a htdocs dir?03:23
DanielBairdokay cool then you can move that dir or whatever03:23
prologicI think that's the simplest way03:23
prologicdeploy just copies htdocs somewhere03:23
prologiccreate a virtualenv on the remote system03:23
DanielBairdyes agree.  the final test of building is the htdocs dir should work on it's own03:24
prologicdoes a fab build deploy03:24
prologicand then we can do (remotely)03:24
prologicfab deploy:host=<hostname>03:24
prologicto update the deployment code03:24
prologicI suppose a remote deploy should03:25
prologicdo a hg revert -a && hg pull -u03:25
prologicbefore running the rest of it's tasks03:25
prologicwe probably should have a way to rollback a deployment update too03:26
DanielBairdfab deploy-latest?03:26
prologicwhat do you think?03:26
DanielBairdhow about having two environments set up, and have a deploy-staging03:26
prologicReckon you can whip up a quick03:26
DanielBairdi suppose you'd still want deploy-revert03:26
prologic"Site is down for maintaince. Please come back later"03:26
prologickind of page?03:26
prologicI might try to use that in a deploy03:27
prologicsome sequence of steps03:27
prologicyes I think that could be good03:27
prologiccustomizing the version we want to deploy03:27
prologicand having a staging/production site(s)03:28
prologicthey could in fact just be03:28
prologicand probably should be03:28
prologicdifferent deployments all together03:28
prologicwith different virtual environments03:28
prologicpossibly with different libraries of things03:28
prologicthere are some hard requirements of things though03:28
prologici.e: all the things the playbooks install03:29
prologicMapServer, Python, GDAL, GEOS, PROJ, etc, etc03:29
DanielBairdonce we have a branded site i'll do a "come back in a tick" page.  i'd also like to have a easily-switched-on warning thing: "Feel free to use the site but note that our hosting organisation is performing maintenance, and performance and functionality may be affected [ok]"03:29
prologicyeah sounds good03:30
prologicso I think supporting staging/production is easy03:30
prologicjust do two separate deploys03:30
prologicI think the deploy task should take in several options03:30
prologiceither a tag, or revision03:30
prologicwhere we can specify the tag/revision to update to03:30
prologicotherwise it updates to the latest revision03:30
prologicas part of the task I might take a backup of the environment03:31
DanielBairdyes that sounds good03:31
prologicso that we can support a rollback03:31
prologicwhat do you think?03:31
prologicI'd only support a single rollback though03:31
DanielBairdalso record (and show somehow) what the tag of the last deployment was, so it's easy to quickly rollback when everything blows up03:31
prologicI was thinking03:32
prologicfab deploy:host=<hostname>03:32
prologicto deploy to the latest version03:32
prologicfab deploy:rollback=yes03:32
prologicto rollback the last deploy to the backup03:32
prologicwhich then deletes the rollback backup03:32
prologicfab deploy:tag=<tag>,hostname=<hostname03:32
prologicto deploy a specifc tagged version03:33
prologicsimilar with revision=<rev>03:33
prologicand obviously this has to be preceded by a03:33
prologicfab build03:33
prologicoh it should also specify the environment03:33
prologicfab build deploy:env=<env>,host=<hostname>03:33
prologicwith optional rollback=yes/no tag=<tag> or rev=<rev>03:34
prologicsound good?03:34
prologicrfc? :)03:34
DanielBairdfab rollback03:34
prologicahh yes03:35
prologicfab rollback:host=<host>03:35
prologicrollback the last deployment03:35
prologicbeautiful :)03:35
DanielBairdwhen you want to rollback, that's when you're a bit panicky and ned ot to be simple03:35
prologicof course03:35
prologicbrilliant :)03:35
prologicany other brilliant ideas? :)03:35
prologicI've written my apache vhost template so far03:35
prologicup to writing the tasks/functions now03:36
prologicthat actually do the shit :)03:36
DanielBaird(back again, had a phone call) i guess all we need is:04:20
DanielBaird- deploy a staging version (latest, or specified) to try out04:20
DanielBaird- deploy a proper version (latest, or specified)04:20
DanielBaird- rollback the proper version and go back to what you had before04:20
DanielBairdmaybe the staging deploy can be a whole other deployment on a different VM, but it'd be nice to have a side-by-side setup on the same server.04:20
DanielBairdactually, the deploy version selection should be "current", "latest", or specific.04:20
prologicI was going to fully utilize mercurial here04:27
prologicso specifying a version is:04:27
prologicif no tag or rev specified04:27
prologicgrab the latest (i.e: tip)04:27
prologicand rollback would just look for a backup of the environment04:28
DanielBairdyeah that'll work nicely.04:31
DanielBairdso can you deploy what you currently have?04:36
DanielBairdthat's the extra bit, which i guess would mean "carefully avoid running any hg command"04:36
DanielBairdit would be bad practice, but it wouldn't be impossible that there's a bug that only shows up on the dep server, and you'd want to make a change to code then deploy to staging to test it04:37
DanielBairdwith the eventual plan to make the change properly, once you diagnose tyhe bug04:38
DanielBairdnot very important i guess, but might be easy to have a "current" version spec that skips the hg step04:38
prologicsee I don't believe in the whole test on a dev server type crap04:39
prologicyou should be able to replicate an entire environment easily04:39
DanielBairdi agree that you *should*...04:40
prologicwhich is why I've put so much effort into the ansible playbooks and fabric tasks04:40
prologicI should be able to reproduce any bug found by any user on my Mac04:40
prologicjust as easily as on the production server04:40
prologicif I can't, I've failed as a developer :)04:40
prologicwhich is why in some respects I like the whole vagrant thing04:40
prologicit's useful04:40
DanielBairdit's not that important.. a principled stand is completely enough reason to not have a deploy-current04:40
prologicI just do the same thing against a local proxmox vz container04:40
prologicthat's why the deployment tasks are going to do:04:41
prologichg revert -a && hg pull -U && hg update -C <revision or tag or tip>04:41
prologicdepending on what you want04:41
prologicso any local changes to a live system will be trashed04:42
DanielBairdthat's absolutely the right way to do it.04:42
prologicand if I catch any dev (myself or anyone) editing things live on a staging, or production environment04:42
prologicI will personally hunt them down (or myself)04:42
prologicand shoot them :)04:42
DanielBairdremember to hg-ignore the config file04:42
prologicyes of course :)04:42
prologicall of our systems we've built here thus far04:42
prologicare all developed against a live shared dev server04:42
prologicwhich I think/believe is just bullshit04:43
prologicthe first thing I did when I started here when they threw me on another  project that used some Java products like jena + vivo04:43
prologicwas write a set of ansible playbooks that replicated their dev/prod environments locally04:43
prologicor on any servers I wanted04:43
prologichaven't zero repeatability is just crap04:44
DanielBairddepends on the system i guess, but yeah you usually don't want to spend ANY minutes making something run on a dev server when it's different from the final deployment location.04:44
prologicarguably dev, staging, production04:44
DanielBairdi suppose CI is worth a bit of setup04:44
prologicshould never be so different04:44
prologicI've been in so many half baked, half arsed teams04:45
prologicI'm so sick of manual labor04:45
prologicand hand tweaking :)04:45
prologicansible + fabric all the way04:45
prologicif we change something04:45
prologicsomething in the playbacks repo04:45
prologicor the ccav rpeo needs to change04:45
DanielBairdyou only need to share a dev server if you need special functionality04:45
prologicor you want your dev team to see collected changess04:46
prologicnot to share development on :)04:46
prologicyeah CI would be good04:46
DanielBairdi think the idea of a shared dev server comes from those awful java app servers from last century04:46
prologicunfortunately I don't think we're going to have time to do any of that04:46
DanielBairdno probably not04:46
prologicas it stands right now there is very little unit or systems testing04:46
prologictesting and documentation always lags behind04:46
prologicstupid business poressures04:46
prologicI've started a framework for testing04:47
DanielBairdit's not quite so bad when your app is mainly tying together other systems.04:47
prologicand getting CI up and running is as easy as setting up a ShiningPanda account04:47
prologicI think if we have time at the end04:47
prologicI'd like to setup CI04:48
prologicand some basic unit and system tests04:48
prologicwith selenium04:48
prologicwhich shining panda has full support for04:48
prologiceven if we just have a couple of backend unit tests04:48
prologicand a couple of fronted selenium systems tests04:48
DanielBairdyes end-to-end stuff would be good04:48
prologicreally should be doing TDD here04:48
prologicbut as I've had NFI what I'm doing04:49
prologicthat's bee impossible04:49
DanielBairdi really love doing bdd/tdd but haven;t been able to do it on a work project yet04:49
prologicI think it's only possible when04:49
prologica) the business allows time for it04:49
prologicb) you know what you're doing (mostly)04:49
prologicmy previous project here was done with TDD04:49
prologicwith 100% test coverage04:50
prologicfully documented04:50
DanielBairdyou want to be good and familiar with your test library, but also you want to know where you're going.. it's hard to speculate in code while you're writing tests04:50
prologicnot really that hard04:52
prologicit's just you have to have very clear requirements and design first04:53
prologicand know what you're doing in the first place04:53
prologicthen you can do TDD04:53
prologicbut if you're new to a domain04:53
prologichave no clear requirements04:53
prologicit's too hard to do TDD04:53
DanielBairdyeah that's what i mean.. if you're dicking around trying stuff out, tdd is harder04:53
DanielBairdnot new to the domain, necvessarily, just fiddling around with something04:53
prologicdicking around == R&D :)04:53
prologicwhich unfortunately ends up being the real code anyway04:54
prologicwhen you're in a situation like we're in04:54
prologicnot knowing what we're doing04:54
prologicand not having clear requirements04:54
DanielBairdyes the stuff you *should* re-write and clean up, that you never do :(04:54
prologictechnical debt04:54
prologicyeah no I'm not necessarily new to the domain either04:54
prologicI've done Geospatial stuff before too04:54
prologicbut the lack of clear, non-ambigous requirements isn't there04:55
prologicand I've never used some of the requirements before either04:55
prologici.e: map server was pretty much decided for me04:55
prologicwhereas I've used the commercial ESRI ArcGIs before04:55
prologicso that imposed a lot of resitrctions04:55
prologicand in itself imposed a lot of design choices04:55
prologiclike: numpy, shapely, gdal, fiona, etc, etc04:56
prologicI need to stop chatting04:56
prologicand write these two damn functions :)04:56
prologicluck? nah :)04:58
prologictime is what I need!04:58
prologicand a brain that doesn't tire!04:58
prologicalso my sight back would be good too!04:58
DanielBairdyou need to get a cyberman "upgrade"04:59
prologicI see you watched Dr Who last night :)05:02
prologicI love that show!05:02
ircnotifier45eeace5ae00 by prologic: Ensure the permissions of the "webapps" home directory is 177005:03
ircnotifier49914bfdbe8e by prologic: Remove shell=/bin/false for "webapps" user05:41
ircnotifier2ac3eeac8dfb by prologic: Add a custom to /etc/profile.d that adds $HOME/bin to the PATH for all users05:41
ircnotifier382c83a83b1a by prologic: Working on deploy tasks06:27
ircnotifier644c4ab77602 by prologic: Install python packaging and some common python libraires/tolls06:35
ircnotifier15b2d463d06c by prologic: Make the webapps home a virtualenv and setup virtualenvwrapper.sh06:35
ircnotifiera3ccbde97809 by prologic: Fix up webapps virtualenv and bash profile/rc06:45
*** DanielBaird has quit IRC06:54
*** irclogger___ has quit IRC08:24
*** irclogger____ has joined #cmvt08:24
*** prologic_ has joined #cmvt08:25
*** ChanServ has quit IRC11:21
*** ircnotifier has quit IRC11:21
*** prologic has joined #cmvt11:40
*** ChanServ has joined #cmvt11:40
*** ircnotifier has joined #cmvt11:40
*** prologic_ has joined #cmvt12:12
*** prologic has quit IRC12:16
*** ircnotifier has quit IRC13:50
*** ChanServ has quit IRC13:50
*** prologic has joined #cmvt14:08
*** ChanServ has joined #cmvt14:08
*** ircnotifier has joined #cmvt14:08
*** irclogger____ has quit IRC17:31
*** irclogger_____ has joined #cmvt17:31
*** DanielBaird has joined #cmvt23:31

Generated by 2.11.0 by Marius Gedminas - find it at!