IRC Logs for #cmvt Thursday, 2013-05-02

DanielBairdstill bootstrapping ccav.. no problem so far00:30
prologicI'm working on adding roles for building mapserver and it's dependencies now00:30
DanielBairdcool.  i have a nice working d3 graph, i'll poke it into a page layout.  it's hard to resist making it better before adding it to a page00:41
DanielBairdi switched it over to splines instead of jaggy lines.  maybe i should stop there :)00:58
DanielBairdhey ansible finished with my vm.  i'll drop in and test a deployment01:02
DanielBairdafter ansible runs, i should be able to just 'ccav', right?01:06
DanielBairdno i had to run fab build to make the htdocs dir.01:08
prologicthe htdocs should have been built01:13
prologiccan you identify why it failed at all?01:13
prologicDid the ansible play succeed?01:14
prologicwith an OK at the end?01:14
prologicI've only been testing the backend python libraries and all dependencies in the end01:14
prologici.e: gdal, geos, bumpy, shapely, etc all work properly01:14
prologicit would have created (hopefully) the htdocs door in01:15
prologic /usr/local/src/ccav/htdocs01:15
prologicso you'd have to run ccav out of there for the moment01:16
prologicI haven't as such yet hooked this up in a proper virtualenv with Apache or anything yet01:16
DanielBairdPLAY RECAP ********************************************************************01:17
DanielBaird10.1.2.3                   : ok=36   changed=32   unreachable=0    failed=001:17
ircnotifierfd0e72e6e1f7 by prologic: Added and tested proj role against local proxmox01:17
prologicyeah that succeeded then01:17
prologiccheck in /usr/local/src/ccav01:17
prologicthere should be a htdocs dir01:17
prologicotherwise ansible would have reported a failure form the fab build failure01:17
prologicso I'm a little surprised01:18
prologicunless less/coffee isn't returning a failure status code on failure01:18
DanielBairdwhat's ccav's port? 9000?01:18
DanielBairdhmm well i ran ccav from /usr/..../ccav, pointing it at that copy of htdocs.. i have a redirect from my localhost 9999 to the vagrantvm:9000, but no response from there.01:21
prologicoh that's right01:21
prologicthe default dep doesn't use circuits-dev01:21
DanielBairdit serves on  as in, all IPs?01:22
ircnotifier32552e0aa8f7 by prologic: Make default bind the default and consistently so in configuration options01:22
prologicby default - yes01:23
prologicanyway I've fixed the lack of consistency01:23
prologic8000 now (on next build/deploy)01:23
DanielBairdwhen i hit localhost:9999 which is forwarded to vm:9000 it grinds for a long time .. as in, not refused or anything, just empty response01:24
DanielBairdfrom inside the vm i can wget index.html01:24
DanielBairdhmm might be something at my end. i'll check the virtualbox gui01:25
prologicnot sure :)01:27
prologicI will test it out here as well01:27
prologicsee if anything weird is going on01:27
DanielBairdit might be from the additional net interface i added.  i'll fiddle that a bit01:28
DanielBairdhmm something to do with my vagrant networking.  i never actually read the vagrant net docs, might go hav a look01:38
prologicI'm getting mapserver role written up and tested01:40
prologicI'll test ccav out in a sec01:40
DanielBairdi think it's my setup, i can't see a python -m SimpleHTTPServer either01:44
prologicI would have been surprised if it was a circuits problem01:46
prologicit's throughly well tested :)01:46
prologicfuck yeah01:46
DanielBairdi can ping my vm, just not curl from it.  weitrd01:46
prologicmap server is done!01:46
prologicdoes vagrant install one?01:46
DanielBairddunno it's whatever came on the centos base box01:46
DanielBairdi'll check it out01:46
DanielBairdknow anything about ip tables?01:49
prologicre-testing entire playbook again01:49
ircnotifiera85b81b91cd5 by prologic: Added and tested mapserver role and required dependencies01:50
ircnotifier74280e08bb42 by prologic: Added readline-devel to python deps (easier to work with on the CLI)01:50
prologiciptables -L -n -v01:50
DanielBairdi think im configged to reject anythign but ssh01:51
DanielBairdChain INPUT (policy ACCEPT 0 packets, 0 bytes)01:51
DanielBaird pkts bytes target     prot opt in     out     source               destination01:51
DanielBaird 1129 85704 ACCEPT     all  --  *      *             state RELATED,ESTABLISHED01:51
DanielBaird    2   168 ACCEPT     icmp --  *      *  
DanielBaird    0     0 ACCEPT     all  --  lo     *  
DanielBaird    4   176 ACCEPT     tcp  --  *      *             state NEW tcp dpt:2201:51
DanielBaird  109 11699 REJECT     all  --  *      *             reject-with icmp-host-prohibited01:51
DanielBairdChain FORWARD (policy ACCEPT 0 packets, 0 bytes)01:51
DanielBaird pkts bytes target     prot opt in     out     source               destination01:51
DanielBaird    0     0 REJECT     all  --  *      *             reject-with icmp-host-prohibited01:51
DanielBairdChain OUTPUT (policy ACCEPT 709 packets, 82281 bytes)01:51
DanielBaird pkts bytes target     prot opt in     out     source               destination01:51
DanielBairdi think thats the centos standard setup01:52
prologicI don't think it is tbh01:53
prologicat least not for me01:53
prologicmy blank centos has no firewall rules01:53
prologiccan you paste that output to a paste bin site01:53
prologicso I can read it better01:53
DanielBairdnah don't worry for now i'll add an accept rule as per
prologicI'd just blow away the firewall dude01:56
prologicVagrant must be setting that up01:57
prologicor it's part of the Vagrant CentOS image01:57
prologicjust do a01:57
prologiciptables -F01:57
DanielBairdyeah that centos wiki page said it was a standard setup01:57
prologicfirewalls are a stupid idea01:57
prologicperhaps standard in _some_ CentOS installs01:57
prologicin general stageful firewalls are the first thing to suffer and go down01:58
prologicthey don't really protect ;)01:58
prologicand your vm is local anyway01:58
prologiconce you've sorted out your firewall crap :)02:09
prologicI'll get you to re-test this from scratch again02:09
prologicwith the included mapserver and deps02:09
prologicbbs - lunch02:10
*** DanielBaird has joined #cmvt02:11
DanielBairdah, finally.02:24
DanielBairdhad to echo 0> /selinux/enforce02:24
DanielBairdso there's some security policy somewhere that is stopping me from connecting.02:25
DanielBairdanyway for now i'll disable selinux in my vagrant provisioning script02:26
DanielBairdi've updated ccav and playbooks repos and re-doing vagrant vm from scratch02:34
DanielBairdre-running, i mean.02:38
prologicback from lunch too02:45
prologicyeah don't do selinux02:45
prologicit's  a PITA02:45
prologicand so incomplete02:45
prologicccav just works for me02:49
prologicawesome :)02:49
prologicShould I now try and get it all working seamlessly with Apache + mod_proxy on the default port 80?02:50
prologicAnd get it all built/running inside a virtualenv02:50
DanielBairdis it going to end up in plone?  andrew and mark(?) were talking about plone as the host02:51
DanielBairdso when you vm are you using a centos iso to start with?02:51
prologicumm no02:52
prologicI don't think it'll end up in Plone02:52
prologicNah I'm using CentOS 6.3 prebuilt images/templates02:53
DanielBairdhmm i'm getting authentication errors from ansible02:53
DanielBairdi can ssh as root into the vm, that's what ansible is doing, right?02:54
prologicyou need to specify -u <user> -k02:54
prologicdoes vagrant automatically put your key on the vm?02:54
prologicdon't need -k in that case02:55
prologicbut definitely need -u <user>02:55
prologicand -s if you have to sudo up02:55
prologicI've left the user/sudo/pass all configurable from the command line now02:55
DanielBairdi'll try the non-root guy02:55
prologicrather than hard coded in the site.yml02:55
prologicso for my test test beds I've been doing:02:55
prologicansible-playbook -u root -k -l test_local_proxmox site.tml02:56
prologicand for02:56
prologicansible-playbook -u ec2-user -s -l test_jmills_nectar site.yml02:56
prologicyou'll want something similar02:56
prologicwith an appropriate group/ip entry in hosts file02:56
DanielBairdim trying ansible-playbook -s -u vagrant -k -l vagrantvm site.yml02:56
DanielBairdthe non -s version hung doing something to .ssh02:57
DanielBairdnot hung.  died.  anyway this one is working afaict02:57
prologic-u species user to connect as02:58
prologic-k prompts for password for user given by -u02:58
prologic-s tells ansible to use sudo to root02:58
DanielBairdyeah vagrant give you a user called 'vagrant' with pwd 'vagrant', the root user's password is 'vagrant', and shares your project dir into the machine at the path /vagrant.  so when you log in as vagrant, you're left in the /home/vagrant dir, and you have to cd to /vagrant to see your code.02:59
DanielBairdthe upshot being that you type "vagrant" a LOT02:59
prologicbecause that makes sense :)03:01
prologicI think I prefer my way :) haha03:01
prologicwait 5s (tops)03:02
prologicssh root@
prologicvaoila :)03:02
prologicvoila :)03:02
DanielBairdwell at least i sometimes get the pleasure of watching gdal compile03:03
DanielBairdactually the compile info is hidden. so really i get to watch nothing happen for ten minutes, then a line saying "changed"03:03
DanielBairdbut anway this is just when i destroy the vm03:03
prologic*nods* :)03:04
DanielBairdsubsequerntly, when we're not testing full dependencies, i can `vagrant up` then ssh in as normal03:04
prologicI think I still like openvz better than vagrant thogh03:04
prologicit's far far quicker :)03:04
prologicI suspect I'll be done testing full deploys soon03:05
prologicanother full day or two I think to get apache + mod_proxy + app in virtualenv03:05
prologicand any security stuff03:05
prologicbtw I think you're confused with the Climate Hub Repositories03:05
prologicthat runs on Plone03:05
prologicthis (Climate Hub Visualizations/Mapping) won't03:05
prologicccav will probably run Apache + mod_proxy -> ccav backend03:06
prologicSo I'm going to work on that now and tomorrow03:06
DanielBairdcool np03:06
prologicso on Monday we might (fingers crossed) ready to do some more backend <-> fronted integration with API(s)03:07
DanielBairdi'll get the graph in and hook up a json file to load03:07
DanielBairdthen you can switch out the json get response with whatever awesome data you can come up with03:08
prologicsounds perfect03:08
DanielBairdso i just tried setting up a virtualenv for my work-on-the-ui project03:16
DanielBairdthere's something funcked up in my python setup03:17
ircnotifier0ccb02219d89 by prologic: Use explicit paths to gdal-config and geos-config because for some stupid reasons CentOS 6.3 on the NECTAR cloud is somehow totally different from any other CentOS 6.3 (or maybe it's just the sudo thing)03:18
DanielBairdi ran python develop in my new virtualenv03:18
DanielBairdthat worked fine installing stuff until it screwed up bitching about bad local header03:19
DanielBairdnow if i run python develop again, it immediately dies with 'no module setuptools'03:19
ircnotifier86f3c38031b4 by prologic: Need to be specific about --with-ogr too03:20
DanielBairdis it weird that, inside a virtualenv, is installing virtualenv?03:21
DanielBairdactualyl here's the whole thing03:23
ircnotifier31652ef05cd4 by prologic: Removed unnecessary setup_requires dependencies03:23
prologicSee if this fixes the issue at hand03:24
prologictry it again03:25
prologicupdate your repo(s)03:25
prologicsee if that fixes it03:25
prologicI have a funny feeling I might have been accidentally screwing you over :)03:25
prologicvirtualenv in setup_requies makes no sense03:25
prologicit's an outside dependency03:25
prologicalso I had been doing fab build03:26
prologicor fab develop03:26
prologicfor a while now03:26
prologicso maybe I've been avoiding the issue somehow :)03:26
prologicmy bad :)03:26
prologicyou can smack me in the head one day :)03:26
DanielBairdRunning from numpy source directory.03:28
DanielBairderror: SandboxViolation: open('/dev/null', 'w') {}03:28
DanielBairdThe package setup script has attempted to modify files on your system03:28
DanielBairdthat are not within the EasyInstall build area, and has been aborted.03:28
DanielBairdThis package cannot be safely installed by EasyInstall, and may not03:28
DanielBairdsupport alternate installation locations even if you run its setup03:28
DanielBairdscript by hand.  Please inform the package's author and the EasyInstall03:28
DanielBairdmaintainers to find out if a fix or workaround is available.03:28
DanielBairdnumpy install got cranky, is it a dev version or something?03:28
prologicyes yes03:30
prologicthat's why I don't ever do python setup.y develop anymore03:30
prologicit doesn't work for bumpy :)03:30
prologicinside a virtualenv03:30
prologicgo figure03:30
prologicuse "fab build"03:30
prologicor "fab develop"03:30
prologicat least we sorted out the fuck up though?03:30
prologicI believe it's a known issue with distribute and virtualenv that I've reported but haven't heard back on03:31
prologicso the solution is to pip install manually03:31
prologicwhich is what the fab tasks do03:31
DanielBairdhuh okay .. so i never need to, fab build replaces it?03:32
DanielBairdfab build died coz matplotlib needs numpy.03:33
DanielBairdi might blow this whole thing away and start with a fresh checkout03:34
prologicfab build/develop replaces/wraps it03:34
prologicdoes extra stuff too03:34
DanielBairdwhats the hg to find out if i have un-pushed commits?03:35
prologicremember to run ./ first though03:35
prologicmkvirtualenv foo03:35
prologicfab develop03:35
prologicdo what the playbook does :)03:35
prologicun-pushed changessets03:36
prologicthat's call outgoing changesets03:36
prologichg outgoing03:36
prologichg out03:36
DanielBairdawesome, nothing un-pushed.03:36
DanielBairdhmm fresh new repo and virtualenv, bootstrap then fab develop, still matplotlib dies from no numpy03:41
DanielBairdis bootstrap supposed to get numpy?03:42
DanielBairdno i see it in fab build03:43
DanielBairdi'll try pip installing it manually03:43
prologicit should install numpy03:53
prologicsee what the mnaul install of bumpy does in the virtualenv03:54
prologicmaybe there's a missing step03:54
DanielBairdnumpy whiles like abitch when it's installing, but in the end says it worked.  i'm trying fab develop again to see if it gets through this time03:54
DanielBaird* whiles = whines03:54
DanielBairdif there weren't a million lines of compiler warnings, i could scroll back and see what happened when fab tried installing numpy the first time03:58
DanielBairdokay ran ccav, it's working.  so i'm just gonna act like everything's normal, and jimmy my graph into a layout04:00
prologicexcept that it's not :)04:01
prologicbumpy failed to install04:01
prologichave to look into why04:01
prologicbut first still fixing map server on nectar cloud04:02
prologicbah for differences04:02
DanielBairdyeah it's weird, fab tries to install numpy first, then other stuff, and fails.. i install numpy manually, then run fab and it's fine.04:04
DanielBairdactually maybe that means numpy had a dep that was installed after04:04
DanielBairdas in, it's dep was installed by the fab requirements.txt line.  fab tries to install numpy before it runs the requirements.txt04:05
prologichang on04:06
prologicI'll check I haven't screwed up again04:06
prologicoh I'm such an idiot04:09
prologicI've got pip function doing04:09
prologicwith settings(hide=("stdout", "stderr"), warn_only=True):04:09
prologic   local("pip ...")04:09
prologicI'm so stupid04:09
DanielBairdoh wait pip is a function you wrote04:12
DanielBairdthought it might have been supplied with ansible04:12
DanielBairdsry fabric04:12
ircnotifier0454ac17143f by prologic: DON'T warn_only for failures. Let fabric abort.04:18
prologicgawd I'm such a daubass :)04:19
prologicwe'll find out why bumpy fails to install now04:19
prologicgod damn auto correct04:19
prologicyeah pip I wrote04:20
prologicit does 2 things04:20
prologicchecks if the list of modules you give it are already installed04:20
prologicand only tries to install what's not already04:20
DanielBairdhey here';s a plan04:20
prologicit makes use of my requires() function04:20
DanielBairdwhen we're deploying we should start a background process04:21
DanielBairdthat mines bitcoin while the installation is happening04:21
prologicyay map server finally built successfully on netar04:21
DanielBairdthen suspends the mining when the install stops.04:21
prologicand slow down the compile?04:21
DanielBairdeventually we'll have mined a whole coin, all on some other guy's VM resources04:21
DanielBairdor technically our resources, but it won't be holding up the site itself, just adding time to the deploy phase.04:22
DanielBairdsomething to think about.  i wonder if the fabric authors have already added something like that to their tool04:23
DanielBairdremember when spare CPU cycles were supposed to be used by SETI?04:23
prologicit could easily be done :)04:31
prologictempting :)04:31
prologicI already earn $1.30/day from mining04:31
DanielBairdwhat's your power cost?04:32
DanielBairdor is that factored in04:32
DanielBairdor does griffith uni sponsor your mining operation04:32
prologicnah all home equipment04:33
prologicI'd say power is roughly 300W of power04:33
prologicfor about 168Mh/s04:33
prologicI'm going to buy some hardware next week04:33
prologicwhich should bump me up to over 600Mh/s04:34
prologicat an additional 80W of power04:34
prologicbut we have solar too04:34
prologicso I'm not so concerned with the power bill04:34
DanielBairdah that's interesting.  you're turning sunlight into bitcoins04:34
prologicbut yeah I'm mining at a loss of course04:34
prologicat currently difficulty04:34
prologicit is not worth mining with anything but FPGA or ASIC chips04:34
DanielBairdi guess you're investing in potential higher value of coins04:35
prologicthat's the plan - yes :)04:35
DanielBairdthe future higher value i mean04:35
prologicI hope to get to $15/day in a couple of weeks04:35
DanielBairdim tempted to start using requirejs04:35
prologicthen maybe in 6months I might be able to get to $200/day04:35
prologicdo it04:36
prologicI like requirejs04:36
DanielBairdnow that there'll be lots of code to do stuff like hide and show the alert messages etc, that should all be in modules04:36
prologicnice approach to modules for Javascript04:36
DanielBairdyeah i think i'll do it.  there's no meeting for this until wednesday or something.  so noone will bitch if i still don't have a graph showing today04:37
prologicyeah but magic isn't necessarily good :)04:37
DanielBairdmagic is good for the wizard, bad for the demons who have to run around behnd the scenes making it happen04:37
prologicbut Sam approached me this afternoon04:37
prologicHe's hoping we can have something demo-able by Friday next week04:38
prologicI said tentatively we could04:38
prologicobviously at the current state we're at04:38
prologicbut at least a deploy to NECTAR with a working graph, map and some layers04:38
DanielBairdhmm.. yeah okay imaybe i'll jsut get a graph up so he can say what things he wants to demo, and we work on those04:38
prologicI might have to pull some later nighters on the backend to get the layers going again but oh well04:38
prologicI think he'll be happy with anything tbh04:39
prologicso let's continue to focus on integration and deployment04:39
prologicand when we're happy enough with that04:39
prologicget the backend powering your fronted with some data and layers04:39
prologicspeaking of which04:40
prologicNECTAR deploy finally succeeded04:40
DanielBairdacutally require is not much of a drag even to get started.  i have so little code that isn't already modular04:40
prologicI'll commit this04:40
DanielBairdif i start using requirejs is gerhard or someone gonna have a problem with it later?04:40
DanielBairdit does change the code04:40
ircnotifier6d097c7ff0bc by prologic: Added swig to mapserver deps and swig configure step for mapscript extensions04:40
DanielBairdthe look of the code i mean04:40
prologicI don't think so04:41
prologicI've pretty much convinced Gerhard RequireJS is a good thing04:41
prologicHe'll be fine with it04:41
DanielBairdi guess it compiles into a big block anyway04:41
prologicwe've discussed it at lengths04:41
DanielBairdokay cool04:41
DanielBairdi'll add it now04:41
prologiclet's test this again local again04:42
DanielBairdback.. you want me to kick off another vagrant full build?05:13
DanielBairdno wait.. that's already working05:13
DanielBairdi guess you want to test the setup inside a virtualenv05:13
prologicactually I would like you to re-test the latest playbooks05:27
prologicit's getting late though05:27
prologicso maybe do it in a detached screen session05:27
DanielBairdi'll kick one off05:27
DanielBairdi have separate project dirs for doing vagrant and ui stuff05:27
prologickk nps05:28
prologicI going to write roles for apache now05:28
prologicand a webapp user05:28
prologicand webdevs group05:28
ircnotifier651a1ac89871 by Daniel Baird: Bringing Vagrant config into line with latest deployment05:30
ircnotifier5f5e5bca5fd9 by Daniel Baird: merged05:30
ircnotifier261da87e4b5b by Daniel Baird: Added vagrant host05:32
DanielBairdokay updated everything.. vagranting.  vagaries?  vaguing?05:34
ircnotifierd6dfab30e695 by prologic: Added apache role05:52
ircnotifier0a53595b606c by prologic: Automated merge with
ircnotifier03361b36879f by prologic: Added security role adding a webdev user and prompting for a password (required passlib library)06:23
prologicon that note cya tomorrow06:23
prologicfew more things to do06:23
prologicbut I'll be ready by Monday06:23
DanielBairdnp i'm awa y tomorrow though06:27
DanielBairdi'll check in something tonight for you to look at06:28
ircnotifier44c152abba87 by Daniel Baird: Don't merge JS (require.js will load modules)07:20
ircnotifierf5d0083eb5d8 by Daniel Baird: ignore the distribute.tar.gz file07:20
ircnotifierd3ad6c0a857b by Daniel Baird: drop angular, start requirejs, add graph and table07:20
ircnotifiercd1005130eb4 by Daniel Baird: merged07:20
*** DanielBaird has quit IRC07:20

Generated by 2.11.0 by Marius Gedminas - find it at!