IRC Logs for #cmvt Monday, 2013-04-22

prologicI was just passing requests like00:00
prologichttp://localhost:8000/mapserv?<wms_params>00:00
prologicwhere one of the params was:00:00
prologictime=202500:00
prologicfor example00:00
prologicI imagine we want an API on the backend00:00
prologicthat uses mapscript to serve up ant arbitraryritrary WMS layer right?00:01
DanielHereNowsorry drifted off.00:03
DanielHereNowi guess the raster layers can be handled by leaflet, so i suppose anything that leaflet can talk to will work.00:04
DanielHereNowit's nice to not use the mapserv cgi but for now i suppose it's okay to use it.00:04
DanielHereNowhowever i guess that would mean serving cgi from the dev setup00:04
DanielHereNowhave you looked at the vagrantfile?  I can add mapserver installation to that, into wherever centos's apache defaults to serving cgi from.00:05
DanielHereNowor will circuits do old stype cgi?00:05
DanielHereNowor, should i hold off raster layer loading until we've got a working mapscript api00:06
prologicccav+circuits is already serving up map server as a CGI00:06
prologichmmm00:07
DanielHereNowbut to answer the actual question: yes we want to serve wms from our api.. it can be special to out specific layers though i guess00:07
prologiclet's have a good think about how this should work00:07
DanielHereNowah okay so i wrote the fab files to make a htdocs dir00:08
DanielHereNowis it easy to tell circuits to serve static from that dir?00:08
DanielHereNow    docroot = path.join(path.dirname(__file__), "htdocs")00:09
DanielHereNowi guess is the place.. should it be ../htdocs, given that it's a sibling dir to the ccav dir00:09
prologichttp://pastebin.com/JtSkCQcf00:09
prologichave a look at this for a moment00:09
prologicthis is my data directory as it stands now00:09
prologicwith raw data (downloaded from wallace initiative)00:10
prologicdelta data (computed from current)00:10
prologicand indexes (for time)00:10
prologicyeah circuits.web can serve up any directory of static content00:11
prologicand I can make it configurable easily00:11
prologicno need to worry about Apache right now :)00:11
prologicAlso: http://pastebin.com/7u1Vf09a00:14
DanielHereNowalright i can concat the scenario / model / year to match the names in teh data dir.00:15
DanielHereNowoh right, ccav is a binary running from somewhere else.. i have two clones of ccav, how do i switch which 'ccav' is run?00:17
DanielHereNow(i'm getting the old content, coz ccav is running from the old clone)00:18
ircnotifier71964fb0f550 by prologic: Added docstring for build task00:19
ircnotifiera86b6c7637ae by prologic: Code cleanup00:19
ircnotifier4068b31e4223 by prologic: Used nested context to hide stdout/stderr00:19
DanielBairdbrb00:20
prologicumm00:21
prologicnuke your other repo00:21
prologicyou don't need it anymore :)00:21
prologicdo you have virtualenvwrapper istalled?00:21
DanielBairdyeah i can delete it.  i think i have virtualenv00:23
DanielBairdif my ccav command is looking at the old one, will deleting it make it hard to re-install ccav?00:24
DanielBairdscrew it anyway it's all checked in.  deleting..00:24
prologicit is :)00:26
DanielBairdthere's some setup i run, right.. setup.py was uninformative, wasn't it python setup.py development or something?00:27
prologicjust do:00:27
prologiccd /path/to/new/ccav/repo00:27
prologicfab build00:27
prologicor00:28
prologicfab build:dev=True00:28
DanielBairdi did the first one, now it's silently installing numpy00:29
DanielBairdi guess it'll get there soon00:29
prologicyu[00:29
prologicyup00:29
prologicit _should_ work :)00:29
prologicI haven't seen anything go wrong with this so far00:29
prologicThu I spent a bit of time improving the build task00:30
prologicbrb00:31
DanielBairdshould fab build also make the htdocs dir?  probably should00:32
DanielBairdactually i should rename frontend.all to frontend.build, might be more consistent.  then call that from build00:34
DanielBairdjust saw the mockup for the big multigraph thing they're asking for00:46
DanielBairdthey seem to care a lot about this, I might start work on it00:48
ircnotifiere754970df8de by Daniel Baird: renamed frontend build task00:52
prologicyeah :)00:58
prologicthey do sadly00:58
prologicbut I'm sure we can whip it together00:58
prologiceven with some static json for now00:59
ircnotifier81a3caf5e840 by Daniel Baird: added devbuild task, added frontend building to build task01:00
DanielBairdnot sure if it matches with the page layout i've been considering01:00
DanielBairdbut i can always make the map dramatically smaller when they switch to graph mode, i supoose01:00
DanielBairdbtw i added "fab devbuild" which does a "fab build(dev=True)" but avoids having to hold the shift key :)01:01
prologichaha01:02
prologicyou lazy bugger :)01:02
prologiclol01:03
prologichmm01:03
prologic~/ccav/tmp/mapserver/data01:03
prologic$ unzip RCP3PD_cccma-cgcm31.zip01:03
prologicArchive:  RCP3PD_cccma-cgcm31.zip01:03
prologicwarning [RCP3PD_cccma-cgcm31.zip]:  3998560378 extra bytes at beginning or within zipfile01:03
prologic  (attempting to process anyway)01:03
prologicerror [RCP3PD_cccma-cgcm31.zip]:  start of central directory not found;01:03
prologic  zipfile corrupt.01:03
prologic  (please check that you have transferred or created the zipfile in the01:03
prologic  appropriate BINARY mode and that you have compiled UnZip properly)01:03
DanielBairdhmm01:03
DanielBairdi've only unzipped from osx.. lemme try unzip01:04
prologicyeah unzipping from OSX's GUI01:04
prologicis no good for us :)01:04
prologicnfi how to fab that up :)01:04
DanielBairdyeah i get that error too01:04
prologicso it seems the GUI was able to unzip the archive01:06
prologichrmm01:06
prologicbloody hell01:06
prologicsee I never actually do things form the GUI01:06
prologicnow to figure out what the hell unzip (CLI)'s problem is01:06
prologicboth the CLI and GUI tools01:06
prologicwould be using the same librries01:06
DanielBairdi'm trying zip -FFv RCP3PD_cccma-cgcm31\ \(2\).zip --out fixed.zip01:08
DanielBaird(recommended on some random web forum)01:08
DanielBairdnope didn't work01:09
prologichttps://discussions.apple.com/thread/3524098?start=0&tstart=001:09
prologichrmm01:09
prologicthe OS X GUI Archiver is doing some magic01:09
prologicto try and recover the archive01:09
DanielBairdanother randpage suggested jar xf file01:10
prologicbut most likely the contents of which will be slightly corrupt anyway01:10
DanielBairdim trying that01:10
DanielBairdthe debian thread said it's just a zipping technique that unzip doesn't like01:10
prologicyah I think we're wasting our breath01:10
prologicI believe it is correupt01:10
prologicbut the GUI archiver tool uncompresses it anyway01:10
prologicbut the data would be a bit useless/inaccurate in places01:11
DanielBairdjar xf worked without complaining01:11
DanielBairddunno how to specify target dir using jar01:12
prologichrmm01:12
prologichttp://superuser.com/questions/114011/extract-large-zip-file-50-gb-on-mac-os-x01:14
prologicok01:14
prologicmight have a solution01:14
prologic:)01:14
DanielBairdhmm01:15
prologicok01:15
prologicmight be an Apple issue01:15
prologicsolvable by installing latest 6.0 unzip01:16
prologicmatching Apple's 6.0 zip01:16
prologicgo figure :)01:16
DanielBairdapple love us command line people, obviously01:16
DanielBairdalright.. that's nice, i was worried that JCU had been hosting useless zip files, and they're not easy to fit onto our infrastructure01:17
prologicwell01:22
prologicthe zips are also larger than they're suppose to01:22
prologiczip only supports 4GB archives I think01:22
prologicbut I guess there are extensions to the zip file format that support >4GB01:22
prologicanyway a solvable problem01:23
prologic*phew* :)01:23
prologicAlso, where are the 250m datasets?01:23
DanielBairdback.  i just discovered the port i thought was a hdmi port is really a display port01:27
prologichah01:27
DanielBairdactually i should mention the can't-unzip-large-files problem on the download ui page01:28
DanielBairdaaaanyway, 250m res.01:28
prologicyes that would be good to warn users of that01:28
prologicthat they may need unzip 6.0 from Macports01:28
prologicor some other tool like Unarchier01:28
DanielBairdi was offered 250m at one point to make downloadable on that web page, but couldn't fit it on the hdd of the wallaceinit machine01:28
DanielBairdi'll poke around see if i can find some nwo01:29
DanielBaird* now01:29
prologichmm01:29
prologicdo you have a copy of the full 25m dataset?01:29
prologicas-in the 5km and 1km datasets?01:29
prologicWe do need these available somewhere to download/synchornize :)01:29
prologiceventually01:29
DanielBairdJeremy's got (i think..) 250m somewhere01:30
prologichow large is the dataset compressed?01:33
prologicshould our build task also try to install less?01:34
prologicand check if nodejs/npm is installed?01:35
prologicand if not installed abort telling the user to install nodejs?01:35
prologicor should we just make installing Python + NodeJS mandatory at the start01:35
DanielBairdbrb01:36
DanielBairdback01:40
DanielBairdif the build thing installs dependencies, then yeah it should install node & npm, then npm install -g coffee-script less01:40
DanielBairdi've already added those deps to the vagrantfile01:41
DanielBairdbut if it's in the fab build process it will also work01:41
DanielBairdi think i added a python install to the vagrantfile, whatever we aren't installing in fab build needs to be there01:42
prologicwell ...01:43
prologicthings like:01:43
prologicinstalling Python01:43
prologicinstalling NodeJS01:43
prologic_should_ be taken care of by a configuration management tool  like Puppet, Chef, Ansible, Salt, etc01:44
DanielBairdyes01:44
prologicby the sysadmins of the server in question01:44
prologicso I'll check for nom in the path01:44
prologicand do an nom installe coffee-script less01:44
DanielBairdoh i just looked at the vagrantfile.. installing node on centos is awful01:44
DanielBairdhad to compile from source01:44
prologicyeah01:44
prologicthat's what I mean01:44
prologicThings Config Management should do:01:45
prologic * Install Python01:45
prologic * Install MapServer01:45
prologic * Install NodeJS01:45
prologic * Install Apache01:45
prologicThings Fabric should do:01:45
prologic * Install dependencies (Python, NodeJS, etc)01:45
prologic * Build the web application01:46
DanielBairdanyway yes.. at present the vagrantfile installs gcc, make, git, node, coffeescript and less.  actually i didn't install python, i thought you might be better at getting it right first time.01:46
prologic * Download datasets and install them01:46
prologicetc01:46
DanielBairdwe can move coffee and less into fabric01:46
prologic*nods*01:46
prologicbe better there I think01:46
prologicas long as NodeJS is instlled01:46
DanielBairdbut have a look at the vagrantfile, it's technically a ruby script  but actually it's mostly bash.01:47
DanielBairdand yes, as long as node is there, i can write a npm install of cofffee and less into the fab files01:48
DanielBairdjust before we actually launch i'll freeze the versions too.  that'll be worthwhile01:49
prologicdoes nom support installing from a requirements file with specific versions?01:50
DanielBairdprobably.  it does support comamnd line specification of versions01:51
prologictbh I don't think vagrant isn't going to buy us much per say (because we're both OS X developers)01:51
prologicI already use Homebrew on my Mac01:51
prologicso already have MapServer, Python, etc installed01:51
prologicand NodeJS01:51
prologice.eg:01:51
prologicbrew install mapserver01:51
prologicbrew install python01:51
prologicbrew install nodejs01:51
DanielBairdokay nvm01:57
prologicwell it's still useful I guess to others :)01:57
DanielBairdit pays off when we have to describe the setup for the centos machine we'll be deploying to01:57
prologicbut so far I don't see anyone in the near to medium term future that will be developing on Windows :)01:57
prologicGerhard is on a Mac01:57
prologicas am I01:57
prologicand you :)01:58
prologic_that_ is true (describes the setup)01:58
prologic-but- then again01:58
prologicI think the sysadmin team around here uses Puppet01:58
DanielBairdi'm going to maintain it, because I think it will be worth it, but if i'm the only one using vagrant then I catch up on it later01:58
prologicso I'm not entirely sure how helpful it'll be01:58
prologicI myself prefer to use Ansible for Config Management of servers01:58
DanielBaird* I'll01:59
prologic*nods* it's no problem01:59
prologic-but-01:59
prologicset it up in such a way that you assume01:59
prologicmost things will be done by fabric01:59
prologici.e: install Python, MapServer and NodeJS01:59
prologicI think those are our only reps right now01:59
prologicthere's a GIS CentOS repo too I think you can download to /etc/yum.repo.d/02:00
DanielBairdanything that fabric doesn't want, i'll just add to the bash setup.  it's currently missing python but like i say I'd expect centos has a yum package02:00
DanielBairdhmm yes mapserver02:00
DanielBairdyeah that'll be okay.. at worst i can just compile it but that does slow a full reset up a lot02:01
DanielBairdoccasionally i like to do a "vagrant destroy; vagrant up" to double-check everything's still working02:01
DanielBairdin other news, the giant, tape-backed storage area where Jeremy keeps his climate layers has perms set to Jeremy-only02:02
DanielBairdi'lll email him asking for 250m layers.  and actually Lauren or Erin might have some access02:03
DanielBairdactually.. is it your local guys who will be setting up the final resting place for this system?02:05
DanielBairdi could use their puppet recipes or whatever puppet calls them02:05
DanielBairdpuppet strings maybe02:05
DanielBairdaww they're just called "modules".  that's not in keeping with the tradition of nerdy wordplay02:07
prologicI'm not 100% sure on this one02:10
prologicwe're getting an 8-core VM02:11
prologicwith 16GB of RAM02:11
prologicand some level of storage02:11
prologicno idea on the OS02:11
prologicor who's managing what at this stage02:11
prologicbut yeah02:11
DanielBairdsomeone mentioned centos, which is why is used that bloody os for the vagrantbox02:11
DanielBairdi'll get to it at some point.  you might have to talk me though getting python set up properly.  or will all the fiddling about with virtualenv and setup and whatnot be inside the fabric file?02:12
prologicyeah02:13
prologicjust yum install python02:13
prologicis fine02:13
prologicyeah we have been using CentOS here for a while02:13
prologicbut we're now looking at moving over to RHEL02:13
prologicfor some reason02:13
DanielBairdthere'll be some kind of enterprise deal02:15
DanielBairdunis lo e to outsource these days02:15
DanielBaird* love02:15
DanielBairdi've added python to the vagrantfile and testing it out02:16
DanielBairdif i fab clean, does that remove all the python setup for ccav too?02:16
DanielBairdhmm for some reason when centos installs git, it wants perl too02:17
DanielBairdactually it's sone kind of git-perl bindings.  meh whatever02:20
prologicumm02:21
prologicyeah fab clean just cleans up build files and dirs02:21
prologicI'm trying to figure out a decent way to bootstrap all of this02:21
prologicin some manner02:21
prologicand check for npm02:21
prologicand install npm deps02:21
DanielBairdor which npm or something like that i guess02:22
DanielBairdif you have node after v 0.6.something, you definitely have npm.02:29
DanielBairdbut i suppose you want to `which npm` anyway02:29
prologicyeah02:30
DanielBairdbut if it's not there, it's a sysadmin problem so fab should bail out or whatever02:31
prologic*nods*02:31
prologicI'm going to create a bootstrap.py as well I think02:31
prologicwhich will do a couple of things02:31
prologiccheck for distribute/pip and install as required02:32
prologiccheck for fabric and install as requried02:32
prologicthen you can do:02:32
prologicfab build deply02:32
prologicdeploy*02:32
DanielBairdso the last line of the bash init stuff will be:02:34
DanielBairdpython bootstrap.py && fab build deploy02:34
prologicyes something like this02:34
prologicsounds perfect :)02:34
prologicthis is for the vagrant file?02:34
DanielBairdyep, i'm not using puppet or any proper tool, just booting a vm and running a bash script02:35
ircnotifier3b593906fdb6 by prologic: Automated merge with https://bitbucket.org/ccaih/ccav02:35
ircnotifier79d045ebe6cd by prologic: Added installation of coffee-script and less to build task02:35
prologichow's this?02:36
prologicre: bash script - yeah that's perfectly fine02:36
prologicnice and simple02:36
DanielBairdi'll add a couple of things02:38
DanielBairdno wait it's okay02:39
DanielBairdactually i will add a -g for the node install02:39
DanielBairdi think the npm installs should in the the normal build, the frontend.build needs coffee erc02:40
DanielBaird* etc02:40
DanielBairdbrb buying coffee pods02:42
prologicyeah I'm gonna go have lunch02:59
DanielBairdback.  see you later03:00
ircnotifierdd88380f998c by Daniel Baird: moved npm tools to main build, added --global option03:00
DanielBairdat some point can you look at the optional deploy location, if you still think that's worth having.  I'm not sure how to handle args with values, and pass args around.  I've added TODOs into the fab files where it would need to go (but to be clear, I'm okay if we just always deploy to htdocs and don't worry about the option)03:01
ircnotifiercced883d463b by prologic: Re-fixed broken merge by me :/03:07
ircnotifieraeb2dfdda162 by prologic: Merged with dd88380f998c03:07
ircnotifier8ee1bf32a4fe by prologic: Fixed broken merge and renamed devbuild task to develop03:07
ircnotifier03410e4a9852 by prologic: Make ccav take a mandatory argumenet (<docroot>) for now.03:11
prologicFailed to load resource: the server responded with a status of 404 (Not Found) http://localhost:9000/js/all_0_1.js03:12
prologicUncaught Error: No module: maplayout angular.js:105803:12
prologicPort error: Could not establish connection. Receiving end does not exist. miscellaneous_bindings:23203:12
prologicError in event handler for 'undefined': Cannot read property 'hideTooltipSetting' of undefined TypeError: Cannot read property 'hideTooltipSetting' of undefined03:12
prologic    at chrome-extension://kkelicaakdanhinjdeammmilcgefonfh/js/content.js:18:1503:12
prologic    at miscellaneous_bindings:276:1103:12
prologic    at [object Object].dispatch (event_bindings:203:41)03:12
prologic    at Object.<anonymous> (miscellaneous_bindings:235:27) event_bindings:20703:12
prologicgot some things to iron out on your end :)03:12
prologicok bbs - lunch!03:12
DanielBairdhmm yes wonder when that happened.03:12
DanielBairdhad to specify .js on the end of the coffeescript target.  must've just changed in the latest coffeescript.03:25
DanielBairdhmm i think i need mercurial advice.  i have some kind of merge but i don't know how to resolve it03:38
prologicok bac03:40
prologic+k03:40
prologicahh03:40
DanielBairdi forgot to pull before pushing03:40
prologiclet's video chat03:40
prologicSkype or iMessages?03:40
DanielBairdhmm let's go skype03:41
ircnotifierd82f88f68826 by Daniel Baird: merged 72:d7af5cd2dc9a and 71:03410e4a985204:00
ircnotifier29b53072c2cc by Daniel Baird: adding graph pane04:00
ircnotifierb62850c22178 by Daniel Baird: changed quotes04:00
ircnotifierffcc3fb5f322 by Daniel Baird: Vagrantfile now installs python04:00
ircnotifierd7af5cd2dc9a by Daniel Baird: removed unused file04:00
ircnotifiere664e183fed0 by Daniel Baird: Fixed coffeescript compilation to suit latest coffeescript04:00
ircnotifier8f0bc71b7318 by Daniel Baird: added data dir, now that it's not ignored04:19
ircnotifierbf5190f1a67f by Daniel Baird: Testing for coffeescript and lessc before installing05:25
DanielBairdthat is the laziest shell-level test, but makes it quicker.. i think npm goes off to compare versions everytime you install again05:27
prologichmm05:53
prologicI'll commit the "better" way of doing this :)05:53
ircnotifierdd1acb1602b7 by prologic: Use has_binary function instead of bash idioims05:55
DanielBairdcool even faster as well06:49
ircnotifierd2c3b75f70f0 by prologic: Several major improvements on the build task and it's dependencies06:57
prologicok I'm outta here06:57
prologicman times flies06:57
DanielBairdse ay06:58
DanielBaird* see ya06:58
*** DanielBaird has quit IRC08:12
*** DanielBaird has joined #cmvt08:57
*** DanielBaird has quit IRC09:06
*** DanielBaird has joined #cmvt10:14
*** DanielBaird has quit IRC10:18
*** DanielBaird has joined #cmvt11:14
*** DanielBaird has quit IRC11:19
*** DanielBaird has joined #cmvt12:14
*** DanielBaird has quit IRC12:19
*** DanielBaird has joined #cmvt13:20
*** DanielBaird has quit IRC13:25
*** DanielBaird has joined #cmvt13:58
*** DanielBaird has quit IRC14:24
*** DanielBaird has joined #cmvt15:18
*** DanielBaird has quit IRC15:27
*** DanielBaird has joined #cmvt16:24
*** DanielBaird has quit IRC16:28
*** DanielBaird has joined #cmvt17:24
*** DanielBaird has quit IRC17:29
*** DanielBaird has joined #cmvt18:24
*** DanielBaird has quit IRC18:29
*** DanielBaird has joined #cmvt19:25
*** DanielBaird has quit IRC19:30
*** DanielBaird has joined #cmvt20:25
*** DanielBaird has quit IRC20:30
*** DanielBaird has joined #cmvt21:27
*** DanielBaird has quit IRC21:31
*** DanielBaird has joined #cmvt22:27
*** DanielBaird has quit IRC22:31
*** DanielBaird has joined #cmvt23:09
prologicmorn'n23:45

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!