IRC Logs for #cmvt Tuesday, 2013-07-02

prologichow are we going?00:03
DanielBairdhard to chase the niggling syntax errors through the commit / deploy loop, but i'm nearly there.  i'm currently testing asynchronous loading of the variable descriptions, once it works i'll do the same handling for all the region data as well.00:05
prologiccool00:07
prologicshould I keep trying to create a json schema api?00:08
prologicI've coded up a simple base class for supporting json schema based api resources00:11
prologicso my api resources can now subclass Resource and automatically get serialized to json00:12
prologicand set their content type with profile= correctly00:12
DanielBairdthis ui doesn't need a completely discoverable api.. i don't know how much effort you should put into making the api more generally applicable. it depends on the project's goals00:12
prologicprobably want to semi automatically generate the schema for resources too00:12
prologicbut for now I'll just do so by hand00:12
prologic*nods*00:13
prologicI know :)00:13
prologiclemme know if you need me to do something more on the backend00:13
prologicI know the extra effort is probably not worth it right now00:20
prologicbut it would be nice if your ui code could implement a json schema client00:20
prologicthat way I can change the api without breaking your code :)00:20
prologicchange urls00:20
prologicand even change the structure of things00:20
prologic:)00:20
prologicbut oh well00:20
prologicwe're running out of time I fear00:20
prologic:)00:20
DanielBairdhow do i get the region names?00:21
DanielBairdand states they belong in, and whatnot00:21
DanielBairdi imagined that would be in the region summaries, but it's okay if it's under /regions/..  but what's there is just shapefiles i think00:22
prologicumm00:24
prologicwhat do you need now?00:24
DanielBairdin the region table, i show the region name and it's parent state00:24
prologicdo we need a way to query the regions Shapefile?00:24
DanielBairdhmm..00:25
prologicI think in the summaires00:26
prologicthe region is currently an id00:26
prologicbut really should be a url to /regions/<id>00:26
prologichmm00:26
DanielBairdi think i'll need region meta as json, and the region polygon as a WKT or geoJSON00:27
prologicfor the sake of adhocness00:27
DanielBairdor even just a geoJSON file with region meta info in it00:27
prologicand kiss00:27
prologiccan we just have an /api/regions resource00:27
prologicthat queries the shapefile00:27
prologicpassing it the id you get from the summaries00:27
DanielBairdyep that will work okay.  can i get all regions in one fetch by GETting /api/regions/ ?00:28
prologichttp://codepad.org/4ewKxEbp00:31
prologicThese are the properties of one of the regions00:31
prologicYes you could fetch all regions00:32
prologichowever you would not want to fetch all geometries however00:32
DanielBairdLooks like REG_NAME_6 and / or REG_NAME_7 is the region name.  No state info, not sure how we get that.  I guess ARC is a region type?00:35
prologicI think so00:36
prologicI'm not sure either00:36
DanielBairdand yeah even if i could read shapefiles in JS, I definitely won't want to be grabbing entire polygons with a million points00:36
prologiclike I've said before00:36
prologicwe don't currently have this kind of data00:36
prologici.e: regions <-> state boundaries00:36
prologicwe'd have to either find some00:36
prologicor make it up00:37
prologicI don't like the later prospect00:37
prologicI can however build you an /api/regions resource00:37
prologicwhich will let you query the Shapefile in question with some query string parameters00:37
prologicand/or include the geometry as geoJSON00:37
DanielBairdyeah.. IBRA regions aren't distinct to state boundaries.  I'm showing state because that's what the mockup used for grouping into the donut graphs.00:37
prologic*nods*00:38
prologicshall I build you a regions api00:38
prologicand we'll think about how we can solve the other issue later?00:38
prologicprobably take ma a couple of hours00:38
DanielBairdyeah all I'll need is /regions/IBRA/ to return a list of all the ibra regions, with their names and ids, and states even if we just call everything 'other' for now.  Then the same for LGA, NRM00:39
DanielBairdor i guess if the shapefiles are in the way, /regions/IBRA/all  or something.00:39
prologic*nods*00:40
prologicyeah IBRA regions just simply don't have that kind of data00:40
prologicif we can somehow find relevant data00:40
prologicwe can insert it into the Shapefile's properties00:40
prologicanyway00:40
prologicI'll build a generic Shapefile resource00:40
prologicbit like a table/record query resource really00:40
prologiconly it also supports geometries00:40
prologicin theory if we moved all this to databases00:41
DanielBairdif i had lauren i'd just get her to work out state by finding the state poly with the most overlap.00:41
prologicthe backend code would not change much00:41
prologicwell00:41
prologicin theory I could do that?00:41
prologici.e:00:41
prologicI could build a tool to compute such infomration00:41
prologicbe intersecting every region against every state00:41
prologicand coming up with some meta data around what region is in what state boundary00:42
DanielBairdi guess.. is there a tame postgrad hanging around who and knock something up in R?00:42
prologicit would be trivial to compuete00:42
prologicand maybe a day or so to write the code to compete it00:42
DanielBairdwho *could*00:42
prologichaha00:42
prologicdunno00:42
prologicbut it would be just as quick for me to do it00:42
prologicI already have code that does this in similar but different ways00:42
prologicthat's how I compete summaries for you right now00:42
prologicby intersecting all regions against all models00:43
DanielBairdhmm yes i guess it's not too far different than summarising across the region.  it's kind of a summary of the state each pixel falls into00:43
prologicin fact00:43
prologicnow that I think about it00:43
prologicin theory I could build such an api/resource00:43
prologicdynamically00:43
prologicby utilizing geometry indexes on shapefiles00:43
prologicand then later on we can cache that00:44
prologici.e:00:44
prologicyou could compete the informaiotn you need client-side00:44
prologicby performing two queries00:44
prologicyou need to get the region information anyway00:44
prologicso get it's bounding box00:44
prologicthen query the state boundaries by that bounding box00:45
prologicand get the state peroperties00:45
DanielBairdbut in any case, unless it's quite quick to do, for now just give every region a state of 'other' or 'Australia' or something so i can start writing in the region data fetch00:45
DanielBairdthat's true, but it's two fetches for every region.. i'll always need all regions in a given region type, all at once00:45
DanielBairdgotta populate the table00:45
prologicyou sure you don't want me to do this properly firs time around00:45
prologicand include state boundaries as well00:46
prologic(which I have data for)00:46
prologicso we have IBRA and STATE00:46
prologicand let you query both resources00:46
prologichmm00:46
prologicyou do have a point00:46
prologiceven if you fetch all regions00:46
prologicyou'd still have to fetch the state information for every region00:47
DanielBairdyeah00:47
prologichmm00:47
prologiccan a his enabled database perform a join by intersecting geometries from two tables?00:48
prologicthat's really what you need00:48
DanielBairdpragmatically, the fewer fetches before i can display the screen, the better, so ideally the region names are in the summary json, and just repeated over and over :)00:48
DanielBairdPostGIS probably can00:48
prologichmmm00:49
prologicor I just modify gensummaries00:49
prologicto include the region's name00:49
prologicand state00:49
prologicby also intersecting the region poly with all state polys00:49
DanielBairdyeah that's the simplest thing that will work00:49
prologicyou get your 1 fetch00:49
prologicand it'd just take extra time during deploy00:49
prologicand probably quickest to implement too00:50
prologicbut I suspect you'll still want to be able to query features (Shapefiles)00:50
DanielBairdarguably, you could make a case for the region name being a ligitimate part of a 'summary'00:50
prologicIBRA, NRM, LGA, etc00:50
DanielBairdyeah i'd still like to get simplified polys00:50
prologickk00:50
prologicalright00:50
prologicI'll modify gensummaries00:50
prologicand start including the state boundaries in the build00:50
DanielBairdi can grab the polys one at a time or whatever, the map is not the primary interface (on the graph screen, at least)00:51
prologickk00:55
prologichttp://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1270.0.55.003July%202011?OpenDocument01:07
prologichmm01:07
prologicnone of these shape files include state bounadires01:07
prologicgot NRM and LGA01:07
prologicwhich we need01:07
prologicneed to find a shape file for australian state boundaries01:07
prologichttp://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1259.0.30.001July%202011?OpenDocument01:09
prologicI think I found it here01:09
prologicState and Territory ASGC Ed 2011 Digital Boundaries in ESRI Shapefile Format01:09
prologichttp://codepad.org/YiXm5lP101:12
prologicbeautiful01:12
prologicfound the data I need :)01:12
prologicnow to include it in the deploy01:12
prologicand modify gensummaries01:12
prologicbrb01:12
DanielBairdthats looking promising01:13
*** DanielBaird has joined #cmvt01:13
DanielBairddoes circuits do CORS?01:34
prologicCORS?01:34
prologicahh i see01:35
prologicno it has no such feature01:35
DanielBairdyeah i'm trying to set up my local host install to get data from the VM, so i don't have to commit every console.log thing i do01:35
prologicbe pretty easy to add or do in application code on the backend though01:35
prologicget the same data locally?01:36
prologicrun: fab data01:36
DanielBairdhmm yes i guess that will work too..01:36
prologicshould01:36
prologicI test locally - independent of any vm or deployment01:36
prologicthen test the dev deployment01:36
DanielBairdyes actually, good idea i can just occasionally do a remote deploy to test that the data is getting built properly on deploy01:37
prologic*nods*01:37
DanielBairdFatal error: requires(('asc2tif', 'downloadall', 'extractall')) failed01:37
DanielBairdcan i just pip install those things?01:37
prologicI suggest you do (first)01:37
prologicfab develop01:37
prologicactually01:39
prologicaccording to the CORDS wikipedia article01:39
prologicthere is no reason why any circuits.web built app could not already support this01:40
prologicit's just that because circuits.web doesn't implement it01:40
prologicit should allow it anyway01:40
DanielBairdyeah i think it's just some headers.01:40
prologici.e: it will not restrict requests01:40
prologic*nods*01:40
prologica properly implemented web application *should* deny requests to it's resources01:40
DanielBairdif you're not worried about cross-doimain stuff, you can alllow everything01:40
prologicthat do not originate from it's own domain01:40
prologicor specifically allow requests from an allowed set of other domains01:41
prologicby design/default01:41
prologicisn't everything allowed?01:41
prologicor does the HTTP server need to respond with Access-Control-Allow-Origin of some kind01:42
DanielBairdi think it's browsers that are stopping cross-domain stuff01:43
DanielBairdbut if the server sends the right header in response, the browser permits it01:43
prologicyes I think you're right01:44
prologicso yeah01:44
prologicI could allow cross origin requests to all domains quite easily I guess01:44
prologichrmm01:56
prologicI have a problem01:56
DanielBairdback.  wassup?01:57
prologiceach shapefile has different keys for properties01:57
prologicthere is no consistent way of identifying what the region's name is going to be in01:57
prologicI'd probably have to configure and hard code in a mapping01:58
prologicof region type -> key names ro commonly used bits of information we want01:58
DanielBairdahh that sucks a bit01:58
DanielBairdconsistent within a single region type, though?01:59
DanielBairdso it's three sets of config01:59
prologicI'll just do that for now01:59
prologicsucks01:59
prologicbut yeah common problem01:59
prologicat old govt. we actually remapped the properties as well01:59
prologicbut at the database level01:59
prologicusing views01:59
prologicand in some cases02:00
prologicactually changed the actual data during import02:00
prologicI'll just create a static mapping for now02:00
DanielBairdoff to lunch, bbl02:07
prologicgod this code is as ugly as all hell02:43
DanielBairdyou been looking at my code again?02:56
prologicwhat? no03:08
prologiclol03:08
prologicmy code is but ugly03:08
prologicat least the gensummaries03:08
prologicuggh03:08
prologiccan't get it to match any states atm03:08
prologicstupid pos03:08
prologicbbs gonna go get lunch03:13
prologicman03:42
prologicI don't understand why I'm not matching these f'n states03:42
prologicboth datasets are in the same spatial reference03:42
prologicWTF!03:59
prologichow weird03:59
prologicI have to re-open the Shapefile03:59
prologicevery time I want to filter it by bounding box03:59
prologicI think I've almost got this working04:16
prologicsorry about the time :)04:16
prologicunexpected issues lol04:16
prologicwho would have thought!04:17
prologicyeap cya07:14
DanielBairdhoi22:32
DanielBairdi mean, hi22:32
prologicmorning22:44
prologicSo I have to help Sam Wolski with some dexterity types stuff for the Plone Climate Hub website today - but that shouldn't take too long23:03
prologicI did complete the modifications to gensummaries to include region name and state name as well as their ids23:03
DanielBairdnp23:05
DanielBairdi'm wrangling jquery Promise objects, trying to get the async loading to work for region data, but i'll drop out and do some BCCVL stuff today anyway23:06
prologichmm ok23:07
prologicis there anything else I can add/improve today for you with ccav?23:07
prologicOr shall we leave it where it is for now till I get back?23:07
DanielBairdhmm i think it's okay.  the next thing would be to add current summaries, atm I've copied stuff from a scenario/model/year dir into 'current'.  but that's working for me, so no rush.23:10
prologiccurrent summaries?23:13
prologicelaborate23:13
DanielBairdsummaries for the current time23:14
prologiclike the 2009 current raster layers?23:14
DanielBairdyes23:14
prologicwhich I think are on wallace23:14
prologicok23:14
prologicI can do that this morning/today23:15
prologicwhat else?23:15
DanielBairdcan't think of anything else for now.. eventually there's the polygons, the full set of years etc.. but none of that is required at present23:15
prologicin fact adding current summaries23:18
prologicis as simple as adding the current.zip url to the data fabric task23:18
prologicI think/hope :)23:18
prologicI could probably add a feature api23:18
prologicto query shape files and get geojson polys23:18
prologicthat is possibly doable today23:18
prologicWith "current"23:42
prologicI'm treating it as a model23:42
prologicwith model and scenario evaluating to "current"23:42
prologicand year to "76to05"23:42
prologicso that I don't have to write a special case to handle processing it in fabfile/data.py or scripts/gensummaries23:43
prologicProblems with this approach (for now) ?23:43
DanielBairdso the path will be summaries/current/current/76to05/IBRA ?23:43
prologicno23:44
prologicsummaries are done per region23:44
prologichang on I'm confused23:44
prologic:)23:44
prologiclet me let it run and test her efirst23:44
prologicI'm weary that I only have ~6hrs to do stuff23:45
prologicso writing as little code as possible whilst still achieving required functionality is ideal :)23:45
DanielBairdthe paths for non-current are http://10.0.0.2/testapi/summaries/RCP3PD/csiro-mk30/2015/IBRA/23:45
prologicoh yes of course23:45
prologicyes you are quite right then23:45
prologicit'll be exactly as you sid23:45
prologicI know bit confoluted23:45
prologicbut no change in code required if I do that23:46
prologicI had to take the current.zip from wallace23:46
DanielBairdnp that's what i originally thought we'd go with23:46
prologicand re-zip it with that directory structure23:46
prologiccool23:46
prologicwe can tidy up urls and build a  better api, etc23:46
prologicas time permits23:46
prologicor if time permits23:46
prologicbut this should work just like any other model23:46
prologicif this works23:46
prologicI'll try and build a feature api to gain access to shapefiles23:47
DanielBairdi've got special handling for current, i'll leave it in and use that url as the special one.  so if we come back later and tidy it up, it's easy to switch back to just /current/IBRA23:47
prologicand I'll see if I have time to chuck in and test NRM and LGA regions as well23:47
prologic*nods*23:47
prologicof course I didn't simplify the current data set at all23:49
prologicso it has the full 19 bioclims23:49
prologicoh well23:49
prologicperhaps we should switch to the full datasets for the other two models now23:49
prologicbtw23:59

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!