IRC Logs for #cmvt Wednesday, 2013-07-03

prologicI'm really sorry for what I'm about to do too btw00:00
prologicyou know how we have /testapi00:00
prologicwhich is mostly a dumb api00:00
prologicserving up statically generated data00:00
prologicwell the features api I'm going to put under /api/features00:00
prologicjust to make your life living hell :)00:00
prologicand yes it will be json schema based (somewhat)00:00
prologicat least I know and am comfortable with enough of it to make it so00:00
prologicI'll probably only support bbox querying/filtering for now though00:01
prologicunless you need something else as well?00:01
prologicfull 19 bioclim current data set is taking too long to test as well00:03
prologicstripped it back to 2 bioclims for now00:03
prologiconce we're happy with functionality00:03
prologicwe can throw the real full datasets in00:03
prologicdeploy will just take hours/days00:04
prologicyeah it is my great hope everything will live under /api eventually00:04
prologicwith partial json schema support at least00:04
DanielBairdhow much difference would it be to make it three bioclims?00:13
DanielBairdactually nvm two is okay00:14
prologicno idea in terms of time to deploy00:24
prologicit currently is taking over 10mins to deploy right now00:24
DanielBairdi guess that's download time too, but yeah two's fine, it just looks odd graphing just two vars.  but the graph visuals are all good no need to fiddle any further, until i get all excited and add fancy scrub handles or something00:28
prologicyeah time to download00:34
prologiccreate appropriate structure00:34
prologicgenerate summaries00:34
prologicgenerate variables00:34
DanielBairdactually my graph module detects the region name is numeric (coz it currently just the region id, i guess) and graphs that.  so there's three vars being graphed at present anyway :)00:35
prologicjust fixing a couple of bugs I noticed with gensummaries00:36
prologicpush that up shortly00:36
*** robert_pyke has joined #cmvt01:10
prologicalmost finished the features api01:33
prologicjust need to test it now01:33
prologicDanielBaird:  okay you've got a features api now01:59
DanielBairdcool, nearly got the final async data loading thing working.02:00
DanielBairdi'm off to get a coffee though. ttyl02:00
prologichi robert_pyke02:00
prologicDanielBaird:  ping?05:36
prologichow's the ajaxu stuff going?05:36
prologicdo we just have (for me) the map layer serving stuff left to go when I get back?05:37
DanielBairdit's loading properly, just re-syncing the reload-the-graph stuff now.05:37
DanielBairdmap layers, and i guess the polygons at whatever simplification level seems okay05:37
prologicwell the polygons you can already get05:38
prologicat /api/features05:38
DanielBairdand we still need states, region names etc in the summaries, i haven't fetched today so maybe that's there already05:38
prologicbut I'm not doing any simplification at this stage05:38
prologicso be careful05:38
prologicmap layers will have to wait till I get back05:38
prologicre map layers05:38
prologicthings to consider are:05:38
prologictile indexing (tool done for this)05:38
prologicdelta processing (tool needs to written)05:39
prologicapi to let you load up any request layer (todo)05:39
DanielBairdyeah no worries, i'll presume that there's some way for me to get a WMS url if i know the scenario/year /etc, and test on other wms layers05:39
prologicDanielBaird: and we still need states, region names etc in the summaries, i haven't fetched today so maybe that's there already <-- already there05:39
prologicif you pull/fetch you'll get that too05:39
DanielBairdyeah just didn't want to commit until i had the data refreshing working again05:40
prologicyes the idea with the mapping apis05:40
prologicwould be to give you WMS access05:40
prologicusing mapscript05:40
prologicand I'd write the api code to generically load up any layer05:40
prologicbased on parameters you pass05:40
DanielBairdyep as long as it boils down to a url it'll just be a quick string assembly once you get back05:41
prologicwell I'm almost outta here05:51
prologicso enjoy your time with Plone and Diazo :)05:51
DanielBairdcool have fun.  taking hte family somewhere?05:53
prologicto Tassy05:58
prologicbrrrr cold :)05:58
DanielBairdawesome, i had a great tassie holiday a few years ago. go on the macquarie harbour / sarah island tour05:58
prologiccool thanks06:09
prologicwe'll check it out06:09
prologichey with this clustering stuff, postgis, etc06:09
prologicstuff rob is going06:09
prologicwhat's the end result?06:09
prologicmap tiles?06:09
prologicnevermind :006:13
DanielBairdthe end result of clustering is geojson features06:13
DanielBairdis that what you mean?06:13
prologicso you end up with a bunch of polygons06:13
prologicrepresenting species distriibutions?06:13
DanielBairdprobably points06:13
DanielBairdit's mostly for displaying occurrences06:14
DanielBairdnot sure if he's doing anything special with polys06:14
DanielBairdor even distributions06:14
DanielBairdbut occurrence clustering is required, and hard to do quickly06:14
prologicsurely it could be precomputed?06:15
prologicand cached?06:15
DanielBairdyeah that's right06:15
prologicso why use postgis then?06:15
prologicprecompute and cache it06:15
prologicjust curious06:15
DanielBairddunno, Rob's honours thesis is about the quickest way to do good clustering.  he's finding centroids of clusters i think..06:16
prologici see06:16
DanielBairdi think he's adding powers to postgis, or something06:16
prologicadding powers to postgis?06:16
DanielBairdbut it's just algorithms, i suspect it could just as easily be applied to not-postgis06:16
DanielBairdyeah there's postgis intergrations in ruby and python, he's adding (i think) additoonal features to those intergration layers06:17
DanielBairdlol same typo twice in a row06:17
DanielBairdif you're storing geo-data in a database, i think you definitely want to use postgis to access it.  you can also just hard-arse the lat/long columns if you like, but it's hard to approach postgis's performance06:18
prologicyeah I realize if you do any kind of his processing in a database06:20
prologicyou need to use a spatially aware database06:20
prologicI don't get the point of postgis here06:20
prologicor why it's being used06:20
prologicwhen you can clearly precmpute the cache the results of the species occurences06:20
prologicand even perhaps store that in a simple shapefile06:20
DanielBairdwhat is there instead of postgis?06:20
prologicI assume clustering is well understadn06:21
prologicand we're not developing an entirely new algorithm here?06:21
robert_pykehello, I am here.06:21
prologicI think my point is06:21
prologicthe computation could be done once06:22
prologicin a  tool06:22
prologicand written out to s shapefile06:22
prologicpostgis (btw) is just postgresql + proj + geos + gdal06:23
prologicso if you're not using postgis (like I'm not)06:23
DanielBairdi think that's the idea, and postgres/postgis/rob's clustering is the tool in that story.06:23
prologicthen I'm using gdal, goes, proj06:23
prologicand several python abstraction libraries on top of those06:23
prologicokay sure06:23
prologicso rob's using postgis as his "tool" to do the calculations06:24
prologicthat's fine06:24
prologicihmo something like this should be written as a tool though06:24
prologicand integrated into a general visualization tool06:24
prologicsuch as the one you and I have been working on06:24
DanielBairdagree, if you mixed our region-awareness with robert's occurrence display/handling, and maybe Edgar's vetting interface, that would make an awesome geo-data browser06:25
prologicwe really ought to work towards this kind of effort06:26
prologicI greatest hope in the months I've spent on this06:26
prologicis to be able to continue to develop it06:26
prologicand re-use it for bCCVL as well06:26
prologicfor their visualization06:26
prologicso what rob's working on should really be done as a small tool06:26
prologicthat integrates with ccav (badly named)06:26
prologicif the output is a simple statically precomputed Shapefile06:27
robert_pykeWhy am I using PostGIS? As you noted, it's a spatially aware database.06:27
prologicthen we can just shove that in and serve it up and write UI logic around it06:27
robert_pykeAs such, we can do our spatial overlap operators on it (bounding box queries).06:27
robert_pykeIt uses spatial columns to speed up lat/lng calculations, such as the overlap.06:27
robert_pykeI've used it previously to perform on the fly clustering, as well as pre-calc'd clustering with cache tables.06:27
robert_pykeIt also outputs GeoJSON, which has been convenient for embedding extra information in feature requests.06:27
robert_pykeOther than that, no special reason.06:27
prologicYeah I get all that06:28
prologicthe same can be achieved with python libraires06:28
prologicgdal, ogr, fiona and shapely06:28
robert_pykeIt's mainly been useful for the GeoJSON aspect, and embedding extra information from the database.06:28
prologicwith very similar performance characteristis06:28
robert_pykesuch as the import time, etc.06:28
prologicso you're using postgis as a tool06:28
prologicfor the same functionality that I'm using06:28
prologicbecause underneath it all uses the same libraries06:28
prologicgdal, ogr, geos, proj06:28
DanielBairdin the bccvl visulaliser applcation, the occurrence data is unlikely to remain long-term in postgres; it's more likely that fresh imports will be dropped in, visualised/clustered, and the cluster result cached for a little while.06:29
prologicI guess what would be (or would have been - you've probably already put a lot of work into this) is a tool (say in Python) that does the computations and writes the results out to a Shapefile06:29
prologicwhich we can then serve up, query and use in the ui06:30
prologicnew data would not be entered by users however06:30
prologicit would be done by a system maintainer06:30
prologicso you just make that part of the "import process"06:30
prologicto do another precompuation06:31
prologicalso Shapefiles are already GeoJSON06:31
prologicas-in each feature or the entire feature collection can already be exported to GeoJSon easily06:31
prologicpostgis just hooked into the ogr ExportToJson() call I imagine06:32
prologicas I said, postgis is just a very thing layer on top of goes, proj4 and gdal06:32
prologicanyway I'm off on my holiday :)06:32
prologicfood for thought :)06:32
prologicI still want to have one visualization system06:32
prologicwe need to integrate our work(s) :)06:33
prologiccyan in a a week and a half06:33
*** DanielBaird has quit IRC08:01
*** DanielBaird has joined #cmvt08:31
*** DanielBaird has quit IRC08:40
*** DanielBaird has joined #cmvt09:06
*** DanielBaird has quit IRC09:11
*** DanielBaird has joined #cmvt10:08
*** DanielBaird has quit IRC10:13
*** DanielBaird has joined #cmvt11:08
*** DanielBaird has quit IRC11:13
*** DanielBaird has joined #cmvt12:09
*** DanielBaird has quit IRC12:13
*** DanielBaird has joined #cmvt13:10
*** DanielBaird has quit IRC13:15
*** DanielBaird has joined #cmvt13:58

Generated by 2.11.0 by Marius Gedminas - find it at!