IRC Logs for #cmvt Sunday, 2013-04-07

*** pvrdwb has joined #cmvt15:13
prologicGood Mroning15:57
pvrdwbi didn't do any Angular over the weekend.. i forgot on thursday, my family are in town16:30
pvrdwbon the other hand, i've been crabbing on the beach with my son and his cousin.  so that might help out project.16:30
pvrdwb* out16:30
pvrdwb** our16:30
prologicgood man :)16:32
prologicI'm glad you put family first! :)16:32
prologicOk well on Friday I basically played around more with WFS and MapServer and GeoJson16:32
prologicgot GeoJSon output to work with WFS at least16:32
prologicgot some basic intersection stuff working with shapely, pyproj and the IBRA7_regions Shapefile16:33
pvrdwbcool.. this morning i worked through the angular tutorial that i think describes how i'd be fetching map layers and hooking them in to thingy. Leaflet.16:33
prologicyeah I read through AngularJS myself on Friday on the bus trip home16:35
prologicI'm in two minds about it :)16:35
prologicOn one hand hopefully it means you don't have to program any javascript per say16:35
prologicOn the other hand, it abuses the DOM16:35
pvrdwbEmber and Knockout are the other structured options, and they similarly dick around in the dom.16:37
pvrdwbember was my other front runner, it uses string templates that you have in your page but inside <script> tags  (so technically the page is valid) but when it assembles the dom it drops loads of sync tags with special IDs to do the data binding16:39
prologicyeah *nods*16:40
prologicat this point I don't really care too much16:40
pvrdwbthey're valid DOM elements, but messy looking in your inspector16:40
prologicI'm sort of over a lot of front-end web technologies :)16:40
prologicso let's go with AngularJS and see how it pans out :)16:40
pvrdwbbut the main reason i went for angular vs ember was ember wants to do a full model/view/controller layout, and i felt like it was unnecessary to re-do structure that properly belongs on the server16:40
prologici.e: for lack of better alternatives - short of writing a desktop app :)16:40
prologicI'm not particularly a fav of MVC or MVVM anyway16:41
prologicoften they're complex patterns on a much simpler probem16:41
pvrdwbyeah Angular was closest to actual web page development, rather than trying to map web apps into the traditional server patterns16:41
prologicbut let's go! :)16:41
prologicso I dunno if you've had any experience with WFS with MapServer16:42
prologicbut it's looking a little disappointing16:42
pvrdwbno not really16:42
prologicI can get GeoJSON output fairly easily16:42
prologicbut it's slow via the CGI16:42
prologicso I think I need to get rid of that whole layer of I/O16:42
prologicand go direct to MapScript16:42
prologiceliminitationg a whole layer of pipes16:42
pvrdwbdoes anyone use WFS properly?16:43
prologicAlso, it seems in order to do basic vector operations with WFS (e.g: intersections, which WFS/MapServer calls WFS) you have to do things like &filter=<some big long ugly xml>16:43
pvrdwbor is it the way things "should" work but no-one actually uses it16:43
prologicWell that's the thing16:44
prologicI can find very little on usages of WFS with MapServer16:44
prologicin ArcGIS Server for example16:44
prologicthey have an entire RESTful interfaces16:44
prologicthat speaks JSON (ESRI JSON)16:44
prologicI honestly actually think that not many GIS apps are built on top of MapServer at all16:44
prologicI think there ar emote built with GeoServer16:45
prologicand the rest are built with ESRI ArcGIS16:45
prologicIHMO :)16:45
prologicnevertheless MapServer is a perfectly fine WMS (Web Map Service) for generating map tiles16:45
prologicbut I'm a little iffy on the whole WFS (Web Feature Serivce) side16:45
pvrdwbyeah i've only really used map server for pushing pixels around16:47
prologicin any case if MapServer's WFS sucks ass16:47
prologicI can build a set of RESTful interfaces to do what we want16:47
prologicit may not be as feature rich16:47
prologicbut then again neither is MapServer's WFS anyway16:47
prologicI think it only supports a handful of operations16:48
pvrdwbis there interface we need defined as "whatever leaflet uses" in any case?16:48
prologicHey AB is asking16:49
prologicYou going to make that meeting  at 1100?16:49
prologicWell leaflet out-of-the-box doesn't speak WFS16:49
prologicit has Vector supports in terms of various Vector objects you can deal with on the Map16:49
prologicBut you'd have to feed/create those yourself16:49
pvrdwbi thought i'd clicked "coming" on the invite.. maybe i didn't16:49
prologicso in the simplest case I'd just feed you some GeoJSon16:49
pvrdwbyeah that's what i've done in open layers.. added map server raster layers, but handled vector stuff directly16:51
prologicthen that's what we'll do then16:51
prologicand I'll either provide that via a spatialite or postgis backend16:52
prologicor directly from the shape files themselves loaded in memory16:52
prologicI might compare and see if spatiality is particularly any faster than shape file + shapely16:52
pvrdwbback.  guess we'll be talking for real in a few mins17:00
prologicI was just going to say, if you're going to sort of rewrite the user stories, let's do so in pivotal tracker?18:03
pvrdwbyeah i think i'll want to do that.18:06
prologicI might go get some lunch shortly18:12
prologicquickly comparing the performance of spatialite18:12
prologichey Dan20:18
prologicIs that the same 250m sample data you linked me before?20:18
prologicRob's thesis in webapp/tests/fixtures/sample_250m_ascii_grid.asc.gz ?20:18
prologicDanielBaird:  ping?21:04
DanielBairdum, i think it is the same.21:04
DanielBairdAndrew thought you didn't have it yet, and Rob pointed to this new repository.  but i think the data is the same21:05
prologicyeah ok nps21:10
prologicOk so here's the thing21:10
prologicI don't get what Rob's claims were with rendering performance21:10
prologicI can render 1k and 50m just fine21:10
prologicthere is no performance issues21:10
DanielBairdworth asking him about.. i think he's using map script but it could be memory bound or something21:15
DanielBairdi think he's off doing courses now though21:17
prologicwell I'm not sure21:19
prologicask him to poke around here when he has some time21:19
prologicbut I looked at his ruby code21:19
prologicand he's testing the time it takes to:21:19
prologica) uncompress the 1.6G .asc.gz file21:19
prologic2) create a map file21:19
prologic3) perform mapscript processing21:20
prologic4) render the image21:20
prologicI dunno what sort of hw specs his test machine is21:20
prologicbut it takes at least 13s on my iMac just to uncompress the .asc.gz itself21:20
prologicmap server however, given GeoTIFF data21:20
prologiccan render the entire map of Australia at 250m at the same level of performance that 5k and 1km can21:21
DanielBairdthey're ascii grids though right?  he's probably giving map server the asciis, so perhaps the slowdown is all in ascii parsing21:21
DanielBairdi presume you're converting to geotiff21:22
prologicI think the slowdown is in two places:21:27
prologica) uncompressing the .sac.gz file in the first place as part of his test21:27
prologicb) dealing with .asc (ASCII Grid) files21:27
prologicrendering performance itself of 250m resolution data is fine21:27
DanielBairdhmm yes looks like he's including unzip in his test timing.  perhaps he's expecting to have the raw data zipped21:31
prologicI think that's a silly idea21:32
prologicconverting it to GeoTIFF saves quite a lot of space21:32
prologicand leaves it in a form that map server can deal with straight away21:32
prologicas well as any other kind of processing21:32
prologice.g: processing with python and bumpy using the fiona library (wrapping gdal)21:33
prologicsorry fiona is for vector (ogr)21:33
prologicmy bad :)21:33
DanielBairdyeah Jeremy's modelling needs asciigrids, so for the Edgar project, where the modelling happens continuously in the background, we kept stuff as asciigrids.  this project doesn't have any reason to keep data as ascii21:34
prologicno, no it doesn't21:35
DanielBairdit's terrible the wasted space in an ascii grid file.. even NULs take many chars to store21:35
prologicbut I still question the timing of the uncompression21:35
prologicand is Jeremy's software so limited that it can't deal with anything else but .asc?21:35
DanielBairdwell you should ask jeremy about how limited his modelling is :) but i think that's what maxent takes.  i don't know about dismo.  i can't imagine it would be a big deal to convert before use in any case21:36
DanielBairdis there any info that an ascii grid carries that doesn't go into a geotiff?? my impression is that geotiff has all the normal fields21:37
prologicit's no wonder he has to run this stuff on a cluster21:37
prologicsac and grotiff are equivilent21:37
DanielBairdyeah i presume it's just the path of least resistance -- asciigrids are easy to deal with as plain text files, and once it works, i bet no academic modellers ever come back to it21:39
DanielBairdin fact maybe R can import ascii more easily than geotiff.. it's maxent in java that does the mathematical stuff but Jeremy uses R for all the custom bits, like excluding offshore pixels etc21:44
prologicman I can exclude all off-whore pixels just be doing21:47
prologicx = gdal.Open(src, gdalconst.GA_ReadOnly)21:47
prologicxs = x.ReadAsArray()21:47
prologicxs[xs == -9999] = np.nan21:47
prologicand it's so uber fast!21:47
prologiceven at 250m21:47
DanielBairdbut putting the -9999s in there, i think that was an R script21:48

Generated by 2.11.0 by Marius Gedminas - find it at!