IRC Logs for #cmvt Monday, 2013-07-15

prologicI am back :)00:17
prologicDanielBaird:  ping?00:28
DanielBairdsorry in meetings still00:41
prologicah nps :)00:42
prologic Let me know when you're back01:29
prologicand what we're doing with ccav01:29
prologicI fetched your recent changes01:29
prologicbut notice 404's to getting summary data01:29
DanielBairdhi, I'm back. i've been doing BCCVL stuff the whole week and not done any work on ccav.  i think the 404s are from fetching LGA or something, that isn't available in the sample data set01:44
prologicwell you are half right01:58
prologicyou're trying to fetch summaries for LGA and NRM regions01:58
prologicwhich I haven't added to the data task to download and process as yet01:58
prologicthe other issue is an incorrect uri for current and IBRA01:59
prologicremember we agreed upon01:59
prologicGET /testapi/summaries/current/current/76to05/IBRA01:59
prologicI'll have to double check01:59
DanielBairdah that's right, i probably have my original special case still in place.01:59
prologicAlso I notice in your code you have the regions hard coded01:59
prologicperhaps it might be more convenient to move that to a ?02:00
prologicor something simple for now?02:00
prologicAlso I realize you're working on BCCVL as well now02:00
prologicas I haven't been told otherwise (AB, Mark, etc are away)02:00
prologicI'm assuming work is to continue on ccav as well02:00
DanielBairdyeah i expect to put in time finishing off ccav, bccvl just needed a big get-started push02:01
prologicDo you have time to work on ccav today?02:03
prologicOtherwise I'll work on improving the API and doing some performance enhancements to the data generation02:03
DanielBairdhmm i must have copied some data across back when i thought we were going to have /current/02:04
DanielBairdi'll remove it so i get the same 404s you get.02:04
DanielBairdum, i'll probably be tied up in bccvl meetings most of today.02:04
prologicyeah everything is generated automatically be fab data (and fab deploy)02:05
prologicif you shove in manual (by hand) data well nothing is guaranteed :)02:05
prologicyeah ok nps02:05
prologicLike I said I'll continue where I left off02:05
prologicimproving things02:05
DanielBairdi might spend some time going through the stuff the ccav needs and adding it all into pivotal or bitbucket issues or something02:05
prologicanything specific you want me to work on?02:05
prologiclet's go bitbucket issues if you don't mind?02:06
prologicI've found the pivotal tracker stores we came up with entirely useless02:06
DanielBairdprobably getting states into the output is the next biggest thing02:06
prologicas you've no doubt seen02:06
DanielBairdagree i think i'd prefer bb issues too02:06
prologicthey haven't really married up with what we're doing exactly02:06
prologicStates into the summaries is done02:06
prologicDo you want me to shove in LGA and NRM as well?02:06
DanielBairdwe *should* have been carefully pruning the stories etc, but that's hard to do when there's no proper client to see them02:06
prologicalso hard to do when we're trying to wear "many hats"02:07
prologicI've found this project thus far to be very much evolutionary in process and development02:07
prologicit's all been rather R&D really02:07
prologicprobably more for me than you :)02:07
DanielBairdyes good idea.. actually if you made "fab testdata" and "fab data" then you could stop having to worry about limiting the datasets for the testing stuff02:08
DanielBairdand yeah when it's just you and me doing any active conversation, there's no motivation to make our conversations any more formal that this02:08
prologicagreed and agreed02:09
prologicI'll create a testdata task just for the purpose02:09
prologicI may as well shove in LGA and NRM regions though02:09
prologicwe already have States02:09
prologicno new code is actually required to support other region types02:10
prologicalso before I left I also added a "features" api02:10
prologicso you could get geojson features, etc02:10
prologicyou probably have not had time to try it though :)02:10
DanielBairdyeah i keep thinking i'll start on the map properly, but then hitting one more little thing to fix on the graph page.. oh actually we need to work out what "download this set" will do02:12
prologicoh ok02:15
prologicwell I'm going to improve the overall performance of gensummaries today/tomorrow02:16
prologicI'm finding it terribly slow02:16
prologicand increasingly so when more regions are added to the mix02:16
prologicwhat is this "download set" though?02:17
prologicwhat kind of data are we serving up?02:17
prologicIs it something I could just pre-generate?02:17
prologicOr something I could dynamically put together based on existing pre-generated data?02:18
DanielBairdactually i don't know for sure, just that there were download buttons on the mockup02:19
DanielBairdi presume it's all the climate vars for the selected regions.02:19
DanielBairdbut honestly it's hard to see why that's any better than just getting all regions and doing filtering in your spreadsheet program02:19
prologicwell perhaps we need to discuss this with AB/SM ?02:23
prologicIn any case I'm going to have LGA/NRM regions added in today02:24
prologicbut possibly nothing else of value02:25
prologicwill be trying to improve performance of gensummaries mainly02:25
prologicI'm going to go get some lunch, I'll be back in a few mins -- will have lunch at my desk today02:32
DanielBairdyou'd have liked bccvl, my UI for that is all Twitter Bootstrap03:10
DanielBairdthe page is allowed to scroll, so there's no need to fanciness with positioning etc03:11
prologicI quite like what Twitter Bootstrap has to offer tbh03:12
DanielBairdyes it's awesome, more or less.  it needs a lot of extra dom elements, and it gets a little annoying once you need to do something not officially supported, but i suspect that's the result of some pretty good tradeoff decisions03:20
prologicwell the thing I find most appealing is a standard-ish way of dealing with UI design03:29
prologicand clean good use of CSS/HTML03:29
DanielBairdyeah the super consistent, clean looking styling is great03:31
prologiceven across many devices03:35
prologicit's good03:35
DanielBairdyeah.. it's the promise of CSS, but actually implemented properly, by experts, and refined over time to work sensibly in every browser03:35
prologichave added in LGA and NRM regions03:37
prologicwill commit that shortly03:37
prologicstill doing performance analysis on gensummaries03:38
prologictell me something03:41
prologicwould it be useful to split up all the raster layers (the models) by intersecting them against every region geometry03:41
prologicand outputting the resulting array as a separately named GeoTIFF file03:42
prologicfor mapping purposes03:42
prologicI'm thinking this because doing this03:42
prologicwould simplify the problem of generating summaries I think03:42
prologicand allow me to take full advantage of multi-processing03:42
prologicsince the problem of intersecting every raster layer with every region geometry is embarrassingly paralell03:43
prologicthe gensummaries itself could then be simplified to simplify find all such files and iterate over them -- also embarrassingly parallel03:44
DanielBairdhowever, everyone else i've mentioned it to thinks that actually it *is* useful to see just that region's pixels03:44
prologicI mean map server does this anyway by default03:44
DanielBairdi agree that it would make the summaries easier03:44
prologicby specifying bounding box03:44
prologicit cuts up the currently visible raster layers anwyay03:45
prologicdynamically though03:45
prologicI'd just be doing the same thing create many potentially 10s of thousands of smaller layers03:45
prologicI think it would make generating summary data easier in terms of simpler code03:45
prologicbut also easier to paraellize03:45
prologicsince it's two discrete pressing steps across many inputs03:46
prologicalso for example03:46
prologicgensummaries could be called per region type03:46
prologicand run them in the background03:46
prologicthe OS would of course run them in separate processes (cpu cores)03:46
prologicso in that sense the time taken would be the time taken for the largest set of regions03:47
DanielBairdit does seem simpler.  what would the storage requirements be for keeping all the regional sub-rasters, for every region / scenario / year / GCM / etc?  if it's not much, then hell yes let's keep them for maybe using in the UI03:54
prologicWell ...03:55
prologicThink about it03:55
prologicit would be as large as the original data set(s)03:56
prologic* the no. of region types * the no. of regions per region type03:56
DanielBairdnumber of files.. do the geotiffs get much smaller when they're reduced to just a region?03:57
prologicyes significantly03:57
prologicbecause the array is smaller03:57
prologicbut the sum of 1 scenario/model/year/bioclim geotiff file cut up into smaller regions of one region type03:58
prologicwould equal the original geotiff03:58
DanielBairdso it's more-or-less the same storage space over again.. that's not bad at all03:58
prologicso no. of geotiff files * region_types * regions_per_type03:58
prologicyea it would be03:58
prologicso say we have 10 geotiffs now03:59
prologicand only one region type (IBRA)03:59
prologicwhich has 90 geometries03:59
prologicwe'd have 90 additional geotiffs03:59
prologic180 for two bioclims03:59
prologicdon't forget we also have to compete differences for all raster layers against current04:04
prologicand store those as well for mapping purposes04:04
prologicI developed the code for this months back04:04
prologicbut haven't integrated it into the current codebase yet04:04
prologicI'll write up an issue for it04:04
prologicthe so-called "delta layers"04:05
DanielBairdoh yeah that's right, deltas04:06
prologicalso while I'm on the subject of mapping04:08
prologicand in the mood to "exactly" describe outstanding tasks04:09
prologicwe (really me)04:09
prologicalso need to produce a higher level tool that uses the existing "tileindex" tool I wrote specifically for the job to generate tile indexes for all layers04:09
prologicincluding normal layers04:09
prologicand delta layers04:09
prologicbasically this allows you from a mapping point of view to take into account "time"04:10
prologicthe tool was specifically written as an enhancement to gdal's identically named tileindex tool but with the support of an extra dimension "time".04:10
prologicit quite literally generates a Shapefile database mapping of geometry, data source and time04:13
DanielBairdyes it's time to push everything into issues, isn't it04:13
prologicso I have to figure out how this tool I wrote works again04:13
prologicand how to utilize it against our entire data set(s)04:13
prologicdon't you hate forgetting how things work that you wrote :)04:13
prologic well we mostly know what we're doing now04:14
prologicsince we have something to work with :)04:14
prologicand not "guessing" :)04:14
prologici.e: things are coming out of "realizations"04:14
prologicso I'll document this issue as well now04:14
DanielBairdand we'd like to wrap it up.. so having a defined list of stuff means we can plan to get it completed04:14
prologicand lastly04:17
prologicwe need to discuss, plan and develop our "WMS"/"WMS-T" service for you04:18
prologicso that you can actually request layers and delta layers04:18
prologicwith some flexibility on your end04:18
prologici.e: my plan was to use map script and python04:18
prologicand provide you the enough flexibility to request any layer from any scneario+model+year04:19
prologicand tell me if you want normal or delta04:19
prologicetc etc04:19
prologicwould you like to write up said issue?04:19
prologicsince it's more about how you want it done04:20
prologicand less about me04:20
prologicI'll simply be coding up what you think would be sufficient04:20
DanielBairdyes okay.. i'll poke in a placeholder and fill it out a bit later04:20
prologicwithout the need to generate or write many map files04:20
prologicthis should be done dyncmially04:20
DanielBairdI'm expecting jeremy to pop up sometime and drag me off to a meeting04:20
DanielBairdaaaany second now..04:20
prologicquite write up a placeholder :)04:21
prologicthat's the only task left that I can thin of04:21
prologicthat actually requires some level of discussion bweteen us04:21
prologicalso I've not done anything with amp script before04:21
prologicbut I assume it's fairly straight forward04:21
DanielBairdthis is a thing that Robert can help with04:29
DanielBairdhe's been talking about mapscript a lot over the last week or so04:29
prologichis input would be very valulable04:51
prologicbasically for that task04:51
prologicI want to avoid generating map files at all04:52
prologicand make the whole process of serving up map tiles dynamic04:52
prologicwith an optional tile cache in front of it04:52
*** DanielBaird has quit IRC06:31
*** DanielBaird has joined #cmvt06:48
DanielBairdokay bailing.. was sucked into meetings all afternoon.  see you07:05
*** DanielBaird has quit IRC08:08
*** DanielBaird has joined #cmvt08:38
*** DanielBaird has quit IRC08:47
*** DanielBaird has joined #cmvt09:13
*** DanielBaird has quit IRC09:18
*** DanielBaird has joined #cmvt10:14
*** DanielBaird has quit IRC10:18
*** DanielBaird has joined #cmvt11:42
*** DanielBaird has quit IRC11:47
*** DanielBaird has joined #cmvt12:42
*** DanielBaird has quit IRC12:47
*** DanielBaird has joined #cmvt13:43
*** DanielBaird has quit IRC13:48
*** DanielBaird has joined #cmvt13:58
*** DanielBaird has quit IRC14:24
*** DanielBaird has joined #cmvt14:55
*** DanielBaird has quit IRC15:04
*** DanielBaird has joined #cmvt15:30
*** DanielBaird has quit IRC15:36
*** DanielBaird has joined #cmvt16:32
*** DanielBaird has quit IRC16:36
*** DanielBaird has joined #cmvt17:38
*** DanielBaird has quit IRC17:40
*** DanielBaird has joined #cmvt18:41
*** DanielBaird has quit IRC18:45
*** DanielBaird has joined #cmvt19:41
*** DanielBaird has quit IRC19:45
*** DanielBaird has joined #cmvt20:42
*** DanielBaird has quit IRC20:46
*** DanielBaird has joined #cmvt21:42
*** DanielBaird has quit IRC21:47
*** DanielBaird has joined #cmvt22:40
prologicDanielBaird:  Good Morning23:25
*** robert_pyke has joined #cmvt23:36
prologicmorning rob23:38

Generated by 2.11.0 by Marius Gedminas - find it at!