| prologic | I am back :) | 00:17 |
|---|---|---|
| prologic | DanielBaird: ping? | 00:28 |
| prologic | :) | 00:28 |
| DanielBaird | sorry in meetings still | 00:41 |
| prologic | ah nps :) | 00:42 |
| prologic | Let me know when you're back | 01:29 |
| prologic | and what we're doing with ccav | 01:29 |
| prologic | I fetched your recent changes | 01:29 |
| prologic | but notice 404's to getting summary data | 01:29 |
| DanielBaird | hi, I'm back. i've been doing BCCVL stuff the whole week and not done any work on ccav. i think the 404s are from fetching LGA or something, that isn't available in the sample data set | 01:44 |
| prologic | well you are half right | 01:58 |
| prologic | you're trying to fetch summaries for LGA and NRM regions | 01:58 |
| prologic | which I haven't added to the data task to download and process as yet | 01:58 |
| prologic | -but- | 01:58 |
| prologic | the other issue is an incorrect uri for current and IBRA | 01:59 |
| prologic | remember we agreed upon | 01:59 |
| prologic | GET /testapi/summaries/current/current/76to05/IBRA | 01:59 |
| prologic | IIRC | 01:59 |
| prologic | I'll have to double check | 01:59 |
| DanielBaird | ah that's right, i probably have my original special case still in place. | 01:59 |
| prologic | Also I notice in your code you have the regions hard coded | 01:59 |
| prologic | perhaps it might be more convenient to move that to a config.coffee ? | 02:00 |
| prologic | or something simple for now? | 02:00 |
| prologic | Also I realize you're working on BCCVL as well now | 02:00 |
| prologic | as I haven't been told otherwise (AB, Mark, etc are away) | 02:00 |
| prologic | I'm assuming work is to continue on ccav as well | 02:00 |
| DanielBaird | yeah i expect to put in time finishing off ccav, bccvl just needed a big get-started push | 02:01 |
| prologic | *nods* | 02:02 |
| prologic | Do you have time to work on ccav today? | 02:03 |
| prologic | Otherwise I'll work on improving the API and doing some performance enhancements to the data generation | 02:03 |
| DanielBaird | hmm i must have copied some data across back when i thought we were going to have /current/ | 02:04 |
| DanielBaird | i'll remove it so i get the same 404s you get. | 02:04 |
| DanielBaird | um, i'll probably be tied up in bccvl meetings most of today. | 02:04 |
| prologic | ahh | 02:04 |
| prologic | yeah everything is generated automatically be fab data (and fab deploy) | 02:05 |
| prologic | if you shove in manual (by hand) data well nothing is guaranteed :) | 02:05 |
| prologic | yeah ok nps | 02:05 |
| prologic | Like I said I'll continue where I left off | 02:05 |
| prologic | improving things | 02:05 |
| DanielBaird | i might spend some time going through the stuff the ccav needs and adding it all into pivotal or bitbucket issues or something | 02:05 |
| prologic | anything specific you want me to work on? | 02:05 |
| prologic | let's go bitbucket issues if you don't mind? | 02:06 |
| prologic | I've found the pivotal tracker stores we came up with entirely useless | 02:06 |
| DanielBaird | probably getting states into the output is the next biggest thing | 02:06 |
| prologic | as you've no doubt seen | 02:06 |
| DanielBaird | agree i think i'd prefer bb issues too | 02:06 |
| prologic | they haven't really married up with what we're doing exactly | 02:06 |
| prologic | States into the summaries is done | 02:06 |
| prologic | Do you want me to shove in LGA and NRM as well? | 02:06 |
| DanielBaird | we *should* have been carefully pruning the stories etc, but that's hard to do when there's no proper client to see them | 02:06 |
| prologic | also hard to do when we're trying to wear "many hats" | 02:07 |
| prologic | I've found this project thus far to be very much evolutionary in process and development | 02:07 |
| prologic | it's all been rather R&D really | 02:07 |
| prologic | probably more for me than you :) | 02:07 |
| DanielBaird | yes good idea.. actually if you made "fab testdata" and "fab data" then you could stop having to worry about limiting the datasets for the testing stuff | 02:08 |
| DanielBaird | and yeah when it's just you and me doing any active conversation, there's no motivation to make our conversations any more formal that this | 02:08 |
| prologic | agreed and agreed | 02:09 |
| prologic | I'll create a testdata task just for the purpose | 02:09 |
| prologic | I may as well shove in LGA and NRM regions though | 02:09 |
| prologic | we already have States | 02:09 |
| prologic | no new code is actually required to support other region types | 02:10 |
| prologic | also before I left I also added a "features" api | 02:10 |
| prologic | so you could get geojson features, etc | 02:10 |
| prologic | you probably have not had time to try it though :) | 02:10 |
| DanielBaird | yeah i keep thinking i'll start on the map properly, but then hitting one more little thing to fix on the graph page.. oh actually we need to work out what "download this set" will do | 02:12 |
| prologic | oh ok | 02:15 |
| prologic | sure | 02:15 |
| DanielBaird | or | 02:15 |
| prologic | well I'm going to improve the overall performance of gensummaries today/tomorrow | 02:16 |
| prologic | I'm finding it terribly slow | 02:16 |
| prologic | and increasingly so when more regions are added to the mix | 02:16 |
| prologic | what is this "download set" though? | 02:17 |
| prologic | what kind of data are we serving up? | 02:17 |
| prologic | Is it something I could just pre-generate? | 02:17 |
| prologic | Or something I could dynamically put together based on existing pre-generated data? | 02:18 |
| DanielBaird | actually i don't know for sure, just that there were download buttons on the mockup | 02:19 |
| DanielBaird | i presume it's all the climate vars for the selected regions. | 02:19 |
| DanielBaird | but honestly it's hard to see why that's any better than just getting all regions and doing filtering in your spreadsheet program | 02:19 |
| prologic | well perhaps we need to discuss this with AB/SM ? | 02:23 |
| prologic | https://bitbucket.org/ccaih/ccav/issue/6/performance-enhancements-for-gensummaries | 02:24 |
| prologic | In any case I'm going to have LGA/NRM regions added in today | 02:24 |
| prologic | but possibly nothing else of value | 02:25 |
| prologic | will be trying to improve performance of gensummaries mainly | 02:25 |
| prologic | I'm going to go get some lunch, I'll be back in a few mins -- will have lunch at my desk today | 02:32 |
| DanielBaird | you'd have liked bccvl, my UI for that is all Twitter Bootstrap | 03:10 |
| DanielBaird | the page is allowed to scroll, so there's no need to fanciness with positioning etc | 03:11 |
| prologic | I quite like what Twitter Bootstrap has to offer tbh | 03:12 |
| DanielBaird | yes it's awesome, more or less. it needs a lot of extra dom elements, and it gets a little annoying once you need to do something not officially supported, but i suspect that's the result of some pretty good tradeoff decisions | 03:20 |
| prologic | well the thing I find most appealing is a standard-ish way of dealing with UI design | 03:29 |
| prologic | :) | 03:29 |
| prologic | and clean good use of CSS/HTML | 03:29 |
| DanielBaird | yeah the super consistent, clean looking styling is great | 03:31 |
| prologic | *nods* | 03:34 |
| prologic | even across many devices | 03:35 |
| prologic | it's good | 03:35 |
| DanielBaird | yeah.. it's the promise of CSS, but actually implemented properly, by experts, and refined over time to work sensibly in every browser | 03:35 |
| prologic | *nods* | 03:37 |
| prologic | have added in LGA and NRM regions | 03:37 |
| prologic | will commit that shortly | 03:37 |
| prologic | still doing performance analysis on gensummaries | 03:38 |
| prologic | tell me something | 03:41 |
| prologic | would it be useful to split up all the raster layers (the models) by intersecting them against every region geometry | 03:41 |
| DanielBaird | yes..? | 03:41 |
| DanielBaird | hmm | 03:42 |
| prologic | and outputting the resulting array as a separately named GeoTIFF file | 03:42 |
| prologic | for mapping purposes | 03:42 |
| prologic | I'm thinking this because doing this | 03:42 |
| prologic | would simplify the problem of generating summaries I think | 03:42 |
| prologic | and allow me to take full advantage of multi-processing | 03:42 |
| prologic | since the problem of intersecting every raster layer with every region geometry is embarrassingly paralell | 03:43 |
| prologic | thoughts? | 03:43 |
| prologic | the gensummaries itself could then be simplified to simplify find all such files and iterate over them -- also embarrassingly parallel | 03:44 |
| DanielBaird | however, everyone else i've mentioned it to thinks that actually it *is* useful to see just that region's pixels | 03:44 |
| prologic | yeah | 03:44 |
| prologic | I mean map server does this anyway by default | 03:44 |
| DanielBaird | i agree that it would make the summaries easier | 03:44 |
| prologic | by specifying bounding box | 03:44 |
| prologic | it cuts up the currently visible raster layers anwyay | 03:45 |
| prologic | dynamically though | 03:45 |
| prologic | I'd just be doing the same thing create many potentially 10s of thousands of smaller layers | 03:45 |
| prologic | well | 03:45 |
| prologic | I think it would make generating summary data easier in terms of simpler code | 03:45 |
| prologic | but also easier to paraellize | 03:45 |
| prologic | since it's two discrete pressing steps across many inputs | 03:46 |
| prologic | also for example | 03:46 |
| prologic | gensummaries could be called per region type | 03:46 |
| prologic | and run them in the background | 03:46 |
| prologic | the OS would of course run them in separate processes (cpu cores) | 03:46 |
| prologic | so in that sense the time taken would be the time taken for the largest set of regions | 03:47 |
| DanielBaird | it does seem simpler. what would the storage requirements be for keeping all the regional sub-rasters, for every region / scenario / year / GCM / etc? if it's not much, then hell yes let's keep them for maybe using in the UI | 03:54 |
| prologic | Well ... | 03:55 |
| prologic | Think about it | 03:55 |
| prologic | it would be as large as the original data set(s) | 03:56 |
| prologic | * the no. of region types * the no. of regions per region type | 03:56 |
| prologic | right? | 03:56 |
| DanielBaird | number of files.. do the geotiffs get much smaller when they're reduced to just a region? | 03:57 |
| prologic | yes significantly | 03:57 |
| prologic | because the array is smaller | 03:57 |
| prologic | but the sum of 1 scenario/model/year/bioclim geotiff file cut up into smaller regions of one region type | 03:58 |
| prologic | would equal the original geotiff | 03:58 |
| DanielBaird | so it's more-or-less the same storage space over again.. that's not bad at all | 03:58 |
| prologic | so no. of geotiff files * region_types * regions_per_type | 03:58 |
| prologic | yea it would be | 03:58 |
| prologic | so say we have 10 geotiffs now | 03:59 |
| prologic | and only one region type (IBRA) | 03:59 |
| prologic | which has 90 geometries | 03:59 |
| prologic | we'd have 90 additional geotiffs | 03:59 |
| prologic | 180 for two bioclims | 03:59 |
| prologic | don't forget we also have to compete differences for all raster layers against current | 04:04 |
| prologic | and store those as well for mapping purposes | 04:04 |
| prologic | I developed the code for this months back | 04:04 |
| prologic | but haven't integrated it into the current codebase yet | 04:04 |
| prologic | I'll write up an issue for it | 04:04 |
| prologic | the so-called "delta layers" | 04:05 |
| DanielBaird | oh yeah that's right, deltas | 04:06 |
| prologic | also while I'm on the subject of mapping | 04:08 |
| prologic | and in the mood to "exactly" describe outstanding tasks | 04:09 |
| prologic | we (really me) | 04:09 |
| prologic | also need to produce a higher level tool that uses the existing "tileindex" tool I wrote specifically for the job to generate tile indexes for all layers | 04:09 |
| prologic | including normal layers | 04:09 |
| prologic | and delta layers | 04:09 |
| prologic | basically this allows you from a mapping point of view to take into account "time" | 04:10 |
| prologic | the tool was specifically written as an enhancement to gdal's identically named tileindex tool but with the support of an extra dimension "time". | 04:10 |
| prologic | it quite literally generates a Shapefile database mapping of geometry, data source and time | 04:13 |
| DanielBaird | yes it's time to push everything into issues, isn't it | 04:13 |
| prologic | so I have to figure out how this tool I wrote works again | 04:13 |
| prologic | and how to utilize it against our entire data set(s) | 04:13 |
| prologic | don't you hate forgetting how things work that you wrote :) | 04:13 |
| prologic | well we mostly know what we're doing now | 04:14 |
| prologic | since we have something to work with :) | 04:14 |
| prologic | and not "guessing" :) | 04:14 |
| DanielBaird | yeah | 04:14 |
| prologic | i.e: things are coming out of "realizations" | 04:14 |
| prologic | so I'll document this issue as well now | 04:14 |
| DanielBaird | and we'd like to wrap it up.. so having a defined list of stuff means we can plan to get it completed | 04:14 |
| prologic | and lastly | 04:17 |
| prologic | we need to discuss, plan and develop our "WMS"/"WMS-T" service for you | 04:18 |
| prologic | so that you can actually request layers and delta layers | 04:18 |
| prologic | with some flexibility on your end | 04:18 |
| prologic | i.e: my plan was to use map script and python | 04:18 |
| prologic | and provide you the enough flexibility to request any layer from any scneario+model+year | 04:19 |
| prologic | and tell me if you want normal or delta | 04:19 |
| prologic | etc etc | 04:19 |
| prologic | would you like to write up said issue? | 04:19 |
| prologic | since it's more about how you want it done | 04:20 |
| prologic | and less about me | 04:20 |
| prologic | I'll simply be coding up what you think would be sufficient | 04:20 |
| DanielBaird | yes okay.. i'll poke in a placeholder and fill it out a bit later | 04:20 |
| prologic | without the need to generate or write many map files | 04:20 |
| prologic | this should be done dyncmially | 04:20 |
| prologic | *nods* | 04:20 |
| DanielBaird | I'm expecting jeremy to pop up sometime and drag me off to a meeting | 04:20 |
| DanielBaird | aaaany second now.. | 04:20 |
| prologic | haha | 04:21 |
| prologic | quite write up a placeholder :) | 04:21 |
| prologic | that's the only task left that I can thin of | 04:21 |
| prologic | that actually requires some level of discussion bweteen us | 04:21 |
| prologic | also I've not done anything with amp script before | 04:21 |
| prologic | but I assume it's fairly straight forward | 04:21 |
| DanielBaird | this is a thing that Robert can help with | 04:29 |
| DanielBaird | he's been talking about mapscript a lot over the last week or so | 04:29 |
| prologic | *nods* | 04:51 |
| prologic | his input would be very valulable | 04:51 |
| prologic | basically for that task | 04:51 |
| prologic | I want to avoid generating map files at all | 04:52 |
| prologic | and make the whole process of serving up map tiles dynamic | 04:52 |
| prologic | with an optional tile cache in front of it | 04:52 |
| *** DanielBaird has quit IRC | 06:31 | |
| *** DanielBaird has joined #cmvt | 06:48 | |
| DanielBaird | okay bailing.. was sucked into meetings all afternoon. see you | 07:05 |
| *** DanielBaird has quit IRC | 08:08 | |
| *** DanielBaird has joined #cmvt | 08:38 | |
| *** DanielBaird has quit IRC | 08:47 | |
| *** DanielBaird has joined #cmvt | 09:13 | |
| *** DanielBaird has quit IRC | 09:18 | |
| *** DanielBaird has joined #cmvt | 10:14 | |
| *** DanielBaird has quit IRC | 10:18 | |
| *** DanielBaird has joined #cmvt | 11:42 | |
| *** DanielBaird has quit IRC | 11:47 | |
| *** DanielBaird has joined #cmvt | 12:42 | |
| *** DanielBaird has quit IRC | 12:47 | |
| *** DanielBaird has joined #cmvt | 13:43 | |
| *** DanielBaird has quit IRC | 13:48 | |
| *** DanielBaird has joined #cmvt | 13:58 | |
| *** DanielBaird has quit IRC | 14:24 | |
| *** DanielBaird has joined #cmvt | 14:55 | |
| *** DanielBaird has quit IRC | 15:04 | |
| *** DanielBaird has joined #cmvt | 15:30 | |
| *** DanielBaird has quit IRC | 15:36 | |
| *** DanielBaird has joined #cmvt | 16:32 | |
| *** DanielBaird has quit IRC | 16:36 | |
| *** DanielBaird has joined #cmvt | 17:38 | |
| *** DanielBaird has quit IRC | 17:40 | |
| *** DanielBaird has joined #cmvt | 18:41 | |
| *** DanielBaird has quit IRC | 18:45 | |
| *** DanielBaird has joined #cmvt | 19:41 | |
| *** DanielBaird has quit IRC | 19:45 | |
| *** DanielBaird has joined #cmvt | 20:42 | |
| *** DanielBaird has quit IRC | 20:46 | |
| *** DanielBaird has joined #cmvt | 21:42 | |
| *** DanielBaird has quit IRC | 21:47 | |
| *** DanielBaird has joined #cmvt | 22:40 | |
| prologic | DanielBaird: Good Morning | 23:25 |
| DanielBaird | hi | 23:25 |
| *** robert_pyke has joined #cmvt | 23:36 | |
| prologic | morning rob | 23:38 |
| robert_pyke | G'day | 23:38 |
Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!