| prologic | pdurbin: robert_ I’ve found a few things that may be useful with my problem above: | 00:25 |
|---|---|---|
| prologic | https://pypi.python.org/pypi/objectpath/0.2 | 00:25 |
| prologic | http://www.jsoniq.org/ | 00:25 |
| prologic | https://pypi.python.org/pypi/jsonpath-rw/1.3.0 | 00:25 |
| prologic | the first and last look really good | 00:25 |
| prologic | and might just do what I need | 00:26 |
| robert_ | well, COM is a component framework | 00:27 |
| prologic | no it’s not :) | 00:27 |
| prologic | it’s a binary protocol :P | 00:27 |
| prologic | maybe it is if you mean the C#/VB/.NET implemtnations of COM :) | 00:28 |
| prologic | and all the other stuff they’ve done: DCOM, COM++, ActiveX, etc, etc | 00:28 |
| robert_ | lol, COM actually has its own allocator. | 00:29 |
| robert_ | see: PM | 00:31 |
| prologic | WoW | 00:33 |
| prologic | just Wow! | 00:33 |
| prologic | :( | 00:33 |
| prologic | this is too low level to be useful as a reuseable component in any sense of the word | 00:33 |
| robert_ | yeah, see? | 00:34 |
| robert_ | it IS a framework | 00:34 |
| robert_ | they built the entire thing in C++ | 00:34 |
| prologic | yes :) | 00:43 |
| prologic | if you define a framework as a set of rules, policies and procedures in any given langauge | 00:43 |
| prologic | sure absolutely :) | 00:43 |
| prologic | but a Component Architecture? | 00:43 |
| prologic | yeah right | 00:44 |
| robert_ | lol | 00:45 |
| robert_ | it's a C++ collection of code that enables the definition and consumption of "software components" | 00:45 |
| prologic | Okay | 00:46 |
| prologic | I’m down toa choice of ObjectPath | 00:46 |
| prologic | or jsonpath_rw | 00:46 |
| prologic | sure :) | 00:47 |
| prologic | call em whateer you want | 00:47 |
| prologic | if you look around the web | 00:47 |
| prologic | I think there is no clear definition for “component model” or “component architecture" | 00:47 |
| prologic | I think Micirsoft rather successfully bastadized that :) | 00:47 |
| prologic | in my mind it’s always been about composition of units of behavior | 00:48 |
| prologic | building more complex components from simpper ones | 00:48 |
| prologic | composing themtoegether in a cooperative message bus if you will | 00:48 |
| prologic | it should be possible for circuits (one day) and any similar component architecture to have a visual/grpahical ide whereby you can plug ’n play with components of various complexity and composition | 00:49 |
| robert_ | well, not the COM ABI | 00:52 |
| robert_ | but the component framework part | 00:52 |
| robert_ | that's what circuits does | 00:53 |
| prologic | *nods* | 00:53 |
| robert_ | I was just equate component frameworks with COM, lol | 00:54 |
| robert_ | frame of reference, I guess. | 00:55 |
| pdurbin | prologic: so it's between those two | 00:55 |
| prologic | pdurbin: any experience/opioni? | 00:56 |
| prologic | how have you solved this (if at all)? | 00:56 |
| prologic | besides data in sql databases :) | 00:56 |
| pdurbin | I've used jq on the client side. that's about it :) | 00:56 |
| prologic | and exposing things like where clasues via restful web apis | 00:56 |
| prologic | jq the cli tool you linked me before? | 00:56 |
| pdurbin | yeah. "sed for json" | 00:57 |
| prologic | and on the client side you say? :) | 00:57 |
| prologic | how do you run cli tools written in C client-side? :P | 00:57 |
| pdurbin | curl | jq '.' | 00:58 |
| robert_ | I've already got an XML data format that I'm using | 00:58 |
| prologic | or client in this case is a shell? :) | 00:58 |
| prologic | ahh | 00:58 |
| prologic | so client != web browser client : | 00:58 |
| prologic | gotcha | 00:58 |
| pdurbin | curl is my web browser ;) | 00:58 |
| prologic | haha | 00:58 |
| prologic | you sound a lot like me :) | 00:58 |
| prologic | I might consider this tool to help directly solve my problem at hand | 00:59 |
| prologic | the problem is actually two fold | 00:59 |
| prologic | improving the query capabily of my apis | 00:59 |
| pdurbin | prologic: what if you throw all your data in mongodb and do fancy queries or whatever in there | 00:59 |
| prologic | and extracting some data out of them for our client — very specific data | 00:59 |
| prologic | can’t really do that | 01:00 |
| prologic | it’s over 500GB of raw data | 01:00 |
| prologic | and over 400M files | 01:00 |
| prologic | be too hard to throw it into MongoDB now :) | 01:00 |
| pdurbin | :) | 01:00 |
| prologic | it’s the data behind http://ccav.terranova.org.au/ | 01:00 |
| prologic | :) | 01:00 |
| prologic | throw me that jq link again? | 01:00 |
| pdurbin | big data | 01:00 |
| prologic | *nods* | 01:00 |
| prologic | big data indeed | 01:00 |
| prologic | and this isn’t even that big | 01:00 |
| pdurbin | http://stedolan.github.io/jq/ | 01:01 |
| prologic | we *were* going to do the same sort of data processing on 1km datasets | 01:01 |
| prologic | i.e: 30TB of raw data unprocessed (i.e: before my data processing touches it) | 01:01 |
| prologic | the one thst’s on ccav.terranova.org.au right now is based on 5km datasets | 01:01 |
| prologic | so if that yields 500GB of data and 400M files | 01:01 |
| prologic | what do you think 1km will be :) | 01:01 |
| prologic | 5x larger? 10x? more :) | 01:02 |
| prologic | I would need to consider a different file system perhaps | 01:02 |
| prologic | pdurbin: what do you do for work anyway? | 01:04 |
| prologic | have i asked before? :) | 01:04 |
| pdurbin | prologic: I work on this: https://github.com/IQSS/dataverse | 01:07 |
| prologic | So you work for Harvard? | 01:12 |
| pdurbin | yeah | 01:12 |
| prologic | Nice :) | 01:12 |
| prologic | I’ll take a look at dataverse a bit more closely later | 01:12 |
| prologic | looks interesting :) | 01:12 |
| prologic | I guess we work (incidently) in siilar fields | 01:13 |
| pdurbin | prologic: some day we'd like to have `pip install dataverse` ... a python client for our APIs: https://github.com/IQSS/dvn-client-python | 01:13 |
| prologic | Ahh :) | 01:14 |
| prologic | Do you have many developers working on this? | 01:14 |
| prologic | 20 commits - no :) | 01:14 |
| prologic | You guys could awlays hire me :) | 01:15 |
| pdurbin | that python thing? some collaborators are helping | 01:15 |
| prologic | I’m looking for new work :) | 01:15 |
| prologic | hrmm | 01:21 |
| prologic | https://pypi.python.org/pypi/jsonpath-rw/1.3.0 | 01:21 |
| prologic | this does not provide querying | 01:21 |
| prologic | just path traversal | 01:21 |
| prologic | doesn’t provide much more than what I already have with jsonselect | 01:21 |
| prologic | I’m starting to think there isn’t much in the way of querying/filtering json data with Python | 01:28 |
| prologic | other than roll your own | 01:28 |
| prologic | who wants that :) | 01:28 |
| pdurbin | prologic: "jq can be built in library form now" -- https://github.com/stedolan/jq/issues/12#issuecomment-17457566 | 01:33 |
| pdurbin | http://www.drmaciver.com/2013/12/pyjq-a-work-in-progress/ | 01:33 |
| prologic | oh? | 01:35 |
| prologic | hrmm :) | 01:35 |
| prologic | that puts it as a contender to ObjectPath | 01:35 |
| prologic | jsonpath_rq is useless ihmo | 01:35 |
| prologic | I mean it’s nice and all, but offers not much more than what I already have | 01:35 |
| prologic | http://www.logilab.org/project/rql | 01:36 |
| prologic | man logiclab geez | 01:37 |
| prologic | get on wht it seriously | 01:37 |
| prologic | any project tehy’ve ever develoepd is | 01:37 |
| prologic | hard to understand, almost zero documetnation | 01:37 |
| prologic | and you just turn away | 01:37 |
| prologic | pdurbin: so… if I’m reading that Issue | 01:40 |
| prologic | and the linked one | 01:40 |
| prologic | correclty | 01:40 |
| prologic | then jq can be “built” as a library now | 01:40 |
| pdurbin | yeah | 01:40 |
| prologic | and presumably one could write CPython extnesions to it | 01:40 |
| pdurbin | yeah | 01:40 |
| prologic | or use cffi for example | 01:40 |
| pdurbin | buh | 01:40 |
| prologic | or ctypes | 01:40 |
| prologic | but I have to assume this is not available to me “right now” :) | 01:40 |
| pdurbin | prologic: did you look at https://github.com/mwilliamson/jq.py ? | 01:41 |
| pdurbin | (different guy) | 01:41 |
| prologic | hmm no lemme looksie :) | 01:43 |
| pdurbin | huh, and yet another: https://github.com/doloopwhile/pyjq | 01:44 |
| prologic | hehe thanks :) | 01:45 |
| prologic | see it’s not what you know | 01:45 |
| prologic | it’s who you know :) | 01:45 |
| prologic | oooooer | 01:47 |
| pdurbin | prologic: oh, since you do maps and python, you might be interested in this: http://worldmap.harvard.edu https://github.com/cga-harvard/cga-worldmap | 01:47 |
| prologic | A Python binding for ./jq | 01:47 |
| prologic | fuck yeah :) | 01:47 |
| prologic | sorry :P | 01:47 |
| prologic | someone’s already done the hard work | 01:47 |
| pdurbin | heh | 01:47 |
| prologic | even better | 01:47 |
| pdurbin | right | 01:47 |
| prologic | :) | 01:48 |
| prologic | yeah strangely I’ve been doing GIS for the last 2 years now | 01:48 |
| prologic | with big data stuff | 01:48 |
| prologic | who knew | 01:48 |
| pdurbin | we're starting to integrate worldmap with dataverse | 01:48 |
| pdurbin | the idea is that if you upload spatial data, you should see a "map it" button | 01:49 |
| pdurbin | http://datascience.iq.harvard.edu/blog/bari-project-integrating-dataverse-worldmap | 01:50 |
| prologic | ahh nice | 01:52 |
| pdurbin | prologic: shapefile or geojson? which is better? | 01:53 |
| prologic | depends | 01:55 |
| prologic | Shapefile(s) have builtin indexes | 01:55 |
| prologic | GeoJSON is just a nice serialization format | 01:55 |
| prologic | of the Geomery and it’s Properties | 01:55 |
| prologic | You’d have to do indexing seaprately if you use GeoJSON | 01:55 |
| prologic | I opted for Shapefile(s) myself in this project | 01:56 |
| prologic | no benefit for postgis or similar GIS-enabeld rdbms | 01:56 |
| prologic | as they all use (at least the OSS ones) underneath the GEOS/GDAL libraries anyway | 01:56 |
| prologic | including spatialite | 01:56 |
| pdurbin | no benefit? postgis doesn't benefit from shapefiles? | 01:57 |
| prologic | what I mean to say is... | 02:03 |
| prologic | if you’re serving up raster images | 02:03 |
| prologic | think GeoTiff | 02:03 |
| prologic | or Vectors | 02:03 |
| prologic | think Shapefile(s) / GeoJSON | 02:04 |
| prologic | PostGIS buys you nothing in terms of performance or functionality | 02:04 |
| prologic | unless you want to mix data with other kinds of data | 02:04 |
| prologic | e.g: non-GIS data | 02:04 |
| prologic | I’ve compared the performance of SpatialLite/PostGIS and plain ol Python + Shapely + Fiona against the same datasets | 02:04 |
| prologic | PostGIS basically uses GEOS and GDAL anyway | 02:05 |
| prologic | as does for example SHapelf/Fiona (Python) | 02:05 |
| prologic | which also use GDAL/GEOS | 02:05 |
| pdurbin | hmm. ok. I'll chew on that while I sleep. thanks | 02:07 |
| prologic | nps :) | 02:08 |
| prologic | just my 2c worth of experience so far :P | 02:08 |
| prologic | pdurbin: http://codepad.org/f8CL1P1x | 03:34 |
| prologic | got the pyjq extension to compile on OS X at work ;) | 03:34 |
| robert_ | PC-GEOS <3 | 04:04 |
| *** koobs has quit IRC | 04:19 | |
| *** koobs has joined #circuits | 04:51 | |
| kdb_ | Welcome back koobs :) | 04:51 |
| *** koobs has quit IRC | 07:15 | |
| *** koobs has joined #circuits | 08:14 | |
| kdb_ | Welcome back koobs :) | 08:14 |
| prologic | even'n all | 08:19 |
| prologic | http://linuxgizmos.com/tiny-open-source-module-aims-to-make-iot-apps-easy/ | 08:43 |
| prologic | oooer | 08:43 |
| pdurbin | prologic: good job | 11:35 |
| prologic | oh :) | 12:06 |
| prologic | pdurbin, thanks :P | 12:06 |
| prologic | I try | 12:06 |
| prologic | he | 12:06 |
| prologic | heh | 12:06 |
| pdurbin | :) | 12:09 |
| *** An_Ony_M1ose has joined #circuits | 20:38 | |
| kdb_ | Hi an_ony_m1ose | 20:38 |
| *** ninkotech has quit IRC | 20:44 | |
| *** ninkotech has joined #circuits | 20:56 | |
| kdb_ | Welcome back ninkotech :) | 20:56 |
| prologic | morn’n all | 21:44 |
| prologic | pdurbin: so with jq, if I want to get a list of objects whoose .name matches a aubset of names? | 23:02 |
| robert_ | prologic: how goes it? | 23:09 |
| prologic | hey robert_ | 23:22 |
| prologic | hacking some crap together :) | 23:22 |
| prologic | you know the usual kind of write once throw away stuff | 23:22 |
| prologic | it’s awful :) | 23:22 |
| robert_ | yeah | 23:22 |
| robert_ | heh | 23:22 |
| robert_ | prologic: what would be your best advice for an admin console implemented over a UNIX socket? (something similar to AST's manager) | 23:24 |
| prologic | not familar with AST’s manager? | 23:32 |
| robert_ | http://1.bp.blogspot.com/-fzvIMYaJdVg/TqZmy27htLI/AAAAAAAAAck/ClAozBiBERI/s1600/Spee-Dial-CLI.jpg | 23:39 |
| prologic | oh right | 23:40 |
| prologic | so sorry what are you asking exactly? | 23:41 |
| robert_ | it's an administrative console you can connect to and use to debug problems at run-time with AST; how would you implement onesuch admin console with circuits? | 23:42 |
| robert_ | sort of like Debugger(), but "not really" | 23:42 |
| prologic | so you want an interactive console into the system? | 23:43 |
| robert_ | first, it'd always be available; second, it would log problems, and you could start/stop/etc. the application | 23:43 |
| robert_ | indeed | 23:43 |
| prologic | pretty easy really | 23:43 |
| prologic | just your normal code.interact() | 23:44 |
| prologic | I think | 23:44 |
| robert_ | it's going to be a daemon, so I don't want to have to restart it as an interactive app | 23:44 |
| prologic | look up python itneractive shell | 23:44 |
| prologic | and then wrap this up in a UNIX socket server | 23:44 |
| prologic | let’s call the component InteractiveDebugger() :) | 23:45 |
| prologic | once registered will also register a UNIXServer socket to a given path | 23:45 |
| prologic | and listen for events from connected sockets | 23:46 |
| prologic | and execute them interactively like a python shell | 23:46 |
| prologic | setup your locals to have the root (top-level) manager as “root" | 23:46 |
| robert_ | oh, basically eval and friends after authentication, etc. etc. | 23:46 |
| prologic | or app | 23:46 |
| robert_ | yeah | 23:46 |
| prologic | use the code module I think | 23:47 |
| robert_ | http://code.activestate.com/recipes/355319-using-codeinteractiveconsole-to-embed-a-python-she/ | 23:48 |
Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!
