IRC Logs for #circuits Tuesday, 2014-06-10

prologicpdurbin: robert_ I’ve found a few things that may be useful with my problem above:00:25
prologichttps://pypi.python.org/pypi/objectpath/0.200:25
prologichttp://www.jsoniq.org/00:25
prologichttps://pypi.python.org/pypi/jsonpath-rw/1.3.000:25
prologicthe first and last look really good00:25
prologicand might just do what I need00:26
robert_well, COM is a component framework00:27
prologicno it’s not :)00:27
prologicit’s a binary protocol :P00:27
prologicmaybe it is if you mean the C#/VB/.NET implemtnations of COM :)00:28
prologicand all the other stuff they’ve done: DCOM, COM++, ActiveX, etc, etc00:28
robert_lol, COM actually has its own allocator.00:29
robert_see: PM00:31
prologicWoW00:33
prologicjust Wow!00:33
prologic:(00:33
prologicthis is too low level to be useful as a reuseable component in any sense of the word00:33
robert_yeah, see?00:34
robert_it IS a framework00:34
robert_they built the entire thing in C++00:34
prologicyes :)00:43
prologicif you define a framework as a set of rules, policies and procedures in any given langauge00:43
prologicsure absolutely :)00:43
prologicbut a Component Architecture?00:43
prologicyeah right00:44
robert_lol00:45
robert_it's a C++ collection of code that enables the definition and consumption of "software components"00:45
prologicOkay00:46
prologicI’m down toa  choice of ObjectPath00:46
prologicor jsonpath_rw00:46
prologicsure :)00:47
prologiccall em whateer you want00:47
prologicif you look around the web00:47
prologicI think there is no clear definition for “component model” or “component architecture"00:47
prologicI think Micirsoft rather successfully bastadized that :)00:47
prologicin my mind it’s always been about composition of units of behavior00:48
prologicbuilding more complex components from simpper ones00:48
prologiccomposing themtoegether in a cooperative message bus if you will00:48
prologicit should be possible for circuits (one day) and any similar component architecture to have a visual/grpahical ide whereby you can plug ’n play with components of various complexity and composition00:49
robert_well, not the COM ABI00:52
robert_but the component framework part00:52
robert_that's what circuits does00:53
prologic*nods*00:53
robert_I was just equate component frameworks with COM, lol00:54
robert_frame of reference, I guess.00:55
pdurbinprologic: so it's between those two00:55
prologicpdurbin: any experience/opioni?00:56
prologichow have you solved this (if at all)?00:56
prologicbesides data in sql databases :)00:56
pdurbinI've used jq on the client side. that's about it :)00:56
prologicand exposing things like where clasues via restful web apis00:56
prologicjq the cli tool you linked me before?00:56
pdurbinyeah. "sed for json"00:57
prologicand on the client side you say? :)00:57
prologichow do you run cli tools written in C client-side? :P00:57
pdurbincurl | jq '.'00:58
robert_I've already got an XML data format that I'm using00:58
prologicor client in this case is a shell? :)00:58
prologicahh00:58
prologicso client != web browser client :00:58
prologicgotcha00:58
pdurbincurl is my web browser ;)00:58
prologichaha00:58
prologicyou sound a lot like me :)00:58
prologicI might consider this tool to help directly solve my problem at hand00:59
prologicthe problem is actually two fold00:59
prologicimproving the query capabily of my apis00:59
pdurbinprologic: what if you throw all your data in mongodb and do fancy queries or whatever in there00:59
prologicand extracting some data out of them for our client — very specific data00:59
prologiccan’t really do that01:00
prologicit’s over 500GB of raw data01:00
prologicand over 400M files01:00
prologicbe too hard to throw it into MongoDB now :)01:00
pdurbin:)01:00
prologicit’s the data behind http://ccav.terranova.org.au/01:00
prologic:)01:00
prologicthrow me that jq link again?01:00
pdurbinbig data01:00
prologic*nods*01:00
prologicbig data indeed01:00
prologicand this isn’t even that big01:00
pdurbinhttp://stedolan.github.io/jq/01:01
prologicwe *were* going to do the same sort of data processing on 1km datasets01:01
prologici.e: 30TB of raw data unprocessed (i.e: before my data processing touches it)01:01
prologicthe one thst’s on ccav.terranova.org.au right now is based on 5km datasets01:01
prologicso if that yields 500GB of data and 400M files01:01
prologicwhat do you think 1km will be :)01:01
prologic5x larger? 10x? more :)01:02
prologicI would need to consider a different file system perhaps01:02
prologicpdurbin: what do you do for work anyway?01:04
prologichave i asked before? :)01:04
pdurbinprologic: I work on this: https://github.com/IQSS/dataverse01:07
prologicSo you work for Harvard?01:12
pdurbinyeah01:12
prologicNice :)01:12
prologicI’ll take a look at dataverse a bit more closely later01:12
prologiclooks interesting :)01:12
prologicI guess we work (incidently) in siilar fields01:13
pdurbinprologic: some day we'd like to have `pip install dataverse` ... a python client for our APIs: https://github.com/IQSS/dvn-client-python01:13
prologicAhh :)01:14
prologicDo you have many developers working on this?01:14
prologic20 commits - no :)01:14
prologicYou guys could awlays hire me :)01:15
pdurbinthat python thing? some collaborators are helping01:15
prologicI’m looking for new work :)01:15
prologichrmm01:21
prologichttps://pypi.python.org/pypi/jsonpath-rw/1.3.001:21
prologicthis does not provide querying01:21
prologicjust path traversal01:21
prologicdoesn’t provide much more than what I already have with jsonselect01:21
prologicI’m starting to think there isn’t much in the way of querying/filtering json data with Python01:28
prologicother than roll your own01:28
prologicwho wants that :)01:28
pdurbinprologic: "jq can be built in library form now" -- https://github.com/stedolan/jq/issues/12#issuecomment-1745756601:33
pdurbinhttp://www.drmaciver.com/2013/12/pyjq-a-work-in-progress/01:33
prologicoh?01:35
prologichrmm :)01:35
prologicthat puts it as a contender to ObjectPath01:35
prologicjsonpath_rq is useless ihmo01:35
prologicI mean it’s nice and all, but offers not much more than what I already have01:35
prologichttp://www.logilab.org/project/rql01:36
prologicman logiclab geez01:37
prologicget on wht it seriously01:37
prologicany project tehy’ve ever develoepd is01:37
prologichard to understand, almost zero documetnation01:37
prologicand you just turn away01:37
prologicpdurbin: so… if I’m reading that Issue01:40
prologicand the linked one01:40
prologiccorreclty01:40
prologicthen jq can be “built” as a library now01:40
pdurbinyeah01:40
prologicand presumably one could write CPython extnesions to it01:40
pdurbinyeah01:40
prologicor use cffi for example01:40
pdurbinbuh01:40
prologicor ctypes01:40
prologicbut I have to assume this is not available to me “right now” :)01:40
pdurbinprologic: did you look at https://github.com/mwilliamson/jq.py ?01:41
pdurbin(different guy)01:41
prologichmm no lemme looksie :)01:43
pdurbinhuh, and yet another: https://github.com/doloopwhile/pyjq01:44
prologichehe thanks :)01:45
prologicsee it’s not what you know01:45
prologicit’s who you know :)01:45
prologicoooooer01:47
pdurbinprologic: oh, since you do maps and python, you might be interested in this: http://worldmap.harvard.edu https://github.com/cga-harvard/cga-worldmap01:47
prologicA Python binding for ./jq01:47
prologicfuck yeah :)01:47
prologicsorry :P01:47
prologicsomeone’s already done the hard work01:47
pdurbinheh01:47
prologiceven better01:47
pdurbinright01:47
prologic:)01:48
prologicyeah strangely I’ve been doing GIS for the last 2 years now01:48
prologicwith big data stuff01:48
prologicwho knew01:48
pdurbinwe're starting to integrate worldmap with dataverse01:48
pdurbinthe idea is that if you upload spatial data, you should see a "map it" button01:49
pdurbinhttp://datascience.iq.harvard.edu/blog/bari-project-integrating-dataverse-worldmap01:50
prologicahh nice01:52
pdurbinprologic: shapefile or geojson? which is better?01:53
prologicdepends01:55
prologicShapefile(s) have builtin indexes01:55
prologicGeoJSON is just a nice serialization format01:55
prologicof the Geomery and it’s Properties01:55
prologicYou’d have to do indexing seaprately if you use GeoJSON01:55
prologicI opted for Shapefile(s) myself in this project01:56
prologicno benefit for postgis or similar GIS-enabeld rdbms01:56
prologicas they all use (at least the OSS ones) underneath the GEOS/GDAL libraries anyway01:56
prologicincluding spatialite01:56
pdurbinno benefit? postgis doesn't benefit from shapefiles?01:57
prologicwhat I mean to say is...02:03
prologicif you’re serving up raster images02:03
prologicthink GeoTiff02:03
prologicor Vectors02:03
prologicthink Shapefile(s) / GeoJSON02:04
prologicPostGIS buys you nothing in terms of performance or functionality02:04
prologicunless you want to mix data with other kinds of data02:04
prologice.g: non-GIS data02:04
prologicI’ve compared the performance of SpatialLite/PostGIS and plain ol Python + Shapely + Fiona against the same datasets02:04
prologicPostGIS basically uses GEOS and GDAL anyway02:05
prologicas does for example SHapelf/Fiona (Python)02:05
prologicwhich also use GDAL/GEOS02:05
pdurbinhmm. ok. I'll chew on that while I sleep. thanks02:07
prologicnps :)02:08
prologicjust my 2c worth of experience so far :P02:08
prologicpdurbin: http://codepad.org/f8CL1P1x03:34
prologicgot the pyjq extension to compile on OS X at work ;)03:34
robert_PC-GEOS <304:04
*** koobs has quit IRC04:19
*** koobs has joined #circuits04:51
kdb_Welcome back koobs :)04:51
*** koobs has quit IRC07:15
*** koobs has joined #circuits08:14
kdb_Welcome back koobs :)08:14
prologiceven'n all08:19
prologichttp://linuxgizmos.com/tiny-open-source-module-aims-to-make-iot-apps-easy/08:43
prologicoooer08:43
pdurbinprologic: good job11:35
prologicoh :)12:06
prologicpdurbin, thanks :P12:06
prologicI try12:06
prologiche12:06
prologicheh12:06
pdurbin:)12:09
*** An_Ony_M1ose has joined #circuits20:38
kdb_Hi an_ony_m1ose20:38
*** ninkotech has quit IRC20:44
*** ninkotech has joined #circuits20:56
kdb_Welcome back ninkotech :)20:56
prologicmorn’n all21:44
prologicpdurbin: so with jq, if I want to get a list of objects whoose .name matches a aubset of names?23:02
robert_prologic: how goes it?23:09
prologichey robert_23:22
prologichacking some crap together :)23:22
prologicyou know the usual kind of write once throw away stuff23:22
prologicit’s awful :)23:22
robert_yeah23:22
robert_heh23:22
robert_prologic: what would be your best advice for an admin console implemented over a UNIX socket? (something similar to AST's manager)23:24
prologicnot familar with AST’s manager?23:32
robert_http://1.bp.blogspot.com/-fzvIMYaJdVg/TqZmy27htLI/AAAAAAAAAck/ClAozBiBERI/s1600/Spee-Dial-CLI.jpg23:39
prologicoh right23:40
prologicso sorry what are you asking exactly?23:41
robert_it's an administrative console you can connect to and use to debug problems at run-time with AST; how would you implement onesuch admin console with circuits?23:42
robert_sort of like Debugger(), but "not really"23:42
prologicso you want an interactive console into the system?23:43
robert_first, it'd always be available; second, it would log problems, and you could start/stop/etc. the application23:43
robert_indeed23:43
prologicpretty easy really23:43
prologicjust your normal code.interact()23:44
prologicI think23:44
robert_it's going to be a daemon, so I don't want to have to restart it as an interactive app23:44
prologiclook up python itneractive shell23:44
prologicand then wrap this up in a UNIX socket server23:44
prologiclet’s call the component InteractiveDebugger() :)23:45
prologiconce registered will also register a UNIXServer socket to a given path23:45
prologicand listen for events from connected sockets23:46
prologicand execute them interactively like a python shell23:46
prologicsetup your locals to have the root (top-level) manager as “root"23:46
robert_oh, basically eval and friends after authentication, etc. etc.23:46
prologicor app23:46
robert_yeah23:46
prologicuse the code module I think23:47
robert_http://code.activestate.com/recipes/355319-using-codeinteractiveconsole-to-embed-a-python-she/23:48

Generated by irclog2html.py 2.11.0 by Marius Gedminas - find it at mg.pov.lt!