IRC Logs for #circuits Thursday, 2014-01-30

*** Myke_ has joined #circuits08:39
*** Ossoleil has joined #circuits08:44
*** Ossoleil has quit IRC08:51
*** Ossoleil has joined #circuits09:14
*** Ossoleil has quit IRC10:26
*** Ossoleil has joined #circuits10:39
*** kephu has joined #circuits10:45
kephuI'm investigating libs for setting up a rpc client-server, and came across circuits. One question about it though, that I wasn't able to find in the docs: how do I ensure the connection between the two is secure?10:46
kephulike, SSL/TLS between them, or some other manner of secure communication10:47
*** Ossoleil has quit IRC11:05
*** Ossoleil has joined #circuits11:07
kephuso, um... anyone?11:07
*** Ossoleil has quit IRC11:08
prologichi kephu11:49
prologicsorry was away watching tv, dinner, family etc ;)11:49
prologicre ssl11:50
prologicour socket components all take an optional kwarg11:50
prologicby default False11:50
prologicjust wraps the standard python library ssl socket11:50
prologicare you using the dev branch?11:50
prologic1.2.1 on pypi is vastly out of date11:50
prologicI/we were hoping to release 3.0 early this year and start more regular release cycles but that hasn't happened yet - work, family, commitments, etc11:51
kephuprologic: I think I got 2.something12:01
prologicI meant to say 2.1.012:01
prologicyou need/should use 3.0.dev12:01
kephuyep, 2.1.012:01
kephuis it stable enough though?12:01
prologicall but the websockets test12:02
prologicwe do CI on every commit12:02
prologicwe try not to ever break anything12:02
prologicif we do, it's only for a few revisinos or so12:02
kephuisn't websockets sort of what I need here?12:02
kephusorry if its a noob question, I'm sorta new to the whole rpc thing12:02
kephualso, do you have a quick example of how to go about writing a simple ssl-based client-server?12:04
prologicwhen you said rpc12:04
prologicI assumed you meant JSON-RPC12:04
prologicor XML-RPC12:04
kephuwell, yes12:04
prologicyou are of course welcome to fix the websocekts client12:04
prologicI believe it's our client component that's broken12:04
prologicnot the server12:04
kephuI'm thinking about json, personally12:04
prologictest passes locally on all our develop environments12:04
prologicbut no on CI ( and shiningpanda)12:05
prologicthere should be one or two ssl examples in examples/web/12:05
kephu(btw, you mentioned python standard lib for ssl... looks like 2.7 doesn't have it?)12:05
prologicno it does12:05
prologic-but- you obviously must have the right python support in place12:06
prologicdistro dependent12:06
prologicif you were compiling python from source (which I do)12:06
prologicyou'd have to have openssl-dev12:06
prologicor openssl-devel12:06
prologicfor example12:06
prologicI use CRUX ( almost everywehre12:06
kephustraight from in my case12:06
kephuwhat with me slavin' away under the heavy burden of my corporate overlords12:07
kephu(i kid, of course, the job is pretty sweet)12:07
prologicif you're using their binaries12:08
prologicit should have ssl12:08
prologicas long as your system does12:08
prologicjust pass ssl=True to the client/server12:09
prologicsee the examples/web/12:09
prologicalso welcome to the wonderful world of circuits12:09
prologicand components!12:09
prologichorizontally scable software12:09
kephuobtw, since I'm pretty sure this'll come up at some point from the higher-ups, how much of a pain is it to integrate circuits with some other language?12:10
prologicjust the same as anything else in Python12:11
prologicpick your interface of choice12:11
prologicXML-RPC, JSON-RPC12:11
prologica lower-level C/C++ extension library via Python extensions, ctypes or similar12:11
prologicetc etc12:11
prologicjust to be clear12:12
prologicwhat circuits provides and offers12:12
prologicis a way to horizontally scale software development12:12
prologicwith highly reuseable components12:12
kephuso if I'm sticking with json-rpc, all I gotta say is "install a json-rpc lib on your end, learn it, use it"?12:12
prologicwith clearly defined interfaces12:12
prologica powerful message bus12:12
prologicand a vast library of already built and supported components by the core development team12:12
kephuok so in turn let me elaborate a bit on what I need it for, then12:13
prologicpretty much12:13
prologicPython + circuits <-> JSON-RPC <-> Java + Java App12:13
prologicor whatever12:13
kephuI wrote an automation software, and I'd like it to be able to run in rpc mode, so that you can call it to run some tasks from an external machine12:13
kephue.g. chaining a bunch of automation instances to access some stuff in, say, a vpn12:14
prologicsounds resonable12:14
kephucool, for a second there I was worried I chose the wrong tool for the job :P12:16
prologicwell there are plenty of tools12:17
prologiccircuits is unique in it's architecture12:17
kephuok, about that example though: how can I verify on the server, that a request comes from a trusted source?12:17
prologicTwisted for example is an alternative12:18
prologicor Tornado12:18
prologicbut each have their pros/cons12:18
kephuyeah, it was a cursory look at the architecture that sold me on the idea behind circuits?12:18
prologicthat's sort of outside of the scope of circuits12:18
prologicreally a more general programming question12:18
kephu(why did i add a question mark)12:18
prologicusually I see key pairs exchanged12:18
prologicas a means of "trust"12:18
prologicand "authantication"12:18
kephuhm, will have to look into that then12:20
kephubut that ssl thing, it's a start12:20
*** Ossoleil has joined #circuits12:23
prologicyeah well ssl will give you at least a secure connection12:24
prologicor communictaions channel12:24
prologictrust is an issue of12:24
prologicor authentication12:25
*** Romster has joined #circuits12:25
*** Ossoleil has quit IRC12:29
kephuok, I think I have an idea here12:31
Romsteri'm full of ideas but programming them is my my problem :)12:33
prologickephu, cool :)12:33
prologicwe're always here to help12:33
prologicjust hang around :)12:33
Romsteroh you are alive pro12:33
prologicwell last I checked my heart was beating12:33
prologicand my lungs moving12:33
prologicso yes :)12:33
Romsteri got a basic database working now with sqlalchemy i'll work on that more and add database to my version-sort codebase. but then i need a hand on how i'd use circuits to do the search page.12:34
Romsteri am really rusty with python and slowly remembering bits of it.12:35
Romsterlooking over now.12:36
Romster>>> import circuits12:37
Romster>>> print circuits.__version__12:37
Romsterso it was that long ago i last messed with it.12:37
prologicit's 3.0.0.dev12:39
prologicTraceback (most recent call last):12:39
prologic  File "<string>", line 1, in <module>12:39
prologicAttributeError: 'module' object has no attribute '__version__'12:39
Romsterso 2.1.0 is way old even12:39
prologicTraceback (most recent call last):12:39
prologic  File "<string>", line 1, in <module>12:39
prologicAttributeError: 'module' object has no attribute 'version'12:39
prologicdoesn't work in XChat ;)12:39
Romstererr -_-12:40
Romsteryour still on xchat, go install hexchat i packaged it in opt/hexchat12:40
Romsteri've since migrated12:40
Romsterit's xchat but only a fork and bugs removed.12:40
Romsteryou will have to configure the channels etc again though.12:41
Romsterso that workon is one of your virtualenv's?12:41
prologicI highly recommend you work with virtualenv12:42
prologicand most so recommend virtualenvwrapper12:42
Romsterseriously though migrate to hexchat12:42
prologicpip install virtualenvwrapper12:42
prologicadd to your .bashrc12:43
prologicsource it12:43
prologicmkvirtualenv foo12:43
Romsterxchat is decaying no one is maintaining it upstream12:43
prologicworkon foo12:43
prologicI'm used to X-Chat :)12:43
prologicit has all my settings12:43
prologicand window positioning12:43
Romsterhexchat looks like xchat12:43
prologicmaybe later when I feel like breaking my desktop :)12:44
prologictrying to get docker working here12:44
kephuprologic: so, 2 questions about grabbing the freshest version: 1) do I just grab it from repo, and 2) what's the uninstall procedure here? Just wipe the dir from site-packages?12:44
Romstersome small differences but i've got used to those.12:45
Romsteri'll try but i'll most likely just install circuits system wide12:46
*** Ossoleil has joined #circuits12:47
Romsteryou can run hexchat and xchat at the same time. while you migrate.12:47
Romster < nothing about intergrating with a database engine.12:51
Romsterok forms are covered i can work off that but need a example how sqlalchemy fits in with circuits and forms. and how to safely parse user input.12:52
prologicnormally if you're using pip12:56
prologicwhich you should12:56
prologicand you should be using virtualenv12:56
prologicpip uninstall circuits12:56
prologicand you can do:12:56
prologicpip install -e hg+
prologicor simply clone the repo with mercurial12:56
prologicand python install12:56
prologicor develop12:57
prologicas per normal12:57
Romsterprologic, you work with sqlite3 alot. is it necessary to use 2 tables and join them when i'll have a column in the table that will have may identical items. does sqlite3 compress this in the database or best i use two tables and do a join via id.12:57
prologicRomster> < nothing about intergrating with a database engine. <-- that's because it's just like anything in python12:57
Romsterguess i'll go the pip way and virtualenv as you said.12:57
prologicany way you'd use SA right now in testing12:57
prologicyou'd do so with circuits to12:57
prologicit imposes no restrictions on you12:57
Romsteri can probably manage with mako used it before and forms... just need a sql query example in circuits.12:58
Romsteri'm making a basic search page.12:58
prologiccheck out this12:58
prologicfor some nice sexy ajax/bootstrap forms12:58
prologicwith circuits.web12:58
prologicRomster, just follow the SA tutorials :)12:59
prologicand write SA + make code in your request handlers13:00
prologicre normalized databases13:00
prologicyeah if you have two tables of many items with referentials relatiopnahips13:00
prologicyou'll want to key them by some kind of unique id13:00
prologicand use join queries13:00
prologicand probably index some fields you want to search on13:00
Romsterif i do two tables. the question is if i use one table i could have 50 urls of the same site on unique version numbers.13:01
*** Ossoleil has quit IRC13:01
Romsterwould sqlite3 compress it's database or i'm best to store the url one time in a url table and id join.13:01
prologicdatabases are clever yes :)13:02
prologicit'll perform optimal queries based on your query13:02
prologicand indexes13:02
Romsterso duplicate strings can compress internally without me trying to reinvent the wheel?13:02
Romsterlike i insert 50 records with same url and filename but all different version strings13:03
Romsteror i use a url_id url table and use the url_id on them 50 records which would use less space.13:04
Romsterhmm idealy i could do the filename to a file_id as well and use 3 tables.13:04
Romsterunless the database has it's own internal deduplication code.13:05
Romster you seen that i'm gonna be databasing that stuff.13:05
Romsterbut from many urls.13:06
Romsterwhat generates that is in python already13:06
prologicif you have a unieuq filename and url pair13:06
prologicbut differnt versions13:06
prologicyou'll want a version table13:06
prologicand a file/url table13:06
prologicwhere you link the file/url table with the version13:06
Romsterthought as much13:06
Romsteryeah i kow about joins so i'm best to do a 3 way join on select for the results.13:07
Romstertable url url_id url ; table file file_id file ; table version version_id version13:08
Romsteroh acttually13:08
Romstertable version version_id url_id file_id version13:09
Romsterthen on the version table i can join to the url and filename13:09
Romsterto save massive duplication of text13:09
Romsterprologic, yeah this database will get huge.13:10
Romsteri'll basically use scrapy to index project distfiles of my choosing.13:11
kephuprologic: (sorry for the delay) pip is broken in my work env13:11
Romsterstore in database as version sorted. search portal on to list13:11
Romsterit's probably a bit adventitious of a project for myself.13:12
Romsterto top it off later i'll have a side div with lateist file versions of tracked distfiles of official project pages.13:12
Romsteri should actually write all these features to the index page.13:13
Romsteri might just end up storing the split string in the database and sort them on retrieval? as sql is not guaranteed to display them the same way as stored?13:18
Romsterunless i had some column to sort by, and by that point i may as well jsut do the complex sort on display. and possibly cache the results.13:19
Romsterfor that query13:19
Romsteri'll keep working on it.13:21
Romsterlots of kinks to sort out13:21
prologickephu, fix it :)13:22
prologicmake sure you use latest python13:23
prologiclatest setuptools 2.x something13:23
prologicand latest pip13:23
prologicthe python packaging ecosystem went through some rough packages there for a while13:23
prologicso best to use the latest stuff now that's it's slowly going back to normal13:23
prologicRomster, you think too much :)13:24
prologicstart small13:24
prologicwork your way up :)13:24
kephuprologic: no, i mean the url pip expects is unreachable, and apparently unproxy-able via cmdline13:24
prologicoh geez13:24
prologicdo you work for corparate or govenment?13:24
prologicwith a locked down network?13:24
Romsterlol i'm trying to think small database these results then make a search page to do a simple select sql query then i'll work on it to improve it.13:25
Romsteri'll get that stage done before i do anything else.13:25
*** Ossoleil has joined #circuits13:26
Romsteri sadly think for about 20 computers a day at work all running windows all with different issues to fix -_- and i get home wanting todo more computer stuff in linux.13:26
prologicgood :)13:26
Romsteri'm finding it hard.13:26
Romsterlits tupples constructs i got all these buzz words i need to associate them with the correct stuff.13:27
Romsterok last one is probably more C than python13:28
Romsterremeber 3 types and i can only recall lists tuples oh now i remember dicts13:28
Romsteri guess i know enough just piecing it all together.13:29
Romsteri struggle at.13:29
prologicPython is pretty simple really13:30
Romsteryou know all that ugly bash i had for sorting files well i got a friend to help with the rewrite to python13:31
Romsterso i decided to continue in python.13:31
prologicit only has 31 keywords to remember13:31
prologic>>> import keyword13:31
prologic>>> dir(keyword)13:31
prologic['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'iskeyword', 'kwlist', 'main']13:31
prologic>>> keyword.kwlist13:31
prologic['and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'exec', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'not', 'or', 'pass', 'print', 'raise', 'return', 'try', 'while', 'with', 'yield']13:31
prologic>>> len(keyword.kwlist)13:31
Romsterok not so bad and not having to worry about not using depreciated buggy glibc functions as in C13:33
Romsterremember i did write some small parts of python many years ago in i think it was kdb at the time or irc services i forget now.13:34
Romsteri've forgotten all that.13:34
Romsteri do understand objects better now than then.13:35
Romstermethods i'm still grasping.13:35
*** ircnotifier has quit IRC13:35
kephumethods (or functions rather) are mad powerful in python13:36
Romsterhonesty wish i had a brain like pros but i have this hard to learn one. but i think i'm starting to get the hang of it.13:36
*** ircnotifier has joined #circuits13:36
kephualso in any functional-ish language i suppose13:36
kephui mean not that python is one, but some borrowed concepts are pretty awesome13:37
prologicsorry had to reboot there13:37
prologicmissed a bit13:37
kephuwoah i only just realized13:37
kephuwhat a twist13:37
Romsterdead link13:38
kephucouldn't bold it in the shoddy web-based irc client13:39
kephuso i asterisk'd it up13:39
Romsteryou didn't miss much prologic13:39
kephubtw is there any way to grab that fresh version of circuits without having to install hg?13:39
prologicdid you use the wrong url?13:39
prologicyes of course13:39
prologicgo fetch the tip.tar.gz13:39
prologicor tip.zip13:39
prologicyou'll find it on the web page13:39
kephusoo, branch default i presume?13:40
kephuwait, no, branch default is 2.1.0 apparently13:42
kephuah, wait, no, i suck at this13:42
kephuthink i got it now13:42
kephutip's __version__ is 2.1.0, too13:43
kephuokay apparently i do suck at this13:44
kephua little help?13:44
Romsteri press branches and it's not listing any or perhaps i should be looking for tags13:45
kephuyeah, tip is in tags13:45
kephubut after downloading & unzipping it, I took a look at circuits/, and it indicates __version__ = "2.1.0"13:46
Romsteryeah so dev is in tip13:46
Romsteryou downloaded stable not dev13:47
Romsterah i cna't find crap sorry i'm no help here13:48
Romsterhg clone it i guess13:48
Romsterpip install -e hg+
prologickephu, circuits-dev13:49
Romsteri went back though my log13:49
Romsterno god damn wonder why is it in a new hg repo than a branch in the existing one...13:49
prologicwe maintain two separate repos13:49
prologicwe'll probably get rid of this once we release 3.0 proper13:50
prologicit seems to confuse :)13:50
Romsterit got me13:50
prologicbrb - rebooting13:50
*** ircnotifier has quit IRC13:50
kephuok, installed13:51
*** ircnotifier has joined #circuits13:51
kephuso, just to make sure, if I want json-rpc, I should look into circuits.web, right?13:52
prologicthere are two examples in examples/web/13:55
prologicand jsonrpc13:55
kephunot entirely sure how to send to it, but I'll likely get there13:57
Romsteri'm off to game a bit then sleep. ttyl13:59
*** Romster has quit IRC14:00
prologickephu, just with regular rpc14:03
prologicxmlrpc client14:03
prologicor jsonrpc client14:03
*** An_Ony_Moose has quit IRC14:22
*** An_Ony_Moose has joined #circuits14:24
*** kephu has quit IRC14:26
*** Ossoleil has quit IRC17:42
*** anunnaki has quit IRC23:17
*** anunnaki has joined #circuits23:17

Generated by 2.11.0 by Marius Gedminas - find it at!