IRC Logs for #circuits Saturday, 2013-04-13

prologicspaceone, so you've managed to build an entire parser and protocol handler in 100 lines?11:20
*** koobs has quit IRC13:35
*** koobs has joined #circuits13:35
ronnywhich protocol?13:55
*** Osso has joined #circuits15:22
spaceoneprologic: no16:09
spaceonedid you look at it?16:09
spaceoneronny: HTTP16:15
ronnyi see16:57
spaceoneronny: what do you see?17:04
ronnyspaceone, btw is the parser cowboy style or rfc conforming?17:04
spaceoneronny: it is full rfc conform17:06
ronnyhmm, then its neat17:06
ronnyhow do you verify its conform?17:06
spaceonebecause no implementation is rfc conform my mission was to write it17:06
spaceoneronny: i read the rfc 2 times17:06
spaceonei am testing on my own17:06
ronnythen you are likely wrong :)17:07
spaceonei need some slave to write tests17:07
ronnywhats the problem with http-parser?17:07
spaceoneit is bullshit17:07
spaceoneit is not rfc conform17:07
spaceoneand it is not object orientated17:07
ronnywhats wrong?17:07
spaceonealle regex's17:08
ronnyi see, so you mean the python parser hack it has17:08
ronnydid you check the c bits?17:08
spaceonethe c things are most likely the same, but i did not check17:08
ronnythe c bits are actually a sound statemachine17:09
spaceoneOo what do you mean?17:09
spaceonei did not understand the slang17:09
ronnydo you know what a statemachine is?17:09
ronnythen you have absolutely no business in writing parsers :)17:10
spaceonelol, i have17:10
ronnywhere is the code?17:10
spaceonemy code?17:10
spaceoneon github17:10
ronnyhmm, and what bit exactly is the rfc conforming parser ?17:14
spaceoneall the parse methods17:15
ronnytht code is so full of wtfs17:15
ronnyyou inherit a descriptor from bytes17:15
spaceonenot for people who understands OOP and python17:16
ronnyand all it does is call bytes17:16
ronnyi rest my case, those who dont understand statemachines shouldnt be writing parsers :)17:17
spaceonei only don't know the word17:17
spaceoneit is not translateable to german17:17
spaceonetell me exactly where the mankos are17:17
ronnythe german word ist zustandsmaschiene oder finiter automat17:18
spaceonei mean it is only Method which is inheriting from bytes (and protocol form tuple) which does not do anything more than existing17:18
ronnydo you have an actual example of how this thing is supposed to parse a http message off a stream17:18
spaceonewait a minute17:19
ronnybtw, why the hell is Protocol a tuple and a descriptor?17:24
spaceonei am open for feedback17:24
ronnyits like you have absolutely zero idea of separation of concerns17:25
spaceoneyou can set request.protocol = (1,1) then17:25
spaceoneyou don't have to request.protocol = Protocol((1,1))17:26
ronnyno, thats not what you mean17:26
ronnyyou did put assignment handling of a instance of the type inside of the type17:27
ronnyso now your Protocol is also Responsible for the assignment stuff17:27
spaceonejeah, show me a better solution... using @property ?17:27
spaceoneit is way more code and does the same17:28
spaceonein revision 1 it is done with @property... the idea of the data descriptors come from circuits.web17:29
spaceoneand i did not want to create 2 classes for every thing, so i put it into 117:29
ronnythe fact that those properties are in fact single use things tells me WTF ARE YOU DOING DUDE17:31
ronnydid you jsut finish reading some kind of book and want to hammer everything that looks like a nail17:31
spaceoneronny: what other things do you have? (i mean i can remove all descriptors and replace them through @property)17:34
ronnyand whats even worse, all descriptors follow the same type wrapping patterns17:34
ronnyso instead of undoing code duplication you just moved it17:34
spaceonethey are not all doing the same17:35
ronnyeach one follows the pattern of wrap somthing in some kind of type if its not of that type17:35
spaceonethe most, yes17:36
spaceonei want a reliability17:36
ronnyprologic, btw, do you have a protocol that feeds lines till an empty line arrives, then switches to raw data?17:38
ronnyspaceone, either you suddenly get a very important idea now or there is no point to help you17:38
ronnyalso no chunked17:39
spaceoneronny: there is no request chunked17:40
spaceoneand response chunked is on todo17:40
ronnyspaceone, btw, did you get the idea i was talking about?17:42
ronnyspaceone, just fyi - im strongly under the impression you never had any solid computer-science education17:45
spaceoneronny: of course i understand your 'i am doing assignments on the wrong place' idea, i understand the 'waiting for a \n thing'17:46
spaceoneronny: no i did not17:46
spaceoneronny: i am currently software engineering apprentice17:46
ronnyi see, you have much to learn17:47
spaceoneronny: i want you to not only criticise me but helping me understanding software engineering, telling me exactly whats wrong and why17:47
ronnyright now im under the imporession that task takes a a few hours of discussion at least17:48
spaceonewhat of my ideas are wrong, and where is it done better (i mean on circuits.web is the most very similar)17:48
ronnywhere in germany are you located?17:49
spaceoneronny: bremen, why?17:49
spaceoneronny: can you give me some clear points?17:49
ronnyspaceone, atm im not sure where to start, much is gut feeling and its hard to just propperly express since i dont clearly get where you are coming from and what background is missing17:50
spaceoneronny: i mean do you think there are some great ideas in the project?17:50
spaceoneronny: do you think it is ok that each class does it's parsing job? instead of one method which parses everything?17:51
ronnyspaceone, the layering is completely strange17:52
ronnyits pretty much looking like the throwaway messes i did years ago before studying17:52
ronnyspaceone, the basic problem is that you didnt really write a http parser, you wrote something that pulls apart a number of lines that look like a http message header17:56
ronnyand you yank it right into some kind of high level model17:57
ronnyand have zero separation wrt protocol vs repressentation17:57
spaceonebut isn't that what you want?17:57
spaceoneto combine parsing and model17:58
spaceoneas classmethods17:58
ronnyits a total mess if you later it wrong17:58
spaceoneto do things where they belong to17:58
ronnyi mean a protocol implementation is something you stream data in and it does actions17:58
ronnywith yours, you cant even stream data in17:58
ronnyyou have to do half the protocol at a different level17:58
ronnyand then push part of it into a high level model17:59
spaceoneyes, you must wait until the first \r\n is available17:59
spaceonethen you must wait until \r\n\r\n is available17:59
spaceoneand then until the body is complete17:59
ronnybut why?17:59
ronnya propper protocol is just feed the data and operates on it17:59
ronnyyour thin can do none of that18:00
ronnyits just something that pulls lines appart18:00
spaceoneit would be correct to parse each byte?18:00
ronnyin the strict theoretical sense yes, but you can get by fine with higher level actions and statemachines18:00
ronnythere is a bit more pragmatic way given that you can receive lines till you get a empty one18:01
spaceonewhat aspect are in there?18:01
ronnyand the first of the line will be the status line, and all the other lines are headers or header continuations18:02
ronnyoh, btw, did you implement header continuations?18:02
spaceoneof course i did18:02
ronnybasically parsing a http message till the end of header is the following statemachine18:03
spaceoneronny: do you understand the (maybe wrong) ideas i had?18:03
ronnystatus -> header; header -> header ; header -> header_cont; header_cont -> header: header_cont -> header_cont: header -> end;18:03
ronnythis is a directed graph with a few loops18:04
ronnyoh and i forgot header_cont -> end18:04
spaceoneronny: yes, that would be easy to adapt18:04
spaceoneit is just moving some lines of code into the _on_read18:04
spaceoneso i don't have to wait until a \r\n\r\n is available18:05
ronnythe thing is, that is the main and most importand machinery of a http parser18:05
ronnythe rest is just more pretty data18:05
ronnymost of the things you parsed and transformed, are expected to be byte strings or "wsgi" strings by most frameworks i work with18:06
ronnythe more important part is chunkoung out the path and method, and geting the header strings18:07
spaceoneso my goal now is to support streaming for each header line? remove descriptors?18:07
ronnyi not sure how to set a good goal for that lib18:07
spaceoneronny: there are only bytes18:07
spaceonehttp is bytes18:07
ronnyspaceone, fun fact about wsgi - they messed up shit big time, in python3 "wsgi" strings are bytest raped to be unicode by decode with latin118:08
spaceoneronny: there is no line which has anything to do with wsgi18:08
ronnyyou may smile insane now :)18:08
ronnyspaceone, wsgi is one of the things you feed http in more often than not18:09
ronny(at least on python)18:09
ronnyalltho i suppose its less of an issue with circuits18:09
spaceonei did not fully understand the wsgi thing18:10
spaceoneyou are putting http things into a special format18:10
spaceonethen you again have to parse it in you application18:10
ronnyspaceone, it follows the cgi environment18:10
spaceoneyes, and who needs this?18:10
ronnyhttp headers would be transfered n whats basically a mapping18:10
spaceonehttp speaks about ressources18:11
spaceonenot about calling cgi scritps18:11
ronnywell, im not saying its a good idea, its just that it is the standard18:11
spaceoneokay, but http is also the standard18:11
spaceoneand the standard i want to reach18:11
spaceonei really can not see an advantage of wsgi, except that everyone is able to support it18:13
ronnythats correct ^^18:13
ronnyas for parsing http18:13
ronnystore stuff as bytes, parse more detailed when needed18:14
spaceonealso don't you think that the wsgi things must be done after parsing the http message?18:14
ronnybrb, called into real life18:14
spaceoneronny: i will be back in 30 mins18:14
spaceonemaybe 20~~18:15
ronnyspaceone, conforming wsgi happens after parsing headers and before reading the body18:37
spaceoneronny: why before reading body?18:42
ronnybecause the body can be gigabytes18:42
ronnyand the body is streamed18:43
spaceonehmm, is a socket able to not read the body?18:43
spaceone/ read it when it is needed18:43
spaceoneother the body is in the mem18:44
spaceoneronny: there is an important thing you said: 'store stuff as bytes, parse more detailed when needed'18:46
spaceonein my application i am sure that i need everything18:46
ronnywhat does the application do?18:46
spaceonei need every header, i need the method, i need the uri18:46
spaceoneif there is a body i also need the body18:47
spaceoneronny: depends on the resource, but basically it is doing everything18:47
spaceonemy application is a application which conforms the http MUSTs and which wants to rule the REST constraints18:48
ronnythats not an answer at all :)18:48
spaceoneronny: what do you want to hear?18:49
ronnywell you basically just say "it does kinda everything and has to follow the HTTTP MUSTs and REST"18:49
ronnyi asked what does the app do18:49
ronnyanswers look like "address book" "media streaming"18:50
spaceonethe app mapps Host header and URI to a domain, maps uri path to a resource, maps method to resource method, parses input body by Content-Type header, checks if-none-match, if-unmodified-since headers, checks expectations form expect header18:52
spaceonegives parsed querystring options to the method18:52
spaceoneif the method requires these18:52
spaceone(de)compresses input/output18:53
spaceoneauthenticates credentials18:53
ronnythat is technical details needed to implemente a app18:54
ronnybut what is the actual purpose of that app18:54
ronnybecause you just described "every webframework there is"18:54
spaceones/app/my framework18:54
spaceoneit just wants to give a nice and readable api for programmers who want to conform http and design their architecture in a REST style18:55
ronnyso you want to make a framework, and dont even have a few own apps you want to implement with it18:55
ronnythen you are doing it wrong :)18:55
spaceonei have18:56
ronnythen why dont you talk about those?18:56
spaceonebecuase they are not my goal18:56
spaceonei have a CMS18:56
spaceonein my old framework i had chat, forum (community), helpdesk(kind of ticket system), navigation, accounts and profiles, shoutbox, cms, pollings18:58
spaceonenews system18:58
spaceonepersonal messages (PM)18:58
ronnynow we are finally talking sensible use-cases18:58
spaceoneadmin panel18:59
spaceoneeverything of that should follow the same strucutrre and design18:59
spaceonei want to prevent code duplication18:59
spaceonei want to give a nice API18:59
ronnywell, you need message queues and database interaction18:59
spaceonei want to combine human web with programmable web18:59
ronnythat tends to be more than you can bite if you have no complete computer science education :)19:00
ronnyin particular since the app is rather large19:00
spaceonei don't think so19:00
ronnyto finish something like that will easily end in manyears19:01
spaceoneso the measurement(?) is time19:01
ronnyon of the measurements19:02
ronnytime is by no means equal19:02
ronnythe more experience the implementor has, the more he can do in a given time frame19:02
spaceoneand i have some experience19:02
spaceonemore than the most 19yrs old programmers can say about themself19:03
ronnyi'd guess you have at most 2 years of toying around19:03
spaceone'have' ?19:03
spaceoneif you mean 'had'19:03
ronnyexperience is something you accumulate19:04
spaceoneyes, becuase i began with bullshit language called php19:04
ronnywel, if you have "experience" then you should be able to tell me what basic building blocks you need to build that app on19:06
ronnyin terms of semi-infrastructure19:06
spaceonei can only try to explain19:07
spaceonethe 'building blocks' is a API which does not needs to be changed anymore19:07
spaceone(expect HTTP2.0 is released)19:07
spaceonelol nice word play19:08
spaceoneronny: i want a reliable structure which does things in *one* way19:08
spaceonea way which requires small code19:09
spaceonea way which let the programmer be powerful19:09
ronnyok, you literally have no idea what you are doing, and just have a few buzzwords in mind19:09
spaceonegive me an example of what kind of explaination you want and i will give you my ideas19:10
spaceoneronny: btw what is your goal?19:10
ronnyspaceone, when having an app in mind like you said, you should be able to identify basic infrastructure components and how they would work together19:10
ronnyjust a http api is not gona cut it19:10
ronnyand the idea of a api you dont need to change, well, not so realistic19:11
ronnyi tried something like that about 10 years ago, failed beautifully horrible19:11
spaceoneronny: when do you need to change an api? i mean if every HTTP19:11
spaceonefeature is build in you don't need to change anything19:11
ronnyi suppose its best to have you experience it :)19:12
spaceoneronny: do you think all i am talking is bullshit?19:14
spaceonedo you think i am a worse programmer?19:15
ronnyspaceone, what you are currently sprouting is roughly equivalent to the mistakes i did roughly 8-12 years ago19:15
spaceoneronny: can you show me one programmer which never did that mistakes?19:16
ronnythere are some :)19:17
spaceonei guess so, but very rare19:18
ronnyyou should flip your approach a bit19:19
spaceoneflip == swap ?19:20
ronnyinstead on creating own things, start to work on projects where you know the people involved are better than you19:20
spaceoneronny: okay, i did by using circuits19:20
ronnyit tends to be a great learning experience19:20
spaceoneronny: on my first project i also did it19:20
ronnyspaceone, i mean dont make frameworks or that apps19:21
ronnyin fact for a while, dont make own apps, but help on others projects /read them19:21
spaceoneronny: so i should implement my cms/whatever using existing frameworks19:21
spaceoneor work on existing frameworks19:22
ronnymore like bring a cms thats close closer19:22
spaceonecan you explain 'close closer' please19:22
spaceone/describe in other words19:23
ronnyas in take a cms that does some/most things you like, and try adding one of the things you want19:24
ronnyin small deliveralbe steps19:24
spaceonebut i did not find any cms i like19:24
ronnywell, i did not even find a programming language i like :)19:25
ronnyyou have to make a compromise somewhere19:25
spaceonebut there isn't even any i like a little bit19:25
ronnyand why do you dislike them?19:26
spaceonenot easy adaptable, not integrateable into existing webistes, no REST design, don't following basic http rules19:27
ronnyi see19:28
ronnyi suppsoe you want everything or nothing19:28
ronnythen at least tell me what infrastructure components you need to build the thing19:28
spaceonegive me some more time19:29
spaceonefirst layer: http parsing19:29
ronnyhttp is a implementation detail19:29
spaceone2nd: bringing http into objects19:29
ronnythat faslls all into REST19:29
spaceone3rd: mapping http-things to resources19:30
ronnybut thats like only one kind of frontend you need, and literally none of the backends19:30
ronnystill REST19:30
spaceoneafter this is done, i just want a simple method returning data (from a model)19:31
spaceoneafter that the it should be brought into a representation19:31
ronnyplease drop the imlementation details19:32
ronnyim talkign about some bigger pictures here19:32
spaceonegive me an example please19:33
ronnystuff like databases, message queues,19:33
spaceonei can give you class names, but that would also be implementation detail19:33
ronnyevent streaming19:33
spaceonethe things like database is something which is a implementation detail of the specific app/reosurce/module19:34
ronnythe elevel im talking about is the semantic component stucture of the application19:34
ronnybecause for certain featues you will need certain data interactions that need infrastructure things like databases, message quees, websocket handlers, http handlers19:35
spaceonewhat do you mean by message queue?19:36
ronnyspaceone, thats a thing you use to push notifications and background work to different processes19:37
ronnythink zmq/rabbitmq/amqp19:37
ronnyhmm, seems like i'll be afk for the rest of the night19:40
*** Osso has quit IRC19:41
spaceoneronny: can you show me some of your projects?19:48
spaceoneprologic: ping me when online please!19:49
ronnyspaceone, you can scrape my bitbucket and github, my name is RonnyPfannschmidt on both, also take a look at pypy, pocoo/flask and sqlalchemy20:49
ronnyin particular sqlalchemy and its history is an example of something really good (sqla is of epic quality)20:50
spaceoneronny: jeah i know sqlalchemy21:28
spaceoneronny: you are also flask developpert?21:29
ronnyim partially involved21:29
ronnyim slowly doing supporting the werkzeug python3 port atm (doing mroe review than coding atm)21:29
ronnyi just recovered from some bad things21:29
spaceonei really like sqlalchemy21:30
spaceonei dont like falsk and werkzeug21:30
spaceoneronny: where are you from?21:31
ronnyspaceone, germany21:31
ronnyatm turingia, in ~2 weeks bonn21:31
spaceonewhere exactly21:32
spaceoneronny: you are german?21:32
spaceoneor just living there21:32
spaceoneso you speak also german21:32
spaceonenice to know21:32
spaceoneronny: can you translate the word 'abbildung' for me?21:33
ronnyusually its called figure21:33
ronnyat least thats what latex uses when not seeting the language to german :P21:34
ronnywhats the context?21:34
spaceone"objektorientierte abbildung von http"21:34
ronnythen its a mapping21:36
ronnyobjectoriented mapping of http21:36
ronnysounds scary :)21:36
spaceonei sounds nice in german21:37
spaceoneronny: how old are you?21:38
spaceonedid you study software engineering?21:39
ronnygeneral computer science, and did a lot of opensource on the side21:40
spaceoneronny: do you think you understood REST ?21:42
ronnyi got a pretty good understanding of it21:43
ronnybut im not 100% certain atm, its been a while since i did read that dissertation21:43
spaceoneokay, REST is more than most people think... and REST has less do with HTTP than most people think21:44
spaceonei am sure i did not understand it fully21:45
ronnyhttp is just a implementation of that structural pattern21:45
spaceonei also was to lazy to read all of the pages of the dissertation, i read only the page about REST21:45
ronnywell, plain http is pretty close21:46
ronnyit can be proxied, cached21:46
spaceonei think i understood http21:46
spaceonei just had a look on the http parser.c21:47
spaceonefor me it does not look nice, do you want to see how a http parser should look like, ronny?21:47
ronnyits c, its a bit nasty21:48
ronnybut it implements a fit statemachine21:48
ronnythats ragel21:48
spaceonebut there is fully clear what is meant21:48
ronnywell yeah, it expresses a statemachine directly :)21:49
spaceoneronny: i need you to more understand things21:50
ronnyi know ragle and even a few http parser written in it21:51
spaceones//i need you so that i can understand things better/21:51
spaceoneif we come back to the streaming thing21:52
spaceoneare there clients which does not send all headers in one chunk?21:52
spaceone(i know that this is no argument)21:52
ronnycertainly :)21:52
spaceonebut in generel i can't imagine that there are clients which does not send all headers in one chunk21:53
spaceonethis is only a few bytes21:53
ronnyspaceone, i knowsomeone that has high frequency internal deployment with ~15 req/seconds21:55
ronnyif the headers span more than one packet they will not arrive in one chunk21:55
spaceoneronny: do you agree that i mapped all http verbs of a http message into objects?21:59
ronnyspaceone, wait, where?22:00
spaceoneronny: in my httoop22:00
ronnyalso what do you consider verb there22:00
spaceoneronny: every part of a http message22:00
ronnydid you account for the protocol being extensible?22:01
ronnyspaceone, well, thing is, to give a propper answer, i'd have to do generative tests against the full rfc22:02
spaceoneronny: just look at the message22:02
spaceonea http message can currently only contain:22:02
spaceonerequest: method, uri, protocol, version; headers; body22:03
spaceoneresponse: protocol version; status ( code, reason); headers; body22:03
ronnyseems complete22:03
spaceonewhat extensions could exists?22:03
spaceonehas nothing to do with the message format22:04
ronnynote that i'll now hit the bed, i need sleep22:04
ronnylike seriously - didnt have much the last few days22:04
spaceoneronny: hmm... i had not enough sleep since 3 month :D22:04
ronnythats a misstake22:05
spaceoneronny: i hope we can continue tomorrow22:05
spaceoneronny: hmm, i had to finish project22:05
ronnygrabs ome good sleep22:05
ronnylack of sleep costs you productivity22:05
spaceonei know22:05
spaceoneand i also felt22:05
spaceonebut the project was not finishable in that time22:06
spaceonei now got my results22:06
spaceonefunctionality and architectuer: 100%; code documentation: 100%; project documentaiton: 98%22:06
spaceoneit was arround 1000 hours work22:07
*** ronny has quit IRC23:51

Generated by 2.11.0 by Marius Gedminas - find it at!