IRC Logs for #io Friday, 2018-12-07

*** ntalin has joined #io17:13
ntalinjer: Do you have a link to docs on the Callisto language? Did the 17:15
ntalin"Io Tome" ever pan out?17:15
jerntalin, it did not.17:32
jeri'm afraid i got too involved in and intersted with functional programming and my io work ceased17:33
ntalinjer: Aw. Did Callisto do anything differently from the "worlds" for Smalltalk and JS that Alan Kay was working on?17:47
jerntalin, it never really got too far off the ground if i'm honest17:54
jeri shifted gears to working on a different idea, ahead of time compilation through llvm on a different implementation called `acute` -- the simplest io that you could build, while still being source code compatible with io, and compile it down to machine code with a runtime support library17:55
ntalinAoT compilation for Io? Now I'm really interested. Aren't fexprs (which unevaluated message handling is equivalent to, I think) uncompilable in most languages?18:08
ntalinjer ^18:14
jerheh sure, so my technique relied on whole program knowledge18:15
jerall dependencies, all code, and for the whole thing to be compiled as one unit18:16
jergiven that, i could determine when/where messages were being manipulated18:16
jeracross which objects18:16
jerand those would be effectively eval'd and not compiled18:16
jerbut most io code tended to be, i assume still is, mostly static18:16
jerstatic in terms of not having the AST changed underneith them18:17
ntalinRight, Wand's proof stipulates that compilation of fexprs in an implicitly-evaluating language must be via whole-program optimization. Does Io have 1st class environments like Kernel?18:17
jerso late dynamic binding works fine, requires runtime support18:17
jerbut many operations can be replaced with builtins that have direct or a series of direct instructions to make them work18:17
jerrequires a change in how objects get represented, but it's totally doable18:17
jerntalin, in iovm? not at the moment, no18:18
jerthat said, not a terribly difficult thing to build into the vm18:18
ntalinThey can be added after the fact to the language? Geez, it's powerful.18:19
jertotally18:20
jerin io, code is data18:20
jerand you can hijack the entire vm18:20
jerhave code run without the rest of the app knowing about it18:20
ntalinWould it be possible to have a Io-like language with capabilities?18:20
jerthis gives you the ability to use it for "great good" as well as "pure evil" =D18:20
jerntalin, decorators on function/object definitions, requires tweaks to how messages work, haven't fully figured out how to make this work18:21
jernot decorators in the python sense, just to clarify18:21
jermore hints18:21
ntalinI'd like to build real world things with a Io-like language, but you just know people off the Internet are gonna go pure evil.18:21
jerntalin, you're talking to one of them -- most of my examples that i've built for people publicly, have been things that could be considered pure evil18:21
jeri wrote a DSL in Io that'd accept valid C code and execute it as Io code18:22
jerw/o any parsers =]18:22
jerwayback machine probably still has it18:22
ntalinThat's really cool. It sounds a bit like the ZetaC compiler for Lisp Machines, which wasn't terribly efficient but allowed debugging C code with all the power of Lisp.18:22
ntalinDo you happen to have a link?18:23
jerhttps://web.archive.org/web/20080626093912/http://blurgle.blogspot.com:80/2006/09/fun-with-c-dsl.html18:24
jerhad to find it18:24
jer=]18:24
jerit was horrible18:24
ntalinAwesome, thanks. Bookmarked.18:24
jeralso the code almost certainly won't work anymore18:24
jerit used the old message traversal mechanisms18:24
jeri didn't redo it when we changed how messages were linked to one another in ~2012-ish18:24
ntalinI was thinking of writing my own Io anyways.18:24
jergotcha18:24
jera simple iovm isn't hard to build18:25
jerjust gets complicated the more stdlib you want18:25
jer=D18:25
ntalinI have a thing for toy languages.18:25
jerheh18:25
jeri'm mostly writing Erlang and Elixir these days18:25
ntalinI'm doing C++ for my classes. I just wrote an AVL and red-black tree with raw pointers.18:26
ntalinkiiiiill meeeee18:26
ntalinHow's Erlang? I'm considering going into distributed systems because then I might use interesting paradigms like the Actor Model.18:27
jerfor interpreting that blog post, Message attached, use to be `b` in this message sequence `a b`; while `nextMessage` used to be what's now basically a while loop around `Message next` until you get to a semicolon, and then taking the next message after the semi colon -- i.e., `c` in this bit: `a b; c`18:28
jerthe whole erlang ecosystem is great imo18:28
ntalinYeah, I've heard great things about OTP.18:28
jerso i'm working on a machine learning deployment platform18:28
jermuch of our systems are written in Elixir18:29
ntalinDoes it need high performance?18:29
jerno, it's not the fastest language in the world18:29
ntalin(from the Erlang/Elixir)18:29
jerwhat it gives you are soft realtime guarantees18:29
jerso predictable latencies18:29
jeralso gives you the tooling to treat multiuser systems, or distributed applications with some common sense -- think about your project in terms of risk, try and minimize the riskiest areas of your app so if they go bad, they impact the fewest users possible18:30
ntalinI ask because I just found a supercompiler for Erlang, erlscp.18:30
jeralso wrap your system in a supervision tree structure to ensure that the system restarts properly18:30
jerwhen i say Erlang, Elixir, LFE, ... they're all basically different syntaxes on top of erlang18:30
jerthe libs are all the same18:30
ntalinI thought LFE used the Erlang libraries and Elixir had its own?18:31
jerelixir has some of its own libs, but those work in erlang too: `'Elixir.Enum':sum([1,2,3]).` for instance, is just Enum.sum([1,2,3]) in elixir18:31
ntalinAh, cool.18:32
jerLFE piggybacks on top of OTP directly, is effectively a syntax for erlang18:32
jerelixir is a little different in that it namespaces its stuff18:32
jerit compiles to a different part of the compilation pipeline18:32
jer(LFE hits core erlang, elixir does not)18:32
jernot sure elixir could hit core erlang if i'm honest -- which means elixir's compiler is left out of some optimizations that can be done at that level18:33
ntalinDo you feel that programming in Erlang is more functional or object-oriented (in the message passing sense)?18:34
jernowadays where my interest lies in software development is with embedded and distributed systems18:34
jer(not IoT to clarify)18:35
jerntalin, definitely think that OO is more of a set of design patterns, while FP is a paradigm18:35
jerso the only way i can answer that is, it's more of an OO language than most OO languages are18:35
jerbut it's a FP language18:35
jeryou don't pass state directly to other objects -- you send messages, and they get their own copy18:36
ntalinThen what's Io's paradigm?18:36
jernothing is shared18:36
jerio takes a shared memory single executor paradigm to define its objects18:36
jeras such you're simply passing messages between areas of memory18:36
jerio's definitely imperative, erlang is declarative18:36
jerio implements OO design patterns too18:37
jerbut it embraces almost nothing of FP18:37
jerit's not declarative at all18:37
jeryou can program in a declarative way in io, but it's lipstick on a pig18:37
ntalinI'd imagine so. The speed tradeoff is because it makes few enough assumptions that you can monkeypatch to your heart's delight.18:37
ntalinIs bytecode compilation possible with Io?18:38
jerit's possible, it's what we already have really18:38
jera bytecode interpreter in io would be one that just serializes messages18:38
jerwe had one of those at one point18:38
jerdidn't speed things up at all18:38
jerso we rolled back18:38
ntalinWould it maybe be faster with a partial evaluator?18:39
jermaybe? but any speed gains you'd get that way, would also be gained with the current evaluation model18:39
ntalinWhat's the current eval model?18:39
jeri honestly think the only way you make io faster is to build a new vm that's backed by primitives, which can be overridden (to maintain io's flexibility) at the cost of performance18:39
jerntalin, it's a tree walker18:40
jersource code is read, all the lines, then it's parsed, it's passed through a message rewriter (for operator shuffling), and parsed again (for priority)18:40
ntalinWhere can I read more about this?18:40
jernot sure it's specifically documented at all, but a tree walker virtual machine is a common approach for interpreters18:41
jerit's in the dragon book18:41
jeriirc18:41
jeranyway what you end up with after parsing io code, is a tree of messages18:41
jerjust messages18:41
ntalinIs it covered in Lisp In Small Pieces? I'm planning to read that once I finish SICP.18:41
jereven `Object clone` by itself, is two message sends; `Object` to the Lobby, and `clone` to whatever the Lobby returend for `Object`18:42
jerntalin, dunno18:42
jeranyway, all io code descends from one root message18:42
jerso you just have to walk the tree18:42
jerthink of all the left branches being the next message, and all the right branches at each node being the args18:42
jeryou lean right when walking18:42
ntalinI always thought that the message tree walking was just like wrap in Kernel. Is that the wrong way to think of it?18:43
jeri don't remember; but it's just effectively tree traversal18:43
jerwith special rules on when to follow what things18:43
jeri.e., you might be walking one path then have to jump to another path because of inheritance18:44
jerrules18:44
jerthe thing i like most building these days is hardware18:44
jerand this just gave me an idea18:44
jer=D18:44
ntalinOh?18:44
jeri built a CPU that is based on a transport triggered architecture18:44
ntalinI've been thinking about that exact topic!18:45
jercan be thought of, that execution of an instruction happens as a side effect of the transport of data in the system18:45
jerthis is a perfect model of computation for an io machine18:45
ntalinI know nothing about hardware, but thought that the bus messages were just like messages in an OO language!18:45
ntalinI'M NOT ALONE!18:45
ntalinHURRAH!18:45
jerheh18:46
jerso without getting too technical in how a cpu works18:46
ntalinEach CPU having its own RAM would be equivalent to an actor, no?18:46
jermy instructions have one form https://gist.github.com/jeremytregunna/870e317f15c985eeea62e6c6bbaa4d9518:46
jeryou can think of the source unit as the sender, the source subunit as the caller (subtle difference; one is an object/actor, the other is the message data is coming from); the destination unit is the target, and the destination subunit as the message to be called18:47
jerthe M* C* lets you conditionally execute those instructions18:47
jerbased on ALU flags18:47
jerBS gives you the ability to parallelize instructions across different data busses (same set of functional units, so not quite true ILP)18:48
jerand the IMM bit lets you load 31-bit immediates into the system18:48
jerthose immediates could be, for instance, unsigned floating point values18:48
jeror signed 16b/8b fp values18:49
jeranyway18:49
ntalinI'll have to read more about this. There goes the rest of my nonexistent free time.18:50
jeri built this cpu originally as a control unit for my ML accelerator, but then for that hardware project i switched to use the TPU's instruction set so i could use tensorflow w/o having to write my own damned compiler =D18:50
jeri play around with fpga's these days =D18:50
jermy cpu has the performance of a p54c pentium 7518:50
jerruns at just under 200mhz18:51
jer(well most of it, some of it runs slower)18:51
jerunfortunately one downside of TTAs is that they suffer the same problem as VLIW architectures in that you need smart compilers18:51
jercompiler has to know exactly what time to read data, etc.18:51
ntalinWhat are your thoughts on Mercury? I've been thinking that it could be used for real time and embedded software because it has a compile-time garbage collectors and the type/mode system can statically assert that only the stack is used.18:51
jeri'm not familiar with mercury18:52
ntalin(plus it's easy to write backends, from what I've heard)18:52
ntalinIt's a functional/logic language.18:52
jercompile time garbage collectors typically would lean more functional than not =D18:52
ntalinIt's a love child of Prolog and Haskell18:52
jerahhh18:52
jererlang is a prolog descendant18:52
ntalinKind of. It has the syntax and the pattern matching is kind of like unification, but I've always felt that Erlang is more Lispy in practice.18:53
ntalin(well, it's a descendant, but not like it, is what I mean)18:53
jerthe prolog erlang is derived from is very, very old, and it's no longer a prolog18:59
jerbut yes some bits still remain as hints of its past18:59
jerIo is more of a lisp than erlang is18:59
ntalinIt is.18:59
jeryou can convert erlang code into data, but erlang code isn't itself data. messages are data though; io is the same way, but all code is messages in io, thereby all code is data.19:00
ntalinIDK why people insist that Lisp is a functional language. It can be used that way, but then there's no advantage over OCaml or Haskell. Macros are the opposite of referential transparency. Just because they're closer to it than the alternatives doesn't give them a free pass, and they get used, unlike template languages.19:01
ntalinWhat's different about the Prolog Erlang comes from?19:02
ntalinWas it closer to early message passing languages like Hewitt's Planner?19:03
jerthose are days way before my time, i think joe armstrong has written about it19:09
ntalinHow is it that Io ended up resembling Smalltalk-71/72 so closely?19:10
jermight be interesting to make a system based around a unified memory, where the combination of a module address that contained a list of functions, and a function address inside that module gave you a list of instructions to be performed19:11
jerntalin, intent, and somewhat accident =]19:11
ntalinHave you ever heard of the FFP machine?19:12
jersteve was very into Self, i was very interested (at the time) in newtonscript19:12
ntalinIt was a computer architecture meant to execute the functional array language described in Backus's Turing Award lecture.19:12
jerno i haven't19:12
jerah19:13
ntalinhttp://www.cs.unc.edu/techreports/87-014.pdf19:13
ntalinI've been thinking that Lisp (as it exists, Scheme for reference) isn't as fundamental as people claim, that eval/apply can be separated into more powerful languages. In that view, Kernel/Io is what you get when you remove apply and keep eval, and APL/Joy/FP is what you get when you remove eval and keep apply.19:15
jerwell io is what you get when you make eval only take one type, then compile your language into objects of that type, and walk the tree of objects applying that singular function19:16
jer=D19:16
jerntalin, i'll take a look19:18
jerthese days i identify more as an fpga developer than a software developer tbh =]19:19
ntalinThen hopefully the work on the FFP machine is useful. :)19:19
jerthough for work, i manage a team of software engineers and devops engineers19:19
jer(write some codem myself too at work, but it's only a secondary responsibility of mine now)19:19
jerand only because we're still a startup =D19:19
ntalinWhere do you work?19:19
jermetis machine, i'm the director of engineering19:21
jerwe're a machine learning startup19:21
jerfocusing on making ml models run in production environments at scale19:21
ntalinCool.19:22
jerntalin, a lot of this paper so far on the 2nd page is sounding familiar19:22
jerbut i can't place it19:22
jermy current work if i'm honest, is trying to figure out jesd250a (GDDR6) spec enough that i can write a controller19:23
jermy ML accelerator is currently bandwidth limited on DDR319:23
ntalinBTW, there's a massively parallel APL compiler for GPUs called co-dfns. https://github.com/Co-dfns/Co-dfns19:24
ntalinHave you done APL before?19:24
jeri have, but don't have a keyboard for it anymore =p19:24
ntalinEmacs has an excellent mode for GNU APL which allows the use of a standard keyboard.19:24
ntalinMaybe it could be adapted.19:24
jerGPUs don't interest me though19:24
jerhighly efficient SIMD machines sure19:25
ntalinAre you using TPUs instead?19:25
jermy ML accelerator works as a TPUv1 =D19:25
ntalinOh wait, you already said that.19:25
jeryeah19:25
ntalinWhat languages do you use for parallel programming? Is it mostly existing C/C++/Fortran libraries?19:26
jerthe kind of machine i described before (unified memory, where the combination of addresses lead to a series of instructions) -- sounds exactly like a microcode decoder19:27
ntalin(FYI, co-dfns is AGPL-licensed and the source is unreadable)19:27
jermostly python these days if i'm honest19:27
jernot the most efficient, but there's a crapton of parallel libraries in python that hit TPUs, GPUs, CPUs19:27
ntalinSo C/C++/Fortran libraries.19:27
jerbasically19:27
jeri don't care about that though =]19:27
jeri've got a 1080 Ti with 11gb of memory, it's great for running many kinds of models on19:28
jermy ML aceclerator is still far slower than it =D19:28
jeron everything i've tested19:28
jerbut that's to be expected19:28
jerit has a core count advantage of a couple hundred times19:29
jeri've only got 196 multiply-accumulate units on my accelerator right now, it's got 3584 =D19:30
ntalinWhat kinds of problems is Metis built for? Is it focused on online learning?19:30
jerso Metis isn't my company; Metis Machine is, they're different19:30
jerMetis is an online learning company19:30
jerMetis Machine is a ML deployment platform19:30
jer(we may end up rebranding ourselves for that confusion reason)19:31
ntalinI didn't mean education, I meant online learning as in the NN technique.19:31
jerbut even with my 196 MACs, i still am faster (only marginally) than my CPU (Ryzen 7 1800X) at executing my test model19:32
jernope19:32
jerwe have no education bits as part of our platform19:32
ntalin.-.19:32
jerour bet is that you already have models, or if you're just starting out, that you'll take a sample model (we've got a few sample models) and run those... we'll train them on our platform, fetch any data you need for training, deploy those to score / inference on our platform, OR, send them to your users' mobile devices19:33
jerthat's what we do19:33
jerour mobile platform isn't up yet, but should be tuesday =]19:33
ntalinWould ML benefit from architectures which bare all of their parallelism given a fully optimizing compiler for an APL/Futhark-like language?19:35
jerpossibly19:36
jernot all ML algorithms are equal, or benefit from massive parallelism19:36
ntalinIs it because linear optimization is hard to parallelize? (I don't know that much about this - I'm about to take linear algebra and have been studying on my own)19:37
jerso consider a recommendation engine -- what you've got for instance, let's say it's a social graph, and you're trying to recommend people to follow ... this is inherently a linear time serial algorithm, and won't benefit greatly from more cores19:40
jerwhat it benefits from is fast execute and memory bandwidth being available19:40
jerthe thing with making ML go fast is making sure you can keep your algorithms fed with data19:40
jerand utilization high19:41
jerif your utilization is low, you're gonna be slow, TPU, GPU or CPU be damned19:41
jerin fact, if utilization is slow, you might be better on a cpu19:41
jerwhich has faster, though smaller amounts of cores, and caches19:41
ntalinIf you want to give Mercury a first shot, it has a BEAM backend.19:47
ntalinDo you use Emacs?19:47
jeri do not19:47
ntalinVim? Atom? Eclipse?19:47
jeri'm so much a vi user, i even recently found a vi reimplementation i wrote over a weekend like 12 years ago19:47
ntalinWow19:48
jerhttp://jer.tregunna.ca/9pcaLL19:49
jer=D19:49
jerthat's actually my ReLU implementation loaded into it, in verilog19:49
jer(part of my ML accelerator's activation pipeline)19:49
ntalin(I asked about Emacs because Mercury has really good integration with it - you get all sorts of automated linter warnings and tips before you even compile)19:49
jerah19:50
ntalinIt feels interactive, but in a very different way from REPL-based languages.19:52
ntalinI haven't even checked if it offers a REPL (I don't think it does) - I haven't felt the need.19:53
ntalinAnyways, if you want to try it just for Mercury, Emacs has a package called Evil which gives you Vim keybindings.19:54
ntalin(I've also heard that the Haskell experience is similar with the right packages, although I never bothered to set up a full environment for it)19:54
jermy ML accelerator at 14 x 14, uses under peak load, 0.580 mW of power20:00
jerat 1.8V20:00
jer178mhz20:00
jer70.4 GOPS20:00
jerdefinitely suitable for embedded use cases in devices for model inference =D20:01
jeri do want to get one of google's TPUs built for the same purpose to benchmark against it20:01
jerbut i'm thinking i might try and start my own hardware company, and convert this design into one suitable for an ASIC, convert that design to a mask, and get that made under 180nm cmos, since it's reasonably a cheap process20:02
jercouple hundred thousand chips made, for low 6 digits20:02
ntalinI've heard that a lot of ML algorithms run with exponential complexity when being trained and O(n) time when actually used. Is that why it's embeddable?20:02
jer(estimating my die area would be around 11 x 11 mm)20:03
jerntalin, well my design doesn't do training right now, unlikely to be something i want to do with it anyway20:03
jerGPU is fine for training, because they're cheap, and generally good for that kind of work20:03
jerat least it's fine if it's a NN with good utilization20:04
jerso what my core needs, is a pretrained model20:04
ntalinRight, it's meant for actual use (like how Google Duplex and Siri are probably trained using server farms and supercomputers but fit inside phones by the end)20:04
jerit'll do soft realtime inference20:04
jerat low power20:04
ntalinDo I have the wrong idea?20:05
ntalinHow do you analyze an existing model, by the way? I thought that most relevant internal information got destroyed on the loss function.20:07
ntalin(if it isn't obvious yet, I know hardly anything about ML)20:08
jerthat's ok, i'm not strong on ML either, but also prepping for a demo so can't go into details right now20:12
jersuffice to say, i'm not worried about the training problem20:12
jerjust the inference/scoring issues20:12
ntalinThe loss function: I-I|-II-L20:21

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!