IRC Logs for #crux-devel Wednesday, 2018-02-21

jaeger25 successful jobs in the container (which is running in docker on a VM, heh)00:03
j_vnice00:36
jaegerSo it's randomly (at least I think it's random) failing in the VMs... but it didn't fail in docker or on physical hardware00:39
j_vare there certain VM backends that exhibit the trouble?00:46
jaegerI haven't tested kvm yet (I intend to) but esxi and virtualbox both have the errors00:48
j_vit's kvm i'm having trouble getting a fresh kernel built for... i'm beginning to think i may have changed something in my main system kernel config that is breaking my normal config i use for vm's... i did slim down my config recently and silly me overwrote the backup00:51
jaegeroops00:51
j_vyeah... i'll get through it. i'd done stuff like this before, but i thought i'd put enough safe guards in to prevent me fat fingering it. but alas, seems i didn't do so well on this one.00:52
jaegerthis is ridiculous:01:12
jaegerlaptop: 25/001:12
jaegercontainer: 25/001:12
jaegerVM1 (esxi): 16/901:12
jaegerVM2 (esxi): 0/2501:12
jaegerVM3 (virtualbox): 13/1201:12
jaegerResults are all over the place01:12
jaegeron the VM Where it failed 25 builds, it's oddly consistent in its failures:01:14
jaegerhttp://ix.io/NgZ01:14
jaegernot 100% consistent but still01:14
*** Workster has quit IRC01:30
*** Workster has joined #crux-devel01:31
*** Workster has quit IRC01:42
*** Workster has joined #crux-devel01:42
*** Workster has quit IRC01:42
*** Workster has joined #crux-devel01:42
j_vi notice that all the failures there were permissions01:49
jaegeryeah01:55
j_vit's strange... is there something wrong with the footprint on that machine? because in the footprint i'm looking at, none of the *.c files are executable.02:37
*** Workster has quit IRC02:39
*** Workster has joined #crux-devel02:39
*** Workster has quit IRC02:39
*** Workster has joined #crux-devel02:39
jaegerI don't even know what the hell is wrong, heh03:02
jaegermaybe I screwed that up03:02
jaegerI've even tried explicitly setting umask during the build, doesn't help03:27
j_vyou checked the footprint file to see how the *.c files are listed in it? because according to the footprint in the repo, i think those files should have passed.03:30
jaegeryeah, I think I messed up the footprint on that host before the builds, going to redo them03:34
jaegeralso trying in a kvm VM now03:34
*** ___18VADBHIX has quit IRC03:47
*** ___18VADBHIX has joined #crux-devel03:49
jaegerfixing the footprint thing changed the stats to 17/8 on that VM, still crap06:02
jaegerfinal counts:06:03
jaegerlaptop: 25/006:03
jaegercontainer: 25/006:03
jaegerVM1 (esxi): 16/906:03
jaegerVM2 (esxi): 17/806:03
jaegerVM3 (virtualbox): 13/1206:03
jaegerVM4 (qemu/KVM): 23/206:03
*** Workster has quit IRC06:28
*** Workster has joined #crux-devel06:29
*** chinarulezzz_ has joined #crux-devel06:30
*** Workster has quit IRC07:04
*** Workster has joined #crux-devel07:05
*** frinnst2 has joined #crux-devel08:47
*** Workster has quit IRC11:31
*** pxxcqylziiiprbxi has joined #crux-devel11:31
*** Workster has joined #crux-devel11:31
*** Workster has quit IRC12:01
*** Workster has joined #crux-devel12:01
*** Workster has quit IRC13:21
*** Workster has joined #crux-devel13:21
*** Workster has quit IRC13:21
*** Workster has joined #crux-devel13:21
j_vdamn, that really sucks14:58
j_vand makes no sense14:58
jaegerAgreed on both counts14:58
j_vdo you use a script for the VM builds? i mean to setup the dependencies?14:59
jaegerI didn't in this case, go doesn't need anything really outside of core15:00
j_vtrue15:02
jaegerThe laptop and vritualbox VM have X installed, the others don't... the virtualbox VM also has a lot of other packages for Mate installed. the container and the qemu/KVM VM were both fresh clean installs15:02
jaegerThe other esxi VM is my docker host, so it doesn't have much installed besides docker15:03
jaegerI didn't really expect any of that to make any difference and it doesn't seem to15:03
j_vdefinitedly weird... setting up my container builds was because i kept running into inconsistencies before. so this is kind of worrying to me as well15:04
jaegergo is the exception, not the rule, from my experience... so at least it's not everything15:05
j_vwell, then, perhaps i've been thinking about this thing from the wrong angle15:05
pedjago apps are fiddly, pretty much all require a sacrifice of a different kind to build :)15:07
pedjaor different build tool, or environment setup, or...15:08
pedjathis is the latest shiny, afaict https://github.com/magefile/mage15:09
pedjaI just wish they standardise on *something*, but that's wishful thinking15:11
j_vfinally, figured out what i was doing wrong with my kvm kernel builds... the virtio-disk driver was being built as a module. damn that's dense of me... i know better than that15:11
j_vi feel so stooopid15:14
j_vi was sure i checked that several times, but apparently not. now i can start test that go build in VMs here as well15:19
fun_What was the problem with the module?15:21
pedjait has to be compiled in to work, I guess :)15:22
j_vyes, i would have sworn that i checked to make sure it was builtin. doh15:23
pedjaall virtio drivers on ISO kernel are compiled in, apart from net and baloon, iirc15:23
fun_Ah! Right. Otherwise the guest won't boot.15:24
fun_Silly me15:24
j_vyep... i thought i was missing some fs driver or feature. ok, i guess i need to get over it.15:25
*** Workster has quit IRC15:40
*** Workster has joined #crux-devel15:40
*** Workster has quit IRC15:40
*** Workster has joined #crux-devel15:40
fun_Maybe the clock source makes the reproductibility problem with go.15:51
fun_Regarding VM builds15:53
jaegerperhaps... will investigate16:16
j_vgot a build run of 10 for go running on a kvm instance right now. if all ten succeed, i'll run it another for another 25/30.16:26
*** john_cephalopoda has joined #crux-devel16:48
jaegerok17:03
j_vi build with the added line 'export GOCACHE=off' to the Pkgfile, then updated the footprint before starting the run17:05
jaegersame17:05
jaegerSo far the clocksource change has not fixed the problem. Numbers may end up different but still getting failures17:06
j_vcan you paste a recent failure?17:07
jaegeryou want the whole log or just the end?17:08
j_vi think the failure point should be enough, but the whole log wouldn't hurt... up to you17:09
jaegerhttp://ix.io/NuK <-- here's the short version, if you want more from a specific one just let me know17:09
j_vok17:09
jaegerI've got to leave soon for a lunch meeting but will be back after17:09
jaegerthat output is just a 'grep -B5 failed' on the logfile17:10
j_vno problem... i'll be here on and off anyways, so no prob17:10
j_vok, thanks... i'll try looking at this from a few different angles17:12
jaegercool17:13
j_vbtw, 10 successes for 10 on kvm build17:17
j_vso i'm starting a 30 build run on that VM... i'll look at installing virtualbox at some point today; haven't messed with it in a while17:18
*** Workster has quit IRC18:01
*** nloztugacayskvam has joined #crux-devel18:01
*** Workster has joined #crux-devel18:02
*** Henschi has joined #crux-devel18:23
*** onodera has joined #crux-devel18:23
*** Workster has quit IRC18:26
*** vyjtpdptkdjwkjut has joined #crux-devel18:26
*** Workster has joined #crux-devel18:27
*** Workster has quit IRC18:47
*** sqpytzyhxllskkrr has joined #crux-devel18:47
*** Workster has joined #crux-devel18:47
jaeger15/10 on the esxi VM1 using hpet clocksource... trying acpi_pm now18:47
jaegerheh, first build was already a failure18:50
j_von plain kvm, so container, i'm at 14/0 (not including the previous 10)18:52
j_vs/so container/no container/18:52
fun_Esx does risky tricks when told that it runs a linux image18:53
jaegervirtualbox and kvm both also have problems with it, for me at least. I wonder if there's something in the kernel config that causes it18:54
jaegerI haven't tried it on the ryzen box yet, will give that a shot too18:55
jaegerj_v: if your 30-build run finishes successfully, will you paste your kernel config for me to try?18:59
j_vjaeger: yeah, no problem... you obviously would probably still need to adjust it, since i have it stripped down for use on my amd host19:03
jaegersure, but it might be worth a test19:06
j_vhere it is: http://ix.io/Ny319:06
jaegerthanks19:06
j_vno prob... would be cool to even isolate what causes the failures19:07
jaegeryeah19:08
j_vi have a script that builds a crux package if you want... it's nice for keeping copies archived of working kernels19:09
j_vhttp://ix.io/Ny719:10
j_valso sets up the needed stuff for building out of tree modules19:10
jaegerI don't use a package for my kernels but looks interseting19:11
jaegerinteresting, even19:11
jaegerI do keep multiple kernels around, though, just in case19:11
j_vi stole most of that part for out tree modules from alpinelinux's kernel build APKBUILD19:12
j_vi was pretty shameless about that since i had no intention of actually publishing it19:12
jaegerheh19:12
jaegerhow did you invoke the KVM VM?19:14
jaegerI doubt it matters but I'm curious19:14
j_vgive a moment to drag it up, i use home grown script that uses dmenu and config files to choose different setups, so have to look in the logs to see the full invocation19:18
jaegerno rush at all19:18
jaegerhttp://ix.io/Nyr is what I'm using, pretty simple19:20
j_vhttp://ix.io/Nys for the full qemu commandline... the 'once' part is because i had to run the ISO to chroot into it first to fix the kernel (wasn't bootable until i installed a working kernel in there) and then rebooted into the raw image19:22
j_vthe extra.raw is so i can save stuff or load stuff at qemu stop/start19:23
jaegerlooks pretty straightforward, nothing complicated19:24
j_vyeah, i should be using 9p or similar to do the work of the extra.raw image, but haven't gotten around to it19:24
j_vusing the curses interface is smart19:25
j_vi do that when i'm messing around with (net|open)bsd19:26
jaegerI just like the simplicity and the ability to run in tmux :)19:27
j_vyeah, plus you can detach it or loose x11 with no danger of loosing the process19:27
j_vs/or loose/without loosing/19:28
j_vno, the first was right19:28
jaegeralmost. "lose" would have been correct but I figured that's what you meant19:28
j_vah, yeah, my english is not always what it should be for a native english speaker with a decent vocabulary19:29
jaegerEnglish is silly in many ways, too19:30
j_vno doubt19:34
frinnstjaeger: how are you building go that gives you troubles. in a container on that vm or something?19:35
frinnstI got a crux esxi vm too to play with if that could be helpful19:35
jaegerI'm building on several different hosts per that list I pasted before... intel laptop (physical), esxi VM1, esxi VM2, virtualbox VM3, qemu/KVM VM4, and container (in docker on VM2)19:37
jaegerIn all cases it's just contrib/go with 'export GOCACHE=off' added19:37
jaegerI need to push that update, it's not the cause of this problem anyway19:38
jaegerfor x in $(seq 1 25); do fakeroot pkgmk -f; done 2>&1 | tee builds.log19:39
jaeger(I'm building in my homedir, not contrib)19:39
jaegerSo those are also possible factors... homedir build as well as fakeroot19:40
frinnstyeah fakeroot has caused some problems previously. iirc it created another footprint for syslinux19:40
jaegergoing to try 25 root builds on one of the esxi hosts next19:41
jaegerwith that said, I used fakeroot on ALL of these systems19:41
frinnst=======> Building '/home/ports/pkg/go#1.10-1.pkg.tar.gz'.19:42
frinnsthere we go19:42
jaegerwoo19:42
frinnstjust one cpu tho. I can add more. how many have you tried with? any value causing more problems than others?19:42
jaegerdifferent number of cores doesn't seem to have any effect. laptop has 8, VMs have either 2 or 4, container had 419:44
frinnstalright ill let this brew and see if anything happens over night19:45
frinnstits a very thin vm, 1 cpu and 512mb ram :)19:45
jaegerthanks, more input is welcome :)19:46
jaegerj_v: I'm trying your kernel config unmodified, for science (tm)19:48
frinnstcrazy slow, more ram!19:48
jaegerdownload more!19:49
*** frinnst2 has quit IRC19:49
frinnstlots of missing stuff after the first build19:56
frinnsta few hundred of these: MISSING   -rw-r--r--      root/root       usr/lib/go/pkg/obj/go-build/e5/e56a6917d7ec86e319ad18203ebeb19f61d960511156fd60ebb297339c84fd09-d19:56
frinnstMISSING   -rw-r--r--      root/root       usr/lib/go/pkg/obj/go-build/e5/e5714d77f4459f2adf978c938d9ae32d0abe14d0073a8297736902e75657473d-a19:56
j_vthat's the stuff that adding 'export GOCACHE=off' (under the other exports in the Pkgfile) removes from the footprint19:58
frinnsthm, i did that19:58
j_vahh, pkgmk -uf?19:59
frinnstnever updated the footprint19:59
frinnstretrying20:00
jaegersorry, I should have clarified that. GOCACHE=off removes everything in the 'go-build' dir in the footprint20:02
jaegeras j_v said, yeah20:02
frinnstno worries. first build finished fine20:05
frinnstafk20:05
j_vfakeroot can mess up some stuff. i know that texlive fails miserably when built with fakeroot. for weird fs-like issues, too. i remember researching it when i first started writing my pkgwrap stuff, was one of the impetus for starting it in the first place.20:16
j_vi remember ran into something written by a fakeroot dev saying fakeroot wasn't meant to be a build environment.20:17
j_vbut more for packaging20:17
j_va few things are hard to get around it, or without something like it, like doing mknod in an unprivileged container... i use fakeroot to build core/filesystem, but that's about it.20:19
*** Workster has quit IRC20:19
*** knqefticqizuiycx has joined #crux-devel20:19
*** Workster has joined #crux-devel20:19
j_vi'm looking testing setting my container program, nsjail, to setuid. that may help get around the mknod type limitations. and should also make using overlayfs work, too.20:22
jaegerhttps://docs.google.com/spreadsheets/d/1O6oJBg9Pw9REvLGWvqQqVD9Y4cLoL8YNnzxKS9F8gzc/edit?usp=sharing20:26
jaegerCan you guys see that?20:26
jaegerFigured it might help to organize the numbers a little20:26
jaegerWhere numbers are missing, those runs aren't done yet20:27
j_vcool20:30
j_vok, so i go 30/0 (40/0 including previous run) on qemu kvm, no fakeroot20:33
j_vi'll probably try a run using fakeroot now, because my gut reaction is that fakeroot is at least partially the issue20:35
jaegermore data points won't hurt, certainly20:36
j_vmy gut reactions have been wrong plenty of times, unfortunately20:37
jaegerI have a LOT of CPU time dedicated to building go right now, heh20:50
*** xor29ah has quit IRC20:51
j_vyeah... hope i didn't encourage you to get too sidetracked... not saying you ARE sidetracked, just hope you don't feel like you are.20:52
jaegerAll good. I'm waiting for a couple of things at work so have a little time to mess with it20:54
*** xor29ah has joined #crux-devel20:54
j_vcool20:55
fun_fakeroot uses LD_PRELOAD, which is ignored by statically linked programs like go :)21:00
*** Henschi has quit IRC21:00
jaegerMakes sense, I guess21:03
j_vso i figured out my trouble with building kernel for kvm, but still can't track down why my container program nsjail doen't work in qemu VM. i'm pretty sure it did at some point.21:19
jaegersounds strange21:19
j_vthinking all the meltdown and spectre updates may have something to do with it, but i'll have to boot the physical machine into an older kernel to prove that theory out.21:20
jaegeryeah, could be21:21
frinnst21 successes. I think that's 100%. still some to go21:23
frinnstzzz for now21:23
jaegernice21:23
jaegerand night21:23
fun_jaeger: your docker container built go successfully while its host failed?21:33
*** Workster has quit IRC21:34
jaegerindeed. :) Charming, isn't it?21:34
fun_Hehe21:34
*** Workster has joined #crux-devel21:34
*** Workster has quit IRC21:34
*** Workster has joined #crux-devel21:34
fun_So, kernel/clock/fakeroot out. Filesystem in VM must be.21:41
fun_I remember that I had to disable sendfile usage from nginx config when running in VM.21:43
jaegerwell, I haven't finished the root vs. fakeroot tests yet but we'll see21:45
jaegeroddly j_v's kernel config worked fine with fakeroot21:45
j_vjaeger: can you paste a copy of the original config you were using so i can look at the diff21:53
j_v?21:53
jaegersure, sec22:05
jaegerhttp://ix.io/ND022:07
john_cephalopodaHere's some log of my go build, not sure if I posted it before: https://bpaste.net/raw/502d3fb9404722:13
jaegeryeah, that'll be fixed by the next push22:13
j_vthanks22:19
j_vafk for a bit, bbl22:20
*** onodera has quit IRC22:41
*** john_cephalopoda has quit IRC23:06
j_v30/0 in KVM using fakeroot as unprivileged user23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!