IRC Logs for #crux-devel Monday, 2019-03-18

frinnsthah curfew seems to be awesome.00:05
frinnstwacky races with zombies00:05
frinnstnot sure what the shitty rating is all about00:06
ryuofrinnst: does that make it extra flushable? :)05:53
juefrinnst: no, cairo builds fine for me, with 3.4 and 3.507:18
frinnstcheers. must be somehthing else then07:25
juefrinnst: is meson a requirement for newest glib?07:48
frinnstno but who knows how long they will maintain autotools side by side?08:17
juethat's right, sooner or later they will drop autotools support, IMO08:34
frinnstI prepared all my gnome ports yesterday but never pushed anything08:45
frinnstglib was hell :-)08:45
*** xor29ah has quit IRC10:47
*** xor29ah has joined #crux-devel10:53
juefrinnst: 32 packages here, which references the glib libtool file ;)11:05
frinnstyep, fun times. began around 23:00 last night :-)11:05
jueguess that could be a major thing for 3.6, to remove all or at least almost all libtool stuff11:09
Romsteroh god and watch the world burn11:09
juenow that we are talking about, I remember that tilmann has for many years the idea to remove all *.la files ;)11:11
juebecause he considered it as crap :)11:12
jueRomster: yeah, indeed :-)11:14
juebtw, here's the little awk script I wrote for the jpeg issue, which makes it simpler to find the owers of files11:22
juewith it:11:23
jue./pkgowner $(grep -lrs /usr/lib) | sort -u11:23
frinnstshame the 'finddeps' output is not very prt-get friendly11:46
frinnstperhaps a flag to make it so would be handy. would make it easy to brute force it11:47
frinnstin this casae it would obviously need to find recursive deps so not very relevant, but still11:47
frinnstoh, it seems to handle it pretty well actually11:50
frinnstsuspected it would freak out on (core), (xorg) etc11:51
frinnst-- Packages not found11:51
frinnstnot very pretty :-)11:51
juehmm, finddeps will not help with the *.la issue, guess you are talking about a differnet problem?12:03
pedjaafaict, pretty much every distribution removes .la files12:24
pedjaat least fedora, opensuse and arch do12:25
pedjamaybe that would explain libvirt breakage that has been annoying me for a while now :)12:26
juefrinnst: FYI, ninja 1.9.0 works fine for me12:37
Romsteri did ask about libtool slaying some time long ago too.13:02
pedja'libtool slaying' :) I like that13:07
pedjaRomster, I was thinking about opening a bug about python3 symlinks, with the diff against my port which works with boost. Interested?13:08
Romsteryeah do that i haven't looked into it yet post the patch13:14
Romsteri just hadn't got to that yet.13:14
pedjaRomster, OK.13:15
frinnstgood to know. I'm very behind at the moment. Lots of stuff that I lag behind on. sry about that13:33
frinnstwith increased sunlight I should have increased energy once again :)13:34
frinnstbtw any thoughts about the install thingy on the mailinglist?13:34
Romsteri do a bit here and there.13:39
Romsternearly 1am and i work at 9am so i can't cut into my sleep to much13:40
frinnstI usually get more energy to do crux stuff late at night too. then I suffer the day after or the entire week13:46
frinnstwe're not 21 any more :-)14:22
Romsterbut i still get in the zone late at night :D14:23
Romsterbut not tonight i usually do that on fridays when i can afford to stay up late as i don't work weekends.14:25
pedjanight owl? join the club :)14:25
Romsterbut now i am off to bed. because i must sleep else i'll be tired at work all day.14:25
Romsterdon't ever pull a 24 hour no sleep and work the next day it is n't fun14:25
pedjagood night, sleep tight, don' let the monsters that live down under bite14:26
Romsterhah, well i am still alive so far :D laters everyone14:26
pedjathere is no more contrib in ports category drop-down in the flyspray? or was I dreaming it ever been there?14:39
frinnstworks for me14:49
frinnstjust under username14:49
pedjanot that dropdown, under task type14:50
frinnstyou broke it!14:50
pedjadon't have that kind of power :)14:50
pedjaso it's
frinnst doesnt work?14:51
pedjait works14:52
pedjait's nbd reattaching bugs to crux-contrib, but it might be confusing when opening the bug for contrib ports, I think14:53
pedjaspeaking of bugs, is fs#1647 relevant anymore?14:55
ryuo... seriously?14:58
ryuoa security flaw in ed?14:58
ryuowhat's the world coming to? lmao14:58
ryuoOh. patch.14:59
ryuohuh. just occurred to me.15:26
ryuoanother use for pkgutils library could be website integration.15:26
*** onodera has joined #crux-devel16:56
jueI've created a tarball from patch.git HEAD ->
juelooks like there are some more issues than the one listed in FS#164717:27
jueI'd prefer to use that tarball than to bother with many patches, for reference the arch17:29
juewhat do you think?17:30
pedjafine with me. tarball instead of bunch of patches is indeed a cleaner solution :)17:33
pedjaI tried to pick only relevant commits when I opened that bug, fwiv17:36
juewell, if arch is correct, we have two additional CVE issues17:40
jueCVE-2018-6951 and 695217:41
pedjawhen was the upstream last proper release?17:57
pedjaupstream being patch17:57
jue2.7.6 is around a year old,
juefrinnst, jaeger: have you seen the message from Steffen in crux-contrib?18:27
tek__I fixed that half an hour ago21:59
ryuotek__ speaks!22:04
ryuotek__: we were waiting on your feedback on LXC. i believe Romster posted a bug about it.22:04
tek__will check that in a second, doing various stuffs atm22:05
ryuoif you don't have the time, i think Romster was willing to take over that port...22:05
*** onodera has quit IRC22:06
tek__frinnst: ping22:08
ryuotek__: i'm working on joining the crux team in some capacity. my main specialty is programming.22:08
tek__did you consider the important people yet? :-)22:09
ryuoimportant people? i figured that'd be jaeger, frinnst, and jue here. it seems like they're the core team.22:09
tek__a) I'm joking b) exactly!22:10
ryuoit seems to me that CRUX has a serious need for developers that can do more than scripting...22:10
tek__evolvng our toolchain certainly wont hurt :)22:11
ryuomy current project is converting the C++ code of pkgutils into a usable C library.22:11
ryuoand making it more robust in the process. i've identified some flaws in the current software that aren't easy to fix.22:12
ryuothe idea is to allow the guts of pkgutils to be used from scripting languages while retaining the older binary interface.22:12
ryuo(the external commands)22:13
ryuoOr, just expose it for reuse in other C or C++ applications.22:13
tek__sounds nike a nice plan, I like C more than C++, for what it's worth. What's the motivation for using C here?22:13
ryuoWell C is easier to bind in general.22:14
ryuoThat, and i suspect more people will want to tinker with pkgutils if it's in C.22:14
tek__frinnst: created a pull request for httpup on github. Can we try to get jw to merge your curl stuff and the buffer thingy? If he wont react, we should move the new code to crux.nu22:15
ryuosome parts of the library are generic code for data structures i need to write for this library to be manageable22:15
ryuothey won't be needed by the language interfaces but it could simplify other C frontends.22:15
ryuoit made more sense to put them in the library where both can benefit from the generic stuff.22:16
ryuolike, my dynamic array code is here.22:16
tek__which parts of pkgutils are supposed to be transitioned to C? Only the C++ snippets? pkgmk, e.g. is bahs22:16
ryuopkgmk i intend to leave alone.22:17
ryuoit's fine as it is.22:17
ryuothere's no way to really improve it further that i can tell.22:17
ryuonot without a redesign, and that's not practical.22:17
tek__parsing shell-ish Pkgfiles is not fun without a library anyway..22:18
ryuoit's pretty much impossible to do it correctly without a shell22:18
tek__and why bother..22:18
ryuoand the only interface i found to a shell is a lesser known feature of BASH.22:18
ryuoyou can dynamically create builtin commands.22:18
ryuosee help enable22:19
ryuooption -f22:19
ryuoi wrote a bash builtin to allow you to dump shell variables to JSON.22:20
ryuomostly a POC.22:20
ryuoother than that i saw no particular use.22:20
ryuoi thought it could be useful for parsing ports tree22:20
ryuobut it has security risks22:20
ryuoshell can execute arbitrary commands and all.22:20
tek__I rewrote prt-get and parts of pkgutils in C and put it under   jfyi..22:21
ryuoi see.22:22
tek__I figured the same wrt secuirty back then :)22:22
ryuoi'm hoping to finish my library in a few months.22:22
ryuoi found some flaws in pkgutils while i was working...22:22
ryuothe code lacks a design that can completely rollback a failed package upgrade or install.22:22
ryuoit appears to just crash and leave things in an indeterminate state.22:23
ryuoof course, the main time this could occur is in a disk full situation.22:23
ryuoif it's due to hardware failure, there's nothing i can do about that.22:24
ryuomy idea is to keep the old files intact until i have unpacked the entire archive...22:25
ryuowriting new files to some prefixed name that doesn't already exist.22:25
ryuoif i can successfully write them all22:26
ryuothen i can unlink the old files and rename the new ones.22:26
ryuobasically make sure i can reserve enough space for the new files without overwriting the old ones.22:26
tek__you want to do this in place on a per-file basis?22:26
ryuoi have to do it in the real root's directories.22:27
ryuootherwise i risk EXDEV during renaming.22:27
ryuoi can't see another way to retain the old package contents if the upgrade can't be applied for some reason.22:28
tek__I would *way* prefer to the temporary stuff in one place, heck even it was tar'd up before being unpacked to / in one swoop22:29
ryuothere's a flaw with that approach.22:29
ryuowhen the system has multiple mount points for system directories.22:29
ryuorename can't be used across mount points.22:29
ryuoand if i have to copy to move the data, i risk running out of space there.22:30
tek__what a mess :-)22:30
ryuoindeed. the simplest solution seemed to be to write to new files in each directory...22:31
ryuoand only rename once all were successfully written22:31
ryuothe alternative is to rename the originals and unlink them only after the transaction is successful.22:31
ryuotek__: i understand your approach, but i'm trying to use one that's the easiest to work with from C...22:32
ryuoand rename is only guaranteed to work within the same directory when it comes to EXDEV22:32
ryuobecause same directory can't cross mount points.22:33
ryuoif not for my time spent on frugalware i doubt i'd even know this was a problem...22:34
ryuoi ran into EXDEV when i was designing an installer in C.22:34
tek__I get your point22:36
ryuotek__: ok, sorry.22:36
tek__no problem22:36
tek__there is still a timeframe between putting the files into the directories and renaming them22:37
tek__or am I missing something22:37
tek__you cannot queue the rename calls into one batch22:37
ryuono, that's not the issue.22:37
ryuolet me quote from the manpage.22:38
ryuo       EXDEV  oldpath and newpath are not on the same mounted filesystem.22:38
ryuoif i run into this issue because the user has part of the system mounted on more than one filesystem22:38
ryuoi can't guarantee a safe transaction.22:38
ryuoi would need to copy the data across filesystems instead, which may also fail due to space reasons on that filesystem.22:39
ryuoit seems easier to just write it to a tmp file in the final destination instead.22:39
ryuoso rename won't be able to fail for this reason.22:39
ryuobasically i'm trying to take an immutable approach of sorts. retain the old package files until i'm certain the new ones have been written successfully.22:41
ryuouse rename to replace the originals only then, which shouldn't fail at this point because of the precautions i took.22:41
ryuoand since it only works within the same filesystems22:42
ryuothe tmpfiles must be in the final destination dir22:42
ryuoi was planning to use some hidden file prefix so they can be easily purged if i happen to leave some stragglers by accident22:43
tek__yes but we might still have to cope if the renaming of 1..n files is interreupted at e.g. file n-1 how are you rolling back? are you doing something like rename org_file old_org_file; rename new_org_file orgfile  and then an unlink of all old_org_files in the end?22:44
ryuofair enough. rename could be interrupted.22:44
ryuoi guess i hadn't thought of everything.22:45
ryuoOk, so obviously the original files need to be retained long enough to be purged after the final part is committed.22:45
tek__but the approach will cope with that, you can rollback to that.22:45
ryuoyour step includes an extra one i hadn't thought of.22:45
tek__btw.. somethling like a pkgexport $foo before extracting $foo_new would suffice, too :-)22:46
ryuothe need to keep the originals around under another name.22:46
ryuonot necessarily tek__, if you're suggesting removing the original first.22:46
ryuothe new package contents may occupy more space than the original.22:46
ryuothat difference might result in disk space running out.22:47
ryuoso i think what i will do is22:47
ryuorename old files22:47
tek__we can fix this, sure. Yet I wonder how relevant this issue is with current disk sizes (no offense meant!)22:47
ryuotek__: indeed, i'm aware disks are massive. disk exhaustion is rare.22:48
ryuobut since i'm rewriting pkgutils anyway it seems a good idea to revisit the design.22:48
ryuothe aspects that have little impact on the actual behavior people care about.22:49
ryuoi need to understand how pkgutils works internally to replicate it.22:49
ryuoi intend to modify it with algorithmic improvements for...22:49
ryuoprimarily correctness, secondarily speed22:49
ryuopkgutils is mostly IO bound so algorithms improvements won't help a lot.22:50
ryuoi did find some inefficiencies in pkgadd though. ;p22:50
ryuothe way it applies its rules makes it a bit slower.22:50
ryuosome optimization to that made upgrades about 5-10% faster22:50
tek__my pkg implementation above also is faster than prt-get, BTW :-)22:51
ryuoi see. maybe we can have prt-get integrate with my library later.22:51
tek__well we should do parallel xz/bz2/gzip before *all* other things. This shit is single-threaded right now >_>22:51
tek__that's certainly an option22:51
ryuowell, fork+exec isn't cheap either.22:52
ryuoit's the primary performance killer of shell scripts.22:52
tek__within the bounds of of nproc(1)? srsyly?22:52
ryuowhat do you mean?22:52
ryuodo you remember the original revdep?22:53
tek__we call tar -cJvf for packaging up stuff after pkgmk22:53
tek__I do and this is not what I'm after.22:53
ryuowell, it has overhead... in small amounts it's barely noticeable.22:53
tek__un/packing  pkg archives is slow because we only use one CPU core22:53
ryuoyou think i should see if libarchive can use multiple threads?22:54
tek__you do nproc-1 more forks. This is so irrelevant for performance, even more because there is nothing else going on during (un)packing stages22:54
tek__it can through env variables _IIRC_22:54
tek__dont quote me on that22:54
ryuoi see.22:55
ryuowell, that's easy to do with pkgmk22:55
ryuothe rest needs libarchive to support it.22:55
tek__might be an xz feature, though..22:55
ryuoyes, at the cost of more RAM.22:55
ryuolast i checked xz's parallel mode is very RAM hungry.22:56
ryuoit's fairly easy to do for pkgmk. you just swap the compression program.22:56
pedjazstd might be an option22:57
ryuoas long as libarchive can read it22:57
tek__well make it configurable in pkgmk.conf then, it just sucks big time that I have 8 cores and 32GB sitting there and maxing out one core and nothging else22:57
ryuoit has to be... not everything has the luxury of that.22:57
tek__never argued othewhise. atm there is no option at all. I dont have the luxury of waiting that long ;P22:58
pedjazstd is stinking fast in both compressing and decompressing on my potatoputer, fwiv, so it can only get better on modern hw :)22:59
pedjaand libarchive links to it23:00
pedjathat was fairly recent addition, iirc23:00
ryuoseems libarchive is still single threaded23:00
ryuoit's easy to make pkgmk use multiple threads23:01
ryuobut seems hard to do during installation...23:01
pedjaisn't librachive 'just' a wrapper using external libs to do the actual work?23:02
ryuoyes, i guess? i never looked that closely.23:02
pedjanever really looked at how it works23:02
ryuoif we wanted that kind of support, it might be a requirements to use another decompression library23:04
ryuohm. gzip, bzip2, xz, lz23:06
ryuoit might be doable to decompress it ourselves and only use libarchive for the actual TAR.23:07
tek__maybe it alos can pass on stuff the like XZ_OPTS envvar to have multiple threads..23:08
ryuolibxz has no special environment stuff that i can tell23:10
ryuoit seems libarchive has code to use XZ with multiple threads23:13
ryuoonly XZ23:14
ryuoand it's only for the compressor23:14
ryuoi have an idea23:15
ryuobut that's for later.23:15
ryuobut, if we want to make compression able to leverage parallelism...23:16
ryuoboth pkgadd and pkgmk need to support it.23:16
ryuoi'll probably add a flag to specify how many threads pkgadd should use.23:17
ryuobut i'll initially make it single threaded.23:17
ryuoit should be possible if i decouple libarchive from the compressor and write my own compressor.23:17
ryuoprovided someone smarter than me has already solved that part for each algorithm we use23:18
ryuoi see 2 main options23:18
ryuouse an external command like tar does23:18
ryuouse library code.23:18
ryuotek__: i'll explore some options for making pkgadd able to leverage multiple threads for decompression.23:19
tek__sure, tell me if you want input from my input23:19
ryuobut i'll need to see how much of a difference it makes to realworld tests.23:19
ryuoso i'll do a prototype that uses pigz or so23:20
ryuoif that produces results it may be worth implementing.23:20
tek__build libreoffice or even better texlive to see the realworld benefits. and please do it with more than 256MB of RAM23:20
ryuotek__: it's pretty obvious pkgmk can benefit.23:20
ryuopkgadd is a different focus though so i'd need to see if it would benefit.23:21
ryuoi mean... you're extracting files all over the place.23:21
ryuodecompression overhead may not matter.23:21
ryuoit's usually compression that's the slow operation.23:21
tek__give me a few minutes and I am sure to be able to prove you wrong23:22
ryuotek__: on what?23:22
tek__on a different note.. I pulled all IRC logs for this channel23:22
tek__the decompression thingy23:22
tek__what about lxc? I did not see a bug23:22
ryuough. moment.23:22
tek__now I do23:23
tek__wait :-)23:23
tek__stupid flyspray projects.23:23
ryuobasically i updated the LXC port on a CRUX test machine Romster lent me.23:23
ryuobut, you're the contrib maintainer, so can't just go in changing it.23:24
pedjaryuo, off-topic, but if you are curious, this is a project written in Ada
ryuoi had no network connectivity for a moment.23:27
ryuotek__: where does it say that, tek?23:30
*** Workster has joined #crux-devel23:31
ryuoOh, the "production environments"?23:31
tek__say what?23:32
ryuo"what about 3.x (also @ryuo)? Website recommends 2"23:32
ryuoi chose 3.x because it was the latest. i figured CRUX normally doesn't care about LTS stuff.23:33
tek__discard that comment23:34
tek__I really wish the diff was attached instead of pasted ._.23:36
ryuook, i'll try a quick test with a tar archive.23:43
tek__plain tar?23:43
ryuognutar probably. is it a problem?23:43
ryuoi'll try a kernel tarball.23:43
tek__we try to parallelize CPU intense operations. plain tar should max out your CPU too depending on your IO but the real benefit come for cpu-intense (de)compression not plain file appending :)23:44
ryuoi'll start with gzip23:46
ryuotek__: fair enough, but i think the others would want pkgadd to be independent of external programs so any parallelism support is likely to need to be a library solution.23:49
ryuoso that static binary is actually dependable23:50
ryuohm. big difference with pigz.23:50
ryuomuch less of one with decompression23:50
ryuoi'm testing with quad core 16G server on my lan23:51
ryuotek__: i'll see if i can patch pkgmk to support parallel compression, but it'll probably require packages not in core to work.23:53
ryuothat's the easy part.23:53
tek__nice. ping if you want input23:54
ryuoi intend to keep pkgmk as is, so there's no harm in expanding it now.23:54
ryuobut it's going to mean using the --use-compress-program argument.23:55
tek__ btw   :o23:58
ryuointeresting. bsdtar only sees a neglible boost from using pigz...23:58
ryuoerr not neglible23:58
ryuobut it's much smaller than compression23:58
ryuo5.8s to 4.5s23:59
ryuoi suspect it's because compression writes to a single file23:59

Generated by 2.14.0 by Marius Gedminas - find it at!