IRC Logs for #crux-devel Tuesday, 2019-03-19

ryuowhich is very easy to do sequentially00:00
ryuoextraction involves more random IO...00:00
ryuohm. time to test with xz00:02
ryuopedja: i'll see if i can tweak your parallel compression patch. it needs to be an optin optional extra. best chance it has to be added.00:04
ryuobasically, test for the program existence. if it doesn't exist, fallback to conventional methods.00:04
pedjamy patch?00:05
ryuopedja: yea?00:05
ryuohttps://crux.nu/bugs/index.php?do=details&task_id=148400:05
ryuooh.00:05
ryuoit says fun... i thought it said pedja.00:05
ryuoDOH00:05
ryuooh.00:05
ryuoeditted00:05
pedjaheh00:06
pedjayeah, it was not assigned to anyone00:06
pedjaryuo, out of curiosity, how are you testing this?00:08
ryuotesting what?00:08
pedjade/compression00:08
ryuomy server with conventional HDDs.00:08
ryuoi suspect that threaded decompression won't have as much of an impact as it will for compression.00:09
pedjatiming tarball compression/decompression using xz, pigz, etc?00:10
ryuoyes.00:10
ryuobash's builtin time function00:10
ryuono, it's not perfect, but it's a good way to see if there's an obvious benefit.00:11
pedjaI might try that with my cuda package, 1.1gb tar.gz :)00:11
tek__dont forget to drop FS caches before running this :-)00:11
ryuoi'm going to try hacking pkgmk to accept a generic argument for number of compressor processes or threads00:11
ryuoi'll probably need to use a pipeline...00:12
pedjapkgmk/pkgadd take a while with it00:12
ryuodue to how limited bsdtar's support for this us.00:12
ryuois*00:12
ryuoHm.00:35
ryuofun's patch is more in line with using a custom compressor command.00:35
ryuomy idea was to patch pkgmk to take a generic argument and look for an external command matching the chosen compression mode00:36
ryuofalling back to the default method if it doesn't exist.00:36
ryuowow. talk about a kick in the pants.01:32
ryuoxz has no support for threaded decompression, only threaded compression.01:32
ryuoseems pxz offers nothing of value. xz already has a comparable feature.01:33
ryuoi'll just use xz for that mode.01:34
ryuoseems i'm done. testing now.02:17
ryuoOk. seems the patch is working.02:46
ryuothis seems elegant. i hope it's acceptable.02:51
ryuofrinnst, jue, jaeger, tek__: http://dpaste.com/0SDNWK602:54
ryuothis is a patch for an easy way to enable parallel compression support for the built package.02:55
ryuothoughts?02:55
frinnsthow well does that work, anyways?03:37
frinnstnever really played around with parallel compression/decompression. But iirc the implementations are somewhat questionable03:38
frinnsttek__: heh i didnt even remember I had httpup on github. it should be on crux.nu, agree?03:38
ryuofrinnst: it can make a serious difference in how long it takes to create the tarball.03:39
frinnstof course03:40
frinnstfirefox takes ages to compress03:40
ryuoi tried it out with xz's threaded mode.03:40
ryuoit cut the time to a fourth03:40
ryuoi used firefox-bin for a test03:40
ryuofrinnst: i see no harm in having it as an option. i tried to retain the original default so as to not change it for existing setups.03:42
ryuobut, it's possible they have implementation bugs.03:42
ryuothe only parallel option that's in core is xz at present.03:42
ryuofrankly, i don't care much either way. this only makes a major dent in builds that aren't build script heavy.03:43
ryuothe overhead involved in building packages is usually where the bulk of time is spent.03:43
ryuobut i thought i'd propose it since tek__ seems to think it could be a useful improvement.03:44
ryuoi modified it so it won't need the external programs that aren't in core to work. the user will have to install them if they want the support to work.03:44
ryuoi do have my reservations about it though. most compressors aren't parallel for some reason.03:47
ryuothey're too complicated for me to want to try to implement it. i don't even know the first thing about information theory.03:48
frinnstagreed03:51
ryuofrinnst: either way, if you don't think pkgmk should officially support a form of parallelism for compression, i entirely understand.03:52
ryuobut it seemed worth investigating.03:52
ryuoit has potential if nothing else.03:52
ryuobut, it's heavily limited from what i can tell.03:52
ryuothere is no threaded decompression at all for XZ.03:52
ryuoand, the resulting output is larger than a single threaded process.03:53
ryuoi noticed this when i was experimenting.03:53
ryuoit doesn't appear to effect the decompression results...03:54
ryuomaybe it would be better to just incorporate faster algorithms if we want this phase to be faster.03:54
ryuoif this is such a great idea... why aren't other distributions doing it already? hm.03:55
ryuoi'll see if ARCH supports it at all.03:56
ryuook.. ARCH isn't using parallel compression with makepkg.03:58
ryuohm.04:00
frinnstright, so just pushed glib meson build to 3.5. prepare for a fuckton of breakage05:03
ryuofrinnst: is that more or less than a shitton?05:06
frinnstmore05:07
frinnst<jue> frinnst: 32 packages here, which references the glib libtool file ;)05:08
frinnstwow backlog is huuuge05:11
frinnstwent to bed at 7 last night so missed everything05:11
frinnst7pm-3am05:11
ryuotranslation: talk less, work more. got it. ;p05:13
frinnst:>05:14
frinnstgoing to bed early feels soooooooo good05:15
ryuooh, where did we go wrong? frinnst is addicted to... sleep! D:06:41
ryuoyou know, it never occurred to me before but06:44
ryuomeltdown was effectively intel's doping scandal.06:44
ryuothey got caught cheating by taking shortcuts in their implementation.06:45
ryuothen again it's not the first time they've taken shortcuts to seem faster.06:46
ryuoanyone remember when intel's compiler would produce code that performed worse on AMD?06:46
frinnstyeah. not surprised :-)06:54
frinnst"OMG NOT A GENUINE INTEL CPU - TURN OFF EVERYTHING!"06:54
Romstertexlive takes ages to compress07:41
frinnstyeah would be awesome to get it working. but i fear there are issues with most/all multi-threaded compression thingies08:00
frinnstwe need pied piper!08:00
*** frinnst has quit IRC08:09
*** frinnst2 has joined #crux-devel08:12
juefrinnst2: actually, you did it :)09:17
frinnst2:-)09:38
frinnst2but not to 3.4 :-)09:38
*** frinnst has joined #crux-devel11:05
*** frinnst has quit IRC11:05
*** frinnst has joined #crux-devel11:05
*** frinnst2 has quit IRC12:14
*** Workster has quit IRC14:45
*** deus_ex has joined #crux-devel15:05
*** Workster has joined #crux-devel15:07
*** pedja has quit IRC15:07
*** Romster has joined #crux-devel15:20
juefrinnst: gdk-pixbuf fails for me, cause it tries to build the man-pages16:36
juefrinnst: builds with -Dman=false, without the man-pages oc16:43
frinnstah ok cheers18:03
frinnstpushed18:04
frinnsti'll do a complete build to make sure I didnt miss anything18:10
jaegerhas stuff slowed down enough for me to build another ISO? :D18:41
jaegerjust joking, started another bootstrap19:39
ryuointeresting. this adapter to convert the optical drive bay into another hard drive bay works but...20:18
ryuothe bezel won't stay on right.20:21
*** xor29ah has quit IRC22:46
*** heroux has quit IRC22:47
*** heroux has joined #crux-devel22:48
*** xor29ah has joined #crux-devel22:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!