IRC Logs for #crux-devel Saturday, 2013-02-09

juegood morning00:19
juejaeger: sorry, was out yesterday and cannot help here because I've no idea about postfix/mailman00:25
juejaeger: but it's probably best to clean the mail queues and restart postfix00:30
juesome numbers: we had 292 [notify] messages in our 2.8 branch, if we resend all of them we get 47800 messages if we have 164 subscribers, which sounds resonable to me00:53
jue(I have no admin rights for mailmain, so I cannot lookup the number there)00:54
*** mike_k has joined #crux-devel01:13
*** deus_ex has quit IRC02:10
teK_jaeger: sounds right, yes02:14
teK_ was afk yesterdazy02:14
*** deus_ex has joined #crux-devel05:05
pedjajue: I assigned FS#885 to jaeger (bug in nvidia port), altough you are listed as a maintainer.Should I reassign it to you, or?05:36
jaegerto me is fine, I was just waiting for the next nvidia update to change the maintainer05:39
jaegerI have to stop by work for a short time this morning to do a router reboot and some changes. I will look at the mail problem again and nvidia when I get back home05:40
teK_jaeger: did flush the queue?05:43
jaegernot yet, I will do that when I get back05:43
frinnstjue yeah nice05:44
frinnstwill you push the fix? I dont feel like doing anything today. found out an irc-buddy ive known for 15 years died. I'll probably be a bit afk for a while05:45
teK_jaeger: the queue seems to be somewhat empty05:46
teK_% sudo mailq | tail -105:47
teK_postqueue: warning: Mail system is down -- accessing queue directly05:47
teK_-- 645 Kbytes in 40 Requests.05:47
teK_mailman. D:05:47
teK_and I have no idea about the site/adminpw of mailman05:53
jaegerif you have no idea what the site pass is we should reset it, because I don't either. that's easy enough. anyway, you can run /usr/mailman/bin/show_qfiles <queue_dir>/*.pck to see the mails in the various queues06:03
jaegerin, in.oops, retry, etc.06:03
jaegerand yeah, mailman is where they are currently, not postfix06:03
jaegerso if we restarted postfix now without clearing mailman we'll get a lot more messages06:03
jaegerfrinnst: sorry to hear that. :(06:03
jaegerLeaving for work. I might log in from there but more likely when I get back home06:04
teK_yeah I inspected the queue06:10
*** horrorStruck has quit IRC07:13
*** horrorStruck has joined #crux-devel07:19
jaegerok, back home now09:27
juehi jaeger09:48
jueI'm still puzzled about the reason for the mess09:50
jaegerMe too, though for now I just want to get mail working again09:51
jaegerI think it had to be caused by something errant in the ports/opt.git update hook09:51
jueyep, that's most important09:51
jaegerOnce mailman is cleaned out, any objections to me disabling the update hook for now?09:51
jueno, should give us a better feeling :)09:52
jaegeralright, I've scripted a check for the queues and it's running but it takes a long time10:05
jaegereach message is in a .pck (pickled) file so has to be examined by show_qfiles and grep10:05
jaegerI'm removing any message with "Subject: ports/opt (master):" in it10:06
jaegerI need to leave again to help my mom and brother with something but I'll finish this up later. It's going to take quite a while to run anyway10:17
jaegerIf you want to watch its progress tail -f /tmp/*-clean.log10:20
teK_great jaeger11:49
*** mike_k has quit IRC14:17
jaegerok, back now. sorry I've been out so much today14:18
jaegerso my script cleaned only 1588 mails out of the mailman queues, there are still a LOT left14:19
jaegerbut they don't seem to be the master ones14:19
jaegerchecking subjects now to see if any are useful. otherwise I might throw them all out14:22
jaegerlots of them are obvious spam14:22
jaegerMail is re-enabled now14:38
jaegerI'll clear out the moderated ports/opt (master) messages, too14:40
teK_I don't really speak perl so I cannot review hooks/updates..14:41
jaegerI can read enough to know what the script does, the problem is I want to see what args it got... I'm going to probably rewrite the hook in bash or python if nobody ejects14:41
jaegerer... objects14:41
jaegercurse you, fingers!14:41
teK_now only roelof is left spamming our list :P14:41
jaegerI'm going to change the mailman site pass, too14:42
jaegersince none of us knows it14:42
teK_it'd be nice to get hold of it :)14:42
jaegerI'll send it to you in a min, gonna generate a random one14:42
teK_sure, no hurry14:43
teK_Looking if there is a problem with hotmail.14:48
jaegeryeah... :/14:48
jaegerdisabled the update hook for now14:48
jaegerthe hook seems to want to send to both crux-commits@ and crux@ (if there's a [notify] tag)14:49
jaegerthe crux-commits archives are empty since august 200814:49
teK_uhm. strange..14:50
jaegeryet another issue to diagnose, hehe14:50
jaeger <-- here is the hook script14:52
jaegerit should be called like so: update <ref> <old_sha1> <new_sha1>14:52
jaegerMy guess is that sip wrote the original version, I think he was a perl guy14:53
jaegeranyway, it finds the branch, old hash, new hash, and working dir14:53
jaegerit runs 'git merge-base <old_sha1> <new_sha1>'14:54
jaegerI guess that finds the best common ancestor14:54
teK_according to the man page it does *g*14:54
jaegerthen runs 'git rev-list --pretty=oneline <new_sha1> <common_ancestor_sha1>'14:55
teK_maybe the script got called with a really old old_sha1 and a current new_sha114:55
jaegerthat gets the subject lines14:55
jaegermaybe, but the weirdest part is that it somehow got called with 'master' as the ref14:55
jaegerit splits the commit subjects (log lines) and iterates through them... it sends to regardless but adds if there's a [notify] tag14:56
jaegerthen it calls 'git log -p -n 1 <sha1_of_current_commit> | mail -r $from -s $subject $to'14:57
jaegerfeel free to correct me if I'm reading any of that wrong. it's a fairly simple chain of events, the input is what I wish I had now14:58
jaegerI'll add some logging to the rewrite if I do that14:58
jaegerooh, here's a thought14:58
jaegerthe update hook's exit status determines whether or not the push fails14:59
teK_maybe there's a template on the interwebs how to do it 'properly' it's not an exotic task I guess14:59
jaegerso it's possible the mails could have all gone through before the push failed, which is why we don't see anything in master, maybe?14:59
jaegerwild guess, I haven't put a lot of thought into that yet14:59
jaeger <-- reference14:59
jaegerpost-update might be a better place for this15:00
jaegerhrmm, I wonder which I used for my rsync regeneration15:00
teK_I think15:00
jaegeryep, I used post-update15:00
jaegerwhich is what we use for the rsync regen on as well15:01
teK_wrt exit status: I still try to understand what you wrote15:01
jaegerok, let me try to rephrase it15:01
teK_yeah It ouched that for switching from 2.8 to 3.0 branches15:01
jaeger1) I do a local push15:01
jaeger2) update hook runs, sends emails, exits with non-zero status15:01
jaeger3) git-receive-pack does not run because of non-zero status15:02
jaegerthus master is not update15:02
jaegerdoes that sound like a possibility?15:02
jaegerif we used post-update instead, it's run AFTER git-receive-pack succeeds15:02
teK_yes but I don't see the link to (re?)sending all the old notify-cruft we sawq15:02
jaegerwell, the notifies were all 'ports/opt (master)' instead of 'ports/opt (3.0)', I wondered if someone had merged 3.0 into master15:03
teK_it wasn't me :D15:04
jaegerI think it was sepen but it's ok, I still think he's a decent guy :D15:04
teK_but yeah15:05
teK_sounds reasonable15:05
teK_nice debugging skills15:05
jaegerwell, I'm not 100% sure but I *think* this might be what happened15:05
jaegerif I had the input args to the update hook at the time it ran I might know for sure15:05
teK_yeah at least $ref = master is just not right15:06
jaegerwith all that said I think the update hook is also where the problem with the revert emails happens15:06
jaegerA nice thing we could do in the update hook is fail if ref == master15:06
teK_so you commited the rsync hook?15:06
jaegerI haven't changed anything yet, just disabled the update hook15:06
jaegerthe 'post-update' hook is still active, which updates the rsync repos15:06
teK_no.. did you write the rsync-related hook?15:07
jaegerwe just have no email notifications right now15:07
jaegerah, sorry, didn't get what you meant. I did not write it, no15:07
jaegerI used a copy of it on morpheus.net15:07
teK_hm so the questin is..15:07
teK_DAMN IT15:07
jaegerthe rsync update one is simple and not prone to breakage, though15:07
teK_the question is.. why did sip(?) use post-update and update15:07
jaegerI'm not sure, honestly15:08
teK_i.e. was there a reason not to put the notify-stuff into post-update first place15:08
jaegergithooks(5) lists when they all run and some of the pros/cons of each15:08
jaegerit's also possible that that has changed with a new git release since they were created15:08
jaegerI imagine either a) there was a reason then but maybe not now, or b) he (or whoever) didn't know better :)15:08
teK_all in all I don't see a reason why we should not notify the ML (only) after everything git-related worked out15:08
jaegercould just be an honest mistake15:08
teK_it would be superawesome if you a) put failure into update if ref == master and b) move the notification to post-update :)15:10
jaegerthe current discussion (complaining :D) in #crux also brings up something we could do if we wanted - strip out giant footprints from the email and send a link to a URL instead or something like that15:10
jaegeryeah, I think moving it is a good idea and the master check as well, will start with that and we can add features later if needed.15:10
jaegerleft my git book at work, doh15:11
jaegerfortunately it's all online :D15:12
jaegerI like its description of the server-side hooks better than githooks(5)15:13
jaeger(at the very bottom)15:13
teK_well the texlive-mail got rejected for it's size anyway :D15:13
jaegeryeah, I had to manually approve that one15:13
teK_hmhm =)15:13
jaegerI cleared a bunch of other master ones from the ML moderation at the time15:13
jaegerI like the idea of using "update" to check for the master ref15:13
jaegerbecause if you push updates to master and 3.0, for example, only master fails15:13
jaegerthen "post-receive" for the email notification15:14
jaegerthough that raises another question15:14
jaegerif you push master and 3.0 and master fails, does update still send all the emails for both?15:14
jaegerer... post-receive15:14
jaegerI guess there's an easy way to solve that problem. just make the script exit non-zero if master is pushed, BEFORE sending any emails15:15
teK_well it says only pre-receive runs once15:15
teK_it's run only once I suppose15:16
jaegerright, I was just thinking about using post-receive and update, not pre-receive... rethinking now15:16
teK_is pre-receive. It takes a list of references that are being pushed from stdin15:16
teK_same goes for post-receive15:17
jaegermaybe the best course would be this:15:17
jaeger1) use pre-receive to block master updates15:17
jaeger2) use post-receive to email15:17
jaegerif 1) fails, 2) doesn't happen15:17
jaegerWe would have to disable it if we ever wanted to actually USE master but we haven't used it in so long I don't see that being a problem, really15:18
teK_I don't see a use-case for it either15:19
teK_so if there's a push to 3.0 AND master you just let it fail completely in pre-receive15:19
jaegeryes, that's what I'd prefer to do15:19
jaegerforce the user to fix his/her commit first15:19
teK_should work if the git book is right ;015:19
jaegerI will test it on a local server first :)15:19
teK_yeah, maybe we should add a explicit hint in the handbook / contribhowto that deals with git15:20
jaegerMaybe, though it might confuse non-contributing users15:20
jaegerEither way I'm going to start some testing here and write a new notify script15:21
teK_I mean this page:
jaegerI need to reinstall one of my crux installs here, that will be a perfect test :)15:21
teK_non contributing users are not suppsoed to read that anyway :D15:21
jaegerah, yes, I agree completely that we can add more to the wiki :)15:21
jaegerjust thought adding it to the main handbook would be confusing15:21
teK_just as a reference.15:21
jaegersince not all users will contribute15:21
jaegerI still wonder about the crux-commits list, that's kinda odd15:23
teK_btw.. what's the maximum net bandwidth utilisation you have seen for a gigabit link via for example ftp?15:23
jaegerWould you mind looking at the admin page for that and see if you see anything obvious, like a disabled switch?15:23
jaegerI've seen about 700 megabit/sec max15:23
teK_yes, maybe it's a mailman thing and everything sendt to this list jsut gets dumped15:23
teK_I'm asking because I get ~100MiB/s15:24
jaegerwell, it's been a long time since I did raw speed tests. I could test again if you like15:25
teK_but this is with jumboframes so tcp/ip header overhead should have much less impact as it has for 100megabit links without jumbo frames15:25
jaegerah, yes, jumbo frames would help15:25
teK_just wondering :)15:25
teK_yeah but I get 10MiB/s for 100megabit and 100MiB/s for gigabit15:25
jaegerI just enabled jumbo frames at work today for my 10GbE storage devices, that's why I was at work this morning :)15:25
teK_performance is just shitty for anything above or at gigabit level if you don't have jumboframes15:26
teK_just wondering why there's the same percentage of overhead for both links15:26
jaegerwell, depends on the data flowing... even with jumbo frames if you send a ton of small packets it sucks :)15:27
teK_I transfer 60GiB files over FTP15:28
teK_i.e. the easiest work load with continous reading/writing to disk15:28
jaegerI had to send 16GiB to an FTP server in Switzerland last week at 280kiB/s :)15:29
teK_not so nice. They are oracle data files :O15:29
jaegertook a while15:29
jaegerwell, it's nice that you can send large files that will take advantage of jumbo frames15:29
teK_just use rapid share, they operate from switzerland, iirc :P15:29
teK_btw.. I asked you once about why scp would transfer some files at 6MiB/s and others with 10MiB/s15:30
teK_it's the same work/payload but transfered to another backup server15:30
teK_just for the record15:30
jaegerI sorta remember that, was odd15:30
teK_lesson learned: just don't go with what's on the colorful brochures *g*15:32
frinnstso, somebody merged 3.0 to master and pushed?17:02
jaegerthat's my guess17:02
frinnstok, everything fixed now? anything i can do?17:03
jaegermailman is sorted out, git never seemed to be messed up, really (update failed?), so it's probably ok right now, as far as I can tell17:03
jaegerI still plan to rewrite the update hook but things are working17:03
jaegerthank you, though :)17:03
jaegerthat's what pre-receive's input looks like18:48
jaegerI created an empty repo with an empty 3.0 branch and pulled git:// 3.0 into it18:48
jaegerall the hook does is print out its input and exit 1, so the push fails18:48
jaegerTo git@daedalus.intranet:jaeger.git ! [remote rejected] 3.0 -> 3.0 (pre-receive hook declined)18:49
jaegererror: failed to push some refs to 'git@daedalus.intranet:jaeger.git'18:49
jaegerfor reference18:49
jaegerwill test with post-receive and update as well18:49
jaeger <-- the update hook's args18:51
jaegerpre-receive and post-receive get refs on stdin, update gets arguments18:52
jaegerhere's an interesting tidbit: git merge-base 000000...bb6221 fails because 000000 isn't a valid hash18:56
jaegerremote: fatal: Not a valid commit name 000000000000000000000000000000000000000018:58
jaegerso this command in the perl script would have failed: remote: fatal: Not a valid commit name 000000000000000000000000000000000000000019:00
jaegerarg, this: my $base = `git merge-base $old_hash $new_hash`;19:00
jaegerI really wish I could see the arguments it got when it went haywire, it was obviously different from my test19:01
jaegerah, figured out the problem there, I was being dumb and pushing to 3.0 instead of to master19:15
jaegermore testing ahead19:15
jaegerwould have looked more like this:19:16
jaegerrefs/heads/master 50d93bf14a537ac09809a60936ad3860812b6069 2d177c803b733a2a150aee5dc44131ffaf3e93b319:16
jaeger$ git merge-base 50d93b 2d177c19:16
jaeger$ git rev-list --pretty=oneline 2d177c ^50d93b | wc -l19:16
jaegerboom, 7.3k emails19:16
jaegerthat number doesn't match up but I think I'm on the right track with what happened, at least19:17
jaegerah ha, the mail manpage explains why the reverts cause bounces19:21
jaeger       -s subject19:21
jaeger              Specify subject on command line (only the first argument after the -s flag is used as a subject; be careful to quote subjects19:21
jaeger              containing spaces).19:21
jaegerThe reverts have spaces in them that aren't getting escaped properly so parts of the subject are being fed to mail as recipients19:21
*** mavrick61 has joined #crux-devel19:32
jaeger <--- a new one with all the commit oneliners, an email would be fired off for each of these19:32
jaegerI think I've spammed enough for tonight, will continue digging and rewriting tomorrow19:40
jaegers/spaces/quotes & spaces/ from earlier19:40
jaegerteK_: I think the reason for using the "update" hook is so that you can send an individual email for each commit you pushed instead of a single email for all of them19:57
jaegerbad in the case of a merge of 3.0 into master, good if you want to push 5 commits and only 2 of them are [notify] commits19:58
jaegerone last thing: $ grep "\[notify\]" /tmp/jaeger-update.log | wc -l20:02
jaegerthat number does match up20:02

Generated by 2.11.0 by Marius Gedminas - find it at!