LLVM IRC channel flooded?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

LLVM IRC channel flooded?

Renato Golin-2
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: LLVM IRC channel flooded?

David Blaikie


On Tue, May 19, 2015 at 7:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings,

NOC?
 
but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

What distinction are you drawing there? The difference between freshly red and previously red?
 
For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

What does this do differently from the main buildbot page? (other than only show arm bots?) Is it something we could do to the buildbot page (remove always-red builders, recategorize flaky/problematic builders so at least they're off in the "experimental" section, etc)?
 
But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

James Y Knight-2
In reply to this post by Renato Golin-2
Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care.

While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems.

Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster.

E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary.

Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure.


If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement.

On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Yaron Keren
+1 for hiding flaky bots. I routinely see some bot randomally failing after a non-related commit.  
sanitizer-x86_64-linux may be the worst one. This wastes time and hides real problems.

2015-05-19 20:40 GMT+03:00 James Y Knight <[hidden email]>:
Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care.

While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems.

Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster.

E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary.

Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure.


If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement.

On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: LLVM IRC channel flooded?

Renato Golin-2
In reply to this post by David Blaikie
On 19 May 2015 at 18:39, David Blaikie <[hidden email]> wrote:
> NOC?

Sorry, NOC is "network operations centre". the room with big screens
showing the status of a data centre, where operators sit and fix
problems, always looking at the big screens on the wall, in case they
go red.


> What distinction are you drawing there? The difference between freshly red
> and previously red?

Basically, yes.


> What does this do differently from the main buildbot page? (other than only
> show arm bots?) Is it something we could do to the buildbot page (remove
> always-red builders, recategorize flaky/problematic builders so at least
> they're off in the "experimental" section, etc)?

Separating ARM from the rest is the most important thing to me. but
classifying them and only showing the information I want is also
important.

James Knight has summarised well the problems I have with the current
buildbot page.

cheers,
--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Chris Matthews
In reply to this post by James Y Knight-2
When we built green dragon we tried to be really accountable for this sort of cruft, with a goal of 99% useful notifications, or nothing.

On green dragon we curate both which builds notify the IRC and which builds show up on the main page.  Anything that fails for reasons unrelated to the commit is not allowed to do either. We use the phased build approach to make sure we notify only once per failure.  Builds that are red for more than a week are disabled, if we can’t fix it in a week, its not worth building anymore.   Because of that, libcxx builds, LLDB and performance builds do not notify and some are disabled.  When we email the blamelist, I am CCed on every email, and they are not filtered from my inbox.  If the blame list is long, it only emails me, and I track down who broke it.

Of course green dragon only runs a small proportion of the total builds.  

If you can’t look at the build page, and know that everything that is red is a real problem, we have a real problem.  Even within builds, if most of the steps are marked as failures, you don’t know what when wrong.

On May 19, 2015, at 10:40 AM, James Y Knight <[hidden email]> wrote:

Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care.

While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems.

Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster.

E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary.

Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure.


If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement.

On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

David Blaikie
In reply to this post by James Y Knight-2


On Tue, May 19, 2015 at 10:40 AM, James Y Knight <[hidden email]> wrote:
Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care.

While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems.

Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster.

E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary.

Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure.

Whenever you get crappy fail-mail, please forward it to llvm-dev, cc'ing the bot owner and request the issue be addressed or the bot be removed. Yeah, I know it's not an ideal process, but it's something to keep issues visible/pushed on.

But, yes, having some more formal process to deal with this sort of thing would be nice (I can imagine some process along the lines of "bots start in experimental and need a track record of low flake/false positive results for some period of time before being promoted out of experimental so they can send mail to blame lists and IRC, etc" coupled with some mechanism for demoting a buildbot back into experimental if it starts behaving poorly)

- David
 


If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement.

On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: LLVM IRC channel flooded?

Matthias Braun-2
In reply to this post by Renato Golin-2
+1 I completely agree with everything said in this thread. I wish I had a  “this error had nothing to do with the commits” link in the bot mails and if a certain amount of people clicked that link the bot would be stopped from sending mails or spamming the IRC channel…

I know setting up buildbots and keeping them running in a stable fashion is a hard task (I have done that for other projects) and we have to thank the people doing that ungrateful job, but wasting everyones time with too many false positives isn’t good either.

- Matthias

> On May 19, 2015, at 11:18 AM, Renato Golin <[hidden email]> wrote:
>
> On 19 May 2015 at 18:39, David Blaikie <[hidden email]> wrote:
>> NOC?
>
> Sorry, NOC is "network operations centre". the room with big screens
> showing the status of a data centre, where operators sit and fix
> problems, always looking at the big screens on the wall, in case they
> go red.
>
>
>> What distinction are you drawing there? The difference between freshly red
>> and previously red?
>
> Basically, yes.
>
>
>> What does this do differently from the main buildbot page? (other than only
>> show arm bots?) Is it something we could do to the buildbot page (remove
>> always-red builders, recategorize flaky/problematic builders so at least
>> they're off in the "experimental" section, etc)?
>
> Separating ARM from the rest is the most important thing to me. but
> classifying them and only showing the information I want is also
> important.
>
> James Knight has summarised well the problems I have with the current
> buildbot page.
>
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Chris Matthews
In reply to this post by David Blaikie
Just some stats, after looking through lab.llvm.org:8011

Maybe these should be marked as experimental, and removed from the builders link on the main page.

Never passed at all:
libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03
libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan
libcxx-libcxxabi-x86_64-linux-ubuntu-tsan
libcxx-libcxxabi-x86_64-linux-ubuntu-gcc
libcxx-libcxxabi-x86_64-apple-darwin14-system-lib
lldb-x86_64-ubuntu-14.04-android
llgo-x86_64-linux

Not pass in at least a month:

llvm-clang-lld-x86_64-debian-fast
clang-native-mingw32-win7
clang-x86_64-linux-selfhost-abi-test
clang-x64-ninja-win7-debug
perf-x86_64-penryn-O3-polly-detect-only
sanitizer-x86_64-linux-bootstrap
sanitizer_x86_64-freebsd
sanitizer-windows
libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang
clang-amd64-openbsd
lldb-x86_64-debian-clang
lldb-x86_64-freebsd
lldb-x86_64-ubuntu-14.10


On May 19, 2015, at 11:32 AM, David Blaikie <[hidden email]> wrote:



On Tue, May 19, 2015 at 10:40 AM, James Y Knight <[hidden email]> wrote:
Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care.

While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems.

Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster.

E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary.

Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure.

Whenever you get crappy fail-mail, please forward it to llvm-dev, cc'ing the bot owner and request the issue be addressed or the bot be removed. Yeah, I know it's not an ideal process, but it's something to keep issues visible/pushed on.

But, yes, having some more formal process to deal with this sort of thing would be nice (I can imagine some process along the lines of "bots start in experimental and need a track record of low flake/false positive results for some period of time before being promoted out of experimental so they can send mail to blame lists and IRC, etc" coupled with some mechanism for demoting a buildbot back into experimental if it starts behaving poorly)

- David
 


If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement.

On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Tobias Grosser-5
On 05/19/2015 08:50 PM, Chris Matthews wrote:

> Just some stats, after looking through lab.llvm.org
> <http://lab.llvm.org>:8011
>
> Maybe these should be marked as experimental, and removed from the
> builders link on the main page.
>
> Never passed at all:
> libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03
> libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan
> libcxx-libcxxabi-x86_64-linux-ubuntu-tsan
> libcxx-libcxxabi-x86_64-linux-ubuntu-gcc
> libcxx-libcxxabi-x86_64-apple-darwin14-system-lib
> lldb-x86_64-ubuntu-14.04-android
> llgo-x86_64-linux
>
> Not pass in at least a month:
>
> llvm-clang-lld-x86_64-debian-fast
> clang-native-mingw32-win7
> clang-x86_64-linux-selfhost-abi-test
> clang-x64-ninja-win7-debug
> perf-x86_64-penryn-O3-polly-detect-only
> sanitizer-x86_64-linux-bootstrap
> sanitizer_x86_64-freebsd
> sanitizer-windows
> libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang
> clang-amd64-openbsd
> lldb-x86_64-debian-clang
> lldb-x86_64-freebsd
> lldb-x86_64-ubuntu-14.10

Chris, thanks for going through this!

I am all in favor of removing/disabling these bots (and would be OK with
being even more aggressive).

The one Polly bot listed is a performance buildbot which has emails or
IRC messages disabled. I now removed it completely from the buildbot
list to also keep the web interface clean.

Best,
Tobias




_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

David Blaikie


On Tue, May 19, 2015 at 12:25 PM, Tobias Grosser <[hidden email]> wrote:
On 05/19/2015 08:50 PM, Chris Matthews wrote:
Just some stats, after looking through lab.llvm.org
<http://lab.llvm.org>:8011

Maybe these should be marked as experimental, and removed from the
builders link on the main page.

Never passed at all:
libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03
libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan
libcxx-libcxxabi-x86_64-linux-ubuntu-tsan
libcxx-libcxxabi-x86_64-linux-ubuntu-gcc
libcxx-libcxxabi-x86_64-apple-darwin14-system-lib
lldb-x86_64-ubuntu-14.04-android
llgo-x86_64-linux

Not pass in at least a month:

llvm-clang-lld-x86_64-debian-fast
clang-native-mingw32-win7
clang-x86_64-linux-selfhost-abi-test
clang-x64-ninja-win7-debug
perf-x86_64-penryn-O3-polly-detect-only
sanitizer-x86_64-linux-bootstrap
sanitizer_x86_64-freebsd
sanitizer-windows
libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang
clang-amd64-openbsd
lldb-x86_64-debian-clang
lldb-x86_64-freebsd
lldb-x86_64-ubuntu-14.10

Chris, thanks for going through this!

Yep, totally - is there an easy way to gather this data on an ongoing basis? (perhaps every month or so?)
 
I am all in favor of removing/disabling these bots (and would be OK with being even more aggressive).

Yep, agreed - I'd say removing the buildbots that've never passed and at least moving the ones that haven't passed recently to "experimental" is a totally reasonable approach.
 

The one Polly bot listed is a performance buildbot which has emails or IRC messages disabled. I now removed it completely from the buildbot list to also keep the web interface clean.

Best,
Tobias






_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Reid Kleckner-2
In reply to this post by Chris Matthews
On Tue, May 19, 2015 at 11:50 AM, Chris Matthews <[hidden email]> wrote:
sanitizer-windows

So, I tried to track down what went wrong here, and the oldest build I can find is:

This raises a different problem: the buildmaster doesn't hold onto enough logs. That build is from five days ago, and already I can't find the relevant blamelist causing the breakage. =/

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Renato Golin-2
In reply to this post by David Blaikie
On 19 May 2015 at 20:44, David Blaikie <[hidden email]> wrote:
>> I am all in favor of removing/disabling these bots (and would be OK with
>> being even more aggressive).
>
> Yep, agreed - I'd say removing the buildbots that've never passed and at
> least moving the ones that haven't passed recently to "experimental" is a
> totally reasonable approach.

+1!

--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Renato Golin-2
In reply to this post by Chris Matthews
On 19 May 2015 at 19:50, Chris Matthews <[hidden email]> wrote:
> Maybe these should be marked as experimental, and removed from the builders
> link on the main page.

Right now, I have two "failing" LNT bots. One of them is a known LNT
server instability, and I brought the bot down myself. If you shut
down the bot gracefully, no one gets an email, so if you fix it and
bring it back, no one gets annoyed.

If the bot owners are not willing to do that kind of management, or
are unresponsive, we should take the bots out of the "official" list
and not report anything from them. If anyone wants to put a bot up and
not care about it, they can also put up their own buildmaster, so that
we don't have to mix lost bots with production bots. No emails, no
reds on the production page.

cheers,
--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

David Blaikie


On Tue, May 19, 2015 at 1:13 PM, Renato Golin <[hidden email]> wrote:
On 19 May 2015 at 19:50, Chris Matthews <[hidden email]> wrote:
> Maybe these should be marked as experimental, and removed from the builders
> link on the main page.

Right now, I have two "failing" LNT bots. One of them is a known LNT
server instability, and I brought the bot down myself. If you shut
down the bot gracefully, no one gets an email, so if you fix it and
bring it back, no one gets annoyed.

If the bot owners are not willing to do that kind of management, or
are unresponsive, we should take the bots out of the "official" list
and not report anything from them. If anyone wants to put a bot up and
not care about it, they can also put up their own buildmaster, so that
we don't have to mix lost bots with production bots. No emails, no
reds on the production page.

Yep, having a limited lifetime in the "experimental" category would be useful too - if you're not working to get it out of there, just remove it entirely.
 

cheers,
--renato


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: LLVM IRC channel flooded?

Diego Novillo-3
In reply to this post by Renato Golin-2
On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
> Folks,
>
> I know it's a reasonably valuable thing to have the buildbot IRC bot
> publishing results, but the channel is kind of flooded with the
> messages, and the more bots we put up, the worse it will be.

I agree. It's very hard to keep track of real conversations some time.

I would prefer to have a known central web page where bots publish
their status instead of spamming a conversation channel. It serves no
useful purpose.


Diego.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Andrew Wilkins
In reply to this post by Chris Matthews
On Wed, 20 May 2015 at 02:55 Chris Matthews <[hidden email]> wrote:
Just some stats, after looking through lab.llvm.org:8011

Maybe these should be marked as experimental, and removed from the builders link on the main page.

Never passed at all:
libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03
libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan
libcxx-libcxxabi-x86_64-linux-ubuntu-tsan
libcxx-libcxxabi-x86_64-linux-ubuntu-gcc
libcxx-libcxxabi-x86_64-apple-darwin14-system-lib
lldb-x86_64-ubuntu-14.04-android
llgo-x86_64-linux

Hi,

llgo-x86_64-linux is mine. Sorry, I had disabled the slave agent to avoid spurious emails, but hadn't considered its impact on the status pages. I'm fine with disabling it altogether for now; I'm waiting on a fix to Ninja to be merged.

Cheers,
Andrew
 
Not pass in at least a month:

llvm-clang-lld-x86_64-debian-fast
clang-native-mingw32-win7
clang-x86_64-linux-selfhost-abi-test
clang-x64-ninja-win7-debug
perf-x86_64-penryn-O3-polly-detect-only
sanitizer-x86_64-linux-bootstrap
sanitizer_x86_64-freebsd
sanitizer-windows
libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang
clang-amd64-openbsd
lldb-x86_64-debian-clang
lldb-x86_64-freebsd
lldb-x86_64-ubuntu-14.10


On May 19, 2015, at 11:32 AM, David Blaikie <[hidden email]> wrote:



On Tue, May 19, 2015 at 10:40 AM, James Y Knight <[hidden email]> wrote:
Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care.

While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems.

Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster.

E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary.

Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure.

Whenever you get crappy fail-mail, please forward it to llvm-dev, cc'ing the bot owner and request the issue be addressed or the bot be removed. Yeah, I know it's not an ideal process, but it's something to keep issues visible/pushed on.

But, yes, having some more formal process to deal with this sort of thing would be nice (I can imagine some process along the lines of "bots start in experimental and need a track record of low flake/false positive results for some period of time before being promoted out of experimental so they can send mail to blame lists and IRC, etc" coupled with some mechanism for demoting a buildbot back into experimental if it starts behaving poorly)

- David
 


If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement.

On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Eric Fiselier
>Maybe these should be marked as experimental, and removed from the builders link on the main page.
>
>Never passed at all:
>libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03
>libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan
>libcxx-libcxxabi-x86_64-linux-ubuntu-tsan
>libcxx-libcxxabi-x86_64-linux-ubuntu-gcc
>libcxx-libcxxabi-x86_64-apple-darwin14-system-lib

Sorry about these bots. When I originally put them up I had hoped to
deal with the failures in short order. However that hasn't happened.

I'm happy to move these bots to an experimental section and off the main page.

/Eric


On Tue, May 19, 2015 at 6:55 PM, Andrew Wilkins <[hidden email]> wrote:

> On Wed, 20 May 2015 at 02:55 Chris Matthews <[hidden email]>
> wrote:
>>
>> Just some stats, after looking through lab.llvm.org:8011
>>
>> Maybe these should be marked as experimental, and removed from the
>> builders link on the main page.
>>
>> Never passed at all:
>> libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03
>> libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan
>> libcxx-libcxxabi-x86_64-linux-ubuntu-tsan
>> libcxx-libcxxabi-x86_64-linux-ubuntu-gcc
>> libcxx-libcxxabi-x86_64-apple-darwin14-system-lib
>> lldb-x86_64-ubuntu-14.04-android
>> llgo-x86_64-linux
>
>
> Hi,
>
> llgo-x86_64-linux is mine. Sorry, I had disabled the slave agent to avoid
> spurious emails, but hadn't considered its impact on the status pages. I'm
> fine with disabling it altogether for now; I'm waiting on a fix to Ninja to
> be merged.
>
> Cheers,
> Andrew
>
>>
>> Not pass in at least a month:
>>
>> llvm-clang-lld-x86_64-debian-fast
>> clang-native-mingw32-win7
>> clang-x86_64-linux-selfhost-abi-test
>> clang-x64-ninja-win7-debug
>> perf-x86_64-penryn-O3-polly-detect-only
>> sanitizer-x86_64-linux-bootstrap
>> sanitizer_x86_64-freebsd
>> sanitizer-windows
>> libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang
>> clang-amd64-openbsd
>> lldb-x86_64-debian-clang
>> lldb-x86_64-freebsd
>> lldb-x86_64-ubuntu-14.10
>>
>>
>> On May 19, 2015, at 11:32 AM, David Blaikie <[hidden email]> wrote:
>>
>>
>>
>> On Tue, May 19, 2015 at 10:40 AM, James Y Knight <[hidden email]>
>> wrote:
>>>
>>> Yes, I also find the amount of bot spam in #llvm is basically
>>> intolerable. It makes it difficult to see actual people talking. At first, I
>>> just put all the bots on /ignore. Now I have an xchat script to move the
>>> botspam to another tab (tabify-004.pl). I'd recommend that the bots should
>>> just be moved to #llvm-bots and fix the problem for everyone. Those who are
>>> committing changes can join that channel, too, and others don't care.
>>>
>>> While we're on this subject, I also find the official buildbot page
>>> (lab.llvm.org:8011) almost unusable, since so many columns are either always
>>> red, or else are so flaky that they basically randomly alternate between
>>> passing and failing. So, at a glance, it's impossible to tell whether the
>>> current state of the tree is good. (I certainly haven't memorized which ones
>>> are "supposed" to be red, and which are not. Maybe others have). Having
>>> flaky and always-failing builds show up on the buildbot pages, and notifying
>>> IRC, really has negative utility, since it not only is not providing useful
>>> information, but is serving to obscure the actual important failures, and
>>> causing people to spend time investigating non-problems.
>>>
>>> Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page
>>> instead, which was a great recommendation -- that page shows problems much
>>> more clearly. But it's unfortunate that there *needs* to be a separate "sane
>>> builders only" buildmaster.
>>>
>>> E.g. (and not to pick on this particular bot, this is just one example of
>>> many):
>>> http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 --
>>> passed, while the previous failed. But, it's not caused by the commit, it's
>>> just arbitrary.
>>>
>>> Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this
>>> bot failed?
>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" --
>>> answer: because it's randomly broken. Wasted the questioner's time trying to
>>> investigate the failure.
>>
>>
>> Whenever you get crappy fail-mail, please forward it to llvm-dev, cc'ing
>> the bot owner and request the issue be addressed or the bot be removed.
>> Yeah, I know it's not an ideal process, but it's something to keep issues
>> visible/pushed on.
>>
>> But, yes, having some more formal process to deal with this sort of thing
>> would be nice (I can imagine some process along the lines of "bots start in
>> experimental and need a track record of low flake/false positive results for
>> some period of time before being promoted out of experimental so they can
>> send mail to blame lists and IRC, etc" coupled with some mechanism for
>> demoting a buildbot back into experimental if it starts behaving poorly)
>>
>> - David
>>
>>>
>>>
>>>
>>> If all the flaky or always-broken builder configurations got hidden from
>>> the main pages of buildbot, and stopped sending emails/IRC notifications to
>>> anyone but their "owner", that would be a substantial improvement.
>>>
>>> On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]>
>>> wrote:
>>>>
>>>> Folks,
>>>>
>>>> I know it's a reasonably valuable thing to have the buildbot IRC bot
>>>> publishing results, but the channel is kind of flooded with the
>>>> messages, and the more bots we put up, the worse it will be.
>>>>
>>>> I think we still need the NOC warnings, but not over IRC. The Buildbot
>>>> NOC page is horrible and useless, since it doesn't know the difference
>>>> between "it's red and I know it" from "it's broken".
>>>>
>>>> For that reason, I have built my own NOC page:
>>>>
>>>> http://people.linaro.org/~renato.golin/llvm/arm-bots/
>>>>
>>>> But that machine is too slow to cope with all bots. We may need a
>>>> project to build such a system on a larger scale.
>>>>
>>>> However, for now, I think not printing the green results in IRC would
>>>> go a long way of cleaning the channel up.
>>>>
>>>> Any thoughts?
>>>>
>>>> cheers,
>>>> --renato
>>>> _______________________________________________
>>>> cfe-dev mailing list
>>>> [hidden email]
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> [hidden email]         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Philip Reames-4
In reply to this post by James Y Knight-2


On 05/19/2015 10:40 AM, James Y Knight wrote:
Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care.

While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems.

Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster.

E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary.

Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure.


If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement.
+1, this is a really good summary of issues.  I'm in full support of any and all efforts to reduce noise here. 

I've gotten to the point where I only watch a small handful bots.  Anything other than that, I pretty much ignore unless someone emails me directly or replies to a commit.  I'm not quite to point of marking buildbot emails as spam, but I'm definitely not paying them much attention either.  One particular irritant is getting emails 12-24 hours later about someone else's breakage that has *already been fixed*.  The long cycling bots are really irritating in that respect. 


On Tue, May 19, 2015 at 10:15 AM, Renato Golin <[hidden email]> wrote:
Folks,

I know it's a reasonably valuable thing to have the buildbot IRC bot
publishing results, but the channel is kind of flooded with the
messages, and the more bots we put up, the worse it will be.

I think we still need the NOC warnings, but not over IRC. The Buildbot
NOC page is horrible and useless, since it doesn't know the difference
between "it's red and I know it" from "it's broken".

For that reason, I have built my own NOC page:

http://people.linaro.org/~renato.golin/llvm/arm-bots/

But that machine is too slow to cope with all bots. We may need a
project to build such a system on a larger scale.

However, for now, I think not printing the green results in IRC would
go a long way of cleaning the channel up.

Any thoughts?

cheers,
--renato
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] LLVM IRC channel flooded?

Renato Golin-2
On 20 May 2015 at 18:47, Philip Reames <[hidden email]> wrote:
> One particular irritant is getting emails 12-24 hours later about someone else's
> breakage that has *already been fixed*.  The long cycling bots are really
> irritating in that respect.

That's not that easy to fix, and I think we'll have to cope with that
forever. Not all machines are fast, and some buildbots do a full
self-host, with compiler-rt and running all tests. Others do a full
benchmark run of LNT, running it 5-8 times, which can take several
hours on an ARM box.

The benchmark bots should be marked not to spam, since they're not
there to pick up errors, but the full self-hosting ones do need to
warn on errors. For example, right now I have a bug only on a thumbv7a
self-hosting bot, and not on others. I'm now bisecting it to find the
culprit, but this is not always clear, as the longer it takes for me
to realise, the harder it will be to fix it.

The only way out of it is for people to look at the fast bots, and if
they're fixed, check the commit that did it and see if the slow bot
has been fixed by the same commit later.

Buildbot owners will eventually pick those problems up, but as I said,
the longer it takes, the harder it is to get to the bottom of it, and
the higher is the probability of getting more regressions introduced
because the bot is red and won't warn.

cheers,
--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
12