DejaGNU test fixes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

DejaGNU test fixes

matthijs
Hi all,

while writing a testcase thate needed to do a grep containg {, I found that
the DejaGNU test framework didn't handle those very well. It's a bit of a fuss
to escape accolades properly, but most of all the framework seemed to silently
ignore errors in the escaping (and just not run the command then). See [1].

Fixing the framework resulted in 80 of the tests failing. I spent the
afternoon fixing 60 tests. Of these, I fixed 42 that have been running fine so
far, but generated output on stderr or were marked as failing for some other
reason. Additionally, I fixed 18 test cases that were not run or only partly
run before (though all of those passed after fixing the test case).

I leave the remaining 20 failing testcases, since I do not see a clear
resolution for them. In a lot of cases disabling or fixing a warning might be
enough, but since I can't always see what a test is supposed to test, I'm
afraid that that will actually render the testcase useless.

Of these final 20 tests, 3 are related to bugpoint producing stderr output (but I
can't see if that is expected output or not). Two are related to an invalid
-mcpu option, I think the corresponding RUN lines should be removed or
changed? One is related to the -mattr=sse1 option that is not supported
(anymore?). The remaining 14 are warnings produced by llvm-gcc.

I've attached the error output for the remaining 20 tests, please check it and
fix them where possible.

Gr.

Matthijs

[1]: http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20080609/063543.html

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

test8.log (43K) Download Attachment
signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: DejaGNU test fixes

Dale Johannesen

On Jun 10, 2008, at 9:32 AMPDT, Matthijs Kooijman wrote:

> Hi all,
>
> while writing a testcase thate needed to do a grep containg {, I  
> found that
> the DejaGNU test framework didn't handle those very well. It's a bit  
> of a fuss
> to escape accolades properly, but most of all the framework seemed  
> to silently
> ignore errors in the escaping (and just not run the command then).  
> See [1].
>
> Fixing the framework resulted in 80 of the tests failing. I spent the
> afternoon fixing 60 tests. Of these, I fixed 42 that have been  
> running fine so
> far, but generated output on stderr or were marked as failing for  
> some other
> reason. Additionally, I fixed 18 test cases that were not run or  
> only partly
> run before (though all of those passed after fixing the test case).
>
> I leave the remaining 20 failing testcases, since I do not see a clear
> resolution for them. In a lot of cases disabling or fixing a warning  
> might be
> enough, but since I can't always see what a test is supposed to  
> test, I'm
> afraid that that will actually render the testcase useless.
>
> Of these final 20 tests, 3 are related to bugpoint producing stderr  
> output (but I
> can't see if that is expected output or not). Two are related to an  
> invalid
> -mcpu option, I think the corresponding RUN lines should be removed or
> changed? One is related to the -mattr=sse1 option that is not  
> supported
> (anymore?). The remaining 14 are warnings produced by llvm-gcc.
>
> I've attached the error output for the remaining 20 tests, please  
> check it and
> fix them where possible.
My failure log is not identical to yours, but I've fixed some of the  
problems:

Using -mattr=sse1 was wrong, it's spelled sse.
9 tests got warnings from the gcc FE.  I believe these are all  
legitimate,
so I just suppressed the warnings with -w, which will not interfere with
running the llvm pieces.  (Perhaps it is better to do this in the  
script?  Code
that gets warnings should be testable.)
1 was using %llvmgxx (which includes -c) instead of %link to link.  
Now that
is fixed and the test fails with a more reasonable error.

> Gr.
>
> Matthijs
>
> [1]: http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20080609/063543.html

> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

test8.log (44K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: DejaGNU test fixes

Gordon Henriksen-3
In reply to this post by matthijs
Does merely sending output to stderr legitimately constitute a test  
failure? This doesn't seem entirely legitimate to me. I send  
checkpoints to stderr in the ocaml test case (because linking each  
individual test would be too slow); I was surprised and confused to  
see it failing this morning, when the exit code in fact correctly  
reports success.

— Gordon


On Jun 10, 2008, at 12:32, Matthijs Kooijman wrote:

> Hi all,
>
> while writing a testcase thate needed to do a grep containg {, I  
> found that the DejaGNU test framework didn't handle those very well.  
> It's a bit of a fuss to escape accolades properly, but most of all  
> the framework seemed to silently ignore errors in the escaping (and  
> just not run the command then). See [1].
>
> Fixing the framework resulted in 80 of the tests failing. I spent  
> the afternoon fixing 60 tests. Of these, I fixed 42 that have been  
> running fine so far, but generated output on stderr or were marked  
> as failing for some other reason. Additionally, I fixed 18 test  
> cases that were not run or only partly run before (though all of  
> those passed after fixing the test case).
>
> I leave the remaining 20 failing testcases, since I do not see a  
> clear resolution for them. In a lot of cases disabling or fixing a  
> warning might be enough, but since I can't always see what a test is  
> supposed to test, I'm afraid that that will actually render the  
> testcase useless.
>
> Of these final 20 tests, 3 are related to bugpoint producing stderr  
> output (but I can't see if that is expected output or not). Two are  
> related to an invalid -mcpu option, I think the corresponding RUN  
> lines should be removed or changed? One is related to the -
> mattr=sse1 option that is not supported (anymore?). The remaining 14  
> are warnings produced by llvm-gcc.
>
> I've attached the error output for the remaining 20 tests, please  
> check it and fix them where possible.


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: DejaGNU test fixes

Matthijs Kooijman - Inter-Actief
> Does merely sending output to stderr legitimately constitute a test  
> failure?
Yes and no :-)

I think that being notified of stderr output is generally a good idea. If you
really think that for a particular testcase stderr output is expected and
doesn not mean failure, that particular test can redirect stderr and things
work fine.

However, the most compelling reason for marking stderr output as a failure is
my lack of TCL skills :-) I haven't found a way to distinguish between stderr
output of the program and parsing errors of the command line. So, rather than
ignoring command line parse options, I'd rather fail on stderr output.

So, it's not a policy decision to mark stderr output as a failure, but now
that it is happening I don't think it is so bad to do so.

Gr.

Matthijs

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

signature.asc (196 bytes) Download Attachment