Proposal for improving the llvm nightly tester

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposal for improving the llvm nightly tester

jakh
hi all,
After having some discussions in the IRC, I am trying here to come up with a proposal for GSoC 2008 for improving the llvm nightly tester[1].Following are the ideas and suggestions that came up in the discussion, if you have  any comment  or any other suggestion please add them to the list. I have some doubts in some places.

1.  Improvements to the perl script which manage actual testing (./util/NewNightlyTest.pl)
     (a).  compiling llvm-gcc
     (b).  allowing someone to use an existing tree versus checking out a new one

2.  The improvements to the email that get sent to the llvm-testresults
      (a). finding a point in time where some test failed
      (b). comparing two different sets of test results that aren't just 2 days apart
  here if you have any suggestions which help someone analysis test results would be grateful.

3. Running llvm testsuite for all targets, target variants and operating systems(using an simulator for hardware which is not available) ex:gcc project's compile farm [2]

4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed to the llvm-testresults.

Most of the server side and the database design has already done by "tonic"(IRC nickname at #llvm), so suggestions regarding others would be important, so that we have a fully featured nightly tester .And finally all the php scripts need to be updated according to the required new features. 
Thanks in advance. Please add your comments.!
Regards,
-Rajika

[1] - http://llvm.org/nightlytest/
[2] - http://gcc.gnu.org/wiki/CompileFarm

--
gautamabuddha - http://www.gautamabuddha.org/
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for improving the llvm nightly tester

Dale Johannesen
My main complaints about the nightly tester are:
Loading the web pages is waaay tooo sllooooow
The information provided is not terribly useful for tracking down bugs.  What bug fixers need is environment+command line options adequate to reproduce the problem.  Access to the run logs would be a good start.

On Apr 4, 2008, at 9:22 PM, Rajika Kumarasiri wrote:

hi all,
After having some discussions in the IRC, I am trying here to come up with a proposal for GSoC 2008 for improving the llvm nightly tester[1].Following are the ideas and suggestions that came up in the discussion, if you have  any comment  or any other suggestion please add them to the list. I have some doubts in some places.

1.  Improvements to the perl script which manage actual testing (./util/NewNightlyTest.pl)
     (a).  compiling llvm-gcc
     (b).  allowing someone to use an existing tree versus checking out a new one

2.  The improvements to the email that get sent to the llvm-testresults
      (a). finding a point in time where some test failed
      (b). comparing two different sets of test results that aren't just 2 days apart
  here if you have any suggestions which help someone analysis test results would be grateful.

3. Running llvm testsuite for all targets, target variants and operating systems(using an simulator for hardware which is not available) ex:gcc project's compile farm [2]

4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed to the llvm-testresults.

Most of the server side and the database design has already done by "tonic"(IRC nickname at #llvm), so suggestions regarding others would be important, so that we have a fully featured nightly tester .And finally all the php scripts need to be updated according to the required new features. 
Thanks in advance. Please add your comments.!
Regards,
-Rajika

[1] - http://llvm.org/nightlytest/
[2] - http://gcc.gnu.org/wiki/CompileFarm

--
gautamabuddha - http://www.gautamabuddha.org/
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for improving the llvm nightly tester

jakh
Hi Dale,

On Sat, Apr 5, 2008 at 10:27 AM, Dale Johannesen <[hidden email]> wrote:
My main complaints about the nightly tester are:
Loading the web pages is waaay tooo sllooooow
One of  the complaints there was that the nightly tester is too slow. I am looking forward to hear any idea from some one to improve that.
 
The information provided is not terribly useful for tracking down bugs.  What bug fixers need is environment+command line options adequate to reproduce the
problem.  Access to the run logs would be a good start.
This information can be added to the email that get delivered to the llvm-testresults. Thanks for the inputs!

-Rajika

On Apr 4, 2008, at 9:22 PM, Rajika Kumarasiri wrote:

hi all,
After having some discussions in the IRC, I am trying here to come up with a proposal for GSoC 2008 for improving the llvm nightly tester[1].Following are the ideas and suggestions that came up in the discussion, if you have  any comment  or any other suggestion please add them to the list. I have some doubts in some places.

1.  Improvements to the perl script which manage actual testing (./util/NewNightlyTest.pl)
     (a).  compiling llvm-gcc
     (b).  allowing someone to use an existing tree versus checking out a new one

2.  The improvements to the email that get sent to the llvm-testresults
      (a). finding a point in time where some test failed
      (b). comparing two different sets of test results that aren't just 2 days apart
  here if you have any suggestions which help someone analysis test results would be grateful.

3. Running llvm testsuite for all targets, target variants and operating systems(using an simulator for hardware which is not available) ex:gcc project's compile farm [2]

4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed to the llvm-testresults.

Most of the server side and the database design has already done by "tonic"(IRC nickname at #llvm), so suggestions regarding others would be important, so that we have a fully featured nightly tester .And finally all the php scripts need to be updated according to the required new features. 
Thanks in advance. Please add your comments.!
Regards,
-Rajika

[1] - http://llvm.org/nightlytest/
[2] - http://gcc.gnu.org/wiki/CompileFarm

--
gautamabuddha - http://www.gautamabuddha.org/
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev




--
gautamabuddha - http://www.gautamabuddha.org/
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for improving the llvm nightly tester

Tanya Lattner-2
In reply to this post by jakh

> 1.  Improvements to the perl script which manage actual testing
> (./util/NewNightlyTest.pl)
>     (a).  compiling llvm-gcc
>     (b).  allowing someone to use an existing tree versus checking out a
> new one
> 2.  The improvements to the email that get sent to the llvm-testresults
>      (a). finding a point in time where some test failed
>      (b). comparing two different sets of test results that aren't just 2
> days apart
>  here if you have any suggestions which help someone analysis test results
> would be grateful.

I think you need to expand on this. Basically, these would be improvements
to how the data is displayed and analyzed. The second part is making the
emails that get sent out better. They are separate things.

You should come up with ideas on your own about how someone might want to
interpret the data and allow that flexibility via the web pages.

> 3. Running llvm testsuite for all targets, target variants and operating
> systems(using an simulator for hardware which is not available) ex:gcc
> project's compile farm [2]

How will you test this?

> 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed
> to the llvm-testresults.

Are these different or new tests? Why wouldn't they just be a part of the
llvm-test module? Then you don't have to do any modifications to the
scripts.

-Tanya


> Most of the server side and the database design has already done by
> "tonic"(IRC nickname at #llvm), so suggestions regarding others would be
> important, so that we have a fully featured nightly tester .And finally all
> the php scripts need to be updated according to the required new features.
> Thanks in advance. Please add your comments.!
> Regards,
> -Rajika
>
> [1] - http://llvm.org/nightlytest/
> [2] - http://gcc.gnu.org/wiki/CompileFarm
>
>
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for improving the llvm nightly tester

Duncan Sands
> > 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed
> > to the llvm-testresults.
>
> Are these different or new tests? Why wouldn't they just be a part of the
> llvm-test module? Then you don't have to do any modifications to the
> scripts.

Probably means the gcc testsuite.

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for improving the llvm nightly tester

Devang Patel

On Apr 7, 2008, at 10:36 AM, Duncan Sands wrote:

>>> 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test  
>>> results emailed
>>> to the llvm-testresults.
>>
>> Are these different or new tests? Why wouldn't they just be a part  
>> of the
>> llvm-test module? Then you don't have to do any modifications to the
>> scripts.
>
> Probably means the gcc testsuite.

Eventually, we want to to move all lang. FE tests from llvm/test to  
llvm-gcc testsuite once nightly tester script start supporting llvm-
gcc-4.2 builds.

-
Devang

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for improving the llvm nightly tester

Tanya Lattner-2
In reply to this post by Duncan Sands

>>> 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed
>>> to the llvm-testresults.
>>
>> Are these different or new tests? Why wouldn't they just be a part of the
>> llvm-test module? Then you don't have to do any modifications to the
>> scripts.
>
> Probably means the gcc testsuite.

Well then this should be clarified in the proposal. :)

-Tanya

>
> Ciao,
>
> Duncan.
>
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for improving the llvm nightly tester

jakh
Thanks all for your ideas, Please review the proposal at Google web app. I have added my proposal there.
Thanks!
-Rajika

On Tue, Apr 8, 2008 at 12:07 AM, Tanya M. Lattner <[hidden email]> wrote:

>>> 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed
>>> to the llvm-testresults.
>>
>> Are these different or new tests? Why wouldn't they just be a part of the
>> llvm-test module? Then you don't have to do any modifications to the
>> scripts.
>
> Probably means the gcc testsuite.

Well then this should be clarified in the proposal. :)

-Tanya

>
> Ciao,
>
> Duncan.
>
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



--
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev