Proposal for GSoC project for improving llvm-test testsuite

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposal for GSoC project for improving llvm-test testsuite

jakh
hello everybody,
I would like to improve the llvm-test suite[1] as a part of GSoC 2008 program for LLVM.
I have few concerns/problems regarding this, please give me your feedbacks and suggestions to come up with the proper proposal.

Goal: Improve the llvm-test testsuite.

Idea: Extend the llvm-test testsuite to include new programs and benchmarks[2]. Test programs should be more CPU intensive  with few library dependencies  at the end which are delivered in source format.

I have following few problems, please help me to understand them.

1. Before everything I need to clarify something.
Normally we add tests in a software  to check the correctness of that software code base. Here in this project llvm trying to specific on third party software codes. So I am not sure how a test be structured within llvm to check llvm code base correctness using those third party codes(This is what I understood, please correct me if I am wrong). I'd glad if I can get more information on this.

2. As described llvm testsuite program should be more CPU intensive programs, is there is any reason for that and how  a llvm test suite program can make more CPU intensive, any points that should consider to write a more CPU intensive test program for llvm?

3. Since we need to avoid many library dependencies, we need to reduce dependencies to standard C/C++ libraries and llvm libraries itself.Is this correct ? If so any points that need to consider writing tests which has few library decencies?

4. What a test program should cover?
As I read in the testsuite guide [3], test programs in llvm-test covers two things in general. It check for a particular LLVM feature or trigger a specific bug in LLVM. So in addition to that any other point that need to consider, along with the structure of a test case would be helpful
And also as described test programs are code fragments and whole programs. I think in here also I may need to write either of them depending of the benchmark trying[2].

5. Finally if I am going cover test programs for[3], may be setting few targets( targets in the sense that I am writing test programs for A variety of C++ benchmarks,BioPerf,LLC bench etc.., ) would be help to define the time line. May be more important one can be done in the first touch.

Please guide me on these. Please correct me if I am wrong in anything. Thanks in advance!

Me:
I am a computer science undergraduate[4]. I'm interested in compiler technology and have experience in C/C++ programming.My resume [5].
 

[1] - http://llvm.org/OpenProjects.html#llvmtest
[2] - http://nondot.org/sabre/LLVMNotes/#benchmarks
[3] - http://llvm.org/docs/TestingGuide.html#org
[4] - http://www.cse.mrt.ac.lk/
[5] - http://rajikacc.googlepages.com/resume_rajika.pdf


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Duncan Sands
I would like to see:

(1) LLVM testsuite run every night for all targets, target variants and
operating systems, for example by using an emulator to emulate targets
for which the hardware is not available.  The gcc project has a compile
farm which might be usable for this, but we would need to ask them if it
is ok.
(2) Likewise for the gcc testsuite, with results emailed to the nightly
test mailing list.  I would like the Fortran and Ada testsuites to be run
as well as the C/C++/ObjC tests.

Just some thoughts!

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Joachim Durchholz
In reply to this post by jakh

Am Mittwoch, den 19.03.2008, 10:22 +0530 schrieb Rajika Kumarasiri:
> 2. As described llvm testsuite program should be more CPU intensive
> programs, is there is any reason for that and how  a llvm test suite
> program can make more CPU intensive, any points that should consider
> to write a more CPU intensive test program for llvm?

I guess that's just trying to get the most effect out of machine time.
Running a database application compiled with LLVM will do more to test
the disk subsystem rather than the proper compilation of an application
program.
I think that's twofold.
Tests that run a *lot* of complicated code would test the correctness of
the compiler. Think code written by crazy people that does crazy things;
such code would hit a lot more code paths in the LLVM compiler than
something written by in-house application programmers that slavishly
obey a set of style guidelines.
The other situation is the usual CPU-intensive benchmarks. Number
crunching of any kind, large symbolic computations (such as the program
that first proved the four-color conjecture), scheduling programs for
machine parks or logistics, running any kind of program on a JVM
interpreter. This sort of stuff burns a whole lot of CPU cycles, and you
can easily see whether the optimization passes are doing their work (and
if you hit a case that takes longer than it should, there's an
opportunity to identify yet another optimization technique or a mising
tweak in an existing one that helps this particular application and -
hopefully - all others, too).

> 3. Since we need to avoid many library dependencies, we need to reduce
> dependencies to standard C/C++ libraries and llvm libraries itself.Is
> this correct ?

I suspect dependencies on LLVM libs are OK. An application that needs
lots of standard libraries would more likely hit version mismatch
problems. Or the library in question would be unavailable for all
platform (not so much a problem with open source libs, but closed ones
are probably a no-no).

> 4. What a test program should cover?
> As I read in the testsuite guide [3], test programs in llvm-test
> covers two things in general. It check for a particular LLVM feature
> or trigger a specific bug in LLVM. So in addition to that any other
> point that need to consider, along with the structure of a test case
> would be helpful
> And also as described test programs are code fragments and whole
> programs. I think in here also I may need to write either of them
> depending of the benchmark trying[2].

There are two test suites. One lives in llvm/test and I gather it checks
for regressions and essential features (mostly). It is designed to run
quickly (a few minutes).
The other lives in projects/llvm-test and exercizes the full array of
available tests, in particular those that take a lot of time (such tests
help to identify optimizations that could improve).

To further complicate matters, the llvm/test suite uses dejagnu to
manage tests and collect results, while projects/llvm-test uses just
makefiles.

(I'm pretty sure I overlooked some aspects, I just read some tests,
makefiles, and test results.)

Hope I'm not too far off the mark :-)

Regards,
Jo

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

John Criswell
In reply to this post by jakh
Rajika Kumarasiri wrote:
> hello everybody,
> I would like to improve the llvm-test suite[1] as a part of GSoC 2008 program for LLVM.
> I have few concerns/problems regarding this, please give me your feedbacks and suggestions to come up with the proper proposal.
>  
So, here's the type of programs that I would like to see added to
llvm-test.  These are biased by past and current research agendas, so
I'd be interested in knowing whether others would find such additions
useful:

1) Pointer intensive applications (especially those written in C).  We
have a number of transforms that either improve memory performance
(Automatic Pool Allocation) or whose overhead can be greater for pointer
intensive programs (SAFECode).  Having such programs would be useful for
our research; they could also be used by others who are working on
pointer-analysis based optimizations for LLVM.

2) Programs that are memory intensive (for the same reasons that I want
pointer intensive programs).

3) Programs that are known to have found bugs in LLVM.  There are many
programs that LLVM compiles correctly.  However, if you find one that
exposes one or more bugs in LLVM, it might make a good candidate.  
kimwitu++ is in the test suite because it triggered 4 bugs and motivated
1 JIT optimization.

-- John T.

> Goal: Improve the llvm-test testsuite.
>
> Idea: Extend the llvm-test testsuite to include new programs and benchmarks[2]. Test programs should be more CPU intensive  with few library dependencies  at the end which are delivered in source format.
>
> I have following few problems, please help me to understand them.
>
> 1. Before everything I need to clarify something.
> Normally we add tests in a software  to check the correctness of that software code base. Here in this project llvm trying to specific on third party software codes. So I am not sure how a test be structured within llvm to check llvm code base correctness using those third party codes(This is what I understood, please correct me if I am wrong). I'd glad if I can get more information on this.
>
> 2. As described llvm testsuite program should be more CPU intensive programs, is there is any reason for that and how  a llvm test suite program can make more CPU intensive, any points that should consider to write a more CPU intensive test program for llvm?
>
> 3. Since we need to avoid many library dependencies, we need to reduce dependencies to standard C/C++ libraries and llvm libraries itself.Is this correct ? If so any points that need to consider writing tests which has few library decencies?
>  
> 4. What a test program should cover?
> As I read in the testsuite guide [3], test programs in llvm-test covers two things in general. It check for a particular LLVM feature or trigger a specific bug in LLVM. So in addition to that any other point that need to consider, along with the structure of a test case would be helpful
> And also as described test programs are code fragments and whole programs. I think in here also I may need to write either of them depending of the benchmark trying[2].
>
> 5. Finally if I am going cover test programs for[3], may be setting few targets( targets in the sense that I am writing test programs for A variety of C++ benchmarks,BioPerf,LLC bench etc.., ) would be help to define the time line. May be more important one can be done in the first touch.
>
> Please guide me on these. Please correct me if I am wrong in anything. Thanks in advance!
>
> Me:
> I am a computer science undergraduate[4]. I'm interested in compiler technology and have experience in C/C++ programming.My resume [5].
>
>
> [1] - http://llvm.org/OpenProjects.html#llvmtest
> [2] - http://nondot.org/sabre/LLVMNotes/#benchmarks
> [3] - http://llvm.org/docs/TestingGuide.html#org
> [4] - http://www.cse.mrt.ac.lk/
> [5] - http://rajikacc.googlepages.com/resume_rajika.pdf
>
>
>  

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Devang Patel
In reply to this post by Joachim Durchholz

On Mar 19, 2008, at 5:13 AM, Joachim Durchholz wrote:

> There are two test suites. One lives in llvm/test and I gather it  
> checks
> for regressions and essential features (mostly). It is designed to run
> quickly (a few minutes).

The goal here is to test features and check regressions. The tests are  
written by LLVM developers. Usually the tests are based on new  
features OR derived from bug reports. It is expected that each llvm  
check-in is proceeded by at least on llvm/test check.

> The other lives in projects/llvm-test and exercizes the full array of
> available tests, in particular those that take a lot of time (such  
> tests
> help to identify optimizations that could improve).

The primary goal here is to measure and monitor performance. This  
suite includes benchmarks and applications from various domains. This  
suite uses makefiles to accommodate benchmark and application specific  
build instructions.

> To further complicate matters, the llvm/test suite uses dejagnu to
> manage tests and collect results, while projects/llvm-test uses just
> makefiles.

-
Devang



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Devang Patel
In reply to this post by Duncan Sands

On Mar 19, 2008, at 1:39 AM, Duncan Sands wrote:

> I would like to see:
>
> (1) LLVM testsuite run every night for all targets, target variants  
> and
> operating systems, for example by using an emulator to emulate targets
> for which the hardware is not available.  The gcc project has a  
> compile
> farm which might be usable for this, but we would need to ask them  
> if it
> is ok.
> (2) Likewise for the gcc testsuite, with results emailed to the  
> nightly
> test mailing list.  I would like the Fortran and Ada testsuites to  
> be run
> as well as the C/C++/ObjC tests.
>
> Just some thoughts!

I agree this would be useful. However these two items require access  
to lots of machine cycles. Do you anticipate lots of work, enough to  
justify for a GSoC proposal, in existing test setup to accommodate  
these two specific goals ?

>
>
> Ciao,
>
> Duncan.
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-
Devang



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Devang Patel
In reply to this post by jakh

On Mar 18, 2008, at 9:52 PM, Rajika Kumarasiri wrote:

hello everybody,
I would like to improve the llvm-test suite[1] as a part of GSoC 2008 program for LLVM.
I have few concerns/problems regarding this, please give me your feedbacks and suggestions to come up with the proper proposal.

Goal: Improve the llvm-test testsuite.

There are two parts :
1) llvm-project/llvm/test in svn. This part checks correctness and features.
2) Whole program testsuite, which is located at llvm-project/test-suite. This part includes benchmarks and applications.



Idea: Extend the llvm-test testsuite to include new programs and benchmarks[2]. Test programs should be more CPU intensive  with few library dependencies  at the end which are delivered in source format.

I have following few problems, please help me to understand them.

1. Before everything I need to clarify something.
Normally we add tests in a software  to check the correctness of that software code base.

This is done in 1)st part.

Here in this project llvm trying to specific on third party software codes. So I am not sure how a test be structured within llvm to check llvm code base correctness using those third party codes(This is what I understood, please correct me if I am wrong). I'd glad if I can get more information on this.

When LLVM fails to build third party code, you want to derive small test cases to reproduce such failures. This small and independent tests are added in 1st part.

2. As described llvm testsuite program should be more CPU intensive programs, is there is any reason for that

The goal here (in 2)nd part) is to measure the performance of the code generated by LLVM.

and how  a llvm test suite program can make more CPU intensive, any points that should consider to write a more CPU intensive test program for llvm?

IIUC, You do not want to write new test suite program here. You want to adopt existing programs and benchmarks and update them such that they fit in 2)nd part.  

Let's take BioPerf as an example. You want to take this existing benchmark and appropriately update its makefiles so that it builds on 2)nd part of llvm test-suite. I am not familiar with this benchmark itself, but it uses many external libraries the you want to minimize library use if possible. Finally, the benchmark is likely to use some sort of input data to measure performance. You want to adjust input data set such that it does not take hours to run this benchmark and at the same time it takes enough time to notice performance changes. 

3. Since we need to avoid many library dependencies, we need to reduce dependencies to standard C/C++ libraries and llvm libraries itself.Is this correct ? If so any points that need to consider writing tests which has few library decencies?

4. What a test program should cover?
As I read in the testsuite guide [3], test programs in llvm-test covers two things in general. It check for a particular LLVM feature or trigger a specific bug in LLVM. So in addition to that any other point that need to consider, along with the structure of a test case would be helpful
And also as described test programs are code fragments and whole programs. I think in here also I may need to write either of them depending of the benchmark trying[2].

5. Finally if I am going cover test programs for[3], may be setting few targets( targets in the sense that I am writing test programs for A variety of C++ benchmarks,BioPerf,LLC bench etc.., ) would be help to define the time line. May be more important one can be done in the first touch.




Please guide me on these. Please correct me if I am wrong in anything. Thanks in advance!

Me:
I am a computer science undergraduate[4]. I'm interested in compiler technology and have experience in C/C++ programming.My resume [5].
 

[1] - http://llvm.org/OpenProjects.html#llvmtest
[2] - http://nondot.org/sabre/LLVMNotes/#benchmarks
[3] - http://llvm.org/docs/TestingGuide.html#org
[4] - http://www.cse.mrt.ac.lk/
[5] - http://rajikacc.googlepages.com/resume_rajika.pdf

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-
Devang




_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Duncan Sands
In reply to this post by Devang Patel
Hi,

> I agree this would be useful. However these two items require access  
> to lots of machine cycles.

that's why I suggested using gcc's cfarm compile farm.

> Do you anticipate lots of work, enough to  
> justify for a GSoC proposal, in existing test setup to accommodate  
> these two specific goals ?

I think it would take at least two weeks to set this up.  I don't
know how long a SoC project is supposed to take, but presumably more
than this.  So this could be part of a larger testing framework.  I
must admit that setting this kind of thing up isn't very exciting or
cutting edge, but it is important.  My impression is that LLVM isn't
tested half as hard as it needs to be.

It would also be nice to have some kind of automated patch queue,
to which patches can be submitted and which runs the testsuite with
them applied.

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Duncan Sands
In reply to this post by Devang Patel
While on the subject of the testsuite, some random thoughts:

- show how test results changed from one week and one month
ago, and compared to the last release, rather than just
comparing with the previous run as now.  Good for spotting
trends.
- fix the testsuite so that buildlogs are really available:
I occasionally see that a nightly tester failed the build, and
in the message it gives a url where you can get the build log.
Only the logs aren't there...

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

jakh
Hello every body,
Thank you all for your great suggestions and feedbacks, I'd forward the detailed proposal within couple of days. And also if I need to clarify any thing I'll ask in the list. Thanks again.
Regards,
Rajika

On Thu, Mar 20, 2008 at 10:35 PM, Duncan Sands <[hidden email]> wrote:
While on the subject of the testsuite, some random thoughts:

- show how test results changed from one week and one month
ago, and compared to the last release, rather than just
comparing with the previous run as now.  Good for spotting
trends.
- fix the testsuite so that buildlogs are really available:
I occasionally see that a nightly tester failed the build, and
in the message it gives a url where you can get the build log.
Only the logs aren't there...

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



--
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

Christopher Lamb
Another suggestion: a web service that would allow you to submit a test build and it would provide you feedback on deviations from known good for that target. I know a lot of people keep two trees around for this purpose, and I'd personally like to ditch the duplicate if I could.

--
Chris

On Mar 21, 2008, at 9:22 PM, Rajika Kumarasiri wrote:
Hello every body,
Thank you all for your great suggestions and feedbacks, I'd forward the detailed proposal within couple of days. And also if I need to clarify any thing I'll ask in the list. Thanks again.
Regards,
Rajika

On Thu, Mar 20, 2008 at 10:35 PM, Duncan Sands <[hidden email]> wrote:
While on the subject of the testsuite, some random thoughts:

- show how test results changed from one week and one month
ago, and compared to the last release, rather than just
comparing with the previous run as now.  Good for spotting
trends.
- fix the testsuite so that buildlogs are really available:
I occasionally see that a nightly tester failed the build, and
in the message it gives a url where you can get the build log.
Only the logs aren't there...

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



--
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
_______________________________________________
LLVM Developers mailing list

--
Christopher Lamb




_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Proposal for GSoC project for improving llvm-test testsuite

jakh
This will seems to be a very interesting and also very useful, suggesstion. I'll add this also for the GSoC propsal since I have some experince in web services. We can defeniitely use Axis2/C[1] web service frame work for this purpose.
Thanks!
Regards ,
Rajika

[1] - http://ws.apache.org/axis2/c/

On Tue, Mar 25, 2008 at 11:24 AM, Christopher Lamb <[hidden email]> wrote:
Another suggestion: a web service that would allow you to submit a test build and it would provide you feedback on deviations from known good for that target. I know a lot of people keep two trees around for this purpose, and I'd personally like to ditch the duplicate if I could.

--
Chris

On Mar 21, 2008, at 9:22 PM, Rajika Kumarasiri wrote:
Hello every body,
Thank you all for your great suggestions and feedbacks, I'd forward the detailed proposal within couple of days. And also if I need to clarify any thing I'll ask in the list. Thanks again.
Regards,
Rajika

On Thu, Mar 20, 2008 at 10:35 PM, Duncan Sands <[hidden email]> wrote:
While on the subject of the testsuite, some random thoughts:

- show how test results changed from one week and one month
ago, and compared to the last release, rather than just
comparing with the previous run as now.  Good for spotting
trends.
- fix the testsuite so that buildlogs are really available:
I occasionally see that a nightly tester failed the build, and
in the message it gives a url where you can get the build log.
Only the logs aren't there...

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



--
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
_______________________________________________
LLVM Developers mailing list

--
Christopher Lamb




_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev




--
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev