[llvm-dev] Using lnt to run SPEC tests: possible or not?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[llvm-dev] Using lnt to run SPEC tests: possible or not?

Robin Eklind via llvm-dev
It appears that lnt has the capability to run the
SPEC2000/2006/2017(maybe) tests from looking at the options and from
poking around in the mailing list archives.  Or does it?  I experimented
a bit and cannot get it to work right.  However, it isn't really
documented anywhere so maybe I didn't set it up properly.  Plus there's
all the mention of llvmgcc...

So is anyone doing this?  Is there some setup information that I missed?
  Thanks for any tips on how to do this!
--

-Bill Seurer

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] Using lnt to run SPEC tests: possible or not?

Robin Eklind via llvm-dev
Hi Bill,

Yes, it is possible to run SPEC tests through LNT/test-suite.
You won’t get official reportable SPEC numbers as you don’t use the official build-and-run framework from SPEC, but OTOH you get the extra features in LNT/test-suite working out of the box, e.g. the ones documented at http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.

To make use of this, I think you need to use the cmake driver in the test-suite.
Which, if you use the LNT wrapper to drive runs, is invoked using “lnt runtest test-suite”.

A few example of the important options on the command line that I hope will get you started:
  • Running SPEC2K6int reference inputs: lnt runtest test-suite --only-test=External/SPEC/CINT2006 --cmake-define TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define TEST_SUITE_RUN_TYPE=ref
  • Running SPEC2K6int train inputs: lnt runtest test-suite --only-test=External/SPEC/CINT2006 --cmake-define TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define TEST_SUITE_RUN_TYPE=train
  • Running SPEC2Kfp reference inputs:  lnt runtest test-suite --only-test=External/SPEC/CFP2000 --cmake-define TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define TEST_SUITE_RUN_TYPE=ref
     
Thanks,

Kristof


On 19 Jun 2018, at 23:53, Bill Seurer via llvm-dev <[hidden email]> wrote:

It appears that lnt has the capability to run the SPEC2000/2006/2017(maybe) tests from looking at the options and from poking around in the mailing list archives.  Or does it?  I experimented a bit and cannot get it to work right.  However, it isn't really documented anywhere so maybe I didn't set it up properly.  Plus there's all the mention of llvmgcc...

So is anyone doing this?  Is there some setup information that I missed?  Thanks for any tips on how to do this!
--

-Bill Seurer

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] Using lnt to run SPEC tests: possible or not?

Robin Eklind via llvm-dev
Thanks Kristof.

I am hitting some odd cmake issues (it isn't handing pthread stuff
properly for some reason) that I haven't seen before but this should be
me going.  I've used lnt for running the llvm test suite many times
using "runtest nt" without problem.

Hmmm.  powerpc64 isn't listed in the known architectures in
test_suite.py so I may need to update some of these files.

On 06/20/18 02:02, Kristof Beyls wrote:

> Hi Bill,
>
> Yes, it is possible to run SPEC tests through LNT/test-suite.
> You won’t get official reportable SPEC numbers as you don’t use the
> official build-and-run framework from SPEC, but OTOH you get the extra
> features in LNT/test-suite working out of the box, e.g. the ones
> documented at
> http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.
>
> To make use of this, I think you need to use the cmake driver in the
> test-suite.
> Which, if you use the LNT wrapper to drive runs, is invoked using “lnt
> runtest test-suite”.
>
> A few example of the important options on the command line that I hope
> will get you started:
>
>   * Running SPEC2K6int reference inputs:lnt runtest test-suite
>     --only-test=External/SPEC/CINT2006 --cmake-define
>     TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>     TEST_SUITE_RUN_TYPE=ref
>
>   * Running SPEC2K6int train inputs:lnt runtest test-suite
>     --only-test=External/SPEC/CINT2006 --cmake-define
>     TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
>     TEST_SUITE_RUN_TYPE=train
>   * Running SPEC2Kfp reference inputs: lnt runtest test-suite
>     --only-test=External/SPEC/CFP2000 --cmake-define
>     TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define
>     TEST_SUITE_RUN_TYPE=ref
>
> Thanks,
>
> Kristof
>
>
>> On 19 Jun 2018, at 23:53, Bill Seurer via llvm-dev
>> <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> It appears that lnt has the capability to run the
>> SPEC2000/2006/2017(maybe) tests from looking at the options and from
>> poking around in the mailing list archives.  Or does it?  I
>> experimented a bit and cannot get it to work right.  However, it isn't
>> really documented anywhere so maybe I didn't set it up properly.  Plus
>> there's all the mention of llvmgcc...
>>
>> So is anyone doing this?  Is there some setup information that I
>> missed?  Thanks for any tips on how to do this!
>> --
>>
>> -Bill Seurer
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email] <mailto:[hidden email]>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>


--

-Bill Seurer

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] Using lnt to run SPEC tests: possible or not?

Robin Eklind via llvm-dev
If you don't want to submit the results to an LNT server you can also just run the test-suite directly without going through LNT.

You should be able to do something like:
cmake -DCMAKE_C_COMPILER=/path/to/my/clang -DTEST_SUITE_SPECXXX_ROOT -DTEST_SUITE_SUBDIRS=External/SPECXXX -GNinja /path/to/test-suite
# NOte: use ccmake or cmake -L to see a list of variables you can tweak.
ninja
llvm-lit

to build and run SPEC via the test-suite infrastructure.

- Matthias

On Jun 20, 2018, at 12:20 PM, Bill Seurer via llvm-dev <[hidden email]> wrote:

Thanks Kristof.

I am hitting some odd cmake issues (it isn't handing pthread stuff properly for some reason) that I haven't seen before but this should be me going.  I've used lnt for running the llvm test suite many times using "runtest nt" without problem.

Hmmm.  powerpc64 isn't listed in the known architectures in test_suite.py so I may need to update some of these files.

On 06/20/18 02:02, Kristof Beyls wrote:
Hi Bill,
Yes, it is possible to run SPEC tests through LNT/test-suite.
You won’t get official reportable SPEC numbers as you don’t use the official build-and-run framework from SPEC, but OTOH you get the extra features in LNT/test-suite working out of the box, e.g. the ones documented at http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.
To make use of this, I think you need to use the cmake driver in the test-suite.
Which, if you use the LNT wrapper to drive runs, is invoked using “lnt runtest test-suite”.
A few example of the important options on the command line that I hope will get you started:
 * Running SPEC2K6int reference inputs:lnt runtest test-suite
   --only-test=External/SPEC/CINT2006 --cmake-define
   TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
   TEST_SUITE_RUN_TYPE=ref
 * Running SPEC2K6int train inputs:lnt runtest test-suite
   --only-test=External/SPEC/CINT2006 --cmake-define
   TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define
   TEST_SUITE_RUN_TYPE=train
 * Running SPEC2Kfp reference inputs: lnt runtest test-suite
   --only-test=External/SPEC/CFP2000 --cmake-define
   TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define
   TEST_SUITE_RUN_TYPE=ref
Thanks,
Kristof
On 19 Jun 2018, at 23:53, Bill Seurer via llvm-dev <[hidden email] <[hidden email]>> wrote:

It appears that lnt has the capability to run the SPEC2000/2006/2017(maybe) tests from looking at the options and from poking around in the mailing list archives. Or does it?  I experimented a bit and cannot get it to work right.  However, it isn't really documented anywhere so maybe I didn't set it up properly.  Plus there's all the mention of llvmgcc...

So is anyone doing this?  Is there some setup information that I missed?  Thanks for any tips on how to do this!
-- 

-Bill Seurer

_______________________________________________
LLVM Developers mailing list
[hidden email] <[hidden email]>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


-- 

-Bill Seurer

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev