Re: [llvm-testresults] cfarm-x86-64 x86_64 nightly tester results

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-testresults] cfarm-x86-64 x86_64 nightly tester results

Duncan Sands
This nightly tester is now using an llvm-g++ that produces the new ODR linkage
types.  This means that many more functions are being considered by the
inter-procedural optimization passes (for example, "linkonce" functions defined
in a header).  The result seems to be pretty huge swings (both good and bad) in
the C++ tests in the testsuite, see below.  Note that this tester is often under
heavy load due to other users, so take timing results with a pinch of salt (a
pinch is about +-10% in my experience with this tester).

Ciao,

Duncan.

> Significant changes in test results:
> GCCAS:
>  singlesource/Benchmarks/Adobe-C++/functionobjects: -102.68% (0.5960 => 1.2080)
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -44.04% (5.3323 => 7.6804)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: -10.34% (5.6883 => 6.2763)
>  singlesource/Benchmarks/Adobe-C++/simple_types_loop_invariant: -19.15% (3.4882 => 4.1562)
>  singlesource/Benchmarks/Adobe-C++/stepanov_abstraction: -97.63% (1.0080 => 1.9921)
>  singlesource/Benchmarks/Adobe-C++/stepanov_vector: -171.51% (0.8280 => 2.2481)
>  singlesource/Benchmarks/Misc-C++/bigfib: -43.51% (0.5240 => 0.7520)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -110.33% (1.0080 => 2.1201)
>  singlesource/Benchmarks/Shootout-C++/wordfreq: -43.20% (0.5000 => 0.7160)
>  multisource/Applications/hbd/hbd: -31.91% (1.7801 => 2.3481)
>  multisource/Applications/kimwitu++/kc: -18.78% (32.6580 => 38.7903)
>  multisource/Applications/minisat/minisat: -72.05% (1.2160 => 2.0921)
>  multisource/Benchmarks/PAQ8p/paq8p: -144.14% (3.2082 => 7.8324)
>  multisource/Benchmarks/Trimaran/enc-3des/enc-3des: -9.06% (2.6041 => 2.8401)
>  multisource/Benchmarks/tramp3d-v4/tramp3d-v4: -195.05% (22.2893 => 65.7641)
> CBE:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: 10.56% (3.22 => 2.88)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 69.10% (9.71 => 3.00)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -382.67% (0.75 => 3.62)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 18.66% (11.90 => 9.68)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -1811.11% (0.54 => 10.32)
>  singlesource/Benchmarks/Misc-C++/stepanov_v1p2: 16.90% (25.33 => 21.05)
>  singlesource/Benchmarks/Misc/ffbench: -17.61% (3.18 => 3.74)
>  singlesource/Benchmarks/Shootout-C++/random: 5.51% (3.81 => 3.60)
>  singlesource/Benchmarks/Shootout/hash: -25.67% (9.70 => 12.19)
>  singlesource/Benchmarks/Shootout/methcall: 23.51% (7.74 => 5.92)
>  multisource/Applications/lambda-0.1.3/lambda: 18.80% (8.67 => 7.04)
>  multisource/Applications/minisat/minisat: 13.07% (24.25 => 21.08)
>  multisource/Benchmarks/ASCI_Purple/SMG2000/smg2000: -7.93% (10.84 => 11.70)
> JIT:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -7.34% (14.30 => 15.35)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 36.05% (25.05 => 16.02)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -381.65% (1.09 => 5.25)
>  singlesource/Benchmarks/CoyoteBench/lpbench: -25.22% (15.11 => 18.92)
>  singlesource/Benchmarks/Misc-C++/ray: 5.34% (7.87 => 7.45)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 14.86% (6.46 => 5.50)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -1698.18% (0.55 => 9.89)
>  singlesource/Benchmarks/Shootout-C++/objinst: 5.25% (7.81 => 7.40)
>  singlesource/Benchmarks/Shootout/methcall: -16.47% (8.56 => 9.97)
>  multisource/Applications/aha/aha: 59.87% (15.40 => 6.18)
>  multisource/Applications/hexxagon/hexxagon: 9.50% (15.27 => 13.82)
>  multisource/Benchmarks/PAQ8p/paq8p: 5.16% (173.51 => 164.55)
> LLC:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: 14.13% (3.75 => 3.22)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 69.51% (6.92 => 2.11)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -563.16% (0.57 => 3.78)
>  singlesource/Benchmarks/Misc-C++/ray: 6.45% (7.60 => 7.11)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 12.32% (6.09 => 5.34)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -2438.89% (0.36 => 9.14)
>  singlesource/Benchmarks/Misc/ffbench: -7.76% (3.61 => 3.89)
>  multisource/Applications/lambda-0.1.3/lambda: 18.24% (9.76 => 7.98)
> LLC-BETA:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: 14.89% (3.76 => 3.20)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 69.51% (6.92 => 2.11)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -555.26% (0.76 => 4.98)
>  singlesource/Benchmarks/CoyoteBench/lpbench: 20.54% (18.99 => 15.09)
>  singlesource/Benchmarks/Misc-C++/ray: 6.58% (7.60 => 7.10)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 11.38% (6.15 => 5.45)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -2430.56% (0.36 => 9.11)
>  singlesource/Benchmarks/Misc/ffbench: 6.78% (3.54 => 3.30)
>  singlesource/Benchmarks/Misc/flops-1: -7.75% (4.00 => 4.31)
>  singlesource/Benchmarks/Shootout/ary3: 23.85% (6.96 => 5.30)
> LLC compile:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -19.52% (2.5201 => 3.0121)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -317.65% (0.0680 => 0.2840)
>  multisource/Benchmarks/PAQ8p/paq8p: -19.62% (1.8761 => 2.2441)
> LLC-BETA compile:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -18.74% (2.4761 => 2.9401)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 13.42% (1.7881 => 1.5481)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -406.67% (0.0600 => 0.3040)
>  multisource/Benchmarks/PAQ8p/paq8p: -11.42% (1.9961 => 2.2241)
> Bytecode:
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 13.35% (205736 => 178276)
>  singlesource/Benchmarks/Adobe-C++/stepanov_abstraction: -24.39% (29780 => 37044)
>  singlesource/Benchmarks/Adobe-C++/stepanov_vector: -50.45% (29196 => 43924)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -7.53% (13596 => 14620)
>  singlesource/Benchmarks/Misc-C++/bigfib: -26.93% (19936 => 25304)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -333.72% (10664 => 46252)
>  singlesource/Benchmarks/Misc-C++/stepanov_v1p2: -13.34% (9056 => 10264)
>  singlesource/Benchmarks/Shootout-C++/ary: 5.35% (4560 => 4316)
>  singlesource/Benchmarks/Shootout-C++/ary3: 5.10% (4780 => 4536)
>  singlesource/Benchmarks/Shootout-C++/hash: -175.14% (3636 => 10004)
>  singlesource/Benchmarks/Shootout-C++/hash2: -217.73% (3812 => 12112)
>  singlesource/Benchmarks/Shootout-C++/lists: -47.33% (4420 => 6512)
>  singlesource/Benchmarks/Shootout-C++/lists1: -40.03% (5396 => 7556)
>  singlesource/Benchmarks/Shootout-C++/moments: -14.37% (7880 => 9012)
>  singlesource/Benchmarks/Shootout-C++/reversefile: 7.33% (10640 => 9860)
>  singlesource/Benchmarks/Shootout-C++/spellcheck: 5.53% (16196 => 15300)
>  singlesource/Benchmarks/Shootout-C++/wordfreq: -76.77% (10212 => 18052)
>  singlesource/UnitTests/2006-12-04-DynAllocAndRestore: 10.61% (528 => 472)
>  multisource/Applications/minisat/minisat: 5.46% (48116 => 45488)
>  multisource/Benchmarks/PAQ8p/paq8p: -10.47% (196084 => 216608)
>  multisource/Benchmarks/Prolangs-C++/garage/garage: 5.45% (6748 => 6380)
>  multisource/Benchmarks/tramp3d-v4/tramp3d-v4: -234.99% (687532 => 2303148)
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-testresults] cfarm-x86-64 x86_64 nightly tester results

Chris Lattner-2

On Mar 9, 2009, at 8:53 AM, Duncan Sands wrote:

> This nightly tester is now using an llvm-g++ that produces the new  
> ODR linkage
> types.  This means that many more functions are being considered by  
> the
> inter-procedural optimization passes (for example, "linkonce"  
> functions defined
> in a header).  The result seems to be pretty huge swings (both good  
> and bad) in
> the C++ tests in the testsuite, see below.  Note that this tester is  
> often under
> heavy load due to other users, so take timing results with a pinch  
> of salt (a
> pinch is about +-10% in my experience with this tester).

Awesome Duncan, thank you for working on this!  From the LLC tests, I  
see:

> LLC:
> singlesource/Benchmarks/Adobe-C++/loop_unroll: 14.13% (3.75 => 3.22)
> singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding:  
> 69.51% (6.92 => 2.11)
> singlesource/Benchmarks/CoyoteBench/fftbench: -563.16% (0.57 => 3.78)
> singlesource/Benchmarks/Misc-C++/ray: 6.45% (7.60 => 7.11)
> singlesource/Benchmarks/Misc-C++/sphereflake: 12.32% (6.09 => 5.34)
> singlesource/Benchmarks/Misc-C++/stepanov_container: -2438.89% (0.36  
> => 9.14)
> singlesource/Benchmarks/Misc/ffbench: -7.76% (3.61 => 3.89)
> multisource/Applications/lambda-0.1.3/lambda: 18.24% (9.76 => 7.98)

Can you check to see if the stepanov_container/fftbench regressions  
are real?  If so, it would be very interesting to know what is "going  
wrong" on them.

-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-testresults] cfarm-x86-64 x86_64 nightly tester results

Duncan Sands
Hi Chris,

> Can you check to see if the stepanov_container/fftbench regressions  
> are real?  If so, it would be very interesting to know what is "going  
> wrong" on them.

I think these may not be real.  This version of llvm-gcc was built with
checking enabled - does this turn on checking in libstdc++?  It seems that
a bunch of linkonce libstdc++ checking code is now being inlined (presumably
because it compiled to something smaller than before).  Previously it was
not being inlined so was discarded, and the version from the system libstdc++
was used instead.  My guess is that the system libstdc++ version is faster
because it doesn't do any checking.  To test this theory I've told this
nightly tester to build llvm-gcc with checking disabled.  If I'm correct
then stepanov_container/fftbench will speed up hugely in a few days time
when the tester starts using the new llvm-gcc.

Ciao,

Duncan.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-testresults] cfarm-x86-64 x86_64 nightly tester results

Nick Lewycky
Duncan Sands wrote:

> Hi Chris,
>
>> Can you check to see if the stepanov_container/fftbench regressions  
>> are real?  If so, it would be very interesting to know what is "going  
>> wrong" on them.
>
> I think these may not be real.  This version of llvm-gcc was built with
> checking enabled - does this turn on checking in libstdc++?  It seems that
> a bunch of linkonce libstdc++ checking code is now being inlined (presumably
> because it compiled to something smaller than before).  Previously it was
> not being inlined so was discarded, and the version from the system libstdc++
> was used instead.  My guess is that the system libstdc++ version is faster
> because it doesn't do any checking.  To test this theory I've told this
> nightly tester to build llvm-gcc with checking disabled.  If I'm correct
> then stepanov_container/fftbench will speed up hugely in a few days time
> when the tester starts using the new llvm-gcc.

If it helps, my nightly tester "desire" updates its version of llvm-gcc
before every nightly test.

Nick

> Ciao,
>
> Duncan.
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-testresults] cfarm-x86-64 x86_64 nightly tester results

Chris Lattner-2
In reply to this post by Duncan Sands

On Mar 10, 2009, at 8:24 AM, Duncan Sands wrote:

> Hi Chris,
>
>> Can you check to see if the stepanov_container/fftbench regressions
>> are real?  If so, it would be very interesting to know what is "going
>> wrong" on them.
>
> I think these may not be real.  This version of llvm-gcc was built  
> with
> checking enabled - does this turn on checking in libstdc++?

I don't know if it turns on checking in libstdc++, but I would be very  
surprised by that.

> It seems that
> a bunch of linkonce libstdc++ checking code is now being inlined  
> (presumably
> because it compiled to something smaller than before).  Previously  
> it was
> not being inlined so was discarded, and the version from the system  
> libstdc++
> was used instead.  My guess is that the system libstdc++ version is  
> faster
> because it doesn't do any checking.  To test this theory I've told  
> this
> nightly tester to build llvm-gcc with checking disabled.  If I'm  
> correct
> then stepanov_container/fftbench will speed up hugely in a few days  
> time
> when the tester starts using the new llvm-gcc.

Sounds great, thanks for hunting this down Duncan!

-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev