Adding SQLite3 to llvm test-suite

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding SQLite3 to llvm test-suite

Török Edwin
Hi,

SQLite3 has a very simple build system, and it comes with an extensive
testsuite (over 40000 tests).
It hasn't shown any bugs in LLVM, but it is fairly simple to build [even
turn off features via -D], and can be CPU intensive.

I have integrated SQLite3 into llvm-test's build system, it is too large
to send as attachment (784K), you can get it from here:
http://edwintorok.googlepages.com/SQLite_llvm_test.tar.gz

The testsuite is using tcl, and needs tcl headers available at
build-time so I didn't add that [they are mostly I/O bound anyway]
Instead I found a "speedtest" script in the sqlite distribution. I
modified it to generate sql files instead of directly running sqlite.

The results are here, LLVM is ~20% slower than GCC:

Program         | GCCAS  Bytecode LLC compile LLC-BETA compile JIT
codegen | GCC     CBE     LLC     LLC-BETA JIT     | GCC/CBE GCC/LLC
GCC/LLC-BETA LLC/LLC-BETA
lemon/lemon     | 0.3833 149404   0.5699      *              
0.5166      |    0.04    0.04    0.05 *           0.62 | -       -      
n/a          n/a
sqlite3/sqlite3 | 4.8063 1744588  5.3996      *              
3.7497      |    7.09    8.19    8.78 *          13.68 | 0.87    0.81  
n/a          n/a

There are more speedtest scripts in its testsuite, but the generated sql
is large (>10 Mb), and I think the above shows a clear difference
between LLVM and GCC.

Can somebody please review and commit this to llvm-test?

Best regards,
--Edwin

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Adding SQLite3 to llvm test-suite

Evan Cheng-2
Hi Edwin,

It works fine for me. Thanks!

We prefer a flat directory structure. Is it possible for you to  
separate it out to sqlite3 and lemon rather than having them as sub-
directories under SQLite? Also, can you increase lemon's test size?

Please file bug about the performance issue. Thanks!

Evan

On Mar 22, 2008, at 8:20 AM, Török Edwin wrote:

> Hi,
>
> SQLite3 has a very simple build system, and it comes with an extensive
> testsuite (over 40000 tests).
> It hasn't shown any bugs in LLVM, but it is fairly simple to build  
> [even
> turn off features via -D], and can be CPU intensive.
>
> I have integrated SQLite3 into llvm-test's build system, it is too  
> large
> to send as attachment (784K), you can get it from here:
> http://edwintorok.googlepages.com/SQLite_llvm_test.tar.gz
>
> The testsuite is using tcl, and needs tcl headers available at
> build-time so I didn't add that [they are mostly I/O bound anyway]
> Instead I found a "speedtest" script in the sqlite distribution. I
> modified it to generate sql files instead of directly running sqlite.
>
> The results are here, LLVM is ~20% slower than GCC:
>
> Program         | GCCAS  Bytecode LLC compile LLC-BETA compile JIT
> codegen | GCC     CBE     LLC     LLC-BETA JIT     | GCC/CBE GCC/LLC
> GCC/LLC-BETA LLC/LLC-BETA
> lemon/lemon     | 0.3833 149404   0.5699      *
> 0.5166      |    0.04    0.04    0.05 *           0.62 | -       -
> n/a          n/a
> sqlite3/sqlite3 | 4.8063 1744588  5.3996      *
> 3.7497      |    7.09    8.19    8.78 *          13.68 | 0.87    0.81
> n/a          n/a
>
> There are more speedtest scripts in its testsuite, but the generated  
> sql
> is large (>10 Mb), and I think the above shows a clear difference
> between LLVM and GCC.
>
> Can somebody please review and commit this to llvm-test?
>
> Best regards,
> --Edwin
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Adding SQLite3 to llvm test-suite

Török Edwin
Evan Cheng wrote:
> Hi Edwin,
>  

Hi Evan,

> It works fine for me. Thanks!
>
> We prefer a flat directory structure. Is it possible for you to  
> separate it out to sqlite3 and lemon rather than having them as sub-
> directories under SQLite?

Ok.
The new package is here:
http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
Anything else I should change?

>  Also, can you increase lemon's test size?
>  

Lemon doesn't have a testsuite (those 40000+ tests are all for SQLite),
however I found some projects that use lemon (its syntax is not
compatible with yacc), and copied their grammar files into lemon's
directory.
I also wrote a wrapper main function for lemon that forks/executes for
each file (lemon keeps some state in global variables, so I have to fork).

The difference isn't much, it now takes 0.12 seconds to run lemon.

> Please file bug about the performance issue. Thanks!
>  

Done, bug #2174.

Best regards,
--Edwin
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Adding SQLite3 to llvm test-suite

Evan Cheng-2

On Mar 25, 2008, at 12:56 PM, Török Edwin wrote:

> Evan Cheng wrote:
>> Hi Edwin,
>>
>
> Hi Evan,
>
>> It works fine for me. Thanks!
>>
>> We prefer a flat directory structure. Is it possible for you to
>> separate it out to sqlite3 and lemon rather than having them as sub-
>> directories under SQLite?
>
> Ok.
> The new package is here:
> http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
> Anything else I should change?

Looks good. Please commit.

>
>
>> Also, can you increase lemon's test size?
>>
>
> Lemon doesn't have a testsuite (those 40000+ tests are all for  
> SQLite),
> however I found some projects that use lemon (its syntax is not
> compatible with yacc), and copied their grammar files into lemon's
> directory.
> I also wrote a wrapper main function for lemon that forks/executes for
> each file (lemon keeps some state in global variables, so I have to  
> fork).
>
> The difference isn't much, it now takes 0.12 seconds to run lemon.

Still much too small to be really useful as a benchmark. Can you just  
run it 10 times?

Evan

>
>
>> Please file bug about the performance issue. Thanks!
>>
>
> Done, bug #2174.
>
> Best regards,
> --Edwin
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Adding SQLite3 to llvm test-suite

Török Edwin
On Wed, Mar 26, 2008 at 4:09 AM, Evan Cheng <[hidden email]> wrote:
>
>  On Mar 25, 2008, at 12:56 PM, Török Edwin wrote:
>  > The new package is here:
>  > http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
>  > Anything else I should change?
>
>  Looks good. Please commit.

Should I speak with Chris about commit access?

I made the change below for lemon:
http://edwintorok.googlepages.com/sqlite_lemon_llvmtest_2.tar.gz

>
>
>  >
>  > The difference isn't much, it now takes 0.12 seconds to run lemon.
>
>  Still much too small to be really useful as a benchmark. Can you just
>  run it 10 times?
>

Ok, now it takes ~1.3 seconds.

Best regards,
--Edwin

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Adding SQLite3 to llvm test-suite

Evan Cheng-2

On Mar 26, 2008, at 12:03 AM, Török Edvin wrote:

> On Wed, Mar 26, 2008 at 4:09 AM, Evan Cheng <[hidden email]>  
> wrote:
>>
>> On Mar 25, 2008, at 12:56 PM, Török Edwin wrote:
>>> The new package is here:
>>> http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
>>> Anything else I should change?
>>
>> Looks good. Please commit.
>
> Should I speak with Chris about commit access?

I've committed it for you. Ping Chris about commit access next time.

>
>
> I made the change below for lemon:
> http://edwintorok.googlepages.com/sqlite_lemon_llvmtest_2.tar.gz
>
>>
>>
>>>
>>> The difference isn't much, it now takes 0.12 seconds to run lemon.
>>
>> Still much too small to be really useful as a benchmark. Can you just
>> run it 10 times?
>>
>
> Ok, now it takes ~1.3 seconds.

Thanks. I've bumped it up  a bit more.

Evan

>
>
> Best regards,
> --Edwin
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Adding SQLite3 to llvm test-suite

Török Edwin
Evan Cheng wrote:
> On Mar 26, 2008, at 12:03 AM, Török Edvin wrote:
>  
>> Should I speak with Chris about commit access?
>>    
>
> I've committed it for you. Ping Chris about commit access next time.

Thanks!

--Edwin
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev