How to compile apps to bc files with the new llvm-gcc4?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to compile apps to bc files with the new llvm-gcc4?

Domagoj Babic
Hi,

I'm trying to compile some apps with the new llvm-gcc4 on amd64 linux
to .bc files, rather than to the native code. The same process I used to
build those apps before with llvm-gcc3 doesn't work:

1)
export CC=llvm-gcc CXX=llvm-g++ CFLAGS="-g -fno-inline" CXXFLAGS="-g
-fno-inline"
2) cd <whatever-app>; ./configure
3) make CFLAGS+=-emit-llvm CXXFLAGS+=-emit-llvm
[Reid told me not to pass -emit-llvm to configure scripts.]

The process fails for the same reason on several apps I tried:

--------------------------------------------------------------
$ llvm-gcc -shared  .libs/libdspam.o .libs/heap.o .libs/diction.o
.libs/base64.o .libs/buffer.o .libs/util.o .libs/nodetree.o
.libs/error.o .libs/decode.o .libs/pref.o .libs/config_shared.o
.libs/bnr.o .libs/hash.o .libs/list.o .libs/ldap_client.o
.libs/tokenizer.o .libs/hash_drv.o  -lm -ldl  -Wl,-soname
-Wl,libdspam.so.7 -o .libs/libdspam.so.7.0.0 -v

Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../dst-directory/configure --disable-shared
--prefix=/usr/local/devapps/llvm --enable-languages=c,c++
--enable-llvm=/work/projects/llvm-obj --program-prefix=llvm-
--enable-checking --disable-multilib
Thread model: posix
gcc version 4.0.1 LLVM (Apple Computer, Inc. build 5421)
 /usr/local/devapps/llvm/libexec/gcc/x86_64-unknown-linux-gnu/4.0.1/collect2
--eh-frame-hdr -m elf_x86_64 -shared -o .libs/libdspam.so.7.0.0
/usr/lib/../lib64/crti.o
/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/crtbeginS.o
-L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1
-L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/../../../../lib64
-L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/../../..
-L/lib/../lib64 -L/usr/lib/../lib64 .libs/libdspam.o .libs/heap.o
.libs/diction.o .libs/base64.o .libs/buffer.o .libs/util.o
.libs/nodetree.o .libs/error.o .libs/decode.o .libs/pref.o
.libs/config_shared.o .libs/bnr.o .libs/hash.o .libs/list.o
.libs/ldap_client.o .libs/tokenizer.o .libs/hash_drv.o -lm -ldl
-soname libdspam.so.7 -lgcc -lc -lgcc
/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/crtendS.o
/usr/lib/../lib64/crtn.o
/usr/bin/ld:.libs/libdspam.o: file format not recognized; treating as
linker script
/usr/bin/ld:.libs/libdspam.o:1: syntax error
collect2: ld returned 1 exit status
--------------------------------------------------------------

If I execute the last command manually with llvm-ld, removing -shared
and -Wl, it goes through.


So, obviously I'm doing something wrong.

What do I need to change in the build process to compile the same apps
I could compile before with llvm-gcc3 to bc files?


Thx in advance,
            Domagoj
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: How to compile apps to bc files with the new llvm-gcc4?

Chandler Carruth
Unless I'm missing something, the problem lies directly with the fact that you are trying to do a link-stage operation with the GCC frontend. GCC, by default, probably runs "ld" or another system linker, which grabs the executable from binutils. This linker hasn't been modified (yet) to include support for .bc files, but you're compiler is emitting byte code to the *.o files. That is why the linker barfs, and tries to interpret as a script.

I am actually planning on working on integration between binutils' ld and LLVM to allow just this functionality. I'll be working on this over the holidays, and hopefully will get something working by mid January. Until then, directly using the LLVM linking facilities to handle the bytecode is probably the best if not only option.

-Chandler

PS: I'm still kinda new, so someone else may come along and correct me, but that's the best I can figure out...

On 12/10/06, Domagoj Babic <[hidden email]> wrote:
Hi,

I'm trying to compile some apps with the new llvm-gcc4 on amd64 linux
to .bc files, rather than to the native code. The same process I used to
build those apps before with llvm-gcc3 doesn't work:

1)
export CC=llvm-gcc CXX=llvm-g++ CFLAGS="-g -fno-inline" CXXFLAGS="-g
-fno-inline"
2) cd <whatever-app>; ./configure
3) make CFLAGS+=-emit-llvm CXXFLAGS+=-emit-llvm
[Reid told me not to pass -emit-llvm to configure scripts.]

The process fails for the same reason on several apps I tried:

--------------------------------------------------------------
$ llvm-gcc -shared  .libs/libdspam.o .libs/heap.o .libs/diction.o
.libs/base64.o .libs/buffer.o .libs/util.o .libs/nodetree.o
.libs/error.o .libs/decode.o .libs/pref.o .libs/config_shared.o
.libs/bnr.o .libs/hash.o .libs/list.o .libs/ldap_client.o
.libs/tokenizer.o .libs/hash_drv.o  -lm -ldl  -Wl,-soname
-Wl,libdspam.so.7 -o .libs/libdspam.so.7.0.0 -v

Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../dst-directory/configure --disable-shared
--prefix=/usr/local/devapps/llvm --enable-languages=c,c++
--enable-llvm=/work/projects/llvm-obj --program-prefix=llvm-
--enable-checking --disable-multilib
Thread model: posix
gcc version 4.0.1 LLVM (Apple Computer, Inc. build 5421)
/usr/local/devapps/llvm/libexec/gcc/x86_64-unknown-linux-gnu/4.0.1/collect2
--eh-frame-hdr -m elf_x86_64 -shared -o .libs/libdspam.so.7.0.0
/usr/lib/../lib64/crti.o
/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/crtbeginS.o
-L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1
-L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/../../../../lib64
-L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/../../..
-L/lib/../lib64 -L/usr/lib/../lib64 .libs/libdspam.o .libs/heap.o
.libs/diction.o .libs/base64.o .libs/buffer.o .libs/util.o
.libs/nodetree.o .libs/error.o .libs/decode.o .libs/pref.o
.libs/config_shared.o .libs/bnr.o .libs/hash.o .libs/list.o
.libs/ldap_client.o .libs/tokenizer.o .libs/hash_drv.o -lm -ldl
-soname libdspam.so.7 -lgcc -lc -lgcc
/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/crtendS.o
/usr/lib/../lib64/crtn.o
/usr/bin/ld:.libs/libdspam.o: file format not recognized; treating as
linker script
/usr/bin/ld:.libs/libdspam.o:1: syntax error
collect2: ld returned 1 exit status
--------------------------------------------------------------

If I execute the last command manually with llvm-ld, removing -shared
and -Wl, it goes through.


So, obviously I'm doing something wrong.

What do I need to change in the build process to compile the same apps
I could compile before with llvm-gcc3 to bc files?


Thx in advance,
            Domagoj
_______________________________________________
LLVM Developers mailing list
[hidden email]         <a href="http://llvm.cs.uiuc.edu" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://llvm.cs.uiuc.edu
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: How to compile apps to bc files with the new llvm-gcc4?

Reid Spencer-2
Chandler/Domagoj,

On Sun, 2006-12-10 at 10:53 -0500, Chandler Carruth wrote:
> Unless I'm missing something, the problem lies directly with the fact
> that you are trying to do a link-stage operation with the GCC
> frontend. GCC, by default, probably runs "ld" or another system
> linker, which grabs the executable from binutils. This linker hasn't
> been modified (yet) to include support for .bc files, but you're
> compiler is emitting byte code to the *.o files. That is why the
> linker barfs, and tries to interpret as a script.

Correct. In a discussion with Domagoj on IRC, it turns out that he
actually doesn't want to link a native executable. He just wants to get
a single linked .bc file for each program. llvm-gcc3 used to do this
because it linked with gccld which produces a .bc file and a shell
script for running that bc file with lli. I suggested to him that he
change his makefiles to simply link the llvm-gcc4 produced bytecode
files with gccld as did llvm-gcc3.

> I am actually planning on working on integration between binutils' ld
> and LLVM to allow just this functionality. I'll be working on this
> over the holidays, and hopefully will get something working by mid
> January. Until then, directly using the LLVM linking facilities to
> handle the bytecode is probably the best if not only option.

We look forward to LTO in binutils. This will be a great addition to
LLVM (and binutils) that will make using bytecode more transparent than
ever.

>
> -Chandler
>
> PS: I'm still kinda new, so someone else may come along and correct
> me, but that's the best I can figure out...

You got it right, Chandler :)

Reid.

>
> On 12/10/06, Domagoj Babic <[hidden email]> wrote:
>         Hi,
>        
>         I'm trying to compile some apps with the new llvm-gcc4 on
>         amd64 linux
>         to .bc files, rather than to the native code. The same process
>         I used to
>         build those apps before with llvm-gcc3 doesn't work:
>        
>         1)
>         export CC=llvm-gcc CXX=llvm-g++ CFLAGS="-g -fno-inline"
>         CXXFLAGS="-g
>         -fno-inline"
>         2) cd <whatever-app>; ./configure
>         3) make CFLAGS+=-emit-llvm CXXFLAGS+=-emit-llvm
>         [Reid told me not to pass -emit-llvm to configure scripts.]
>        
>         The process fails for the same reason on several apps I tried:
>        
>         --------------------------------------------------------------
>         $ llvm-gcc
>         -shared  .libs/libdspam.o .libs/heap.o .libs/diction.o
>         .libs/base64.o .libs/buffer.o .libs/util.o .libs/nodetree.o
>         .libs/error.o .libs/decode.o .libs/pref.o .libs/config_shared.o
>         .libs/bnr.o .libs/hash.o .libs/list.o .libs/ldap_client.o
>         .libs/tokenizer.o .libs/hash_drv.o  -lm -ldl  -Wl,-soname
>         -Wl,libdspam.so.7 -o .libs/libdspam.so.7.0.0 -v
>        
>         Using built-in specs.
>         Target: x86_64-unknown-linux-gnu
>         Configured with: ../dst-directory/configure --disable-shared
>         --prefix=/usr/local/devapps/llvm --enable-languages=c,c++
>         --enable-llvm=/work/projects/llvm-obj --program-prefix=llvm-
>         --enable-checking --disable-multilib
>         Thread model: posix
>         gcc version 4.0.1 LLVM (Apple Computer, Inc. build 5421)
>         /usr/local/devapps/llvm/libexec/gcc/x86_64-unknown-linux-gnu/4.0.1/collect2
>         --eh-frame-hdr -m elf_x86_64 -shared
>         -o .libs/libdspam.so.7.0.0
>         /usr/lib/../lib64/crti.o
>         /usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/crtbeginS.o
>         -L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1
>         -L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/../../../../lib64
>         -L/usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/../../..
>         -L/lib/../lib64
>         -L/usr/lib/../lib64 .libs/libdspam.o .libs/heap.o
>         .libs/diction.o .libs/base64.o .libs/buffer.o .libs/util.o
>         .libs/nodetree.o .libs/error.o .libs/decode.o .libs/pref.o
>         .libs/config_shared.o .libs/bnr.o .libs/hash.o .libs/list.o
>         .libs/ldap_client.o .libs/tokenizer.o .libs/hash_drv.o -lm
>         -ldl
>         -soname libdspam.so.7 -lgcc -lc -lgcc
>         /usr/local/devapps/llvm/lib/gcc/x86_64-unknown-linux-gnu/4.0.1/crtendS.o
>         /usr/lib/../lib64/crtn.o
>         /usr/bin/ld:.libs/libdspam.o: file format not recognized;
>         treating as
>         linker script
>         /usr/bin/ld:.libs/libdspam.o:1: syntax error
>         collect2: ld returned 1 exit status
>         --------------------------------------------------------------
>        
>         If I execute the last command manually with llvm-ld, removing
>         -shared
>         and -Wl, it goes through.
>        
>        
>         So, obviously I'm doing something wrong.
>        
>         What do I need to change in the build process to compile the
>         same apps
>         I could compile before with llvm-gcc3 to bc files?
>        
>        
>         Thx in advance,
>                     Domagoj
>         _______________________________________________
>         LLVM Developers mailing list
>         [hidden email]         http://llvm.cs.uiuc.edu
>         http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: How to compile apps to bc files with the new llvm-gcc4?

Scott Michel
Reid Spencer wrote:

> Chandler/Domagoj,
>
> On Sun, 2006-12-10 at 10:53 -0500, Chandler Carruth wrote:
>
>>Unless I'm missing something, the problem lies directly with the fact
>>that you are trying to do a link-stage operation with the GCC
>>frontend. GCC, by default, probably runs "ld" or another system
>>linker, which grabs the executable from binutils. This linker hasn't
>>been modified (yet) to include support for .bc files, but you're
>>compiler is emitting byte code to the *.o files. That is why the
>>linker barfs, and tries to interpret as a script.
>
>
> Correct. In a discussion with Domagoj on IRC, it turns out that he
> actually doesn't want to link a native executable. He just wants to get
> a single linked .bc file for each program. llvm-gcc3 used to do this
> because it linked with gccld which produces a .bc file and a shell
> script for running that bc file with lli. I suggested to him that he
> change his makefiles to simply link the llvm-gcc4 produced bytecode
> files with gccld as did llvm-gcc3.

Here's where my issue about DTRT with llvm-gcc (or whatever your prefix
to llvm tools happens to be) resurfaces. I'm keenly aware that Chris
thinks that one really ought to be careful with having llvm-gcc emit
bytecode. But frankly, if that's the case, what's the point of emitting
byte code in the first place?

It might be a good interim step to hack the frontend so that llvm-ld
gets invoked (and I'm aware of problems like passing '-gcse' to the ld
phase when the frontend collects the arguments.) Even if emitting
bytecode only happens with "-O4", you still run into the same problem at
link time. Unless you override LD in your makefile or some other hack,
which, BTW, doesn't necessarily work either.

>>I am actually planning on working on integration between binutils' ld
>>and LLVM to allow just this functionality. I'll be working on this
>>over the holidays, and hopefully will get something working by mid
>>January. Until then, directly using the LLVM linking facilities to
>>handle the bytecode is probably the best if not only option.
>
>
> We look forward to LTO in binutils. This will be a great addition to
> LLVM (and binutils) that will make using bytecode more transparent than
> ever.

While that would be cool, it still remains that they potentially aren't
going to be installed in the right place for gcc to pick them up, or
worse, you get mismatches between the distro's concept of binutils vs.
llvm-gcc's concept of binutils. That becomes the headache that I suspect
we're all trying to avoid.
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: How to compile apps to bc files with the new llvm-gcc4?

Domagoj Babic
Hi,

On 12/11/06, Scott Michel <[hidden email]> wrote:
> Here's where my issue about DTRT with llvm-gcc (or whatever your prefix
> to llvm tools happens to be) resurfaces. I'm keenly aware that Chris
> thinks that one really ought to be careful with having llvm-gcc emit
> bytecode. But frankly, if that's the case, what's the point of emitting
> byte code in the first place?

For my application, byte-code emission was the crucial reason why I
decided to go with LLVM rather than other front-ends. In general, I
think I made a good decision, but when I need to spend 2 days to compile
a simple app to bytecode files or a week to fix code after
ISA/types/streams/interface change, it gets me thinking....

> It might be a good interim step to hack the frontend so that llvm-ld
> gets invoked (and I'm aware of problems like passing '-gcse' to the ld
> phase when the frontend collects the arguments.) Even if emitting
> bytecode only happens with "-O4", you still run into the same problem at
> link time. Unless you override LD in your makefile or some other hack,
> which, BTW, doesn't necessarily work either.

I hacked a replacement for llvm-collect2 (explained below), but such
hackery is really not a way to go in the long term. I'm very hopeful
that Chandler's work will solve the problem for good.


Hack explained:
1) rename
llvm/libexec/gcc/x86_64-unknown-linux-gnu/4.0.1/collect2
to
collect2-llvm
2) put the attached script collect2 in the same directory
3) export CC=llvm-gcc CXX=llvm-g++

The script will call gccld if -enable-gccld is passed. So you can
configure without the switch, and make with something like:

make CFLAGS+=-emit-llvm CXXFLAGS+=-emit-llvm LDFLAGS+=-enable-gccld
[If your Makefile doesn't handle LDFLAGS properly, use -Wl,-enable-gccld.]

Possible outcome:
$ ./a.out
bash: ./a.out: No such file or directory
$ file a.out
a.out: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.6.0, dynamically linked (uses shared libs), for GNU/Linux
2.6.0, not stripped
$ ls -l a.out
-rwxr-xr-x 1 <login> <login> 180422 2006-12-10 16:25 a.out

If you do
$ strings a.out | grep \.so

on the code compiled with llvm-collect2 and llvm-collect2 called by the
script, you might notice a difference in the libraries it gets linked
with, so in my case it links with a library that doesn't exist on the
system, probably the script doesn't handle paths/LD_LIBRARY_PATH
correctly. At that point I didn't have the patience to try to figure
exactly what's going on, so I just added a symbolic link:
$ ln -s /lib64/ld-linux-x86-64.so.2 /lib/ld64.so.1

[ collect2 patches welcome ;-) ]

I successfully compiled 20 or so apps, some fairly large ones (like wine)
with the script.


Finally, I need to mention that all the credit for the idea to write
such script goes to Reid. All the bugs are exclusively mine :-)

Domagoj

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

collect2.bz2 (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How to compile apps to bc files with the new llvm-gcc4?

Scott Michel
Domagoj Babic wrote:
> I hacked a replacement for llvm-collect2 (explained below), but such
> hackery is really not a way to go in the long term. I'm very hopeful
> that Chandler's work will solve the problem for good.

Interim (pontoon) bridges are the way forward for most successful
campaigns; often, you can't wait for the engineers to erect the
permanent bridge if you're looking to make progress. If it works for the
Navy and the Marine Corps, it'll probably work for us. :-)

Actually, this may be a much cleaner interim solution that the hack I
was planning inside the gcc4. It does require that the frontend's
gcc/gcc.c LINKER_NAME define be set appropriately. Getting that set
correctly might be an interesting configure hack, but easier than
hacking command specs in gcc/gcc.c.


-scooter
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev