llvm-abi: A library for generating ABI-compliant LLVM IR

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

llvm-abi: A library for generating ABI-compliant LLVM IR

Stephen Cross
Hi everyone,

(Also CC'ed cfe-dev since this seems relevant to Clang, particularly
the questions at the end.)

I've been working on a library to generate LLVM IR that complies with
platform ABIs (the current focus is on C but I'm also interested in
ABIs for other languages).

You can find it here: https://github.com/scross99/llvm-abi

To explain further (for those who are unfamiliar), LLVM frontends have
to modify function argument types, attributes etc. in order to ensure
the backend generates code that satisfies the ABI; this is needed
because LLVM's type system can't encode all the necessary information.
This is a complex task and involves substantial amounts of
target-dependent logic. Clang performs this encoding and indeed much
of the current functionality is derived from Clang's source.

This project originated as a necessary piece of functionality for the
Loci compiler frontend [1] and is now an external dependency; my aim
is to make this usable for other LLVM frontends that also need to
generate ABI-compliant IR (I assume this is a fairly large subset of
the frontends).

I'd be very interested in any suggestions/queries/comments.

I made a few interesting discoveries while working on this:

* Clang generates 8 byte alignment for 16+ byte arrays on x86-64, even
though the AMD64 ABI seems to require that arrays of 16+ bytes are
aligned to 16 bytes. Is this a bug or am I missing something obvious?

* Clang determines the features for a CPU (e.g. whether we have AVX
support on x86-64 CPUs), even though this functionality is already
available in LLVM (but appears to be very difficult to query). Would
it be possible to expose the information from LLVM and hence eliminate
the duplication in Clang?

* Clang determines a 'generic CPU' if the user doesn't specify a CPU.
My understanding is that we don't usually generate code for the native
CPU because it may have features unavailable on other similar CPUs.
LLVM can provide full details of the native CPU but can't determine a
generic CPU; could this functionality be added to LLVM?

(Separately I've also been considering a proposal to add ABI
information directly inside LLVM IR in a language-independent way and
I'll discuss this in a later email.)

Thanks,
Stephen

[1] https://github.com/scross99/locic
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Reid Kleckner-2
I agree it would be really nice to build a library for ABI lowering, but any solution that isn't clang or isn't ultimately picked up by clang will necessarily be incomplete. Perhaps that is OK for your frontend's uses, but I think it's the main reason that we haven't done something like this in LLVM already.

Any solution that doesn't involve actual Clang ASTs is unlikely to be able to represent all C-with-extensions types (unions, bitfields, alignment attributes, transparent_union attribute). I took a look at Type.hpp in your project, and it seems to be missing some of these things. Keeping such a library up to date with new extensions is going to be a maintenance burden.

That said, I wish you luck, and I hope the project eases some of the difficulties for new frontends. It is very possible that the corner cases that keep me up at night are not the problems that users actually face. :)

On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross <[hidden email]> wrote:
Hi everyone,

(Also CC'ed cfe-dev since this seems relevant to Clang, particularly
the questions at the end.)

I've been working on a library to generate LLVM IR that complies with
platform ABIs (the current focus is on C but I'm also interested in
ABIs for other languages).

You can find it here: https://github.com/scross99/llvm-abi

To explain further (for those who are unfamiliar), LLVM frontends have
to modify function argument types, attributes etc. in order to ensure
the backend generates code that satisfies the ABI; this is needed
because LLVM's type system can't encode all the necessary information.
This is a complex task and involves substantial amounts of
target-dependent logic. Clang performs this encoding and indeed much
of the current functionality is derived from Clang's source.

This project originated as a necessary piece of functionality for the
Loci compiler frontend [1] and is now an external dependency; my aim
is to make this usable for other LLVM frontends that also need to
generate ABI-compliant IR (I assume this is a fairly large subset of
the frontends).

I'd be very interested in any suggestions/queries/comments.

I made a few interesting discoveries while working on this:

* Clang generates 8 byte alignment for 16+ byte arrays on x86-64, even
though the AMD64 ABI seems to require that arrays of 16+ bytes are
aligned to 16 bytes. Is this a bug or am I missing something obvious?

* Clang determines the features for a CPU (e.g. whether we have AVX
support on x86-64 CPUs), even though this functionality is already
available in LLVM (but appears to be very difficult to query). Would
it be possible to expose the information from LLVM and hence eliminate
the duplication in Clang?

* Clang determines a 'generic CPU' if the user doesn't specify a CPU.
My understanding is that we don't usually generate code for the native
CPU because it may have features unavailable on other similar CPUs.
LLVM can provide full details of the native CPU but can't determine a
generic CPU; could this functionality be added to LLVM?

(Separately I've also been considering a proposal to add ABI
information directly inside LLVM IR in a language-independent way and
I'll discuss this in a later email.)

Thanks,
Stephen

[1] https://github.com/scross99/locic
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Stephen Cross
Hi Reid,

Thanks for your response.

The issue is that every LLVM frontend needing ABI compliance has to
re-implement the same target-dependent logic, which is a significant
burden; the ABI compliance code inside Clang isn't really usable for
other frontends as-is. We haven't got many good options here :-). I
think a lot of people would've hoped that LLVM would provide the means
for achieving ABI compliance (at least for C), though I'm well aware
of the complexity involved and understand the decisions taken thus
far.

Within this context, llvm-abi is an immediately actionable way of
sharing code between frontends, that should lead to a higher quality
codebase for a significantly reduced effort on everyone's part. It
seems like this would further build on LLVM's success because I think
many people would like to generate code that can interact with C APIs.

Ultimately it would be great to see this functionality be provided in
an accessible form inside LLVM and hence for Clang to use that
functionality. This would move much of the target-dependent logic out
of Clang while at the same time making this functionality available to
other LLVM-based tools.

I am sure this is a substantial and complex long term project, but I
think it is a worthwhile aim. Do you (and the LLVM community as a
whole) agree? (Acknowledging that some of the details would need to be
worked out.)

I have a long term and substantial interest in this (as I expect do
other frontend developers), so I'm willing to contribute significant
effort to move this forward. Hopefully the llvm-abi library can
provide a better understanding of what needs to be represented and
help non-Clang frontends :-).

Implementation-wise the changes to LLVM could involve encoding ABI
information into LLVM IR or simply providing C++ APIs for generating
ABI-compliant code inside LLVM (much like llvm-abi). Each solution has
advantages and disadvantages and I'm planning to make a proposal about
this on the mailing list later (hence I'd suggest leaving that
discussion until I make the proposal).

In terms of the mentioned deficiencies of the Type structure, these
are all problems that I intend to address (it would be great if you
could provide details on any of the more difficult areas).

Thanks again,
Stephen

On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]> wrote:

> I agree it would be really nice to build a library for ABI lowering, but any
> solution that isn't clang or isn't ultimately picked up by clang will
> necessarily be incomplete. Perhaps that is OK for your frontend's uses, but
> I think it's the main reason that we haven't done something like this in
> LLVM already.
>
> Any solution that doesn't involve actual Clang ASTs is unlikely to be able
> to represent all C-with-extensions types (unions, bitfields, alignment
> attributes, transparent_union attribute). I took a look at Type.hpp in your
> project, and it seems to be missing some of these things. Keeping such a
> library up to date with new extensions is going to be a maintenance burden.
>
> That said, I wish you luck, and I hope the project eases some of the
> difficulties for new frontends. It is very possible that the corner cases
> that keep me up at night are not the problems that users actually face. :)
>
> On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross <[hidden email]> wrote:
>>
>> Hi everyone,
>>
>> (Also CC'ed cfe-dev since this seems relevant to Clang, particularly
>> the questions at the end.)
>>
>> I've been working on a library to generate LLVM IR that complies with
>> platform ABIs (the current focus is on C but I'm also interested in
>> ABIs for other languages).
>>
>> You can find it here: https://github.com/scross99/llvm-abi
>>
>> To explain further (for those who are unfamiliar), LLVM frontends have
>> to modify function argument types, attributes etc. in order to ensure
>> the backend generates code that satisfies the ABI; this is needed
>> because LLVM's type system can't encode all the necessary information.
>> This is a complex task and involves substantial amounts of
>> target-dependent logic. Clang performs this encoding and indeed much
>> of the current functionality is derived from Clang's source.
>>
>> This project originated as a necessary piece of functionality for the
>> Loci compiler frontend [1] and is now an external dependency; my aim
>> is to make this usable for other LLVM frontends that also need to
>> generate ABI-compliant IR (I assume this is a fairly large subset of
>> the frontends).
>>
>> I'd be very interested in any suggestions/queries/comments.
>>
>> I made a few interesting discoveries while working on this:
>>
>> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64, even
>> though the AMD64 ABI seems to require that arrays of 16+ bytes are
>> aligned to 16 bytes. Is this a bug or am I missing something obvious?
>>
>> * Clang determines the features for a CPU (e.g. whether we have AVX
>> support on x86-64 CPUs), even though this functionality is already
>> available in LLVM (but appears to be very difficult to query). Would
>> it be possible to expose the information from LLVM and hence eliminate
>> the duplication in Clang?
>>
>> * Clang determines a 'generic CPU' if the user doesn't specify a CPU.
>> My understanding is that we don't usually generate code for the native
>> CPU because it may have features unavailable on other similar CPUs.
>> LLVM can provide full details of the native CPU but can't determine a
>> generic CPU; could this functionality be added to LLVM?
>>
>> (Separately I've also been considering a proposal to add ABI
>> information directly inside LLVM IR in a language-independent way and
>> I'll discuss this in a later email.)
>>
>> Thanks,
>> Stephen
>>
>> [1] https://github.com/scross99/locic
>> _______________________________________________
>> cfe-dev mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Hal Finkel
----- Original Message -----

> From: "Stephen Cross" <[hidden email]>
> To: "Reid Kleckner" <[hidden email]>
> Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers Mailing List" <[hidden email]>
> Sent: Monday, June 29, 2015 3:03:40 PM
> Subject: Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR
>
> Hi Reid,
>
> Thanks for your response.
>
> The issue is that every LLVM frontend needing ABI compliance has to
> re-implement the same target-dependent logic, which is a significant
> burden; the ABI compliance code inside Clang isn't really usable for
> other frontends as-is. We haven't got many good options here :-). I
> think a lot of people would've hoped that LLVM would provide the
> means
> for achieving ABI compliance (at least for C), though I'm well aware
> of the complexity involved and understand the decisions taken thus
> far.
>
> Within this context, llvm-abi is an immediately actionable way of
> sharing code between frontends, that should lead to a higher quality
> codebase for a significantly reduced effort on everyone's part. It
> seems like this would further build on LLVM's success because I think
> many people would like to generate code that can interact with C
> APIs.
>
> Ultimately it would be great to see this functionality be provided in
> an accessible form inside LLVM and hence for Clang to use that
> functionality. This would move much of the target-dependent logic out
> of Clang while at the same time making this functionality available
> to
> other LLVM-based tools.
>
> I am sure this is a substantial and complex long term project, but I
> think it is a worthwhile aim. Do you (and the LLVM community as a
> whole) agree? (Acknowledging that some of the details would need to
> be
> worked out.)
>
> I have a long term and substantial interest in this (as I expect do
> other frontend developers), so I'm willing to contribute significant
> effort to move this forward. Hopefully the llvm-abi library can
> provide a better understanding of what needs to be represented and
> help non-Clang frontends :-).
>
> Implementation-wise the changes to LLVM could involve encoding ABI
> information into LLVM IR or simply providing C++ APIs for generating
> ABI-compliant code inside LLVM (much like llvm-abi). Each solution
> has
> advantages and disadvantages and I'm planning to make a proposal
> about
> this on the mailing list later (hence I'd suggest leaving that
> discussion until I make the proposal).
>
> In terms of the mentioned deficiencies of the Type structure, these
> are all problems that I intend to address (it would be great if you
> could provide details on any of the more difficult areas).

First, let me say that I also think this is an incredibly important area in definite need of a solution. However, I also think that we really need to work from Clang's logic here: We don't want to have two different implementations of the ABI rules. They're complicated (and subtle), and it is hard enough to have one correct implementation in our C/C++ compiler. Given that we're talking about the C/C++ ABI, this is a perfectly appropriate place for the logic to live. The issue, as you well know, is that other languages need to interact with C/C++ functions, and so need to understand the C/C++ ABI. My recommended strategy is this:

 1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can be used by external clients (just as Clang's AST headers are available to external clients).

 2. Write your library as a wrapper around that logic so that simple cases can be handled without burdening other frontends with the subtleties of Clang's AST data structures.

 -Hal

>
> Thanks again,
> Stephen
>
> On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
> wrote:
> > I agree it would be really nice to build a library for ABI
> > lowering, but any
> > solution that isn't clang or isn't ultimately picked up by clang
> > will
> > necessarily be incomplete. Perhaps that is OK for your frontend's
> > uses, but
> > I think it's the main reason that we haven't done something like
> > this in
> > LLVM already.
> >
> > Any solution that doesn't involve actual Clang ASTs is unlikely to
> > be able
> > to represent all C-with-extensions types (unions, bitfields,
> > alignment
> > attributes, transparent_union attribute). I took a look at Type.hpp
> > in your
> > project, and it seems to be missing some of these things. Keeping
> > such a
> > library up to date with new extensions is going to be a maintenance
> > burden.
> >
> > That said, I wish you luck, and I hope the project eases some of
> > the
> > difficulties for new frontends. It is very possible that the corner
> > cases
> > that keep me up at night are not the problems that users actually
> > face. :)
> >
> > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
> > <[hidden email]> wrote:
> >>
> >> Hi everyone,
> >>
> >> (Also CC'ed cfe-dev since this seems relevant to Clang,
> >> particularly
> >> the questions at the end.)
> >>
> >> I've been working on a library to generate LLVM IR that complies
> >> with
> >> platform ABIs (the current focus is on C but I'm also interested
> >> in
> >> ABIs for other languages).
> >>
> >> You can find it here: https://github.com/scross99/llvm-abi
> >>
> >> To explain further (for those who are unfamiliar), LLVM frontends
> >> have
> >> to modify function argument types, attributes etc. in order to
> >> ensure
> >> the backend generates code that satisfies the ABI; this is needed
> >> because LLVM's type system can't encode all the necessary
> >> information.
> >> This is a complex task and involves substantial amounts of
> >> target-dependent logic. Clang performs this encoding and indeed
> >> much
> >> of the current functionality is derived from Clang's source.
> >>
> >> This project originated as a necessary piece of functionality for
> >> the
> >> Loci compiler frontend [1] and is now an external dependency; my
> >> aim
> >> is to make this usable for other LLVM frontends that also need to
> >> generate ABI-compliant IR (I assume this is a fairly large subset
> >> of
> >> the frontends).
> >>
> >> I'd be very interested in any suggestions/queries/comments.
> >>
> >> I made a few interesting discoveries while working on this:
> >>
> >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
> >> even
> >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
> >> aligned to 16 bytes. Is this a bug or am I missing something
> >> obvious?
> >>
> >> * Clang determines the features for a CPU (e.g. whether we have
> >> AVX
> >> support on x86-64 CPUs), even though this functionality is already
> >> available in LLVM (but appears to be very difficult to query).
> >> Would
> >> it be possible to expose the information from LLVM and hence
> >> eliminate
> >> the duplication in Clang?
> >>
> >> * Clang determines a 'generic CPU' if the user doesn't specify a
> >> CPU.
> >> My understanding is that we don't usually generate code for the
> >> native
> >> CPU because it may have features unavailable on other similar
> >> CPUs.
> >> LLVM can provide full details of the native CPU but can't
> >> determine a
> >> generic CPU; could this functionality be added to LLVM?
> >>
> >> (Separately I've also been considering a proposal to add ABI
> >> information directly inside LLVM IR in a language-independent way
> >> and
> >> I'll discuss this in a later email.)
> >>
> >> Thanks,
> >> Stephen
> >>
> >> [1] https://github.com/scross99/locic
> >> _______________________________________________
> >> cfe-dev mailing list
> >> [hidden email]
> >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
> >
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>

--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Andrew Wilkins
On Tue, 30 Jun 2015 at 06:02 Hal Finkel <[hidden email]> wrote:
----- Original Message -----
> From: "Stephen Cross" <[hidden email]>
> To: "Reid Kleckner" <[hidden email]>
> Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers Mailing List" <[hidden email]>
> Sent: Monday, June 29, 2015 3:03:40 PM
> Subject: Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM  IR
>
> Hi Reid,
>
> Thanks for your response.
>
> The issue is that every LLVM frontend needing ABI compliance has to
> re-implement the same target-dependent logic, which is a significant
> burden; the ABI compliance code inside Clang isn't really usable for
> other frontends as-is. We haven't got many good options here :-). I
> think a lot of people would've hoped that LLVM would provide the
> means
> for achieving ABI compliance (at least for C), though I'm well aware
> of the complexity involved and understand the decisions taken thus
> far.
>
> Within this context, llvm-abi is an immediately actionable way of
> sharing code between frontends, that should lead to a higher quality
> codebase for a significantly reduced effort on everyone's part. It
> seems like this would further build on LLVM's success because I think
> many people would like to generate code that can interact with C
> APIs.
>
> Ultimately it would be great to see this functionality be provided in
> an accessible form inside LLVM and hence for Clang to use that
> functionality. This would move much of the target-dependent logic out
> of Clang while at the same time making this functionality available
> to
> other LLVM-based tools.
>
> I am sure this is a substantial and complex long term project, but I
> think it is a worthwhile aim. Do you (and the LLVM community as a
> whole) agree? (Acknowledging that some of the details would need to
> be
> worked out.)
>
> I have a long term and substantial interest in this (as I expect do
> other frontend developers), so I'm willing to contribute significant
> effort to move this forward. Hopefully the llvm-abi library can
> provide a better understanding of what needs to be represented and
> help non-Clang frontends :-).
>
> Implementation-wise the changes to LLVM could involve encoding ABI
> information into LLVM IR or simply providing C++ APIs for generating
> ABI-compliant code inside LLVM (much like llvm-abi). Each solution
> has
> advantages and disadvantages and I'm planning to make a proposal
> about
> this on the mailing list later (hence I'd suggest leaving that
> discussion until I make the proposal).
>
> In terms of the mentioned deficiencies of the Type structure, these
> are all problems that I intend to address (it would be great if you
> could provide details on any of the more difficult areas).

First, let me say that I also think this is an incredibly important area in definite need of a solution. However, I also think that we really need to work from Clang's logic here: We don't want to have two different implementations of the ABI rules. They're complicated (and subtle), and it is hard enough to have one correct implementation in our C/C++ compiler. Given that we're talking about the C/C++ ABI, this is a perfectly appropriate place for the logic to live. The issue, as you well know, is that other languages need to interact with C/C++ functions, and so need to understand the C/C++ ABI. My recommended strategy is this:

 1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can be used by external clients (just as Clang's AST headers are available to external clients).

 2. Write your library as a wrapper around that logic so that simple cases can be handled without burdening other frontends with the subtleties of Clang's AST data structures.

This is what I was thinking of doing for llgo in the long term. At the moment we have support for x86-64 only: http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be nice to have a simplified API to TargetInfo, ideally with a C API that other languages to can easily bind to. Stephen, if you do go down that route, I'd be keen to hear about it.

Cheers,
Andrew

 -Hal

>
> Thanks again,
> Stephen
>
> On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
> wrote:
> > I agree it would be really nice to build a library for ABI
> > lowering, but any
> > solution that isn't clang or isn't ultimately picked up by clang
> > will
> > necessarily be incomplete. Perhaps that is OK for your frontend's
> > uses, but
> > I think it's the main reason that we haven't done something like
> > this in
> > LLVM already.
> >
> > Any solution that doesn't involve actual Clang ASTs is unlikely to
> > be able
> > to represent all C-with-extensions types (unions, bitfields,
> > alignment
> > attributes, transparent_union attribute). I took a look at Type.hpp
> > in your
> > project, and it seems to be missing some of these things. Keeping
> > such a
> > library up to date with new extensions is going to be a maintenance
> > burden.
> >
> > That said, I wish you luck, and I hope the project eases some of
> > the
> > difficulties for new frontends. It is very possible that the corner
> > cases
> > that keep me up at night are not the problems that users actually
> > face. :)
> >
> > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
> > <[hidden email]> wrote:
> >>
> >> Hi everyone,
> >>
> >> (Also CC'ed cfe-dev since this seems relevant to Clang,
> >> particularly
> >> the questions at the end.)
> >>
> >> I've been working on a library to generate LLVM IR that complies
> >> with
> >> platform ABIs (the current focus is on C but I'm also interested
> >> in
> >> ABIs for other languages).
> >>
> >> You can find it here: https://github.com/scross99/llvm-abi
> >>
> >> To explain further (for those who are unfamiliar), LLVM frontends
> >> have
> >> to modify function argument types, attributes etc. in order to
> >> ensure
> >> the backend generates code that satisfies the ABI; this is needed
> >> because LLVM's type system can't encode all the necessary
> >> information.
> >> This is a complex task and involves substantial amounts of
> >> target-dependent logic. Clang performs this encoding and indeed
> >> much
> >> of the current functionality is derived from Clang's source.
> >>
> >> This project originated as a necessary piece of functionality for
> >> the
> >> Loci compiler frontend [1] and is now an external dependency; my
> >> aim
> >> is to make this usable for other LLVM frontends that also need to
> >> generate ABI-compliant IR (I assume this is a fairly large subset
> >> of
> >> the frontends).
> >>
> >> I'd be very interested in any suggestions/queries/comments.
> >>
> >> I made a few interesting discoveries while working on this:
> >>
> >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
> >> even
> >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
> >> aligned to 16 bytes. Is this a bug or am I missing something
> >> obvious?
> >>
> >> * Clang determines the features for a CPU (e.g. whether we have
> >> AVX
> >> support on x86-64 CPUs), even though this functionality is already
> >> available in LLVM (but appears to be very difficult to query).
> >> Would
> >> it be possible to expose the information from LLVM and hence
> >> eliminate
> >> the duplication in Clang?
> >>
> >> * Clang determines a 'generic CPU' if the user doesn't specify a
> >> CPU.
> >> My understanding is that we don't usually generate code for the
> >> native
> >> CPU because it may have features unavailable on other similar
> >> CPUs.
> >> LLVM can provide full details of the native CPU but can't
> >> determine a
> >> generic CPU; could this functionality be added to LLVM?
> >>
> >> (Separately I've also been considering a proposal to add ABI
> >> information directly inside LLVM IR in a language-independent way
> >> and
> >> I'll discuss this in a later email.)
> >>
> >> Thanks,
> >> Stephen
> >>
> >> [1] https://github.com/scross99/locic
> >> _______________________________________________
> >> cfe-dev mailing list
> >> [hidden email]
> >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
> >
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>

--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Stephen Cross
Hi everyone,

I was wondering if anyone could answer the questions in my first
email. These were:

* Why does Clang generate 8 byte alignment for 16+ byte arrays on
x86-64, even though the AMD64 ABI seems to require 16 byte alignment?
* Clang has some logic in lib/Basic/Targets.cpp to determine CPU
features, but LLVM also does the same task in its backend. Would it be
possible to modify LLVM to expose this logic and modify Clang to use
that?
* (Slightly modified) Clang has some logic in lib/Driver/Tools.cpp
('getCPUName') for selecting a generic CPU. Would it be sensible to
expose this in a header in include/clang/Driver/?

> Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can be used by external clients (just as Clang's AST headers are available to external clients).

Yes, this sounds like a good approach. While I'm not entirely
convinced this is where we want to end up (it doesn't seem right to
split the ABI implementation between the frontend and the backend), I
definitely agree this is a sensible starting point. Unless anyone is
against this I'll start working out the changes needed under the
assumption this is how I should proceed.

My approach would be to just refactor the code to separate classes
within CodeGen and add the necessary headers to
include/clang/CodeGen/. It might make sense to create a separate
library directory (e.g. lib/ABI) but I expect there will be circular
dependency issues to overcome. The very first thing is probably to
split each target's code out of TargetInfo.cpp; I'd suggest that the
structure of CodeGen could mirror <llvm>/lib/Target/ so we'd have
directories for ARM, PowerPC, X86 etc. What do you think?

Note that there's also functionality in CGCall.cpp (and a few other
files) which would need to be separated out to be usable for other
frontends.

> Write your library as a wrapper around that logic so that simple cases can be handled without burdening other frontends with the subtleties of Clang's AST data structures.

This will be how it ends up, but while the modifications to Clang are
ongoing llvm-abi is and will continue to be an entirely separate
ABI-implementing codebase. This is just because the Loci frontend
needs this functionality right now. I'm also expecting there will be
functionality going into llvm-abi that for good reason won't be found
in Clang. There's some duplication of effort here but it's
unavoidable.

I like the idea of frontends using the library as a convenient API and
the library's implementation using code from Clang. I know that some
frontends (including the Loci frontend) are designed to be able to be
built on systems that could have slightly older versions of LLVM/Clang
(for Loci, it's LLVM 3.3+), so this factor will have to be considered
when integrating the code from Clang.

> At the moment we have support for x86-64 only: http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go.

I'll add other architectures to llvm-abi soon (it's largely a fairly
trivial process of moving code from Clang and replacing its in-tree
dependencies) so hopefully this will be useful. I can see from other
non-Clang frontends that non-X86 support does tend to be at least
quite incomplete.

> It would be nice to have a simplified API to TargetInfo, ideally with a C API that other languages to can easily bind to.

Absolutely, I agree. I appear to be in the rare situation of having a
frontend written in C++ (i.e. not self-hosting) but I know most
frontends rely on having a C API.

It seems like the approach to this should be to have a C API in
llvm-abi that wraps around its C++ API. Unfortunately my time is
currently fairly thin-spread; would you be able to contribute to this
effort?

Thanks,
Stephen

On Tue, Jun 30, 2015 at 12:47 AM, Andrew Wilkins <[hidden email]> wrote:

> On Tue, 30 Jun 2015 at 06:02 Hal Finkel <[hidden email]> wrote:
>>
>> ----- Original Message -----
>> > From: "Stephen Cross" <[hidden email]>
>> > To: "Reid Kleckner" <[hidden email]>
>> > Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers
>> > Mailing List" <[hidden email]>
>> > Sent: Monday, June 29, 2015 3:03:40 PM
>> > Subject: Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant
>> > LLVM  IR
>> >
>> > Hi Reid,
>> >
>> > Thanks for your response.
>> >
>> > The issue is that every LLVM frontend needing ABI compliance has to
>> > re-implement the same target-dependent logic, which is a significant
>> > burden; the ABI compliance code inside Clang isn't really usable for
>> > other frontends as-is. We haven't got many good options here :-). I
>> > think a lot of people would've hoped that LLVM would provide the
>> > means
>> > for achieving ABI compliance (at least for C), though I'm well aware
>> > of the complexity involved and understand the decisions taken thus
>> > far.
>> >
>> > Within this context, llvm-abi is an immediately actionable way of
>> > sharing code between frontends, that should lead to a higher quality
>> > codebase for a significantly reduced effort on everyone's part. It
>> > seems like this would further build on LLVM's success because I think
>> > many people would like to generate code that can interact with C
>> > APIs.
>> >
>> > Ultimately it would be great to see this functionality be provided in
>> > an accessible form inside LLVM and hence for Clang to use that
>> > functionality. This would move much of the target-dependent logic out
>> > of Clang while at the same time making this functionality available
>> > to
>> > other LLVM-based tools.
>> >
>> > I am sure this is a substantial and complex long term project, but I
>> > think it is a worthwhile aim. Do you (and the LLVM community as a
>> > whole) agree? (Acknowledging that some of the details would need to
>> > be
>> > worked out.)
>> >
>> > I have a long term and substantial interest in this (as I expect do
>> > other frontend developers), so I'm willing to contribute significant
>> > effort to move this forward. Hopefully the llvm-abi library can
>> > provide a better understanding of what needs to be represented and
>> > help non-Clang frontends :-).
>> >
>> > Implementation-wise the changes to LLVM could involve encoding ABI
>> > information into LLVM IR or simply providing C++ APIs for generating
>> > ABI-compliant code inside LLVM (much like llvm-abi). Each solution
>> > has
>> > advantages and disadvantages and I'm planning to make a proposal
>> > about
>> > this on the mailing list later (hence I'd suggest leaving that
>> > discussion until I make the proposal).
>> >
>> > In terms of the mentioned deficiencies of the Type structure, these
>> > are all problems that I intend to address (it would be great if you
>> > could provide details on any of the more difficult areas).
>>
>> First, let me say that I also think this is an incredibly important area
>> in definite need of a solution. However, I also think that we really need to
>> work from Clang's logic here: We don't want to have two different
>> implementations of the ABI rules. They're complicated (and subtle), and it
>> is hard enough to have one correct implementation in our C/C++ compiler.
>> Given that we're talking about the C/C++ ABI, this is a perfectly
>> appropriate place for the logic to live. The issue, as you well know, is
>> that other languages need to interact with C/C++ functions, and so need to
>> understand the C/C++ ABI. My recommended strategy is this:
>>
>>  1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it
>> can be used by external clients (just as Clang's AST headers are available
>> to external clients).
>>
>>  2. Write your library as a wrapper around that logic so that simple cases
>> can be handled without burdening other frontends with the subtleties of
>> Clang's AST data structures.
>
>
> This is what I was thinking of doing for llgo in the long term. At the
> moment we have support for x86-64 only:
> http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be nice to
> have a simplified API to TargetInfo, ideally with a C API that other
> languages to can easily bind to. Stephen, if you do go down that route, I'd
> be keen to hear about it.
>
> Cheers,
> Andrew
>
>>  -Hal
>>
>> >
>> > Thanks again,
>> > Stephen
>> >
>> > On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
>> > wrote:
>> > > I agree it would be really nice to build a library for ABI
>> > > lowering, but any
>> > > solution that isn't clang or isn't ultimately picked up by clang
>> > > will
>> > > necessarily be incomplete. Perhaps that is OK for your frontend's
>> > > uses, but
>> > > I think it's the main reason that we haven't done something like
>> > > this in
>> > > LLVM already.
>> > >
>> > > Any solution that doesn't involve actual Clang ASTs is unlikely to
>> > > be able
>> > > to represent all C-with-extensions types (unions, bitfields,
>> > > alignment
>> > > attributes, transparent_union attribute). I took a look at Type.hpp
>> > > in your
>> > > project, and it seems to be missing some of these things. Keeping
>> > > such a
>> > > library up to date with new extensions is going to be a maintenance
>> > > burden.
>> > >
>> > > That said, I wish you luck, and I hope the project eases some of
>> > > the
>> > > difficulties for new frontends. It is very possible that the corner
>> > > cases
>> > > that keep me up at night are not the problems that users actually
>> > > face. :)
>> > >
>> > > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
>> > > <[hidden email]> wrote:
>> > >>
>> > >> Hi everyone,
>> > >>
>> > >> (Also CC'ed cfe-dev since this seems relevant to Clang,
>> > >> particularly
>> > >> the questions at the end.)
>> > >>
>> > >> I've been working on a library to generate LLVM IR that complies
>> > >> with
>> > >> platform ABIs (the current focus is on C but I'm also interested
>> > >> in
>> > >> ABIs for other languages).
>> > >>
>> > >> You can find it here: https://github.com/scross99/llvm-abi
>> > >>
>> > >> To explain further (for those who are unfamiliar), LLVM frontends
>> > >> have
>> > >> to modify function argument types, attributes etc. in order to
>> > >> ensure
>> > >> the backend generates code that satisfies the ABI; this is needed
>> > >> because LLVM's type system can't encode all the necessary
>> > >> information.
>> > >> This is a complex task and involves substantial amounts of
>> > >> target-dependent logic. Clang performs this encoding and indeed
>> > >> much
>> > >> of the current functionality is derived from Clang's source.
>> > >>
>> > >> This project originated as a necessary piece of functionality for
>> > >> the
>> > >> Loci compiler frontend [1] and is now an external dependency; my
>> > >> aim
>> > >> is to make this usable for other LLVM frontends that also need to
>> > >> generate ABI-compliant IR (I assume this is a fairly large subset
>> > >> of
>> > >> the frontends).
>> > >>
>> > >> I'd be very interested in any suggestions/queries/comments.
>> > >>
>> > >> I made a few interesting discoveries while working on this:
>> > >>
>> > >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
>> > >> even
>> > >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
>> > >> aligned to 16 bytes. Is this a bug or am I missing something
>> > >> obvious?
>> > >>
>> > >> * Clang determines the features for a CPU (e.g. whether we have
>> > >> AVX
>> > >> support on x86-64 CPUs), even though this functionality is already
>> > >> available in LLVM (but appears to be very difficult to query).
>> > >> Would
>> > >> it be possible to expose the information from LLVM and hence
>> > >> eliminate
>> > >> the duplication in Clang?
>> > >>
>> > >> * Clang determines a 'generic CPU' if the user doesn't specify a
>> > >> CPU.
>> > >> My understanding is that we don't usually generate code for the
>> > >> native
>> > >> CPU because it may have features unavailable on other similar
>> > >> CPUs.
>> > >> LLVM can provide full details of the native CPU but can't
>> > >> determine a
>> > >> generic CPU; could this functionality be added to LLVM?
>> > >>
>> > >> (Separately I've also been considering a proposal to add ABI
>> > >> information directly inside LLVM IR in a language-independent way
>> > >> and
>> > >> I'll discuss this in a later email.)
>> > >>
>> > >> Thanks,
>> > >> Stephen
>> > >>
>> > >> [1] https://github.com/scross99/locic
>> > >> _______________________________________________
>> > >> cfe-dev mailing list
>> > >> [hidden email]
>> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> > >
>> > >
>> > _______________________________________________
>> > cfe-dev mailing list
>> > [hidden email]
>> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >
>>
>> --
>> Hal Finkel
>> Assistant Computational Scientist
>> Leadership Computing Facility
>> Argonne National Laboratory
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Hal Finkel
----- Original Message -----

> From: "Stephen Cross" <[hidden email]>
> To: "Andrew Wilkins" <[hidden email]>, "Hal Finkel" <[hidden email]>
> Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers Mailing List" <[hidden email]>
> Sent: Tuesday, June 30, 2015 7:19:10 PM
> Subject: Re: [LLVMdev] [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR
>
> Hi everyone,
>
> I was wondering if anyone could answer the questions in my first
> email. These were:
>
> * Why does Clang generate 8 byte alignment for 16+ byte arrays on
> x86-64, even though the AMD64 ABI seems to require 16 byte alignment?
> * Clang has some logic in lib/Basic/Targets.cpp to determine CPU
> features, but LLVM also does the same task in its backend. Would it
> be
> possible to modify LLVM to expose this logic and modify Clang to use
> that?

The problem is that the mapping is not direct, and a lot of the code in Clang is dealing with legacy from how GCC sets target features. We try to keep the LLVM backend features free from that kind of stuff.

 -Hal

> * (Slightly modified) Clang has some logic in lib/Driver/Tools.cpp
> ('getCPUName') for selecting a generic CPU. Would it be sensible to
> expose this in a header in include/clang/Driver/?
>
> > Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it
> > can be used by external clients (just as Clang's AST headers are
> > available to external clients).
>
> Yes, this sounds like a good approach. While I'm not entirely
> convinced this is where we want to end up (it doesn't seem right to
> split the ABI implementation between the frontend and the backend), I
> definitely agree this is a sensible starting point. Unless anyone is
> against this I'll start working out the changes needed under the
> assumption this is how I should proceed.
>
> My approach would be to just refactor the code to separate classes
> within CodeGen and add the necessary headers to
> include/clang/CodeGen/. It might make sense to create a separate
> library directory (e.g. lib/ABI) but I expect there will be circular
> dependency issues to overcome. The very first thing is probably to
> split each target's code out of TargetInfo.cpp; I'd suggest that the
> structure of CodeGen could mirror <llvm>/lib/Target/ so we'd have
> directories for ARM, PowerPC, X86 etc. What do you think?
>
> Note that there's also functionality in CGCall.cpp (and a few other
> files) which would need to be separated out to be usable for other
> frontends.
>
> > Write your library as a wrapper around that logic so that simple
> > cases can be handled without burdening other frontends with the
> > subtleties of Clang's AST data structures.
>
> This will be how it ends up, but while the modifications to Clang are
> ongoing llvm-abi is and will continue to be an entirely separate
> ABI-implementing codebase. This is just because the Loci frontend
> needs this functionality right now. I'm also expecting there will be
> functionality going into llvm-abi that for good reason won't be found
> in Clang. There's some duplication of effort here but it's
> unavoidable.
>
> I like the idea of frontends using the library as a convenient API
> and
> the library's implementation using code from Clang. I know that some
> frontends (including the Loci frontend) are designed to be able to be
> built on systems that could have slightly older versions of
> LLVM/Clang
> (for Loci, it's LLVM 3.3+), so this factor will have to be considered
> when integrating the code from Clang.
>
> > At the moment we have support for x86-64 only:
> > http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go.
>
> I'll add other architectures to llvm-abi soon (it's largely a fairly
> trivial process of moving code from Clang and replacing its in-tree
> dependencies) so hopefully this will be useful. I can see from other
> non-Clang frontends that non-X86 support does tend to be at least
> quite incomplete.
>
> > It would be nice to have a simplified API to TargetInfo, ideally
> > with a C API that other languages to can easily bind to.
>
> Absolutely, I agree. I appear to be in the rare situation of having a
> frontend written in C++ (i.e. not self-hosting) but I know most
> frontends rely on having a C API.
>
> It seems like the approach to this should be to have a C API in
> llvm-abi that wraps around its C++ API. Unfortunately my time is
> currently fairly thin-spread; would you be able to contribute to this
> effort?
>
> Thanks,
> Stephen
>
> On Tue, Jun 30, 2015 at 12:47 AM, Andrew Wilkins <[hidden email]>
> wrote:
> > On Tue, 30 Jun 2015 at 06:02 Hal Finkel <[hidden email]> wrote:
> >>
> >> ----- Original Message -----
> >> > From: "Stephen Cross" <[hidden email]>
> >> > To: "Reid Kleckner" <[hidden email]>
> >> > Cc: "Clang Developers List" <[hidden email]>, "LLVM
> >> > Developers
> >> > Mailing List" <[hidden email]>
> >> > Sent: Monday, June 29, 2015 3:03:40 PM
> >> > Subject: Re: [cfe-dev] llvm-abi: A library for generating
> >> > ABI-compliant
> >> > LLVM  IR
> >> >
> >> > Hi Reid,
> >> >
> >> > Thanks for your response.
> >> >
> >> > The issue is that every LLVM frontend needing ABI compliance has
> >> > to
> >> > re-implement the same target-dependent logic, which is a
> >> > significant
> >> > burden; the ABI compliance code inside Clang isn't really usable
> >> > for
> >> > other frontends as-is. We haven't got many good options here
> >> > :-). I
> >> > think a lot of people would've hoped that LLVM would provide the
> >> > means
> >> > for achieving ABI compliance (at least for C), though I'm well
> >> > aware
> >> > of the complexity involved and understand the decisions taken
> >> > thus
> >> > far.
> >> >
> >> > Within this context, llvm-abi is an immediately actionable way
> >> > of
> >> > sharing code between frontends, that should lead to a higher
> >> > quality
> >> > codebase for a significantly reduced effort on everyone's part.
> >> > It
> >> > seems like this would further build on LLVM's success because I
> >> > think
> >> > many people would like to generate code that can interact with C
> >> > APIs.
> >> >
> >> > Ultimately it would be great to see this functionality be
> >> > provided in
> >> > an accessible form inside LLVM and hence for Clang to use that
> >> > functionality. This would move much of the target-dependent
> >> > logic out
> >> > of Clang while at the same time making this functionality
> >> > available
> >> > to
> >> > other LLVM-based tools.
> >> >
> >> > I am sure this is a substantial and complex long term project,
> >> > but I
> >> > think it is a worthwhile aim. Do you (and the LLVM community as
> >> > a
> >> > whole) agree? (Acknowledging that some of the details would need
> >> > to
> >> > be
> >> > worked out.)
> >> >
> >> > I have a long term and substantial interest in this (as I expect
> >> > do
> >> > other frontend developers), so I'm willing to contribute
> >> > significant
> >> > effort to move this forward. Hopefully the llvm-abi library can
> >> > provide a better understanding of what needs to be represented
> >> > and
> >> > help non-Clang frontends :-).
> >> >
> >> > Implementation-wise the changes to LLVM could involve encoding
> >> > ABI
> >> > information into LLVM IR or simply providing C++ APIs for
> >> > generating
> >> > ABI-compliant code inside LLVM (much like llvm-abi). Each
> >> > solution
> >> > has
> >> > advantages and disadvantages and I'm planning to make a proposal
> >> > about
> >> > this on the mailing list later (hence I'd suggest leaving that
> >> > discussion until I make the proposal).
> >> >
> >> > In terms of the mentioned deficiencies of the Type structure,
> >> > these
> >> > are all problems that I intend to address (it would be great if
> >> > you
> >> > could provide details on any of the more difficult areas).
> >>
> >> First, let me say that I also think this is an incredibly
> >> important area
> >> in definite need of a solution. However, I also think that we
> >> really need to
> >> work from Clang's logic here: We don't want to have two different
> >> implementations of the ABI rules. They're complicated (and
> >> subtle), and it
> >> is hard enough to have one correct implementation in our C/C++
> >> compiler.
> >> Given that we're talking about the C/C++ ABI, this is a perfectly
> >> appropriate place for the logic to live. The issue, as you well
> >> know, is
> >> that other languages need to interact with C/C++ functions, and so
> >> need to
> >> understand the C/C++ ABI. My recommended strategy is this:
> >>
> >>  1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so
> >>  that it
> >> can be used by external clients (just as Clang's AST headers are
> >> available
> >> to external clients).
> >>
> >>  2. Write your library as a wrapper around that logic so that
> >>  simple cases
> >> can be handled without burdening other frontends with the
> >> subtleties of
> >> Clang's AST data structures.
> >
> >
> > This is what I was thinking of doing for llgo in the long term. At
> > the
> > moment we have support for x86-64 only:
> > http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be
> > nice to
> > have a simplified API to TargetInfo, ideally with a C API that
> > other
> > languages to can easily bind to. Stephen, if you do go down that
> > route, I'd
> > be keen to hear about it.
> >
> > Cheers,
> > Andrew
> >
> >>  -Hal
> >>
> >> >
> >> > Thanks again,
> >> > Stephen
> >> >
> >> > On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
> >> > wrote:
> >> > > I agree it would be really nice to build a library for ABI
> >> > > lowering, but any
> >> > > solution that isn't clang or isn't ultimately picked up by
> >> > > clang
> >> > > will
> >> > > necessarily be incomplete. Perhaps that is OK for your
> >> > > frontend's
> >> > > uses, but
> >> > > I think it's the main reason that we haven't done something
> >> > > like
> >> > > this in
> >> > > LLVM already.
> >> > >
> >> > > Any solution that doesn't involve actual Clang ASTs is
> >> > > unlikely to
> >> > > be able
> >> > > to represent all C-with-extensions types (unions, bitfields,
> >> > > alignment
> >> > > attributes, transparent_union attribute). I took a look at
> >> > > Type.hpp
> >> > > in your
> >> > > project, and it seems to be missing some of these things.
> >> > > Keeping
> >> > > such a
> >> > > library up to date with new extensions is going to be a
> >> > > maintenance
> >> > > burden.
> >> > >
> >> > > That said, I wish you luck, and I hope the project eases some
> >> > > of
> >> > > the
> >> > > difficulties for new frontends. It is very possible that the
> >> > > corner
> >> > > cases
> >> > > that keep me up at night are not the problems that users
> >> > > actually
> >> > > face. :)
> >> > >
> >> > > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
> >> > > <[hidden email]> wrote:
> >> > >>
> >> > >> Hi everyone,
> >> > >>
> >> > >> (Also CC'ed cfe-dev since this seems relevant to Clang,
> >> > >> particularly
> >> > >> the questions at the end.)
> >> > >>
> >> > >> I've been working on a library to generate LLVM IR that
> >> > >> complies
> >> > >> with
> >> > >> platform ABIs (the current focus is on C but I'm also
> >> > >> interested
> >> > >> in
> >> > >> ABIs for other languages).
> >> > >>
> >> > >> You can find it here: https://github.com/scross99/llvm-abi
> >> > >>
> >> > >> To explain further (for those who are unfamiliar), LLVM
> >> > >> frontends
> >> > >> have
> >> > >> to modify function argument types, attributes etc. in order
> >> > >> to
> >> > >> ensure
> >> > >> the backend generates code that satisfies the ABI; this is
> >> > >> needed
> >> > >> because LLVM's type system can't encode all the necessary
> >> > >> information.
> >> > >> This is a complex task and involves substantial amounts of
> >> > >> target-dependent logic. Clang performs this encoding and
> >> > >> indeed
> >> > >> much
> >> > >> of the current functionality is derived from Clang's source.
> >> > >>
> >> > >> This project originated as a necessary piece of functionality
> >> > >> for
> >> > >> the
> >> > >> Loci compiler frontend [1] and is now an external dependency;
> >> > >> my
> >> > >> aim
> >> > >> is to make this usable for other LLVM frontends that also
> >> > >> need to
> >> > >> generate ABI-compliant IR (I assume this is a fairly large
> >> > >> subset
> >> > >> of
> >> > >> the frontends).
> >> > >>
> >> > >> I'd be very interested in any suggestions/queries/comments.
> >> > >>
> >> > >> I made a few interesting discoveries while working on this:
> >> > >>
> >> > >> * Clang generates 8 byte alignment for 16+ byte arrays on
> >> > >> x86-64,
> >> > >> even
> >> > >> though the AMD64 ABI seems to require that arrays of 16+
> >> > >> bytes are
> >> > >> aligned to 16 bytes. Is this a bug or am I missing something
> >> > >> obvious?
> >> > >>
> >> > >> * Clang determines the features for a CPU (e.g. whether we
> >> > >> have
> >> > >> AVX
> >> > >> support on x86-64 CPUs), even though this functionality is
> >> > >> already
> >> > >> available in LLVM (but appears to be very difficult to
> >> > >> query).
> >> > >> Would
> >> > >> it be possible to expose the information from LLVM and hence
> >> > >> eliminate
> >> > >> the duplication in Clang?
> >> > >>
> >> > >> * Clang determines a 'generic CPU' if the user doesn't
> >> > >> specify a
> >> > >> CPU.
> >> > >> My understanding is that we don't usually generate code for
> >> > >> the
> >> > >> native
> >> > >> CPU because it may have features unavailable on other similar
> >> > >> CPUs.
> >> > >> LLVM can provide full details of the native CPU but can't
> >> > >> determine a
> >> > >> generic CPU; could this functionality be added to LLVM?
> >> > >>
> >> > >> (Separately I've also been considering a proposal to add ABI
> >> > >> information directly inside LLVM IR in a language-independent
> >> > >> way
> >> > >> and
> >> > >> I'll discuss this in a later email.)
> >> > >>
> >> > >> Thanks,
> >> > >> Stephen
> >> > >>
> >> > >> [1] https://github.com/scross99/locic
> >> > >> _______________________________________________
> >> > >> cfe-dev mailing list
> >> > >> [hidden email]
> >> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >> > >
> >> > >
> >> > _______________________________________________
> >> > cfe-dev mailing list
> >> > [hidden email]
> >> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >> >
> >>
> >> --
> >> Hal Finkel
> >> Assistant Computational Scientist
> >> Leadership Computing Facility
> >> Argonne National Laboratory
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> [hidden email]         http://llvm.cs.uiuc.edu
> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Andrew Wilkins
In reply to this post by Stephen Cross
On Wed, 1 Jul 2015 at 08:19 Stephen Cross <[hidden email]> wrote:
Hi everyone,

I was wondering if anyone could answer the questions in my first
email. These were:

* Why does Clang generate 8 byte alignment for 16+ byte arrays on
x86-64, even though the AMD64 ABI seems to require 16 byte alignment?
* Clang has some logic in lib/Basic/Targets.cpp to determine CPU
features, but LLVM also does the same task in its backend. Would it be
possible to modify LLVM to expose this logic and modify Clang to use
that?
* (Slightly modified) Clang has some logic in lib/Driver/Tools.cpp
('getCPUName') for selecting a generic CPU. Would it be sensible to
expose this in a header in include/clang/Driver/?

> Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can be used by external clients (just as Clang's AST headers are available to external clients).

Yes, this sounds like a good approach. While I'm not entirely
convinced this is where we want to end up (it doesn't seem right to
split the ABI implementation between the frontend and the backend), I
definitely agree this is a sensible starting point. Unless anyone is
against this I'll start working out the changes needed under the
assumption this is how I should proceed.

My approach would be to just refactor the code to separate classes
within CodeGen and add the necessary headers to
include/clang/CodeGen/. It might make sense to create a separate
library directory (e.g. lib/ABI) but I expect there will be circular
dependency issues to overcome. The very first thing is probably to
split each target's code out of TargetInfo.cpp; I'd suggest that the
structure of CodeGen could mirror <llvm>/lib/Target/ so we'd have
directories for ARM, PowerPC, X86 etc. What do you think?

Note that there's also functionality in CGCall.cpp (and a few other
files) which would need to be separated out to be usable for other
frontends.

> Write your library as a wrapper around that logic so that simple cases can be handled without burdening other frontends with the subtleties of Clang's AST data structures.

This will be how it ends up, but while the modifications to Clang are
ongoing llvm-abi is and will continue to be an entirely separate
ABI-implementing codebase. This is just because the Loci frontend
needs this functionality right now. I'm also expecting there will be
functionality going into llvm-abi that for good reason won't be found
in Clang. There's some duplication of effort here but it's
unavoidable.

I like the idea of frontends using the library as a convenient API and
the library's implementation using code from Clang. I know that some
frontends (including the Loci frontend) are designed to be able to be
built on systems that could have slightly older versions of LLVM/Clang
(for Loci, it's LLVM 3.3+), so this factor will have to be considered
when integrating the code from Clang.

> At the moment we have support for x86-64 only: http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go.

I'll add other architectures to llvm-abi soon (it's largely a fairly
trivial process of moving code from Clang and replacing its in-tree
dependencies) so hopefully this will be useful. I can see from other
non-Clang frontends that non-X86 support does tend to be at least
quite incomplete.

> It would be nice to have a simplified API to TargetInfo, ideally with a C API that other languages to can easily bind to.

Absolutely, I agree. I appear to be in the rare situation of having a
frontend written in C++ (i.e. not self-hosting) but I know most
frontends rely on having a C API.

It seems like the approach to this should be to have a C API in
llvm-abi that wraps around its C++ API. Unfortunately my time is
currently fairly thin-spread; would you be able to contribute to this
effort?

Sounds good. My time is likewise thinly spread, but I'll be happy to contribute to the effort when I can.

Cheers,
Andrew
 
Thanks,
Stephen

On Tue, Jun 30, 2015 at 12:47 AM, Andrew Wilkins <[hidden email]> wrote:
> On Tue, 30 Jun 2015 at 06:02 Hal Finkel <[hidden email]> wrote:
>>
>> ----- Original Message -----
>> > From: "Stephen Cross" <[hidden email]>
>> > To: "Reid Kleckner" <[hidden email]>
>> > Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers
>> > Mailing List" <[hidden email]>
>> > Sent: Monday, June 29, 2015 3:03:40 PM
>> > Subject: Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant
>> > LLVM  IR
>> >
>> > Hi Reid,
>> >
>> > Thanks for your response.
>> >
>> > The issue is that every LLVM frontend needing ABI compliance has to
>> > re-implement the same target-dependent logic, which is a significant
>> > burden; the ABI compliance code inside Clang isn't really usable for
>> > other frontends as-is. We haven't got many good options here :-). I
>> > think a lot of people would've hoped that LLVM would provide the
>> > means
>> > for achieving ABI compliance (at least for C), though I'm well aware
>> > of the complexity involved and understand the decisions taken thus
>> > far.
>> >
>> > Within this context, llvm-abi is an immediately actionable way of
>> > sharing code between frontends, that should lead to a higher quality
>> > codebase for a significantly reduced effort on everyone's part. It
>> > seems like this would further build on LLVM's success because I think
>> > many people would like to generate code that can interact with C
>> > APIs.
>> >
>> > Ultimately it would be great to see this functionality be provided in
>> > an accessible form inside LLVM and hence for Clang to use that
>> > functionality. This would move much of the target-dependent logic out
>> > of Clang while at the same time making this functionality available
>> > to
>> > other LLVM-based tools.
>> >
>> > I am sure this is a substantial and complex long term project, but I
>> > think it is a worthwhile aim. Do you (and the LLVM community as a
>> > whole) agree? (Acknowledging that some of the details would need to
>> > be
>> > worked out.)
>> >
>> > I have a long term and substantial interest in this (as I expect do
>> > other frontend developers), so I'm willing to contribute significant
>> > effort to move this forward. Hopefully the llvm-abi library can
>> > provide a better understanding of what needs to be represented and
>> > help non-Clang frontends :-).
>> >
>> > Implementation-wise the changes to LLVM could involve encoding ABI
>> > information into LLVM IR or simply providing C++ APIs for generating
>> > ABI-compliant code inside LLVM (much like llvm-abi). Each solution
>> > has
>> > advantages and disadvantages and I'm planning to make a proposal
>> > about
>> > this on the mailing list later (hence I'd suggest leaving that
>> > discussion until I make the proposal).
>> >
>> > In terms of the mentioned deficiencies of the Type structure, these
>> > are all problems that I intend to address (it would be great if you
>> > could provide details on any of the more difficult areas).
>>
>> First, let me say that I also think this is an incredibly important area
>> in definite need of a solution. However, I also think that we really need to
>> work from Clang's logic here: We don't want to have two different
>> implementations of the ABI rules. They're complicated (and subtle), and it
>> is hard enough to have one correct implementation in our C/C++ compiler.
>> Given that we're talking about the C/C++ ABI, this is a perfectly
>> appropriate place for the logic to live. The issue, as you well know, is
>> that other languages need to interact with C/C++ functions, and so need to
>> understand the C/C++ ABI. My recommended strategy is this:
>>
>>  1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it
>> can be used by external clients (just as Clang's AST headers are available
>> to external clients).
>>
>>  2. Write your library as a wrapper around that logic so that simple cases
>> can be handled without burdening other frontends with the subtleties of
>> Clang's AST data structures.
>
>
> This is what I was thinking of doing for llgo in the long term. At the
> moment we have support for x86-64 only:
> http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be nice to
> have a simplified API to TargetInfo, ideally with a C API that other
> languages to can easily bind to. Stephen, if you do go down that route, I'd
> be keen to hear about it.
>
> Cheers,
> Andrew
>
>>  -Hal
>>
>> >
>> > Thanks again,
>> > Stephen
>> >
>> > On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
>> > wrote:
>> > > I agree it would be really nice to build a library for ABI
>> > > lowering, but any
>> > > solution that isn't clang or isn't ultimately picked up by clang
>> > > will
>> > > necessarily be incomplete. Perhaps that is OK for your frontend's
>> > > uses, but
>> > > I think it's the main reason that we haven't done something like
>> > > this in
>> > > LLVM already.
>> > >
>> > > Any solution that doesn't involve actual Clang ASTs is unlikely to
>> > > be able
>> > > to represent all C-with-extensions types (unions, bitfields,
>> > > alignment
>> > > attributes, transparent_union attribute). I took a look at Type.hpp
>> > > in your
>> > > project, and it seems to be missing some of these things. Keeping
>> > > such a
>> > > library up to date with new extensions is going to be a maintenance
>> > > burden.
>> > >
>> > > That said, I wish you luck, and I hope the project eases some of
>> > > the
>> > > difficulties for new frontends. It is very possible that the corner
>> > > cases
>> > > that keep me up at night are not the problems that users actually
>> > > face. :)
>> > >
>> > > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
>> > > <[hidden email]> wrote:
>> > >>
>> > >> Hi everyone,
>> > >>
>> > >> (Also CC'ed cfe-dev since this seems relevant to Clang,
>> > >> particularly
>> > >> the questions at the end.)
>> > >>
>> > >> I've been working on a library to generate LLVM IR that complies
>> > >> with
>> > >> platform ABIs (the current focus is on C but I'm also interested
>> > >> in
>> > >> ABIs for other languages).
>> > >>
>> > >> You can find it here: https://github.com/scross99/llvm-abi
>> > >>
>> > >> To explain further (for those who are unfamiliar), LLVM frontends
>> > >> have
>> > >> to modify function argument types, attributes etc. in order to
>> > >> ensure
>> > >> the backend generates code that satisfies the ABI; this is needed
>> > >> because LLVM's type system can't encode all the necessary
>> > >> information.
>> > >> This is a complex task and involves substantial amounts of
>> > >> target-dependent logic. Clang performs this encoding and indeed
>> > >> much
>> > >> of the current functionality is derived from Clang's source.
>> > >>
>> > >> This project originated as a necessary piece of functionality for
>> > >> the
>> > >> Loci compiler frontend [1] and is now an external dependency; my
>> > >> aim
>> > >> is to make this usable for other LLVM frontends that also need to
>> > >> generate ABI-compliant IR (I assume this is a fairly large subset
>> > >> of
>> > >> the frontends).
>> > >>
>> > >> I'd be very interested in any suggestions/queries/comments.
>> > >>
>> > >> I made a few interesting discoveries while working on this:
>> > >>
>> > >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
>> > >> even
>> > >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
>> > >> aligned to 16 bytes. Is this a bug or am I missing something
>> > >> obvious?
>> > >>
>> > >> * Clang determines the features for a CPU (e.g. whether we have
>> > >> AVX
>> > >> support on x86-64 CPUs), even though this functionality is already
>> > >> available in LLVM (but appears to be very difficult to query).
>> > >> Would
>> > >> it be possible to expose the information from LLVM and hence
>> > >> eliminate
>> > >> the duplication in Clang?
>> > >>
>> > >> * Clang determines a 'generic CPU' if the user doesn't specify a
>> > >> CPU.
>> > >> My understanding is that we don't usually generate code for the
>> > >> native
>> > >> CPU because it may have features unavailable on other similar
>> > >> CPUs.
>> > >> LLVM can provide full details of the native CPU but can't
>> > >> determine a
>> > >> generic CPU; could this functionality be added to LLVM?
>> > >>
>> > >> (Separately I've also been considering a proposal to add ABI
>> > >> information directly inside LLVM IR in a language-independent way
>> > >> and
>> > >> I'll discuss this in a later email.)
>> > >>
>> > >> Thanks,
>> > >> Stephen
>> > >>
>> > >> [1] https://github.com/scross99/locic
>> > >> _______________________________________________
>> > >> cfe-dev mailing list
>> > >> [hidden email]
>> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> > >
>> > >
>> > _______________________________________________
>> > cfe-dev mailing list
>> > [hidden email]
>> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >
>>
>> --
>> Hal Finkel
>> Assistant Computational Scientist
>> Leadership Computing Facility
>> Argonne National Laboratory
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Pat Gavlin

Chiming in from the LLILC project:


Overall, the idea of an ABI support library sounds great. It'd certainly save us a fair amount of duplicated effort in our frontend, as the CoreCLR ABI (mostly) corresonds to the native C ABI of its host platform. What we have today[1, 2] is largely inspired by Clang, and so looks pretty similar to Stephen's existing work in llvm-abi.


I like the idea of frontends using the library as a convenient API and
> the library's implementation using code from Clang.

Although I understand the motivations behind wrapping the Clang implementation, I'm a bit leery of this as a long-term direction. Admittedly, this is selfishly motivated: LLILC currently has no Clang dependency, and I think we'd prefer to avoid taking one if possible. A reasonable compromise would be the refactoring Stephen suggested below (i.e. moving ABI-related code into its own library within Clang): 

My approach would be to just refactor the code to separate classes
> within CodeGen and add the necessary headers to
> include/clang/CodeGen/. It might make sense to create a separate
> library directory (e.g. lib/ABI) but I expect there will be circular
> dependency issues to overcome.

Although the refactoring could be tricky, this would at least let external users limit their dependency to the new library rather than pulling in libCodeGen.

Thanks,
- Pat

[1] https://github.com/dotnet/llilc/blob/master/include/Reader/abi.h
[2] https://github.com/dotnet/llilc/blob/master/lib/Reader/abi.cpp


From: [hidden email] <[hidden email]> on behalf of Andrew Wilkins <[hidden email]>
Sent: Tuesday, June 30, 2015 6:24 PM
To: Stephen Cross; Hal Finkel
Cc: Clang Developers List; LLVM Developers Mailing List
Subject: Re: [LLVMdev] [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR
 
On Wed, 1 Jul 2015 at 08:19 Stephen Cross <[hidden email]> wrote:
Hi everyone,

I was wondering if anyone could answer the questions in my first
email. These were:

* Why does Clang generate 8 byte alignment for 16+ byte arrays on
x86-64, even though the AMD64 ABI seems to require 16 byte alignment?
* Clang has some logic in lib/Basic/Targets.cpp to determine CPU
features, but LLVM also does the same task in its backend. Would it be
possible to modify LLVM to expose this logic and modify Clang to use
that?
* (Slightly modified) Clang has some logic in lib/Driver/Tools.cpp
('getCPUName') for selecting a generic CPU. Would it be sensible to
expose this in a header in include/clang/Driver/?

> Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can be used by external clients (just as Clang's AST headers are available to external clients).

Yes, this sounds like a good approach. While I'm not entirely
convinced this is where we want to end up (it doesn't seem right to
split the ABI implementation between the frontend and the backend), I
definitely agree this is a sensible starting point. Unless anyone is
against this I'll start working out the changes needed under the
assumption this is how I should proceed.

My approach would be to just refactor the code to separate classes
within CodeGen and add the necessary headers to
include/clang/CodeGen/. It might make sense to create a separate
library directory (e.g. lib/ABI) but I expect there will be circular
dependency issues to overcome. The very first thing is probably to
split each target's code out of TargetInfo.cpp; I'd suggest that the
structure of CodeGen could mirror <llvm>/lib/Target/ so we'd have
directories for ARM, PowerPC, X86 etc. What do you think?

Note that there's also functionality in CGCall.cpp (and a few other
files) which would need to be separated out to be usable for other
frontends.

> Write your library as a wrapper around that logic so that simple cases can be handled without burdening other frontends with the subtleties of Clang's AST data structures.

This will be how it ends up, but while the modifications to Clang are
ongoing llvm-abi is and will continue to be an entirely separate
ABI-implementing codebase. This is just because the Loci frontend
needs this functionality right now. I'm also expecting there will be
functionality going into llvm-abi that for good reason won't be found
in Clang. There's some duplication of effort here but it's
unavoidable.

I like the idea of frontends using the library as a convenient API and
the library's implementation using code from Clang. I know that some
frontends (including the Loci frontend) are designed to be able to be
built on systems that could have slightly older versions of LLVM/Clang
(for Loci, it's LLVM 3.3+), so this factor will have to be considered
when integrating the code from Clang.

> At the moment we have support for x86-64 only: http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go.

I'll add other architectures to llvm-abi soon (it's largely a fairly
trivial process of moving code from Clang and replacing its in-tree
dependencies) so hopefully this will be useful. I can see from other
non-Clang frontends that non-X86 support does tend to be at least
quite incomplete.

> It would be nice to have a simplified API to TargetInfo, ideally with a C API that other languages to can easily bind to.

Absolutely, I agree. I appear to be in the rare situation of having a
frontend written in C++ (i.e. not self-hosting) but I know most
frontends rely on having a C API.

It seems like the approach to this should be to have a C API in
llvm-abi that wraps around its C++ API. Unfortunately my time is
currently fairly thin-spread; would you be able to contribute to this
effort?

Sounds good. My time is likewise thinly spread, but I'll be happy to contribute to the effort when I can.

Cheers,
Andrew
 
Thanks,
Stephen

On Tue, Jun 30, 2015 at 12:47 AM, Andrew Wilkins <[hidden email]> wrote:
> On Tue, 30 Jun 2015 at 06:02 Hal Finkel <[hidden email]> wrote:
>>
>> ----- Original Message -----
>> > From: "Stephen Cross" <[hidden email]>
>> > To: "Reid Kleckner" <[hidden email]>
>> > Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers
>> > Mailing List" <[hidden email]>
>> > Sent: Monday, June 29, 2015 3:03:40 PM
>> > Subject: Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant
>> > LLVM  IR
>> >
>> > Hi Reid,
>> >
>> > Thanks for your response.
>> >
>> > The issue is that every LLVM frontend needing ABI compliance has to
>> > re-implement the same target-dependent logic, which is a significant
>> > burden; the ABI compliance code inside Clang isn't really usable for
>> > other frontends as-is. We haven't got many good options here :-). I
>> > think a lot of people would've hoped that LLVM would provide the
>> > means
>> > for achieving ABI compliance (at least for C), though I'm well aware
>> > of the complexity involved and understand the decisions taken thus
>> > far.
>> >
>> > Within this context, llvm-abi is an immediately actionable way of
>> > sharing code between frontends, that should lead to a higher quality
>> > codebase for a significantly reduced effort on everyone's part. It
>> > seems like this would further build on LLVM's success because I think
>> > many people would like to generate code that can interact with C
>> > APIs.
>> >
>> > Ultimately it would be great to see this functionality be provided in
>> > an accessible form inside LLVM and hence for Clang to use that
>> > functionality. This would move much of the target-dependent logic out
>> > of Clang while at the same time making this functionality available
>> > to
>> > other LLVM-based tools.
>> >
>> > I am sure this is a substantial and complex long term project, but I
>> > think it is a worthwhile aim. Do you (and the LLVM community as a
>> > whole) agree? (Acknowledging that some of the details would need to
>> > be
>> > worked out.)
>> >
>> > I have a long term and substantial interest in this (as I expect do
>> > other frontend developers), so I'm willing to contribute significant
>> > effort to move this forward. Hopefully the llvm-abi library can
>> > provide a better understanding of what needs to be represented and
>> > help non-Clang frontends :-).
>> >
>> > Implementation-wise the changes to LLVM could involve encoding ABI
>> > information into LLVM IR or simply providing C++ APIs for generating
>> > ABI-compliant code inside LLVM (much like llvm-abi). Each solution
>> > has
>> > advantages and disadvantages and I'm planning to make a proposal
>> > about
>> > this on the mailing list later (hence I'd suggest leaving that
>> > discussion until I make the proposal).
>> >
>> > In terms of the mentioned deficiencies of the Type structure, these
>> > are all problems that I intend to address (it would be great if you
>> > could provide details on any of the more difficult areas).
>>
>> First, let me say that I also think this is an incredibly important area
>> in definite need of a solution. However, I also think that we really need to
>> work from Clang's logic here: We don't want to have two different
>> implementations of the ABI rules. They're complicated (and subtle), and it
>> is hard enough to have one correct implementation in our C/C++ compiler.
>> Given that we're talking about the C/C++ ABI, this is a perfectly
>> appropriate place for the logic to live. The issue, as you well know, is
>> that other languages need to interact with C/C++ functions, and so need to
>> understand the C/C++ ABI. My recommended strategy is this:
>>
>>  1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it
>> can be used by external clients (just as Clang's AST headers are available
>> to external clients).
>>
>>  2. Write your library as a wrapper around that logic so that simple cases
>> can be handled without burdening other frontends with the subtleties of
>> Clang's AST data structures.
>
>
> This is what I was thinking of doing for llgo in the long term. At the
> moment we have support for x86-64 only:
> http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be nice to
> have a simplified API to TargetInfo, ideally with a C API that other
> languages to can easily bind to. Stephen, if you do go down that route, I'd
> be keen to hear about it.
>
> Cheers,
> Andrew
>
>>  -Hal
>>
>> >
>> > Thanks again,
>> > Stephen
>> >
>> > On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
>> > wrote:
>> > > I agree it would be really nice to build a library for ABI
>> > > lowering, but any
>> > > solution that isn't clang or isn't ultimately picked up by clang
>> > > will
>> > > necessarily be incomplete. Perhaps that is OK for your frontend's
>> > > uses, but
>> > > I think it's the main reason that we haven't done something like
>> > > this in
>> > > LLVM already.
>> > >
>> > > Any solution that doesn't involve actual Clang ASTs is unlikely to
>> > > be able
>> > > to represent all C-with-extensions types (unions, bitfields,
>> > > alignment
>> > > attributes, transparent_union attribute). I took a look at Type.hpp
>> > > in your
>> > > project, and it seems to be missing some of these things. Keeping
>> > > such a
>> > > library up to date with new extensions is going to be a maintenance
>> > > burden.
>> > >
>> > > That said, I wish you luck, and I hope the project eases some of
>> > > the
>> > > difficulties for new frontends. It is very possible that the corner
>> > > cases
>> > > that keep me up at night are not the problems that users actually
>> > > face. :)
>> > >
>> > > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
>> > > <[hidden email]> wrote:
>> > >>
>> > >> Hi everyone,
>> > >>
>> > >> (Also CC'ed cfe-dev since this seems relevant to Clang,
>> > >> particularly
>> > >> the questions at the end.)
>> > >>
>> > >> I've been working on a library to generate LLVM IR that complies
>> > >> with
>> > >> platform ABIs (the current focus is on C but I'm also interested
>> > >> in
>> > >> ABIs for other languages).
>> > >>
>> > >> You can find it here: https://github.com/scross99/llvm-abi
>> > >>
>> > >> To explain further (for those who are unfamiliar), LLVM frontends
>> > >> have
>> > >> to modify function argument types, attributes etc. in order to
>> > >> ensure
>> > >> the backend generates code that satisfies the ABI; this is needed
>> > >> because LLVM's type system can't encode all the necessary
>> > >> information.
>> > >> This is a complex task and involves substantial amounts of
>> > >> target-dependent logic. Clang performs this encoding and indeed
>> > >> much
>> > >> of the current functionality is derived from Clang's source.
>> > >>
>> > >> This project originated as a necessary piece of functionality for
>> > >> the
>> > >> Loci compiler frontend [1] and is now an external dependency; my
>> > >> aim
>> > >> is to make this usable for other LLVM frontends that also need to
>> > >> generate ABI-compliant IR (I assume this is a fairly large subset
>> > >> of
>> > >> the frontends).
>> > >>
>> > >> I'd be very interested in any suggestions/queries/comments.
>> > >>
>> > >> I made a few interesting discoveries while working on this:
>> > >>
>> > >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
>> > >> even
>> > >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
>> > >> aligned to 16 bytes. Is this a bug or am I missing something
>> > >> obvious?
>> > >>
>> > >> * Clang determines the features for a CPU (e.g. whether we have
>> > >> AVX
>> > >> support on x86-64 CPUs), even though this functionality is already
>> > >> available in LLVM (but appears to be very difficult to query).
>> > >> Would
>> > >> it be possible to expose the information from LLVM and hence
>> > >> eliminate
>> > >> the duplication in Clang?
>> > >>
>> > >> * Clang determines a 'generic CPU' if the user doesn't specify a
>> > >> CPU.
>> > >> My understanding is that we don't usually generate code for the
>> > >> native
>> > >> CPU because it may have features unavailable on other similar
>> > >> CPUs.
>> > >> LLVM can provide full details of the native CPU but can't
>> > >> determine a
>> > >> generic CPU; could this functionality be added to LLVM?
>> > >>
>> > >> (Separately I've also been considering a proposal to add ABI
>> > >> information directly inside LLVM IR in a language-independent way
>> > >> and
>> > >> I'll discuss this in a later email.)
>> > >>
>> > >> Thanks,
>> > >> Stephen
>> > >>
>> > >> [1] https://github.com/scross99/locic
>> > >> _______________________________________________
>> > >> cfe-dev mailing list
>> > >> [hidden email]
>> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> > >
>> > >
>> > _______________________________________________
>> > cfe-dev mailing list
>> > [hidden email]
>> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >
>>
>> --
>> Hal Finkel
>> Assistant Computational Scientist
>> Leadership Computing Facility
>> Argonne National Laboratory
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Stephen Cross
> Overall, the idea of an ABI support library sounds great. It'd certainly save us a fair amount of duplicated effort in our frontend, as the CoreCLR ABI (mostly) corresonds to the native C ABI of its host platform.

Yes indeed. I'm starting to wonder just how many frontends exist that
implement this logic.

Of course I'm naturally going to ask whether you'd be able to help
with any aspects of llvm-abi? :-) In particular it'd be great to have
people contributing the parts that they specifically need (e.g. it
sounds like WIndows support would be important for you).

> Although I understand the motivations behind wrapping the Clang implementation, I'm a bit leery of this as a long-term direction. Admittedly, this is selfishly motivated: LLILC currently has no Clang dependency, and I think we'd prefer to avoid taking one if possible. A reasonable compromise would be the refactoring Stephen suggested below (i.e. moving ABI-related code into its own library within Clang):

Yes, I know. It's worth pointing out that llvm-abi doesn't need to
depend on a built version of Clang, just the same source code, so we
can do fun things like Git submodules or subtrees (presumably using
the Clang Git Mirror). So we'd stay in-sync with the upstream code
from Clang but llvm-abi would actually build the relevant parts of
Clang as part of its own build. I'm not necessarily advocating this
direction but just raising it as a potential solution. It really
depends on exactly what problems you would encounter with a dependency
on Clang.

> Although the refactoring could be tricky, this would at least let external users limit their dependency to the new library rather than pulling in libCodeGen.

Yes, I'd like to do this but the current Clang ABI code depends
heavily on other code in libCodeGen. This is something that I'll
ultimately try to do but it's not currently clear exactly what
restructuring is appropriate.

On Wed, Jul 1, 2015 at 8:33 PM, Pat Gavlin <[hidden email]> wrote:

> Chiming in from the LLILC project:
>
>
> Overall, the idea of an ABI support library sounds great. It'd certainly
> save us a fair amount of duplicated effort in our frontend, as the CoreCLR
> ABI (mostly) corresonds to the native C ABI of its host platform. What we
> have today[1, 2] is largely inspired by Clang, and so looks pretty similar
> to Stephen's existing work in llvm-abi.
>
>
>> I like the idea of frontends using the library as a convenient API and
>> the library's implementation using code from Clang.
>
> Although I understand the motivations behind wrapping the Clang
> implementation, I'm a bit leery of this as a long-term direction.
> Admittedly, this is selfishly motivated: LLILC currently has no Clang
> dependency, and I think we'd prefer to avoid taking one if possible. A
> reasonable compromise would be the refactoring Stephen suggested below (i.e.
> moving ABI-related code into its own library within Clang):
>
>> My approach would be to just refactor the code to separate classes
>> within CodeGen and add the necessary headers to
>> include/clang/CodeGen/. It might make sense to create a separate
>> library directory (e.g. lib/ABI) but I expect there will be circular
>> dependency issues to overcome.
>
> Although the refactoring could be tricky, this would at least let external
> users limit their dependency to the new library rather than pulling in
> libCodeGen.
>
> Thanks,
> - Pat
>
> [1] https://github.com/dotnet/llilc/blob/master/include/Reader/abi.h
> [2] https://github.com/dotnet/llilc/blob/master/lib/Reader/abi.cpp
>
> ________________________________
> From: [hidden email] <[hidden email]> on behalf of
> Andrew Wilkins <[hidden email]>
> Sent: Tuesday, June 30, 2015 6:24 PM
> To: Stephen Cross; Hal Finkel
> Cc: Clang Developers List; LLVM Developers Mailing List
> Subject: Re: [LLVMdev] [cfe-dev] llvm-abi: A library for generating
> ABI-compliant LLVM IR
>
> On Wed, 1 Jul 2015 at 08:19 Stephen Cross <[hidden email]> wrote:
>>
>> Hi everyone,
>>
>> I was wondering if anyone could answer the questions in my first
>> email. These were:
>>
>> * Why does Clang generate 8 byte alignment for 16+ byte arrays on
>> x86-64, even though the AMD64 ABI seems to require 16 byte alignment?
>> * Clang has some logic in lib/Basic/Targets.cpp to determine CPU
>> features, but LLVM also does the same task in its backend. Would it be
>> possible to modify LLVM to expose this logic and modify Clang to use
>> that?
>> * (Slightly modified) Clang has some logic in lib/Driver/Tools.cpp
>> ('getCPUName') for selecting a generic CPU. Would it be sensible to
>> expose this in a header in include/clang/Driver/?
>>
>> > Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can
>> > be used by external clients (just as Clang's AST headers are available to
>> > external clients).
>>
>> Yes, this sounds like a good approach. While I'm not entirely
>> convinced this is where we want to end up (it doesn't seem right to
>> split the ABI implementation between the frontend and the backend), I
>> definitely agree this is a sensible starting point. Unless anyone is
>> against this I'll start working out the changes needed under the
>> assumption this is how I should proceed.
>>
>> My approach would be to just refactor the code to separate classes
>> within CodeGen and add the necessary headers to
>> include/clang/CodeGen/. It might make sense to create a separate
>> library directory (e.g. lib/ABI) but I expect there will be circular
>> dependency issues to overcome. The very first thing is probably to
>> split each target's code out of TargetInfo.cpp; I'd suggest that the
>> structure of CodeGen could mirror <llvm>/lib/Target/ so we'd have
>> directories for ARM, PowerPC, X86 etc. What do you think?
>>
>> Note that there's also functionality in CGCall.cpp (and a few other
>> files) which would need to be separated out to be usable for other
>> frontends.
>>
>> > Write your library as a wrapper around that logic so that simple cases
>> > can be handled without burdening other frontends with the subtleties of
>> > Clang's AST data structures.
>>
>> This will be how it ends up, but while the modifications to Clang are
>> ongoing llvm-abi is and will continue to be an entirely separate
>> ABI-implementing codebase. This is just because the Loci frontend
>> needs this functionality right now. I'm also expecting there will be
>> functionality going into llvm-abi that for good reason won't be found
>> in Clang. There's some duplication of effort here but it's
>> unavoidable.
>>
>> I like the idea of frontends using the library as a convenient API and
>> the library's implementation using code from Clang. I know that some
>> frontends (including the Loci frontend) are designed to be able to be
>> built on systems that could have slightly older versions of LLVM/Clang
>> (for Loci, it's LLVM 3.3+), so this factor will have to be considered
>> when integrating the code from Clang.
>>
>> > At the moment we have support for x86-64 only:
>> > http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go.
>>
>> I'll add other architectures to llvm-abi soon (it's largely a fairly
>> trivial process of moving code from Clang and replacing its in-tree
>> dependencies) so hopefully this will be useful. I can see from other
>> non-Clang frontends that non-X86 support does tend to be at least
>> quite incomplete.
>>
>> > It would be nice to have a simplified API to TargetInfo, ideally with a
>> > C API that other languages to can easily bind to.
>>
>> Absolutely, I agree. I appear to be in the rare situation of having a
>> frontend written in C++ (i.e. not self-hosting) but I know most
>> frontends rely on having a C API.
>>
>> It seems like the approach to this should be to have a C API in
>> llvm-abi that wraps around its C++ API. Unfortunately my time is
>> currently fairly thin-spread; would you be able to contribute to this
>> effort?
>
>
> Sounds good. My time is likewise thinly spread, but I'll be happy to
> contribute to the effort when I can.
>
> Cheers,
> Andrew
>
>>
>> Thanks,
>> Stephen
>>
>> On Tue, Jun 30, 2015 at 12:47 AM, Andrew Wilkins <[hidden email]> wrote:
>> > On Tue, 30 Jun 2015 at 06:02 Hal Finkel <[hidden email]> wrote:
>> >>
>> >> ----- Original Message -----
>> >> > From: "Stephen Cross" <[hidden email]>
>> >> > To: "Reid Kleckner" <[hidden email]>
>> >> > Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers
>> >> > Mailing List" <[hidden email]>
>> >> > Sent: Monday, June 29, 2015 3:03:40 PM
>> >> > Subject: Re: [cfe-dev] llvm-abi: A library for generating
>> >> > ABI-compliant
>> >> > LLVM  IR
>> >> >
>> >> > Hi Reid,
>> >> >
>> >> > Thanks for your response.
>> >> >
>> >> > The issue is that every LLVM frontend needing ABI compliance has to
>> >> > re-implement the same target-dependent logic, which is a significant
>> >> > burden; the ABI compliance code inside Clang isn't really usable for
>> >> > other frontends as-is. We haven't got many good options here :-). I
>> >> > think a lot of people would've hoped that LLVM would provide the
>> >> > means
>> >> > for achieving ABI compliance (at least for C), though I'm well aware
>> >> > of the complexity involved and understand the decisions taken thus
>> >> > far.
>> >> >
>> >> > Within this context, llvm-abi is an immediately actionable way of
>> >> > sharing code between frontends, that should lead to a higher quality
>> >> > codebase for a significantly reduced effort on everyone's part. It
>> >> > seems like this would further build on LLVM's success because I think
>> >> > many people would like to generate code that can interact with C
>> >> > APIs.
>> >> >
>> >> > Ultimately it would be great to see this functionality be provided in
>> >> > an accessible form inside LLVM and hence for Clang to use that
>> >> > functionality. This would move much of the target-dependent logic out
>> >> > of Clang while at the same time making this functionality available
>> >> > to
>> >> > other LLVM-based tools.
>> >> >
>> >> > I am sure this is a substantial and complex long term project, but I
>> >> > think it is a worthwhile aim. Do you (and the LLVM community as a
>> >> > whole) agree? (Acknowledging that some of the details would need to
>> >> > be
>> >> > worked out.)
>> >> >
>> >> > I have a long term and substantial interest in this (as I expect do
>> >> > other frontend developers), so I'm willing to contribute significant
>> >> > effort to move this forward. Hopefully the llvm-abi library can
>> >> > provide a better understanding of what needs to be represented and
>> >> > help non-Clang frontends :-).
>> >> >
>> >> > Implementation-wise the changes to LLVM could involve encoding ABI
>> >> > information into LLVM IR or simply providing C++ APIs for generating
>> >> > ABI-compliant code inside LLVM (much like llvm-abi). Each solution
>> >> > has
>> >> > advantages and disadvantages and I'm planning to make a proposal
>> >> > about
>> >> > this on the mailing list later (hence I'd suggest leaving that
>> >> > discussion until I make the proposal).
>> >> >
>> >> > In terms of the mentioned deficiencies of the Type structure, these
>> >> > are all problems that I intend to address (it would be great if you
>> >> > could provide details on any of the more difficult areas).
>> >>
>> >> First, let me say that I also think this is an incredibly important
>> >> area
>> >> in definite need of a solution. However, I also think that we really
>> >> need to
>> >> work from Clang's logic here: We don't want to have two different
>> >> implementations of the ABI rules. They're complicated (and subtle), and
>> >> it
>> >> is hard enough to have one correct implementation in our C/C++
>> >> compiler.
>> >> Given that we're talking about the C/C++ ABI, this is a perfectly
>> >> appropriate place for the logic to live. The issue, as you well know,
>> >> is
>> >> that other languages need to interact with C/C++ functions, and so need
>> >> to
>> >> understand the C/C++ ABI. My recommended strategy is this:
>> >>
>> >>  1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it
>> >> can be used by external clients (just as Clang's AST headers are
>> >> available
>> >> to external clients).
>> >>
>> >>  2. Write your library as a wrapper around that logic so that simple
>> >> cases
>> >> can be handled without burdening other frontends with the subtleties of
>> >> Clang's AST data structures.
>> >
>> >
>> > This is what I was thinking of doing for llgo in the long term. At the
>> > moment we have support for x86-64 only:
>> > http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be nice
>> > to
>> > have a simplified API to TargetInfo, ideally with a C API that other
>> > languages to can easily bind to. Stephen, if you do go down that route,
>> > I'd
>> > be keen to hear about it.
>> >
>> > Cheers,
>> > Andrew
>> >
>> >>  -Hal
>> >>
>> >> >
>> >> > Thanks again,
>> >> > Stephen
>> >> >
>> >> > On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
>> >> > wrote:
>> >> > > I agree it would be really nice to build a library for ABI
>> >> > > lowering, but any
>> >> > > solution that isn't clang or isn't ultimately picked up by clang
>> >> > > will
>> >> > > necessarily be incomplete. Perhaps that is OK for your frontend's
>> >> > > uses, but
>> >> > > I think it's the main reason that we haven't done something like
>> >> > > this in
>> >> > > LLVM already.
>> >> > >
>> >> > > Any solution that doesn't involve actual Clang ASTs is unlikely to
>> >> > > be able
>> >> > > to represent all C-with-extensions types (unions, bitfields,
>> >> > > alignment
>> >> > > attributes, transparent_union attribute). I took a look at Type.hpp
>> >> > > in your
>> >> > > project, and it seems to be missing some of these things. Keeping
>> >> > > such a
>> >> > > library up to date with new extensions is going to be a maintenance
>> >> > > burden.
>> >> > >
>> >> > > That said, I wish you luck, and I hope the project eases some of
>> >> > > the
>> >> > > difficulties for new frontends. It is very possible that the corner
>> >> > > cases
>> >> > > that keep me up at night are not the problems that users actually
>> >> > > face. :)
>> >> > >
>> >> > > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
>> >> > > <[hidden email]> wrote:
>> >> > >>
>> >> > >> Hi everyone,
>> >> > >>
>> >> > >> (Also CC'ed cfe-dev since this seems relevant to Clang,
>> >> > >> particularly
>> >> > >> the questions at the end.)
>> >> > >>
>> >> > >> I've been working on a library to generate LLVM IR that complies
>> >> > >> with
>> >> > >> platform ABIs (the current focus is on C but I'm also interested
>> >> > >> in
>> >> > >> ABIs for other languages).
>> >> > >>
>> >> > >> You can find it here: https://github.com/scross99/llvm-abi
>> >> > >>
>> >> > >> To explain further (for those who are unfamiliar), LLVM frontends
>> >> > >> have
>> >> > >> to modify function argument types, attributes etc. in order to
>> >> > >> ensure
>> >> > >> the backend generates code that satisfies the ABI; this is needed
>> >> > >> because LLVM's type system can't encode all the necessary
>> >> > >> information.
>> >> > >> This is a complex task and involves substantial amounts of
>> >> > >> target-dependent logic. Clang performs this encoding and indeed
>> >> > >> much
>> >> > >> of the current functionality is derived from Clang's source.
>> >> > >>
>> >> > >> This project originated as a necessary piece of functionality for
>> >> > >> the
>> >> > >> Loci compiler frontend [1] and is now an external dependency; my
>> >> > >> aim
>> >> > >> is to make this usable for other LLVM frontends that also need to
>> >> > >> generate ABI-compliant IR (I assume this is a fairly large subset
>> >> > >> of
>> >> > >> the frontends).
>> >> > >>
>> >> > >> I'd be very interested in any suggestions/queries/comments.
>> >> > >>
>> >> > >> I made a few interesting discoveries while working on this:
>> >> > >>
>> >> > >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
>> >> > >> even
>> >> > >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
>> >> > >> aligned to 16 bytes. Is this a bug or am I missing something
>> >> > >> obvious?
>> >> > >>
>> >> > >> * Clang determines the features for a CPU (e.g. whether we have
>> >> > >> AVX
>> >> > >> support on x86-64 CPUs), even though this functionality is already
>> >> > >> available in LLVM (but appears to be very difficult to query).
>> >> > >> Would
>> >> > >> it be possible to expose the information from LLVM and hence
>> >> > >> eliminate
>> >> > >> the duplication in Clang?
>> >> > >>
>> >> > >> * Clang determines a 'generic CPU' if the user doesn't specify a
>> >> > >> CPU.
>> >> > >> My understanding is that we don't usually generate code for the
>> >> > >> native
>> >> > >> CPU because it may have features unavailable on other similar
>> >> > >> CPUs.
>> >> > >> LLVM can provide full details of the native CPU but can't
>> >> > >> determine a
>> >> > >> generic CPU; could this functionality be added to LLVM?
>> >> > >>
>> >> > >> (Separately I've also been considering a proposal to add ABI
>> >> > >> information directly inside LLVM IR in a language-independent way
>> >> > >> and
>> >> > >> I'll discuss this in a later email.)
>> >> > >>
>> >> > >> Thanks,
>> >> > >> Stephen
>> >> > >>
>> >> > >> [1] https://github.com/scross99/locic
>> >> > >> _______________________________________________
>> >> > >> cfe-dev mailing list
>> >> > >> [hidden email]
>> >> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >> > >
>> >> > >
>> >> > _______________________________________________
>> >> > cfe-dev mailing list
>> >> > [hidden email]
>> >> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >> >
>> >>
>> >> --
>> >> Hal Finkel
>> >> Assistant Computational Scientist
>> >> Leadership Computing Facility
>> >> Argonne National Laboratory
>> >> _______________________________________________
>> >> LLVM Developers mailing list
>> >> [hidden email]         http://llvm.cs.uiuc.edu
>> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Pat Gavlin
> In particular it'd be great to have people contributing the parts that they specifically need (e.g. it sounds like WIndows support would be important for you).

Certainly. Like everyone else, my time is spread a bit thin, but I'm certainly happy to help where I can. :-)

- Pat
________________________________________
From: Stephen Cross <[hidden email]>
Sent: Wednesday, July 01, 2015 2:23 PM
To: Pat Gavlin
Cc: Andrew Wilkins; Hal Finkel; Clang Developers List; LLVM Developers Mailing List
Subject: Re: [LLVMdev] [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

> Overall, the idea of an ABI support library sounds great. It'd certainly save us a fair amount of duplicated effort in our frontend, as the CoreCLR ABI (mostly) corresonds to the native C ABI of its host platform.

Yes indeed. I'm starting to wonder just how many frontends exist that
implement this logic.

Of course I'm naturally going to ask whether you'd be able to help
with any aspects of llvm-abi? :-) In particular it'd be great to have
people contributing the parts that they specifically need (e.g. it
sounds like WIndows support would be important for you).

> Although I understand the motivations behind wrapping the Clang implementation, I'm a bit leery of this as a long-term direction. Admittedly, this is selfishly motivated: LLILC currently has no Clang dependency, and I think we'd prefer to avoid taking one if possible. A reasonable compromise would be the refactoring Stephen suggested below (i.e. moving ABI-related code into its own library within Clang):

Yes, I know. It's worth pointing out that llvm-abi doesn't need to
depend on a built version of Clang, just the same source code, so we
can do fun things like Git submodules or subtrees (presumably using
the Clang Git Mirror). So we'd stay in-sync with the upstream code
from Clang but llvm-abi would actually build the relevant parts of
Clang as part of its own build. I'm not necessarily advocating this
direction but just raising it as a potential solution. It really
depends on exactly what problems you would encounter with a dependency
on Clang.

> Although the refactoring could be tricky, this would at least let external users limit their dependency to the new library rather than pulling in libCodeGen.

Yes, I'd like to do this but the current Clang ABI code depends
heavily on other code in libCodeGen. This is something that I'll
ultimately try to do but it's not currently clear exactly what
restructuring is appropriate.

On Wed, Jul 1, 2015 at 8:33 PM, Pat Gavlin <[hidden email]> wrote:

> Chiming in from the LLILC project:
>
>
> Overall, the idea of an ABI support library sounds great. It'd certainly
> save us a fair amount of duplicated effort in our frontend, as the CoreCLR
> ABI (mostly) corresonds to the native C ABI of its host platform. What we
> have today[1, 2] is largely inspired by Clang, and so looks pretty similar
> to Stephen's existing work in llvm-abi.
>
>
>> I like the idea of frontends using the library as a convenient API and
>> the library's implementation using code from Clang.
>
> Although I understand the motivations behind wrapping the Clang
> implementation, I'm a bit leery of this as a long-term direction.
> Admittedly, this is selfishly motivated: LLILC currently has no Clang
> dependency, and I think we'd prefer to avoid taking one if possible. A
> reasonable compromise would be the refactoring Stephen suggested below (i.e.
> moving ABI-related code into its own library within Clang):
>
>> My approach would be to just refactor the code to separate classes
>> within CodeGen and add the necessary headers to
>> include/clang/CodeGen/. It might make sense to create a separate
>> library directory (e.g. lib/ABI) but I expect there will be circular
>> dependency issues to overcome.
>
> Although the refactoring could be tricky, this would at least let external
> users limit their dependency to the new library rather than pulling in
> libCodeGen.
>
> Thanks,
> - Pat
>
> [1] https://github.com/dotnet/llilc/blob/master/include/Reader/abi.h
> [2] https://github.com/dotnet/llilc/blob/master/lib/Reader/abi.cpp
>
> ________________________________
> From: [hidden email] <[hidden email]> on behalf of
> Andrew Wilkins <[hidden email]>
> Sent: Tuesday, June 30, 2015 6:24 PM
> To: Stephen Cross; Hal Finkel
> Cc: Clang Developers List; LLVM Developers Mailing List
> Subject: Re: [LLVMdev] [cfe-dev] llvm-abi: A library for generating
> ABI-compliant LLVM IR
>
> On Wed, 1 Jul 2015 at 08:19 Stephen Cross <[hidden email]> wrote:
>>
>> Hi everyone,
>>
>> I was wondering if anyone could answer the questions in my first
>> email. These were:
>>
>> * Why does Clang generate 8 byte alignment for 16+ byte arrays on
>> x86-64, even though the AMD64 ABI seems to require 16 byte alignment?
>> * Clang has some logic in lib/Basic/Targets.cpp to determine CPU
>> features, but LLVM also does the same task in its backend. Would it be
>> possible to modify LLVM to expose this logic and modify Clang to use
>> that?
>> * (Slightly modified) Clang has some logic in lib/Driver/Tools.cpp
>> ('getCPUName') for selecting a generic CPU. Would it be sensible to
>> expose this in a header in include/clang/Driver/?
>>
>> > Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can
>> > be used by external clients (just as Clang's AST headers are available to
>> > external clients).
>>
>> Yes, this sounds like a good approach. While I'm not entirely
>> convinced this is where we want to end up (it doesn't seem right to
>> split the ABI implementation between the frontend and the backend), I
>> definitely agree this is a sensible starting point. Unless anyone is
>> against this I'll start working out the changes needed under the
>> assumption this is how I should proceed.
>>
>> My approach would be to just refactor the code to separate classes
>> within CodeGen and add the necessary headers to
>> include/clang/CodeGen/. It might make sense to create a separate
>> library directory (e.g. lib/ABI) but I expect there will be circular
>> dependency issues to overcome. The very first thing is probably to
>> split each target's code out of TargetInfo.cpp; I'd suggest that the
>> structure of CodeGen could mirror <llvm>/lib/Target/ so we'd have
>> directories for ARM, PowerPC, X86 etc. What do you think?
>>
>> Note that there's also functionality in CGCall.cpp (and a few other
>> files) which would need to be separated out to be usable for other
>> frontends.
>>
>> > Write your library as a wrapper around that logic so that simple cases
>> > can be handled without burdening other frontends with the subtleties of
>> > Clang's AST data structures.
>>
>> This will be how it ends up, but while the modifications to Clang are
>> ongoing llvm-abi is and will continue to be an entirely separate
>> ABI-implementing codebase. This is just because the Loci frontend
>> needs this functionality right now. I'm also expecting there will be
>> functionality going into llvm-abi that for good reason won't be found
>> in Clang. There's some duplication of effort here but it's
>> unavoidable.
>>
>> I like the idea of frontends using the library as a convenient API and
>> the library's implementation using code from Clang. I know that some
>> frontends (including the Loci frontend) are designed to be able to be
>> built on systems that could have slightly older versions of LLVM/Clang
>> (for Loci, it's LLVM 3.3+), so this factor will have to be considered
>> when integrating the code from Clang.
>>
>> > At the moment we have support for x86-64 only:
>> > http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go.
>>
>> I'll add other architectures to llvm-abi soon (it's largely a fairly
>> trivial process of moving code from Clang and replacing its in-tree
>> dependencies) so hopefully this will be useful. I can see from other
>> non-Clang frontends that non-X86 support does tend to be at least
>> quite incomplete.
>>
>> > It would be nice to have a simplified API to TargetInfo, ideally with a
>> > C API that other languages to can easily bind to.
>>
>> Absolutely, I agree. I appear to be in the rare situation of having a
>> frontend written in C++ (i.e. not self-hosting) but I know most
>> frontends rely on having a C API.
>>
>> It seems like the approach to this should be to have a C API in
>> llvm-abi that wraps around its C++ API. Unfortunately my time is
>> currently fairly thin-spread; would you be able to contribute to this
>> effort?
>
>
> Sounds good. My time is likewise thinly spread, but I'll be happy to
> contribute to the effort when I can.
>
> Cheers,
> Andrew
>
>>
>> Thanks,
>> Stephen
>>
>> On Tue, Jun 30, 2015 at 12:47 AM, Andrew Wilkins <[hidden email]> wrote:
>> > On Tue, 30 Jun 2015 at 06:02 Hal Finkel <[hidden email]> wrote:
>> >>
>> >> ----- Original Message -----
>> >> > From: "Stephen Cross" <[hidden email]>
>> >> > To: "Reid Kleckner" <[hidden email]>
>> >> > Cc: "Clang Developers List" <[hidden email]>, "LLVM Developers
>> >> > Mailing List" <[hidden email]>
>> >> > Sent: Monday, June 29, 2015 3:03:40 PM
>> >> > Subject: Re: [cfe-dev] llvm-abi: A library for generating
>> >> > ABI-compliant
>> >> > LLVM  IR
>> >> >
>> >> > Hi Reid,
>> >> >
>> >> > Thanks for your response.
>> >> >
>> >> > The issue is that every LLVM frontend needing ABI compliance has to
>> >> > re-implement the same target-dependent logic, which is a significant
>> >> > burden; the ABI compliance code inside Clang isn't really usable for
>> >> > other frontends as-is. We haven't got many good options here :-). I
>> >> > think a lot of people would've hoped that LLVM would provide the
>> >> > means
>> >> > for achieving ABI compliance (at least for C), though I'm well aware
>> >> > of the complexity involved and understand the decisions taken thus
>> >> > far.
>> >> >
>> >> > Within this context, llvm-abi is an immediately actionable way of
>> >> > sharing code between frontends, that should lead to a higher quality
>> >> > codebase for a significantly reduced effort on everyone's part. It
>> >> > seems like this would further build on LLVM's success because I think
>> >> > many people would like to generate code that can interact with C
>> >> > APIs.
>> >> >
>> >> > Ultimately it would be great to see this functionality be provided in
>> >> > an accessible form inside LLVM and hence for Clang to use that
>> >> > functionality. This would move much of the target-dependent logic out
>> >> > of Clang while at the same time making this functionality available
>> >> > to
>> >> > other LLVM-based tools.
>> >> >
>> >> > I am sure this is a substantial and complex long term project, but I
>> >> > think it is a worthwhile aim. Do you (and the LLVM community as a
>> >> > whole) agree? (Acknowledging that some of the details would need to
>> >> > be
>> >> > worked out.)
>> >> >
>> >> > I have a long term and substantial interest in this (as I expect do
>> >> > other frontend developers), so I'm willing to contribute significant
>> >> > effort to move this forward. Hopefully the llvm-abi library can
>> >> > provide a better understanding of what needs to be represented and
>> >> > help non-Clang frontends :-).
>> >> >
>> >> > Implementation-wise the changes to LLVM could involve encoding ABI
>> >> > information into LLVM IR or simply providing C++ APIs for generating
>> >> > ABI-compliant code inside LLVM (much like llvm-abi). Each solution
>> >> > has
>> >> > advantages and disadvantages and I'm planning to make a proposal
>> >> > about
>> >> > this on the mailing list later (hence I'd suggest leaving that
>> >> > discussion until I make the proposal).
>> >> >
>> >> > In terms of the mentioned deficiencies of the Type structure, these
>> >> > are all problems that I intend to address (it would be great if you
>> >> > could provide details on any of the more difficult areas).
>> >>
>> >> First, let me say that I also think this is an incredibly important
>> >> area
>> >> in definite need of a solution. However, I also think that we really
>> >> need to
>> >> work from Clang's logic here: We don't want to have two different
>> >> implementations of the ABI rules. They're complicated (and subtle), and
>> >> it
>> >> is hard enough to have one correct implementation in our C/C++
>> >> compiler.
>> >> Given that we're talking about the C/C++ ABI, this is a perfectly
>> >> appropriate place for the logic to live. The issue, as you well know,
>> >> is
>> >> that other languages need to interact with C/C++ functions, and so need
>> >> to
>> >> understand the C/C++ ABI. My recommended strategy is this:
>> >>
>> >>  1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it
>> >> can be used by external clients (just as Clang's AST headers are
>> >> available
>> >> to external clients).
>> >>
>> >>  2. Write your library as a wrapper around that logic so that simple
>> >> cases
>> >> can be handled without burdening other frontends with the subtleties of
>> >> Clang's AST data structures.
>> >
>> >
>> > This is what I was thinking of doing for llgo in the long term. At the
>> > moment we have support for x86-64 only:
>> > http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be nice
>> > to
>> > have a simplified API to TargetInfo, ideally with a C API that other
>> > languages to can easily bind to. Stephen, if you do go down that route,
>> > I'd
>> > be keen to hear about it.
>> >
>> > Cheers,
>> > Andrew
>> >
>> >>  -Hal
>> >>
>> >> >
>> >> > Thanks again,
>> >> > Stephen
>> >> >
>> >> > On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <[hidden email]>
>> >> > wrote:
>> >> > > I agree it would be really nice to build a library for ABI
>> >> > > lowering, but any
>> >> > > solution that isn't clang or isn't ultimately picked up by clang
>> >> > > will
>> >> > > necessarily be incomplete. Perhaps that is OK for your frontend's
>> >> > > uses, but
>> >> > > I think it's the main reason that we haven't done something like
>> >> > > this in
>> >> > > LLVM already.
>> >> > >
>> >> > > Any solution that doesn't involve actual Clang ASTs is unlikely to
>> >> > > be able
>> >> > > to represent all C-with-extensions types (unions, bitfields,
>> >> > > alignment
>> >> > > attributes, transparent_union attribute). I took a look at Type.hpp
>> >> > > in your
>> >> > > project, and it seems to be missing some of these things. Keeping
>> >> > > such a
>> >> > > library up to date with new extensions is going to be a maintenance
>> >> > > burden.
>> >> > >
>> >> > > That said, I wish you luck, and I hope the project eases some of
>> >> > > the
>> >> > > difficulties for new frontends. It is very possible that the corner
>> >> > > cases
>> >> > > that keep me up at night are not the problems that users actually
>> >> > > face. :)
>> >> > >
>> >> > > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
>> >> > > <[hidden email]> wrote:
>> >> > >>
>> >> > >> Hi everyone,
>> >> > >>
>> >> > >> (Also CC'ed cfe-dev since this seems relevant to Clang,
>> >> > >> particularly
>> >> > >> the questions at the end.)
>> >> > >>
>> >> > >> I've been working on a library to generate LLVM IR that complies
>> >> > >> with
>> >> > >> platform ABIs (the current focus is on C but I'm also interested
>> >> > >> in
>> >> > >> ABIs for other languages).
>> >> > >>
>> >> > >> You can find it here: https://github.com/scross99/llvm-abi
>> >> > >>
>> >> > >> To explain further (for those who are unfamiliar), LLVM frontends
>> >> > >> have
>> >> > >> to modify function argument types, attributes etc. in order to
>> >> > >> ensure
>> >> > >> the backend generates code that satisfies the ABI; this is needed
>> >> > >> because LLVM's type system can't encode all the necessary
>> >> > >> information.
>> >> > >> This is a complex task and involves substantial amounts of
>> >> > >> target-dependent logic. Clang performs this encoding and indeed
>> >> > >> much
>> >> > >> of the current functionality is derived from Clang's source.
>> >> > >>
>> >> > >> This project originated as a necessary piece of functionality for
>> >> > >> the
>> >> > >> Loci compiler frontend [1] and is now an external dependency; my
>> >> > >> aim
>> >> > >> is to make this usable for other LLVM frontends that also need to
>> >> > >> generate ABI-compliant IR (I assume this is a fairly large subset
>> >> > >> of
>> >> > >> the frontends).
>> >> > >>
>> >> > >> I'd be very interested in any suggestions/queries/comments.
>> >> > >>
>> >> > >> I made a few interesting discoveries while working on this:
>> >> > >>
>> >> > >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
>> >> > >> even
>> >> > >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
>> >> > >> aligned to 16 bytes. Is this a bug or am I missing something
>> >> > >> obvious?
>> >> > >>
>> >> > >> * Clang determines the features for a CPU (e.g. whether we have
>> >> > >> AVX
>> >> > >> support on x86-64 CPUs), even though this functionality is already
>> >> > >> available in LLVM (but appears to be very difficult to query).
>> >> > >> Would
>> >> > >> it be possible to expose the information from LLVM and hence
>> >> > >> eliminate
>> >> > >> the duplication in Clang?
>> >> > >>
>> >> > >> * Clang determines a 'generic CPU' if the user doesn't specify a
>> >> > >> CPU.
>> >> > >> My understanding is that we don't usually generate code for the
>> >> > >> native
>> >> > >> CPU because it may have features unavailable on other similar
>> >> > >> CPUs.
>> >> > >> LLVM can provide full details of the native CPU but can't
>> >> > >> determine a
>> >> > >> generic CPU; could this functionality be added to LLVM?
>> >> > >>
>> >> > >> (Separately I've also been considering a proposal to add ABI
>> >> > >> information directly inside LLVM IR in a language-independent way
>> >> > >> and
>> >> > >> I'll discuss this in a later email.)
>> >> > >>
>> >> > >> Thanks,
>> >> > >> Stephen
>> >> > >>
>> >> > >> [1] https://github.com/scross99/locic
>> >> > >> _______________________________________________
>> >> > >> cfe-dev mailing list
>> >> > >> [hidden email]
>> >> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >> > >
>> >> > >
>> >> > _______________________________________________
>> >> > cfe-dev mailing list
>> >> > [hidden email]
>> >> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >> >
>> >>
>> >> --
>> >> Hal Finkel
>> >> Assistant Computational Scientist
>> >> Leadership Computing Facility
>> >> Argonne National Laboratory
>> >> _______________________________________________
>> >> LLVM Developers mailing list
>> >> [hidden email]         http://llvm.cs.uiuc.edu
>> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev