[llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Jonas Paulsson via llvm-dev
Hi All,

This week I've been focused on removing some of the blockers for people transitioning from ORCv1 to ORCv2.


When LLVM codegens floating point constants for COFF we produce named constant pool entries of the form __real@<bitval>. These are stored in COFF COMDAT sections [1] which allow duplicate symbol definitions to be discarded (roughly equivalent to the concept of a weak symbol). This posed two problems: (1) ORC had no idea about the (rough) mapping between COMDAT and Weak, and (2) did not support late introduction of weak symbols via MaterializationResponsibility::defineMaterializing. This caused use of the same floating point constant in multiple files to result in duplicate definition errors for the constant pool entries. 

Both issues have been addressed in 84217ad6611, and floating point constants should now work in COFF. If you run in to any trouble with them please file a bug and CC me.

Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on llvm-dev):

Use of emulated-tls affects the name mangling of LLVM IR symbols. For a thread local symbol with name <Name>, when emulated-tls is enabled we will produce at least one emulated-tls symbol: __emutls_v.<Name> (which must then be linker-mangled to get the final symbol name). If that symbol has a non-zero initializer a second symbol will also be produced: __emutls_t.<Name>. ORCv2 requires up-front specification of the exact set of linker-level symbol names that a module produces, so that it can dispatch and track compiles, however none of the existing mangling code took emulated-tls into account. This has been fixed in ce2207abaf9 (with follow up fixes for the various examples I broke), and you should now be able to use thread-locals in ORCv2.

Besides these two bugs, I have continued work on the big static initializer patch. Unfortunately issue #2 interacts with it, so I have to do some integration work before it is ready to land, but I'm hopeful that it will be ready some time next week.

Finally, I have added a new example (bb7a5707ac0)to llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin demonstrating how to write a custom ObjectLinkingLayer plugin. ObjectLinkingLayer plugins can be used to receive and respond to events in the linking layer, and to inspect and modify JITLink LinkGraphs during the linking process. ORC uses this internally to register EH frames, but the system is likely to be of interest to anyone who wants low level control/visibility into the JIT linker.

— Lang.


_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Jonas Paulsson via llvm-dev
Hi Lang,

Thank you very much for writing these.  I suspect that they are likely
to be drowned in the noise of this list and hidden from a lot of the
people that would find them valuable.

Would it be possible for them to be hosted somewhere permanently?
Perhaps they could either be incorporated as a subsection of LLVM Weekly
or put in a different location on the same server?

David

On 24/01/2020 03:27, Lang Hames via llvm-dev wrote:

> Hi All,
>
> This week I've been focused on removing some of the blockers for people
> transitioning from ORCv1 to ORCv2.
>
> Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337):
>
> When LLVM codegens floating point constants for COFF we produce named
> constant pool entries of the form __real@<bitval>. These are stored in
> COFF COMDAT sections [1] which allow duplicate symbol definitions to be
> discarded (roughly equivalent to the concept of a weak symbol). This
> posed two problems: (1) ORC had no idea about the (rough) mapping
> between COMDAT and Weak, and (2) did not support late introduction of
> weak symbols via MaterializationResponsibility::defineMaterializing.
> This caused use of the same floating point constant in multiple files to
> result in duplicate definition errors for the constant pool entries.
>
> Both issues have been addressed in 84217ad6611, and floating point
> constants should now work in COFF. If you run in to any trouble with
> them please file a bug and CC me.
>
> Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on
> llvm-dev):
>
> Use of emulated-tls affects the name mangling of LLVM IR symbols. For a
> thread local symbol with name <Name>, when emulated-tls is enabled we
> will produce at least one emulated-tls symbol: __emutls_v.<Name> (which
> must then be linker-mangled to get the final symbol name). If that
> symbol has a non-zero initializer a second symbol will also be produced:
> __emutls_t.<Name>. ORCv2 requires up-front specification of the exact
> set of /linker-level/ symbol names that a module produces, so that it
> can dispatch and track compiles, however none of the existing mangling
> code took emulated-tls into account. This has been fixed in ce2207abaf9
> (with follow up fixes for the various examples I broke), and you should
> now be able to use thread-locals in ORCv2.
>
> Besides these two bugs, I have continued work on the big static
> initializer patch. Unfortunately issue #2 interacts with it, so I have
> to do some integration work before it is ready to land, but I'm hopeful
> that it will be ready some time next week.
>
> Finally, I have added a new example (bb7a5707ac0)to
> llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin
> demonstrating how to write a custom ObjectLinkingLayer plugin.
> ObjectLinkingLayer plugins can be used to receive and respond to events
> in the linking layer, and to inspect and modify JITLink LinkGraphs
> during the linking process. ORC uses this internally to register EH
> frames, but the system is likely to be of interest to anyone who wants
> low level control/visibility into the JIT linker.
>
> — Lang.
>
> [1]
> https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only 
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Jonas Paulsson via llvm-dev
In reply to this post by Jonas Paulsson via llvm-dev

I'm very pleased that PR40074/44337 has been squashed, great work! This means I can now use unmodified LLVM for my application and perhaps soon target a stable release. Will it be part of the LLVM 10 release?

Machiel

On 24-Jan-20 04:27, Lang Hames via llvm-dev wrote:
Hi All,

This week I've been focused on removing some of the blockers for people transitioning from ORCv1 to ORCv2.


When LLVM codegens floating point constants for COFF we produce named constant pool entries of the form __real@<bitval>. These are stored in COFF COMDAT sections [1] which allow duplicate symbol definitions to be discarded (roughly equivalent to the concept of a weak symbol). This posed two problems: (1) ORC had no idea about the (rough) mapping between COMDAT and Weak, and (2) did not support late introduction of weak symbols via MaterializationResponsibility::defineMaterializing. This caused use of the same floating point constant in multiple files to result in duplicate definition errors for the constant pool entries. 

Both issues have been addressed in 84217ad6611, and floating point constants should now work in COFF. If you run in to any trouble with them please file a bug and CC me.

Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on llvm-dev):

Use of emulated-tls affects the name mangling of LLVM IR symbols. For a thread local symbol with name <Name>, when emulated-tls is enabled we will produce at least one emulated-tls symbol: __emutls_v.<Name> (which must then be linker-mangled to get the final symbol name). If that symbol has a non-zero initializer a second symbol will also be produced: __emutls_t.<Name>. ORCv2 requires up-front specification of the exact set of linker-level symbol names that a module produces, so that it can dispatch and track compiles, however none of the existing mangling code took emulated-tls into account. This has been fixed in ce2207abaf9 (with follow up fixes for the various examples I broke), and you should now be able to use thread-locals in ORCv2.

Besides these two bugs, I have continued work on the big static initializer patch. Unfortunately issue #2 interacts with it, so I have to do some integration work before it is ready to land, but I'm hopeful that it will be ready some time next week.

Finally, I have added a new example (bb7a5707ac0)to llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin demonstrating how to write a custom ObjectLinkingLayer plugin. ObjectLinkingLayer plugins can be used to receive and respond to events in the linking layer, and to inspect and modify JITLink LinkGraphs during the linking process. ORC uses this internally to register EH frames, but the system is likely to be of interest to anyone who wants low level control/visibility into the JIT linker.

— Lang.


_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Jonas Paulsson via llvm-dev
Hi Machiel,

I have requested that 84217ad6611 be merged into the LLVM 10 release branch in http://llvm.org/PR44700. It's a low-risk fix, so I'm hopeful that it will make it in.

-- Lang.

On Fri, Jan 24, 2020 at 8:06 AM Machiel van Hooren <[hidden email]> wrote:

I'm very pleased that PR40074/44337 has been squashed, great work! This means I can now use unmodified LLVM for my application and perhaps soon target a stable release. Will it be part of the LLVM 10 release?

Machiel

On 24-Jan-20 04:27, Lang Hames via llvm-dev wrote:
Hi All,

This week I've been focused on removing some of the blockers for people transitioning from ORCv1 to ORCv2.


When LLVM codegens floating point constants for COFF we produce named constant pool entries of the form __real@<bitval>. These are stored in COFF COMDAT sections [1] which allow duplicate symbol definitions to be discarded (roughly equivalent to the concept of a weak symbol). This posed two problems: (1) ORC had no idea about the (rough) mapping between COMDAT and Weak, and (2) did not support late introduction of weak symbols via MaterializationResponsibility::defineMaterializing. This caused use of the same floating point constant in multiple files to result in duplicate definition errors for the constant pool entries. 

Both issues have been addressed in 84217ad6611, and floating point constants should now work in COFF. If you run in to any trouble with them please file a bug and CC me.

Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on llvm-dev):

Use of emulated-tls affects the name mangling of LLVM IR symbols. For a thread local symbol with name <Name>, when emulated-tls is enabled we will produce at least one emulated-tls symbol: __emutls_v.<Name> (which must then be linker-mangled to get the final symbol name). If that symbol has a non-zero initializer a second symbol will also be produced: __emutls_t.<Name>. ORCv2 requires up-front specification of the exact set of linker-level symbol names that a module produces, so that it can dispatch and track compiles, however none of the existing mangling code took emulated-tls into account. This has been fixed in ce2207abaf9 (with follow up fixes for the various examples I broke), and you should now be able to use thread-locals in ORCv2.

Besides these two bugs, I have continued work on the big static initializer patch. Unfortunately issue #2 interacts with it, so I have to do some integration work before it is ready to land, but I'm hopeful that it will be ready some time next week.

Finally, I have added a new example (bb7a5707ac0)to llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin demonstrating how to write a custom ObjectLinkingLayer plugin. ObjectLinkingLayer plugins can be used to receive and respond to events in the linking layer, and to inspect and modify JITLink LinkGraphs during the linking process. ORC uses this internally to register EH frames, but the system is likely to be of interest to anyone who wants low level control/visibility into the JIT linker.

— Lang.


_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Jonas Paulsson via llvm-dev
In reply to this post by Jonas Paulsson via llvm-dev
Hi David,

Thank you very much for writing these.  I suspect that they are likely 
to be drowned in the noise of this list and hidden from a lot of the 
people that would find them valuable.

Would it be possible for them to be hosted somewhere permanently? 
Perhaps they could either be incorporated as a subsection of LLVM Weekly 
or put in a different location on the same server?
 
The threads are archived on lists.llvm.org, but I'm not sure how discoverable they are there.

I'm keen to make sure this is visible to the right people, and I'm not picky about the format. Does anyone else have any thoughts on this?

Alex, how would you feel about me supplying a weekly JIT update as a subsection for LLVM Weekly, rather than starting a separate thread? (And thank you very much for posting links to the archives for the last two ORC JIT Weekly threads).

-- Lang.

On Fri, Jan 24, 2020 at 5:08 AM David Chisnall via llvm-dev <[hidden email]> wrote:
Hi Lang,

Thank you very much for writing these.  I suspect that they are likely
to be drowned in the noise of this list and hidden from a lot of the
people that would find them valuable.

Would it be possible for them to be hosted somewhere permanently?
Perhaps they could either be incorporated as a subsection of LLVM Weekly
or put in a different location on the same server?

David

On 24/01/2020 03:27, Lang Hames via llvm-dev wrote:
> Hi All,
>
> This week I've been focused on removing some of the blockers for people
> transitioning from ORCv1 to ORCv2.
>
> Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337):
>
> When LLVM codegens floating point constants for COFF we produce named
> constant pool entries of the form __real@<bitval>. These are stored in
> COFF COMDAT sections [1] which allow duplicate symbol definitions to be
> discarded (roughly equivalent to the concept of a weak symbol). This
> posed two problems: (1) ORC had no idea about the (rough) mapping
> between COMDAT and Weak, and (2) did not support late introduction of
> weak symbols via MaterializationResponsibility::defineMaterializing.
> This caused use of the same floating point constant in multiple files to
> result in duplicate definition errors for the constant pool entries.
>
> Both issues have been addressed in 84217ad6611, and floating point
> constants should now work in COFF. If you run in to any trouble with
> them please file a bug and CC me.
>
> Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on
> llvm-dev):
>
> Use of emulated-tls affects the name mangling of LLVM IR symbols. For a
> thread local symbol with name <Name>, when emulated-tls is enabled we
> will produce at least one emulated-tls symbol: __emutls_v.<Name> (which
> must then be linker-mangled to get the final symbol name). If that
> symbol has a non-zero initializer a second symbol will also be produced:
> __emutls_t.<Name>. ORCv2 requires up-front specification of the exact
> set of /linker-level/ symbol names that a module produces, so that it
> can dispatch and track compiles, however none of the existing mangling
> code took emulated-tls into account. This has been fixed in ce2207abaf9
> (with follow up fixes for the various examples I broke), and you should
> now be able to use thread-locals in ORCv2.
>
> Besides these two bugs, I have continued work on the big static
> initializer patch. Unfortunately issue #2 interacts with it, so I have
> to do some integration work before it is ready to land, but I'm hopeful
> that it will be ready some time next week.
>
> Finally, I have added a new example (bb7a5707ac0)to
> llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin
> demonstrating how to write a custom ObjectLinkingLayer plugin.
> ObjectLinkingLayer plugins can be used to receive and respond to events
> in the linking layer, and to inspect and modify JITLink LinkGraphs
> during the linking process. ORC uses this internally to register EH
> frames, but the system is likely to be of interest to anyone who wants
> low level control/visibility into the JIT linker.
>
> — Lang.
>
> [1]
> https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Jonas Paulsson via llvm-dev
Alex, how would you feel about me supplying a weekly JIT update as a subsection for LLVM Weekly, rather than starting a separate thread? (And thank you very much for posting links to the archives for the last two ORC JIT Weekly threads).

Hi Lang,
I guess you forgot to CC Alex? :)
I cc'ed him, not sure if the email is the correct one though.

On 28. Jan 2020, at 23:10, Lang Hames via llvm-dev <[hidden email]> wrote:

Hi David,

Thank you very much for writing these.  I suspect that they are likely 
to be drowned in the noise of this list and hidden from a lot of the 
people that would find them valuable.

Would it be possible for them to be hosted somewhere permanently? 
Perhaps they could either be incorporated as a subsection of LLVM Weekly 
or put in a different location on the same server?
 
The threads are archived on lists.llvm.org, but I'm not sure how discoverable they are there.

I'm keen to make sure this is visible to the right people, and I'm not picky about the format. Does anyone else have any thoughts on this?

Alex, how would you feel about me supplying a weekly JIT update as a subsection for LLVM Weekly, rather than starting a separate thread? (And thank you very much for posting links to the archives for the last two ORC JIT Weekly threads).

-- Lang.

On Fri, Jan 24, 2020 at 5:08 AM David Chisnall via llvm-dev <[hidden email]> wrote:
Hi Lang,

Thank you very much for writing these.  I suspect that they are likely
to be drowned in the noise of this list and hidden from a lot of the
people that would find them valuable.

Would it be possible for them to be hosted somewhere permanently?
Perhaps they could either be incorporated as a subsection of LLVM Weekly
or put in a different location on the same server?

David

On 24/01/2020 03:27, Lang Hames via llvm-dev wrote:
> Hi All,
>
> This week I've been focused on removing some of the blockers for people
> transitioning from ORCv1 to ORCv2.
>
> Issue #1 (http://llvm.org/PR40074, http://llvm.org/PR44337):
>
> When LLVM codegens floating point constants for COFF we produce named
> constant pool entries of the form __real@<bitval>. These are stored in
> COFF COMDAT sections [1] which allow duplicate symbol definitions to be
> discarded (roughly equivalent to the concept of a weak symbol). This
> posed two problems: (1) ORC had no idea about the (rough) mapping
> between COMDAT and Weak, and (2) did not support late introduction of
> weak symbols via MaterializationResponsibility::defineMaterializing.
> This caused use of the same floating point constant in multiple files to
> result in duplicate definition errors for the constant pool entries.
>
> Both issues have been addressed in 84217ad6611, and floating point
> constants should now work in COFF. If you run in to any trouble with
> them please file a bug and CC me.
>
> Issue #2: Emulated TLS and name mangling (raised by Geoff Levner on
> llvm-dev):
>
> Use of emulated-tls affects the name mangling of LLVM IR symbols. For a
> thread local symbol with name <Name>, when emulated-tls is enabled we
> will produce at least one emulated-tls symbol: __emutls_v.<Name> (which
> must then be linker-mangled to get the final symbol name). If that
> symbol has a non-zero initializer a second symbol will also be produced:
> __emutls_t.<Name>. ORCv2 requires up-front specification of the exact
> set of /linker-level/ symbol names that a module produces, so that it
> can dispatch and track compiles, however none of the existing mangling
> code took emulated-tls into account. This has been fixed in ce2207abaf9
> (with follow up fixes for the various examples I broke), and you should
> now be able to use thread-locals in ORCv2.
>
> Besides these two bugs, I have continued work on the big static
> initializer patch. Unfortunately issue #2 interacts with it, so I have
> to do some integration work before it is ready to land, but I'm hopeful
> that it will be ready some time next week.
>
> Finally, I have added a new example (bb7a5707ac0)to
> llvm/examples/LLJITExamples: LLJITWithObjectLinkingLayerPlugin
> demonstrating how to write a custom ObjectLinkingLayer plugin.
> ObjectLinkingLayer plugins can be used to receive and respond to events
> in the linking layer, and to inspect and modify JITLink LinkGraphs
> during the linking process. ORC uses this internally to register EH
> frames, but the system is likely to be of interest to anyone who wants
> low level control/visibility into the JIT linker.
>
> — Lang.
>
> [1]
> https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#comdat-sections-object-only
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ORC JIT Weekly #2 -- COFF COMDAT Constants and Emulated TLS

Jonas Paulsson via llvm-dev
In reply to this post by Jonas Paulsson via llvm-dev
On Tue, 28 Jan 2020 at 22:11, Lang Hames <[hidden email]> wrote:

>
> Hi David,
>
>> Thank you very much for writing these.  I suspect that they are likely
>> to be drowned in the noise of this list and hidden from a lot of the
>> people that would find them valuable.
>>
>> Would it be possible for them to be hosted somewhere permanently?
>> Perhaps they could either be incorporated as a subsection of LLVM Weekly
>> or put in a different location on the same server?
>
>
> The threads are archived on lists.llvm.org, but I'm not sure how discoverable they are there.
>
> I'm keen to make sure this is visible to the right people, and I'm not picky about the format. Does anyone else have any thoughts on this?
>
> Alex, how would you feel about me supplying a weekly JIT update as a subsection for LLVM Weekly, rather than starting a separate thread? (And thank you very much for posting links to the archives for the last two ORC JIT Weekly threads).

Hi Lang,

Sorry, I missed this question previously. I'd be very happy with that.
If you just email me some markdown by end of day on a Sunday, I can be
sure to paste it in while doing the final clean-up and send on Monday.

Best,

Alex
_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev