Stack traces from JIT code

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Stack traces from JIT code

Priyendra Deshwal-2
[Resending this email after subscribing to the group. Got an error message the previous time around]

Hey guys,

In our project we have the following workflow:

- Given a user request, we construct a C++ program to service the request
- We create a compiler invocation to compile that program into an IR module
- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

Will that work? Is there a simpler way?

Thanks,
-- Priyendra

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Stack traces from JIT code

Reid Kleckner-2
Writing your own JITEventListener is probably the best way to capture traces from production.  Using the existing gdb/JIT interface support is probably far too heavyweight for production.  If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production.


On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <[hidden email]> wrote:
[Resending this email after subscribing to the group. Got an error message the previous time around]

Hey guys,

In our project we have the following workflow:

- Given a user request, we construct a C++ program to service the request
- We create a compiler invocation to compile that program into an IR module
- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

Will that work? Is there a simpler way?

Thanks,
-- Priyendra

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Stack traces from JIT code

Andrew MacPherson
We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind.

Cheers,
Andrew


On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <[hidden email]> wrote:
Writing your own JITEventListener is probably the best way to capture traces from production.  Using the existing gdb/JIT interface support is probably far too heavyweight for production.  If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production.


On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <[hidden email]> wrote:
[Resending this email after subscribing to the group. Got an error message the previous time around]

Hey guys,

In our project we have the following workflow:

- Given a user request, we construct a C++ program to service the request
- We create a compiler invocation to compile that program into an IR module
- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

Will that work? Is there a simpler way?

Thanks,
-- Priyendra

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Stack traces from JIT code

Priyendra Deshwal-2
Thanks for the info guys. I will go ahead and implement the JITEventListener. Thanks for the MCJIT tip Andrew - will keep a look out for that.


On Wed, Nov 13, 2013 at 9:23 AM, Andrew MacPherson <[hidden email]> wrote:
We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind.

Cheers,
Andrew


On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <[hidden email]> wrote:
Writing your own JITEventListener is probably the best way to capture traces from production.  Using the existing gdb/JIT interface support is probably far too heavyweight for production.  If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production.


On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <[hidden email]> wrote:
[Resending this email after subscribing to the group. Got an error message the previous time around]

Hey guys,

In our project we have the following workflow:

- Given a user request, we construct a C++ program to service the request
- We create a compiler invocation to compile that program into an IR module
- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

Will that work? Is there a simpler way?

Thanks,
-- Priyendra

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev




_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Stack traces from JIT code

Kaylor, Andrew
In reply to this post by Andrew MacPherson

From NotifyObjectEmitted you should be able to get to any function that’s visible via either the ELF headers or the DWARF information.  Do you have functions that don’t show up in either of those places?

 

-Andy

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Andrew MacPherson
Sent: Wednesday, November 13, 2013 9:23 AM
To: Reid Kleckner
Cc: [hidden email]
Subject: Re: [LLVMdev] Stack traces from JIT code

 

We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind.

Cheers,
Andrew

 

On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <[hidden email]> wrote:

Writing your own JITEventListener is probably the best way to capture traces from production.  Using the existing gdb/JIT interface support is probably far too heavyweight for production.  If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production.

 

On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <[hidden email]> wrote:

[Resending this email after subscribing to the group. Got an error message the previous time around]

 

Hey guys,

 

In our project we have the following workflow:

 

- Given a user request, we construct a C++ program to service the request

- We create a compiler invocation to compile that program into an IR module

- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

 

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

 

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

 

Will that work? Is there a simpler way?

 

Thanks,

-- Priyendra

 

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Stack traces from JIT code

Andrew MacPherson
Hi Andy,

In the NotifyObjectEmitted method of our derived JITEventListener class we use the begin_symbol() iterator to walk the object's symbols looking for functions and only functions marked with ExternalLinkage seem to show up. I'm not sure how I would access the Dwarf info from within there, is there a way?

Thanks,
Andrew


On Wed, Nov 13, 2013 at 8:23 PM, Kaylor, Andrew <[hidden email]> wrote:

From NotifyObjectEmitted you should be able to get to any function that’s visible via either the ELF headers or the DWARF information.  Do you have functions that don’t show up in either of those places?

 

-Andy

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Andrew MacPherson
Sent: Wednesday, November 13, 2013 9:23 AM
To: Reid Kleckner
Cc: [hidden email]
Subject: Re: [LLVMdev] Stack traces from JIT code

 

We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind.

Cheers,
Andrew

 

On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <[hidden email]> wrote:

Writing your own JITEventListener is probably the best way to capture traces from production.  Using the existing gdb/JIT interface support is probably far too heavyweight for production.  If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production.

 

On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <[hidden email]> wrote:

[Resending this email after subscribing to the group. Got an error message the previous time around]

 

Hey guys,

 

In our project we have the following workflow:

 

- Given a user request, we construct a C++ program to service the request

- We create a compiler invocation to compile that program into an IR module

- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

 

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

 

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

 

Will that work? Is there a simpler way?

 

Thanks,

-- Priyendra

 

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Stack traces from JIT code

Kaylor, Andrew

Yeah, take a look at the IntelJITEventListener code (in lib/ExecutionEngine/IntelJITEvents).  It uses the debug info to find function names.

 

-Andy

 

From: Andrew MacPherson [mailto:[hidden email]]
Sent: Wednesday, November 13, 2013 1:12 PM
To: Kaylor, Andrew
Cc: Reid Kleckner; [hidden email]
Subject: Re: [LLVMdev] Stack traces from JIT code

 

Hi Andy,

In the NotifyObjectEmitted method of our derived JITEventListener class we use the begin_symbol() iterator to walk the object's symbols looking for functions and only functions marked with ExternalLinkage seem to show up. I'm not sure how I would access the Dwarf info from within there, is there a way?

Thanks,
Andrew

 

On Wed, Nov 13, 2013 at 8:23 PM, Kaylor, Andrew <[hidden email]> wrote:

From NotifyObjectEmitted you should be able to get to any function that’s visible via either the ELF headers or the DWARF information.  Do you have functions that don’t show up in either of those places?

 

-Andy

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Andrew MacPherson
Sent: Wednesday, November 13, 2013 9:23 AM
To: Reid Kleckner
Cc: [hidden email]
Subject: Re: [LLVMdev] Stack traces from JIT code

 

We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind.

Cheers,
Andrew

 

On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <[hidden email]> wrote:

Writing your own JITEventListener is probably the best way to capture traces from production.  Using the existing gdb/JIT interface support is probably far too heavyweight for production.  If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production.

 

On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <[hidden email]> wrote:

[Resending this email after subscribing to the group. Got an error message the previous time around]

 

Hey guys,

 

In our project we have the following workflow:

 

- Given a user request, we construct a C++ program to service the request

- We create a compiler invocation to compile that program into an IR module

- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

 

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

 

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

 

Will that work? Is there a simpler way?

 

Thanks,

-- Priyendra

 

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Stack traces from JIT code

Andrew MacPherson
Thanks Andy. It looks like the IntelJITEventListener uses the ObjectImage begin_symbols() iterator as well so would not include functions without ExternalLinkage but I hadn't seen these Dwarf parsing functions before, they look quite useful!

Cheers,
Andrew


On Thu, Nov 14, 2013 at 12:05 AM, Kaylor, Andrew <[hidden email]> wrote:

Yeah, take a look at the IntelJITEventListener code (in lib/ExecutionEngine/IntelJITEvents).  It uses the debug info to find function names.

 

-Andy

 

From: Andrew MacPherson [mailto:[hidden email]]
Sent: Wednesday, November 13, 2013 1:12 PM
To: Kaylor, Andrew
Cc: Reid Kleckner; [hidden email]


Subject: Re: [LLVMdev] Stack traces from JIT code

 

Hi Andy,

In the NotifyObjectEmitted method of our derived JITEventListener class we use the begin_symbol() iterator to walk the object's symbols looking for functions and only functions marked with ExternalLinkage seem to show up. I'm not sure how I would access the Dwarf info from within there, is there a way?

Thanks,
Andrew

 

On Wed, Nov 13, 2013 at 8:23 PM, Kaylor, Andrew <[hidden email]> wrote:

From NotifyObjectEmitted you should be able to get to any function that’s visible via either the ELF headers or the DWARF information.  Do you have functions that don’t show up in either of those places?

 

-Andy

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Andrew MacPherson
Sent: Wednesday, November 13, 2013 9:23 AM
To: Reid Kleckner
Cc: [hidden email]
Subject: Re: [LLVMdev] Stack traces from JIT code

 

We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind.

Cheers,
Andrew

 

On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <[hidden email]> wrote:

Writing your own JITEventListener is probably the best way to capture traces from production.  Using the existing gdb/JIT interface support is probably far too heavyweight for production.  If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production.

 

On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <[hidden email]> wrote:

[Resending this email after subscribing to the group. Got an error message the previous time around]

 

Hey guys,

 

In our project we have the following workflow:

 

- Given a user request, we construct a C++ program to service the request

- We create a compiler invocation to compile that program into an IR module

- We use JIT (and optionally MCJIT) to convert the IR into executable code and run it

 

Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward.

 

However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization.

 

Will that work? Is there a simpler way?

 

Thanks,

-- Priyendra

 

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 

 



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev