ORC and relocations

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

ORC and relocations

Eugene Rozenfeld

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

ORC and relocations

Eugene Rozenfeld

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Lang Hames
Hi Eugene,

There's no way to 'skip' application of a relocation.

You could use the NotifyLoadedFtor and libObject to scan the object ahead of time and record the relocations that you'd like to override, then you could supply a custom Resolver that just returns '0' for the symbols that you want to handle manually. 

There's no way to handle distinct relocations for the same symbol differently: For performance reasons we only look up each symbol once and re-use the supplied address to apply every relocation for that symbol.

Is that sufficient to support your use-case?

Cheers,
Lang.

On Wed, Jun 24, 2015 at 12:03 AM, Eugene Rozenfeld <[hidden email]> wrote:

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene

 



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Eugene Rozenfeld

Hi Lang,

 

Thank you for your reply. I tried to supply a custom Resolver that just returns 0 for the symbols I want to handle manually, but that results in a fatal error below. If that can be changed, then yes,

it would be sufficient for my use case.

 

Thanks,

 

Eugene

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

        << format("0x%lx", Addr) << "\n");

      // This list may have been updated when we called getSymbolAddress, so

      // don't change this code to get the list earlier.

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, Addr);

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Wednesday, June 24, 2015 4:42 PM
To: Eugene Rozenfeld
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

There's no way to 'skip' application of a relocation.

 

You could use the NotifyLoadedFtor and libObject to scan the object ahead of time and record the relocations that you'd like to override, then you could supply a custom Resolver that just returns '0' for the symbols that you want to handle manually. 

 

There's no way to handle distinct relocations for the same symbol differently: For performance reasons we only look up each symbol once and re-use the supplied address to apply every relocation for that symbol.

 

Is that sufficient to support your use-case?

 

Cheers,

Lang.

 

On Wed, Jun 24, 2015 at 12:03 AM, Eugene Rozenfeld <[hidden email]> wrote:

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene

 

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Eugene Rozenfeld
In reply to this post by Lang Hames

Hi Lang,

 

Can you please let me know you think it would be right to modify RuntimeDyldImpl::resolveExternalSymbols to allow resolvers to return 0 addresses? Something like this would be ideal for me:

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      if (Addr) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

Thanks,

 

Eugene

From: Eugene Rozenfeld
Sent: Wednesday, June 24, 2015 4:47 PM
To: 'Lang Hames'
Cc: [hidden email]
Subject: RE: ORC and relocations

 

Hi Lang,

 

Thank you for your reply. I tried to supply a custom Resolver that just returns 0 for the symbols I want to handle manually, but that results in a fatal error below. If that can be changed, then yes,

it would be sufficient for my use case.

 

Thanks,

 

Eugene

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

        << format("0x%lx", Addr) << "\n");

      // This list may have been updated when we called getSymbolAddress, so

      // don't change this code to get the list earlier.

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, Addr);

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

From: Lang Hames [[hidden email]]
Sent: Wednesday, June 24, 2015 4:42 PM
To: Eugene Rozenfeld
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

There's no way to 'skip' application of a relocation.

 

You could use the NotifyLoadedFtor and libObject to scan the object ahead of time and record the relocations that you'd like to override, then you could supply a custom Resolver that just returns '0' for the symbols that you want to handle manually. 

 

There's no way to handle distinct relocations for the same symbol differently: For performance reasons we only look up each symbol once and re-use the supplied address to apply every relocation for that symbol.

 

Is that sufficient to support your use-case?

 

Cheers,

Lang.

 

On Wed, Jun 24, 2015 at 12:03 AM, Eugene Rozenfeld <[hidden email]> wrote:

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene

 

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Lang Hames
Hi Eugene,

Returning null addresses to the linker is unusual enough that I'd prefer to keep it as an error.

Could you return a non-zero marker value, e.g. all-ones? If not, we should add an option to RuntimeDyld like "AllowUnresolvedSymbols" and change the error condition to if '(!Addr && !AllowUnresolvedSymbols)'.

Cheers,
Lang.


On Thu, Jun 25, 2015 at 6:35 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Can you please let me know you think it would be right to modify RuntimeDyldImpl::resolveExternalSymbols to allow resolvers to return 0 addresses? Something like this would be ideal for me:

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      if (Addr) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

Thanks,

 

Eugene

From: Eugene Rozenfeld
Sent: Wednesday, June 24, 2015 4:47 PM
To: 'Lang Hames'
Cc: [hidden email]
Subject: RE: ORC and relocations

 

Hi Lang,

 

Thank you for your reply. I tried to supply a custom Resolver that just returns 0 for the symbols I want to handle manually, but that results in a fatal error below. If that can be changed, then yes,

it would be sufficient for my use case.

 

Thanks,

 

Eugene

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

        << format("0x%lx", Addr) << "\n");

      // This list may have been updated when we called getSymbolAddress, so

      // don't change this code to get the list earlier.

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, Addr);

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

From: Lang Hames [[hidden email]]
Sent: Wednesday, June 24, 2015 4:42 PM
To: Eugene Rozenfeld
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

There's no way to 'skip' application of a relocation.

 

You could use the NotifyLoadedFtor and libObject to scan the object ahead of time and record the relocations that you'd like to override, then you could supply a custom Resolver that just returns '0' for the symbols that you want to handle manually. 

 

There's no way to handle distinct relocations for the same symbol differently: For performance reasons we only look up each symbol once and re-use the supplied address to apply every relocation for that symbol.

 

Is that sufficient to support your use-case?

 

Cheers,

Lang.

 

On Wed, Jun 24, 2015 at 12:03 AM, Eugene Rozenfeld <[hidden email]> wrote:

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene

 

 



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Eugene Rozenfeld

Hi Lang,

 

Yes, I can return a non-zero marker value. Are you ok with this version?

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      // If Resolver returned UINT64_MAX, the client wants to handle this symbol
      // manually and we shouldn't resolve its relocations.

      if (Addr != UINT64_MAX) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

Thanks,

 

Eugene

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, June 26, 2015 10:28 AM
To: Eugene Rozenfeld
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Returning null addresses to the linker is unusual enough that I'd prefer to keep it as an error.

 

Could you return a non-zero marker value, e.g. all-ones? If not, we should add an option to RuntimeDyld like "AllowUnresolvedSymbols" and change the error condition to if '(!Addr && !AllowUnresolvedSymbols)'.

 

Cheers,

Lang.

 

 

On Thu, Jun 25, 2015 at 6:35 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Can you please let me know you think it would be right to modify RuntimeDyldImpl::resolveExternalSymbols to allow resolvers to return 0 addresses? Something like this would be ideal for me:

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      if (Addr) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

Thanks,

 

Eugene

From: Eugene Rozenfeld
Sent: Wednesday, June 24, 2015 4:47 PM
To: 'Lang Hames'
Cc: [hidden email]
Subject: RE: ORC and relocations

 

Hi Lang,

 

Thank you for your reply. I tried to supply a custom Resolver that just returns 0 for the symbols I want to handle manually, but that results in a fatal error below. If that can be changed, then yes,

it would be sufficient for my use case.

 

Thanks,

 

Eugene 

 

From: Lang Hames [[hidden email]]
Sent: Wednesday, June 24, 2015 4:42 PM
To: Eugene Rozenfeld
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

There's no way to 'skip' application of a relocation.

 

You could use the NotifyLoadedFtor and libObject to scan the object ahead of time and record the relocations that you'd like to override, then you could supply a custom Resolver that just returns '0' for the symbols that you want to handle manually. 

 

There's no way to handle distinct relocations for the same symbol differently: For performance reasons we only look up each symbol once and re-use the supplied address to apply every relocation for that symbol.

 

Is that sufficient to support your use-case?

 

Cheers,

Lang.

 

On Wed, Jun 24, 2015 at 12:03 AM, Eugene Rozenfeld <[hidden email]> wrote:

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene

 

 

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Lang Hames
Hi Eugene,

Sorry for the delayed reply. This looks good to me - I've applied it (along with some extra code to make it testable) in r241383.

Cheers,
Lang.

On Mon, Jun 29, 2015 at 7:31 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Yes, I can return a non-zero marker value. Are you ok with this version?

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      // If Resolver returned UINT64_MAX, the client wants to handle this symbol
      // manually and we shouldn't resolve its relocations.

      if (Addr != UINT64_MAX) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

Thanks,

 

Eugene

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, June 26, 2015 10:28 AM
To: Eugene Rozenfeld
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Returning null addresses to the linker is unusual enough that I'd prefer to keep it as an error.

 

Could you return a non-zero marker value, e.g. all-ones? If not, we should add an option to RuntimeDyld like "AllowUnresolvedSymbols" and change the error condition to if '(!Addr && !AllowUnresolvedSymbols)'.

 

Cheers,

Lang.

 

 

On Thu, Jun 25, 2015 at 6:35 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Can you please let me know you think it would be right to modify RuntimeDyldImpl::resolveExternalSymbols to allow resolvers to return 0 addresses? Something like this would be ideal for me:

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      if (Addr) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

Thanks,

 

Eugene

From: Eugene Rozenfeld
Sent: Wednesday, June 24, 2015 4:47 PM
To: 'Lang Hames'
Cc: [hidden email]
Subject: RE: ORC and relocations

 

Hi Lang,

 

Thank you for your reply. I tried to supply a custom Resolver that just returns 0 for the symbols I want to handle manually, but that results in a fatal error below. If that can be changed, then yes,

it would be sufficient for my use case.

 

Thanks,

 

Eugene 

 

From: Lang Hames [[hidden email]]
Sent: Wednesday, June 24, 2015 4:42 PM
To: Eugene Rozenfeld
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

There's no way to 'skip' application of a relocation.

 

You could use the NotifyLoadedFtor and libObject to scan the object ahead of time and record the relocations that you'd like to override, then you could supply a custom Resolver that just returns '0' for the symbols that you want to handle manually. 

 

There's no way to handle distinct relocations for the same symbol differently: For performance reasons we only look up each symbol once and re-use the supplied address to apply every relocation for that symbol.

 

Is that sufficient to support your use-case?

 

Cheers,

Lang.

 

On Wed, Jun 24, 2015 at 12:03 AM, Eugene Rozenfeld <[hidden email]> wrote:

Hello,

 

I’m working on LLILC (a jit for the CoreCLR built on ORC), in particular, on using LLILC as an ngen jit. I would like to have an ability to be notified of relocations that ObjectLinkingLayer  is applying and to be able to tell the linking layer not to resolve certain relocations for external symbols (so that the client can do some custom resolutions later). The only way I found of looking at relocations in the client is via NotifyLoadedFtor notifications but I couldn’t find a way of blocking a relocation resolution.

 

One way to achieve that is to change the Resolver::findSymbol api to allow clients to indicate that the relocations for the symbol shouldn’t be resolved and update RuntimeDyldImpl::resolveExternalSymbols accordingly. Would this be a reasonable approach?

 

Thanks,

 

Eugene

 

 

 



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Eugene Rozenfeld

Hi Lang,

 

It turns out I also need an ability to tell the object linking layer not to apply any relocations. I need to skip this step below.

The only way I can see I can achieve that is by creating my own ObjectLinkingLayer that would duplicate almost all of orc::ObjectLinkingLayer.

I’d like to avoid that. An alternative it to pass a flag to orc::ObjectLinkingLayer constructor and orc::ObjectLinkingLayer::ConcreteLinkedObjectSet constructor

to indicate whether relocation resolution should be performed. Would you be ok with such a change?

 

Thanks,

 

Eugene

 

template <typename NotifyLoadedFtor = DoNothingOnNotifyLoaded>

class ObjectLinkingLayer : public ObjectLinkingLayerBase {

private:

 

  template <typename MemoryManagerPtrTtypename SymbolResolverPtrT>

  class ConcreteLinkedObjectSet : public LinkedObjectSet {

  public:

    ConcreteLinkedObjectSet(MemoryManagerPtrT MemMgr,

                            SymbolResolverPtrT Resolver)

      : LinkedObjectSet(*MemMgr, *Resolver), MemMgr(std::move(MemMgr)),

        Resolver(std::move(Resolver)) { }

 

    void Finalize() override {

      State = Finalizing;

      RTDyld->resolveRelocations();

      RTDyld->registerEHFrames();

      MemMgr->finalizeMemory();

      OwnedBuffers.clear();

      State = Finalized;

    }

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, July 3, 2015 6:36 PM
To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Sorry for the delayed reply. This looks good to me - I've applied it (along with some extra code to make it testable) in r241383.

 

Cheers,

Lang.

 

On Mon, Jun 29, 2015 at 7:31 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Yes, I can return a non-zero marker value. Are you ok with this version?

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      // If Resolver returned UINT64_MAX, the client wants to handle this symbol
      // manually and we shouldn't resolve its relocations.

      if (Addr != UINT64_MAX) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

Thanks,

 

Eugene


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Lang Hames
Hi Eugene,

Skipping the call to resolveRelocations would disable many (if not all) internal relocations too. Is that the desired behavior?

At that point there's not much left for RuntimeDyld (or the ObjectLinkingLayer) to do. Would something like a NoopLinkingLayer be a workable solution?

Cheers,
Lang.


On Wed, Jul 22, 2015 at 7:26 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

It turns out I also need an ability to tell the object linking layer not to apply any relocations. I need to skip this step below.

The only way I can see I can achieve that is by creating my own ObjectLinkingLayer that would duplicate almost all of orc::ObjectLinkingLayer.

I’d like to avoid that. An alternative it to pass a flag to orc::ObjectLinkingLayer constructor and orc::ObjectLinkingLayer::ConcreteLinkedObjectSet constructor

to indicate whether relocation resolution should be performed. Would you be ok with such a change?

 

Thanks,

 

Eugene

 

template <typename NotifyLoadedFtor = DoNothingOnNotifyLoaded>

class ObjectLinkingLayer : public ObjectLinkingLayerBase {

private:

 

  template <typename MemoryManagerPtrTtypename SymbolResolverPtrT>

  class ConcreteLinkedObjectSet : public LinkedObjectSet {

  public:

    ConcreteLinkedObjectSet(MemoryManagerPtrT MemMgr,

                            SymbolResolverPtrT Resolver)

      : LinkedObjectSet(*MemMgr, *Resolver), MemMgr(std::move(MemMgr)),

        Resolver(std::move(Resolver)) { }

 

    void Finalize() override {

      State = Finalizing;

      RTDyld->resolveRelocations();

      RTDyld->registerEHFrames();

      MemMgr->finalizeMemory();

      OwnedBuffers.clear();

      State = Finalized;

    }

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, July 3, 2015 6:36 PM
To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Sorry for the delayed reply. This looks good to me - I've applied it (along with some extra code to make it testable) in r241383.

 

Cheers,

Lang.

 

On Mon, Jun 29, 2015 at 7:31 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Yes, I can return a non-zero marker value. Are you ok with this version?

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      // If Resolver returned UINT64_MAX, the client wants to handle this symbol
      // manually and we shouldn't resolve its relocations.

      if (Addr != UINT64_MAX) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

Thanks,

 

Eugene



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Eugene Rozenfeld

Yes, I’m handling all internal and external relocations manually in NotifyLoadedFtor and I already verified that I get the behavior I need if I comment out the call to resolveRelocations.

 

I would like to reuse ObjectLinkingLayer::addObjectSet (which eventually calls RuntimeDyld::loadObject), which has the right calls to the memory manager and also RuntimeDyld::registerEHFrames.

 

I understand that resolveRelocations is normally the main job of ObjectLinkingLayer and as I said I can create my own ObjectLinkingLayer or something instead of it that has these calls but it doesn’t feel right if all I need to reuse ObjectLinkingLayer is to avoid the resolveRelocations call.

 

Eugene

 

From: Lang Hames [mailto:[hidden email]]
Sent: Wednesday, July 22, 2015 8:17 PM
To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Skipping the call to resolveRelocations would disable many (if not all) internal relocations too. Is that the desired behavior?

 

At that point there's not much left for RuntimeDyld (or the ObjectLinkingLayer) to do. Would something like a NoopLinkingLayer be a workable solution?

 

Cheers,

Lang.

 

 

On Wed, Jul 22, 2015 at 7:26 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

It turns out I also need an ability to tell the object linking layer not to apply any relocations. I need to skip this step below.

The only way I can see I can achieve that is by creating my own ObjectLinkingLayer that would duplicate almost all of orc::ObjectLinkingLayer.

I’d like to avoid that. An alternative it to pass a flag to orc::ObjectLinkingLayer constructor and orc::ObjectLinkingLayer::ConcreteLinkedObjectSet constructor

to indicate whether relocation resolution should be performed. Would you be ok with such a change?

 

Thanks,

 

Eugene

 

template <typename NotifyLoadedFtor = DoNothingOnNotifyLoaded>

class ObjectLinkingLayer : public ObjectLinkingLayerBase {

private:

 

  template <typename MemoryManagerPtrTtypename SymbolResolverPtrT>

  class ConcreteLinkedObjectSet : public LinkedObjectSet {

  public:

    ConcreteLinkedObjectSet(MemoryManagerPtrT MemMgr,

                            SymbolResolverPtrT Resolver)

      : LinkedObjectSet(*MemMgr, *Resolver), MemMgr(std::move(MemMgr)),

        Resolver(std::move(Resolver)) { }

 

    void Finalize() override {

      State = Finalizing;

      RTDyld->resolveRelocations();

      RTDyld->registerEHFrames();

      MemMgr->finalizeMemory();

      OwnedBuffers.clear();

      State = Finalized;

    }

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, July 3, 2015 6:36 PM
To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Sorry for the delayed reply. This looks good to me - I've applied it (along with some extra code to make it testable) in r241383.

 

Cheers,

Lang.

 

On Mon, Jun 29, 2015 at 7:31 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Yes, I can return a non-zero marker value. Are you ok with this version?

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      // If Resolver returned UINT64_MAX, the client wants to handle this symbol
      // manually and we shouldn't resolve its relocations.

      if (Addr != UINT64_MAX) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

Thanks,

 

Eugene

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Lang Hames
Hi Eugene,

Sorry for the delayed reply. Custom relocations weren't something I had in mind when I designed Orc, so they raise some interesting design questions which I don't have good answers to yet. (E.g. The interface for the Orc layer concept assumes that there's a RuntimeDyld instance embedded at the bottom of the stack. That's why addModuleSet takes a MemoryManager and SymbolResolver. If there's no RuntimeDyld instance at the bottom of the stack, it's not obvious that the interface should require those).

For now I recommend just creating a local copy of the ObjectLinkingLayer and removing the call to resolveRelocations. I'll keep thinking about the best way to support this use case going forward.

Cheers,
Lang.


On Wed, Jul 22, 2015 at 11:51 PM, Eugene Rozenfeld <[hidden email]> wrote:

Yes, I’m handling all internal and external relocations manually in NotifyLoadedFtor and I already verified that I get the behavior I need if I comment out the call to resolveRelocations.

 

I would like to reuse ObjectLinkingLayer::addObjectSet (which eventually calls RuntimeDyld::loadObject), which has the right calls to the memory manager and also RuntimeDyld::registerEHFrames.

 

I understand that resolveRelocations is normally the main job of ObjectLinkingLayer and as I said I can create my own ObjectLinkingLayer or something instead of it that has these calls but it doesn’t feel right if all I need to reuse ObjectLinkingLayer is to avoid the resolveRelocations call.

 

Eugene

 

From: Lang Hames [mailto:[hidden email]]
Sent: Wednesday, July 22, 2015 8:17 PM


To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Skipping the call to resolveRelocations would disable many (if not all) internal relocations too. Is that the desired behavior?

 

At that point there's not much left for RuntimeDyld (or the ObjectLinkingLayer) to do. Would something like a NoopLinkingLayer be a workable solution?

 

Cheers,

Lang.

 

 

On Wed, Jul 22, 2015 at 7:26 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

It turns out I also need an ability to tell the object linking layer not to apply any relocations. I need to skip this step below.

The only way I can see I can achieve that is by creating my own ObjectLinkingLayer that would duplicate almost all of orc::ObjectLinkingLayer.

I’d like to avoid that. An alternative it to pass a flag to orc::ObjectLinkingLayer constructor and orc::ObjectLinkingLayer::ConcreteLinkedObjectSet constructor

to indicate whether relocation resolution should be performed. Would you be ok with such a change?

 

Thanks,

 

Eugene

 

template <typename NotifyLoadedFtor = DoNothingOnNotifyLoaded>

class ObjectLinkingLayer : public ObjectLinkingLayerBase {

private:

 

  template <typename MemoryManagerPtrTtypename SymbolResolverPtrT>

  class ConcreteLinkedObjectSet : public LinkedObjectSet {

  public:

    ConcreteLinkedObjectSet(MemoryManagerPtrT MemMgr,

                            SymbolResolverPtrT Resolver)

      : LinkedObjectSet(*MemMgr, *Resolver), MemMgr(std::move(MemMgr)),

        Resolver(std::move(Resolver)) { }

 

    void Finalize() override {

      State = Finalizing;

      RTDyld->resolveRelocations();

      RTDyld->registerEHFrames();

      MemMgr->finalizeMemory();

      OwnedBuffers.clear();

      State = Finalized;

    }

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, July 3, 2015 6:36 PM
To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Sorry for the delayed reply. This looks good to me - I've applied it (along with some extra code to make it testable) in r241383.

 

Cheers,

Lang.

 

On Mon, Jun 29, 2015 at 7:31 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Yes, I can return a non-zero marker value. Are you ok with this version?

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      // If Resolver returned UINT64_MAX, the client wants to handle this symbol
      // manually and we shouldn't resolve its relocations.

      if (Addr != UINT64_MAX) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

Thanks,

 

Eugene

 



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: ORC and relocations

Eugene Rozenfeld

Hi Lang,

 

Sounds good, I’ll create a local copy of ObjectLinkingLayer. Hopefully it won’t be necessary in the future.

 

Thanks,

 

Eugene

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, July 24, 2015 1:24 PM
To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Sorry for the delayed reply. Custom relocations weren't something I had in mind when I designed Orc, so they raise some interesting design questions which I don't have good answers to yet. (E.g. The interface for the Orc layer concept assumes that there's a RuntimeDyld instance embedded at the bottom of the stack. That's why addModuleSet takes a MemoryManager and SymbolResolver. If there's no RuntimeDyld instance at the bottom of the stack, it's not obvious that the interface should require those).

 

For now I recommend just creating a local copy of the ObjectLinkingLayer and removing the call to resolveRelocations. I'll keep thinking about the best way to support this use case going forward.

 

Cheers,

Lang.

 

 

On Wed, Jul 22, 2015 at 11:51 PM, Eugene Rozenfeld <[hidden email]> wrote:

Yes, I’m handling all internal and external relocations manually in NotifyLoadedFtor and I already verified that I get the behavior I need if I comment out the call to resolveRelocations.

 

I would like to reuse ObjectLinkingLayer::addObjectSet (which eventually calls RuntimeDyld::loadObject), which has the right calls to the memory manager and also RuntimeDyld::registerEHFrames.

 

I understand that resolveRelocations is normally the main job of ObjectLinkingLayer and as I said I can create my own ObjectLinkingLayer or something instead of it that has these calls but it doesn’t feel right if all I need to reuse ObjectLinkingLayer is to avoid the resolveRelocations call.

 

Eugene

 

From: Lang Hames [mailto:[hidden email]]
Sent: Wednesday, July 22, 2015 8:17 PM


To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Skipping the call to resolveRelocations would disable many (if not all) internal relocations too. Is that the desired behavior?

 

At that point there's not much left for RuntimeDyld (or the ObjectLinkingLayer) to do. Would something like a NoopLinkingLayer be a workable solution?

 

Cheers,

Lang.

 

 

On Wed, Jul 22, 2015 at 7:26 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

It turns out I also need an ability to tell the object linking layer not to apply any relocations. I need to skip this step below.

The only way I can see I can achieve that is by creating my own ObjectLinkingLayer that would duplicate almost all of orc::ObjectLinkingLayer.

I’d like to avoid that. An alternative it to pass a flag to orc::ObjectLinkingLayer constructor and orc::ObjectLinkingLayer::ConcreteLinkedObjectSet constructor

to indicate whether relocation resolution should be performed. Would you be ok with such a change?

 

Thanks,

 

Eugene

 

template <typename NotifyLoadedFtor = DoNothingOnNotifyLoaded>

class ObjectLinkingLayer : public ObjectLinkingLayerBase {

private:

 

  template <typename MemoryManagerPtrTtypename SymbolResolverPtrT>

  class ConcreteLinkedObjectSet : public LinkedObjectSet {

  public:

    ConcreteLinkedObjectSet(MemoryManagerPtrT MemMgr,

                            SymbolResolverPtrT Resolver)

      : LinkedObjectSet(*MemMgr, *Resolver), MemMgr(std::move(MemMgr)),

        Resolver(std::move(Resolver)) { }

 

    void Finalize() override {

      State = Finalizing;

      RTDyld->resolveRelocations();

      RTDyld->registerEHFrames();

      MemMgr->finalizeMemory();

      OwnedBuffers.clear();

      State = Finalized;

    }

 

 

From: Lang Hames [mailto:[hidden email]]
Sent: Friday, July 3, 2015 6:36 PM
To: Eugene Rozenfeld <[hidden email]>
Cc: [hidden email]
Subject: Re: ORC and relocations

 

Hi Eugene,

 

Sorry for the delayed reply. This looks good to me - I've applied it (along with some extra code to make it testable) in r241383.

 

Cheers,

Lang.

 

On Mon, Jun 29, 2015 at 7:31 PM, Eugene Rozenfeld <[hidden email]> wrote:

Hi Lang,

 

Yes, I can return a non-zero marker value. Are you ok with this version?

 

void RuntimeDyldImpl::resolveExternalSymbols() {

  while (!ExternalSymbolRelocations.empty()) {

    StringMap<RelocationList>::iterator i = ExternalSymbolRelocations.begin();

 

    StringRef Name = i->first();

    if (Name.size() == 0) {

      // This is an absolute symbol, use an address of zero.

      DEBUG(dbgs() << "Resolving absolute relocations."

                   << "\n");

      RelocationList &Relocs = i->second;

      resolveRelocationList(Relocs, 0);

    } else {

      uint64_t Addr = 0;

      RTDyldSymbolTable::const_iterator Loc = GlobalSymbolTable.find(Name);

      if (Loc == GlobalSymbolTable.end()) {

        // This is an external symbol, try to get its address from the symbol

        // resolver.

        Addr = Resolver.findSymbol(Name.data()).getAddress();

        // The call to getSymbolAddress may have caused additional modules to

        // be loaded, which may have added new entries to the

        // ExternalSymbolRelocations map.  Consquently, we need to update our

        // iterator.  This is also why retrieval of the relocation list

        // associated with this symbol is deferred until below this point.

        // New entries may have been added to the relocation list.

        i = ExternalSymbolRelocations.find(Name);

      } else {

        // We found the symbol in our global table.  It was probably in a

        // Module that we loaded previously.

        const auto &SymInfo = Loc->second;

        Addr = getSectionLoadAddress(SymInfo.getSectionID()) +

               SymInfo.getOffset();

      }

 

      // FIXME: Implement error handling that doesn't kill the host program!

      if (!Addr) {

        report_fatal_error("Program used external function '" + Name +

                           "' which could not be resolved!");

      }

 

      // If Resolver returned UINT64_MAX, the client wants to handle this symbol
      // manually and we shouldn't resolve its relocations.

      if (Addr != UINT64_MAX) {

        DEBUG(dbgs() << "Resolving relocations Name: " << Name << "\t"

          << format("0x%lx", Addr) << "\n");

        // This list may have been updated when we called getSymbolAddress, so

        // don't change this code to get the list earlier.

        RelocationList &Relocs = i->second;

        resolveRelocationList(Relocs, Addr);

      }

    }

 

    ExternalSymbolRelocations.erase(i);

  }

}

 

 

Thanks,

 

Eugene

 

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev