[llvm-dev] ASan port for Myriad RTEMS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
I have ported ASan in LLVM to Myriad RTEMS, and I would like to
upstream the port.  Below is the design doc.  Feedback welcome.

https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

The port is expected to work with modified versions of RTEMS and
newlib.  I have a git repo with changes to those projects, that I can
make available if there is interest.

Here is the patch series:
https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
Myriad
https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic for
Myriad
https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
RTEMS platform
https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port to
Myriad RTEMS
https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS memory
map
https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
gap
https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
before entering ASan run-time
https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to be
reused by Myriad RTEMS
https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion when
reporting Mmap failure
https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to support
thread restart
https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack to
be initialized during thread init
https://reviews.llvm.org/D46468 [asan] Add target-specific files for Myriad
RTEMS port
https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
sparcv8

Thanks,

Walter
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
Hey,

I work on fuchsia symbolizer stuff. I don't know if you guys already have an external symbolizer but I'm working on making one right now and I plan on making one backed by LLVM that can be run host-side or target-side. I'd like to contribute that back to llvm ideally. What do you guys have so far? I have a prototype in golang that just spins up an instance of llvm-symbolizer currently. It isn't fully functional at the moment and it has some fuchsia specific stuff as well: https://fuchsia.googlesource.com/tools/+/master/symbolize/ but it should usable soon. The full spec we'd like to implement can be read here: https://fuchsia.googlesource.com/zircon/+/master/docs/symbolizer_markup.md

Best,
Jake

On Fri, May 4, 2018 at 2:00 PM Walter Lee via llvm-dev <[hidden email]> wrote:
I have ported ASan in LLVM to Myriad RTEMS, and I would like to
upstream the port.  Below is the design doc.  Feedback welcome.

https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

The port is expected to work with modified versions of RTEMS and
newlib.  I have a git repo with changes to those projects, that I can
make available if there is interest.

Here is the patch series:
https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
Myriad
https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic for
Myriad
https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
RTEMS platform
https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port to
Myriad RTEMS
https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS memory
map
https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
gap
https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
before entering ASan run-time
https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to be
reused by Myriad RTEMS
https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion when
reporting Mmap failure
https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to support
thread restart
https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack to
be initialized during thread init
https://reviews.llvm.org/D46468 [asan] Add target-specific files for Myriad
RTEMS port
https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
sparcv8

Thanks,

Walter
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev
Hi Walter, 

I've done a first quick scan. 
Overall looks reasonable, but I'd like to try reducing the number of newly introduced platform-specific ifs. 

Vitaly, please also take a look (once my initial comments are addressed). 

One outstanding issue is your problem with initialization vs checking, which requires you to insert so many ifs. 
Is there any chance you can avoid this? 
If you control the OS, surely you can do at least minimal initialization of the shadow at a proper moment... 

--kcc 

On Fri, May 4, 2018 at 2:00 PM Walter Lee via llvm-dev <[hidden email]> wrote:
I have ported ASan in LLVM to Myriad RTEMS, and I would like to
upstream the port.  Below is the design doc.  Feedback welcome.

https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

The port is expected to work with modified versions of RTEMS and
newlib.  I have a git repo with changes to those projects, that I can
make available if there is interest.

Here is the patch series:
https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
Myriad
https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic for
Myriad
https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
RTEMS platform
https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port to
Myriad RTEMS
https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS memory
map
https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
gap
https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
before entering ASan run-time
https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to be
reused by Myriad RTEMS
https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion when
reporting Mmap failure
https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to support
thread restart
https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack to
be initialized during thread init
https://reviews.llvm.org/D46468 [asan] Add target-specific files for Myriad
RTEMS port
https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
sparcv8

Thanks,

Walter
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
On RAM...

You chose the 32-byte shadow granularity to reduce the RAM overhead,
but I am afraid this will actually increase it due to extra alignment requirements, 
especially if an average allocation on your typical application is small.   

The pointers are 32-bit, right? 
Given how RAM-constrained your environment is, maybe you should consider something more like HWASAN instead of ASAN. 
But you may not have enough address bits. :(

--kcc 

 

On Fri, May 4, 2018 at 3:10 PM Kostya Serebryany <[hidden email]> wrote:
Hi Walter, 

I've done a first quick scan. 
Overall looks reasonable, but I'd like to try reducing the number of newly introduced platform-specific ifs. 

Vitaly, please also take a look (once my initial comments are addressed). 

One outstanding issue is your problem with initialization vs checking, which requires you to insert so many ifs. 
Is there any chance you can avoid this? 
If you control the OS, surely you can do at least minimal initialization of the shadow at a proper moment... 

--kcc 

On Fri, May 4, 2018 at 2:00 PM Walter Lee via llvm-dev <[hidden email]> wrote:
I have ported ASan in LLVM to Myriad RTEMS, and I would like to
upstream the port.  Below is the design doc.  Feedback welcome.

https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

The port is expected to work with modified versions of RTEMS and
newlib.  I have a git repo with changes to those projects, that I can
make available if there is interest.

Here is the patch series:
https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
Myriad
https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic for
Myriad
https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
RTEMS platform
https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port to
Myriad RTEMS
https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS memory
map
https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
gap
https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
before entering ASan run-time
https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to be
reused by Myriad RTEMS
https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion when
reporting Mmap failure
https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to support
thread restart
https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack to
be initialized during thread init
https://reviews.llvm.org/D46468 [asan] Add target-specific files for Myriad
RTEMS port
https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
sparcv8

Thanks,

Walter
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev
Hi Jake.  Thanks for the info.  Where can I keep up to date on the
symbolizer status?

Our symbolizer is provided by the Myriad vendor and integrated into its
host test environment.  It doesn't do much: just look for PC string
patterns and symbolize them using addr2line.

Thanks,

Walter

On Fri, May 4, 2018 at 5:36 PM Jake Ehrlich <[hidden email]> wrote:

> Hey,

> I work on fuchsia symbolizer stuff. I don't know if you guys already have
an external symbolizer but I'm working on making one right now and I plan
on making one backed by LLVM that can be run host-side or target-side. I'd
like to contribute that back to llvm ideally. What do you guys have so far?
I have a prototype in golang that just spins up an instance of
llvm-symbolizer currently. It isn't fully functional at the moment and it
has some fuchsia specific stuff as well:
https://fuchsia.googlesource.com/tools/+/master/symbolize/ but it should
usable soon. The full spec we'd like to implement can be read here:
https://fuchsia.googlesource.com/zircon/+/master/docs/symbolizer_markup.md

> Best,
> Jake

> On Fri, May 4, 2018 at 2:00 PM Walter Lee via llvm-dev <
[hidden email]> wrote:

>> I have ported ASan in LLVM to Myriad RTEMS, and I would like to
>> upstream the port.  Below is the design doc.  Feedback welcome.


https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

>> The port is expected to work with modified versions of RTEMS and
>> newlib.  I have a git repo with changes to those projects, that I can
>> make available if there is interest.

>> Here is the patch series:
>> https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
>> Myriad
>> https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic
for
>> Myriad
>> https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
>> RTEMS platform
>> https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port
to
>> Myriad RTEMS
>> https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS
memory
>> map
>> https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
>> gap
>> https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
>> before entering ASan run-time
>> https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
>> https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to
be
>> reused by Myriad RTEMS
>> https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion
when
>> reporting Mmap failure
>> https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
>> https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to
support
>> thread restart
>> https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack
to
>> be initialized during thread init
>> https://reviews.llvm.org/D46468 [asan] Add target-specific files for
Myriad
>> RTEMS port
>> https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
>> sparcv8

>> Thanks,

>> Walter
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
Eh...there isn't a good place to know whats happening but I can keep you apprised of major updates. For the golang prototype you can check for commits here: https://fuchsia.googlesource.com/tools/+/master. Once we start on the LLVM part we'd need reviewers so you could stay apprised by reviewing my code ;)

On Fri, May 4, 2018 at 6:17 PM Walter Lee <[hidden email]> wrote:
Hi Jake.  Thanks for the info.  Where can I keep up to date on the
symbolizer status?

Our symbolizer is provided by the Myriad vendor and integrated into its
host test environment.  It doesn't do much: just look for PC string
patterns and symbolize them using addr2line.

Thanks,

Walter

On Fri, May 4, 2018 at 5:36 PM Jake Ehrlich <[hidden email]> wrote:

> Hey,

> I work on fuchsia symbolizer stuff. I don't know if you guys already have
an external symbolizer but I'm working on making one right now and I plan
on making one backed by LLVM that can be run host-side or target-side. I'd
like to contribute that back to llvm ideally. What do you guys have so far?
I have a prototype in golang that just spins up an instance of
llvm-symbolizer currently. It isn't fully functional at the moment and it
has some fuchsia specific stuff as well:
https://fuchsia.googlesource.com/tools/+/master/symbolize/ but it should
usable soon. The full spec we'd like to implement can be read here:
https://fuchsia.googlesource.com/zircon/+/master/docs/symbolizer_markup.md

> Best,
> Jake

> On Fri, May 4, 2018 at 2:00 PM Walter Lee via llvm-dev <
[hidden email]> wrote:

>> I have ported ASan in LLVM to Myriad RTEMS, and I would like to
>> upstream the port.  Below is the design doc.  Feedback welcome.


https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

>> The port is expected to work with modified versions of RTEMS and
>> newlib.  I have a git repo with changes to those projects, that I can
>> make available if there is interest.

>> Here is the patch series:
>> https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
>> Myriad
>> https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic
for
>> Myriad
>> https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
>> RTEMS platform
>> https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port
to
>> Myriad RTEMS
>> https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS
memory
>> map
>> https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
>> gap
>> https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
>> before entering ASan run-time
>> https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
>> https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to
be
>> reused by Myriad RTEMS
>> https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion
when
>> reporting Mmap failure
>> https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
>> https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to
support
>> thread restart
>> https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack
to
>> be initialized during thread init
>> https://reviews.llvm.org/D46468 [asan] Add target-specific files for
Myriad
>> RTEMS port
>> https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
>> sparcv8

>> Thanks,

>> Walter
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev
Hi Kostya.  Thanks for the quick feedback.  I will work on addressing your
comments.

In regard to initialization checks, I can eliminate most of them by
initializing the shadow memory very early, but I still need to do something
in two places, __asan_handle_no_return and GetFakeStackFast.  Would it be
ok to have guards for those two places only?

Walter
On Fri, May 4, 2018 at 6:10 PM Kostya Serebryany <[hidden email]> wrote:

> Hi Walter,

> I've done a first quick scan.
> Overall looks reasonable, but I'd like to try reducing the number of
newly introduced platform-specific ifs.

> Vitaly, please also take a look (once my initial comments are addressed).

> One outstanding issue is your problem with initialization vs checking,
which requires you to insert so many ifs.
> Is there any chance you can avoid this?
> If you control the OS, surely you can do at least minimal initialization
of the shadow at a proper moment...

> --kcc

> On Fri, May 4, 2018 at 2:00 PM Walter Lee via llvm-dev <
[hidden email]> wrote:

>> I have ported ASan in LLVM to Myriad RTEMS, and I would like to
>> upstream the port.  Below is the design doc.  Feedback welcome.


https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

>> The port is expected to work with modified versions of RTEMS and
>> newlib.  I have a git repo with changes to those projects, that I can
>> make available if there is interest.

>> Here is the patch series:
>> https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
>> Myriad
>> https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic
for
>> Myriad
>> https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
>> RTEMS platform
>> https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port
to
>> Myriad RTEMS
>> https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS
memory
>> map
>> https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
>> gap
>> https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
>> before entering ASan run-time
>> https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
>> https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to
be
>> reused by Myriad RTEMS
>> https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion
when
>> reporting Mmap failure
>> https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
>> https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to
support
>> thread restart
>> https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack
to
>> be initialized during thread init
>> https://reviews.llvm.org/D46468 [asan] Add target-specific files for
Myriad
>> RTEMS port
>> https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
>> sparcv8

>> Thanks,

>> Walter
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev
On Fri, May 4, 2018 at 6:21 PM Kostya Serebryany <[hidden email]> wrote:

> On RAM...

> You chose the 32-byte shadow granularity to reduce the RAM overhead,
> but I am afraid this will actually increase it due to extra alignment
requirements,
> especially if an average allocation on your typical application is small.


Good point.  I will run our test suite with 8-byte shadow granularity and
do some measurements.

In general, the memory constraint hasn't been as big an issue as I
initially thought.  The mitigating factor is that we are focused on unit
tests as opposed to running the full blown application.


> The pointers are 32-bit, right?
> Given how RAM-constrained your environment is, maybe you should consider
something more like HWASAN instead of ASAN.
> https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html
> But you may not have enough address bits. :(

Pointers are 32 bits.  Interesting idea, but I agree I don't think we have
enough bits.  Even ignoring the sparsely used lower addresses, DDR + cache
bit leaves us with only 2 bits.

Thanks,

Walter
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev


On Fri, May 4, 2018 at 6:29 PM Walter Lee <[hidden email]> wrote:
Hi Kostya.  Thanks for the quick feedback.  I will work on addressing your
comments.

In regard to initialization checks, I can eliminate most of them by
initializing the shadow memory very early,

This will be a very good way to handle this. 
 
but I still need to do something
in two places, __asan_handle_no_return

I think it might be fine. Let me look at the code once the rest is done. 
 
and GetFakeStackFast. 

Not sure. Why don't just disable stack-use-after-return? 
 
Would it be
ok to have guards for those two places only?

Walter
On Fri, May 4, 2018 at 6:10 PM Kostya Serebryany <[hidden email]> wrote:

> Hi Walter,

> I've done a first quick scan.
> Overall looks reasonable, but I'd like to try reducing the number of
newly introduced platform-specific ifs.

> Vitaly, please also take a look (once my initial comments are addressed).

> One outstanding issue is your problem with initialization vs checking,
which requires you to insert so many ifs.
> Is there any chance you can avoid this?
> If you control the OS, surely you can do at least minimal initialization
of the shadow at a proper moment...

> --kcc

> On Fri, May 4, 2018 at 2:00 PM Walter Lee via llvm-dev <
[hidden email]> wrote:

>> I have ported ASan in LLVM to Myriad RTEMS, and I would like to
>> upstream the port.  Below is the design doc.  Feedback welcome.


https://docs.google.com/document/d/1oxmk0xUojybDaQDAuTEVpHVMi5xQX74cJPyMJbaSaRM

>> The port is expected to work with modified versions of RTEMS and
>> newlib.  I have a git repo with changes to those projects, that I can
>> make available if there is interest.

>> Here is the patch series:
>> https://reviews.llvm.org/D46451 [asan] Add instrumentation support for
>> Myriad
>> https://reviews.llvm.org/D46452 [sanitizer] Don't add --export-dynamic
for
>> Myriad
>> https://reviews.llvm.org/D46453 [sanitizer] Add definitions for Myriad
>> RTEMS platform
>> https://reviews.llvm.org/D46454 [sanitizer] Trivial portion of the port
to
>> Myriad RTEMS
>> https://reviews.llvm.org/D46456 [asan] Add support for Myriad RTEMS
memory
>> map
>> https://reviews.llvm.org/D46457 [asan] Add a magic shadow value for shadw
>> gap
>> https://reviews.llvm.org/D46459 [asan] On RTEMS, checks for asan_inited
>> before entering ASan run-time
>> https://reviews.llvm.org/D46461 [asan] Set flags appropriately for RTEMS
>> https://reviews.llvm.org/D46462 [sanitizer] Allow Fuchsia symbolizer to
be
>> reused by Myriad RTEMS
>> https://reviews.llvm.org/D46463 [sanitizer] On RTEMS, avoid recursion
when
>> reporting Mmap failure
>> https://reviews.llvm.org/D46465 [asan] Port asan_malloc_linux.cc to RTEMS
>> https://reviews.llvm.org/D46466 [asan] Add AsanThread::Restart() to
support
>> thread restart
>> https://reviews.llvm.org/D46467 [asan] Add argument to allow fake stack
to
>> be initialized during thread init
>> https://reviews.llvm.org/D46468 [asan] Add target-specific files for
Myriad
>> RTEMS port
>> https://reviews.llvm.org/D46469 [sanitizer] Port fast stack unwinder to
>> sparcv8

>> Thanks,

>> Walter
>> _______________________________________________
>> LLVM Developers mailing list
>> [hidden email]
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev


On Fri, May 4, 2018 at 7:09 PM Walter Lee <[hidden email]> wrote:
On Fri, May 4, 2018 at 6:21 PM Kostya Serebryany <[hidden email]> wrote:

> On RAM...

> You chose the 32-byte shadow granularity to reduce the RAM overhead,
> but I am afraid this will actually increase it due to extra alignment
requirements,
> especially if an average allocation on your typical application is small.


Good point.  I will run our test suite with 8-byte shadow granularity and
do some measurements.

try 16 as well. 
Depending on your typical code and default alignment, I would expect 8 or 16 to be the best. 
 

In general, the memory constraint hasn't been as big an issue as I
initially thought.  The mitigating factor is that we are focused on unit
tests as opposed to running the full blown application.


> The pointers are 32-bit, right?
> Given how RAM-constrained your environment is, maybe you should consider
something more like HWASAN instead of ASAN.
> https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html
> But you may not have enough address bits. :(

Pointers are 32 bits.  Interesting idea, but I agree I don't think we have
enough bits.  Even ignoring the sparsely used lower addresses, DDR + cache
bit leaves us with only 2 bits.

Thanks,

Walter

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev
On Mon, May 7, 2018 at 2:05 PM Kostya Serebryany <[hidden email]> wrote:
 
and GetFakeStackFast. 

Not sure. Why don't just disable stack-use-after-return? 

Yeah originally I was going to do that, but:

1. We had a stack use-after-return last month that people had to debug by hand. that would have been caught by ASan with stack-use-after-return.  So it seems like a worthwhile feature to support.

2. Initially I was concerned about code size and one thing I was planning to do was to disable stack-use-after-return instrumentation.  But in practice, we find that this was a non-issue: we are able to run the same number of tests with and without such instrumentation.

3. It seems like a good idea to keep the same default behavior and feature parity with x86 ASan when possible.

Thanks,

Walter


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev


On Mon, May 7, 2018 at 12:35 PM Walter Lee <[hidden email]> wrote:
On Mon, May 7, 2018 at 2:05 PM Kostya Serebryany <[hidden email]> wrote:
 
and GetFakeStackFast. 

Not sure. Why don't just disable stack-use-after-return? 

Yeah originally I was going to do that, but:

I probably misunderstood.  You are not disabling UAR completely, just disabling the fast-path getter? 
Hmm. 
Maybe let's leave this one out for now and once everything else is resolved will look at it again? 


1. We had a stack use-after-return last month that people had to debug by hand. that would have been caught by ASan with stack-use-after-return.  So it seems like a worthwhile feature to support.

2. Initially I was concerned about code size and one thing I was planning to do was to disable stack-use-after-return instrumentation.  But in practice, we find that this was a non-issue: we are able to run the same number of tests with and without such instrumentation.

3. It seems like a good idea to keep the same default behavior and feature parity with x86 ASan when possible.

Thanks,

Walter


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
On Mon, May 7, 2018 at 3:42 PM Kostya Serebryany <[hidden email]> wrote:

> I probably misunderstood.  You are not disabling UAR completely, just
disabling the fast-path getter?

I may not be putting the guard in the right place, but the problem is when
!asan_inited, AsanThreads has not been set up => GetCurrentThread won't
work.  So in that scenario, I want GetFakeStack and GetFakeStackFast to
just return nullptr, and OnMalloc to return 0.

> Hmm.
> Maybe let's leave this one out for now and once everything else is
resolved will look at it again?

Sounds good.
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
In reply to this post by Chris Lattner via llvm-dev
I ran our test suite with grain of 8 and 32, and more tests were able to
avoid running out of memory with grain of 32 than 8.  The ones that run out
of memory on 8 but not 32 all failed trying to allocate a large region from
the heap (350M).  I haven't had any tests that run out of memory for other
reasons.

Given that, I will check in the current selected granularity of 32.  I will
try grain 16 and to collect finer grained data when I have a chance.

Thanks,

Walter
On Mon, May 7, 2018 at 2:07 PM Kostya Serebryany <[hidden email]> wrote:



> On Fri, May 4, 2018 at 7:09 PM Walter Lee <[hidden email]> wrote:

>> On Fri, May 4, 2018 at 6:21 PM Kostya Serebryany <[hidden email]> wrote:

>> > On RAM...

>> > You chose the 32-byte shadow granularity to reduce the RAM overhead,
>> > but I am afraid this will actually increase it due to extra alignment
>> requirements,
>> > especially if an average allocation on your typical application is
small.


>> Good point.  I will run our test suite with 8-byte shadow granularity and
>> do some measurements.


> try 16 as well.
> Depending on your typical code and default alignment, I would expect 8 or
16 to be the best.



>> In general, the memory constraint hasn't been as big an issue as I
>> initially thought.  The mitigating factor is that we are focused on unit
>> tests as opposed to running the full blown application.


>> > The pointers are 32-bit, right?
>> > Given how RAM-constrained your environment is, maybe you should
consider
>> something more like HWASAN instead of ASAN.
>> > https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html
>> > But you may not have enough address bits. :(

>> Pointers are 32 bits.  Interesting idea, but I agree I don't think we
have
>> enough bits.  Even ignoring the sparsely used lower addresses, DDR +
cache
>> bit leaves us with only 2 bits.

>> Thanks,

>> Walter
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] ASan port for Myriad RTEMS

Chris Lattner via llvm-dev
I've finished committing the patch series.  Thanks a lot to Kostya, Vitaly,
Aleksey, and Roland for doing the bulk of the reviews!

Thanks,

Walter
On Fri, May 18, 2018 at 12:06 AM Walter Lee <[hidden email]> wrote:

> I ran our test suite with grain of 8 and 32, and more tests were able to
> avoid running out of memory with grain of 32 than 8.  The ones that run
out
> of memory on 8 but not 32 all failed trying to allocate a large region
from
> the heap (350M).  I haven't had any tests that run out of memory for other
> reasons.

> Given that, I will check in the current selected granularity of 32.  I
will
> try grain 16 and to collect finer grained data when I have a chance.

> Thanks,

> Walter
> On Mon, May 7, 2018 at 2:07 PM Kostya Serebryany <[hidden email]> wrote:



> > On Fri, May 4, 2018 at 7:09 PM Walter Lee <[hidden email]> wrote:

> >> On Fri, May 4, 2018 at 6:21 PM Kostya Serebryany <[hidden email]>
wrote:

> >> > On RAM...

> >> > You chose the 32-byte shadow granularity to reduce the RAM overhead,
> >> > but I am afraid this will actually increase it due to extra alignment
> >> requirements,
> >> > especially if an average allocation on your typical application is
> small.


> >> Good point.  I will run our test suite with 8-byte shadow granularity
and
> >> do some measurements.


> > try 16 as well.
> > Depending on your typical code and default alignment, I would expect 8
or
> 16 to be the best.



> >> In general, the memory constraint hasn't been as big an issue as I
> >> initially thought.  The mitigating factor is that we are focused on
unit
> >> tests as opposed to running the full blown application.


> >> > The pointers are 32-bit, right?
> >> > Given how RAM-constrained your environment is, maybe you should
> consider
> >> something more like HWASAN instead of ASAN.
> >> >
https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html
> >> > But you may not have enough address bits. :(

> >> Pointers are 32 bits.  Interesting idea, but I agree I don't think we
> have
> >> enough bits.  Even ignoring the sparsely used lower addresses, DDR +
> cache
> >> bit leaves us with only 2 bits.

> >> Thanks,

> >> Walter
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev