[llvm-dev] Wrapping lit commands for device testing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[llvm-dev] Wrapping lit commands for device testing

Adam Nemet via llvm-dev
Hi,

Tests in compiler-rt run under both iOS simulators and Android devices, which requires “wrapping” the run line with the “adb” command for Android, and adding a prefix / setting some environment variables for simulators/
In case of Android this is done by changing the compilation command to produce a wrapper which already calls “adb”, and in case of iOS simulators by explicitly adding the “%run” command in front of run lines.
The solution is far from perfect though: many tests (e.g. libFuzzer) extensively rely on using arbitrary shell commands to create and manipulate temporary text files.
Performing those commands in a different environment is done in a very ad-hoc manner: from what I see, for sanitizer tests those temporary files are then usually references in the environment variables present in the run command, which are then parsed to carry those files into the executable environment.
Doing this for libFuzzer would be very difficult and would require a lot of modifications to test.

I would prefer a more transparent approach: if LIT configuration was to provide a hook to perform arbitrary replacements/wrapping on each of the executed lines, the wrapping could be performed without modifying tests at all.

My implementation preference would be adding a method  `def wrap(self, s)` (str -> str) to the format class (lit/formats/*), and then using it (if non-empty) on each line being run.
I think that is preferable to a method on a `config` object, since line transformation is a very heavyweight operation, akin to other methods available on the `format` class (test discovery semantics and test execution semantics).
Would there be any objections against such an implementation plan?

Regards,
George
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [cfe-dev] Wrapping lit commands for device testing

Adam Nemet via llvm-dev
Hello George,

I would be very much interested in a (generic) solution that would allow
running tests on a remote host/device. My intended use case for this is
lldb+android, and I am mostly interested in googletest-style tests. In
fact, I proposed something vaguely similar to this (but limited to gtest)
last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
However, due to some internal reshuffles, this idea never got past the
prototype stage. Nonetheless, I would still like do something like this
eventually (even better if someone else does it for me :) ).

I would love to hear more about your proposal, as my head is currently
filled with questions about how this would work. (I admit I don't know
anything about how libFuzzer is tested.) It's not clear to me whether you
intend to implement a general purpose remote test execution framework, or a
solution tailored to the needs of libFuzzer. Do you automatically handle
copying the inputs to the remote target? How do you identify what the
inputs are? Do you copy over any llvm tools executed in the run lines? What
about their dependent libraries? etc.

Maybe if you could give an example of a typical test you intend to support
and how it would work, a lot of these questions would be answered. If this
turns out to be something that could be reused for our use case, I would be
willing to chip in to help make it happen.

regards,
pavel

On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
[hidden email]> wrote:

> Hi,

> Tests in compiler-rt run under both iOS simulators and Android devices,
which requires “wrapping” the run line with the “adb” command for Android,
and adding a prefix / setting some environment variables for simulators/
> In case of Android this is done by changing the compilation command to
produce a wrapper which already calls “adb”, and in case of iOS simulators
by explicitly adding the “%run” command in front of run lines.
> The solution is far from perfect though: many tests (e.g. libFuzzer)
extensively rely on using arbitrary shell commands to create and manipulate
temporary text files.
> Performing those commands in a different environment is done in a very
ad-hoc manner: from what I see, for sanitizer tests those temporary files
are then usually references in the environment variables present in the run
command, which are then parsed to carry those files into the executable
environment.
> Doing this for libFuzzer would be very difficult and would require a lot
of modifications to test.

> I would prefer a more transparent approach: if LIT configuration was to
provide a hook to perform arbitrary replacements/wrapping on each of the
executed lines, the wrapping could be performed without modifying tests at
all.

> My implementation preference would be adding a method  `def wrap(self,
s)` (str -> str) to the format class (lit/formats/*), and then using it (if
non-empty) on each line being run.
> I think that is preferable to a method on a `config` object, since line
transformation is a very heavyweight operation, akin to other methods
available on the `format` class (test discovery semantics and test
execution semantics).
> Would there be any objections against such an implementation plan?

> Regards,
> George
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [cfe-dev] Wrapping lit commands for device testing

Adam Nemet via llvm-dev
Hi Pavel,

I was thinking about a generic proposal for a while,
but then I’ve realized that a much simpler heuristic suffices for all of my needs:

 - Prepend all commands which should execute on the device with %run, which runs a wrapper python script
 - Inside that script use a simple regexp to find all paths
 - sync all paths to the device
 - run the executable on the device
 - sync all paths form the device back to host

> On Apr 23, 2018, at 8:09 AM, Pavel Labath <[hidden email]> wrote:
>
> Hello George,
>
> I would be very much interested in a (generic) solution that would allow
> running tests on a remote host/device. My intended use case for this is
> lldb+android, and I am mostly interested in googletest-style tests.

I’m not sure about those, all my problems were in LIT tests.

> In
> fact, I proposed something vaguely similar to this (but limited to gtest)
> last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
> However, due to some internal reshuffles, this idea never got past the
> prototype stage. Nonetheless, I would still like do something like this
> eventually (even better if someone else does it for me :) ).
>
> I would love to hear more about your proposal, as my head is currently
> filled with questions about how this would work. (I admit I don't know
> anything about how libFuzzer is tested.) It's not clear to me whether you
> intend to implement a general purpose remote test execution framework, or a
> solution tailored to the needs of libFuzzer.

Seemed generic enough for most of compiler-rt.

> Do you automatically handle
> copying the inputs to the remote target? How do you identify what the
> inputs are?

Yes, the simple heuristic above seems to work here.

> Do you copy over any llvm tools executed in the run lines?

I copy the command directly after %run wrapper, but not FileCheck/etc.
%run syncs it over to the device.

> What
> about their dependent libraries? etc.

Dynamically linked? Mostly we don’t have any, and for those tests where we do,
some rpath substitution magic is required.


> Maybe if you could give an example of a typical test you intend to support
> and how it would work, a lot of these questions would be answered. If this
> turns out to be something that could be reused for our use case, I would be
> willing to chip in to help make it happen.

Basically, any of libFuzzer tests with %run prepended, e.g. https://reviews.llvm.org/D45235

George

>
> regards,
> pavel
>
> On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
> [hidden email]> wrote:
>
>> Hi,
>
>> Tests in compiler-rt run under both iOS simulators and Android devices,
> which requires “wrapping” the run line with the “adb” command for Android,
> and adding a prefix / setting some environment variables for simulators/
>> In case of Android this is done by changing the compilation command to
> produce a wrapper which already calls “adb”, and in case of iOS simulators
> by explicitly adding the “%run” command in front of run lines.
>> The solution is far from perfect though: many tests (e.g. libFuzzer)
> extensively rely on using arbitrary shell commands to create and manipulate
> temporary text files.
>> Performing those commands in a different environment is done in a very
> ad-hoc manner: from what I see, for sanitizer tests those temporary files
> are then usually references in the environment variables present in the run
> command, which are then parsed to carry those files into the executable
> environment.
>> Doing this for libFuzzer would be very difficult and would require a lot
> of modifications to test.
>
>> I would prefer a more transparent approach: if LIT configuration was to
> provide a hook to perform arbitrary replacements/wrapping on each of the
> executed lines, the wrapping could be performed without modifying tests at
> all.
>
>> My implementation preference would be adding a method  `def wrap(self,
> s)` (str -> str) to the format class (lit/formats/*), and then using it (if
> non-empty) on each line being run.
>> I think that is preferable to a method on a `config` object, since line
> transformation is a very heavyweight operation, akin to other methods
> available on the `format` class (test discovery semantics and test
> execution semantics).
>> Would there be any objections against such an implementation plan?
>
>> Regards,
>> George
>> _______________________________________________
>> cfe-dev mailing list
>> [hidden email]
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [cfe-dev] Wrapping lit commands for device testing

Adam Nemet via llvm-dev
Thank you for the explanation George. I have too came to believe that a
general all-encompassing solution would be incredibly complicated. Our
(lldb) use case is in many ways similar to compiler-rt (we want to test
code which was not compiled for the host), but then it also has it's own
"thing" (we have *two* processes/binaries, one debugging another,
theoretically they don't even have to have the same bitness).

I am glad you were able to make your use case work. I'll probably try
creating something a bit custom for lldb, when I actually get around to it.

regards,
pavel
On Sat, 5 May 2018 at 03:38, George Karpenkov <[hidden email]> wrote:

> Hi Pavel,

> I was thinking about a generic proposal for a while,
> but then I’ve realized that a much simpler heuristic suffices for all of
my needs:

>    - Prepend all commands which should execute on the device with %run,
which runs a wrapper python script
>    - Inside that script use a simple regexp to find all paths
>    - sync all paths to the device
>    - run the executable on the device
>    - sync all paths form the device back to host

> > On Apr 23, 2018, at 8:09 AM, Pavel Labath <[hidden email]> wrote:
> >
> > Hello George,
> >
> > I would be very much interested in a (generic) solution that would allow
> > running tests on a remote host/device. My intended use case for this is
> > lldb+android, and I am mostly interested in googletest-style tests.

> I’m not sure about those, all my problems were in LIT tests.

> > In
> > fact, I proposed something vaguely similar to this (but limited to
gtest)
> > last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html
> .
> > However, due to some internal reshuffles, this idea never got past the
> > prototype stage. Nonetheless, I would still like do something like this
> > eventually (even better if someone else does it for me :) ).
> >
> > I would love to hear more about your proposal, as my head is currently
> > filled with questions about how this would work. (I admit I don't know
> > anything about how libFuzzer is tested.) It's not clear to me whether
you
> > intend to implement a general purpose remote test execution framework,
or a
> > solution tailored to the needs of libFuzzer.

> Seemed generic enough for most of compiler-rt.

> > Do you automatically handle
> > copying the inputs to the remote target? How do you identify what the
> > inputs are?

> Yes, the simple heuristic above seems to work here.

> > Do you copy over any llvm tools executed in the run lines?

> I copy the command directly after %run wrapper, but not FileCheck/etc.
> %run syncs it over to the device.

> > What
> > about their dependent libraries? etc.

> Dynamically linked? Mostly we don’t have any, and for those tests where
we do,
> some rpath substitution magic is required.


> > Maybe if you could give an example of a typical test you intend to
support
> > and how it would work, a lot of these questions would be answered. If
this
> > turns out to be something that could be reused for our use case, I
would be
> > willing to chip in to help make it happen.

> Basically, any of libFuzzer tests with %run prepended, e.g.
https://reviews.llvm.org/D45235

> George

> >
> > regards,
> > pavel
> >
> > On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
> > [hidden email]> wrote:
> >
> >> Hi,
> >
> >> Tests in compiler-rt run under both iOS simulators and Android devices,
> > which requires “wrapping” the run line with the “adb” command for
Android,
> > and adding a prefix / setting some environment variables for simulators/
> >> In case of Android this is done by changing the compilation command to
> > produce a wrapper which already calls “adb”, and in case of iOS
simulators
> > by explicitly adding the “%run” command in front of run lines.
> >> The solution is far from perfect though: many tests (e.g. libFuzzer)
> > extensively rely on using arbitrary shell commands to create and
manipulate
> > temporary text files.
> >> Performing those commands in a different environment is done in a very
> > ad-hoc manner: from what I see, for sanitizer tests those temporary
files
> > are then usually references in the environment variables present in the
run
> > command, which are then parsed to carry those files into the executable
> > environment.
> >> Doing this for libFuzzer would be very difficult and would require a
lot
> > of modifications to test.
> >
> >> I would prefer a more transparent approach: if LIT configuration was to
> > provide a hook to perform arbitrary replacements/wrapping on each of the
> > executed lines, the wrapping could be performed without modifying tests
at
> > all.
> >
> >> My implementation preference would be adding a method  `def wrap(self,
> > s)` (str -> str) to the format class (lit/formats/*), and then using it
(if

> > non-empty) on each line being run.
> >> I think that is preferable to a method on a `config` object, since line
> > transformation is a very heavyweight operation, akin to other methods
> > available on the `format` class (test discovery semantics and test
> > execution semantics).
> >> Would there be any objections against such an implementation plan?
> >
> >> Regards,
> >> George
> >> _______________________________________________
> >> cfe-dev mailing list
> >> [hidden email]
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev