[llvm-dev] Test Error Paths for Expected & ErrorOr

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev

Yes definitely, testing a small piece of code like the GlobPattern::create() example, it would mostly indicate missing unit tests or insufficient test data.

In contrast to unit tests, however, it can also verify correct handling of errors passed between function call hierarchies in more complex scenarios.
For this I should point to the other example in the code, where it's applied to llvm::object::createBinary():
https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13

Here it detects and runs 44 different control paths, that can hardly be covered by a unit test altogether, because they don't depend on the input to creatBinary() but rather on the environment the test runs in.

Am 27.07.17 um 16:46 schrieb David Blaikie:
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev


On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <[hidden email]> wrote:

Yes definitely, testing a small piece of code like the GlobPattern::create() example, it would mostly indicate missing unit tests or insufficient test data.

In contrast to unit tests, however, it can also verify correct handling of errors passed between function call hierarchies in more complex scenarios.
For this I should point to the other example in the code, where it's applied to llvm::object::createBinary():
https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13

Here it detects and runs 44 different control paths, that can hardly be covered by a unit test altogether, because they don't depend on the input to creatBinary() but rather on the environment the test runs in.

 Yep, testing OS level environmental failures would be great for this - I wonder if there's a good way to distinguish between them (so that this only hits those cases, but doesn't unduly 'cover' other cases that should be targeted by tests, etc). Essentially something more opt-in or some other handshake. (perhaps a certain kind of Error that represents a "this failure is due to the environment, not the caller's arguments"? Not sure)

Hopefully Lang (author of Error/Expected) chimes in - be curious to hear his thoughts on this stuff too.

Thanks again for developing it/bringing it up here! :)

Am 27.07.17 um 16:46 schrieb David Blaikie:
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev
Hi Stefan, David,

This is very interesting stuff - it adds a dimension of error security that Error/Expected can't provide on their own. I think it would be interesting to try to build a tool around this.

Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch? My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.

In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 

Cheers,
Lang.


On Thu, Jul 27, 2017 at 8:56 AM, David Blaikie <[hidden email]> wrote:


On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <[hidden email]> wrote:

Yes definitely, testing a small piece of code like the GlobPattern::create() example, it would mostly indicate missing unit tests or insufficient test data.

In contrast to unit tests, however, it can also verify correct handling of errors passed between function call hierarchies in more complex scenarios.
For this I should point to the other example in the code, where it's applied to llvm::object::createBinary():
https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13

Here it detects and runs 44 different control paths, that can hardly be covered by a unit test altogether, because they don't depend on the input to creatBinary() but rather on the environment the test runs in.

 Yep, testing OS level environmental failures would be great for this - I wonder if there's a good way to distinguish between them (so that this only hits those cases, but doesn't unduly 'cover' other cases that should be targeted by tests, etc). Essentially something more opt-in or some other handshake. (perhaps a certain kind of Error that represents a "this failure is due to the environment, not the caller's arguments"? Not sure)

Hopefully Lang (author of Error/Expected) chimes in - be curious to hear his thoughts on this stuff too.

Thanks again for developing it/bringing it up here! :)

Am 27.07.17 um 16:46 schrieb David Blaikie:
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev


On Fri, Jul 28, 2017 at 2:36 PM Lang Hames <[hidden email]> wrote:
Hi Stefan, David,

This is very interesting stuff - it adds a dimension of error security that Error/Expected can't provide on their own. I think it would be interesting to try to build a tool around this.

Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch? My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.

In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 

Not sure I quite follow this last bit - could you explain by way of an example, perhaps?
 

Cheers,
Lang.


On Thu, Jul 27, 2017 at 8:56 AM, David Blaikie <[hidden email]> wrote:


On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <[hidden email]> wrote:

Yes definitely, testing a small piece of code like the GlobPattern::create() example, it would mostly indicate missing unit tests or insufficient test data.

In contrast to unit tests, however, it can also verify correct handling of errors passed between function call hierarchies in more complex scenarios.
For this I should point to the other example in the code, where it's applied to llvm::object::createBinary():
https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13

Here it detects and runs 44 different control paths, that can hardly be covered by a unit test altogether, because they don't depend on the input to creatBinary() but rather on the environment the test runs in.

 Yep, testing OS level environmental failures would be great for this - I wonder if there's a good way to distinguish between them (so that this only hits those cases, but doesn't unduly 'cover' other cases that should be targeted by tests, etc). Essentially something more opt-in or some other handshake. (perhaps a certain kind of Error that represents a "this failure is due to the environment, not the caller's arguments"? Not sure)

Hopefully Lang (author of Error/Expected) chimes in - be curious to hear his thoughts on this stuff too.

Thanks again for developing it/bringing it up here! :)

Am 27.07.17 um 16:46 schrieb David Blaikie:
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev
In reply to this post by George Karpenkov via llvm-dev

Hi Lang, hi David, thanks for looking into this.

Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch?
In my own code yes, not in LLVM ;) I'd like to run it on a large example, some llvm tool or clang cc1 maybe. I hope to find the time end of this week.

My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.
Very good idea!

In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 
Right, currently I only inject a generic error mock. Handlers may not be prepared for it and testing them is not possible so far. Not sure how a detection for the actual error types could look like, but I am curious for ideas.

I will get back to you with results of a bigger test run asap.

Am 28.07.17 um 23:36 schrieb Lang Hames:

Hi Stefan, David,

This is very interesting stuff - it adds a dimension of error security that Error/Expected can't provide on their own. I think it would be interesting to try to build a tool around this.

Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch? My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.

In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 

Cheers,
Lang.


On Thu, Jul 27, 2017 at 8:56 AM, David Blaikie <[hidden email]> wrote:


On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <[hidden email]> wrote:

Yes definitely, testing a small piece of code like the GlobPattern::create() example, it would mostly indicate missing unit tests or insufficient test data.

In contrast to unit tests, however, it can also verify correct handling of errors passed between function call hierarchies in more complex scenarios.
For this I should point to the other example in the code, where it's applied to llvm::object::createBinary():
https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13

Here it detects and runs 44 different control paths, that can hardly be covered by a unit test altogether, because they don't depend on the input to creatBinary() but rather on the environment the test runs in.

 Yep, testing OS level environmental failures would be great for this - I wonder if there's a good way to distinguish between them (so that this only hits those cases, but doesn't unduly 'cover' other cases that should be targeted by tests, etc). Essentially something more opt-in or some other handshake. (perhaps a certain kind of Error that represents a "this failure is due to the environment, not the caller's arguments"? Not sure)

Hopefully Lang (author of Error/Expected) chimes in - be curious to hear his thoughts on this stuff too.

Thanks again for developing it/bringing it up here! :)

Am 27.07.17 um 16:46 schrieb David Blaikie:
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev


On Mon, Jul 31, 2017 at 8:19 AM Stefan Gränitz <[hidden email]> wrote:

Hi Lang, hi David, thanks for looking into this.



Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch?

In my own code yes, not in LLVM ;) I'd like to run it on a large example, some llvm tool or clang cc1 maybe. I hope to find the time end of this week.



My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.

Very good idea!



In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 

Right, currently I only inject a generic error mock. Handlers may not be prepared for it and testing them is not possible so far. Not sure how a detection for the actual error types could look like, but I am curious for ideas.


Yeah, I imagine that would be tricky - 'true' mutation testing (a compiler that deliberately and systematically miscompiles branches (in a similar way to your approach of systematically producing errors) & helps discover untested parts of the code (any mutation that doesn't result in a test failure is missing testing)) would probably find these, or maybe static analysis.

Alternatively, if this technique were really embedded deep into llvm::Error, then it could differentiate between the various handles in a handleError - except I suppose it'd have no way of creating arbitrary errors required to pass to them - maybe with some API entry point (give for any T that is an ErrorInfo, have ErrorInfo::createTestStub or the like that could be used). It'd be tricky, I'd imagine.

- Dave
 


I will get back to you with results of a bigger test run asap.

Am 28.07.17 um 23:36 schrieb Lang Hames:

Hi Stefan, David,

This is very interesting stuff - it adds a dimension of error security that Error/Expected can't provide on their own. I think it would be interesting to try to build a tool around this.

Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch? My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.

In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 

Cheers,
Lang.


On Thu, Jul 27, 2017 at 8:56 AM, David Blaikie <[hidden email]> wrote:


On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <[hidden email]> wrote:

Yes definitely, testing a small piece of code like the GlobPattern::create() example, it would mostly indicate missing unit tests or insufficient test data.

In contrast to unit tests, however, it can also verify correct handling of errors passed between function call hierarchies in more complex scenarios.
For this I should point to the other example in the code, where it's applied to llvm::object::createBinary():
https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13

Here it detects and runs 44 different control paths, that can hardly be covered by a unit test altogether, because they don't depend on the input to creatBinary() but rather on the environment the test runs in.

 Yep, testing OS level environmental failures would be great for this - I wonder if there's a good way to distinguish between them (so that this only hits those cases, but doesn't unduly 'cover' other cases that should be targeted by tests, etc). Essentially something more opt-in or some other handshake. (perhaps a certain kind of Error that represents a "this failure is due to the environment, not the caller's arguments"? Not sure)

Hopefully Lang (author of Error/Expected) chimes in - be curious to hear his thoughts on this stuff too.

Thanks again for developing it/bringing it up here! :)

Am 27.07.17 um 16:46 schrieb David Blaikie:
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [llvm-dev] Test Error Paths for Expected & ErrorOr

George Karpenkov via llvm-dev
I am back with some news:

Tried Clang cc1, but not much use of Expected/ErrorOr.
Went on instrumenting the llvm-ar tool and ran a few tests: "llvm-ar t /usr/local/lib/libc++.a" has 267 mutation points, many of them duplicates.
Quite a number of error handlers do exit() - I gave up gtest.
Added command line option -force-error=<int> + automate with shell script.
Built and ran with ASAN&UBSAN - no issues found. Disappointing, expected at least a some minor leak! ;)
Added -debug -debug-only=ForceAllErrors for extra debug dump, like short stack trace where instance is broken.

Fork is on GitHub:
https://github.com/weliveindetail/llvm-ForceAllErrors
https://github.com/weliveindetail/llvm-ForceAllErrors/commits/ForceAllErrors

Am 31.07.17 um 17:53 schrieb David Blaikie:
On Mon, Jul 31, 2017 at 8:19 AM Stefan Gränitz <[hidden email]> wrote:

Hi Lang, hi David, thanks for looking into this.

Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch?

In my own code yes, not in LLVM ;) I'd like to run it on a large example, some llvm tool or clang cc1 maybe. I hope to find the time end of this week.

My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.

Very good idea!

In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 

Right, currently I only inject a generic error mock. Handlers may not be prepared for it and testing them is not possible so far. Not sure how a detection for the actual error types could look like, but I am curious for ideas.


Yeah, I imagine that would be tricky - 'true' mutation testing (a compiler that deliberately and systematically miscompiles branches (in a similar way to your approach of systematically producing errors) & helps discover untested parts of the code (any mutation that doesn't result in a test failure is missing testing)) would probably find these, or maybe static analysis.

Alternatively, if this technique were really embedded deep into llvm::Error, then it could differentiate between the various handles in a handleError - except I suppose it'd have no way of creating arbitrary errors required to pass to them - maybe with some API entry point (give for any T that is an ErrorInfo, have ErrorInfo::createTestStub or the like that could be used). It'd be tricky, I'd imagine.

- Dave
 


I will get back to you with results of a bigger test run asap.

Am 28.07.17 um 23:36 schrieb Lang Hames:

Hi Stefan, David,

This is very interesting stuff - it adds a dimension of error security that Error/Expected can't provide on their own. I think it would be interesting to try to build a tool around this.

Did you identify many cases where "real work" (in your example, the nullptr dereference" was being done in an error branch? My suspicion is that that should be rare, but that your tool would be great for exposing logic errors and resource leaks if run with the sanitizers turned on.

In an ideal world we'd go even further and build a clang/LLDB based tool that can identify what kinds of errors a function can produce, then inject instances of those: That would allow us to test actual error handling logic too, not just the generic surrounding logic. 

Cheers,
Lang.


On Thu, Jul 27, 2017 at 8:56 AM, David Blaikie <[hidden email]> wrote:


On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <[hidden email]> wrote:

Yes definitely, testing a small piece of code like the GlobPattern::create() example, it would mostly indicate missing unit tests or insufficient test data.

In contrast to unit tests, however, it can also verify correct handling of errors passed between function call hierarchies in more complex scenarios.
For this I should point to the other example in the code, where it's applied to llvm::object::createBinary():
https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13

Here it detects and runs 44 different control paths, that can hardly be covered by a unit test altogether, because they don't depend on the input to creatBinary() but rather on the environment the test runs in.

 Yep, testing OS level environmental failures would be great for this - I wonder if there's a good way to distinguish between them (so that this only hits those cases, but doesn't unduly 'cover' other cases that should be targeted by tests, etc). Essentially something more opt-in or some other handshake. (perhaps a certain kind of Error that represents a "this failure is due to the environment, not the caller's arguments"? Not sure)

Hopefully Lang (author of Error/Expected) chimes in - be curious to hear his thoughts on this stuff too.

Thanks again for developing it/bringing it up here! :)

Am 27.07.17 um 16:46 schrieb David Blaikie:
I /kind/ of like the idea - but it almost feels like this would be a tool for finding out that test coverage is insufficient, then adding tests that actually exercise the bad input, etc (this should be equally discoverable by code coverage, probably? Maybe not if multiple error paths all collapse together, maybe... )

For instance, with your example, especially once there's an identified bug that helps motivate, would it not be better to add a test that does pass a fileName input that fails GlobPattern::create?

On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <[hidden email]> wrote:
Hello, this is a call for feedback: opinions, improvements, testers..

I use the support classes Expected<T> and ErrorOr<T> quite often
recently and I like the concept a lot! Thanks Lang btw!
However, from time to time I found issues in the execution paths of my
error cases and got annoyed by their naturally low test coverage.

So I started sketching a test that runs all error paths for a given
piece of code to detect these issues. I just pushed it to GitHub and
added a little readme:
https://github.com/weliveindetail/ForceAllErrors-in-LLVM

Are there people on the list facing the same issue?
How do you test your error paths?
Could this be of use for you if it was in a reusable state?
Is there something similar already around?
Anyone seeing bugs or improvements?
Could it maybe even increase coverage in the LLVM test suite some day?

Thanks for all kinds of feedback!
Cheers, Stefan

--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...


-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@...

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Loading...