[EXPERIMENTAL] Generated unit tests

[JanKrivanek]

Context

Generated with very early preview of unit tests generation tool. Trying to evaluate if there is any hope of that being useful and what would need to be worked on to achieve usefulness.

Technical data

Baseline (main):

Test summary: total: 1985, failed: 0, succeeded: 1969, skipped: 16, duration: 9.4s

Target (generated):

Test summary: total: 1038, failed: 6, succeeded: 1022, skipped: 10, duration: 10.6s 

(The after seem to have lest test as I removed net472 targetting. otherwise 198 tests were generated)

AssemblyName	LineRateBefore	LineRateAfter	LineRateDiff	BranchRateBefore	BranchRateAfter	BranchRateDiff
Nerdbank.Streams	71.10 %	69.18 %	-1.92 %	69.36 %	61.43 %	-7.93 %

More comments

• Still just heuristical version (so might not follow convention and references of the repo)
• Very few and dummy code fixers - so we fallback a lot to commenting out tests with compilation diagnostic
• No runtime feedback yet - so tests that crash/fail during runtime are kept

[AArnott]

Interesting project. Once you get the code coverage numbers up and the tests passing, how will you measure effective asserts that guard against regressions?

Also, are the tests that you generated produced with the existing tests in mind or was the generator blind to the existing tests?

[JanKrivanek]

Good questions

• We try to skip what already seems to be test (but that's currently just based off of naming, no converage analysis)
• There were thoughts to employ parameter based testing for better coverage and mutation based testing for verifying the ability to catch regressions
• But before being able to consider it more seriously - so far the main concern is the quality of the tests - it doesn't feel that high. But your PoV on that (based on this PR) - specifically what could have been better, what feels as wrong or useless etc. would be helpful in prioritizing the improvements :-)

[AArnott]

This Nil type, and some other types these new files are testing, are not actually part of Nerdbank.Streams' public API. We link it in from a git submodule and it compiles as internal. This test ought to not even compile. And in this case our policy is to assume the code in the submodule is already well tested in its own repo, such that we focus our testing on the code that is novel here.

[AArnott]

This is peculiar. I don't usually care for Thread.Sleep in tests because it makes tests run slowly.
In this case, this private class isn't even referenced by anything. So why was it generated?