Confusing log message when all tests are filtered out

[genlu]

Description

I noticed a confusing warning in the LUT log after restart LUT (see below). Essentially, after filtering out tests, none left to execute, which is expected because there's no code change since LUT was stopped. But this message makes it sounds like an error. @satya Madala Could you guys change this message at least for filter scenario (if there’s tests discovered but all filtered out).

[15:02:09.546 Info] [TestRunner 1] No test is available in C:\Users\gel\source\repos\XUnitTestProject1\.vs\NUnitCoreTests\lut\0\t\NUnitCoreTests\Debug\netcoreapp2.0\NUnitCoreTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Steps to reproduce

With latest dogfood build:
Open a test project -> Start LUT -> Stop LUT -> Start LUT again

[smadala]

@genlu Is below message looks good?

PS C:\Users\samadala\src\vstest> .\artifacts\Debug\net451\win7-x64\vstest.console.exe .\test\TestAssets\SimpleTestProject\bin\Debug\net451\SimpleTestProject.dll /testcasefilter:"Name=abc"
Microsoft (R) Test Execution Command Line Tool Version 15.8.0-dev
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
No test available for testcase filter `Name=abc` in C:\Users\samadala\src\vstest\.\test\TestAssets\SimpleTestProject\bin\Debug\net451\SimpleTestProject.dll

[smadala]

What is the timeline your looking for this issue to get fixed?

[genlu]

The message above is fine. But for LUT, the filter can be very long (e.g. all tests in project). So can we not include the filter string itself in the message?

In terms of timeline, 15.8 preview 4 would be great. @ManishJayaswal

[ManishJayaswal]

"No test matches the given testcase filter" may be better.

For LUT, in most cases, this would not be an error condition. It would just point to the fact that no tests are covering the line which was edited. Is there any way we can make this message "informational" so that it only shows up with verbose logging? Else this will keep showing as error when technically there is no error- the program is working as expected. If it only shows up in verbose logging then appending the filter text to the message may help with diagnostics so it may be ok.

For timeline - preview 4 of 15.8 is fine.

[smadala]

@ManishJayaswal We didn't get chance fix this for 15.8 preview 4. We will try to get this for preview 5.
@genlu we discuss with about message and log level when we are fixing it.

[ManishJayaswal]

@smadala - we still have a week to fix this in preview 4.

[smadala]

@ManishJayaswal Good to know, I thought 19th June was last check in for M2 approval(Looks like schedule is changed).

@cltshivash @pvlakshm for triage this.

[smadala]

@ManishJayaswal I'm working on this.

@genlu We will show first 60 characters of testcase filter. Is that sounds good?

@ManishJayaswal We can't make this to info level. Because

• The default console log level for dotnet test is minimal which won't print info messages.
• If user try to run the tests and no tests were discovered then warning makes more sense.

Can we do something on LUT side for not to show obvious warning?

[ManishJayaswal]

@smadala - we should show this error if no tests are found in the assembly. However if tests are found in the assembly but everything is getting filtered out then this should probably be an info, right?

[smadala]

@ManishJayaswal

we should show this error if no tests are found in the assembly.

I agree with you. Because

• See the below message in our PR check,

01:17:23 No test is available in D:\j\workspace\Windows_NT_De---9fe03a42\test\DataCollectors\Microsoft.TestPlatform.Extensions.EventLogCollector.UnitTests\bin\Debug\net451\Microsoft.TestPlatform.Extensions.EventLogCollector.UnitTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

I don't know since when the tests has not been running from test-assembly Microsoft.TestPlatform.Extensions.EventLogCollector.UnitTests.dll. Here we are silently printing warning(In some cases warning helps in build configurations they fails the build on warnings).

• I have see VSTest Task in VSTS CI/CD, we try to run tests for product assemblies if it's name match the test-sources regex. This impact perf.

However we may not afford to do this as this change breaks lot of build pipelines once they update the VS/vstest.console.exe.

However if tests are found in the assembly but everything is getting filtered out then this should probably be an info, right?

• No, No tests run on test command means there is something wrong we should report it.
• Even we want to do this, Currently TestPlatform don't have knowledge that there is no tests because of filter/some other reason.

[ManishJayaswal]

How much work would it be for TP to know if no tests were found in assembly or tests were found but all got filtered away. I suspect that initially all tests are getting discovered and then passed through the filter?

[smadala]

Can we do something on LUT side for not to show obvious warning?

Did you think about this?

[smadala]

I have provided enough info why the message level should be warning. Why do you want it to be info other than harmless and meaningful warning message on LUT restart.

Why the LUT sending run request if it expects no tests to run always?

[ManishJayaswal]

The only things we can do on LUT side is to check every error and block this one which does not seem right. LUT needs to send the request because there may be new tests. Gen can confirm that. Gen is there anything LUT can do to not send requests in such cases?

[smadala]

I have completed the PR to address the message text as suggested by @ManishJayaswal with log level warning. If required I will create another PR to address log level once we discuss on it.