GoogleTestAdapter: VsTestConsole can not be used with settings any more since VS 15.5

[csoltenborn]

This issue is the VS Test equivalent of csoltenborn/GoogleTestAdapter#184

Steps to reproduce

1. Download and unzip Google Test Adapter (GTA)
2. Run VsTest.Console.exe from VS installation >= 15.5
   • use /TestAdapterPath or /UseVsixEstensions options (for the latter, of course first install GTA into VS)
   • use /Settings option to provide a run settings file
   • the SampleTests solution of GTA can serve as examples for tests to be run and also contains some run settings files, but that might not even be necessary

See also csoltenborn/GoogleTestAdapter#184 for more detailed reproduction instructions and some more insights. You can also reproduce (and debug) the behavior by running GTA's end-to-end tests (which use VsTest.Console.exe internally). See wiki for build instructions.

Expected behavior

Settings are found and provided to GTA at runtime.

Actual behavior

Settings are not loaded any more since VS 15.5. Instead, an exception occurs, message: "Problem loading settings: Settings Provider named 'GoogleTestAdapterSettings' was not found. The settings can not be loaded."

There is no relevant log output afaik.

Other notes

• The ListDiscoverers, ListExecutors, and ListSettings options do not work anymore as well.
• @smadala has already briefly looked into Exception when using GTA with VsTest.Console.exe (VS 2017 Version 15.5) csoltenborn/GoogleTestAdapter#184 and also provided this possible duplicate.

[smadala]

@csoltenborn
Looks like issue introduced in VS 15.5 because SettingsProvider implementation loading form assemblies which ends with SettingsProvider.dll.

Below diff should fix the issue. I will raise PR to fix this ASAP.

diff --git a/src/Microsoft.TestPlatform.Common/SettingsProvider/SettingsProviderExtensionManager.cs b/src/Microsoft.TestPlatform.Common/SettingsProvider/SettingsProviderExtensionManager.cs
index 44c5c07d..f855cb4b 100644
--- a/src/Microsoft.TestPlatform.Common/SettingsProvider/SettingsProviderExtensionManager.cs
+++ b/src/Microsoft.TestPlatform.Common/SettingsProvider/SettingsProviderExtensionManager.cs
@@ -115,7 +115,7 @@ namespace Microsoft.VisualStudio.TestPlatform.Common.SettingsProvider

                         TestPluginManager.Instance
                             .GetSpecificTestExtensions<TestSettingsProviderPluginInformation, ISettingsProvider, ISettingsProviderCapabilities, TestSettingsProviderMetadata>(
-                                TestPlatformConstants.SettingsProviderEndsWithPattern,
+                                TestPlatformConstants.TestAdapterEndsWithPattern,
                                 out unfilteredTestExtensions,
                                 out testExtensions);

This has been reported from multiple sources making it P1.

• Settings Provider named 'SpecRun' was not found. The settings can not be loaded. here

• Settings Provider named 'GoogleTestAdapterSettings' was not found. The settings can not be loaded.

• An exception occurred while invoking executor 'executor://microsoft/VisualC/VCTest/1.0': Settings Provider named 'VCTest' was not found. The settings can not be loaded

Related issue #1545

[csoltenborn]

Sounds great, thanks in advance! Should have created this issue (much) earlier. As an excuse, there were a couple of MS guys who knew about csoltenborn/GoogleTestAdapter#184...

Now I know why you wanted to see the size of the pull resulting from your issue ;-)

[smadala]

I found workaround while fixing the issue. Please use /Inisolation option to mitigate the issue.

[csoltenborn]

I finally wanted to disable the workaround for this bug (I provided the possibility to specify the settings file via an env variable, from which the settings would be loaded if the bug occured), but found out that two of my end-to-end tests are still having the same problem. Since I'm having difficulties identifying the differences to the non-failing tests (the end-to-end tests are generated and are thus very similar), I would like to ask you to have a look at it if possible? Maybe the bug did not get solved completely...

Reproducing the issue is fairly easy:

• Check out my project at https://github.com/csoltenborn/GoogleTestAdapter
• Follow build instructions in Wiki (set adapter flavor, execute script)
• Open and build solution
• Place Debugger.Launch() statement at the catch clause of CommonFunctions.SafeGetRunSettingsProvider()
• Run the following tests and and enjoy the exception
  • SampleTests_Settings_DisplayName_EQ_TestMath_AddPassesWithTraits_OR_Type_EQ_Small
  • SampleTests_Settings_Type_EQ_Small