wrong setting of TARGET_ARCH cause undetected compiler path

[bdbaddog]

This issue was originally created at: 2010-09-29 08:32:56.
This issue was reported by: mgautier.
mgautier said at 2010-09-29 08:32:56

On a Window 64 bits with a Visual Studio (9.0 Express) 32 bits, there is no 64 bits compiler installed (cl/link).

• In win32.py HOST_ARCH is set to the architecture (ie AMD64)
• In Environment.py (l.959), TARGET_ARCH is set to HOST_ARCH
• In vc.py (l.113) we check if TARGET_ARCH exist and, if yes, assume it is a user request. So we don't check other target platform.
• As there is no 64 bit compiler, the script <VStudio_path>/VC/bin/amd64/vcvarsamd64.bat doesn't exist.
• The environment is not set and when scons tries to lauch cl it fail (cl is not a recognized as an internal or external command)

Commenting line 959 in Environment.py (setting TARGET_ARCH) solve the (my) problem. However it could cause another one (?)...

bdbaddog said at 2010-09-30 07:53:56

Please try SCons 2.0.1 and see if it's fixed there.

mgautier said at 2010-10-07 05:13:34

It was already 2.0.1 (stable), my bad. I toke a look on the development 2.0.1 sources and it's same.

Matthieu

jason_at_intel said at 2010-10-26 17:36:18

to find the install location for express in 2008 look for these registry entries:

r'Software\Wow6432Node\Microsoft\VCExpress\9.0\Setup\VC\ProductDir',
r'Software\Microsoft\VCExpress\9.0\Setup\VC\ProductDir'

This will allow you to use the existing logic you would have with the pro version

mgautier said at 2010-10-29 02:50:27

I think you misunderstand my bug report. I'll try to be more explicit :

The bug appears with VS9.0 Express on win 64b cause VS9.0Express doesn't install a 64bit compiler, however it is not a only VS9.0 Express bug. Paths are correctly "discovered" in the registry base and correctly handled.

It is just that the TARGET_ARCH is force to the HOST_ARCH even if user doesn't want to.

As far I understand, the TARGET_ARCH var is intended to be used in this kind of sconscript :

env = Environment()
if forceTargetArch:
    env['TARGET_ARCH'] = "MySuperArch"
lse:
    # let scons find the better arch
    pass
 
env.Program(foo, ... )

In vc.py, we check if TARGET_ARCH is set, and if yes, assume that is a user request and so force the use of this arch. This behavior is concordant with the use of the previous sconscript. BUT, at the creation of the environment (In Environment.py) TARGET_ARCH is always set to HOST_ARCH. So we always create a "user request" even if the user doesn't want to.

If a "HOST_ARCH compatible compiler" is present it is not a problem as the HOST_ARCH is the better arch. But if there is no "HOST_ARCH compatible compiler" scons doesn't build anything as there is no valid compiler. This is the case with VS9.0 Express (in my case) as it install only a 32 bits compiler on a 64 bits Window, but it could also appears with an other version.

I see two options (and it is to scons dev to chose between them) :

• Setting TARGET_ARCH at creation of the environment is normal, but there is a bug in vc.py when we suppose it's a user request
• The behavior of c.py is normal and so we should not set TARGET_ARCH. (What I've proposed in my first message)

Hope that it's clearer.
Matthieu

gregnoel said at 2010-11-14 18:11:43

Bug party triage.

garyo said at 2012-09-01 10:01:49

Bumping all old issues targeted for past releases to 2.x.

[mwichmann]

@jcbrill here's more of this mess - seems similar to another one recently examined. Any thoughts here? (by the way, if we're poking at you too much, please tell us to stop!)

[jcbrill]

Any thoughts here? (by the way, if we're poking at you too much, please tell us to stop!)

@mwichmann No worries!

I just updated the 32-bit and 64-bit vmware virtual machines this morning that have MSVS 2005/2008/2010 express versions installed with the windows sdks for the 64-bit compilers. It takes some manual tweaking (2008: move/rename batch files) and forwarding to the Sdk batch file (2010) to get the 64-bit builds to work for 2008 and 2010. 2005 is a horse of a different color...

Will start looking into issues raised in original post ...

[bdbaddog]

So am I reading this correctly?
They're saying if you set TARGET_ARCH, and don't specificly request a MSVC version, and the newest MSVC version doesn't have the requested TARGET_ARCH, the configure doesn't find a MSVC install which supports the requested TARGET_ARCH and basically silently fails configuration?

[jcbrill]

I have a bit more information. I'm still working on determining the root cause.

With only MSVS 2008 Express installed on 64-bit windows:

• 9.0Exp is not detected and therefore not added to the installed versions list. The call to _check_cl_exists_in_vc_dir very likely looks to be the culprit.
• However, the build succeeds when MSVC_VERSION='9.0Exp' is used as msvc_find_valid_batch_script adjusts accordingly.

I have not tried it yet, but without explicitly specifying the MSVC_VERSION the build will fail as there are no detected MSVC versions.

The SConstruct which builds for x86 successfully:

import sys

env_list = []

def make_environment(kwargs):
    global env_list
    build_n = len(env_list) + 1
    build = '_build{:03d}'.format(build_n)
    print('Build:', build, kwargs, file=sys.stderr)
    VariantDir(build, '.', duplicate=0)
    env=Environment(**kwargs)
    env.Program(build + '/hello', [build + '/hello.c'])
    env_list.append(env)
    return env

for d in [

    {'MSVC_VERSION': '9.0Exp' },

]:
    make_environment(d)

Debug output fragment:

...
trying to find VC 9.0Exp
found VC in registry: C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
found VC 9.0Exp
HOST_ARCH:x86_64
TARGET_ARCH:None
host platform amd64, target platform amd64 for version 9.0Exp
checking for cl.exe at C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\cl.exe
checking for cl.exe at C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\x86_amd64\cl.exe
no compiler found 9.0Exp
...
msvc_version:None msvs_version:None
installed_vcs:[]
No installed VCs

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
...
get_sdk_vc_script():arch_string:amd64 host_arch:amd64 target_arch:amd64
get_sdk_vc_script():file:bin\vcvarsamd64.bat
vc_script:C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat sdk_script:None
use_script 2 'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\vcvarsall.bat', args:amd64
PATH: C:\Windows\System32;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0
Calling 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat amd64'
use_script 3: failed running VC script 'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\vcvarsall.bat': amd64: Error:The specified configuration type is missing.  The tools for the
configuration might not be installed.
trying target_platform:x86_amd64
found VC in registry: C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
looking in C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
get_installed_sdks()
get_sdk_vc_script():arch_string:amd64_x86_amd64 host_arch:amd64 target_arch:x86_amd64
get_sdk_vc_script():file:None
batch file not found:<SCons.Tool.MSCommon.sdk.WindowsSDK object at 0x000001F23BB594A8>
vc_script:C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat sdk_script:None
use_script 2 'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\vcvarsall.bat', args:x86_amd64
PATH: C:\Windows\System32;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0
Calling 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat x86_amd64'
use_script 3: failed running VC script 'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\vcvarsall.bat': x86_amd64: Error:The specified configuration type is missing.  The tools for the
configuration might not be installed.
trying target_platform:x86
found VC in registry: C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
looking in C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
get_installed_sdks()
get_sdk_vc_script():arch_string:x86 host_arch:x86 target_arch:x86
get_sdk_vc_script():file:bin\vcvars32.bat
sdk_bat_file_path:C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat
vc_script:C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat sdk_script:C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat
use_script 2 'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\vcvarsall.bat', args:x86
PATH: C:\Windows\System32;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0
Calling 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat x86'
Found a working script/target: 'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\vcvarsall.bat'/x86
...
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated VariantDir targets: _build001
cl /Fo_build001\hello.obj /c hello.c /nologo
hello.c
link /nologo /OUT:_build001\hello.exe _build001\hello.obj
scons: done building targets.

output log: 2697-output.txt
cache file: 2697-cache.txt

[jcbrill]

msvc_find_valid_batch_script in vc.py checks more host/target combinations (if necessary) than _check_cl_exists_in_vc_dir which explains why the build is successful but the detection fails.

msvc_find_valid_batch_script excerpt:

    try_target_archs = [target_platform]
    if req_target_platform in ('amd64', 'x86_64'):
        try_target_archs.append('x86_amd64')
    elif req_target_platform in ('x86',):
        try_target_archs.append('x86_x86')
    elif req_target_platform in ('arm',):
        try_target_archs.append('x86_arm')
    elif req_target_platform in ('arm64',):
        try_target_archs.append('x86_arm64')
    elif not req_target_platform:
        if target_platform in ('amd64', 'x86_64'):
            try_target_archs.append('x86_amd64')
            # If the user hasn't specifically requested a TARGET_ARCH,
            # and the TARGET_ARCH is amd64 then also try 32 bits
            # if there are no viable 64 bit tools installed
            try_target_archs.append('x86')

The final addition after the comment in the last clause appears to be relevant. Need to study the code more...

[jcbrill]

The problem still exists, the underlying behavior is explained below, and a solution is proposed which may be overkill for this particular issue but is likely better in the long run.

So am I reading this correctly?
They're saying if you set TARGET_ARCH, and don't specificly request a MSVC version, and the newest MSVC version doesn't have the requested TARGET_ARCH, the configure doesn't find a MSVC install which supports the requested TARGET_ARCH and basically silently fails configuration?

Yes.

The requested TARGET_ARCH is kind of a red herring in this instance as the underlying issue is the difference between initial VC detection and the batch file determination for the build environment.

There are two factors at play: (1) the user requested TARGET_ARCH does not appear available during initial VC detection and (2) the default installation of VS2008 express does not contain tools for 64-bit targets.

When only VS2008 Express (9.0Exp) is installed:

• The build will fail if MSVC_VERSION is not specified. 9.0Exp is not detected regardless of the user's TARGET_ARCH request. Reformatted for illustration:

  scons: Reading SConscript files ...
  Build: _build001 {'TARGET_ARCH': 'x86'}
  scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
  File "..\scons-master\scripts\scons.py", line 98, in <module>
  scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
  File "..\scons-master\scripts\scons.py", line 98, in <module>
  scons: done reading SConscript files.
  scons: Building targets ...
  scons: building associated VariantDir targets: _build001
  cl /Fo_build001\hello.obj /c hello.c /nologo
  'cl' is not recognized as an internal or external command, operable program or batch file.
  scons: *** [_build001\hello.obj] Error 1
  scons: building terminated because of errors.
  

• The build will succeed if MSVC_VERSION is specified. 9.0Exp is still not detected so an erroneous warning is issued but the build is successful.

  Running scons (10:29:42.28) ...
  scons: Reading SConscript files ...
  Build: _build001 {'MSVC_VERSION': '9.0Exp'}
  scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
  File "..\scons-master\scripts\scons.py", line 98, in <module>
  scons: done reading SConscript files.
  scons: Building targets ...
  scons: building associated VariantDir targets: _build001
  cl /Fo_build001\hello.obj /c hello.c /nologo
  hello.c
  link /nologo /OUT:_build001\hello.exe _build001\hello.obj
  scons: done building targets.
  

The root cause of behavior is due to the implementation difference in initialization behavior of installed VCs and the handling of build environments:

• during initialization (cl.exe exists), the alternate default x86 target is not considered on x64 platforms
• during determination of the msvc batchfile and/or arguments for the build, the (unnecessarily complicated) adjustments for default x86 targets are considered which is why the build succeeds.

During initialization of the installed VCs, the TARGET_ARCH is undefined. The behavior is to set the TARGET_ARCH = HOST_ARCH. In this case, HOST_ARCH = amd64 and TARGET_ARCH = amd64. This causes only amd64 and x86_amd64 to be checked, neither of which exist in a default installation of VS2008 Express.

Excerpt from the debug log:

trying to find VC 9.0Exp
found VC in registry: c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
found VC 9.0Exp
HOST_ARCH:x86_64
TARGET_ARCH:None
host platform amd64, target platform amd64 for version 9.0Exp
checking for cl.exe at c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\cl.exe
checking for cl.exe at c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\x86_amd64\cl.exe
no compiler found 9.0Exp

AUTHOR EDIT: remaining content moved to #4131 for discussion.