I have switched my windows/master build to using docker in order to have a consistent development environment. I will paste my full Dockerfile below, but the basic setup is:
Using the kitware/cmb:ci-superbuild-windows10-msvc2019-20191111 image (on dockerhub)
Clone cmb-superbuild:master
Update the common-superbuild subproject to master, in order to get Ben’s recent qt fix.
Configure with python3
Dockerfile:
FROM kitware/cmb:ci-superbuild-windows10-msvc2019-20191111
RUN mkdir /Users/ContainerUser/code
WORKDIR /Users/ContainerUser/code
RUN git clone --recursive https://gitlab.kitware.com/cmb/cmb-superbuild.git
# Need qt fixes 09-Dec-2019:
WORKDIR /Users/ContainerUser/code/cmb-superbuild/superbuild
RUN git checkout master
RUN mkdir -p /Users/ContainerUser/build/dev-superbuild
WORKDIR /Users/ContainerUser/build/dev-superbuild
RUN cmake \
-GNinja \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_INSTALL_PREFIX:PATH=install \
-DENABLE_cmbusersguide:BOOL=OFF \
-DENABLE_DOCUMENTATION:BOOL=OFF \
-DENABLE_paraview:BOOL=ON \
-DENABLE_python3:BOOL=ON \
-DQt5Core_DIR:PATH=/Qt/5.12.5/msvc2017_64/lib/cmake/Qt5 \
-DUSE_SYSTEM_qt5:BOOL=ON \
/Users/ContainerUser/code/cmb-superbuild
RUN ninja
RUN ctest -R modelbuilder
First error is configuring smtk, specifically finding python3. CMake seems to have found the executable, but not an “Interpreter” component.
CMake Error at C:/Program Files/CMake/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find Python3 (missing: Python3_EXECUTABLE Interpreter) (found
version "3.7.4")
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/CMake/share/cmake-3.15/Modules/FindPython/Support.cmake:1653 (find_package_handle_standard_args)
C:/Program Files/CMake/share/cmake-3.15/Modules/FindPython3.cmake:215 (include)
CMakeLists.txt:433 (find_package)
-- Configuring incomplete, errors occurred!
See also "C:/Users/ContainerUser/build/dev-superbuild/superbuild/smtk/build/CMakeFiles/CMakeOutput.log".
See also "C:/Users/ContainerUser/build/dev-superbuild/superbuild/smtk/build/CMakeFiles/CMakeError.log".
CMake Error at C:/Users/ContainerUser/build/dev-superbuild/superbuild/sb-smtk-configure.cmake:47 (message):
Failed with exit code 1
I am pretty sure there is a way to make docker build the image even when an error occurs. I’ll update the Dockerfile to do that on subsequent attempts.
To get some visibility, I modified my Dockerfile to save the image when the ninja command failed. So I have a “full superbuild” image (all 27 GB of it) on our cmbsharedwin laptop (fka Haocheng’s laptop). My going-forward plan is to nag @ben.boeckel to figure out how to get this image to finish building and packaging.
Side note: the syntax for skipping errors in windows/docker is different than regular docker. The windows version doesn’t accept “&&” but is OK with “;” (I didn’t try “|”). So, for example:
Update: It turns out there is a problem with the FindPython3 module in cmake 3.15, that hoses Python3_EXECUTABLE if python 3.8 isn’t found. (Ben please correct me if needed.) We need to update the docker image to use cmake 3.16. I did that manually to confirm that the error configuring smtk is resolved, and I am continuing the build from there. Thanks Ben.
@ben.boeckel This begs the question I never thought to ask yesterday: If cmake 3.15 screwed up FindPython3, how can gitlab ci build smtk on windows? Doesn’t it use the same base image (kitware/cmb:ci-superbuild-windows10-msvc2019-20191111)?
In the meanwhile, docker started building smtk but hit an error compiling one of the test programs (UnsetValueError.cxx). I’ll paste the error messages below, but change my script to turn off smtk testing and see what’s next.
CORRECTION (It’s hard to read the docker console)
There is a warning about deriving UnsetValueError from std::runtime_error, but stackoverflow gives me the impression we can ignore it.
There is an error building the unitUnsignedValueError.cxx, that it doesn’t have SMTK_DATA_DIR defined. I presume that is because the superbuild’s smtk.cmake doesn’t define one anywhere. Either way, turning off testing should let me move on.
Another observation: the docker image seems to only use 2 of the 8 cores the on cmbsharedwin machine. So it might be awhile before the next update.
FYI only
[120/1345] Building CXX object smtk\attribute\testing\cxx\CMakeFiles\UnitTests_smtk_attribute_testing_cxx.dir\unitUnsetV
alueError.cxx.obj
FAILED: smtk/attribute/testing/cxx/CMakeFiles/UnitTests_smtk_attribute_testing_cxx.dir/unitUnsetValueError.cxx.obj
“C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.23.28105\bin\Hostx64\x64\cl.exe” /nolo
go /TP -DBOOST_ALL_NO_LIB -DSMTK_SCRATCH_DIR=“C:/Users/ContainerUser/build/dev-superbuild/superbuild/smtk/build/Testing
/Temporary” -DVTK_SUPPORT -I. -IC:\Users\ContainerUser\build\dev-superbuild\superbuild\smtk\src -Ismtk\attribute\testin
g\cxx -IC:\Users\ContainerUser\build\dev-superbuild\superbuild\smtk\src\smtk\extension\vtk\meshing -IC:\Users\ContainerU
ser\build\dev-superbuild\superbuild\smtk\src\thirdparty -Ithirdparty -IC:\Users\ContainerUser\build\dev-superbuild\super
build\smtk\src\thirdparty\pugixml -IC:\Users\ContainerUser\build\dev-superbuild\install\include\boost-1_71 -IC:\Users\Co
ntainerUser\build\dev-superbuild\install\include /DWIN32 /D_WINDOWS /W3 /GR /EHsc /EHsc /wd4996 /wd4503 /MD /O2 /Ob2 /DN
DEBUG /showIncludes /Fosmtk\attribute\testing\cxx\CMakeFiles\UnitTests_smtk_attribute_testing_cxx.dir\unitUnsetValueErro
r.cxx.obj /Fdsmtk\attribute\testing\cxx\CMakeFiles\UnitTests_smtk_attribute_testing_cxx.dir\ /FS -c C:\Users\ContainerUs
er\build\dev-superbuild\superbuild\smtk\src\smtk\attribute\testing\cxx\unitUnsetValueError.cxx
C:\Users\ContainerUser\build\dev-superbuild\superbuild\smtk\src\smtk/attribute/UnsetValueError.h(25): warning C4275: non
dll-interface class ‘std::runtime_error’ used as base for dll-interface class ‘smtk::attribute::UnsetValueError’
c:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.23.28105\include\stdexcept(123): note: s
ee declaration of ‘std::runtime_error’
C:\Users\ContainerUser\build\dev-superbuild\superbuild\smtk\src\smtk/attribute/UnsetValueError.h(24): note: see declarat
ion of ‘smtk::attribute::UnsetValueError’
C:\Users\ContainerUser\build\dev-superbuild\superbuild\smtk\src\smtk\attribute\testing\cxx\unitUnsetValueError.cxx(51):
error C2065: ‘SMTK_DATA_DIR’: undeclared identifier
[121/1345] Building CXX object smtk\operation\testing\cxx\CMakeFiles\UnitTests_smtk_operation_testing_cxx.dir\TestMutexe
dOperation.cxx.obj
[122/1345] Building CXX object smtk\operation\pybind11\CMakeFiles_smtkPybindOperation.dir\PybindOperation.cxx.obj
ninja: build stopped: subcommand failed.
CMake Error at C:/Users/ContainerUser/build/dev-superbuild/superbuild/sb-smtk-build.cmake:47 (message):
Failed with exit code 1
Well, after updating my Dockerfile to upgrade cmake to version 16.1 and waiting 3+ hours, smtk STILL fails to configure, with the same error “Could not find Python3 (missing: Python3_EXECUTABLE Interpreter)”
Turns out that, although I updated “chocolatey’s” cmake, the smtk configure is apparently getting called with the cmake that gets installed with VS 2019. I think I’ve stumbled through how to put choclately’s cmake as the first PATH item, so we’ll make that the next attempt.
It’s not the cmake version, or, perhaps, it’s not just the cmake version.
The problem still occurs with cmake 3.16.1 – unable to configure smtk because Python3 interpreter not found. With nothing to lose, I added Python3_EXECUTABLE to the smtk.cmake file. Here’s the diff:
index 07f8a9b..c94d236 100644
--- a/projects/smtk.cmake
+++ b/projects/smtk.cmake
@@ -1,5 +1,8 @@
include(smtk-version)
+message("CMAKE_VERSION ${CMAKE_VERSION}")
+message("superbuild_install_location ${superbuild_install_location}")
+
set(smtk_extra_cmake_args)
if (WIN32)
# On Windows we expect the Python source for module to be
@@ -7,7 +10,8 @@ if (WIN32)
# place than SMTK would put it by default. Tell SMTK where
# to install Python source for the smtk module:
list(APPEND smtk_extra_cmake_args
- "-DSMTK_PYTHON_MODULEDIR:PATH=bin/Lib/site-packages")
+ "-DSMTK_PYTHON_MODULEDIR:PATH=bin/Lib/site-packages"
+ "-DPython3_EXECUTABLE:FILEPATH=${superbuild_install_location}/Python/python.exe")
endif ()
For whatever reason, that fixed the problem configuring smtk. I’ll post the next roadblock in a separate reply.
Given the workaround for configuring smtk, the next issue was smtkprojectmanager failing to configure because of Python3, this time it couldn’t find Python3_INCLUDE_DIR, if I remember correctly. Rather than screw around with this, I just added ALL of the cmake cache variables that python generates to the smtkprojectmanager.cmake file. That fixed the configure problem, so the next roadblock was building project manager. The error message is one we have seen before, that the link command is using python37.dll instead of .lib. I’ll paste the console messages here, and make a note to myself that this is from log4.
-- Build files have been written to: C:/Users/ContainerUser/build/dev-superbuild/superbuild/smtkprojectmanager/build
[286/293] Performing build step for 'smtkprojectmanager'
FAILED: superbuild/smtkprojectmanager/stamp/smtkprojectmanager-build
cmd.exe /C "cd /D C:\Users\ContainerUser\build\dev-superbuild\superbuild\smtkprojectmanager\build && "C:\Program Files\cmake\bin\cmake.exe" -P C:/Users/ContainerUser/build/dev-superbuild/superbuild/sb-smtkprojectmanager-build.cmake && "C:\Program Files\cmake\bin\cmake.exe" -E touch C:/Users/ContainerUser/build/dev-superbuild/superbuild/smtkprojectmanager/stamp/smtkprojectmanager-build"
[1/19] Building CXX object config\CMakeFiles\projectConfig.dir\Registry.cxx.obj
[2/19] Building CXX object config\CMakeFiles\projectConfig.dir\Truchas.cxx.obj
[3/19] Building CXX object config\CMakeFiles\projectConfig.dir\ACE3P.cxx.obj
[4/19] Building CXX object config\CMakeFiles\projectConfig.dir\Simulation.cxx.obj
[5/19] Linking CXX shared library bin\projectConfig.dll
Creating library config\projectConfig.lib and object config\projectConfig.exp
[6/19] Automatic MOC for target smtkProjectManagerPlugin
[7/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\smtkProjectManagerPluginPlugin.cxx.obj
[8/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\smtkProjectManagerPlugin_autogen\mocs_compilation.cpp.obj
[9/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectCloseBehavior.cxx.obj
[10/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectExportBehavior.cxx.obj
[11/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectNewBehavior.cxx.obj
[12/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectImportModelBehavior.cxx.obj
[13/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectMenu.cxx.obj
[14/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectSaveBehavior.cxx.obj
[15/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectOpenBehavior.cxx.obj
[16/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectMenuImplementation.cxx.obj
[17/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKRecentProjectsMenu.cxx.obj
[18/19] Building CXX object plugin\CMakeFiles\smtkProjectManagerPlugin.dir\pqSMTKProjectLoader.cxx.obj
[19/19] Linking CXX shared module bin\smtk-3.3.0\smtkProjectManagerPlugin\smtkProjectManagerPlugin.dll
FAILED: bin/smtk-3.3.0/smtkProjectManagerPlugin/smtkProjectManagerPlugin.dll
cmd.exe /C "cd . && "C:\Program Files\cmake\bin\cmake.exe" -E vs_link_dll --intdir=plugin\CMakeFiles\smtkProjectManagerPlugin.dir --rc="C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64\rc.exe" --mt="C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64\mt.exe" --manifests -- "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.23.28105\bin\Hostx64\x64\link.exe" /nologo @CMakeFiles\smtkProjectManagerPlugin.rsp /out:bin\smtk-3.3.0\smtkProjectManagerPlugin\smtkProjectManagerPlugin.dll /implib:plugin\smtkProjectManagerPlugin.lib /pdb:bin\smtk-3.3.0\smtkProjectManagerPlugin\smtkProjectManagerPlugin.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO && cd ."
LINK: command "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.23.28105\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\smtkProjectManagerPlugin.rsp /out:bin\smtk-3.3.0\smtkProjectManagerPlugin\smtkProjectManagerPlugin.dll /implib:plugin\smtkProjectManagerPlugin.lib /pdb:bin\smtk-3.3.0\smtkProjectManagerPlugin\smtkProjectManagerPlugin.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO /MANIFEST /MANIFESTFILE:bin\smtk-3.3.0\smtkProjectManagerPlugin\smtkProjectManagerPlugin.dll.manifest" failed (exit code 1107) with the following output:
C:\Users\ContainerUser\build\dev-superbuild\install\Python\python37.dll : fatal error LNK1107: invalid or corrupt file: cannot read at 0x2F8
ninja: build stopped: subcommand failed.
CMake Error at C:/Users/ContainerUser/build/dev-superbuild/superbuild/sb-smtkprojectmanager-build.cmake:47 (message):
Failed with exit code 1
Putting smtkprojectmanager aside for the moment, the next error occurs building cmb, specifically, linking modelbuilder, where there is a problem with the python dll.
Well… the linking issue is fixed by changing cmake_minimum_required from 3.9.0 to 3.10.0, go figure.
1. I will try that with smtkprojectmanager next.
2. Moving on, the modelbuilder packaging fails to find python 3. I will try the logic below, which is copied from paraview superbuld (win32/paraview.bundle.cmake); @ben.boeckel please let me know if I’m on the wrong track.
set(exclude_regexes)
if (python3_enabled)
if (python3_built_by_superbuild)
list(APPEND library_paths
"${superbuild_install_location}/Python")
else()
list(APPEND exclude_regexes
".*python3[0-9]+.dll")
endif()
endif ()
Rats, we gotta take back the cmake_minimum_version() “fix”. It turns out that things only looked fixed because we reran configure. I know that Ben is already looking into this in a slightly different context.
So, mostly fyi, I can manually generate a modelbuilder package file, but I cannot seem to get it out of the container in order to try it out. The “docker cp” command doesn’t work on running containers with windows/HyperV. Also I cannot “docker commit” the container to save the package file, presumably because of the same Windows/HyperV limitation. I also wasn’t able to upload to my ftp server, but instead got a message “no data connection”. Maybe that’s a firewall thing?
@ben.boeckel I have the windows laptop (cmbsharedwin) with a new build overnight using the hand-modified FindPython3 module. It hits the same problem linking modelbuilder, pointing to the python3 .dll instead of the .lib file. Whenever you can look at this please…
Great news – on Friday (20-Dec) we were able to build and package the baseline modelbuilder on windows (docker), and verify that it would run on another machine. A big milestone, thanks mostly to Ben.
There are some minor updates needed to cmb-superbuild, including a workaround needed until a VTK MR (!6331) makes its way to cmb. The superbuild MR that currently builds and packages on windows is Windows Packaging (!579).
With that in place, I also finished the ACE3P Plugin needed for the SLAC project, and added it to the superbuild and got that to build on windows. The superbuild MR for that is WIP ACE3P Plugin (!580). It is marked WIP because it is based on the Windows Packaging MR.
There remain some issues to work out, that, I think, are relatively minor:
The superbuild MRs are failing two CI tests: unitAttributeAssociationConstraints and modelAttributesPy. I think the superbuild is pulling smtk:master.
I only tested the ACE3P plugin on linux so far. The cumulus jobs panel works OK, and I updated the workflow files to exoirt OK (the “current” ACE3P workflow files are now in the plugin, superseding the simulation-workflow repository). When I tried to submit a job to NERSC, there was an error importing girder-client, but I think I know the problem and how to fix it.
I also need to figure out how to script the complete windows-docker build process. The biggest impediment seems to be that our ci-windows10-base-20191111 doesn’t support all of the docker syntax (try passing a build argument to it). So it takes some trial and error to figure things out sometimes.
Well that didn’t last long – I thought I was almost there, but instead I am backed up to a new “smtk wont configure” problem that just showed up out of the blue today (even though almost nothing changed this week…)
I posted the console messages here. To be more specific, cmake configures OK and it is the generate step that no longer succeeds. There are two types of errors:
1. The first look like this:
e[0me[91mCMake Error: install(EXPORT "SMTKParaViewExtensionPlugins" ...) includes target "smtkPVServerExtCS" which requires target "smtkPVServerExt" that is not in any export set.
2. And the second look like this:
e[0me[91mCMake Error in smtk/session/vtk/CMakeLists.txt:
export called with target "smtkVTKSession" which requires target
"smtkIOVTK" that is not in any export set.
I have only done a little googling so far. @ben.boeckel if you are online this weekend and can advise, I would appreciate any insights into what could possibly be causing this. Also, I’ll be back in the office on Monday and will bring the f’ing windows laptop with me.
For what it’s worth, I tried moving back to the smtk code on Dec 19 (when I think it built) but still get the same errors. And I am pretty sure I was using the same cmake as last week (3.16.2).
. The question of how my build completed is a valid question however…