Building Linux Brave from NFS mounted folder fails. Suspecting ninja race

Description of the issue:
When building brave on linux from source, the build process fails. It does complete when built on a local mounted disk.

Steps to Reproduce (add as many as necessary):

  1. cd to a nfs mounted folder
  2. git clone / checkout / npm init / npm run build
  3. build exits with error

Actual Result (gifs and screenshots are welcome!):
see below.

Expected result:
when I

  1. cd to a local disk
  2. git clone / checkout / npm init / npm run build
  3. brave is build

Reproduces how often:

Any time, any release channel. Tested with 0.66.99, 0.66.101 and 0.70.14. All have the same result.

Operating System and Brave Version(See the About Brave page in the main menu):

OS: Linux (LFS alike)
Brave: 0.66.101

Additional Information:

Hi devs,

I’m trying to build brave on linux from source and fail when the build directory is located on a network file system (NFSv4) mounted path.

Here’s what I basically do and which runs fine on a local disk:

/scratch/local2/ is a local xfs formatted and mounted drive:

mkdir -p /scratch/local2/brave-0.66.101-0/build
cd /scratch/local2/brave-0.66.101-0/build
git clone https://github.com/brave/brave-browser
cd brave-browser
git checkout v0.66.101
npm config set color false
npm install
npm run init
npm run build

That’s more or less ‘how-to-build-from-source’ as documented.

This works fine and the resulting brave binary runs like a boss.

If I try this on a NFS mounted drive it fails.

mount host:/drive /mnt/1
mkdir -p /mnt/1/brave-browser-0.66.101-1/build
cd /mnt/1/brave-browser-0.66.101-1/build
git clone https://github.com/brave/brave-browser
cd brave-browser
git checkout v0.66.101
npm config set color false
npm install
npm run init
npm run build
[...]
[11427/37622] CXX obj/third_party/webrtc/pc/peerconnection/peer_connection.o
[11428/37622] CXX obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_core_validation/core_validation.o
ninja: build stopped: subcommand failed.
null
null
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! brave@0.66.101 build: `node ./scripts/commands.js build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the brave@0.66.101 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /scratch/local2/brave-build/home/.npm/_logs/2019-08-02T12_18_47_709Z-debug.log

and the afore mentioned debug log looks like this:

0 info it worked if it ends with ok
1 verbose cli [ '/pkg/node-10.16.0-0/bin/node',
1 verbose cli   '/pkg/node-10.16.0-0/bin/npm',
1 verbose cli   'run',
1 verbose cli   'build' ]
2 info using npm@6.9.0
3 info using node@v10.16.0
4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
5 info lifecycle brave@0.66.101~prebuild: brave@0.66.101
6 info lifecycle brave@0.66.101~build: brave@0.66.101
7 verbose lifecycle brave@0.66.101~build: unsafe-perm in lifecycle true
8 verbose lifecycle brave@0.66.101~build: PATH: /pkg/node-10.16.0-0/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/scratch/local2/brave-build/nfs/brave-browser/node_modules/.bin:/pkg/node-10.16.0-0/bin:/bin:/usr/bin:/usr/sbin:/usr/local/package/bin:/usr/local/bin
9 verbose lifecycle brave@0.66.101~build: CWD: /scratch/local2/brave-build/nfs/brave-browser
10 silly lifecycle brave@0.66.101~build: Args: [ '-c', 'node ./scripts/commands.js build' ]
11 silly lifecycle brave@0.66.101~build: Returned: code: 1  signal: null
12 info lifecycle brave@0.66.101~build: Failed to exec build script
13 verbose stack Error: brave@0.66.101 build: `node ./scripts/commands.js build`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/pkg/node-10.16.0-0/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
13 verbose stack     at EventEmitter.emit (events.js:198:13)
13 verbose stack     at ChildProcess.<anonymous> (/pkg/node-10.16.0-0/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:198:13)
13 verbose stack     at maybeClose (internal/child_process.js:982:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid brave@0.66.101
15 verbose cwd /scratch/local2/brave-build/nfs/brave-browser
16 verbose Linux 4.19.57.mx64.276
17 verbose argv "/pkg/node-10.16.0-0/bin/node" "/pkg/node-10.16.0-0/bin/npm" "run" "build"
18 verbose node v10.16.0
19 verbose npm  v6.9.0
20 error code ELIFECYCLE
21 error errno 1
22 error brave@0.66.101 build: `node ./scripts/commands.js build`
22 error Exit status 1
23 error Failed at the brave@0.66.101 build script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

I tried 0.66.99, 0.66.101 and 0.70.14 and it looks like there is some ninja race condition.

When I dig into the STDOUT/STDERR log of npm run build I find the following:

[...]
[11341/37622] CXX obj/third_party/webrtc/video/video/transport_adapter.o
[11342/37622] CC obj/third_party/yasm/re2c/actions.o
[11343/37622] CXX obj/third_party/webrtc/rtc_base/experiments/field_trial_parser/field_trial_units.o
[11344/37622] CC obj/third_party/yasm/re2c/dfa.o
[11345/37622] LINK ./genstring
[11346/37622] CC obj/third_party/yasm/re2c/code.o
[11347/37622] SOLINK ./libVkLayer_stateless_validation.so
FAILED: libVkLayer_stateless_validation.so libVkLayer_stateless_validation.so.TOC
python "../../build/toolchain/gcc_solink_wrapper.py" --readelf="readelf" --nm="nm" --sofile="./libVkLayer_stateless_validation.so" --tocfile="./libVkLayer_stateless_validation.so.TOC" --output="./libVkLayer_stateless_va
lidation.so" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -shared -Wl,-Bsymbolic,--exclude-libs,ALL -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,defs -Wl,--as-needed -fuse-ld=lld -Wl,
--color-diagnostics -m64 -Wl,--gdb-index -rdynamic -nostdlib++ --sysroot=../../build/linux/debian_sid_amd64-sysroot -L../../build/linux/debian_sid_amd64-sysroot/usr/local/lib/x86_64-linux-gnu -L../../build/linux/debian_
sid_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath=\$ORIGIN -Werror -o "./libVkLayer_stateless_validation.so" -Wl,-soname="libVkLayer_stateless_valida
tion.so" @"./libVkLayer_stateless_validation.so.rsp"
ld.lld: error: undefined symbol: AllVkImageUsageFlagBits
>>> referenced by parameter_validation_utils.cpp:0 (../../third_party/angle/third_party/vulkan-validation-layers/src/layers/parameter_validation_utils.cpp:0)
>>>               obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_stateless_validation/parameter_validation_utils.o:(StatelessValidation::manual_PreCallValidateCreateImageView(VkDevice_T*, VkImageView
CreateInfo const*, VkAllocationCallbacks const*, VkImageView_T**))

ld.lld: error: undefined symbol: GeneratedVulkanHeaderVersion
>>> referenced by parameter_validation_utils.cpp:1042 (../../third_party/angle/third_party/vulkan-validation-layers/src/layers/parameter_validation_utils.cpp:1042)
>>>               obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_stateless_validation/parameter_validation_utils.o:(StatelessValidation::manual_PreCallValidateCreateGraphicsPipelines(VkDevice_T*, VkP
ipelineCache_T*, unsigned int, VkGraphicsPipelineCreateInfo const*, VkAllocationCallbacks const*, VkPipeline_T**))

ld.lld: error: undefined symbol: GeneratedVulkanHeaderVersion
>>> referenced by parameter_validation_utils.cpp:1390 (../../third_party/angle/third_party/vulkan-validation-layers/src/layers/parameter_validation_utils.cpp:1390)
>>>               obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_stateless_validation/parameter_validation_utils.o:(StatelessValidation::manual_PreCallValidateCreateGraphicsPipelines(VkDevice_T*, VkP
ipelineCache_T*, unsigned int, VkGraphicsPipelineCreateInfo const*, VkAllocationCallbacks const*, VkPipeline_T**))

ld.lld: error: undefined symbol: GeneratedVulkanHeaderVersion
>>> referenced by parameter_validation_utils.cpp:1465 (../../third_party/angle/third_party/vulkan-validation-layers/src/layers/parameter_validation_utils.cpp:1465)
>>>               obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_stateless_validation/parameter_validation_utils.o:(StatelessValidation::manual_PreCallValidateCreateGraphicsPipelines(VkDevice_T*, VkPipelineCache_T*, unsigned int, VkGraphicsPipelineCreateInfo const*, VkAllocationCallbacks const*, VkPipeline_T**))

ld.lld: error: undefined symbol: AllVkCompareOpEnums
>>> referenced by parameter_validation_utils.cpp:1486 (../../third_party/angle/third_party/vulkan-validation-layers/src/layers/parameter_validation_utils.cpp:1486)
>>>               obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_stateless_validation/parameter_validation_utils.o:(StatelessValidation::manual_PreCallValidateCreateGraphicsPipelines(VkDevice_T*, VkPipelineCache_T*, unsigned int, VkGraphicsPipelineCreateInfo const*, VkAllocationCallbacks const*, VkPipeline_T**))

ld.lld: error: undefined symbol: AllVkStencilOpEnums
>>> referenced by parameter_validation_utils.cpp:1502 (../../third_party/angle/third_party/vulkan-validation-layers/src/layers/parameter_validation_utils.cpp:1502)
>>>               obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_stateless_validation/parameter_validation_utils.o:(StatelessValidation::manual_PreCallValidateCreateGraphicsPipelines(VkDevice_T*, VkPipelineCache_T*, unsigned int, VkGraphicsPipelineCreateInfo const*, VkAllocationCallbacks const*, VkPipeline_T**))

[...]

ld.lld: error: undefined symbol: AllVkBlendOpEnums
>>> referenced by parameter_validation_utils.cpp:1636 (../../third_party/angle/third_party/vulkan-validation-layers/src/layers/parameter_validation_utils.cpp:1636)
>>>               obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_stateless_validation/parameter_validation_utils.o:(StatelessValidation::manual_PreCallValidateCreateGraphicsPipelines(VkDevice_T*, VkPipelineCache_T*, unsigned int, VkGraphicsPipelineCreateInfo const*, VkAllocationCallbacks const*, VkPipeline_T**))

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[11348/37622] ACTION //third_party/yasm:generate_license(//build/toolchain/linux:clang_x64)
[11349/37622] CXX obj/third_party/webrtc/pc/rtc_pc_base/jsep_transport.o
[11350/37622] CXX obj/third_party/webrtc/video/video/quality_threshold.o
[11351/37622] CXX obj/third_party/webrtc/system_wrappers/system_wrappers/cpu_info.o
[...]

But it has nothing to do with this undefined symbols in particular. The build system has 80 cores + 1 TB RAM and ninja is building highly parallel. It almost always fails at SOLINK ./libVkLayer_stateless_validation.so but sometimes it’s AllVkBlendOpEnums as undefined and sometimes AllVkImageUsageFlagBits.

I once found some NON_BLOCK npm/node issue, where the linker commandline was bigger than 64k and was cut off the last object files in the command line. So there were simply missing in the resulting archive.

I’d like to get a much more verbose output of the building process like ninja -C <path> -v -k 1 but I don’t have a clue how to implant this into the building pipeline (npm run build).

I’m almost certain I’m on the complete wrong way misdirected by my ignorance of knowing anything about node, npm or ninja. But hey: build it local it works, build in on a nfs mount it fails. Simple.

So here is my build script:

#!/bin/sh

if [ $USER != 'build' ];then
  echo "# run me as user 'build'"
  echo "sudo -u build $0"
  exit 1
fi

set -ex

# brave branch to build
BRAVE_VERSION=0.66.101

BUILD_LOCAL=/scratch/local2/brave-build/local
mkdir -vp ${BUILD_LOCAL}

BUILD_NFS=/scratch/local2/brave-build/nfs
# be sure to have some nfs mounted here
# beware to have at least 200 GB RAM here, otherwise use a disk
# mkdir /mnt/shm
# mount --bind /dev/shm /mnt/shm  # /dev cannot be exported
# exportfs localhost:/mnt/shm
# mount localhost:/mnt/shm /scratch/local2/brave-build/nfs


BUILD_GIT=/scratch/local2/brave-build/git
mkdir -vp ${BUILD_GIT}

LOGS=/scratch/local2/brave-build
HOME=/scratch/local2/brave-build/home
TMPDIR=/scratch/local2/brave-build/tmp
PATH=/bin:/usr/bin:/usr/sbin:/usr/local/package/bin:/usr/local/bin

export HOME
export TMPDIR
export PATH

mkdir -vp $HOME
mkdir -vp $TMPDIR

# generic build
build_brave() {

    git clone ${BUILD_GIT}/brave-browser

    cd brave-browser
    git checkout v${BRAVE_VERSION}

    npm config set color false
    npm install
    npm run init
    npm run build

}

#
# build brave on local drive
#

cd ${BUILD_GIT}

[ -d brave-browser ] || git clone https://github.com/brave/brave-browser

cd ${BUILD_NFS}

build_brave  2>&1 | tee ${LOGS}/build-nfs.log

cd ${BUILD_LOCAL}

build_brave  2>&1 | tee ${LOGS}/build-local.log

Any hint would be helpful.

Peter

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.