C++ Toolchain Configuration

Overview

To invoke the compiler with the right options, Bazel needs some knowledge about the compiler internals, such as include directories and important flags. In other words, Bazel needs a simplified model of the compiler to understand its workings.

Bazel needs to know the following:

• Whether the compiler supports thinLTO, modules, dynamic linking, or PIC (position independent code).
• Paths to the required tools such as gcc, ld, ar, objcopy, and so on.
• The built-in system include directories. Bazel needs these to validate that all headers that were included in the source file were properly declared in the BUILD file.
• The default sysroot.
• Which flags to use for compilation, linking, archiving.
• Which flags to use for the supported compilation modes (opt, dbg, fastbuild).
• Make variables specifically required by the compiler.

If the compiler has support for multiple architectures, Bazel needs to configure them separately.

CcToolchainConfigInfo is a provider that provides the necessary level of granularity for configuring the behavior of Bazel’s C++ rules. By default, Bazel automatically configures CcToolchainConfigInfo for your build, but you have the option to configure it manually. For that, you need a Starlark rule that provides the CcToolchainConfigInfo and you need to point the toolchain_config attribute of the cc_toolchain to your rule. You can create the CcToolchainConfigInfo by calling cc_common.create_cc_toolchain_config_info(). You can find Starlark constructors for all structs you’ll need in the process in @rules_cc//cc:cc_toolchain_config_lib.bzl.

When a C++ target enters the analysis phase, Bazel selects the appropriate cc_toolchain target based on the BUILD file, and obtains the CcToolchainConfigInfo provider from the target specified in the cc_toolchain.toolchain_config attribute. The cc_toolchain target passes this information to the C++ target through a CcToolchainProvider.

For example, a compile or link action, instantiated by a rule such as cc_binary or cc_library, needs the following information:

• The compiler or linker to use
• Command-line flags for the compiler/linker
• Configuration flags passed through the --copt/--linkopt options
• Environment variables
• Artifacts needed in the sandbox in which the action executes

All of the above information except the artifacts required in the sandbox is specified in the Starlark target that the cc_toolchain points to.

The artifacts to be shipped to the sandbox are declared in the cc_toolchain target. For example, with the cc_toolchain.linker_files attribute you can specify the linker binary and toolchain libraries to ship into the sandbox.

Toolchain selection

The toolchain selection logic operates as follows:

1. User specifies a cc_toolchain_suite target in the BUILD file and points Bazel to the target using the --crosstool_top option.

2. The cc_toolchain_suite target references multiple toolchains. The values of the --cpu and --compiler flags determine which of those toolchains is selected, either based only on the --cpu flag value, or based on a joint --cpu | --compiler value. The selection process is as follows:

• If the --compiler option is specified, Bazel selects the corresponding entry from the cc_toolchain_suite.toolchains attribute with --cpu | --compiler. If Bazel does not find a corresponding entry, it throws an error.

• If the --compiler option is not specified, Bazel selects the corresponding entry from the cc_toolchain_suite.toolchains attribute with just --cpu.

• If no flags are specified, Bazel inspects the host system and selects a --cpu value based on its findings. See the inspection mechanism code.

Once a toolchain has been selected, corresponding feature and action_config objects in the Starlark rule govern the configuration of the build (that is, items described later). These messages allow the implementation of fully fledged C++ features in Bazel without modifying the Bazel binary. C++ rules support multiple unique actions documented in detail in the Bazel source code.

Features

A feature is an entity that requires command-line flags, actions, constraints on the execution environment, or dependency alterations. A feature can be something as simple as allowing BUILD files to select configurations of flags, such as treat_warnings_as_errors, or interact with the C++ rules and include new compile actions and inputs to the compilation, such as header_modules or thin_lto.

Ideally, CcToolchainConfigInfo contains a list of features, where each feature consists of one or more flag groups, each defining a list of flags that apply to specific Bazel actions.

A feature is specified by name, which allows full decoupling of the Starlark rule configuration from Bazel releases. In other words, a Bazel release does not affect the behavior of CcToolchainConfigInfo configurations as long as those configurations do not require the use of new features.

A feature is enabled in one of the following ways:

• The feature’s enabled field is set to true.
• Bazel or the rule owner explicitly enable it.
• The user enables it through the --feature Bazel option or features rule attribute.

Features can have interdependencies, depend on command line flags, BUILD file settings, and other variables.

Feature relationships

Dependencies are typically managed directly with Bazel, which simply enforces the requirements and manages conflicts intrinsic to the nature of the features defined in the build. The toolchain specification allows for more granular constraints for use directly within the Starlark rule that govern feature support and expansion. These are:

Constraint	Description
requires = [feature_set (features = ['feature-name-1', 'feature-name-2']),]	Feature-level. The feature is supported only if the specified required features are enabled. For example, when a feature is only supported in certain build modes (opt, dbg, or fastbuild). If requires contains multiple feature_sets the feature is supported if any of the feature_sets is satisfied (when all specified features are enabled).
implies = ['feature']	Feature-level. This feature implies the specified feature(s). Enabling a feature also implicitly enables all features implied by it (that is, it functions recursively). Also provides the ability to factor common subsets of functionality out of a set of features, such as the common parts of sanitizers. Implied features cannot be disabled.
provides = ['feature']	Feature-level. Indicates that this feature is one of several mutually exclusive alternate features. For example, all of the sanitizers could specify provides = ["sanitizer"]. This improves error handling by listing the alternatives if the user asks for two or more mutually exclusive features at once.
with_features = [with_feature_set(features = ['feature-1'], not_features = ['feature-2'],),]	Flag set-level. A feature can specify multiple flag sets with multiple. When with_features is specified, the flag set will only expand to the build command if there is at least one with_feature_set for which all of the features in the specified features set are enabled, and all the features specified in not_features set are disabled. If with_features is not specified, the flag set will be applied unconditionally for every action specified.

Actions

Actions provide the flexibility to modify the circumstances under which an action executes without assuming how the action will be run. An action_config specifies the tool binary that an action invokes, while a feature specifies the configuration (flags) that determine how that tool behaves when the action is invoked.

Features reference actions to signal which Bazel actions they affect since actions can modify the Bazel action graph. The CcToolchainConfigInfo provider contains actions that have flags and tools associated with them, such as c++-compile. Flags are assigned to each action by associating them with a feature.

Each action name represents a single type of action performed by Bazel, such as compiling or linking. There is, however, a many-to-one relationship between actions and Bazel action types, where a Bazel action type refers to a Java class that implements an action (such as CppCompileAction). In particular, the “assembler actions” and “compiler actions” in the table below are CppCompileAction, while the link actions are CppLinkAction.

Assembler actions

Action	Description
preprocess-assemble	Assemble with preprocessing. Typically for .S files.
assemble	Assemble without preprocessing. Typically for .s files.

Compiler actions

Action	Description
cc-flags-make-variable	Propagates CC_FLAGS to genrules.
c-compile	Compile as C.
c++-compile	Compile as C++.
c++-header-parsing	Run the compiler’s parser on a header file to ensure that the header is self-contained, as it will otherwise produce compilation errors. Applies only to toolchains that support modules.

Link actions

Action	Description
c++-link-dynamic-library	Link a shared library containing all of its dependencies.
c++-link-nodeps-dynamic-library	Link a shared library only containing cc_library sources.
c++-link-executable	Link a final ready-to-run library.

AR actions

AR actions assemble object files into archive libraries (.a files) via ar and encode some semantics into the name.

Action	Description
c++-link-static-library	Create a static library (archive).

LTO actions

Action	Description
lto-backend	ThinLTO action compiling bitcodes into native objects.
lto-index	ThinLTO action generating global index.

Using action_config

The action_config is a Starlark struct that describes a Bazel action by specifying the tool (binary) to invoke during the action and sets of flags, defined by features. These flags apply constraints to the action’s execution.

The action_config() constructor has the following parameters:

Attribute	Description
action_name	The Bazel action to which this action corresponds. Bazel uses this attribute to discover per-action tool and execution requirements.
tools	The executable to invoke. The tool applied to the action will be the first tool in the list with a feature set that matches the feature configuration. Default value must be provided.
flag_sets	A list of flags that applies to a group of actions. Same as for a feature.
env_sets	A list of environment constraints that applies to a group of actions. Same as for a feature.

An action_config can require and imply other features and action_configs as dictated by the feature relationships described earlier. This behavior is similar to that of a feature.

The last two attributes are redundant against the corresponding attributes on features and are included because some Bazel actions require certain flags or environment variables and the goal is to avoid unnecessary action_config+feature pairs. Typically, sharing a single feature across multiple action_configs is preferred.

You can not define more than one action_config with the same action_name within the same toolchain. This prevents ambiguity in tool paths and enforces the intention behind action_config - that an action’s properties are clearly described in a single place in the toolchain.

Using tool constructor

Anaction_config can specify a set of tools via its tools parameter. The tool() constructor takes in the following parameters:

Field	Description
path	Path to the tool in question (relative to the current location).
with_features	A list of feature sets out of which at least one must be satisfied for this tool to apply.

For a given action_config, only a single tool applies its tool path and execution requirements to the Bazel action. A tool is selected by iterating through the tools attribute on an action_config until a tool with a with_feature set matching the feature configuration is found (see Feature relationships earlier on this page for more information). You should end your tool lists with a default tool that corresponds to an empty feature configuration.

Example usage

Features and actions can be used together to implement Bazel actions with diverse cross-platform semantics. For example, debug symbol generation on macOS requires generating symbols in the compile action, then invoking a specialized tool during the link action to create compressed dsym archive, and then decompressing that archive to produce the application bundle and .plist files consumable by Xcode.

With Bazel, this process can instead be implemented as follows, with unbundle-debuginfo being a Bazel action:

load(“@rules_cc//cc:defs.bzl”, “ACTION_NAMES”)

action_configs = [ action_config ( action_name = ACTION_NAMES.cpp_link_executable, tools = [ tool( with_features = [ with_feature(features=[“generate-debug-symbols”]), ], path = “toolchain/mac/ld-with-dsym-packaging”, ), tool (path = “toolchain/mac/ld”), ], ), ]

features = [ feature( name = “generate-debug-symbols”, flag_sets = [ flag_set ( actions = [ ACTION_NAMES.c_compile, ACTION_NAMES.cpp_compile ], flag_groups = [ flag_group( flags = [“-g”], ), ], ) ], implies = [“unbundle-debuginfo”], ), ]

This same feature can be implemented entirely differently for Linux, which uses fission, or for Windows, which produces .pdb files. For example, the implementation for fission-based debug symbol generation might look as follows:

load(“@rules_cc//cc:defs.bzl”, “ACTION_NAMES”)

action_configs = [ action_config ( name = ACTION_NAMES.cpp_compile, tools = [ tool( path = “toolchain/bin/gcc”, ), ], ), ]

features = [ feature ( name = “generate-debug-symbols”, requires = [with_feature_set(features = [“dbg”])], flag_sets = [ flag_set( actions = [ACTION_NAMES.cpp_compile], flag_groups = [ flag_group( flags = [“-gsplit-dwarf”], ), ], ), flag_set( actions = [ACTION_NAMES.cpp_link_executable], flag_groups = [ flag_group( flags = [“-Wl”, “—gdb-index”], ), ], ), ], ), ]

Flag groups

CcToolchainConfigInfo allows you to bundle flags into groups that serve a specific purpose. You can specify a flag within using pre-defined variables within the flag value, which the compiler expands when adding the flag to the build command. For example:

flag_group ( flags = [”%”], )

In this case, the contents of the flag will be replaced by the output file path of the action.

Flag groups are expanded to the build command in the order in which they appear in the list, top-to-bottom, left-to-right.

For flags that need to repeat with different values when added to the build command, the flag group can iterate variables of type list. For example, the variable include_path of type list:

flag_group ( iterate_over = “include_paths”, flags = [“-I%”], )

expands to -I<path> for each path element in the include_paths list. All flags (or flag_groups) in the body of a flag group declaration are expanded as a unit. For example:

flag_group ( iterate_over = “include_paths”, flags = [“-I”, ”%”], )

expands to -I <path> for each path element in the include_paths list.

A variable can repeat multiple times. For example:

flag_group ( iterate_over = “include_paths”, flags = [“-iprefix=%”, “-isystem=%”], )

expands to:

-iprefix=<inc0> -isystem=<inc0> -iprefix=<inc1> -isystem=<inc1>

Variables can correspond to structures accessible using dot-notation. For example:

flag_group ( flags = [“-l%”], )

Structures can be nested and may also contain sequences. To prevent name clashes and to be explicit, you must specify the full path through the fields. For example:

flag_group ( iterate_over = “libraries_to_link”, flag_groups = [ flag_group ( iterate_over = “libraries_to_link.shared_libraries”, flags = [“-l%”], ), ], )

Conditional expansion

Flag groups support conditional expansion based on the presence of a particular variable or its field using the expand_if_available, expand_if_not_available, expand_if_true, expand_if_false, or expand_if_equal attributes. For example:

flag_group ( iterate_over = “libraries_to_link”, flag_groups = [ flag_group ( iterate_over = “libraries_to_link.shared_libraries”, flag_groups = [ flag_group ( expand_if_available = “libraries_to_link.shared_libraries.is_whole_archive”, flags = [“—whole_archive”], ), flag_group ( flags = [“-l%”], ), flag_group ( expand_if_available = “libraries_to_link.shared_libraries.is_whole_archive”, flags = [“—no_whole_archive”], ), ], ), ], )

Note: The --whole_archive and --no_whole_archive options are added to the build command only when a currently iterated library has an is_whole_archive field.

CcToolchainConfigInfo reference

This section provides a reference of build variables, features, and other information required to successfully configure C++ rules.

CcToolchainConfigInfo build variables

The following is a reference of CcToolchainConfigInfo build variables.

Note: The Action column indicates the relevant action type, if applicable.

Variable	Action	Description
source_file	compile	Source file to compile.
input_file	strip	Artifact to strip.
output_file	compile, strip	Compilation output.
output_assembly_file	compile	Emitted assembly file. Applies only when the compile action emits assembly text, typically when using the —save_temps flag. The contents are the same as for output_file.
output_preprocess_file	compile	Preprocessed output. Applies only to compile actions that only preprocess the source files, typically when using the —save_temps flag. The contents are the same as for output_file.
includes	compile	Sequence of files the compiler must unconditionally include in the compiled source.
include_paths	compile	Sequence directories in which the compiler searches for headers included using #include<foo.h> and #include “foo.h”.
quote_include_paths	compile	Sequence of -iquote includes - directories in which the compiler searches for headers included using #include “foo.h”.
system_include_paths	compile	Sequence of -isystem includes - directories in which the compiler searches for headers included using #include <foo.h>.
dependency_file	compile	The .d dependency file generated by the compiler.
preprocessor_defines	compile	Sequence of defines, such as —DDEBUG.
pic	compile	Compiles the output as position-independent code.
gcov_gcno_file	compile	The gcov coverage file.
per_object_debug_info_file	compile	The per-object debug info (.dwp) file.
stripopts	strip	Sequence of stripopts.
legacy_compile_flags	compile	Sequence of flags from legacy CROSSTOOL fields such as compiler_flag, optional_compiler_flag, cxx_flag, and optional_cxx_flag.
user_compile_flags	compile	Sequence of flags from either the copt rule attribute or the —copt, —cxxopt, and —conlyopt flags.
unfiltered_compile_flags	compile	Sequence of flags from the unfiltered_cxx_flag legacy CROSSTOOL field or the unfiltered_compile_flags feature. These are not filtered by the nocopts rule attribute.
sysroot		The sysroot.
runtime_library_search_directories	link	Entries in the linker runtime search path (usually set with the -rpath flag).
library_search_directories	link	Entries in the linker search path (usually set with the -L flag).
libraries_to_link	link	Flags providing files to link as inputs in the linker invocation.
def_file_path	link	Location of def file used on Windows with MSVC.
linker_param_file	link	Location of linker param file created by bazel to overcome command line length limit.
output_execpath	link	Execpath of the output of the linker.
generate_interface_library	link	”yes” or “no” depending on whether interface library should be generated.
interface_library_builder_path	link	Path to the interface library builder tool.
interface_library_input_path	link	Input for the interface library ifso builder tool.
interface_library_output_path	link	Path where to generate interface library using the ifso builder tool.
legacy_link_flags	link	Linker flags coming from the legacy CROSSTOOL fields.
user_link_flags	link	Linker flags coming from the —linkopt or linkopts attribute.
linkstamp_paths	link	A build variable giving linkstamp paths.
force_pic	link	Presence of this variable indicates that PIC/PIE code should be generated (Bazel option --force_pic was passed).
strip_debug_symbols	link	Presence of this variable indicates that the debug symbols should be stripped.
is_cc_test	link	Truthy when current action is a cc_test linking action, false otherwise.
is_using_fission	compile, link	Presence of this variable indicates that fission (per-object debug info) is activated. Debug info will be in .dwo files instead of .o files and the compiler and linker need to know this.
fdo_instrument_path	compile, link	Path to the directory that stores FDO instrumentation profile.
fdo_profile_path	compile	Path to FDO profile.
fdo_prefetch_hints_path	compile	Path to the cache prefetch profile.
cs_fdo_instrument_path	compile, link	Path to the directory that stores context sensitive FDO instrumentation profile.

Well-known features

The following is a reference of features and their activation conditions.

Feature	Documentation
opt | dbg | fastbuild	Enabled by default based on compilation mode.
static_linking_mode | dynamic_linking_mode	Enabled by default based on linking mode.
per_object_debug_info	Enabled if the supports_fission feature is specified and enabled and the current compilation mode is specified in the —fission flag.
supports_start_end_lib	If enabled (and the option —start_end_lib is set), Bazel will not link against static libraries but instead use the —start-lib/—end-lib linker options to link against objects directly. This speeds up the build since Bazel doesn’t have to build static libraries.
supports_interface_shared_libraries	If enabled (and the option —interface_shared_objects is set), Bazel will link targets that have linkstatic set to False (cc_tests by default) against interface shared libraries. This makes incremental relinking faster.
supports_dynamic_linker	If enabled, C++ rules will know the toolchain can produce shared libraries.
static_link_cpp_runtimes	If enabled, Bazel will link the C++ runtime statically in static linking mode and dynamically in dynamic linking mode. Artifacts specified in the cc_toolchain.static_runtime_lib or cc_toolchain.dynamic_runtime_lib attribute (depending on the linking mode) will be added to the linking actions.
supports_pic	If enabled, toolchain will know to use PIC objects for dynamic libraries. The pic variable is present whenever PIC compilation is needed. If not enabled by default, and --force_pic is passed, Bazel will request supports_pic and validate that the feature is enabled. If the feature is missing, or couldn’t be enabled, --force_pic cannot be used.
static_linking_mode | dynamic_linking_mode	Enabled by default based on linking mode.
no_legacy_features	Prevents Bazel from adding legacy features to the C++ configuration when present. See the complete list of features below.
shorten_virtual_includes	If enabled, virtual include header files are linked under bin/_virtual_includes/<hash of target path> instead of bin/<target package path>/_virtual_includes/<target name>. Useful on Windows to avoid long path issue with MSVC.

Legacy features patching logic

Bazel applies the following changes to the toolchain’s features for backwards compatibility:

• Moves legacy_compile_flags feature to the top of the toolchain
• Moves default_compile_flags feature to the top of the toolchain
• Adds dependency_file (if not present) feature to the top of the toolchain
• Adds pic (if not present) feature to the top of the toolchain
• Adds per_object_debug_info (if not present) feature to the top of the toolchain
• Adds preprocessor_defines (if not present) feature to the top of the toolchain
• Adds includes (if not present) feature to the top of the toolchain
• Adds include_paths (if not present) feature to the top of the toolchain
• Adds fdo_instrument (if not present) feature to the top of the toolchain
• Adds fdo_optimize (if not present) feature to the top of the toolchain
• Adds cs_fdo_instrument (if not present) feature to the top of the toolchain
• Adds cs_fdo_optimize (if not present) feature to the top of the toolchain
• Adds fdo_prefetch_hints (if not present) feature to the top of the toolchain
• Adds autofdo (if not present) feature to the top of the toolchain
• Adds build_interface_libraries (if not present) feature to the top of the toolchain
• Adds dynamic_library_linker_tool (if not present) feature to the top of the toolchain
• Adds shared_flag (if not present) feature to the top of the toolchain
• Adds linkstamps (if not present) feature to the top of the toolchain
• Adds output_execpath_flags (if not present) feature to the top of the toolchain
• Adds runtime_library_search_directories (if not present) feature to the top of the toolchain
• Adds library_search_directories (if not present) feature to the top of the toolchain
• Adds archiver_flags (if not present) feature to the top of the toolchain
• Adds libraries_to_link (if not present) feature to the top of the toolchain
• Adds force_pic_flags (if not present) feature to the top of the toolchain
• Adds user_link_flags (if not present) feature to the top of the toolchain
• Adds legacy_link_flags (if not present) feature to the top of the toolchain
• Adds static_libgcc (if not present) feature to the top of the toolchain
• Adds fission_support (if not present) feature to the top of the toolchain
• Adds strip_debug_symbols (if not present) feature to the top of the toolchain
• Adds coverage (if not present) feature to the top of the toolchain
• Adds llvm_coverage_map_format (if not present) feature to the top of the toolchain
• Adds gcc_coverage_map_format (if not present) feature to the top of the toolchain
• Adds fully_static_link (if not present) feature to the bottom of the toolchain
• Adds user_compile_flags (if not present) feature to the bottom of the toolchain
• Adds sysroot (if not present) feature to the bottom of the toolchain
• Adds unfiltered_compile_flags (if not present) feature to the bottom of the toolchain
• Adds linker_param_file (if not present) feature to the bottom of the toolchain
• Adds compiler_input_flags (if not present) feature to the bottom of the toolchain
• Adds compiler_output_flags (if not present) feature to the bottom of the toolchain

This is a long list of features. The plan is to get rid of them once Crosstool in Starlark is done. For the curious reader see the implementation in CppActionConfigs, and for production toolchains consider adding no_legacy_features to make the toolchain more standalone.