1.. SPDX-License-Identifier: GPL-2.0
  2
  3=====================================
  4Using Propeller with the Linux kernel
  5=====================================
  6
  7This enables Propeller build support for the kernel when using Clang
  8compiler. Propeller is a profile-guided optimization (PGO) method used
  9to optimize binary executables. Like AutoFDO, it utilizes hardware
 10sampling to gather information about the frequency of execution of
 11different code paths within a binary. Unlike AutoFDO, this information
 12is then used right before linking phase to optimize (among others)
 13block layout within and across functions.
 14
 15A few important notes about adopting Propeller optimization:
 16
 17#. Although it can be used as a standalone optimization step, it is
 18   strongly recommended to apply Propeller on top of AutoFDO,
 19   AutoFDO+ThinLTO or Instrument FDO. The rest of this document
 20   assumes this paradigm.
 21
 22#. Propeller uses another round of profiling on top of
 23   AutoFDO/AutoFDO+ThinLTO/iFDO. The whole build process involves
 24   "build-afdo - train-afdo - build-propeller - train-propeller -
 25   build-optimized".
 26
 27#. Propeller requires LLVM 19 release or later for Clang/Clang++
 28   and the linker(ld.lld).
 29
 30#. In addition to LLVM toolchain, Propeller requires a profiling
 31   conversion tool: https://github.com/google/autofdo with a release
 32   after v0.30.1: https://github.com/google/autofdo/releases/tag/v0.30.1.
 33
 34The Propeller optimization process involves the following steps:
 35
 36#. Initial building: Build the AutoFDO or AutoFDO+ThinLTO binary as
 37   you would normally do, but with a set of compile-time / link-time
 38   flags, so that a special metadata section is created within the
 39   kernel binary. The special section is only intend to be used by the
 40   profiling tool, it is not part of the runtime image, nor does it
 41   change kernel run time text sections.
 42
 43#. Profiling: The above kernel is then run with a representative
 44   workload to gather execution frequency data. This data is collected
 45   using hardware sampling, via perf. Propeller is most effective on
 46   platforms supporting advanced PMU features like LBR on Intel
 47   machines. This step is the same as profiling the kernel for AutoFDO
 48   (the exact perf parameters can be different).
 49
 50#. Propeller profile generation: Perf output file is converted to a
 51   pair of Propeller profiles via an offline tool.
 52
 53#. Optimized build: Build the AutoFDO or AutoFDO+ThinLTO optimized
 54   binary as you would normally do, but with a compile-time /
 55   link-time flag to pick up the Propeller compile time and link time
 56   profiles. This build step uses 3 profiles - the AutoFDO profile,
 57   the Propeller compile-time profile and the Propeller link-time
 58   profile.
 59
 60#. Deployment: The optimized kernel binary is deployed and used
 61   in production environments, providing improved performance
 62   and reduced latency.
 63
 64Preparation
 65===========
 66
 67Configure the kernel with::
 68
 69   CONFIG_AUTOFDO_CLANG=y
 70   CONFIG_PROPELLER_CLANG=y
 71
 72Customization
 73=============
 74
 75The default CONFIG_PROPELLER_CLANG setting covers kernel space objects
 76for Propeller builds. One can, however, enable or disable Propeller build
 77for individual files and directories by adding a line similar to the
 78following to the respective kernel Makefile:
 79
 80- For enabling a single file (e.g. foo.o)::
 81
 82   PROPELLER_PROFILE_foo.o := y
 83
 84- For enabling all files in one directory::
 85
 86   PROPELLER_PROFILE := y
 87
 88- For disabling one file::
 89
 90   PROPELLER_PROFILE_foo.o := n
 91
 92- For disabling all files in one directory::
 93
 94   PROPELLER__PROFILE := n
 95
 96
 97Workflow
 98========
 99
100Here is an example workflow for building an AutoFDO+Propeller kernel:
101
1021) Assuming an AutoFDO profile is already collected following
103   instructions in the AutoFDO document, build the kernel on the host
104   machine, with AutoFDO and Propeller build configs ::
105
106      CONFIG_AUTOFDO_CLANG=y
107      CONFIG_PROPELLER_CLANG=y
108
109   and ::
110
111      $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<autofdo-profile-name>
112
1132) Install the kernel on the test machine.
114
1153) Run the load tests. The '-c' option in perf specifies the sample
116   event period. We suggest using a suitable prime number, like 500009,
117   for this purpose.
118
119   - For Intel platforms::
120
121      $ perf record -e BR_INST_RETIRED.NEAR_TAKEN:k -a -N -b -c <count> -o <perf_file> -- <loadtest>
122
123   - For AMD platforms::
124
125      $ perf record --pfm-event RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k -a -N -b -c <count> -o <perf_file> -- <loadtest>
126
127   Note you can repeat the above steps to collect multiple <perf_file>s.
128
1294) (Optional) Download the raw perf file(s) to the host machine.
130
1315) Use the create_llvm_prof tool (https://github.com/google/autofdo) to
132   generate Propeller profile. ::
133
134      $ create_llvm_prof --binary=<vmlinux> --profile=<perf_file>
135                         --format=propeller --propeller_output_module_name
136                         --out=<propeller_profile_prefix>_cc_profile.txt
137                         --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt
138
139   "<propeller_profile_prefix>" can be something like "/home/user/dir/any_string".
140
141   This command generates a pair of Propeller profiles:
142   "<propeller_profile_prefix>_cc_profile.txt" and
143   "<propeller_profile_prefix>_ld_profile.txt".
144
145   If there are more than 1 perf_file collected in the previous step,
146   you can create a temp list file "<perf_file_list>" with each line
147   containing one perf file name and run::
148
149      $ create_llvm_prof --binary=<vmlinux> --profile=@<perf_file_list>
150                         --format=propeller --propeller_output_module_name
151                         --out=<propeller_profile_prefix>_cc_profile.txt
152                         --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt
153
1546) Rebuild the kernel using the AutoFDO and Propeller
155   profiles. ::
156
157      CONFIG_AUTOFDO_CLANG=y
158      CONFIG_PROPELLER_CLANG=y
159
160   and ::
161
162      $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<profile_file> CLANG_PROPELLER_PROFILE_PREFIX=<propeller_profile_prefix>