1x86 Topology
  2============
  3
  4This documents and clarifies the main aspects of x86 topology modelling and
  5representation in the kernel. Update/change when doing changes to the
  6respective code.
  7
  8The architecture-agnostic topology definitions are in
  9Documentation/cputopology.txt. This file holds x86-specific
 10differences/specialities which must not necessarily apply to the generic
 11definitions. Thus, the way to read up on Linux topology on x86 is to start
 12with the generic one and look at this one in parallel for the x86 specifics.
 13
 14Needless to say, code should use the generic functions - this file is *only*
 15here to *document* the inner workings of x86 topology.
 16
 17Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
 18
 19The main aim of the topology facilities is to present adequate interfaces to
 20code which needs to know/query/use the structure of the running system wrt
 21threads, cores, packages, etc.
 22
 23The kernel does not care about the concept of physical sockets because a
 24socket has no relevance to software. It's an electromechanical component. In
 25the past a socket always contained a single package (see below), but with the
 26advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
 27there might be still references to sockets in the code, but they are of
 28historical nature and should be cleaned up.
 29
 30The topology of a system is described in the units of:
 31
 32    - packages
 33    - cores
 34    - threads
 35
 36* Package:
 37
 38  Packages contain a number of cores plus shared resources, e.g. DRAM
 39  controller, shared caches etc.
 40
 41  AMD nomenclature for package is 'Node'.
 42
 43  Package-related topology information in the kernel:
 44
 45  - cpuinfo_x86.x86_max_cores:
 46
 47    The number of cores in a package. This information is retrieved via CPUID.
 48
 49  - cpuinfo_x86.phys_proc_id:
 50
 51    The physical ID of the package. This information is retrieved via CPUID
 52    and deduced from the APIC IDs of the cores in the package.
 53
 54  - cpuinfo_x86.logical_id:
 55
 56    The logical ID of the package. As we do not trust BIOSes to enumerate the
 57    packages in a consistent way, we introduced the concept of logical package
 58    ID so we can sanely calculate the number of maximum possible packages in
 59    the system and have the packages enumerated linearly.
 60
 61  - topology_max_packages():
 62
 63    The maximum possible number of packages in the system. Helpful for per
 64    package facilities to preallocate per package information.
 65
 66
 67* Cores:
 68
 69  A core consists of 1 or more threads. It does not matter whether the threads
 70  are SMT- or CMT-type threads.
 71
 72  AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
 73  "core".
 74
 75  Core-related topology information in the kernel:
 76
 77  - smp_num_siblings:
 78
 79    The number of threads in a core. The number of threads in a package can be
 80    calculated by:
 81
 82	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
 83
 84
 85* Threads:
 86
 87  A thread is a single scheduling unit. It's the equivalent to a logical Linux
 88  CPU.
 89
 90  AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
 91  uses "thread".
 92
 93  Thread-related topology information in the kernel:
 94
 95  - topology_core_cpumask():
 96
 97    The cpumask contains all online threads in the package to which a thread
 98    belongs.
 99
100    The number of online threads is also printed in /proc/cpuinfo "siblings."
101
102  - topology_sibling_mask():
103
104    The cpumask contains all online threads in the core to which a thread
105    belongs.
106
107   - topology_logical_package_id():
108
109    The logical package ID to which a thread belongs.
110
111   - topology_physical_package_id():
112
113    The physical package ID to which a thread belongs.
114
115   - topology_core_id();
116
117    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
118    "core_id."
119
120
121
122System topology examples
123
124Note:
125
126The alternative Linux CPU enumeration depends on how the BIOS enumerates the
127threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
128That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
129the same whether threads are enabled or not. That's merely an implementation
130detail and has no practical impact.
131
1321) Single Package, Single Core
133
134   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
135
1362) Single Package, Dual Core
137
138   a) One thread per core
139
140	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
141		    -> [core 1] -> [thread 0] -> Linux CPU 1
142
143   b) Two threads per core
144
145	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
146				-> [thread 1] -> Linux CPU 1
147		    -> [core 1] -> [thread 0] -> Linux CPU 2
148				-> [thread 1] -> Linux CPU 3
149
150      Alternative enumeration:
151
152	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
153				-> [thread 1] -> Linux CPU 2
154		    -> [core 1] -> [thread 0] -> Linux CPU 1
155				-> [thread 1] -> Linux CPU 3
156
157      AMD nomenclature for CMT systems:
158
159	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
160				     -> [Compute Unit Core 1] -> Linux CPU 1
161		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
162				     -> [Compute Unit Core 1] -> Linux CPU 3
163
1644) Dual Package, Dual Core
165
166   a) One thread per core
167
168	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
169		    -> [core 1] -> [thread 0] -> Linux CPU 1
170
171	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
172		    -> [core 1] -> [thread 0] -> Linux CPU 3
173
174   b) Two threads per core
175
176	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
177				-> [thread 1] -> Linux CPU 1
178		    -> [core 1] -> [thread 0] -> Linux CPU 2
179				-> [thread 1] -> Linux CPU 3
180
181	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
182				-> [thread 1] -> Linux CPU 5
183		    -> [core 1] -> [thread 0] -> Linux CPU 6
184				-> [thread 1] -> Linux CPU 7
185
186      Alternative enumeration:
187
188	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
189				-> [thread 1] -> Linux CPU 4
190		    -> [core 1] -> [thread 0] -> Linux CPU 1
191				-> [thread 1] -> Linux CPU 5
192
193	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
194				-> [thread 1] -> Linux CPU 6
195		    -> [core 1] -> [thread 0] -> Linux CPU 3
196				-> [thread 1] -> Linux CPU 7
197
198      AMD nomenclature for CMT systems:
199
200	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
201				     -> [Compute Unit Core 1] -> Linux CPU 1
202		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
203				     -> [Compute Unit Core 1] -> Linux CPU 3
204
205	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
206				     -> [Compute Unit Core 1] -> Linux CPU 5
207		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
208				     -> [Compute Unit Core 1] -> Linux CPU 7