Loading...
1.. SPDX-License-Identifier: GPL-2.0
2
3.. _fsverity:
4
5=======================================================
6fs-verity: read-only file-based authenticity protection
7=======================================================
8
9Introduction
10============
11
12fs-verity (``fs/verity/``) is a support layer that filesystems can
13hook into to support transparent integrity and authenticity protection
14of read-only files. Currently, it is supported by the ext4, f2fs, and
15btrfs filesystems. Like fscrypt, not too much filesystem-specific
16code is needed to support fs-verity.
17
18fs-verity is similar to `dm-verity
19<https://www.kernel.org/doc/Documentation/admin-guide/device-mapper/verity.rst>`_
20but works on files rather than block devices. On regular files on
21filesystems supporting fs-verity, userspace can execute an ioctl that
22causes the filesystem to build a Merkle tree for the file and persist
23it to a filesystem-specific location associated with the file.
24
25After this, the file is made readonly, and all reads from the file are
26automatically verified against the file's Merkle tree. Reads of any
27corrupted data, including mmap reads, will fail.
28
29Userspace can use another ioctl to retrieve the root hash (actually
30the "fs-verity file digest", which is a hash that includes the Merkle
31tree root hash) that fs-verity is enforcing for the file. This ioctl
32executes in constant time, regardless of the file size.
33
34fs-verity is essentially a way to hash a file in constant time,
35subject to the caveat that reads which would violate the hash will
36fail at runtime.
37
38Use cases
39=========
40
41By itself, fs-verity only provides integrity protection, i.e.
42detection of accidental (non-malicious) corruption.
43
44However, because fs-verity makes retrieving the file hash extremely
45efficient, it's primarily meant to be used as a tool to support
46authentication (detection of malicious modifications) or auditing
47(logging file hashes before use).
48
49A standard file hash could be used instead of fs-verity. However,
50this is inefficient if the file is large and only a small portion may
51be accessed. This is often the case for Android application package
52(APK) files, for example. These typically contain many translations,
53classes, and other resources that are infrequently or even never
54accessed on a particular device. It would be slow and wasteful to
55read and hash the entire file before starting the application.
56
57Unlike an ahead-of-time hash, fs-verity also re-verifies data each
58time it's paged in. This ensures that malicious disk firmware can't
59undetectably change the contents of the file at runtime.
60
61fs-verity does not replace or obsolete dm-verity. dm-verity should
62still be used on read-only filesystems. fs-verity is for files that
63must live on a read-write filesystem because they are independently
64updated and potentially user-installed, so dm-verity cannot be used.
65
66fs-verity does not mandate a particular scheme for authenticating its
67file hashes. (Similarly, dm-verity does not mandate a particular
68scheme for authenticating its block device root hashes.) Options for
69authenticating fs-verity file hashes include:
70
71- Trusted userspace code. Often, the userspace code that accesses
72 files can be trusted to authenticate them. Consider e.g. an
73 application that wants to authenticate data files before using them,
74 or an application loader that is part of the operating system (which
75 is already authenticated in a different way, such as by being loaded
76 from a read-only partition that uses dm-verity) and that wants to
77 authenticate applications before loading them. In these cases, this
78 trusted userspace code can authenticate a file's contents by
79 retrieving its fs-verity digest using `FS_IOC_MEASURE_VERITY`_, then
80 verifying a signature of it using any userspace cryptographic
81 library that supports digital signatures.
82
83- Integrity Measurement Architecture (IMA). IMA supports fs-verity
84 file digests as an alternative to its traditional full file digests.
85 "IMA appraisal" enforces that files contain a valid, matching
86 signature in their "security.ima" extended attribute, as controlled
87 by the IMA policy. For more information, see the IMA documentation.
88
89- Integrity Policy Enforcement (IPE). IPE supports enforcing access
90 control decisions based on immutable security properties of files,
91 including those protected by fs-verity's built-in signatures.
92 "IPE policy" specifically allows for the authorization of fs-verity
93 files using properties ``fsverity_digest`` for identifying
94 files by their verity digest, and ``fsverity_signature`` to authorize
95 files with a verified fs-verity's built-in signature. For
96 details on configuring IPE policies and understanding its operational
97 modes, please refer to :doc:`IPE admin guide </admin-guide/LSM/ipe>`.
98
99- Trusted userspace code in combination with `Built-in signature
100 verification`_. This approach should be used only with great care.
101
102User API
103========
104
105FS_IOC_ENABLE_VERITY
106--------------------
107
108The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file. It takes
109in a pointer to a struct fsverity_enable_arg, defined as
110follows::
111
112 struct fsverity_enable_arg {
113 __u32 version;
114 __u32 hash_algorithm;
115 __u32 block_size;
116 __u32 salt_size;
117 __u64 salt_ptr;
118 __u32 sig_size;
119 __u32 __reserved1;
120 __u64 sig_ptr;
121 __u64 __reserved2[11];
122 };
123
124This structure contains the parameters of the Merkle tree to build for
125the file. It must be initialized as follows:
126
127- ``version`` must be 1.
128- ``hash_algorithm`` must be the identifier for the hash algorithm to
129 use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256. See
130 ``include/uapi/linux/fsverity.h`` for the list of possible values.
131- ``block_size`` is the Merkle tree block size, in bytes. In Linux
132 v6.3 and later, this can be any power of 2 between (inclusively)
133 1024 and the minimum of the system page size and the filesystem
134 block size. In earlier versions, the page size was the only allowed
135 value.
136- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is
137 provided. The salt is a value that is prepended to every hashed
138 block; it can be used to personalize the hashing for a particular
139 file or device. Currently the maximum salt size is 32 bytes.
140- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is
141 provided.
142- ``sig_size`` is the size of the builtin signature in bytes, or 0 if no
143 builtin signature is provided. Currently the builtin signature is
144 (somewhat arbitrarily) limited to 16128 bytes.
145- ``sig_ptr`` is the pointer to the builtin signature, or NULL if no
146 builtin signature is provided. A builtin signature is only needed
147 if the `Built-in signature verification`_ feature is being used. It
148 is not needed for IMA appraisal, and it is not needed if the file
149 signature is being handled entirely in userspace.
150- All reserved fields must be zeroed.
151
152FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for
153the file and persist it to a filesystem-specific location associated
154with the file, then mark the file as a verity file. This ioctl may
155take a long time to execute on large files, and it is interruptible by
156fatal signals.
157
158FS_IOC_ENABLE_VERITY checks for write access to the inode. However,
159it must be executed on an O_RDONLY file descriptor and no processes
160can have the file open for writing. Attempts to open the file for
161writing while this ioctl is executing will fail with ETXTBSY. (This
162is necessary to guarantee that no writable file descriptors will exist
163after verity is enabled, and to guarantee that the file's contents are
164stable while the Merkle tree is being built over it.)
165
166On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a
167verity file. On failure (including the case of interruption by a
168fatal signal), no changes are made to the file.
169
170FS_IOC_ENABLE_VERITY can fail with the following errors:
171
172- ``EACCES``: the process does not have write access to the file
173- ``EBADMSG``: the builtin signature is malformed
174- ``EBUSY``: this ioctl is already running on the file
175- ``EEXIST``: the file already has verity enabled
176- ``EFAULT``: the caller provided inaccessible memory
177- ``EFBIG``: the file is too large to enable verity on
178- ``EINTR``: the operation was interrupted by a fatal signal
179- ``EINVAL``: unsupported version, hash algorithm, or block size; or
180 reserved bits are set; or the file descriptor refers to neither a
181 regular file nor a directory.
182- ``EISDIR``: the file descriptor refers to a directory
183- ``EKEYREJECTED``: the builtin signature doesn't match the file
184- ``EMSGSIZE``: the salt or builtin signature is too long
185- ``ENOKEY``: the ".fs-verity" keyring doesn't contain the certificate
186 needed to verify the builtin signature
187- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not
188 available in the kernel's crypto API as currently configured (e.g.
189 for SHA-512, missing CONFIG_CRYPTO_SHA512).
190- ``ENOTTY``: this type of filesystem does not implement fs-verity
191- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
192 support; or the filesystem superblock has not had the 'verity'
193 feature enabled on it; or the filesystem does not support fs-verity
194 on this file. (See `Filesystem support`_.)
195- ``EPERM``: the file is append-only; or, a builtin signature is
196 required and one was not provided.
197- ``EROFS``: the filesystem is read-only
198- ``ETXTBSY``: someone has the file open for writing. This can be the
199 caller's file descriptor, another open file descriptor, or the file
200 reference held by a writable memory map.
201
202FS_IOC_MEASURE_VERITY
203---------------------
204
205The FS_IOC_MEASURE_VERITY ioctl retrieves the digest of a verity file.
206The fs-verity file digest is a cryptographic digest that identifies
207the file contents that are being enforced on reads; it is computed via
208a Merkle tree and is different from a traditional full-file digest.
209
210This ioctl takes in a pointer to a variable-length structure::
211
212 struct fsverity_digest {
213 __u16 digest_algorithm;
214 __u16 digest_size; /* input/output */
215 __u8 digest[];
216 };
217
218``digest_size`` is an input/output field. On input, it must be
219initialized to the number of bytes allocated for the variable-length
220``digest`` field.
221
222On success, 0 is returned and the kernel fills in the structure as
223follows:
224
225- ``digest_algorithm`` will be the hash algorithm used for the file
226 digest. It will match ``fsverity_enable_arg::hash_algorithm``.
227- ``digest_size`` will be the size of the digest in bytes, e.g. 32
228 for SHA-256. (This can be redundant with ``digest_algorithm``.)
229- ``digest`` will be the actual bytes of the digest.
230
231FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time,
232regardless of the size of the file.
233
234FS_IOC_MEASURE_VERITY can fail with the following errors:
235
236- ``EFAULT``: the caller provided inaccessible memory
237- ``ENODATA``: the file is not a verity file
238- ``ENOTTY``: this type of filesystem does not implement fs-verity
239- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
240 support, or the filesystem superblock has not had the 'verity'
241 feature enabled on it. (See `Filesystem support`_.)
242- ``EOVERFLOW``: the digest is longer than the specified
243 ``digest_size`` bytes. Try providing a larger buffer.
244
245FS_IOC_READ_VERITY_METADATA
246---------------------------
247
248The FS_IOC_READ_VERITY_METADATA ioctl reads verity metadata from a
249verity file. This ioctl is available since Linux v5.12.
250
251This ioctl allows writing a server program that takes a verity file
252and serves it to a client program, such that the client can do its own
253fs-verity compatible verification of the file. This only makes sense
254if the client doesn't trust the server and if the server needs to
255provide the storage for the client.
256
257This is a fairly specialized use case, and most fs-verity users won't
258need this ioctl.
259
260This ioctl takes in a pointer to the following structure::
261
262 #define FS_VERITY_METADATA_TYPE_MERKLE_TREE 1
263 #define FS_VERITY_METADATA_TYPE_DESCRIPTOR 2
264 #define FS_VERITY_METADATA_TYPE_SIGNATURE 3
265
266 struct fsverity_read_metadata_arg {
267 __u64 metadata_type;
268 __u64 offset;
269 __u64 length;
270 __u64 buf_ptr;
271 __u64 __reserved;
272 };
273
274``metadata_type`` specifies the type of metadata to read:
275
276- ``FS_VERITY_METADATA_TYPE_MERKLE_TREE`` reads the blocks of the
277 Merkle tree. The blocks are returned in order from the root level
278 to the leaf level. Within each level, the blocks are returned in
279 the same order that their hashes are themselves hashed.
280 See `Merkle tree`_ for more information.
281
282- ``FS_VERITY_METADATA_TYPE_DESCRIPTOR`` reads the fs-verity
283 descriptor. See `fs-verity descriptor`_.
284
285- ``FS_VERITY_METADATA_TYPE_SIGNATURE`` reads the builtin signature
286 which was passed to FS_IOC_ENABLE_VERITY, if any. See `Built-in
287 signature verification`_.
288
289The semantics are similar to those of ``pread()``. ``offset``
290specifies the offset in bytes into the metadata item to read from, and
291``length`` specifies the maximum number of bytes to read from the
292metadata item. ``buf_ptr`` is the pointer to the buffer to read into,
293cast to a 64-bit integer. ``__reserved`` must be 0. On success, the
294number of bytes read is returned. 0 is returned at the end of the
295metadata item. The returned length may be less than ``length``, for
296example if the ioctl is interrupted.
297
298The metadata returned by FS_IOC_READ_VERITY_METADATA isn't guaranteed
299to be authenticated against the file digest that would be returned by
300`FS_IOC_MEASURE_VERITY`_, as the metadata is expected to be used to
301implement fs-verity compatible verification anyway (though absent a
302malicious disk, the metadata will indeed match). E.g. to implement
303this ioctl, the filesystem is allowed to just read the Merkle tree
304blocks from disk without actually verifying the path to the root node.
305
306FS_IOC_READ_VERITY_METADATA can fail with the following errors:
307
308- ``EFAULT``: the caller provided inaccessible memory
309- ``EINTR``: the ioctl was interrupted before any data was read
310- ``EINVAL``: reserved fields were set, or ``offset + length``
311 overflowed
312- ``ENODATA``: the file is not a verity file, or
313 FS_VERITY_METADATA_TYPE_SIGNATURE was requested but the file doesn't
314 have a builtin signature
315- ``ENOTTY``: this type of filesystem does not implement fs-verity, or
316 this ioctl is not yet implemented on it
317- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
318 support, or the filesystem superblock has not had the 'verity'
319 feature enabled on it. (See `Filesystem support`_.)
320
321FS_IOC_GETFLAGS
322---------------
323
324The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity)
325can also be used to check whether a file has fs-verity enabled or not.
326To do so, check for FS_VERITY_FL (0x00100000) in the returned flags.
327
328The verity flag is not settable via FS_IOC_SETFLAGS. You must use
329FS_IOC_ENABLE_VERITY instead, since parameters must be provided.
330
331statx
332-----
333
334Since Linux v5.5, the statx() system call sets STATX_ATTR_VERITY if
335the file has fs-verity enabled. This can perform better than
336FS_IOC_GETFLAGS and FS_IOC_MEASURE_VERITY because it doesn't require
337opening the file, and opening verity files can be expensive.
338
339.. _accessing_verity_files:
340
341Accessing verity files
342======================
343
344Applications can transparently access a verity file just like a
345non-verity one, with the following exceptions:
346
347- Verity files are readonly. They cannot be opened for writing or
348 truncate()d, even if the file mode bits allow it. Attempts to do
349 one of these things will fail with EPERM. However, changes to
350 metadata such as owner, mode, timestamps, and xattrs are still
351 allowed, since these are not measured by fs-verity. Verity files
352 can also still be renamed, deleted, and linked to.
353
354- Direct I/O is not supported on verity files. Attempts to use direct
355 I/O on such files will fall back to buffered I/O.
356
357- DAX (Direct Access) is not supported on verity files, because this
358 would circumvent the data verification.
359
360- Reads of data that doesn't match the verity Merkle tree will fail
361 with EIO (for read()) or SIGBUS (for mmap() reads).
362
363- If the sysctl "fs.verity.require_signatures" is set to 1 and the
364 file is not signed by a key in the ".fs-verity" keyring, then
365 opening the file will fail. See `Built-in signature verification`_.
366
367Direct access to the Merkle tree is not supported. Therefore, if a
368verity file is copied, or is backed up and restored, then it will lose
369its "verity"-ness. fs-verity is primarily meant for files like
370executables that are managed by a package manager.
371
372File digest computation
373=======================
374
375This section describes how fs-verity hashes the file contents using a
376Merkle tree to produce the digest which cryptographically identifies
377the file contents. This algorithm is the same for all filesystems
378that support fs-verity.
379
380Userspace only needs to be aware of this algorithm if it needs to
381compute fs-verity file digests itself, e.g. in order to sign files.
382
383.. _fsverity_merkle_tree:
384
385Merkle tree
386-----------
387
388The file contents is divided into blocks, where the block size is
389configurable but is usually 4096 bytes. The end of the last block is
390zero-padded if needed. Each block is then hashed, producing the first
391level of hashes. Then, the hashes in this first level are grouped
392into 'blocksize'-byte blocks (zero-padding the ends as needed) and
393these blocks are hashed, producing the second level of hashes. This
394proceeds up the tree until only a single block remains. The hash of
395this block is the "Merkle tree root hash".
396
397If the file fits in one block and is nonempty, then the "Merkle tree
398root hash" is simply the hash of the single data block. If the file
399is empty, then the "Merkle tree root hash" is all zeroes.
400
401The "blocks" here are not necessarily the same as "filesystem blocks".
402
403If a salt was specified, then it's zero-padded to the closest multiple
404of the input size of the hash algorithm's compression function, e.g.
40564 bytes for SHA-256 or 128 bytes for SHA-512. The padded salt is
406prepended to every data or Merkle tree block that is hashed.
407
408The purpose of the block padding is to cause every hash to be taken
409over the same amount of data, which simplifies the implementation and
410keeps open more possibilities for hardware acceleration. The purpose
411of the salt padding is to make the salting "free" when the salted hash
412state is precomputed, then imported for each hash.
413
414Example: in the recommended configuration of SHA-256 and 4K blocks,
415128 hash values fit in each block. Thus, each level of the Merkle
416tree is approximately 128 times smaller than the previous, and for
417large files the Merkle tree's size converges to approximately 1/127 of
418the original file size. However, for small files, the padding is
419significant, making the space overhead proportionally more.
420
421.. _fsverity_descriptor:
422
423fs-verity descriptor
424--------------------
425
426By itself, the Merkle tree root hash is ambiguous. For example, it
427can't a distinguish a large file from a small second file whose data
428is exactly the top-level hash block of the first file. Ambiguities
429also arise from the convention of padding to the next block boundary.
430
431To solve this problem, the fs-verity file digest is actually computed
432as a hash of the following structure, which contains the Merkle tree
433root hash as well as other fields such as the file size::
434
435 struct fsverity_descriptor {
436 __u8 version; /* must be 1 */
437 __u8 hash_algorithm; /* Merkle tree hash algorithm */
438 __u8 log_blocksize; /* log2 of size of data and tree blocks */
439 __u8 salt_size; /* size of salt in bytes; 0 if none */
440 __le32 __reserved_0x04; /* must be 0 */
441 __le64 data_size; /* size of file the Merkle tree is built over */
442 __u8 root_hash[64]; /* Merkle tree root hash */
443 __u8 salt[32]; /* salt prepended to each hashed block */
444 __u8 __reserved[144]; /* must be 0's */
445 };
446
447Built-in signature verification
448===============================
449
450CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y adds supports for in-kernel
451verification of fs-verity builtin signatures.
452
453**IMPORTANT**! Please take great care before using this feature.
454It is not the only way to do signatures with fs-verity, and the
455alternatives (such as userspace signature verification, and IMA
456appraisal) can be much better. It's also easy to fall into a trap
457of thinking this feature solves more problems than it actually does.
458
459Enabling this option adds the following:
460
4611. At boot time, the kernel creates a keyring named ".fs-verity". The
462 root user can add trusted X.509 certificates to this keyring using
463 the add_key() system call.
464
4652. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted
466 detached signature in DER format of the file's fs-verity digest.
467 On success, the ioctl persists the signature alongside the Merkle
468 tree. Then, any time the file is opened, the kernel verifies the
469 file's actual digest against this signature, using the certificates
470 in the ".fs-verity" keyring. This verification happens as long as the
471 file's signature exists, regardless of the state of the sysctl variable
472 "fs.verity.require_signatures" described in the next item. The IPE LSM
473 relies on this behavior to recognize and label fsverity files
474 that contain a verified built-in fsverity signature.
475
4763. A new sysctl "fs.verity.require_signatures" is made available.
477 When set to 1, the kernel requires that all verity files have a
478 correctly signed digest as described in (2).
479
480The data that the signature as described in (2) must be a signature of
481is the fs-verity file digest in the following format::
482
483 struct fsverity_formatted_digest {
484 char magic[8]; /* must be "FSVerity" */
485 __le16 digest_algorithm;
486 __le16 digest_size;
487 __u8 digest[];
488 };
489
490That's it. It should be emphasized again that fs-verity builtin
491signatures are not the only way to do signatures with fs-verity. See
492`Use cases`_ for an overview of ways in which fs-verity can be used.
493fs-verity builtin signatures have some major limitations that should
494be carefully considered before using them:
495
496- Builtin signature verification does *not* make the kernel enforce
497 that any files actually have fs-verity enabled. Thus, it is not a
498 complete authentication policy. Currently, if it is used, one
499 way to complete the authentication policy is for trusted userspace
500 code to explicitly check whether files have fs-verity enabled with a
501 signature before they are accessed. (With
502 fs.verity.require_signatures=1, just checking whether fs-verity is
503 enabled suffices.) But, in this case the trusted userspace code
504 could just store the signature alongside the file and verify it
505 itself using a cryptographic library, instead of using this feature.
506
507- Another approach is to utilize fs-verity builtin signature
508 verification in conjunction with the IPE LSM, which supports defining
509 a kernel-enforced, system-wide authentication policy that allows only
510 files with a verified fs-verity builtin signature to perform certain
511 operations, such as execution. Note that IPE doesn't require
512 fs.verity.require_signatures=1.
513 Please refer to :doc:`IPE admin guide </admin-guide/LSM/ipe>` for
514 more details.
515
516- A file's builtin signature can only be set at the same time that
517 fs-verity is being enabled on the file. Changing or deleting the
518 builtin signature later requires re-creating the file.
519
520- Builtin signature verification uses the same set of public keys for
521 all fs-verity enabled files on the system. Different keys cannot be
522 trusted for different files; each key is all or nothing.
523
524- The sysctl fs.verity.require_signatures applies system-wide.
525 Setting it to 1 only works when all users of fs-verity on the system
526 agree that it should be set to 1. This limitation can prevent
527 fs-verity from being used in cases where it would be helpful.
528
529- Builtin signature verification can only use signature algorithms
530 that are supported by the kernel. For example, the kernel does not
531 yet support Ed25519, even though this is often the signature
532 algorithm that is recommended for new cryptographic designs.
533
534- fs-verity builtin signatures are in PKCS#7 format, and the public
535 keys are in X.509 format. These formats are commonly used,
536 including by some other kernel features (which is why the fs-verity
537 builtin signatures use them), and are very feature rich.
538 Unfortunately, history has shown that code that parses and handles
539 these formats (which are from the 1990s and are based on ASN.1)
540 often has vulnerabilities as a result of their complexity. This
541 complexity is not inherent to the cryptography itself.
542
543 fs-verity users who do not need advanced features of X.509 and
544 PKCS#7 should strongly consider using simpler formats, such as plain
545 Ed25519 keys and signatures, and verifying signatures in userspace.
546
547 fs-verity users who choose to use X.509 and PKCS#7 anyway should
548 still consider that verifying those signatures in userspace is more
549 flexible (for other reasons mentioned earlier in this document) and
550 eliminates the need to enable CONFIG_FS_VERITY_BUILTIN_SIGNATURES
551 and its associated increase in kernel attack surface. In some cases
552 it can even be necessary, since advanced X.509 and PKCS#7 features
553 do not always work as intended with the kernel. For example, the
554 kernel does not check X.509 certificate validity times.
555
556 Note: IMA appraisal, which supports fs-verity, does not use PKCS#7
557 for its signatures, so it partially avoids the issues discussed
558 here. IMA appraisal does use X.509.
559
560Filesystem support
561==================
562
563fs-verity is supported by several filesystems, described below. The
564CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity on
565any of these filesystems.
566
567``include/linux/fsverity.h`` declares the interface between the
568``fs/verity/`` support layer and filesystems. Briefly, filesystems
569must provide an ``fsverity_operations`` structure that provides
570methods to read and write the verity metadata to a filesystem-specific
571location, including the Merkle tree blocks and
572``fsverity_descriptor``. Filesystems must also call functions in
573``fs/verity/`` at certain times, such as when a file is opened or when
574pages have been read into the pagecache. (See `Verifying data`_.)
575
576ext4
577----
578
579ext4 supports fs-verity since Linux v5.4 and e2fsprogs v1.45.2.
580
581To create verity files on an ext4 filesystem, the filesystem must have
582been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on
583it. "verity" is an RO_COMPAT filesystem feature, so once set, old
584kernels will only be able to mount the filesystem readonly, and old
585versions of e2fsck will be unable to check the filesystem.
586
587Originally, an ext4 filesystem with the "verity" feature could only be
588mounted when its block size was equal to the system page size
589(typically 4096 bytes). In Linux v6.3, this limitation was removed.
590
591ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files. It
592can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared.
593
594ext4 also supports encryption, which can be used simultaneously with
595fs-verity. In this case, the plaintext data is verified rather than
596the ciphertext. This is necessary in order to make the fs-verity file
597digest meaningful, since every file is encrypted differently.
598
599ext4 stores the verity metadata (Merkle tree and fsverity_descriptor)
600past the end of the file, starting at the first 64K boundary beyond
601i_size. This approach works because (a) verity files are readonly,
602and (b) pages fully beyond i_size aren't visible to userspace but can
603be read/written internally by ext4 with only some relatively small
604changes to ext4. This approach avoids having to depend on the
605EA_INODE feature and on rearchitecturing ext4's xattr support to
606support paging multi-gigabyte xattrs into memory, and to support
607encrypting xattrs. Note that the verity metadata *must* be encrypted
608when the file is, since it contains hashes of the plaintext data.
609
610ext4 only allows verity on extent-based files.
611
612f2fs
613----
614
615f2fs supports fs-verity since Linux v5.4 and f2fs-tools v1.11.0.
616
617To create verity files on an f2fs filesystem, the filesystem must have
618been formatted with ``-O verity``.
619
620f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files.
621It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be
622cleared.
623
624Like ext4, f2fs stores the verity metadata (Merkle tree and
625fsverity_descriptor) past the end of the file, starting at the first
62664K boundary beyond i_size. See explanation for ext4 above.
627Moreover, f2fs supports at most 4096 bytes of xattr entries per inode
628which usually wouldn't be enough for even a single Merkle tree block.
629
630f2fs doesn't support enabling verity on files that currently have
631atomic or volatile writes pending.
632
633btrfs
634-----
635
636btrfs supports fs-verity since Linux v5.15. Verity-enabled inodes are
637marked with a RO_COMPAT inode flag, and the verity metadata is stored
638in separate btree items.
639
640Implementation details
641======================
642
643Verifying data
644--------------
645
646fs-verity ensures that all reads of a verity file's data are verified,
647regardless of which syscall is used to do the read (e.g. mmap(),
648read(), pread()) and regardless of whether it's the first read or a
649later read (unless the later read can return cached data that was
650already verified). Below, we describe how filesystems implement this.
651
652Pagecache
653~~~~~~~~~
654
655For filesystems using Linux's pagecache, the ``->read_folio()`` and
656``->readahead()`` methods must be modified to verify folios before
657they are marked Uptodate. Merely hooking ``->read_iter()`` would be
658insufficient, since ``->read_iter()`` is not used for memory maps.
659
660Therefore, fs/verity/ provides the function fsverity_verify_blocks()
661which verifies data that has been read into the pagecache of a verity
662inode. The containing folio must still be locked and not Uptodate, so
663it's not yet readable by userspace. As needed to do the verification,
664fsverity_verify_blocks() will call back into the filesystem to read
665hash blocks via fsverity_operations::read_merkle_tree_page().
666
667fsverity_verify_blocks() returns false if verification failed; in this
668case, the filesystem must not set the folio Uptodate. Following this,
669as per the usual Linux pagecache behavior, attempts by userspace to
670read() from the part of the file containing the folio will fail with
671EIO, and accesses to the folio within a memory map will raise SIGBUS.
672
673In principle, verifying a data block requires verifying the entire
674path in the Merkle tree from the data block to the root hash.
675However, for efficiency the filesystem may cache the hash blocks.
676Therefore, fsverity_verify_blocks() only ascends the tree reading hash
677blocks until an already-verified hash block is seen. It then verifies
678the path to that block.
679
680This optimization, which is also used by dm-verity, results in
681excellent sequential read performance. This is because usually (e.g.
682127 in 128 times for 4K blocks and SHA-256) the hash block from the
683bottom level of the tree will already be cached and checked from
684reading a previous data block. However, random reads perform worse.
685
686Block device based filesystems
687~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
688
689Block device based filesystems (e.g. ext4 and f2fs) in Linux also use
690the pagecache, so the above subsection applies too. However, they
691also usually read many data blocks from a file at once, grouped into a
692structure called a "bio". To make it easier for these types of
693filesystems to support fs-verity, fs/verity/ also provides a function
694fsverity_verify_bio() which verifies all data blocks in a bio.
695
696ext4 and f2fs also support encryption. If a verity file is also
697encrypted, the data must be decrypted before being verified. To
698support this, these filesystems allocate a "post-read context" for
699each bio and store it in ``->bi_private``::
700
701 struct bio_post_read_ctx {
702 struct bio *bio;
703 struct work_struct work;
704 unsigned int cur_step;
705 unsigned int enabled_steps;
706 };
707
708``enabled_steps`` is a bitmask that specifies whether decryption,
709verity, or both is enabled. After the bio completes, for each needed
710postprocessing step the filesystem enqueues the bio_post_read_ctx on a
711workqueue, and then the workqueue work does the decryption or
712verification. Finally, folios where no decryption or verity error
713occurred are marked Uptodate, and the folios are unlocked.
714
715On many filesystems, files can contain holes. Normally,
716``->readahead()`` simply zeroes hole blocks and considers the
717corresponding data to be up-to-date; no bios are issued. To prevent
718this case from bypassing fs-verity, filesystems use
719fsverity_verify_blocks() to verify hole blocks.
720
721Filesystems also disable direct I/O on verity files, since otherwise
722direct I/O would bypass fs-verity.
723
724Userspace utility
725=================
726
727This document focuses on the kernel, but a userspace utility for
728fs-verity can be found at:
729
730 https://git.kernel.org/pub/scm/fs/fsverity/fsverity-utils.git
731
732See the README.md file in the fsverity-utils source tree for details,
733including examples of setting up fs-verity protected files.
734
735Tests
736=====
737
738To test fs-verity, use xfstests. For example, using `kvm-xfstests
739<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::
740
741 kvm-xfstests -c ext4,f2fs,btrfs -g verity
742
743FAQ
744===
745
746This section answers frequently asked questions about fs-verity that
747weren't already directly answered in other parts of this document.
748
749:Q: Why isn't fs-verity part of IMA?
750:A: fs-verity and IMA (Integrity Measurement Architecture) have
751 different focuses. fs-verity is a filesystem-level mechanism for
752 hashing individual files using a Merkle tree. In contrast, IMA
753 specifies a system-wide policy that specifies which files are
754 hashed and what to do with those hashes, such as log them,
755 authenticate them, or add them to a measurement list.
756
757 IMA supports the fs-verity hashing mechanism as an alternative
758 to full file hashes, for those who want the performance and
759 security benefits of the Merkle tree based hash. However, it
760 doesn't make sense to force all uses of fs-verity to be through
761 IMA. fs-verity already meets many users' needs even as a
762 standalone filesystem feature, and it's testable like other
763 filesystem features e.g. with xfstests.
764
765:Q: Isn't fs-verity useless because the attacker can just modify the
766 hashes in the Merkle tree, which is stored on-disk?
767:A: To verify the authenticity of an fs-verity file you must verify
768 the authenticity of the "fs-verity file digest", which
769 incorporates the root hash of the Merkle tree. See `Use cases`_.
770
771:Q: Isn't fs-verity useless because the attacker can just replace a
772 verity file with a non-verity one?
773:A: See `Use cases`_. In the initial use case, it's really trusted
774 userspace code that authenticates the files; fs-verity is just a
775 tool to do this job efficiently and securely. The trusted
776 userspace code will consider non-verity files to be inauthentic.
777
778:Q: Why does the Merkle tree need to be stored on-disk? Couldn't you
779 store just the root hash?
780:A: If the Merkle tree wasn't stored on-disk, then you'd have to
781 compute the entire tree when the file is first accessed, even if
782 just one byte is being read. This is a fundamental consequence of
783 how Merkle tree hashing works. To verify a leaf node, you need to
784 verify the whole path to the root hash, including the root node
785 (the thing which the root hash is a hash of). But if the root
786 node isn't stored on-disk, you have to compute it by hashing its
787 children, and so on until you've actually hashed the entire file.
788
789 That defeats most of the point of doing a Merkle tree-based hash,
790 since if you have to hash the whole file ahead of time anyway,
791 then you could simply do sha256(file) instead. That would be much
792 simpler, and a bit faster too.
793
794 It's true that an in-memory Merkle tree could still provide the
795 advantage of verification on every read rather than just on the
796 first read. However, it would be inefficient because every time a
797 hash page gets evicted (you can't pin the entire Merkle tree into
798 memory, since it may be very large), in order to restore it you
799 again need to hash everything below it in the tree. This again
800 defeats most of the point of doing a Merkle tree-based hash, since
801 a single block read could trigger re-hashing gigabytes of data.
802
803:Q: But couldn't you store just the leaf nodes and compute the rest?
804:A: See previous answer; this really just moves up one level, since
805 one could alternatively interpret the data blocks as being the
806 leaf nodes of the Merkle tree. It's true that the tree can be
807 computed much faster if the leaf level is stored rather than just
808 the data, but that's only because each level is less than 1% the
809 size of the level below (assuming the recommended settings of
810 SHA-256 and 4K blocks). For the exact same reason, by storing
811 "just the leaf nodes" you'd already be storing over 99% of the
812 tree, so you might as well simply store the whole tree.
813
814:Q: Can the Merkle tree be built ahead of time, e.g. distributed as
815 part of a package that is installed to many computers?
816:A: This isn't currently supported. It was part of the original
817 design, but was removed to simplify the kernel UAPI and because it
818 wasn't a critical use case. Files are usually installed once and
819 used many times, and cryptographic hashing is somewhat fast on
820 most modern processors.
821
822:Q: Why doesn't fs-verity support writes?
823:A: Write support would be very difficult and would require a
824 completely different design, so it's well outside the scope of
825 fs-verity. Write support would require:
826
827 - A way to maintain consistency between the data and hashes,
828 including all levels of hashes, since corruption after a crash
829 (especially of potentially the entire file!) is unacceptable.
830 The main options for solving this are data journalling,
831 copy-on-write, and log-structured volume. But it's very hard to
832 retrofit existing filesystems with new consistency mechanisms.
833 Data journalling is available on ext4, but is very slow.
834
835 - Rebuilding the Merkle tree after every write, which would be
836 extremely inefficient. Alternatively, a different authenticated
837 dictionary structure such as an "authenticated skiplist" could
838 be used. However, this would be far more complex.
839
840 Compare it to dm-verity vs. dm-integrity. dm-verity is very
841 simple: the kernel just verifies read-only data against a
842 read-only Merkle tree. In contrast, dm-integrity supports writes
843 but is slow, is much more complex, and doesn't actually support
844 full-device authentication since it authenticates each sector
845 independently, i.e. there is no "root hash". It doesn't really
846 make sense for the same device-mapper target to support these two
847 very different cases; the same applies to fs-verity.
848
849:Q: Since verity files are immutable, why isn't the immutable bit set?
850:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a
851 specific set of semantics which not only make the file contents
852 read-only, but also prevent the file from being deleted, renamed,
853 linked to, or having its owner or mode changed. These extra
854 properties are unwanted for fs-verity, so reusing the immutable
855 bit isn't appropriate.
856
857:Q: Why does the API use ioctls instead of setxattr() and getxattr()?
858:A: Abusing the xattr interface for basically arbitrary syscalls is
859 heavily frowned upon by most of the Linux filesystem developers.
860 An xattr should really just be an xattr on-disk, not an API to
861 e.g. magically trigger construction of a Merkle tree.
862
863:Q: Does fs-verity support remote filesystems?
864:A: So far all filesystems that have implemented fs-verity support are
865 local filesystems, but in principle any filesystem that can store
866 per-file verity metadata can support fs-verity, regardless of
867 whether it's local or remote. Some filesystems may have fewer
868 options of where to store the verity metadata; one possibility is
869 to store it past the end of the file and "hide" it from userspace
870 by manipulating i_size. The data verification functions provided
871 by ``fs/verity/`` also assume that the filesystem uses the Linux
872 pagecache, but both local and remote filesystems normally do so.
873
874:Q: Why is anything filesystem-specific at all? Shouldn't fs-verity
875 be implemented entirely at the VFS level?
876:A: There are many reasons why this is not possible or would be very
877 difficult, including the following:
878
879 - To prevent bypassing verification, folios must not be marked
880 Uptodate until they've been verified. Currently, each
881 filesystem is responsible for marking folios Uptodate via
882 ``->readahead()``. Therefore, currently it's not possible for
883 the VFS to do the verification on its own. Changing this would
884 require significant changes to the VFS and all filesystems.
885
886 - It would require defining a filesystem-independent way to store
887 the verity metadata. Extended attributes don't work for this
888 because (a) the Merkle tree may be gigabytes, but many
889 filesystems assume that all xattrs fit into a single 4K
890 filesystem block, and (b) ext4 and f2fs encryption doesn't
891 encrypt xattrs, yet the Merkle tree *must* be encrypted when the
892 file contents are, because it stores hashes of the plaintext
893 file contents.
894
895 So the verity metadata would have to be stored in an actual
896 file. Using a separate file would be very ugly, since the
897 metadata is fundamentally part of the file to be protected, and
898 it could cause problems where users could delete the real file
899 but not the metadata file or vice versa. On the other hand,
900 having it be in the same file would break applications unless
901 filesystems' notion of i_size were divorced from the VFS's,
902 which would be complex and require changes to all filesystems.
903
904 - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's
905 transaction mechanism so that either the file ends up with
906 verity enabled, or no changes were made. Allowing intermediate
907 states to occur after a crash may cause problems.
1.. SPDX-License-Identifier: GPL-2.0
2
3.. _fsverity:
4
5=======================================================
6fs-verity: read-only file-based authenticity protection
7=======================================================
8
9Introduction
10============
11
12fs-verity (``fs/verity/``) is a support layer that filesystems can
13hook into to support transparent integrity and authenticity protection
14of read-only files. Currently, it is supported by the ext4 and f2fs
15filesystems. Like fscrypt, not too much filesystem-specific code is
16needed to support fs-verity.
17
18fs-verity is similar to `dm-verity
19<https://www.kernel.org/doc/Documentation/device-mapper/verity.txt>`_
20but works on files rather than block devices. On regular files on
21filesystems supporting fs-verity, userspace can execute an ioctl that
22causes the filesystem to build a Merkle tree for the file and persist
23it to a filesystem-specific location associated with the file.
24
25After this, the file is made readonly, and all reads from the file are
26automatically verified against the file's Merkle tree. Reads of any
27corrupted data, including mmap reads, will fail.
28
29Userspace can use another ioctl to retrieve the root hash (actually
30the "file measurement", which is a hash that includes the root hash)
31that fs-verity is enforcing for the file. This ioctl executes in
32constant time, regardless of the file size.
33
34fs-verity is essentially a way to hash a file in constant time,
35subject to the caveat that reads which would violate the hash will
36fail at runtime.
37
38Use cases
39=========
40
41By itself, the base fs-verity feature only provides integrity
42protection, i.e. detection of accidental (non-malicious) corruption.
43
44However, because fs-verity makes retrieving the file hash extremely
45efficient, it's primarily meant to be used as a tool to support
46authentication (detection of malicious modifications) or auditing
47(logging file hashes before use).
48
49Trusted userspace code (e.g. operating system code running on a
50read-only partition that is itself authenticated by dm-verity) can
51authenticate the contents of an fs-verity file by using the
52`FS_IOC_MEASURE_VERITY`_ ioctl to retrieve its hash, then verifying a
53digital signature of it.
54
55A standard file hash could be used instead of fs-verity. However,
56this is inefficient if the file is large and only a small portion may
57be accessed. This is often the case for Android application package
58(APK) files, for example. These typically contain many translations,
59classes, and other resources that are infrequently or even never
60accessed on a particular device. It would be slow and wasteful to
61read and hash the entire file before starting the application.
62
63Unlike an ahead-of-time hash, fs-verity also re-verifies data each
64time it's paged in. This ensures that malicious disk firmware can't
65undetectably change the contents of the file at runtime.
66
67fs-verity does not replace or obsolete dm-verity. dm-verity should
68still be used on read-only filesystems. fs-verity is for files that
69must live on a read-write filesystem because they are independently
70updated and potentially user-installed, so dm-verity cannot be used.
71
72The base fs-verity feature is a hashing mechanism only; actually
73authenticating the files is up to userspace. However, to meet some
74users' needs, fs-verity optionally supports a simple signature
75verification mechanism where users can configure the kernel to require
76that all fs-verity files be signed by a key loaded into a keyring; see
77`Built-in signature verification`_. Support for fs-verity file hashes
78in IMA (Integrity Measurement Architecture) policies is also planned.
79
80User API
81========
82
83FS_IOC_ENABLE_VERITY
84--------------------
85
86The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file. It takes
87in a pointer to a :c:type:`struct fsverity_enable_arg`, defined as
88follows::
89
90 struct fsverity_enable_arg {
91 __u32 version;
92 __u32 hash_algorithm;
93 __u32 block_size;
94 __u32 salt_size;
95 __u64 salt_ptr;
96 __u32 sig_size;
97 __u32 __reserved1;
98 __u64 sig_ptr;
99 __u64 __reserved2[11];
100 };
101
102This structure contains the parameters of the Merkle tree to build for
103the file, and optionally contains a signature. It must be initialized
104as follows:
105
106- ``version`` must be 1.
107- ``hash_algorithm`` must be the identifier for the hash algorithm to
108 use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256. See
109 ``include/uapi/linux/fsverity.h`` for the list of possible values.
110- ``block_size`` must be the Merkle tree block size. Currently, this
111 must be equal to the system page size, which is usually 4096 bytes.
112 Other sizes may be supported in the future. This value is not
113 necessarily the same as the filesystem block size.
114- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is
115 provided. The salt is a value that is prepended to every hashed
116 block; it can be used to personalize the hashing for a particular
117 file or device. Currently the maximum salt size is 32 bytes.
118- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is
119 provided.
120- ``sig_size`` is the size of the signature in bytes, or 0 if no
121 signature is provided. Currently the signature is (somewhat
122 arbitrarily) limited to 16128 bytes. See `Built-in signature
123 verification`_ for more information.
124- ``sig_ptr`` is the pointer to the signature, or NULL if no
125 signature is provided.
126- All reserved fields must be zeroed.
127
128FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for
129the file and persist it to a filesystem-specific location associated
130with the file, then mark the file as a verity file. This ioctl may
131take a long time to execute on large files, and it is interruptible by
132fatal signals.
133
134FS_IOC_ENABLE_VERITY checks for write access to the inode. However,
135it must be executed on an O_RDONLY file descriptor and no processes
136can have the file open for writing. Attempts to open the file for
137writing while this ioctl is executing will fail with ETXTBSY. (This
138is necessary to guarantee that no writable file descriptors will exist
139after verity is enabled, and to guarantee that the file's contents are
140stable while the Merkle tree is being built over it.)
141
142On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a
143verity file. On failure (including the case of interruption by a
144fatal signal), no changes are made to the file.
145
146FS_IOC_ENABLE_VERITY can fail with the following errors:
147
148- ``EACCES``: the process does not have write access to the file
149- ``EBADMSG``: the signature is malformed
150- ``EBUSY``: this ioctl is already running on the file
151- ``EEXIST``: the file already has verity enabled
152- ``EFAULT``: the caller provided inaccessible memory
153- ``EINTR``: the operation was interrupted by a fatal signal
154- ``EINVAL``: unsupported version, hash algorithm, or block size; or
155 reserved bits are set; or the file descriptor refers to neither a
156 regular file nor a directory.
157- ``EISDIR``: the file descriptor refers to a directory
158- ``EKEYREJECTED``: the signature doesn't match the file
159- ``EMSGSIZE``: the salt or signature is too long
160- ``ENOKEY``: the fs-verity keyring doesn't contain the certificate
161 needed to verify the signature
162- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not
163 available in the kernel's crypto API as currently configured (e.g.
164 for SHA-512, missing CONFIG_CRYPTO_SHA512).
165- ``ENOTTY``: this type of filesystem does not implement fs-verity
166- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
167 support; or the filesystem superblock has not had the 'verity'
168 feature enabled on it; or the filesystem does not support fs-verity
169 on this file. (See `Filesystem support`_.)
170- ``EPERM``: the file is append-only; or, a signature is required and
171 one was not provided.
172- ``EROFS``: the filesystem is read-only
173- ``ETXTBSY``: someone has the file open for writing. This can be the
174 caller's file descriptor, another open file descriptor, or the file
175 reference held by a writable memory map.
176
177FS_IOC_MEASURE_VERITY
178---------------------
179
180The FS_IOC_MEASURE_VERITY ioctl retrieves the measurement of a verity
181file. The file measurement is a digest that cryptographically
182identifies the file contents that are being enforced on reads.
183
184This ioctl takes in a pointer to a variable-length structure::
185
186 struct fsverity_digest {
187 __u16 digest_algorithm;
188 __u16 digest_size; /* input/output */
189 __u8 digest[];
190 };
191
192``digest_size`` is an input/output field. On input, it must be
193initialized to the number of bytes allocated for the variable-length
194``digest`` field.
195
196On success, 0 is returned and the kernel fills in the structure as
197follows:
198
199- ``digest_algorithm`` will be the hash algorithm used for the file
200 measurement. It will match ``fsverity_enable_arg::hash_algorithm``.
201- ``digest_size`` will be the size of the digest in bytes, e.g. 32
202 for SHA-256. (This can be redundant with ``digest_algorithm``.)
203- ``digest`` will be the actual bytes of the digest.
204
205FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time,
206regardless of the size of the file.
207
208FS_IOC_MEASURE_VERITY can fail with the following errors:
209
210- ``EFAULT``: the caller provided inaccessible memory
211- ``ENODATA``: the file is not a verity file
212- ``ENOTTY``: this type of filesystem does not implement fs-verity
213- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
214 support, or the filesystem superblock has not had the 'verity'
215 feature enabled on it. (See `Filesystem support`_.)
216- ``EOVERFLOW``: the digest is longer than the specified
217 ``digest_size`` bytes. Try providing a larger buffer.
218
219FS_IOC_GETFLAGS
220---------------
221
222The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity)
223can also be used to check whether a file has fs-verity enabled or not.
224To do so, check for FS_VERITY_FL (0x00100000) in the returned flags.
225
226The verity flag is not settable via FS_IOC_SETFLAGS. You must use
227FS_IOC_ENABLE_VERITY instead, since parameters must be provided.
228
229statx
230-----
231
232Since Linux v5.5, the statx() system call sets STATX_ATTR_VERITY if
233the file has fs-verity enabled. This can perform better than
234FS_IOC_GETFLAGS and FS_IOC_MEASURE_VERITY because it doesn't require
235opening the file, and opening verity files can be expensive.
236
237Accessing verity files
238======================
239
240Applications can transparently access a verity file just like a
241non-verity one, with the following exceptions:
242
243- Verity files are readonly. They cannot be opened for writing or
244 truncate()d, even if the file mode bits allow it. Attempts to do
245 one of these things will fail with EPERM. However, changes to
246 metadata such as owner, mode, timestamps, and xattrs are still
247 allowed, since these are not measured by fs-verity. Verity files
248 can also still be renamed, deleted, and linked to.
249
250- Direct I/O is not supported on verity files. Attempts to use direct
251 I/O on such files will fall back to buffered I/O.
252
253- DAX (Direct Access) is not supported on verity files, because this
254 would circumvent the data verification.
255
256- Reads of data that doesn't match the verity Merkle tree will fail
257 with EIO (for read()) or SIGBUS (for mmap() reads).
258
259- If the sysctl "fs.verity.require_signatures" is set to 1 and the
260 file's verity measurement is not signed by a key in the fs-verity
261 keyring, then opening the file will fail. See `Built-in signature
262 verification`_.
263
264Direct access to the Merkle tree is not supported. Therefore, if a
265verity file is copied, or is backed up and restored, then it will lose
266its "verity"-ness. fs-verity is primarily meant for files like
267executables that are managed by a package manager.
268
269File measurement computation
270============================
271
272This section describes how fs-verity hashes the file contents using a
273Merkle tree to produce the "file measurement" which cryptographically
274identifies the file contents. This algorithm is the same for all
275filesystems that support fs-verity.
276
277Userspace only needs to be aware of this algorithm if it needs to
278compute the file measurement itself, e.g. in order to sign the file.
279
280.. _fsverity_merkle_tree:
281
282Merkle tree
283-----------
284
285The file contents is divided into blocks, where the block size is
286configurable but is usually 4096 bytes. The end of the last block is
287zero-padded if needed. Each block is then hashed, producing the first
288level of hashes. Then, the hashes in this first level are grouped
289into 'blocksize'-byte blocks (zero-padding the ends as needed) and
290these blocks are hashed, producing the second level of hashes. This
291proceeds up the tree until only a single block remains. The hash of
292this block is the "Merkle tree root hash".
293
294If the file fits in one block and is nonempty, then the "Merkle tree
295root hash" is simply the hash of the single data block. If the file
296is empty, then the "Merkle tree root hash" is all zeroes.
297
298The "blocks" here are not necessarily the same as "filesystem blocks".
299
300If a salt was specified, then it's zero-padded to the closest multiple
301of the input size of the hash algorithm's compression function, e.g.
30264 bytes for SHA-256 or 128 bytes for SHA-512. The padded salt is
303prepended to every data or Merkle tree block that is hashed.
304
305The purpose of the block padding is to cause every hash to be taken
306over the same amount of data, which simplifies the implementation and
307keeps open more possibilities for hardware acceleration. The purpose
308of the salt padding is to make the salting "free" when the salted hash
309state is precomputed, then imported for each hash.
310
311Example: in the recommended configuration of SHA-256 and 4K blocks,
312128 hash values fit in each block. Thus, each level of the Merkle
313tree is approximately 128 times smaller than the previous, and for
314large files the Merkle tree's size converges to approximately 1/127 of
315the original file size. However, for small files, the padding is
316significant, making the space overhead proportionally more.
317
318.. _fsverity_descriptor:
319
320fs-verity descriptor
321--------------------
322
323By itself, the Merkle tree root hash is ambiguous. For example, it
324can't a distinguish a large file from a small second file whose data
325is exactly the top-level hash block of the first file. Ambiguities
326also arise from the convention of padding to the next block boundary.
327
328To solve this problem, the verity file measurement is actually
329computed as a hash of the following structure, which contains the
330Merkle tree root hash as well as other fields such as the file size::
331
332 struct fsverity_descriptor {
333 __u8 version; /* must be 1 */
334 __u8 hash_algorithm; /* Merkle tree hash algorithm */
335 __u8 log_blocksize; /* log2 of size of data and tree blocks */
336 __u8 salt_size; /* size of salt in bytes; 0 if none */
337 __le32 sig_size; /* must be 0 */
338 __le64 data_size; /* size of file the Merkle tree is built over */
339 __u8 root_hash[64]; /* Merkle tree root hash */
340 __u8 salt[32]; /* salt prepended to each hashed block */
341 __u8 __reserved[144]; /* must be 0's */
342 };
343
344Note that the ``sig_size`` field must be set to 0 for the purpose of
345computing the file measurement, even if a signature was provided (or
346will be provided) to `FS_IOC_ENABLE_VERITY`_.
347
348Built-in signature verification
349===============================
350
351With CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y, fs-verity supports putting
352a portion of an authentication policy (see `Use cases`_) in the
353kernel. Specifically, it adds support for:
354
3551. At fs-verity module initialization time, a keyring ".fs-verity" is
356 created. The root user can add trusted X.509 certificates to this
357 keyring using the add_key() system call, then (when done)
358 optionally use keyctl_restrict_keyring() to prevent additional
359 certificates from being added.
360
3612. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted
362 detached signature in DER format of the file measurement. On
363 success, this signature is persisted alongside the Merkle tree.
364 Then, any time the file is opened, the kernel will verify the
365 file's actual measurement against this signature, using the
366 certificates in the ".fs-verity" keyring.
367
3683. A new sysctl "fs.verity.require_signatures" is made available.
369 When set to 1, the kernel requires that all verity files have a
370 correctly signed file measurement as described in (2).
371
372File measurements must be signed in the following format, which is
373similar to the structure used by `FS_IOC_MEASURE_VERITY`_::
374
375 struct fsverity_signed_digest {
376 char magic[8]; /* must be "FSVerity" */
377 __le16 digest_algorithm;
378 __le16 digest_size;
379 __u8 digest[];
380 };
381
382fs-verity's built-in signature verification support is meant as a
383relatively simple mechanism that can be used to provide some level of
384authenticity protection for verity files, as an alternative to doing
385the signature verification in userspace or using IMA-appraisal.
386However, with this mechanism, userspace programs still need to check
387that the verity bit is set, and there is no protection against verity
388files being swapped around.
389
390Filesystem support
391==================
392
393fs-verity is currently supported by the ext4 and f2fs filesystems.
394The CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity
395on either filesystem.
396
397``include/linux/fsverity.h`` declares the interface between the
398``fs/verity/`` support layer and filesystems. Briefly, filesystems
399must provide an ``fsverity_operations`` structure that provides
400methods to read and write the verity metadata to a filesystem-specific
401location, including the Merkle tree blocks and
402``fsverity_descriptor``. Filesystems must also call functions in
403``fs/verity/`` at certain times, such as when a file is opened or when
404pages have been read into the pagecache. (See `Verifying data`_.)
405
406ext4
407----
408
409ext4 supports fs-verity since Linux v5.4 and e2fsprogs v1.45.2.
410
411To create verity files on an ext4 filesystem, the filesystem must have
412been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on
413it. "verity" is an RO_COMPAT filesystem feature, so once set, old
414kernels will only be able to mount the filesystem readonly, and old
415versions of e2fsck will be unable to check the filesystem. Moreover,
416currently ext4 only supports mounting a filesystem with the "verity"
417feature when its block size is equal to PAGE_SIZE (often 4096 bytes).
418
419ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files. It
420can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared.
421
422ext4 also supports encryption, which can be used simultaneously with
423fs-verity. In this case, the plaintext data is verified rather than
424the ciphertext. This is necessary in order to make the file
425measurement meaningful, since every file is encrypted differently.
426
427ext4 stores the verity metadata (Merkle tree and fsverity_descriptor)
428past the end of the file, starting at the first 64K boundary beyond
429i_size. This approach works because (a) verity files are readonly,
430and (b) pages fully beyond i_size aren't visible to userspace but can
431be read/written internally by ext4 with only some relatively small
432changes to ext4. This approach avoids having to depend on the
433EA_INODE feature and on rearchitecturing ext4's xattr support to
434support paging multi-gigabyte xattrs into memory, and to support
435encrypting xattrs. Note that the verity metadata *must* be encrypted
436when the file is, since it contains hashes of the plaintext data.
437
438Currently, ext4 verity only supports the case where the Merkle tree
439block size, filesystem block size, and page size are all the same. It
440also only supports extent-based files.
441
442f2fs
443----
444
445f2fs supports fs-verity since Linux v5.4 and f2fs-tools v1.11.0.
446
447To create verity files on an f2fs filesystem, the filesystem must have
448been formatted with ``-O verity``.
449
450f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files.
451It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be
452cleared.
453
454Like ext4, f2fs stores the verity metadata (Merkle tree and
455fsverity_descriptor) past the end of the file, starting at the first
45664K boundary beyond i_size. See explanation for ext4 above.
457Moreover, f2fs supports at most 4096 bytes of xattr entries per inode
458which wouldn't be enough for even a single Merkle tree block.
459
460Currently, f2fs verity only supports a Merkle tree block size of 4096.
461Also, f2fs doesn't support enabling verity on files that currently
462have atomic or volatile writes pending.
463
464Implementation details
465======================
466
467Verifying data
468--------------
469
470fs-verity ensures that all reads of a verity file's data are verified,
471regardless of which syscall is used to do the read (e.g. mmap(),
472read(), pread()) and regardless of whether it's the first read or a
473later read (unless the later read can return cached data that was
474already verified). Below, we describe how filesystems implement this.
475
476Pagecache
477~~~~~~~~~
478
479For filesystems using Linux's pagecache, the ``->readpage()`` and
480``->readpages()`` methods must be modified to verify pages before they
481are marked Uptodate. Merely hooking ``->read_iter()`` would be
482insufficient, since ``->read_iter()`` is not used for memory maps.
483
484Therefore, fs/verity/ provides a function fsverity_verify_page() which
485verifies a page that has been read into the pagecache of a verity
486inode, but is still locked and not Uptodate, so it's not yet readable
487by userspace. As needed to do the verification,
488fsverity_verify_page() will call back into the filesystem to read
489Merkle tree pages via fsverity_operations::read_merkle_tree_page().
490
491fsverity_verify_page() returns false if verification failed; in this
492case, the filesystem must not set the page Uptodate. Following this,
493as per the usual Linux pagecache behavior, attempts by userspace to
494read() from the part of the file containing the page will fail with
495EIO, and accesses to the page within a memory map will raise SIGBUS.
496
497fsverity_verify_page() currently only supports the case where the
498Merkle tree block size is equal to PAGE_SIZE (often 4096 bytes).
499
500In principle, fsverity_verify_page() verifies the entire path in the
501Merkle tree from the data page to the root hash. However, for
502efficiency the filesystem may cache the hash pages. Therefore,
503fsverity_verify_page() only ascends the tree reading hash pages until
504an already-verified hash page is seen, as indicated by the PageChecked
505bit being set. It then verifies the path to that page.
506
507This optimization, which is also used by dm-verity, results in
508excellent sequential read performance. This is because usually (e.g.
509127 in 128 times for 4K blocks and SHA-256) the hash page from the
510bottom level of the tree will already be cached and checked from
511reading a previous data page. However, random reads perform worse.
512
513Block device based filesystems
514~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
515
516Block device based filesystems (e.g. ext4 and f2fs) in Linux also use
517the pagecache, so the above subsection applies too. However, they
518also usually read many pages from a file at once, grouped into a
519structure called a "bio". To make it easier for these types of
520filesystems to support fs-verity, fs/verity/ also provides a function
521fsverity_verify_bio() which verifies all pages in a bio.
522
523ext4 and f2fs also support encryption. If a verity file is also
524encrypted, the pages must be decrypted before being verified. To
525support this, these filesystems allocate a "post-read context" for
526each bio and store it in ``->bi_private``::
527
528 struct bio_post_read_ctx {
529 struct bio *bio;
530 struct work_struct work;
531 unsigned int cur_step;
532 unsigned int enabled_steps;
533 };
534
535``enabled_steps`` is a bitmask that specifies whether decryption,
536verity, or both is enabled. After the bio completes, for each needed
537postprocessing step the filesystem enqueues the bio_post_read_ctx on a
538workqueue, and then the workqueue work does the decryption or
539verification. Finally, pages where no decryption or verity error
540occurred are marked Uptodate, and the pages are unlocked.
541
542Files on ext4 and f2fs may contain holes. Normally, ``->readpages()``
543simply zeroes holes and sets the corresponding pages Uptodate; no bios
544are issued. To prevent this case from bypassing fs-verity, these
545filesystems use fsverity_verify_page() to verify hole pages.
546
547ext4 and f2fs disable direct I/O on verity files, since otherwise
548direct I/O would bypass fs-verity. (They also do the same for
549encrypted files.)
550
551Userspace utility
552=================
553
554This document focuses on the kernel, but a userspace utility for
555fs-verity can be found at:
556
557 https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/fsverity-utils.git
558
559See the README.md file in the fsverity-utils source tree for details,
560including examples of setting up fs-verity protected files.
561
562Tests
563=====
564
565To test fs-verity, use xfstests. For example, using `kvm-xfstests
566<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::
567
568 kvm-xfstests -c ext4,f2fs -g verity
569
570FAQ
571===
572
573This section answers frequently asked questions about fs-verity that
574weren't already directly answered in other parts of this document.
575
576:Q: Why isn't fs-verity part of IMA?
577:A: fs-verity and IMA (Integrity Measurement Architecture) have
578 different focuses. fs-verity is a filesystem-level mechanism for
579 hashing individual files using a Merkle tree. In contrast, IMA
580 specifies a system-wide policy that specifies which files are
581 hashed and what to do with those hashes, such as log them,
582 authenticate them, or add them to a measurement list.
583
584 IMA is planned to support the fs-verity hashing mechanism as an
585 alternative to doing full file hashes, for people who want the
586 performance and security benefits of the Merkle tree based hash.
587 But it doesn't make sense to force all uses of fs-verity to be
588 through IMA. As a standalone filesystem feature, fs-verity
589 already meets many users' needs, and it's testable like other
590 filesystem features e.g. with xfstests.
591
592:Q: Isn't fs-verity useless because the attacker can just modify the
593 hashes in the Merkle tree, which is stored on-disk?
594:A: To verify the authenticity of an fs-verity file you must verify
595 the authenticity of the "file measurement", which is basically the
596 root hash of the Merkle tree. See `Use cases`_.
597
598:Q: Isn't fs-verity useless because the attacker can just replace a
599 verity file with a non-verity one?
600:A: See `Use cases`_. In the initial use case, it's really trusted
601 userspace code that authenticates the files; fs-verity is just a
602 tool to do this job efficiently and securely. The trusted
603 userspace code will consider non-verity files to be inauthentic.
604
605:Q: Why does the Merkle tree need to be stored on-disk? Couldn't you
606 store just the root hash?
607:A: If the Merkle tree wasn't stored on-disk, then you'd have to
608 compute the entire tree when the file is first accessed, even if
609 just one byte is being read. This is a fundamental consequence of
610 how Merkle tree hashing works. To verify a leaf node, you need to
611 verify the whole path to the root hash, including the root node
612 (the thing which the root hash is a hash of). But if the root
613 node isn't stored on-disk, you have to compute it by hashing its
614 children, and so on until you've actually hashed the entire file.
615
616 That defeats most of the point of doing a Merkle tree-based hash,
617 since if you have to hash the whole file ahead of time anyway,
618 then you could simply do sha256(file) instead. That would be much
619 simpler, and a bit faster too.
620
621 It's true that an in-memory Merkle tree could still provide the
622 advantage of verification on every read rather than just on the
623 first read. However, it would be inefficient because every time a
624 hash page gets evicted (you can't pin the entire Merkle tree into
625 memory, since it may be very large), in order to restore it you
626 again need to hash everything below it in the tree. This again
627 defeats most of the point of doing a Merkle tree-based hash, since
628 a single block read could trigger re-hashing gigabytes of data.
629
630:Q: But couldn't you store just the leaf nodes and compute the rest?
631:A: See previous answer; this really just moves up one level, since
632 one could alternatively interpret the data blocks as being the
633 leaf nodes of the Merkle tree. It's true that the tree can be
634 computed much faster if the leaf level is stored rather than just
635 the data, but that's only because each level is less than 1% the
636 size of the level below (assuming the recommended settings of
637 SHA-256 and 4K blocks). For the exact same reason, by storing
638 "just the leaf nodes" you'd already be storing over 99% of the
639 tree, so you might as well simply store the whole tree.
640
641:Q: Can the Merkle tree be built ahead of time, e.g. distributed as
642 part of a package that is installed to many computers?
643:A: This isn't currently supported. It was part of the original
644 design, but was removed to simplify the kernel UAPI and because it
645 wasn't a critical use case. Files are usually installed once and
646 used many times, and cryptographic hashing is somewhat fast on
647 most modern processors.
648
649:Q: Why doesn't fs-verity support writes?
650:A: Write support would be very difficult and would require a
651 completely different design, so it's well outside the scope of
652 fs-verity. Write support would require:
653
654 - A way to maintain consistency between the data and hashes,
655 including all levels of hashes, since corruption after a crash
656 (especially of potentially the entire file!) is unacceptable.
657 The main options for solving this are data journalling,
658 copy-on-write, and log-structured volume. But it's very hard to
659 retrofit existing filesystems with new consistency mechanisms.
660 Data journalling is available on ext4, but is very slow.
661
662 - Rebuilding the Merkle tree after every write, which would be
663 extremely inefficient. Alternatively, a different authenticated
664 dictionary structure such as an "authenticated skiplist" could
665 be used. However, this would be far more complex.
666
667 Compare it to dm-verity vs. dm-integrity. dm-verity is very
668 simple: the kernel just verifies read-only data against a
669 read-only Merkle tree. In contrast, dm-integrity supports writes
670 but is slow, is much more complex, and doesn't actually support
671 full-device authentication since it authenticates each sector
672 independently, i.e. there is no "root hash". It doesn't really
673 make sense for the same device-mapper target to support these two
674 very different cases; the same applies to fs-verity.
675
676:Q: Since verity files are immutable, why isn't the immutable bit set?
677:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a
678 specific set of semantics which not only make the file contents
679 read-only, but also prevent the file from being deleted, renamed,
680 linked to, or having its owner or mode changed. These extra
681 properties are unwanted for fs-verity, so reusing the immutable
682 bit isn't appropriate.
683
684:Q: Why does the API use ioctls instead of setxattr() and getxattr()?
685:A: Abusing the xattr interface for basically arbitrary syscalls is
686 heavily frowned upon by most of the Linux filesystem developers.
687 An xattr should really just be an xattr on-disk, not an API to
688 e.g. magically trigger construction of a Merkle tree.
689
690:Q: Does fs-verity support remote filesystems?
691:A: Only ext4 and f2fs support is implemented currently, but in
692 principle any filesystem that can store per-file verity metadata
693 can support fs-verity, regardless of whether it's local or remote.
694 Some filesystems may have fewer options of where to store the
695 verity metadata; one possibility is to store it past the end of
696 the file and "hide" it from userspace by manipulating i_size. The
697 data verification functions provided by ``fs/verity/`` also assume
698 that the filesystem uses the Linux pagecache, but both local and
699 remote filesystems normally do so.
700
701:Q: Why is anything filesystem-specific at all? Shouldn't fs-verity
702 be implemented entirely at the VFS level?
703:A: There are many reasons why this is not possible or would be very
704 difficult, including the following:
705
706 - To prevent bypassing verification, pages must not be marked
707 Uptodate until they've been verified. Currently, each
708 filesystem is responsible for marking pages Uptodate via
709 ``->readpages()``. Therefore, currently it's not possible for
710 the VFS to do the verification on its own. Changing this would
711 require significant changes to the VFS and all filesystems.
712
713 - It would require defining a filesystem-independent way to store
714 the verity metadata. Extended attributes don't work for this
715 because (a) the Merkle tree may be gigabytes, but many
716 filesystems assume that all xattrs fit into a single 4K
717 filesystem block, and (b) ext4 and f2fs encryption doesn't
718 encrypt xattrs, yet the Merkle tree *must* be encrypted when the
719 file contents are, because it stores hashes of the plaintext
720 file contents.
721
722 So the verity metadata would have to be stored in an actual
723 file. Using a separate file would be very ugly, since the
724 metadata is fundamentally part of the file to be protected, and
725 it could cause problems where users could delete the real file
726 but not the metadata file or vice versa. On the other hand,
727 having it be in the same file would break applications unless
728 filesystems' notion of i_size were divorced from the VFS's,
729 which would be complex and require changes to all filesystems.
730
731 - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's
732 transaction mechanism so that either the file ends up with
733 verity enabled, or no changes were made. Allowing intermediate
734 states to occur after a crash may cause problems.