1Written by: Neil Brown
  2Please see MAINTAINERS file for where to send questions.
  3
  4Overlay Filesystem
  5==================
  6
  7This document describes a prototype for a new approach to providing
  8overlay-filesystem functionality in Linux (sometimes referred to as
  9union-filesystems).  An overlay-filesystem tries to present a
 10filesystem which is the result over overlaying one filesystem on top
 11of the other.
 12
 13The result will inevitably fail to look exactly like a normal
 14filesystem for various technical reasons.  The expectation is that
 15many use cases will be able to ignore these differences.
 16
 17This approach is 'hybrid' because the objects that appear in the
 18filesystem do not all appear to belong to that filesystem.  In many
 19cases an object accessed in the union will be indistinguishable
 20from accessing the corresponding object from the original filesystem.
 21This is most obvious from the 'st_dev' field returned by stat(2).
 22
 23While directories will report an st_dev from the overlay-filesystem,
 24all non-directory objects will report an st_dev from the lower or
 25upper filesystem that is providing the object.  Similarly st_ino will
 26only be unique when combined with st_dev, and both of these can change
 27over the lifetime of a non-directory object.  Many applications and
 28tools ignore these values and will not be affected.
 29
 30Upper and Lower
 31---------------
 32
 33An overlay filesystem combines two filesystems - an 'upper' filesystem
 34and a 'lower' filesystem.  When a name exists in both filesystems, the
 35object in the 'upper' filesystem is visible while the object in the
 36'lower' filesystem is either hidden or, in the case of directories,
 37merged with the 'upper' object.
 38
 39It would be more correct to refer to an upper and lower 'directory
 40tree' rather than 'filesystem' as it is quite possible for both
 41directory trees to be in the same filesystem and there is no
 42requirement that the root of a filesystem be given for either upper or
 43lower.
 44
 45The lower filesystem can be any filesystem supported by Linux and does
 46not need to be writable.  The lower filesystem can even be another
 47overlayfs.  The upper filesystem will normally be writable and if it
 48is it must support the creation of trusted.* extended attributes, and
 49must provide valid d_type in readdir responses, so NFS is not suitable.
 50
 51A read-only overlay of two read-only filesystems may use any
 52filesystem type.
 53
 54Directories
 55-----------
 56
 57Overlaying mainly involves directories.  If a given name appears in both
 58upper and lower filesystems and refers to a non-directory in either,
 59then the lower object is hidden - the name refers only to the upper
 60object.
 61
 62Where both upper and lower objects are directories, a merged directory
 63is formed.
 64
 65At mount time, the two directories given as mount options "lowerdir" and
 66"upperdir" are combined into a merged directory:
 67
 68  mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\
 69workdir=/work /merged
 70
 71The "workdir" needs to be an empty directory on the same filesystem
 72as upperdir.
 73
 74Then whenever a lookup is requested in such a merged directory, the
 75lookup is performed in each actual directory and the combined result
 76is cached in the dentry belonging to the overlay filesystem.  If both
 77actual lookups find directories, both are stored and a merged
 78directory is created, otherwise only one is stored: the upper if it
 79exists, else the lower.
 80
 81Only the lists of names from directories are merged.  Other content
 82such as metadata and extended attributes are reported for the upper
 83directory only.  These attributes of the lower directory are hidden.
 84
 85whiteouts and opaque directories
 86--------------------------------
 87
 88In order to support rm and rmdir without changing the lower
 89filesystem, an overlay filesystem needs to record in the upper filesystem
 90that files have been removed.  This is done using whiteouts and opaque
 91directories (non-directories are always opaque).
 92
 93A whiteout is created as a character device with 0/0 device number.
 94When a whiteout is found in the upper level of a merged directory, any
 95matching name in the lower level is ignored, and the whiteout itself
 96is also hidden.
 97
 98A directory is made opaque by setting the xattr "trusted.overlay.opaque"
 99to "y".  Where the upper filesystem contains an opaque directory, any
100directory in the lower filesystem with the same name is ignored.
101
102readdir
103-------
104
105When a 'readdir' request is made on a merged directory, the upper and
106lower directories are each read and the name lists merged in the
107obvious way (upper is read first, then lower - entries that already
108exist are not re-added).  This merged name list is cached in the
109'struct file' and so remains as long as the file is kept open.  If the
110directory is opened and read by two processes at the same time, they
111will each have separate caches.  A seekdir to the start of the
112directory (offset 0) followed by a readdir will cause the cache to be
113discarded and rebuilt.
114
115This means that changes to the merged directory do not appear while a
116directory is being read.  This is unlikely to be noticed by many
117programs.
118
119seek offsets are assigned sequentially when the directories are read.
120Thus if
121  - read part of a directory
122  - remember an offset, and close the directory
123  - re-open the directory some time later
124  - seek to the remembered offset
125
126there may be little correlation between the old and new locations in
127the list of filenames, particularly if anything has changed in the
128directory.
129
130Readdir on directories that are not merged is simply handled by the
131underlying directory (upper or lower).
132
133
134Non-directories
135---------------
136
137Objects that are not directories (files, symlinks, device-special
138files etc.) are presented either from the upper or lower filesystem as
139appropriate.  When a file in the lower filesystem is accessed in a way
140the requires write-access, such as opening for write access, changing
141some metadata etc., the file is first copied from the lower filesystem
142to the upper filesystem (copy_up).  Note that creating a hard-link
143also requires copy_up, though of course creation of a symlink does
144not.
145
146The copy_up may turn out to be unnecessary, for example if the file is
147opened for read-write but the data is not modified.
148
149The copy_up process first makes sure that the containing directory
150exists in the upper filesystem - creating it and any parents as
151necessary.  It then creates the object with the same metadata (owner,
152mode, mtime, symlink-target etc.) and then if the object is a file, the
153data is copied from the lower to the upper filesystem.  Finally any
154extended attributes are copied up.
155
156Once the copy_up is complete, the overlay filesystem simply
157provides direct access to the newly created file in the upper
158filesystem - future operations on the file are barely noticed by the
159overlay filesystem (though an operation on the name of the file such as
160rename or unlink will of course be noticed and handled).
161
162
163Multiple lower layers
164---------------------
165
166Multiple lower layers can now be given using the the colon (":") as a
167separator character between the directory names.  For example:
168
169  mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged
170
171As the example shows, "upperdir=" and "workdir=" may be omitted.  In
172that case the overlay will be read-only.
173
174The specified lower directories will be stacked beginning from the
175rightmost one and going left.  In the above example lower1 will be the
176top, lower2 the middle and lower3 the bottom layer.
177
178
179Non-standard behavior
180---------------------
181
182The copy_up operation essentially creates a new, identical file and
183moves it over to the old name.  The new file may be on a different
184filesystem, so both st_dev and st_ino of the file may change.
185
186Any open files referring to this inode will access the old data and
187metadata.  Similarly any file locks obtained before copy_up will not
188apply to the copied up file.
189
190On a file opened with O_RDONLY fchmod(2), fchown(2), futimesat(2) and
191fsetxattr(2) will fail with EROFS.
192
193If a file with multiple hard links is copied up, then this will
194"break" the link.  Changes will not be propagated to other names
195referring to the same inode.
196
197Symlinks in /proc/PID/ and /proc/PID/fd which point to a non-directory
198object in overlayfs will not contain valid absolute paths, only
199relative paths leading up to the filesystem's root.  This will be
200fixed in the future.
201
202Some operations are not atomic, for example a crash during copy_up or
203rename will leave the filesystem in an inconsistent state.  This will
204be addressed in the future.
205
206Changes to underlying filesystems
207---------------------------------
208
209Offline changes, when the overlay is not mounted, are allowed to either
210the upper or the lower trees.
211
212Changes to the underlying filesystems while part of a mounted overlay
213filesystem are not allowed.  If the underlying filesystem is changed,
214the behavior of the overlay is undefined, though it will not result in
215a crash or deadlock.
216
217Testsuite
218---------
219
220There's testsuite developed by David Howells at:
221
222  git://git.infradead.org/users/dhowells/unionmount-testsuite.git
223
224Run as root:
225
226  # cd unionmount-testsuite
227  # ./run --ov