1File management in the Linux kernel
  2-----------------------------------
  3
  4This document describes how locking for files (struct file)
  5and file descriptor table (struct files) works.
  6
  7Up until 2.6.12, the file descriptor table has been protected
  8with a lock (files->file_lock) and reference count (files->count).
  9->file_lock protected accesses to all the file related fields
 10of the table. ->count was used for sharing the file descriptor
 11table between tasks cloned with CLONE_FILES flag. Typically
 12this would be the case for posix threads. As with the common
 13refcounting model in the kernel, the last task doing
 14a put_files_struct() frees the file descriptor (fd) table.
 15The files (struct file) themselves are protected using
 16reference count (->f_count).
 17
 18In the new lock-free model of file descriptor management,
 19the reference counting is similar, but the locking is
 20based on RCU. The file descriptor table contains multiple
 21elements - the fd sets (open_fds and close_on_exec, the
 22array of file pointers, the sizes of the sets and the array
 23etc.). In order for the updates to appear atomic to
 24a lock-free reader, all the elements of the file descriptor
 25table are in a separate structure - struct fdtable.
 26files_struct contains a pointer to struct fdtable through
 27which the actual fd table is accessed. Initially the
 28fdtable is embedded in files_struct itself. On a subsequent
 29expansion of fdtable, a new fdtable structure is allocated
 30and files->fdtab points to the new structure. The fdtable
 31structure is freed with RCU and lock-free readers either
 32see the old fdtable or the new fdtable making the update
 33appear atomic. Here are the locking rules for
 34the fdtable structure -
 35
 361. All references to the fdtable must be done through
 37   the files_fdtable() macro :
 38
 39	struct fdtable *fdt;
 40
 41	rcu_read_lock();
 42
 43	fdt = files_fdtable(files);
 44	....
 45	if (n <= fdt->max_fds)
 46		....
 47	...
 48	rcu_read_unlock();
 49
 50   files_fdtable() uses rcu_dereference() macro which takes care of
 51   the memory barrier requirements for lock-free dereference.
 52   The fdtable pointer must be read within the read-side
 53   critical section.
 54
 552. Reading of the fdtable as described above must be protected
 56   by rcu_read_lock()/rcu_read_unlock().
 57
 583. For any update to the fd table, files->file_lock must
 59   be held.
 60
 614. To look up the file structure given an fd, a reader
 62   must use either fcheck() or fcheck_files() APIs. These
 63   take care of barrier requirements due to lock-free lookup.
 64   An example :
 65
 66	struct file *file;
 67
 68	rcu_read_lock();
 69	file = fcheck(fd);
 70	if (file) {
 71		...
 72	}
 73	....
 74	rcu_read_unlock();
 75
 765. Handling of the file structures is special. Since the look-up
 77   of the fd (fget()/fget_light()) are lock-free, it is possible
 78   that look-up may race with the last put() operation on the
 79   file structure. This is avoided using atomic_long_inc_not_zero()
 80   on ->f_count :
 81
 82	rcu_read_lock();
 83	file = fcheck_files(files, fd);
 84	if (file) {
 85		if (atomic_long_inc_not_zero(&file->f_count))
 86			*fput_needed = 1;
 87		else
 88		/* Didn't get the reference, someone's freed */
 89			file = NULL;
 90	}
 91	rcu_read_unlock();
 92	....
 93	return file;
 94
 95   atomic_long_inc_not_zero() detects if refcounts is already zero or
 96   goes to zero during increment. If it does, we fail
 97   fget()/fget_light().
 98
 996. Since both fdtable and file structures can be looked up
100   lock-free, they must be installed using rcu_assign_pointer()
101   API. If they are looked up lock-free, rcu_dereference()
102   must be used. However it is advisable to use files_fdtable()
103   and fcheck()/fcheck_files() which take care of these issues.
104
1057. While updating, the fdtable pointer must be looked up while
106   holding files->file_lock. If ->file_lock is dropped, then
107   another thread expand the files thereby creating a new
108   fdtable and making the earlier fdtable pointer stale.
109   For example :
110
111	spin_lock(&files->file_lock);
112	fd = locate_fd(files, file, start);
113	if (fd >= 0) {
114		/* locate_fd() may have expanded fdtable, load the ptr */
115		fdt = files_fdtable(files);
116		__set_open_fd(fd, fdt);
117		__clear_close_on_exec(fd, fdt);
118		spin_unlock(&files->file_lock);
119	.....
120
121   Since locate_fd() can drop ->file_lock (and reacquire ->file_lock),
122   the fdtable pointer (fdt) must be loaded after locate_fd().
123