1Reference counting in pnfs:
  2==========================
  3
  4The are several inter-related caches.  We have layouts which can
  5reference multiple devices, each of which can reference multiple data servers.
  6Each data server can be referenced by multiple devices.  Each device
  7can be referenced by multiple layouts.  To keep all of this straight,
  8we need to reference count.
  9
 10
 11struct pnfs_layout_hdr
 12----------------------
 13The on-the-wire command LAYOUTGET corresponds to struct
 14pnfs_layout_segment, usually referred to by the variable name lseg.
 15Each nfs_inode may hold a pointer to a cache of these layout
 16segments in nfsi->layout, of type struct pnfs_layout_hdr.
 17
 18We reference the header for the inode pointing to it, across each
 19outstanding RPC call that references it (LAYOUTGET, LAYOUTRETURN,
 20LAYOUTCOMMIT), and for each lseg held within.
 21
 22Each header is also (when non-empty) put on a list associated with
 23struct nfs_client (cl_layouts).  Being put on this list does not bump
 24the reference count, as the layout is kept around by the lseg that
 25keeps it in the list.
 26
 27deviceid_cache
 28--------------
 29lsegs reference device ids, which are resolved per nfs_client and
 30layout driver type.  The device ids are held in a RCU cache (struct
 31nfs4_deviceid_cache).  The cache itself is referenced across each
 32mount.  The entries (struct nfs4_deviceid) themselves are held across
 33the lifetime of each lseg referencing them.
 34
 35RCU is used because the deviceid is basically a write once, read many
 36data structure.  The hlist size of 32 buckets needs better
 37justification, but seems reasonable given that we can have multiple
 38deviceid's per filesystem, and multiple filesystems per nfs_client.
 39
 40The hash code is copied from the nfsd code base.  A discussion of
 41hashing and variations of this algorithm can be found at:
 42http://groups.google.com/group/comp.lang.c/browse_thread/thread/9522965e2b8d3809
 43
 44data server cache
 45-----------------
 46file driver devices refer to data servers, which are kept in a module
 47level cache.  Its reference is held over the lifetime of the deviceid
 48pointing to it.
 49
 50lseg
 51----
 52lseg maintains an extra reference corresponding to the NFS_LSEG_VALID
 53bit which holds it in the pnfs_layout_hdr's list.  When the final lseg
 54is removed from the pnfs_layout_hdr's list, the NFS_LAYOUT_DESTROYED
 55bit is set, preventing any new lsegs from being added.
 56
 57layout drivers
 58--------------
 59
 60PNFS utilizes what is called layout drivers. The STD defines 4 basic
 61layout types: "files", "objects", "blocks", and "flexfiles". For each
 62of these types there is a layout-driver with a common function-vectors
 63table which are called by the nfs-client pnfs-core to implement the
 64different layout types.
 65
 66Files-layout-driver code is in: fs/nfs/filelayout/.. directory
 67Objects-layout-driver code is in: fs/nfs/objlayout/.. directory
 68Blocks-layout-driver code is in: fs/nfs/blocklayout/.. directory
 69Flexfiles-layout-driver code is in: fs/nfs/flexfilelayout/.. directory
 70
 71objects-layout setup
 72--------------------
 73
 74As part of the full STD implementation the objlayoutdriver.ko needs, at times,
 75to automatically login to yet undiscovered iscsi/osd devices. For this the
 76driver makes up-calles to a user-mode script called *osd_login*
 77
 78The path_name of the script to use is by default:
 79	/sbin/osd_login.
 80This name can be overridden by the Kernel module parameter:
 81	objlayoutdriver.osd_login_prog
 82
 83If Kernel does not find the osd_login_prog path it will zero it out
 84and will not attempt farther logins. An admin can then write new value
 85to the objlayoutdriver.osd_login_prog Kernel parameter to re-enable it.
 86
 87The /sbin/osd_login is part of the nfs-utils package, and should usually
 88be installed on distributions that support this Kernel version.
 89
 90The API to the login script is as follows:
 91	Usage: $0 -u <URI> -o <OSDNAME> -s <SYSTEMID>
 92	Options:
 93		-u		target uri e.g. iscsi://<ip>:<port>
 94				(always exists)
 95				(More protocols can be defined in the future.
 96				 The client does not interpret this string it is
 97				 passed unchanged as received from the Server)
 98		-o		osdname of the requested target OSD
 99				(Might be empty)
100				(A string which denotes the OSD name, there is a
101				 limit of 64 chars on this string)
102		-s 		systemid of the requested target OSD
103				(Might be empty)
104				(This string, if not empty is always an hex
105				 representation of the 20 bytes osd_system_id)
106
107blocks-layout setup
108-------------------
109
110TODO: Document the setup needs of the blocks layout driver