Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Using the standard Linux command rm to delete multiple files on a Lustre filesystem is not recommended. The rm command will generate a stat() operation on for each file it removes, meaning all of the attributes of a file (filetype, owner, permission, modification time, etc.) will be returned from the metadata server. A large number of stat() operations can place an increased load on the metadata server, resulting in lower performance and instabilities with the filesystem.

Instead, use munlink, a Lustre-specific command that will simply delete a file without performing a stat() operation. Below is an example of how to remove files using our recommended approach which consists of two steps.

The first step deletes all the files and soft links within a directory (and its subdirectories) with the use of munlink:

Code Block
find -P ./processor0 -type f -print0 -o -type l -print0 | xargs -0 munlink

Here is an overview of each step in that command:

  • find find 
    is a command that will search the indicated directory (and subdirectories within). The syntax defines a search for files (or and soft links).
  • -P
    restricts the search within the indicated directory tree and forces NO dereference of symbolic links. This warranties that the find command will not look for files within the links.
  • ./processor0
    is the directory to be deletedfrom which the search will start
  • -type f -print0 -o -type l -print0
    The This indicates that the find command will search the directory processor0 and all subdirectories for anything that is a file (-type f) or (-o) a soft link (-type l), therefore omitting to directories (-type d, this is the lower letter l).
  • -print0

    This dictates how the returned list of files is formatted (indicates the format of the result of the "find" command. This particular format is able to catch strange file names), and ensures that they are in the correct format readable for the next step (xargs -0). It needs to appear twice due to the "or" command option (-o) explained in the previous bullet point.


  • The list of files or links found by the find command is piped to the xargs commandPipe command ( represented by a single pipe line: |) makes the output of "find" to serve as input to the next command (xargs in this case).
  • xargs -0 munlink
    xargs will then convert the received list of files, line by line, into an argument for whatever command is specified at the end (in our this case: munlink). The -0 flag is related to the format of the listed files; if you use -print0 in the find command you must use -0 in the xargs command.
  • munlink
    deletes each file and soft link in the list without overloading the metadata server

The second step is to remove the empty directories and subdirectories in the tree. Once all of the files are and soft links were deleted, you can remove the empty directories with a similar command:

Code Block
find ./processor0 -depth -type d -empty -delete

Again, the find command will search the directory processor0 and all subdirectories for any empty directories (-type d -empty) and delete them. The depth flag instructs find to process each directory's contents before the directory itself, then the most distant branches in the directory tree will be procesed first. The -delete action also implies -depth. To process a directory' contents before the directory itself. Please note that the flags passed to find are evaluated as an expression, so if you pass -delete first, find will attempt to delete below your starting directory.NOTE:

Note

We strongly recommend that all OpenFOAM users make use of munlink for deleting simulation output. Doing so will not only help keep a stable filesystem, but also provide a speedup over using the standard rm -rf command.


Content by Label
showLabelsfalse
max5
spacesUS
showSpacefalse
sortmodified
reversetrue
typepage
cqllabel = "kb-best-practices-article" and type = "page" and space = "US"
labelskb-how-to-article

...