HDFS is the platform file system. There are essentially two ways of accessing this file system :
- using HDFS command line
- through other services like MapReduce or Spark
- with a NFS mount point
A typical workflow would see users upload their data on HDFS using the command line and then process it using Mapreduce or Spark, possibly creating additional data stored on HDFS.
HDFS command line
The command used to interact with the cluster is
hdfs dfs. This command has several sub-commands depending on which operations to do.
For example :
||Lists the files and directories located in
||Uploads one or multiple local files/directories in dst hdfs directory|
||Downloads one or multiple distributed files/directories to the local file system. Beware that the local filesystem might not have the storage capacity to hold the distributed data|
||Delete file at
||Create a directory
||Change the ownership of a file or directory|
||Change the access rights to a file or directory|
||Prints the reference for operation
The full list of commands and their options can be printed by typing
hdfs dfs -help. To get additional detail on a command, use
hdfs dfs -help <command>.
HDFS NFS mount point
The distributed filesystem can be accessed via an nfs mount point on the gateway in
You can browse and interact with HDFS through this mount point with classic unix commands such as
However, all read or written data go through a single NFS server, inducing slow read/write in case of large volume of data. In those cases, prefer using command
-copyToLocal or application such as MapReduce or Spark to interact with large volume of data.