HDFS
HDFS is the platform file system. There are essentially two ways of accessing this file system :
- using HDFS command line
- through other services like MapReduce or Spark
- with a NFS mount point
A typical workflow would see users upload their data on HDFS using the command line and then process it using Mapreduce or Spark, possibly creating additional data stored on HDFS.
HDFS command line
The command used to interact with the cluster is hdfs dfs. This command has several sub-commands depending on which operations to do.
For example :
| Command | Operation |
|---|---|
hdfs dfs -ls <path> |
Lists the files and directories located in <path> |
hdfs dfs -copyFromLocal <localsrc> ... <dst> |
Uploads one or multiple local files/directories in dst hdfs directory |
hdfs dfs -copyToLocal <src> ... <localdst> |
Downloads one or multiple distributed files/directories to the local file system. Beware that the local filesystem might not have the storage capacity to hold the distributed data |
hdfs dfs -rm <path> |
Delete file at <path> |
hdfs dfs -mkdir <path> |
Create a directory <path> |
hdfs dfs -chown <user>:<group> <path> |
Change the ownership of a file or directory |
hdfs dfs -chmod <mode> <path> |
Change the access rights to a file or directory |
hdfs dfs -help <command> |
Prints the reference for operation <command> |
The full list of commands and their options can be printed by typing hdfs dfs -help. To get additional detail on a command, use hdfs dfs -help <command>.
HDFS NFS mount point
The distributed filesystem can be accessed via an nfs mount point on the gateway in /mnt/hdfs_nfs.
You can browse and interact with HDFS through this mount point with classic unix commands such as ls or mv.
However, all read or written data go through a single NFS server, inducing slow read/write in case of large volume of data. In those cases, prefer using command -copyFromLocal and -copyToLocal or application such as MapReduce or Spark to interact with large volume of data.