HDFS

HDFS is the platform file system. There are essentially two ways of accessing this file system :

  • using HDFS command line
  • through other services like MapReduce or Spark

A typical workflow would see users upload their data on HDFS using the command line and then process it using Mapreduce or Spark, possibly creating additional data stored on HDFS.

HDFS command line

The command used to interact with the cluster is hdfs dfs. This command has several sub-commands depending on which operations to do. For example :

Command Operation
hdfs dfs -ls <path> Lists the files and directories located in <path>
hdfs dfs -copyFromLocal <localsrc> ... <dst> Uploads one or multiple local files/directories in dst hdfs directory
hdfs dfs -copyToLocal <src> ... <localdst> Downloads one or multiple distributed files/directories to the local file system. Beware that the local filesystem might not have the storage capacity to hold the distributed data
hdfs dfs -rm <path> Delete file at <path>
hdfs dfs -mkdir <path> Create a directory <path>
hdfs dfs -chown <user>:<group> <path> Change the ownership of a file or directory
hdfs dfs -chmod <mode> <path> Change the access rights to a file or directory
hdfs dfs -help <command> Prints the reference for operation <command>

The full list of commands and their options can be printed by typing hdfs dfs -help. To get additional detail on a command, use hdfs dfs -help <command>.