HadoopClient

A utility to wrap Hadoop console command

class bigflow.util.hadoop_client.HadoopClient(client_path, config_path)

A wrapper class of Hadoop console command

内部类，请用户不要使用该类，未来不保证接口兼容，请直接使用hadoop-client。

参数:
  • client_path (str) -- path of Hadoop client(usually 'hadoop' executable file)
  • config_path (str) -- path of Hadoop configuration
fs_dus(path, args=None)

Wraps console command 'hadoop fs -dus <path>'

参数:path (str) -- path to get size
返回:path size
返回类型:long
fs_get(source, target, args=None)

Wraps console command 'hadoop fs -get <source> <target>'

参数:
  • source (str) -- path of source
  • target (str) -- path of target
fs_mkdir(path, args=None)

Wraps console command 'hadoop fs -mkdir -p <path>'

参数:path (str) -- path to be created
fs_mv(source, target, args=None)

Wraps console command 'hadoop fs -mv <source> <target>'

参数:
  • source (str) -- path of source
  • target (str) -- path of target
fs_put(source, target, args=None, need_mkdir=True)

Wraps console command 'hadoop fs -put <source> <target>'

参数:
  • source (str) -- path of source
  • target (str) -- path of target
fs_rmr(path, args=None)

Wraps console command 'hadoop fs -rmr <path>'

参数:path (str) -- path to be removed
fs_test(path, args=None)

Wraps console command 'hadoop fs -test -e <path>'

参数:path (str) -- path to test
返回:if path exist
返回类型:bool
bigflow.util.hadoop_client.extract_fs_name_from_path(path)

Get fs.defaultFS from path like "hdfs://abcde:22222/a/b/c"