Copying compressed file via SSH to Hadoop HDFS

commentsNo comments    usersigizmund

October 16th, 2009 

Had probably re-invented the way of copying gzipped files via SSH to Hadoop’s HDFS:

gzip -c file.txt | ssh hadoop.gateway.host ‘gunzip -cf – | hdfs -put – input/file.txt’

And all the way back:

ssh hadoop.gateway.host ‘hdfs -cat output/result.txt/* | gzip -c’ | gunzip -c – > resulttxt

Posted in: Uncategorized

Tags: ,



This website uses IntenseDebate comments, but they are not currently loaded because either your browser doesn't support JavaScript, or they didn't load fast enough.

addLeave a comment