Copying compressed file via SSH to Hadoop HDFS

By On October 16, 2009 · Leave a Comment

Had probably re-invented the way of copying gzipped files via SSH to Hadoop’s HDFS:

gzip -c file.txt | ssh hadoop.gateway.host ‘gunzip -cf – | hdfs -put – input/file.txt’

And all the way back:

ssh hadoop.gateway.host ‘hdfs -cat output/result.txt/* | gzip -c’ | gunzip -c – > resulttxt
Tagged with: •
 
If you enjoyed this article, please consider sharing it!
Reddit Facebook Twitter Delicious

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="">