scp large directories / lots of files easily

Recently I came across an issue where I needed to scp hundreds of files and directories totalling several hundred GB. The easiest way was to scp the folders one by one, but some files would be too large for SCP (AIX) to deal with. Also there is no additional compression (aside from what scp provides) for this. The solution was to pax the folders on the fly to a fifo file, (gzip not yet implemented but will be added soon), scp the fifo file (not scp but ssh due to scp limitations) and then on the remote side to extract the files. This is easy to do manually, but very laborious, logging on to each server and running the make fifo's, running the commands in the correct order.

To simplify this I have made a script to automate each step. The only requirement is to have SSH keys set up from the source server to the target server. This is a god method to transfer large amounts of data or lots of files, it is overkill for simple operations.

#--- START ---
#!/usr/bin/ksh 

echo " 
Unrestricted SCP. 
" 

cmd_check() { 
    x=0 
    for cmd in $@ 
    do 
        which ${cmd} 2>&1 > /dev/null 
        if [ $? -ne 0 ] 
        then 
            echo "${cmd} Not Found. Either install command or add path location to the PATH variable" 
            x=$(( ${x} + 1 )) 
        fi 
    done 
    if [ ${x} -ne 0 ] 
    then 
        exit ${x} 
    fi 
} 

cmd_check pax mkfifo ssh rm echo cat 

if [ $# != 4 ] 
then 
    echo "--------------------------------------------------------------------------------" 
    echo "- This script will copy a directory and all the folders and files.             -" 
    echo "- Unlike SCP / tar and gzip there are no file restrictions.                    -" 
    echo "- All transfers / and extracts are done on the go, so there is no space        -" 
    echo "- requirements to land the data                                                -" 
    echo "--------------------------------------------------------------------------------" 
    echo " " 
    echo "Usage: firo_scp.ksh    " 
    echo " " 
    exit 1 
fi 

DIR=${1} 
DIRNAME=`echo ${DIR} | sed 's/.*\///g'` 
USERNAME=${2} 
SERVER=${3} 
RPATH=${4} 
FIFO=./FSCP.$$ 

echo "Making a fifo file" 
mkfifo -m 644 ${FIFO} 

echo "Initiating pax to ${FIFO} fifo for ${DIR}" 
pax -wf ${FIFO} -x pax ${DIR} & 

echo "Making remote fifo file: Authentication Successful should be displayed" 
ssh ${USERNAME}@${SERVER} "mkfifo ${RPATH}/${DIRNAME}.ar" 

echo "catting the local ${FIFO} fifo into remote fifo file: Authentication Successful should be displayed" 
cat ${FIFO} | ssh ${USERNAME}@${SERVER} "cat > ${RPATH}/${DIRNAME}.ar" & 

echo "Initiating remote extract on fifo file: Authentication Successful should be displayed" 
ssh ${USERNAME}@${SERVER} "cat ${RPATH}/${DIRNAME}.ar | pax -rxpax" 

echo "Removing remote fifo file: Authentication Successful should be displayed" 
ssh ${USERNAME}@${SERVER} "rm ${RPATH}/${DIRNAME}.ar" 

echo "Removing local ${FIFO} fifo" 
rm ${FIFO}
#--- END ----

http://pastie.org/614963