iPhone Stanza Recovery

With a recent update of the iphone OS, a jailbreak here and there saw my Stanza library lost. Most of the books were sourced from online catalogs, some of which could be easily recovered, but this was a pain. So I set out with the aim of recovering and backing up my library for future use. Also some of the solutions I found do not name the books properly, so when importing to Calibre it messes up. I have more or less got around this.

The first part was easy, Mac, iTunes and iPhone backup. Check, this will also work if you want to save your iPhone library to your PC. Anyway that's enough rambling, lets get on with the good stuff.

1. Mac - all the tools mentioned are available for Windows, I love command line, so you are on your own (with Cygwin) if you choose that route.
2. iTunes and a backup / iPhone Jailbroken. For Mac you can use "iPhone Backup Extractor" Its free, easy and does the job. I'll let you work it out, likewise use Diskaid to get to your Stanza library, again I'll leave the discovery up to you. If there is interest I will help out.
3. Calibre This is where we will import the books to. This is not a necessity (just makes it easier)
4. There are plenty of tutorials as well using: CalibredbExtract This again is a must if you plan on hosting / Dropboxing Catalogs.

Once you have your extracted Stanza library, you should see something like this:


drwxr-xr-x 393 user user 13362 20 Dec 16:38 0
drwxr-xr-x 393 user user 13362 20 Dec 16:40 1
drwxr-xr-x 392 user user 13328 20 Dec 16:41 2
drwxr-xr-x 389 user user 13226 20 Dec 16:43 3
drwxr-xr-x 394 user user 13396 20 Dec 16:44 4
drwxr-xr-x 393 user user 13362 20 Dec 16:45 5
drwxr-xr-x 390 user user 13260 20 Dec 16:46 6
drwxr-xr-x 397 user user 13498 20 Dec 16:48 7
drwxr-xr-x 385 user user 13090 20 Dec 16:49 8
drwxr-xr-x 387 user user 13158 20 Dec 16:50 9


These are where the books are stored. The epub, is inside each of these folders unpacked as .explode folder. What we need to do is pack it back up, the easy part thanks to: This Blog. The zip part I am grateful for, as well as figuring it out, saved me time. That blog tells you how to do it for each book, with a for loop, some creative shell scripting we can convert all the books back to epub. 1st part done, the next is to rename the and give it a more meaningful .epub name. A bit of searching the xml, sed and awk and we get there.

The script, I copied into each folder and ran from there, and then issued a mv *.epub ../books to put them into one folder. Again if people want me to alter / provide a more rounded solution e-mail me.

for file in $( ls -1 | grep explode )
do
    filer=$( echo ${file} | sed 's/explode//g' )
    zip -Xr9D ${filer}epub mimetype ./${file}/*

    if [ -f ./${file}/OEBPS/content.opf ]
    then 
        TITLE=$(cat ./${file}/OEBPS/content.opf | egrep "dc:title" | sed 's/<\/.*>//g' | sed 's/.*<.*>//g' )
        CREATOR=$(cat ./${file}/OEBPS/content.opf | egrep "dc:creator" | sed 's/<\/.*>//g' | sed 's/.*<.*>//g' | awk -F, ' { print $2 " " $1 } ')
        mv ${filer}epub "${TITLE} - ${CREATOR}.epub"
    fi
done

Thats it, simple, easy, to the point and gets the job done. With a little searching, creative license you can turn the script into a recover, build and publish and make your own repository! That's for another day though.

Working on a few of the Cydgets...

Facebook and Twitter should now be working and no strange things happening!

24Cydget (my favourite) is being worked on and should be better in the next few min. Am woking on a few other things with regards to 24 (good times).

Had a few requests - will look into them, one I am interested in, and while Elements will provide it, sometimes just one Element :-) is useful.

Any issues with the repo should be sorted (new versions - better make process now) - repo refreshed so mismatch should be gone.

Make Cydia Repo Script

Not sure if this is useful or not, but if you find yourself following the tutorials and want a simple ne click script to create your Packages.bz2 and deb files feel free to use the following script. I have the following structure:

/repo/apps/

containing all the apps

/repo/uploads/deb 

Which has the packages and deb files.

cd apps
find . -name .DS_Store -exec rm {} \;
for app in $( ls -1 . )
do
    dpkg -b ${app}
    mv *.deb ../upload/deb
done

cd ../upload/
rm Packages.bz2
dpkg-scanpackages . /dev/null > Packages
bzip2 Packages

Script should be placed (in the above example) in:

/repo/

Cydia Repo for Cydget's

OK. First repo for iPhone, contains 2 Cydget's:

Twitter
Facebook

Basically just loads each respective page. I am investigating the issue with the Twitter not showing the mobile version.

Repo:

http://cranie.com/repo/

Simple!

Get cydia packages from a repository

Ok, not sure how useful this is, or how it will be taken by repo hosts, but, I find, especially when restoring, that having the packages on my computer is very useful!

Script, call it something like getcydia.sh, and run as follows:

getcydia.sh cydia.reponame.com foldertostoredebs

Script is here:

#--- START ---
REPO=${1}
FOLDER=${2}

mkdir ${FOLDER}
cd ${FOLDER}
wget -U "Telesphoreo APT-HTTP/1.0.98" ${REPO}/Packages.bz2
bunzip2 Packages.bz2
cat Packages | grep Filename | sed 's/Filename: //g' | awk -v REPO=${REPO} ' { print "wget -nc -U \"Telesphoreo APT-HTTP/1.0.98\" " REPO "/" $0 } ' | sh
#--- END ---

http://pastie.org/646073

Restore from iPhone backup

3GS jailbreak is now out. I have just updated my phone and done a restore using iTunes. Needless to say this leaves a lot of junk left around. So I decided to restore again and set up my phone as a new one. I synced all my music, apps, vids and, as expected, none of the application save data was available.

So to the restore script. If you made a backup using the script below (and done all) you can run the following script to restore your data).

Run as:

./restore.sh 20091002

Where the 20091002 is the date folder of your backup. This is the first cut of the script, so its just to get the data onto the phone, it will be integrated into the main script. You will need to alter the IP to that of your iPhone:

#--- START ---

USER=mobile
IP=192.168.2.3
export SSH="ssh -i ./iphone ${USER}@${IP}"
export SCP="scp -r -i ./iphone ${USER}@${IP}"
export SCPR="scp -r -i ./iphone "
export SCPS=" ${USER}@${IP}"
SOURCE=${1}

for file in $( ls -1 ${SOURCE} | grep \.app )
do
    APP=$( echo ${file} | sed 's/§/ /g' )
    RDIR=$( ${SSH} "cd /var/mobile/Applications/*/${APP}/../ 2> /dev/null ; pwd" )
    if [ ${RDIR} != "/private/var/mobile" ]
    then
        echo "Restoring ${file}"
        ${SCPR} ${SOURCE}/${file}/Documents ${SCPS}:${RDIR}
    fi
done

for file in $( ls -1 ${SOURCE}/Media )
do
    echo "Restoring ${file} "
    ${SCPR} ${SOURCE}/Media/${file}/ ${SCPS}:/var/mobile/Media/
done
#--- END ---

http://pastie.org/641700

Back

I am back so expect a few more updates soon!

saneproc - quick at a glance process summary

This simple script will show you the number of processes running under each unique user. Simple as that :-). The beauty of this script is that you can pass any stream of data with multiple values and the script will automatically process the data without any changes:

#--- START ---
#!/bin/sh

clear

ps -ef | cut -c 1-8 | sort | awk ' { print $1 } ' | sort |

awk '

BEGIN {
       print "Cranies Process Checker..."
       print " "
       printf ( "%8s%6s%9s\n", "User:", "  |   ", "No. Processes" )
       print "-------------------------"
       usertotal = 1
}

# MAIN

{

if ( $1 == stored ) {
    usertotal += 1
} else {
    if ( stored != "" ) {
        printf ( "%8s%6s%9d\n", stored, "  |   ", usertotal )
    }
    totalused += usertotal
    usertotal = 1
}

stored = $1

}

END {
       printf ( "%8s%6s%9d\n", stored, "  |   ", usertotal )
       print "-------------------------"
       print " "
       print "Total Processes: " totalused
       print "-------------------------"
       print " "
}

 '
#--- END ---

http://pastie.org/616117

Holiday

I am going to be away from a computer from tomorrow for the next week or so. I will try and post some more snippets. Going to aim for a Useful function once a week. So will get that in tomorrow and then continue after with more random scripts. If I manage to update any of my existing ones I will re-post them.

As I am using pastie.org I will update the source and just make a new post once a week with the various updates unless there are any big changes.

Twitter alerter for computer(s)

Recently I was new to the world of Twitter. Once I started I found it very addictive both for the information you could find on there and the possibilities of which it could be used.

The following script should be run via cron, if you run it on anything else other than a Mac you will need to replace the Mac specific HardwareMonitor application with one for your operating system. The code started to add functionality to allow tweeting DM's to the computer and it would reply with certain information like IP address. However, since I found TweetMyMac I have abandoned this functionality. I could be persuaded to re-visit this script an the DM aspect if there was a reason / idea.

The crontab code I use for this on my Mac is:

  15 *  *  *  *  /Users/cranie/mytwitter/twitter

basically (if you are unfamiliar with cron) run crontab -e and enter the above (replacing the /Users... part with the location to your script). This will then run every 15minutes and tweet the appropriate messages if required.

Updates to come:

• Random messages (i.e. damn its 55deg in here or Oh my I am on fire.... etc - ideas welcome)
• DM if there is any useful application for implementing this
• ???

The script:

#--- START ---
#!/bin/sh

# Limit thresholds before script alerts to twitter
CPU_LIM=55
PROC_LIM=100
DISK_LIM=95


USERNAME="your twitter username"
PASSWORD="your twitter password"
URL=http://twitter.com/statuses/update.json

CPU=$(/Applications/HardwareMonitor.app/Contents/MacOS/hwmonitor 2> /dev/null | grep "SMC CPU A DIODE" | sed -e 's/.*: //g' -e 's/ C//g')

if [ ${CPU} -ge ${CPU_LIM} ]
then
    MESSAGE="${MESSAGE}Its roasting in here: CPU Temperature is ${CPU} deg C. "
fi

DISK=$( df -m | grep "/dev/disk0s2" | awk ' { print $5 } ' | sed 's/%//g' )

if [ ${DISK} -ge ${DISK_LIM} ]
then
    MESSAGE="${MESSAGE} I'm getting full! Root disk is ${DISK}% full. "
fi

PROC=$( ps -ef | wc -l | awk ' { print $1 } ')

if [ ${PROC} -ge ${PROC_LIM} ]
then
    MESSAGE="${MESSAGE}${PROC} processes running. "
fi

UPTIME=$( uptime | awk ' { print $3 } ' | sed 's/:.*//g' )
TIMER=$( uptime | awk ' { print $5 } ' | sed -e 's/,//g' -e 's/:.*//g' )
UPTIME_UNIT=$( uptime | awk ' { print $4 } ' | sed 's/,//g' )

if [ "$(( ${UPTIME} % 8 ))" -eq 0 -a "${UPTIME_UNIT}" == "days" -a "${TIMER}" -eq 1 ]
then
    MESSAGE="${MESSAGE}Up ${UPTIME} ${UPTIME_UNIT}. "
fi

if [ "${MESSAGE}" != "" ]
then
     #echo "Posting: ${MESSAGE}"
     curl --basic --user "${USERNAME}:${PASSWORD}" --data-ascii "status=`echo ${MESSAGE}|tr ' ' '+'`" "${URL}" -o /dev/null
else
     echo "Nothing to post"
fi


# --- Checks for ask for info below here:
URL2=https://twitter.com/statuses/mentions.xml

#TIME_CHK=`date | awk ' { split($4,A,":") ; if ( A[2] <= 14 ) { printf("%02d:%02d:%02d\n", A[1] - 1, A[2] -15, A[3]) } else { printf("%02d:%02d:%02d\n", A[1], A[2] - 15, A[3]) } } '`
# MAC
TIME_CHK=`date | awk ' { split($5,A,":") ; if ( A[2] <= 14 ) { printf("%02d:%02d:%02d\n", A[1] - 1, A[2] -15, A[3]) } else { printf("%02d:%02d:%02d\n", A[1], A[2] - 15, A[3]) } } '`
DATE_CHK="`date +'%a %h %d'`"

RUN=`curl --basic --user "${USERNAME}:${PASSWORD}" -k ${URL2} 2> /dev/null | egrep "|" |
awk ' {
    if ( match($1,"created_at") != 0 ) {
        printf $0
    } else {
        print $0
    }
} ' |
grep "" |
sed -e 's/.*//' -e 's/<\/created_at>//' -e 's///' -e 's/<\/text>//' |
awk -v q=\' -v TIME_CHK=${TIME_CHK} -v DATE_CHK="${DATE_CHK}" ' {
    MSG_DATE = $1 " " $2 " " $3
    if ( DATE_CHK == MSG_DATE ) {
         split(TIME_CHK,T,":")
         split($4,M,":")
# +1 hour due to twitter being none BST
HOUR = M[1] + 1
         if ( HOUR >= T[1] && M[2] >= T[2] ) {
             if ( $8 == "show" ) {
                 if ( $9 == "time" ) {
                      EXTRA = EXTRA "My clock tells me its @DATE. "
                 }
                 if ( $9 == "temp" ) {
                      EXTRA = EXTRA "My temp is currently @TEMP. "
                 }
                 if ( $9 == "ip" ) {
                     EXTRA = EXTRA "Sent direct message of my IP address @IP. "
                     DM = DM "My IP address is @IP. "
                 }
             }
        }
    }
} ' `

DATE="`date`"
RUNNER="`echo ${RUN} | sed "s/@DATE/${DATE}/g"`"
RUNNER="`echo ${RUN} | sed "s/@TEMP/${CPU}/g"`"
IP=`curl http://checkip.dyndns.org/ | awk ' { print $6 } ' | sed 's/<.*//g'`
DMER="`echo ${DM} | sed "s/@IP/${IP}/g"`"
if [ "${RUNNER}" != "" ]
then
     curl --basic --user "${USERNAME}:${PASSWORD}" --data-ascii "status=`echo ${RUNNER}|tr ' ' '+'`" "${URL}" -o /dev/null
else
     echo "Nothing to post"
fi

USRL3=http://twitter.com/direct_messages/new.xml
if [ "${DMER}" != "" ]
then
     curl --basic --user "${USERNAME}:${PASSWORD}" -d "text=${DMER}&user=cranies" ${URL3} -o /dev/null
else
     echo "Nothing to post"
fi
#--- END ---

http://pastie.org/615819

Requests?

Whilst on my journey through pasting things in here which are useful to me, if there are any requests (preferably simple ones like I've posted :-) ) then let me know in the comments and I'll do my best to help out where I can.

Alternatively mail me your request and we can see if I can help out on a bigger scale.

Formatting - code in this blog

The code in this blog uses pastie.org - and it is not formatting correctly. I will fix this when I get the chance. For now all the code and descriptions are available. So try to enjoy :-)

Useful functions part1

This is the first in useful functions for shell scripts. This is useful when deploying to several servers at different versions / distributions. It allows you to check all the main programs are available to your script before it runs:

#--- START ---
cmd_check() { 
    x=0 
    for cmd in $@ 
    do 
        which ${cmd} 2>&1 > /dev/null 
        if [ $? -ne 0 ] 
        then 
            echo "${cmd} Not Found. Either install command or add path location to the PATH variable" 
            x=$(( ${x} + 1 )) 
        fi 
    done 
    if [ ${x} -ne 0 ] 
    then 
        exit ${x} 
    fi 
} 

cmd_check pax mkfifo ssh rm echo cat 
#--- END ---

http://pastie.org/614974

When running call the functions cmd_check with all the program names as arguments. If they are not available / in the PATH then the script will echo out an error and exit.

scp large directories / lots of files easily

Recently I came across an issue where I needed to scp hundreds of files and directories totalling several hundred GB. The easiest way was to scp the folders one by one, but some files would be too large for SCP (AIX) to deal with. Also there is no additional compression (aside from what scp provides) for this. The solution was to pax the folders on the fly to a fifo file, (gzip not yet implemented but will be added soon), scp the fifo file (not scp but ssh due to scp limitations) and then on the remote side to extract the files. This is easy to do manually, but very laborious, logging on to each server and running the make fifo's, running the commands in the correct order.

To simplify this I have made a script to automate each step. The only requirement is to have SSH keys set up from the source server to the target server. This is a god method to transfer large amounts of data or lots of files, it is overkill for simple operations.

#--- START ---
#!/usr/bin/ksh 

echo " 
Unrestricted SCP. 
" 

cmd_check() { 
    x=0 
    for cmd in $@ 
    do 
        which ${cmd} 2>&1 > /dev/null 
        if [ $? -ne 0 ] 
        then 
            echo "${cmd} Not Found. Either install command or add path location to the PATH variable" 
            x=$(( ${x} + 1 )) 
        fi 
    done 
    if [ ${x} -ne 0 ] 
    then 
        exit ${x} 
    fi 
} 

cmd_check pax mkfifo ssh rm echo cat 

if [ $# != 4 ] 
then 
    echo "--------------------------------------------------------------------------------" 
    echo "- This script will copy a directory and all the folders and files.             -" 
    echo "- Unlike SCP / tar and gzip there are no file restrictions.                    -" 
    echo "- All transfers / and extracts are done on the go, so there is no space        -" 
    echo "- requirements to land the data                                                -" 
    echo "--------------------------------------------------------------------------------" 
    echo " " 
    echo "Usage: firo_scp.ksh    " 
    echo " " 
    exit 1 
fi 

DIR=${1} 
DIRNAME=`echo ${DIR} | sed 's/.*\///g'` 
USERNAME=${2} 
SERVER=${3} 
RPATH=${4} 
FIFO=./FSCP.$$ 

echo "Making a fifo file" 
mkfifo -m 644 ${FIFO} 

echo "Initiating pax to ${FIFO} fifo for ${DIR}" 
pax -wf ${FIFO} -x pax ${DIR} & 

echo "Making remote fifo file: Authentication Successful should be displayed" 
ssh ${USERNAME}@${SERVER} "mkfifo ${RPATH}/${DIRNAME}.ar" 

echo "catting the local ${FIFO} fifo into remote fifo file: Authentication Successful should be displayed" 
cat ${FIFO} | ssh ${USERNAME}@${SERVER} "cat > ${RPATH}/${DIRNAME}.ar" & 

echo "Initiating remote extract on fifo file: Authentication Successful should be displayed" 
ssh ${USERNAME}@${SERVER} "cat ${RPATH}/${DIRNAME}.ar | pax -rxpax" 

echo "Removing remote fifo file: Authentication Successful should be displayed" 
ssh ${USERNAME}@${SERVER} "rm ${RPATH}/${DIRNAME}.ar" 

echo "Removing local ${FIFO} fifo" 
rm ${FIFO}
#--- END ----

http://pastie.org/614963

iPhone Backup Script

If you jailbreak your iPhone there are multiple ways of doing so. One method requires you to restore your phone, other issues can arise from some of the jailbreak specific applications when upgrading firmware. The best method when upgrading in this situation is to do a clean install and re-apply your jailbreak software (something like package backup can help in this situation). To re-apply the core settings such as SMS / Voicemail / AddressBook can be more hassle.

To get around this I have started to write a backup script (Mac / *nix specific). This will, via SSH, backups installed applications Documents folders, all core settings like SMS / AddressBook / Calendar etc. You will then be able to restore set up the phone as a new phone and then sync all apps and media. Once done you will be able to use the restore function of the script to apply the backed up data.

The script is work in progress and does need to be tidied up a little. The restore function is being written (testing to be done when next firmware / jailbreak is release :-) )

The script and instructions provided below:

• Run the backup script - this should be used in conjunction with iTunes. If anything fails restore from iTunes backup. To run the backup script SSH will need to be turned on, on the phone. You will need the mobile (or root if you prefer) password and the IP address of the phone.
• Run the backup script as follows:
• ./backup.sh -i 192.168.2.3 -u mobile -s
• This will setup the SSH keys and ensure everything is setup for the script to work.
• To run the backups issue the following:
• ./backup.sh -i 192.168.2.3 -u mobile backup
• After you have set up an synced the iPhone you will be able to use:
• ./backup.sh -i 192.168.2.3 -u mobile -d 20090911 restore
• There are options the script can take -a for application only backup, -c for core components and other features will be added as I require them or users request them in the comments!

The script:

#--- START ---
#!/bin/sh
################################################################################
# TITLE: iPhone Backup Script
# AUTHOR: Cranie
# DATE: 2009-08-29
# UPDATED: 
################################################################################

DATE=`date +%Y%m%d`
TMPAPP=/tmp/iphone_backup.$$

usage() {
    echo "    NAME
        ps -- process status"
    echo "  backup.sh -u username -i 127.0.0.1 -c -a -A backup --- is c or a or A not and!"
    echo "  backup.sh -u username -i 127.0.0.1 -o output_folder restore "
    echo "Initial Setup Run:"
    echo "  backup.sh -s -i 127.0.0.1 -u mobile"
    exit 1
}

echo "Thanks For Using iPhone Backup"

backup() {
    echo "Starting backup"
    if [ ! -d ${DATE} ]
    then
        mkdir ${DATE}
    else
        echo "Backup for today exists. Please remove / rename"
        exit 1
    fi
    if [ "${TYPE}" == "apps" -o "${TYPE}" == "" ]
    then
        ${SSH} "find /var/mobile/Applications -type d -name \*.app -ls | sed -e 's/.*Applications\///g' -e 's/ /§/g'" > ${TMPAPP}
        for file in $( cat ${TMPAPP} )
        do
             APP=`echo ${file} | sed 's/.*\///g'`
             APPID=`echo ${file} | sed 's/\/.*//g'`
             echo "Archiving ${APP}"
             mkdir ${DATE}/${APP}
             ${SCP}:/var/mobile/Applications/${APPID}/Documents ${DATE}/${APP}
         done
         rm ${TMPAPP}
    fi
    if [ "${TYPE}" == "core" -o "${TYPE}" == "" ]
    then
        mkdir ${DATE}/Library
        for core in $( echo "SMS Voicemail Notes AddressBook Calendar" )
        do
            mkdir ${DATE}/Library/${core}
            ${SCP}:/var/mobile/Library/${core} ${DATE}/Library/
        done

        mkdir ${DATE}/Media
        for core in $( echo "Audio DCIM Photos Recordings Videos" )
        do
            mkdir ${DATE}/Media/${core}
            ${SCP}:/var/mobile/Media/${core} ${DATE}/Media/
        done
    fi
}

setup() {
    echo "Setting up ssh / checking dependencies"
    echo "At passphrase prompts just press enter"
    if [ -f ./iphone ]
    then
        rm ./iphone 2> /dev/null
        rm ./iphone.pub 2> /dev/null
    fi
    ssh-keygen -t rsa -b 2048 -q -f iphone
    cat ./iphone.pub | ${SSH} "cat > ~/iphone.pub
        if [ ! -d /var/mobile/.ssh ]
        then
            mkdir /var/mobile/.ssh
        fi
        mv ~/iphone.pub ~/.ssh/authorized_keys"
}


if [ $? != 0 ]
then
        usage
fi

while getopts ":hcaAsu:i:" options
do
    case $options in
        s ) echo "Running initial setup. This can be run multiple times but only needed once"
            export RUN=setup
            ;;
        A ) export TYPE=""
            ;;
        a ) export TYPE=apps
            ;;
        c ) export TYPE=core
            ;;
        u ) export USER=${OPTARG}
            ;;
        i ) export IP=${OPTARG}
            export SSH="ssh -i ./iphone ${USER}@${IP}"
            export SCP="scp -r -i ./iphone ${USER}@${IP}"
            ;;
        h ) usage
            ;;
        * ) usage
            ;;
    esac
done
shift $(($OPTIND - 1))

if [ "${RUN}" == "setup" ]
then
    setup ${IP}
fi

if [ "${1}" == "backup" ]
then
    echo "Starting ${TYPE} Backup..."
    backup
fi
#--- END ---

http://pastie.org/614729