Intended Audience
The post is written for the following audiences: me (in case I forgot how I got this working someday), the next person that manages my network, and other Pogoplug users that want to set up rsync and get a general idea of how to add features to the Pogoplugs.
This will be, at times, ridiculously detailed, and at other times it will be frustrating vague. Sorry, but remember that Google is your friend, as is the Pogoplug forums and the Optware community.
Background
I bought a Pogoplug (on sale) for myself and found it easy enough to use that I thought it would be a workable remote file access solution for church. The idea was, instead of using a file server for shared files I would use the Pogo, and back the drive up regularly. So I picked up a Pogoplug Biz for church.
It was not until after I started using it that I discovered
even LAN users had terrible file transfer speeds. I traded some messageswith Cloud Engine’s (who makes the Pogoplug) CTO, and there is no fix and he doesn't expect to have one anytime soon (I read that as ‘never’). Oh well. (To be fair, their CTO did say that, in aggravated cases, they would work out some refund. But I’ve had the device for a year… so I didn’t pursue it.)
Anyway, if I can't get fast, usable two-way access to the Pogoplug drives, then I wanted to at least give my users access to the existing shared drives. I’d get the files from my server to the Pogoplug and then let users access the web site or the Pogodrive software.
I tried to use the Pogoplug Software version on the server, but it only worked locally, not from outside via the internet (I don’t know why). AND, likely due to the beta nature of the software, it tended to fill up my hard disk with temp files. I considered using the Pogoplug "Active Copy" feature, by installing the software version on the file server and leaving the desktop logged in. However, Active Copy does not propagate file deletions or renames. I got tired of trying to make a supposedly awesome product actually do what I wanted to do. Pogoplug’s customer service is not interested in helping me with my problem, so I decided to examine the other feature that attracted me to Pogoplugs in the first place, it's "openness".
My half-baked solution is to use my existing local network file shares (let’s call is drive S for “shared”) but to mirror the share on the Pogoplug so my users could access files remotely (via the Pogoplug P drive or
http://my.pogoplug.com). This would allow my users, at least, read-only access of the files in the office. Maybe I’ll be able to rig some sort of two-way sync, but that is for another day. Since I want to synchronize file changes and deletions one-way only,
rsync came immediately to mind. Thus began this adventure.
Cloud Engines touts the
open source nature of the Pogoplug. Since the Pogoplug runs a flavor of Linux, has SSH access and
has Busybox installed, it makes it
relatively easy to install Optware. It doesn’t hurt that the an informed user like OddballHero is fond of both Optware and Pogoplug. Optware is a package manager, so it helps you download and install programs like rsync.
Some Assumptions and a Word of Warning
I'm assuming you are using either a Windows computer or have access to a Linux PC. Sorry, I don't do Macs; they're not in the scope of my budget. But most of the Linux stuff should be accessible to Mac users.
There is a LOT of assumed knowledge. For example, I am assuming your Pogoplug is activated and you can get to the web interface. A knowledge of editing text files in Linux will be required at some point. As I am not sure who may have to maintain the system I set up at church, I am going to explain far more than you (as the reader) may find necessary. Or maybe less than you’d like. Just skip any "well duh" sections you come across. On the other hand, I will not explain EVERY possible way to do something. To be blunt, if you are unable to follow these instructions with some help from Google you may want to think twice about doing this.
Enable SSH On Your Pogoplug
SSH (short for Secure SHell) allows you to access to a remote networked computer using a secure (encrypted) connection.
SSH is like a secure version of telnet.
Originally, the Pogoplug shipped with SSH enabled, and a published root user password. Now, by default, SSH is NOT enabled. To
enable SSH on your Pogoplug, log on to http://my.pogoplug.com, click the Settings link, select Security Settings, and then select the checkbox "Enable SSH for this Pogoplug enabled device." You will be asked to type in a password for the root user, WRITE THIS DOWN.
Locate Your Pogoplug
Your Pogoplug gets an IP address dynamically, using DHCP. That is to say, the IP address of your Pogoplug could change on a regular basis.
To locate the IP address, look at the bottom of your Pogoplug for the MAC hardware address. This will be in the format of 16 characters, in 8 pairs, separated by a colon (":"). The characters will be 0-9 and a-f. Write this down. For your Pogodrive, the MAC address should begin with 00:25:31
Now, you'll need to look at your DHCP server and look at the client leases. One of the leases will have the MAC hardware address of your Pogoplug listed. That entry will probably not have a name, but it will have an IP address. For an IPv4 address, the address will look like 4 numbers separated by dots, like 192.168.1.101 or something similar.
To make sure it IS really your Pogoplug, try to ping the IP address. (Google it if you don't know how.) Once you have managed to ping the IP address successfully, examine your ARP table for the MAC hardware address. The command to type is "
arp -a". If you find a MAC address in the ARP table that matches up to your Pogoplug, you can move on to SSH. For this document, I will assume your Pogoplug’s IP address is 192.168.1.101.
Establishing SSH Access
Now that we have found the Pogoplug on the network, we want to login with the SSH service we just enabled. For Windows users,
try downloading PuTTY. If you are on a Linux box, just run the SSH command. In either case, use try using a SSH client to the IP address of the Pogoplug. You should receive a login prompt. Use the SSH password you wrote down earlier.
USB Device Names on the Pogoplug
For the purposes of this document, I'm going to suggest (and assume) that you'll ONLY plug in the USB flash drive for the install. Don't use other drives while you set this all up.
The USB drives on the Pogoplug are labeled
sd?1 where ? is alphabetical (
sda1,
sdb1,
sdc1…), depending on discovery order. For the purposes of this document, I'm going to suggest (and assume) that you'll ONLY plug in the USB flash drive for the install of Optware, rsync, whatever. Don't use other drives while you set this all up. When we’re all done, I’ll help you locate and mount the dedicated to Optware.
For the sake of easy identification, we'll use the front port for our software installation (Optware).
Insert an empty USB drive into the front of the Pogoplug. It can be small, but I would recommend you use a USB 2.0 drive. Everything on this drive will be erased, and you'll be leaving it in the Pogoplug, so make sure it is a drive you can spare.
Dedicating a USB Stick for Optware
I am going to assume you want to run the Optware and other additions on a USB stick. I recommend this because, in theory, there will be less wear-and-tear on the Pogoplug's internal flash memory.
For reference, you can read this Pogoplug article about “
How to Partition and Format a USB Drive” for help.
Once you have the USB flash drive plugged into the front, make sure it is recognized as a functional drive. You can do this by running the "
mount" command and looking for an entry for
/tmp/.cemnt/sda1 or go to the web interface at
http://my.pogoplug.com and find your Pogoplug’s attached flash drive. Once you have verified it is properly discovered, eject the drive from the
http://my.pogoplug.com web site. (No, I haven't figured out how to eject it from SSH.)
Get mke2fs
Check and see if you have the software to format the drive already installed. Try running
mke2fs and see if it is already installed on your Pogoplug (it exists on the Pogoplug Pro, for example).
If mke2fs does not exist, you can download it with the steps below. I'm going to borrow liberally from "
ianjb" from that article I referred to earlier.
Note: the link to mke2fs is from a Pogoplug article, so it should be safe.
Note: anything in /tmp will disappear on reboot. If you end up rebooting the Pogoplug, /tmp will be empty, so you have to download things that are in /tmp all again. But that also means you can make a mess in /tmp and it will go away.
Download mke2fs to
/tmp
cd /tmp
wget http://download.capablenet.com/pogoplug/mke2fs.zip
unzip mke2fs.zip
chmod 755 mke2fs
Now that we have mke2fs on the Pogoplug, we can proceed with the format.
Create the Linux Partition
First, partition the USB drive by typing using fdisk
/sbin/fdisk /dev/sda
This puts you in the fdisk menu. Assuming your USB flash drive is ready for Windows, it has a FAT filesystem on it. We’ll delete it and create a new Linux file system.
Use the command “d” to delete partition 1.
Use the command “n” to create a new partition. When asked, create a “p” primary partition. Use the defaults for the partition number (1), first cylinder (1), and last cylinder (just press ENTER).
Use the command “w” to write the changes to the USB flash drive.
Now you will exit the fdisk menu. Creating the Linux filesystem will cause the Pogoplug to automatically remount the drive. Go back to the web interface and eject it again. (Go to
http://my.pogoplug.com, find your Pogoplug, click the eject symbol next to your drive.)
Format the USB Flash with Ext3 Filesystem
We are still in the
/tmp directory. We’ll use the mke2fs command to format the USB flash drive. I like to label mine, so the
-L OPTWARE part is optional
/tmp/mke2fs -j -L OPTWARE /dev/sda1
Mounting the USB Flash Drive As /opt
Now, the USB flash drive can be accessed at “
/dev/sda1”. Because we’re installing Optware, we’ll attach it (that is, mount it) at the folder “
/opt”. Normally the Pogoplug’s internal flash memory is read-only, so we cannot create files or folders. So we’ll make the Pogoplug internal memory available for both read and write and then create our folder.
mount -o rw,remount /
cd /
mkdir opt
mount /dev/sda1 /opt
mount / -o remount,ro
Installing Optware on the USB Flash
For this section, you can get general instructions for
installing Optware on Pogoplugs at NSLU2-Linux. I am going to change things a little to install Optware directly on the USB flash drive. We’ll install the ipkg manager, then Optware, and finally rsync.
First, you need to find the current ipkg manager by visiting this web site:
On that web page, look for the line that begins with “ipkg-opt”. The file name will look like “ipkg-opt_0.99.163-10_arm.ipk” (current as of September 2011). Make a note of this file name. We’ll download the file (with
wget) to the
/tmp folder and get things installed. Modify the “
wget” and “
tar” commands I give below with the current “ipkg-opt_xxxxxxxxxxxxxxx_arm.ipk” file name.
cd /tmp
wget http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/stable/ipkg-opt_0.99.163-10_arm.ipk
tar xvzf ipkg-opt_0.99.163-10_arm.ipk
The
tar command will extract three files, including
data.tar.gz. We’ll extract the file we want directly to the USB flash drive we mounted to
/opt earlier.
cd /
tar xvzf /tmp/data.tar.gz
Now you have Optware on your USB flash drive, living in the
/opt directory, which is the USB flash drive. Next we’ll tell the ipkg manager to get packages from nslu2-linux.org. The following command is all on one line:
echo 'src cross http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/stable' >> /opt/etc/ipkg.conf
(If you know your way around *NIX, you can just edit
/opt/etc/ipkg.conf and add the line as well.)
Update ipkg Manager
To make it easier to type commands, let’s add the new Optware commands to our environment with the
PATH variable and then update ipkg.
PATH=$PATH:/opt/bin:opt/sbin
ipkg update
Installing Nano (Optional)
The rest of these instructions requires editing text files in the Pogoplug. From here, I’ll tell you what lines to insert into text files, or give you examples of the text files, but I cannot tell you which buttons to press. I am going to suggest that you do a Google search for “nano editor” and do some reading, and visit Nano’s site at
http://www.nano-editor.org/. Otherwise, figure out how to use “
vi” which is already available to you.
You can get some
help installing Nano at this link and some
basic instructions on using Nano to edit files at this link (hi Greg).
To install Nano, type
ipkg install nano
To edit a file named
textfile.conf with Nano, type
nano textfile.conf
Installing Rsync
Finally, we can install rsync. We’ll use the ipkg manager to get it installed. You can read up on configuring rsync at
http://www.samba.org/ftp/rsync/rsyncd.conf.html but I will give you a very basic, but not secure, set up of rsync.
ipkg install rsync
Configure Rsync
You can configure rsync by editing the
/opt/etc/rsyncd.conf file, and read up on the instruction for
rsyncd.conf at
http://www.samba.org/ftp/rsync/rsyncd.conf.html. Any line that begins a “#” sign is ignored, so you can use those for comments. If you installed Nano, use the following command to edit the
/opt/etc/rsyncd.conf file:
nano /opt/etc/rsyncd.conf
Below is an example of
/opt/etc/rsyncd.conf for testing:
# rsyncd.conf
# configure according to your needs
uid = root
gid = root
use chroot = yes
max connections = 5
syslog facility = local3
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
secrets file = /opt/etc/rsyncd.secrets
[tmp]
path = /tmp
comment = Rsync of tmp
read only = no
Note: In my set up, I am using static IP addresses with no network security concerns. Because of this, my example does not use either encryption or passwords. If you have ANY intention of doing rsync over the internet or any unsecured network, look into using rsync over SSH, and set up passwords.
First Run of Rsync
I run rsync on my Pogoplug in daemon mode (to limit connections). To enable the rsync daemon, you need to edit
/opt/etc/default/rsync and change
RSYNC_ENABLE=false to
RSYNC_ENABLE=true.
It also happens that the rsync I installed wants to look for temporary files in the directory
/var/run (which did not exist on my Pogoplug). Just in case, let’s create the
/var/run directory.
cd /var
mkdir run
Note: /var is a link to /tmp/var, so nothing in here sticks around upon reboot.
Now we can start the rsync daemon and see if it works. Type in the following:
/opt/etc/init.d/S57rsyncd start
To check and see if rsync is running, type the following command and see if there is a line for rsync:
ps | grep rsync | grep -v grep
Congratulations, you are running rsync.
Sorry, this is where I start getting more vague. Remember, you can always search on Google!
Installing an Rsync Client
If you are using Linux or a Mac (OS X or later) you should already have a rsync client. For Windows, the free choices I am aware of are cwRsync, DeltaCopy, and QtdSync. I’d recommend either DeltaCopy or QtdSync, take your pick. DeltaCopy has the advantage of being well-known, but QtdSync has better support for email notification.
Testing Rsync
Let’s test the connection. Remember, I am assuming your Pogoplug has an IP address of 192.168.1.101.
If you are running Linux, test the connection with the following commands, which should list the share named “
tmp”, and the contents of
/tmp:
rsync rsync://192.168.1.101
rsync rsync://192.168.1.101/tmp
To connect with rsync from Windows (either DeltaCopy or QtdSync), set up a new “profile” and specify the server with your Pogoplug IP address. In DeltaCopy, run “
DeltaC.exe” and click “
add new profile”, type in the server IP, and click the “
…” box on the right. For QtdSync, click the round yellow “
…” button on the right next to “
User” and type in your IP address.
If the client automatically populates the directory name of “tmp” for you, you’re up and running. Now you have a basic, working rsync server on the Pogoplug.
Saving the Setup to the Pogoplug
Let’s make the settings we’ve got permanent. We’ll make the internal memory of the Pogoplug writable with the following command:
mount -o rw,remount /
You’d need to run that command any time you want to edit files that are not in
/tmp or
/opt (which is the USB flash drive). Once you run the above mount command, you can really and truly mess up the Pogoplug, so be careful. When you are done editing files, protect the Pogoplug memory again with the following command:
mount / -o remount,ro
Assuming that this is your first foray into adding things to the Pogoplug, the files
/etc/profile and
/root/.bash_profile do not exist. Using either
vi or
nano, you can edit or create these files to add some of the settings we’ve used. If you are not sure, try the following commands from the SSH prompt and see if it tells you there is no such file:
ls /etc/profile
ls /root/.bash_profile
In either case, you want the contents of
/etc/profile to look have the following lines at the top if they do not already exist:
#!/bin/bash
export PATH=/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin
And, you want
/root/.bash_profile to have the following line:
export PATH=/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin
If neither file exists, you can create them with the following commands (watch out for line wraps on your screen):
mount -o rw,remount /
echo '#!/bin/bash' > /etc/profile
echo 'export PATH=/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin' >> /etc/profile
echo '#!/bin/bash' > /root/.bash_profile
echo 'export PATH=/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin' >> /root/.bash_profile
mount / -o remount,ro
Next, we must modify the startup command, which can be found in the
/etc/init.d/rcS file. This file will already exist. Edit this file with
vi or
nano to include the following lines:
mkdir /tmp/var/run
mount /dev/sda1 /opt
/opt/etc/init.d/S57rsyncd start
First, make a backup of the
rcS file.
cd /etc/init.d
mount -o rw,remount /
cp rcS rcS.bak
mount / -o remount,ro
Assuming you’ve never edited the
/etc/init.d/rcS file before, you can use the following commands to add these lines to the end of that file. Be really, really sure there are TWO great-than signs (>>) in the commands:
mount -o rw,remount /
echo 'mkdir /tmp/var/run' >> /etc/init.d/rcS
echo 'mount /dev/sda1 /opt' >> /etc/init.d/rcS
echo '/opt/etc/init.d/S57rsyncd start' >> /etc/init.d/rcS
mount / -o remount,ro
Take a look at
/etc/init.d/rcS with the command
cat /etc/init.d/rcS
It should look like this, with the last three lines in bold that you added:
#! /bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devpts none /dev/pts
mount -t tmpfs none /tmp
mount -t usbfs none /proc/bus/usb
mkdir /tmp/var
echo "/tmp/core_%e_%t" > /proc/sys/kernel/core_pattern
hostname Pogoplug
ifconfig lo 127.0.0.1
ifconfig eth0 169.254.37.133
udhcpc -b `hostname`
#telnetd
/etc/init.d/db
/etc/init.d/hbmgr.sh start
#/bin/mount –a
mkdir /tmp/var/run
mount /dev/sda1 /opt
/opt/etc/init.d/S57rsyncd start
Rebooting and Checking Setup
Now that everything seems to be working, you can reboot the Pogoplug and make sure everything works upon reboot with just the USB flash drive installed. Type in the following commands to tell the Pogoplug to save its work and reboot:
sync
sync
reboot
At this point you will be disconnected from the Pogoplug. Give it about 30 seconds, then connect again with your SSH client. Once you are able to log in, type the
mount command and it will show you a mount table. Your screen should look like this:
rootfs on / type rootfs (rw)
/dev/root on / type jffs2 (ro)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw)
none on /tmp type tmpfs (rw)
none on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /opt type ext3 (rw,data=ordered)
/tmp/.cemnt/sda1 on /tmp/.cemnt/mnt_sda1 type ext3 (rw,nosuid,nodev,noexec,noatime,data=ordered)
If it looks like this, try your rsync client again (follow the directions from the “Testing Rsync” section above. If you can connect with a rsync client, you’re all set.
Sorry, this is where I start getting really vague. Remember, you can always search on Google!
Sending Files from A PC to the Pogoplug Rsync Share
I highly recommend that you use a small set of files for testing the rsync client to send file to
/tmp on the Pogodrive. Once you have your command all figured out, then you can add a real hard disk.
Note: NEVER try to send files via rsync to anything other than /tmp or to an attached USB drive.
Note: Understand that the rsync option "--delete" will delete hidden files on the Pogoplug. If you use "--delete" you should also add "--exclude .ceid --exclude .cedata" to your list of options.
For example, this command should be safe, even if your target is the automatically mounted Pogoplug drive:
rsync --exclude .ceid --exclude .cedata -rtv --delete /cygdrive/D/test/ rsync://192.168.1.101/tmp
Finally, when you are trying to send a folder named “test”, there is a difference between
/cygdrive/D/test and
/cygdrive/D/test/ (the “/”). A “/” at the end of a source directory copies the contents only, while no “/” copies the directory as well. For more information, see
http://ss64.com/bash/rsync.html.
Adding a Hard Disk and Changing Rsync
You should probably add another hard disk. As the second disk, it should show up as
/dev/sdb1 on the Pogoplug in SSH. Because the Pogoplug software links
/dev/sdb1 automatically to
/tmp/.cemnt/sdb1 and mounts it as
/tmp/.cemnt/mnt_sdb1, you could go back and edit the
/opt/etc/rsyncd.conf file and change the share name from
[tmp] to something more suitable, and update the path to
/tmp/.cemnt/mnt_sdb1 instead. You can restart rsync by running
/opt/etc/init.d/S57rsyncd start (if it complains, just run it again).
Also, consider making your rsync session more secure. In my setup, I’m merely sending files to the Pogoplug from the same computer all the time, and the sending computer has a static IP address. It is all within a small, trusted network. Thus I can use the “allow” and “deny” settings in the rsync share to “whitelist” my sending PC. Consider doing so as well.
hosts allow = 192.168.1.10
hosts deny = *
You may also want to configure users and passwords using the
rsyncd.secrets file. Do a Google search and you’ll find plenty of help.
A Couple of “Gotchas”
There’s a few small “gotchas”… First, sometimes upon a reboot the Pogodrive will find the other USB drives before the flash drive with Optware installed, making it
/dev/sdb1 instead of
sda1, which throws off all the startup scripts. I don’t know why. I have a solution, but that
requires a separate post here. Just remember that, when you reboot, you may have to check the
mount command to make sure that
/dev/sda1 is mounted on
/opt. If not, power off and reboot a couple of times, it will eventually get it right.
Second, your Optware drive will show up in the Pogoplug web site at
http://my.pogoplug.com – if you don’t want to accidentally put files in there, you can eject it from the web and rsync will keep working.
What’s Next
For homework, here are some things you can find on Google that may be helpful.
- You may want to try to configure the Pogoplug with a static IP. You can do this with a DHCP reservation, or you can use the ifconfig command on the Pogoplug to add a VLAN.
- Configure rsync to run over SSH for the encryption. Add passwords to rsync shares.
- Figure out how to schedule the rsync command so that the Pogoplug stays current.
- Figure out how to get email notices if your rsync command does not work right.
- Install Samba on your Pogoplug.
If you come across a good way to do two-sync, let me know! I’m thinking of using Allways Sync, but not so sure yet.