Tag Archives: openindiana

Maximizing rsync performance between Linux and Solaris

I now am a proud owner of an OpenIndiana server, and I’ve been moving files to it over gigabit ethernet for the past few hours. During this time, I’ve made some important realizations, and I figured I’d note them here for everyone’s benefit.  My transfers started off at about 10MB/s sustained, which is right around 100Mbit/s speeds, but on a gigabit network.

1. Ethernet Cables

Something we don’t think about too often these days is the type/quality of Ethernet cable we’re using in our homes.  I certainly thought I was using CAT5e until I actually looked today and found my desktop machine was hooked up with a plain-Jane CAT5 cable.  Yuck – that’s in the garbage now.  After that change, I noticed a small improvement in sustained transfer speed, but still holding at around 12MB/s.

2. MTU

If you have 2 gigabit cards that support it, and a network switch that supports it, you can get better speeds by increasing the maximum transmission unit of your network card.  In Linux, we do it like this:

hank☢barad-dur:~ % sudo ifconfig eth0 mtu 8000

In Solaris, or its derivatives, you do it like this:

root@nyu:~ # ifconfig e1000g0 mtu 8170

You also have to enable that mtu in /kernel/drv/e1000g.conf! I found that out thanks to this post. It’s quite easy – this is what mine looks like:

# 0 is for normal ethernet frames. 
# 1 is for upto 4k size frames. 
# 2 is for upto 8k size frames. 
# 3 is for upto 16k size frames.

Each number corresponds to the last number in the interface, so e1000g0 is the first number (set to 2 in this example) and so on. My switch only supports 9K Jumbo Frames, so this was fine.

This got me a little more stability, but I was still basically capped at 100Mbit (13MB/s). Time to roll out the big guns!

3. Rsync compression

The -z option in rsync compresses files before they’re sent.  I have nice beefy CPUs on both ends, so I thought that wouldn’t hurt – I was completely wrong about this.  For some reason it slows down the transfer by about 50% here.  CPU usage is very low on both machines, so this is really confusing, but as a general rule, do not use compression when transferring files with rsync on a LAN.  So, now that it’s off, I’m up to about 200Mbits/s.  Not bad, but we can do better!

4. Rsync method

So, when you run an rsync like this:

rsync -arxWh --progress . root@

You’re telling rsync to log in (using rsh or ssh) to using the root account. Now, if rsh is selected, then everything is peachy and you’ll get great rates. But, if ssh is selected, you’ll get encryption bloat, and your throughput will be reduced significantly (not to mention CPU usage will be higher). There’s a fix for this – on the destination system, run an rsync daemon. The instructions to do so can be found all over, but these were helpful for me. I set up the rsyncd.conf and secrets file, and just ran rsync --daemon, which backgrounded. I then executed this on the sending machine:

rsync -arxWh --progress . rsync://hank@

And immediately got another 10MB/s (!!) bump in speed. So, now files are cruising over the network at around 300Mbits/s, which is good enough for now. If I didn’t have a crappy Marvell onboard network interface on my host machine, and actually got a real gigabit card (I have a PCI-E one in the mail that will supposedly do full gigabit), this would be a lot faster. For now, I’ll just have to deal with 1/3 of its potential.

OpenIndiana Jones

So, after a bunch of research about building a DIY NAS, I decided to buy a whole bunch of hardware to do so. But, the real question was which software to use. FreeNAS seems to be the most popular solution, and I heard it was better than something called OpenFiler. Then I stumbled across NexentaStor, which is free for any NAS less than 18TB in size, which is fine for me. I was basically ready to go with that, but then I heard about OpenIndiana and the napp-it web gui. Basically, OpenIndiana is the result of OpenSolaris getting closed by Oracle. Since Oracle shut down the openness, the last open version of the operating system has been “sporked” into OpenIndiana. I just installed it in a VM, and I’m impressed, especially with the pool management of zfs.

But, being a hardcore Linux user for about 8 years, I’ve gotten used to certain things working a certain way. This post is just a little note to myself, and to others potentially, about what I didn’t like about the base install, and how I fixed it.


So, nicely, the machine comes with vim 7.2 installed, which is fantastic.  The problem is it’s in compatible mode by default.  Gotta shut that down.  Solaris apparently keeps the vimrc file hidden away in /usr, so we have to do this:

echo "set nocompatible" | sudo tee -a /usr/share/vim/vimrc

I also added the following lines for good measure to the same file using vim:

syntax on
set bg=dark
set ts=4
set sw=4

Now I have a real working copy of my favorite editor. That’s more than half-way to happiness for me. More to come.

Update: grep

So, now that I’m getting settled, I’ve been doing a bunch of shell work, and there’s something I noticed:

root@nyu:/etc # grep -R 2,2 *
grep: illegal option -- R 
Usage: grep -hblcnsviw pattern file . . .

That’s right – the default grep is crappy Solaris grep, not good old GNU grep! So, I checked it out, and the way to solve this is to use ggrep, which I will alias to grep, of course.

root@nyu:/etc # alias grep="ggrep"
root@nyu:/etc # grep 
Usage: ggrep [OPTION]... PATTERN [FILE]...
Try `ggrep --help' for more information.