[root@nyu ~]# lvextend -l +100%FREE /dev/diclonius/vector
  Extending logical volume vector to 7.28 TiB
  Logical volume vector successfully resized

Thanks Redhat Documentation!

In other news, I ran a performance test on my 5-drive RAID-5 using HD204UI drives from Samsung:

[root@nyu tmp]# dd if=/dev/zero of=foo count=5 bs=$((1024*1024*1024))
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB) copied, 18.1434 s, 296 MB/s
[root@nyu tmp]# dd if=foo of=/dev/null
10485760+0 records in
10485760+0 records out
5368709120 bytes (5.4 GB) copied, 15.2682 s, 352 MB/s

So, 350MB/sec reads and 296MB/sec writes! Not bad!!

The Setup

Though their web interface isn’t nearly as polished as Giganews, why would you need polished for usenet? I decided on Star Wreck Legacy, which is a free and legal collection of films.

It’s a hefty 6.6GB, so it should eliminate any short-term bursting, etc. that may skew measurements. It even has an overabundance of par2 files. Nice…

I use stunnel to create SSL tunnels to the secure news servers, and connect to the unencrypted news port on my local machine that stunnel provides. This tricks hellanzb, my client, into thinking it’s just using a regular news server, while all my traffic just looks like normal SSL traffic, indifferentiable from normal secure web traffic save the ports. Both the tests will be conducted with the SSL on using 18 connections.

I have relatively high bandwidth for this country (20Mbits Down, 8Mbits up), but it’s from Comcast, so we’ll see how that holds up. It’s currently working pretty well consistently. Here’s a recent test I did at Speedtest.net:

Astraweb

Transferred 5957.2MB in 49m 43s at 2044.5KB/s (starwrecklegacy.par2)

Giganews

Transferred 5958.6MB in 49m 45s at 2043.5KB/s (starwrecklegacy.par2)

Conclusion

So, as you can see above, there’s no significant difference between the speed of the 2 services. Yet, Astraweb is massively cheaper. Get it cheap while you can!

SunSpider Benchmark Results

I pointed it at my 16GB of Wikipedia text in my MySQL database.

So, earlier this year, I decided it would be cool to mirror Wikipedia. So, I successfully set up a local copy on my system, and it’s been just sitting there ever since. But lately, I’ve been interested in fulltext indexing offered by various indexing engines, and Sphinx has looked especially tasty. So, I figured I’d sit down and try it today.

I pointed it at my 16GB of Wikipedia text in my MySQL database like so:

sphinx.conf

source src1
{
  type        = mysql
  strip_html      = 0
  index_html_attrs  =
  sql_host      = localhost
  sql_user      = wikipedia
  sql_pass      = wikipedia
  sql_db        = wikidb
  sql_query_pre   =
  sql_query     = \
    SELECT old_id, old_text\
    FROM text
  sql_query_post    =
  sql_query_info    = SELECT * FROM text WHERE old_id=$id
}

Next, I set up the indexing section.

index wikipedia
{
  source      = src1
  path      = /nexus/rofl/sphinx/wikipedia.sphinx
  docinfo     = extern
  morphology      = none
  stopwords     =
  min_word_len    = 1
  charset_type    = utf-8
  min_prefix_len    = 0
  min_infix_len   = 0
}
index wikipediastemmed : wikipedia
{
  path      = /var/data/wikipediastemmed
  morphology    = stem_en
}
indexer
{
  mem_limit     = 512M
}

I left all the other options as default. Next, I turned on the indexing and waited for about 2.5 hours. Now, bear in mind that 2.5 hours isn’t all that long to index this much data, especially given the results I’m about to show you.

Now it’s time to test this out!

hank@rofl:/usr/local/etc$ time search endothermic
## ....................................................................................................
## ....................................................................................................
## ....................................................................................................
= Sterling D. | title = Cold Fire® is a Hot Fire Extinguisher | publisher =
Company press release | date = Nov. 28, 2003 | url= http://www.greaterthings.com/News/ColdFire/pr031122.html | accessdate = August 21, 2006}}</ref>
==References==
<references/>
== External links ==
* [http://www.firefreeze.com Fire Freeze Worldwide Inc.]

[[Category:Firefighting]]
        old_flags=utf-8
20. document=112594001, weight=1
        old_id=112594001
        old_text=#REDIRECT[[Endothermic]]
        old_flags=utf-8

words:
1. 'endothermic': 173 documents, 293 hits

real    0m0.831s
user    0m0.004s
sys     0m0.080s

hank@rofl:/usr/local/etc$ time search "hello & world" >/dev/null

real    0m0.659s
user    0m0.032s
sys     0m0.052s

Look at that time!! 0.8 Seconds to search 16GB of text!

Sphinx is indeed the master of the fulltexting.

I’m very impressed. I’m sure I will find a use for this soon.

Update: It’s actually faster.

Due to the comment from Sphinx’s author below, I ran a searchd instance with gets rid of all the overhead when searching from the command line.

Here are some results I got using the Ruby API that’s included with Sphinx:

irb(main):010:0> t = Time.now; s.query('(Single & mother) & !father'); puts Time.now - t
0.016864
=> nil

It only took 0.017 seconds to find all instances of single and mother without mention of father in Wikipedia’s database.

This is indeed impressive.