postgres@tardis:~$ createdb test
postgres@tardis:~$ psql test
psql (8.4.0)
Type "help" for help.

test=# CREATE TYPE rank AS ENUM ('general', 'sergeant', 'private');
CREATE TYPE
test=# CREATE TABLE military (id SERIAL PRIMARY KEY,
test(#   name VARCHAR(128),
test(#   rank rank);
NOTICE:  CREATE TABLE will create implicit sequence "military_id_seq" for serial column "military.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "military_pkey" for table "military"
CREATE TABLE

Notice the use of SERIAL? That’s postgres’ AUTO_INCREMENT, basically. I like it better. Next, it’s time to make a text file with some data and compress it. Here’s what I put in the file (note that the spaces between the words are TAB characters):

general Lee
sergeant  Hartman
private Pyle

And compress it with gzip, making a nice little file:

hank@tardis:/tmp$ gzip file
hank@tardis:/tmp$ zcat file.gz
general	Lee
sergeant	Hartman
private	Pyle

Now let’s actually make a named pipe for postgres to read from:

hank@tardis:/tmp$ mkfifo namedpipe

Now that we have our named pipe, let’s start reading from it:

test=# COPY military (rank, name) FROM '/tmp/namedpipe' WITH DELIMITER E'\t';

The E'\t' part means to escape characters inside the single-quoted string, turning this into an actual tab character. All that we have to do now is use zcat:

hank@tardis:/tmp$ zcat file.gz > namedpipe

Immediately, there’s some output in the psql session:

COPY 3

So, postgres says it got 3 records successfully. Yay! Now, let’s display them:

test=# select * from military;
 id |  name   |   rank
----+---------+----------
  1 | Lee     | general
  2 | Hartman | sergeant
  3 | Pyle    | private
(3 rows)

So, this is a pretty good method to read in compressed files with postgres. I’ve seen many articles that use similar methods with postgres dump files, but it’s useful for bulk delimited data loading as well, as many times it’s prudent to compress bulk data files and not extract them before loading them. See the postgres COPY page for more information about this awesome function.

Here’s a quick one-liner using some UNIX utilities and Perl to construct some nice averages from CSV data:

for i in `seq 2 20`; do cat crim_rate_2005_by_state.csv | cut -d , -f $i | perl -e '$c=$d=0;$e;while(<>){if(/^\d/){$c+=$_;$d+=1}else{s/\s{2,}/ /g;s/"//g;chomp($e=$_);}} print $e, ": ", $c/$d, "\n"'; done

And now, the spaced out version:

#!/bin/bash
for i in `seq 2 20`; do
  cat crim_rate_2005_by_state.csv | \
  cut -d , -f $i | \
  perl -e '$c=$d=0;
    $e;
    while(<>){
      if(/^\d/){
        $c+=$_;
        $d+=1
      } else {
        s/\s{2,}/ /g;
        s/"//g;
        chomp($e=$_);
      }
    }
    print $e, ": ", $c/$d, "\n"';
done

Output

• Population: 5775431.88461538
• Violent crime rate: 418.930769230769
• Murder/manslaughter rate: 5.59038461538462
• Forcible rape rate: 33.1634615384615
• Robbery rate: 114.455769230769
• Assault rate: 265.728846153846
• Property crime rate: 3339.50961538462
• Burglary rate: 685.671153846154
• Larceny/theft rate: 2273.43269230769
• Motor vehicle theft rate: 380.417307692308
• Violent crime: 26928.3461538462
• Murder and nonnegligent manslaughter: 335.730769230769
• Forcible rape: 1809.67307692308
• Robbery: 8128.30769230769
• Aggravated assault: 16654.6346153846
• Property crime: 196569.711538462
• Burglary: 41756.0961538462
• Larceny-theft: 130880.442307692
• Motor vehicle theft: 23933.1730769231

So, now we have our averages. More work to be done. The data file used is available here:

crim_rate_2005_by_state.csv