Displaying articles with tag perl
Posted by hank,
Sat Dec 08 18:09:00 UTC 2007
Code
Here’s a quick one-liner using some UNIX utilities and Perl to construct some nice averages from CSV data:
for i in `seq 2 20`; do cat crim_rate_2005_by_state.csv | cut -d , -f $i | perl -e '$c=$d=0;$e;while(<>){if(/^\d/){$c+=$_;$d+=1}else{s/\s{2,}/ /g;s/"//g;chomp($e=$_);}} print $e, ": ", $c/$d, "\n"'; done
And now, the spaced out version:
#!/bin/bash
for i in `seq 2 20`; do
cat crim_rate_2005_by_state.csv | \
cut -d , -f $i | \
perl -e '$c=$d=0;
$e;
while(<>){
if(/^\d/){
$c+=$_;
$d+=1
} else {
s/\s{2,}/ /g;
s/"//g;
chomp($e=$_);
}
}
print $e, ": ", $c/$d, "\n"';
done
Output
- Population: 5775431.88461538
- Violent crime rate: 418.930769230769
- Murder/manslaughter rate: 5.59038461538462
- Forcible rape rate: 33.1634615384615
- Robbery rate: 114.455769230769
- Assault rate: 265.728846153846
- Property crime rate: 3339.50961538462
- Burglary rate: 685.671153846154
- Larceny/theft rate: 2273.43269230769
- Motor vehicle theft rate: 380.417307692308
- Violent crime: 26928.3461538462
- Murder and nonnegligent manslaughter: 335.730769230769
- Forcible rape: 1809.67307692308
- Robbery: 8128.30769230769
- Aggravated assault: 16654.6346153846
- Property crime: 196569.711538462
- Burglary: 41756.0961538462
- Larceny-theft: 130880.442307692
- Motor vehicle theft: 23933.1730769231
So, now we have our averages. More work to be done. The data file used is available here:
Tags: perl
Posted by hank,
Sun Nov 25 00:46:00 UTC 2007
So, I have all these JPEGs, and I want to know which ones are corrupt (specifically, ones that end prematurely). qiv will spit out the following to STDERR when it finds one:
Premature end of JPEG file
So, this is nice, except it’s entirely unscriptable. The solution I found was using the following script to the display the images in sequence:
perl -e 'for(glob("*.png *.jpg")){$output = `qiv "$_" 2>&1;`; if($output =~ /Premature/){print $_, "\n";}}'
All this does is mix STDERR with STDOUT for a qiv of the file, and check the output for the word “Premature”. If it finds the word, it prints the filename. Simple.
The only problem is that qiv doesnt have a way to just check whether a JPEG file is corrupt (and if there is a command line utility that does, please let me know). To make it go thru the list, I wrote this little gem:
while(true); do xte "key q"; done
All this does is send the q key to the Xserver infinitely. All I have to do is put focus on the first qiv window to make it and all subsequent qiv windows receive q’s. So, just run it, and click on the window. Then there are lots of flashes, and eventually that perl script will print out the names of the bad files. It’s totally ghetto, but it’s the best I’ve got right now. The point of this post is to hopefully find new ways to do this more programmatically.
Tags: perl
Posted by hank,
Tue Sep 18 00:11:00 UTC 2007
Magic EXIF recursive tagging!
Have you ever had your files all nicely nested in directories, but needed to change their EXIF dates? Here’s what I used today to do it:
# Structure like this: 1997/08/Picture.Whatever Maybe Some Spaces.jpg
# Delete the EXIF tags (DONT DO THIS UNLESS YOU KNOW WHAT YOU'RE DOING!)
find . -mindepth 3 -exec jhead -de "{}" \;
# Make Fresh EXIF tags
find . -mindepth 3 -exec jhead -mkexif "{}" \;
# Set the dates according to the Year and Month information in the filenames
find . -mindepth 3 | perl -ne '@a = split(/\//, $_); chomp($_); print `jhead -da$a[1]:$a[2]:01-2007:09:17 "$_"`, "\n"'
This worked wonderfully.
Tags: perl
Posted by hank,
Fri Aug 17 06:01:00 UTC 2007
Now, be aware that this could be a security issue if you let strangers onto your IRC session. They could do some cool shell tricks to break out of this exec command into the raw shell itself, but I’ll forget about that for now. I wanted an alias that would allow me to send a command to my bot to shorten a URL as well as announce the URL into the channel using some good old-fashioned ANSI Color.
/alias fu exec -o perl -e 'print "\#furryurl \${ARGV[0]}\\n"\; print "C4\$ARGV[1]" if(length(\$ARGV[1]))' -- "$0" "$1-"
Note: The C in C4\$ARGV[1] is really a CONTROL-C.
Run this in irssi, and you’ll get some sweet output:
# Upon running /fu http://www.google.com ralree
# The text for 'ralree' is actually colored red on output.
01:59:03 < hardwarehank> #furryurl http://www.google.com
01:59:04 < hardwarehank> ralree
01:59:04 < rufis> hardwarehank: http://www.furryurl.com/hx
I’m so happy!
Tags: perl
Posted by hank,
Sun Jul 15 00:17:00 UTC 2007
I use GNU Screen. A lot. I found an article linked from DIgg today that showed me how to use a setting called hardstatus to set the little bar at the bottom to my liking. I decided to mess around with it a bit, and I ended up with this:
It shows my hostname, IP, uptime, screen window number and name, date and time, and load averages. Pretty cool, huh?
This is how you do it:
- First, create a
.screenrc file and dump this in there:
hardstatus alwayslastline
backtick 1 60 60 /home/hank/.screen_hardstatus
hardstatus string "%{Gk}%H: %{+s y}%1` | %=%n: %t%= | %m/%d %c | %{+b}%l"
- Now, see the line that starts with backtick? That tells screen what program to run for assigning output to
%1` in the hardstatus string. Let’s make the .screen_hardstatus file as well (make it in your own home directory).
#!/bin/bash
# Script to get run by hardstatus in screen.
# Prints customized single-line system stats
IP=`ifconfig eth0 | grep Mask | cut -d: -f2 | cut -d " " -f1`
UPTIME=`perl -pe 's/^(\d+).*/sprintf("%d", ($1\/(24*3600)))." days"/e' /proc/uptime`
echo -n "$IP | Up: $UPTIME"
I cheated and used perl, but all of this could safely be converted to sed/awk easily. All this does is print my raw dotted quad IP address, a pipe, and then the current uptime in days rounded down.
- Now, just run
screen. You should see something similar to the first picture. I used this page a lot to customize the colors, etc.
Tags: perl
Posted by hank,
Mon Apr 16 01:24:00 UTC 2007
DevastatorIIC happened to link me to this today, which I found to be extremely helpful. It’s a collection of small perl scripts that make doing very complicated things very easy. My personal favorite is vipe, which lets you edit a pipeline with vi in the middle of its execution. Another is vidir, which lets you edit the contents of a directory with vi. Beautiful.
Tags: perl
Posted by hardwarehank,
Sun Jan 21 18:59:09 UTC 2007
So, I am currently hosting this blog on Site5’s $5 Deal. I paid $120 for 2 years of hosting with 55GB HD Space and 5TB/month transfer allowance. So far, I am very happy with it, and I’m going to share some gory details with you.
The deal on the site looked to good to be true, and I thought it might be. So far, I haven’t noticed any deviances or misconceptions at all though.
Here’s what it looks like when you log into the Backstage application. This lets you manage your billing, domains, and contact info, and acts as a gateway into your site management systems. Very Nice!
Then, you can go to each site just by clicking its name. Each site has its own SiteAdmin interface which gives you pretty amazing abilities.
You can manage MySQL databases with phpMyAdmin, see all the Ruby and Perl libraries they have installed, mess around with FTP junk, automatically create about 30 applications, manage files, and lots lots more.
Also, there’s a little tool called FlashBack that I get to beta test for them. It auto-saves the state of your directory so if you delete files or whatnot you can restore them easily. Pretty awesome.
Shell access is enabled by default, so you don’t have to mess around with support on that one. I transferred my domain name to them, and they have a flat rate of $8.88/year for domain name registration. That’s fine with me. Also, if you need SSL, even if you can generate your own certificate like I can (CACert), it still costs $15 to install it on the server. Personally, I don’t want to pay for it, so I’m SSL-less right now. Oh well, I don’t need it anyway.
All in all, these guys blow away Dreamhost, which I had for the past year. Dreamhost cost me twice as much, and their server had very slow network performance for me. The fastest I ever saw was 100K/sec, and usually it was around 40K/sec. Unacceptable. Also, as far as I can tell with ps, this shared host is not as overloaded with people as Dreamhost’s. Dreamhost’s MySQL setup is lame (you have to create a ton of subdomains for no apparent reason), whereas Site5 you always use localhost, speeding installations along. I managed to transfer my entire old SVN checkout of Typo over here, and the only trouble I had was trying to remember to cross my T’s and dot my I’s on the FastCGI/Apache environment. It’s too bad I can’t just run mongrel instances and do Apache forwarding like I used to, but as long as it runs, whatever.
Score: 9/10
Site5 has done a wonderful job so far, and I don’t see it going downhill soon. They make a wonderful Rails host, and I’m proud to be a customer.
Tags: perl
Posted by hardwarehank,
Sun Nov 05 13:39:06 UTC 2006
I wrote a script today that does a POST to a currently logged in
GrandCentral user’s account using Firefox or
Mozilla cookies. It’s quite simple really:
#!/usr/bin/perl -w
# Calls someone using the currently logged in account on GrandCentral.com
# User must log in to create the cookie in FireFox or Mozilla
# Then, the FIREFOX_PROFILE Environment variable must be set to the profile
# directory containing the GrandCentral cookie.
# Finally, the script must be called using 2 10-digit phone numbers
# - The first is the local phone number, which is the one you want to use
# - The second is the remote phone number, which is the one you are calling.
use strict;
use LWP::UserAgent;
use HTTP::Cookies;
my $argcount = $#ARGV + 1;
# Die unless we have exactly 2 arguments and they both are 10-digit numbers
unless($argcount == 2 and $ARGV[0] =~ /^\d{10}$/ and $ARGV[1] =~ /^\d{10}$/) {
die("Usage: $0 <Your 10-digit Phone Number> <Their 10-digit Phone
Number>\n");
}
my ($mynum, $destnum) = @ARGV;
my $cookie_file = $ENV{'FIREFOX_PROFILE'}."/cookies.txt";
print $cookie_file, "\n";
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
$ua->cookie_jar( HTTP::Cookies::Netscape->new( file => $cookie_file ));
my $req = HTTP::Request->new(POST => 'http://www.grandcentral.com/calls/send_call_request');
$req->content_type('application/x-www-form-urlencoded');
$req->content("calltype=call&destno=$destnum&ani=$mynum&_=");
my $res = $ua->request($req);
print $res->as_string;
Horray. Now, log into GrandCentral, point your FIREFOX_PROFILE
Environment variable at your profile directory, and fire:
# Phone numbers have been changed to protect the innocent
./grandcentral_dialer 3748372637 7382737485
Whoa! The phone rings! Amazing!! I should add some more functionality to
it - maybe have it pull from a DB of people and allow searching for phone
numbers. Or maybe I could integrate it into some Linux address book and allow
you to directly call people from that. Hmm…the possibilities are endless.
Tags: perl
Posted by hardwarehank,
Sat Oct 07 12:31:00 UTC 2006
PhishTank is an awesome website that keeps a database of phishing websites that are
user submitted and verified. It integrates with
OpenDNS, so when a site is verified by the community
as a phishing site, OpenDNS users will see a phishing warning instead of the
original website. There are also other perks like spelling correction and
faster DNS resolves, but the phishing this is revolutionary. I joined
PhishTank today, and I’m currently in second place on number of phishing sites
submitted. I submitted 167 of them from Google’s blacklist after filtering it
with a short perl script.
#!/usr/bin/perl
use strict;
use LWP;
die("Specify a link file.") unless $ARGV[0];
open IN, "<", $ARGV[0];
open OUT, ">>", "results.".$ARGV[0];
while(<IN>) {
if(/<a href="(.*?)"/) {
my $browser = LWP::UserAgent->new;
$browser->timeout(3);
my $response = $browser->get($1);
if($response->is_success) {
print OUT "$_\n";
print "Success: $1\n";
} else { print "Failed: $1\n"; }
}
}
:) I’m so happy when I find sites that are already marked as phishers. If
only the process of submission could be automated completely…I’ll play with
the API on PhishNet and maybe it will become reality.
Tags: perl
Blog Posts
