Schadenfreude » scripting

wget mirroring with external references

Erik — Sun, 04 Dec 2011 05:49:32 +0000

I was having trouble mirroring a website that had all its images hosted on a different domain, which happened to be random subdomains of cloudfront.net. I tried adding *.cloudfront.net to the -D parameter, but that didn’t work. It turns out it’s smart enough to figure out that all subdomains in the domain list should be included as well:

wget -mkpEK -D www.allshepherdrescue.org,cloudfront.net -H -t 3 \
     --restrict-file-names=windows http://www.allshepherdrescue.org/

This goes into mirror mode, changes relative links to the proper form, fixes the query string urls to static ones, and downloads all files from the domains in the -D parameter. The manpage details all of this.

Finding bad JPEGs with Xorg hacks in Ubuntu

Erik — Sun, 25 Nov 2007 00:46:00 +0000

So, I have all these JPEGs, and I want to know which ones are corrupt (specifically, ones that end prematurely). qiv will spit out the following to STDERR when it finds one:


Premature end of JPEG file

So, this is nice, except it’s entirely unscriptable. The solution I found was using the following script to the display the images in sequence:


perl -e 'for(glob("*.png *.jpg")){$output = `qiv "$_" 2>&1;`; if($output =~ /Premature/){print $_, "\n";}}'

All this does is mix STDERR with STDOUT for a qiv of the file, and check the output for the word “Premature”. If it finds the word, it prints the filename. Simple.

The only problem is that qiv doesnt have a way to just check whether a JPEG file is corrupt (and if there is a command line utility that does, please let me know). To make it go thru the list, I wrote this little gem:


while(true); do xte "key q"; done

All this does is send the q key to the Xserver infinitely. All I have to do is put focus on the first qiv window to make it and all subsequent qiv windows receive q’s. So, just run it, and click on the window. Then there are lots of flashes, and eventually that perl script will print out the names of the bad files. It’s totally ghetto, but it’s the best I’ve got right now. The point of this post is to hopefully find new ways to do this more programmatically.

Setting EXIF dates with a loop

Erik — Tue, 18 Sep 2007 00:11:00 +0000

Magic EXIF recursive tagging!

Have you ever had your files all nicely nested in directories, but needed to change their EXIF dates? Here’s what I used today to do it:


# Structure like this: 1997/08/Picture.Whatever Maybe Some Spaces.jpg

# Delete the EXIF tags (DONT DO THIS UNLESS YOU KNOW WHAT YOU'RE DOING!)
find . -mindepth 3 -exec jhead -de "{}" \;

# Make Fresh EXIF tags
find . -mindepth 3 -exec jhead -mkexif "{}" \;

# Set the dates according to the Year and Month information in the filenames
find . -mindepth 3 | perl -ne '@a = split(/\//, $_); chomp($_); print `jhead -da$a[1]:$a[2]:01-2007:09:17 "$_"`, "\n"'

This worked wonderfully.

Using seq to zero-pad strings in a series

Erik — Sat, 19 May 2007 08:06:00 +0000

I thought this was pretty cool:


for i in `seq -f "%03g" 1 100`; do wget http://www.gozerog.com/images/Hawking_$i.jpg; done

We were trying to do this the other night using bash, but to no avail since the square brackets only work for local file path expansion. I should have remembered seq. Also, this example shows how to 0-pad a series using seq and bash, noted by the *-f* option. Three cheers for seq!