Picture date-time stamp editing

I use Google's Picasa to edit / track my pictures, but I was very disappointed to learn that it does not have a basic tool to edit the date/time stamp of a JPEG.

Simple scenario: Your not-so-technical friend sends you 200 pics taken with their brand new camera. But of course the camera date was set to Jan 01 2001. So now you need to update all the pictures with the correct date in the header.

Solution: Go download a small free command line tool named jhead.
To change the date simply go:
jhead -da NEWDATE-OLDDATE *.JPG

(Looks like their documentation incorrectly specifies the opposite). And try it on a single file first. Their full documentation is here. It calculates the difference between OLDDATE and NEWDATE and applies that difference to all the pictures. So simple and so powerful.

To have Picasa notice the updated JPEG header, you may have to change the date stamp of the file too (not only the JPEG header). To do that:
jhead -ft *.JPG

Nifty.

Vista Command Prompt

So my new Dell Studio 15 laptop is a (great) step down from my old trusted Dell Latitude X300, but it was unfortunately a matter of funds this time :-( Anyway, long story, but I needed to get the "Create a command prompt here" functionality working on my new (Asta La) Vista system. This is the functionality that allows you to right click on any folder icon in Explorer and open a DOS Command Prompt in that folder.

In an earlier blog entry I showed a simple registry hack to do this on Windows XP and it is just as simple to do it on Windows Vista. Simply open your favorite text editor (e.g. TextPad), copy the following 5 lines, and save the file with a .reg extension, e.g. CommandPromptVista.reg. Then simply execute the file. You could also manual update the registry using RegEdit, but that is much more error prone (and slow).

Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\Directory\shell\CommandPrompt]
@="Command Prompt:"
[HKEY_CLASSES_ROOT\Directory\shell\CommandPrompt\Command]
@="cmd.exe /k cd %1″

Nifty.

Links in emails

OK, Joe in sales has just sent you and 16 others in your department yet another 13 MB PowerPoint presentation for everyone to review. And to make things worse, you now keep getting those annoying "your mailbox is over quota" messages from IT. Just another day in the office?

Well, you know that the best way to share documents is to use a link to a shared drive where people can review or modify the documents, but getting those pesky hyperlinks in to your email document just seems like more trouble than it is worth. Easy to see why Joe likes to email the whole presentation every time. On a side note, you can of course use the Microsoft Office feature called document reviewing, but that would just ruin the whole point of my story :-)

So let's assume that you want to to share a document named CoolPres.ppt that lives on the public Z: network drive in your company and that it has a path of "Z:\Sales\Firm A". If you use MS Word as your email editor, then there are 3 ways of getting a link to the presentation included in your email.

1. Use AutoCorrect. Set up an AutoCorrect entry for the Z: drive, so each time you type in Z: it expands to \\myserver\myshare (or whatever your server + share is called). See previous blog entry on how to do this. This is useful if you often refer to the very same network share. However, it is a little more work if the path you want to share has a space in it. To create a link simply type in the link, but as soon as you hit a space, Word will create a hyperlink. Since our example has a space in "Firm A" press undo right after the hyperlink is created and keep typing the rest. Then put quotes around the whole thing, and go to the end of it and press space. In the end, your link will look something like this: \\myserver\myshare\Sales\Firm A\CoolPres.ppt

Yes, as I said AutoCorrect is useful for creating the same link all the time, not so much for typical linking.

2. Use the file:\\ prefix. Simply type in "file:\\" in front of the path to the file. Note that sometimes you will need to put double quotes around the whole string when you have spaces in your file name or path name. So type the following: file:\\Z:\Sales\Firm A\CoolPres.ppt . As you finish typing the file name, Word will automatically turn your string into a hyperlink.

3. Use MS Word hyperlinking. However, the simplest way create a hyperlink to a file in an email, is to use the built-in Word hyperlink feature. Simply press Ctrl-K to get the hyperlink pop-up, locate your document, e.g. Z:\Sales\Firm A\CoolPres.ppt, and press enter. Done.

Please note that the email recipients do not need to have the Z: drive mapped to the same place as you in any of the cases above. The hyperlink you create is actually not to the Z: drive itself, but to the mapped resource (e.g. \\myserver\myshare) instead. You can see this if you mouse over the link you created. However, the email recipients do of course need permissions to read the folder where the shared document lives.

Nifty.

I like either for grep'ing

There are times when you want to grep for multiple different lines from a file. For example, the text file below could be an example of a log file and I'd like to see all the CmdStat entries I had and what Value line followed each of them.

log.txt:
CmdStat=InFlow
Info=More Data and Values
CmdStat=OutStat
Value=-3
CmdStat=Done
Value=42
Notes=end

Well, the first guess of running "grep CmdStat log.txt" followed by "grep Value log.txt" will certainly generate all the right lines, but it will not tell me the where the Value lines are compare to the CmdStat lines. There could be CmdStat lines without Value lines and vice versa.

The correct solution is to use the very useful -e (either) option. Doing a "grep -eCmdStat= -eValue= log.txt" will yield:
CmdStat=InFlow
CmdStat=OutStat
Value=-3
CmdStat=Done
Value=42

which shows me all the CmdStat and Value entries. Note that I used Value= instead of Value so we do not accidentally get lines with the word Value. Always a good idea to grep for a word as unique as possible.

You may of course also combine -e with other useful options, such as -i (case insensitive) and -v (except). So if I want to weed out all the Info and Notes lines from the file, I could do a "grep -v -eNotes= -eInfo= log.txt"

Nifty.

Grep'ing for a column value

A few days ago I needed to get those lines from a file that only contained a certain value in a certain field. To simplify the example, let's say that I have a file containing id, first name, last name, state, and phone number, and that I am only interested in those entries containing a state of California (e.g. ca or CA).

Here is the sample file named in.txt:
line1 joe smith ca 4085551212
line2 joe carlson az 3334445555
line3 carl smith ny 2049998888
line4 joe smith or 5035551234
line5 mike erwin ca 4159876543
line6 mike erwin CA 4159876543

I can not simply do a "grep -i ca in.txt" as I would get all lines containing "ca" even if "ca" occurred in the name fields (e.g. like in carl or carlson). In the above case I would get all lines, except line 4, which is of course incorrect.

If I did "cut -f4 in.txt | grep -i ca" I would get the correct number of results (i.e. 3), but I would only get the 3 "ca" values and not the whole line as I cut everything else away.

So one way to solve the problem (there may be other more clever ones I'm not aware of), is to use the "awk" command instead of the "grep" command.
cat in.txt | awk '{if ($4=="ca") print $0;}'

I think the above is self explanatory even for non-awk folks, the only trick is that $0 shows the entire line. If you only wanted to get certain columns in the output, you could decide to only print $2,$3,$5 for example to get first name, last name and phone number.

If you want to make the solution case insensitive, simply use the awk command "tolower()" in your if statement:
cat in.txt | awk '{if (tolower($4)=="ca") print $0;}'

Nifty.

Multiline AutoCorrect

In an earlier blog entry I showed how the cool Microsoft Word AutoCorrect feature can be used to both increase your typing speed and your spelling accuracy. This time we'll use AutoCorrect to write not a single word or a single line, but multiple lines.

So let's say that you often want to write the following 3 lines:
For more information about AutoCorrect in Word,
please see the nifty tricks at:
http://niftypctricks.blogspot.com

The way to add these lines to an AutoCorrect entry, is simply to write the lines in your Microsoft Word document, then highlight all of them, and press AutoCorrect (ALT+T,A in Word 2003). Simply enter the AutoCorrect key (e.g. multi3), and your multi-line AutoCorrect entry is ready to go. So next time you type multi3, your 3 lines will show up with all the formatting (bold, underline, hyperlinks, etc) of the original lines.

I do want to point out that in case you multi-line text is more part of your email signature than just a multi-line text, then you really should use an email signature instead (assuming that you are using Word as your email editor). So if you sometimes want to write:
Regards,
Michael E.
Master of AutoCorrect and Signatures

Then you should create another email signature in MS Word with the desired text in it. You can then right-click on your email signature at anytime and switch it to the desired signature.

Nifty.

Nifty Regular Expressions (RegExp)

I personally use TextPad as my default text editor. There are probably some much fancier ones out there, but TextPad has all the features that I demand, like great regExp support, vertical cut/paste, and a few other nifty features. It is free to download, and only something like $15 to buy. Well worth it. After all, that is not much more than that venti Caramel Macchiato that you normally order. And TextPad has great help on regular expression. I strongly encourage you to read their regExp help. And by the way, if you try TextPad, then I would suggest that you first change these 2 settings:
1) Under Preference/General, set Context Menu, so you can quickly send any file to TextPad
2) Under Preference/Editor, set Microsoft compatible.
3) Under Preference/View, set line numbers.
4) Under Preference/Assoc Files, add any file type you wish to open with TextPad (e.g. .txt)

OK, back to our scheduled programming. Below are a few tricks I've come to appreciate with regular expression. I do expect the reader to have basic familiarity with regexp (^ means beginning of line, $ means end of line, . means any character, etc)

Problem: Remove all lines containing a certain text string.
Find: ^.*FINDME.*\n
Replace:
Explained: Locate the string you want, and select the entire line (including the newline character). Simply replace with nothing. Note that if you chose $ instead of \n, then you would end up with a lot of blank lines instead, but assuming that you want to completely remove these lines, you need to use \n at the end.

Problem: Insert line numbers in front of all lines
Find: ^
Replace: \i(100,10)\t
Explained: Find beginning of line, replace with a numeric counter starting at 100 and incrementing by 10 followed by a tab. So now you'll have a first column with numbers 100,110,120, etc. in it. By the way, \i by default starts at 1 and increments by 1.

Problem: You have a text files with dates in a DDMMYYYY (day, month, year) format in column 1 and you would like to quickly convert them over to an YYYYMMDD format. This is quickly done using regular expression.
Find: ^\(..\)\(..\)\(....\)
Replace: \3\2\1
Explained: Create 3 match sets of 2, 2, and 4 characters respectively from the beginning of the line. Simply put the 3 match sets in the desired order. If you are unfamiliar with match sets, then \( and \) define each set and \1 refers to the first set, \2 to the second, etc. Please note that if you use POSIX style regexp then you do not need to escape the parentheses (i.e. use (..) instead of \(..\) ) to create the match sets.

Nifty

Searching XSLT files

I work a lot with XML and XSLT files, and hence I often need to grep files (i.e. search for text strings in files) for certain term. Windows Explorer does a great job for most file types, but unfortunately the Windows Explorer search function does not include any XSLT files when searching. Just try to create a dummy file called a.xslt with a dummy text string FINDME, and try to locate it using Windows Explorer search. You will not find it. Sad, but true. If someone knows of a simple tweak (e.g. registry change) to change this behavior, then please post it in the comment section of this entry. I'm not sure why XSLT files are excluded. Afterall, JavaScript (.js) and C# (.cs) files are included. Maybe XSLT just made the Microsoft black list since it is not a Microsoft standard.

Assuming for now that we cannot change the Windows Explorer search behavior, I wanted to show you how to search all files including XSLT files, using UNIX command tools. If you just need to search the current folder, you can of course you use
grep FINDME *.xslt

And if you have a fixed 3 level folder structure you may do something like this
grep FINDME *\*\*.xslt
but note that this will ONLY look for files that are exactly 3 directories deep (e.g. it will not search the file named a.xslt in the root folder).

So we need a much more flexible solution that can search files located at any directory level. But let me first introduce yet another UNIX command line tool: find. The command find, recursively lists all files and folders matching a given criterion (e.g. name, create date, etc) and may execute a command on each file found. Again, do a "man find" to get an overview of the options.

List all files recursively:
find . -type f

List all files or directories with the string xslt in them:
find . | grep -i xslt

Find any file that contains the text FINDME
find . -type f -exec grep FINDME "{}" ";"

Note that when using -exec you also need to add those special "{}" ";" characters at the end. Don't want to go into great detail here, but you need to always add it. Otherwise you can basically specify any command you want after exec, like we did with grep in the above case.

Nifty.

What's the difference?

If you work with plain text files, you often have 2 almost identical text files and you need to know what the difference between them is. This is quite easy to do with the UNIX command line tool diff. It will show you all the lines that are different and tell you in which file the additional text is. The diff command has a lot of cool options, so don’t forget to do a "man diff" on it first to see all the options. I personally use -w and -b the most as they ignore all white-spaces, and white-spaces at end of line, respectively.

Question:
What is the difference between 2 text files, ignoring all white-spaces?
Answer:
diff -w 1.txt 2.txt

However, sometimes you do not want to get all the differences, but merely know which text string occurs in one file that does not occur in the other.

Question:
You have 2 text files, and want to see which text strings are unique in column 1 in either file.
Answer:
cut -f1 1.txt | sort -u > 1u.txt
cut -f1 2.txt | sort -u > 2u.txt
cat 1u.txt 2u.txt | sort | uniq -u

Explained:
Cut column 1 from file 1 | remove duplicates | save in temporary file. Repeat for file 2. Then concatenate the 2 temp files with the unique entries | sort the content | show only unique entries as duplicates indicate that the line was present in both files.

If you wanted to show which lines were in both files instead, simply change the option for uniq to -d (to show only the duplicates). cat 1u.txt 2u.txt | sort | uniq -d

Note that this solution does not show you which file the unique string came from. Also, cat is like the DOS command type, though much cooler. It concatenates the content of all files given as argument as sends that to stdout (as most UNIX commands do).

Nifty.

Command line tools for dummies: Stringing them together

In the last blog entry we introduced a few basic UNIX command line tools. Now we'll start stringing them together to make them really useful.

We'll use the following 3-column tab delimited file containing:
line1 A Mike
line2 B Andy
line3 D Chang
line4 A Tom
line5 B Mike
line6 C Brad

All UNIX command line tool read from stdin (standard in) and output to stdout (standard out) by default. This allows us to string these commands together so the output of one command is used for input for the next command. On both UNIX and Windows, you use the pipe character (aka vertical bar, aka |) to string commands together.

So "grep Mike file.txt | wc –l" will first find lines with Mike in the file named file.txt and then send those lines (2 of them in our case) on to the wc command, which in turn will count them and output the result (2 in our case). Let’s try a few quick examples.

Question:
How many different grades (column 2) did we hand out?
Answer:
cut –f 2 | file.txt | sort -u | wc –l
4
Explained:
Cut field 2 | sort them uniquely (-u removes duplicates) | count how many. An alternative solution is of course: "cut –f 2 file.txt | sort | uniq | wc –l", but "sort -u" is shorter than "sort uniq".

Question:
What was the distribution of grades (column 2)
Answer:
cut –f 2 file.txt | sort | uniq -c
2 A
2 B
1 C
1 D
Explained:
Cut field 2 | sort them | unique them (remove duplicates) | add a count (note that uniq expects sorted input).

Question:
Same question as the above, but if you have a rather large file with many values, then your distribution will get very long. Often you may just want to see the top 2.
Answer:
cut –f 2 file.txt | sort | uniq -c | sort -rn | head -2
2 A
2 B
Explained:
Cut field 2 | sort them | unique them (remove duplicates) | sort results in reverse and treat as numeric (since we want to sort the added count) | show first 2.

Question:
What grades (column 2) did Mike get?
Answer:
cut –f 2,3 file.txt | grep Mike | cut –f 1
A
B
Explained:
Cut field 2 and 3 (otherwise we could potentially find Mike elsewhere) | find only lines containing Mike | cut out the grade (now in field 1 of the 2 fields we have left).

In the next blog, we'll look at handling multiple input files. But for now, practice with the above.

Nifty.

Command line tools for dummies

Let me start by saying that I receive no kickbacks or any other compensation from any of the products that I mention. I would certainly like to, but I don't :-) So the products mentioned are simply products that I have come to love and simply can't live without in my daily life.

Back in the days at the school of engineering at Santa Clara University, I took a great course in UNIX command line tools (grep, awk, sed, etc), and ever since I have not been able to live without them. I mean how do you quickly view the last few lines of a very large log file without a cool command line tool like tail, or how do you quickly find which XLST file that contains the XML tag named recordIdentifier without using a tool like grep.


So when my career took a turn from a UNIX environment to a Windows environment, I was scrambling to find some cool command line tools for Windows. I found my comfort in the MKS Toolkit, which are all the beloved UNIX command line tools compiled for Windows. It is not the cheapest tool ($479), but it is worth every penny if you for example do a lot of text file handling like me. It comes with hundreds of UNIX command line tools, shells, etc.

So the next few blog entries will be dedicated to learning a few nifty tricks on how to use UNIX command line tools. It is not going to be extremely advanced, but it will solve a lot of the everyday problems you encounter. Warning: Once you learn these little tricks, your co-workers with regard you as a complete nerd or even a hack :-) If you are working on a UNIX platform, the tools are of course readily available at the prompt. If you are on Windows, go purchase a set of command line tools (like the MKS Toolkit I mentioned). If you work on a Mac, you probably haven't heard the term command line before :-)

Anyway, let's start out with the very basics by first introducing a few simple tools. The power of command line tools is in stringing them together, but before we get there, we need to understand what each tool does. Please note that you can always access the online help for each tool by typing "man toolname". These online help pages (aka man pages) contain descriptions, available options, and examples, and they are a great source for exploring the tools even further.

For all the examples below, assume that you have a very simple tab delimited text file named file.txt with 3 columns. An easy way to create it is to create it in Excel and then paste it into your favorite text editor. The file content is shown below:
line1 A Mike
line2 C Chang
line3 A Tom
line4 B Mike

OK, here we go with the top 7 command line tools I use the most. Each tool has a large number of options, but the examples just shows one of them.

grep: Find lines containing or not containing text string
-i: option to ignore case (i.e. case insensitive)
C:\>grep -i mike file.txt
line1 A Mike
line4 B Mike

wc (word count): Show number of lines, word, etc
-l: option to get only the line count
C:\>wc -l file.txt
4 file.txt

tail: Show only last few lines.
head: Show only first few lines.
-10: you can specify how many lines you want
C:\>tail -2 file.txt
line3 A Tom
line4 B Mike
C:\>head -1 file.txt
line1 A Mike

cut: Cut file vertically by characters or fields
-f: option to specify fields to cut
C:\>cut -f1,3 file.txt
line1 Mike
line2 Chang
line3 Tom
line4 Mike

uniq: show unique or repeated lines
-c: option to count number of times a line occurs
C:\>uniq -c file.txt
1 line1 A Mike
1 line2 C Chang
1 line3 A Tom
1 line4 B Mike

sort: Sort input lines
-k: specify which fields to sort on
C:\>sort -k2 file.txt
line1 A Mike
line3 A Tom
line4 B Mike
line2 C Chang

Ok, that was the very very basic stuff. Not terribly useful when you only use one command. The real nifty stuff we'll look at in the next blog entry.

Nifty

Copy your AutoCorrect list

As you may have read in the latest blog, AutoCorrect is a very useful feature in most Microsoft Office applications.

Once you have created your own personal list of favorite words to expand you probably want to bring a copy of that AutoCorrect list with you to all the other computers that you use. This can easily be accomplished a few different ways. There are several comprehensive tools out there, including some Microsoft tools, but I mostly only need to copy the AutoCorrect list file, so I do it the old fashioned way. I copy the physical file.

The AutoCorrect list file has an extension of ACL (AutoCorrect List). The file is named MSOxxxx.acl where xxxx designates the language that the file is for. There is an ACL file for each language you have used. For example, MSO1033.acl is the ACL file for language of English (US). Lastly, the file is typically located in your application data folders, so you'll likely find it in "C:\Documents and Settings\NN.NN\Application Data\Microsoft\Office", where Windows NN.NN is your user name.

So simply copy the file (e.g. "C:\Documents and Settings\NN.NN\Application Data\Microsoft\Office\MSO1033.acl") to the same location on your other PC and you now have the same AutoCorrect list on both.

As you will now start growing your very own AutoCorrect list, you can even share it between home, work, client assignments, etc. I personally have a USB stick (aka Pen Drive) that I always carry with me, and in my tools folder on the USB drive, I always have the latest version of my ACL file.

Nifty.

Use AutoCorrect to type really fast

Like most people, I also use Microsoft Word to write documentation, email, etc. and one of the features I use the most is AutoCorrect. Warning: Once you start using it, it can be hard to live without it.

AutoCorrect is simply a list of characters that will get replaced with other characters. It is the automatic "replace as you type" function, and it is commonly used to fix typical misspellings and create emoticons such as a smiley. For example, if you type in "your here" in Microsoft Word, it will automatically get expanded to a "you're here". To see the complete list, go to Tools->AutoCorrect Options in Microsoft Word. Note that AutoCorrect is language sensitive, so you have a different list for each language you compose messages in. This of course makes sense since you do not want to autocorrect abotu to about for Danish, but only for English.

But instead of using it to correct misspelled words, I use it to write longer words that I use over and over again. Below is a small sample of my own personal list.

Commonly used words and sentences:
sw=software
dev=develop
devn=development
anyq=Any questions, please do not hesitate to contact me.

You can also use AutoCorrect to write the correct capitalization of a given word or sentence. If you enter a word in lowercase in autocorrect, then Microsoft Word will automatically expand to the correct case. E.g. if you have a lowercase entry of sw for software, then sw will expand to software, and SW (in uppercase) will expand SOFTWARE, and Software if in the beginning of a sentence. Works pretty well. But if you always want to have SOFTWARE in uppercase since it may be a trademark name, etc, then all you have to enter is sw for SOFTWARE.

There are some words that you may want to autocorrect some of the times, but not all the times. The easiest way to handle this is to press undo (ALT-BACKSPACE or CTRL-Z) immediately after the expanding. So if you need to enter the address "123 SW MAIN ST", you want to press undo right after SW otherwise you'll send the letter to "123 SOFTWARE MAIN ST".

Lastly, note that you could also have an entry of uppercase SW. However, this means that SW is only expanded if you enter the uppercase SW. If you entered sw it would not get expanded.

I suggest you play around with it yourself. It is kind of like reading a book on riding a bicycle. It may get you an idea, but you'll not get the feel for it, until you jump on the bike yourself.

By the way, this entry was writing using 8 different AutoCorrect entries.

Nifty.

There are stars and then there are stars

Microsoft Word and Microsoft Excel are both fantastic programs that most of us use everyday. Personally I vote for Microsoft Excel as the best and most useful program ever written, but that is a completely different story. But it is funny to see how differently the two programs function despite both being developed by the very same company (but clearly by different departments).

Take searching in Word versus Excel. In Word you can replace an asterisk (*) with a B by simply using Find=* and Replace=B. So A*C becomes ABC. It (kinda) makes sense, but in Excel the asterisk (*) is suddenly used as a wildcard character that you can not easily turn off. So A*C now becomes B as the whole cell is matched. God forbid someone as Microsoft should discover standard regular expressions.

But how do you then replace an plain asterisk (*) inside text in your Excel file. Well, it turns out that you put a tilde (~) in front of the asterisk. Yeah, don't ask me how tilde suddenly became the escape character instead of backslash. So when you use Find=~* and Replace=B, then A*C correctly becomes ABC. Again, God forbid someone as Microsoft should discover regular expressions and using \ as the escape character.

Microsoft, in case you are listening, please add an option to use real UNIX style regular expressions. I mean it is a standard after all. But at least we have a way to replace asterisks in Excel.

Nifty.

DOS Prompt FOR command

The FOR command available in the DOS prompt is surprisingly useful. As you may have seen in the earlier blog the help on the FOR command (i.e. "HELP FOR") has a lot of useful information. I personally find that I use the FOR /L option more than any of the others. For example, if you need to create 100 directories named NewDir0 to NewDir99 you clearly do not want to do that from the Windows Explorer (AKA Windows Exploder). And you do not want to start typing mkdir manually from the command prompt either.

Instead, use the FOR command. But before you start creating a hundred directories, you should always use the echo command to test your FOR loop.

The FOR /L allows you to specify the start, step, and end of the loop, so the command:
C:\>FOR /L %i in (0,1,99) do @echo NewDir%i

Will echo:
NewDir0
NewDir1
...
NewDir99

Once it works the way you want it to, simply replace echo with your desired command (e.g. mkdir) and you whoopty you got yourself 100 directories.

Nifty