Vista Command Prompt

So my new Dell Studio 15 laptop is a (great) step down from my old trusted Dell Latitude X300, but it was unfortunately a matter of funds this time :-( Anyway, long story, but I needed to get the "Create a command prompt here" functionality working on my new (Asta La) Vista system. This is the functionality that allows you to right click on any folder icon in Explorer and open a DOS Command Prompt in that folder.

In an earlier blog entry I showed a simple registry hack to do this on Windows XP and it is just as simple to do it on Windows Vista. Simply open your favorite text editor (e.g. TextPad), copy the following 5 lines, and save the file with a .reg extension, e.g. CommandPromptVista.reg. Then simply execute the file. You could also manual update the registry using RegEdit, but that is much more error prone (and slow).

Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\Directory\shell\CommandPrompt]
@="Command Prompt:"
[HKEY_CLASSES_ROOT\Directory\shell\CommandPrompt\Command]
@="cmd.exe /k cd %1″

Nifty.

Links in emails

OK, Joe in sales has just sent you and 16 others in your department yet another 13 MB PowerPoint presentation for everyone to review. And to make things worse, you now keep getting those annoying "your mailbox is over quota" messages from IT. Just another day in the office?

Well, you know that the best way to share documents is to use a link to a shared drive where people can review or modify the documents, but getting those pesky hyperlinks in to your email document just seems like more trouble than it is worth. Easy to see why Joe likes to email the whole presentation every time. On a side note, you can of course use the Microsoft Office feature called document reviewing, but that would just ruin the whole point of my story :-)

So let's assume that you want to to share a document named CoolPres.ppt that lives on the public Z: network drive in your company and that it has a path of "Z:\Sales\Firm A". If you use MS Word as your email editor, then there are 3 ways of getting a link to the presentation included in your email.

1. Use AutoCorrect. Set up an AutoCorrect entry for the Z: drive, so each time you type in Z: it expands to \\myserver\myshare (or whatever your server + share is called). See previous blog entry on how to do this. This is useful if you often refer to the very same network share. However, it is a little more work if the path you want to share has a space in it. To create a link simply type in the link, but as soon as you hit a space, Word will create a hyperlink. Since our example has a space in "Firm A" press undo right after the hyperlink is created and keep typing the rest. Then put quotes around the whole thing, and go to the end of it and press space. In the end, your link will look something like this: \\myserver\myshare\Sales\Firm A\CoolPres.ppt

Yes, as I said AutoCorrect is useful for creating the same link all the time, not so much for typical linking.

2. Use the file:\\ prefix. Simply type in "file:\\" in front of the path to the file. Note that sometimes you will need to put double quotes around the whole string when you have spaces in your file name or path name. So type the following: file:\\Z:\Sales\Firm A\CoolPres.ppt . As you finish typing the file name, Word will automatically turn your string into a hyperlink.

3. Use MS Word hyperlinking. However, the simplest way create a hyperlink to a file in an email, is to use the built-in Word hyperlink feature. Simply press Ctrl-K to get the hyperlink pop-up, locate your document, e.g. Z:\Sales\Firm A\CoolPres.ppt, and press enter. Done.

Please note that the email recipients do not need to have the Z: drive mapped to the same place as you in any of the cases above. The hyperlink you create is actually not to the Z: drive itself, but to the mapped resource (e.g. \\myserver\myshare) instead. You can see this if you mouse over the link you created. However, the email recipients do of course need permissions to read the folder where the shared document lives.

Nifty.

I like either for grep'ing

There are times when you want to grep for multiple different lines from a file. For example, the text file below could be an example of a log file and I'd like to see all the CmdStat entries I had and what Value line followed each of them.

log.txt:
CmdStat=InFlow
Info=More Data and Values
CmdStat=OutStat
Value=-3
CmdStat=Done
Value=42
Notes=end

Well, the first guess of running "grep CmdStat log.txt" followed by "grep Value log.txt" will certainly generate all the right lines, but it will not tell me the where the Value lines are compare to the CmdStat lines. There could be CmdStat lines without Value lines and vice versa.

The correct solution is to use the very useful -e (either) option. Doing a "grep -eCmdStat= -eValue= log.txt" will yield:
CmdStat=InFlow
CmdStat=OutStat
Value=-3
CmdStat=Done
Value=42

which shows me all the CmdStat and Value entries. Note that I used Value= instead of Value so we do not accidentally get lines with the word Value. Always a good idea to grep for a word as unique as possible.

You may of course also combine -e with other useful options, such as -i (case insensitive) and -v (except). So if I want to weed out all the Info and Notes lines from the file, I could do a "grep -v -eNotes= -eInfo= log.txt"

Nifty.

Grep'ing for a column value

A few days ago I needed to get those lines from a file that only contained a certain value in a certain field. To simplify the example, let's say that I have a file containing id, first name, last name, state, and phone number, and that I am only interested in those entries containing a state of California (e.g. ca or CA).

Here is the sample file named in.txt:
line1 joe smith ca 4085551212
line2 joe carlson az 3334445555
line3 carl smith ny 2049998888
line4 joe smith or 5035551234
line5 mike erwin ca 4159876543
line6 mike erwin CA 4159876543

I can not simply do a "grep -i ca in.txt" as I would get all lines containing "ca" even if "ca" occurred in the name fields (e.g. like in carl or carlson). In the above case I would get all lines, except line 4, which is of course incorrect.

If I did "cut -f4 in.txt | grep -i ca" I would get the correct number of results (i.e. 3), but I would only get the 3 "ca" values and not the whole line as I cut everything else away.

So one way to solve the problem (there may be other more clever ones I'm not aware of), is to use the "awk" command instead of the "grep" command.
cat in.txt | awk '{if ($4=="ca") print $0;}'

I think the above is self explanatory even for non-awk folks, the only trick is that $0 shows the entire line. If you only wanted to get certain columns in the output, you could decide to only print $2,$3,$5 for example to get first name, last name and phone number.

If you want to make the solution case insensitive, simply use the awk command "tolower()" in your if statement:
cat in.txt | awk '{if (tolower($4)=="ca") print $0;}'

Nifty.

Multiline AutoCorrect

In an earlier blog entry I showed how the cool Microsoft Word AutoCorrect feature can be used to both increase your typing speed and your spelling accuracy. This time we'll use AutoCorrect to write not a single word or a single line, but multiple lines.

So let's say that you often want to write the following 3 lines:
For more information about AutoCorrect in Word,
please see the nifty tricks at:
http://niftypctricks.blogspot.com

The way to add these lines to an AutoCorrect entry, is simply to write the lines in your Microsoft Word document, then highlight all of them, and press AutoCorrect (ALT+T,A in Word 2003). Simply enter the AutoCorrect key (e.g. multi3), and your multi-line AutoCorrect entry is ready to go. So next time you type multi3, your 3 lines will show up with all the formatting (bold, underline, hyperlinks, etc) of the original lines.

I do want to point out that in case you multi-line text is more part of your email signature than just a multi-line text, then you really should use an email signature instead (assuming that you are using Word as your email editor). So if you sometimes want to write:
Regards,
Michael E.
Master of AutoCorrect and Signatures

Then you should create another email signature in MS Word with the desired text in it. You can then right-click on your email signature at anytime and switch it to the desired signature.

Nifty.

Nifty Regular Expressions (RegExp)

I personally use TextPad as my default text editor. There are probably some much fancier ones out there, but TextPad has all the features that I demand, like great regExp support, vertical cut/paste, and a few other nifty features. It is free to download, and only something like $15 to buy. Well worth it. After all, that is not much more than that venti Caramel Macchiato that you normally order. And TextPad has great help on regular expression. I strongly encourage you to read their regExp help. And by the way, if you try TextPad, then I would suggest that you first change these 2 settings:
1) Under Preference/General, set Context Menu, so you can quickly send any file to TextPad
2) Under Preference/Editor, set Microsoft compatible.
3) Under Preference/View, set line numbers.
4) Under Preference/Assoc Files, add any file type you wish to open with TextPad (e.g. .txt)

OK, back to our scheduled programming. Below are a few tricks I've come to appreciate with regular expression. I do expect the reader to have basic familiarity with regexp (^ means beginning of line, $ means end of line, . means any character, etc)

Problem: Remove all lines containing a certain text string.
Find: ^.*FINDME.*\n
Replace:
Explained: Locate the string you want, and select the entire line (including the newline character). Simply replace with nothing. Note that if you chose $ instead of \n, then you would end up with a lot of blank lines instead, but assuming that you want to completely remove these lines, you need to use \n at the end.

Problem: Insert line numbers in front of all lines
Find: ^
Replace: \i(100,10)\t
Explained: Find beginning of line, replace with a numeric counter starting at 100 and incrementing by 10 followed by a tab. So now you'll have a first column with numbers 100,110,120, etc. in it. By the way, \i by default starts at 1 and increments by 1.

Problem: You have a text files with dates in a DDMMYYYY (day, month, year) format in column 1 and you would like to quickly convert them over to an YYYYMMDD format. This is quickly done using regular expression.
Find: ^\(..\)\(..\)\(....\)
Replace: \3\2\1
Explained: Create 3 match sets of 2, 2, and 4 characters respectively from the beginning of the line. Simply put the 3 match sets in the desired order. If you are unfamiliar with match sets, then \( and \) define each set and \1 refers to the first set, \2 to the second, etc. Please note that if you use POSIX style regexp then you do not need to escape the parentheses (i.e. use (..) instead of \(..\) ) to create the match sets.

Nifty

Searching XSLT files

I work a lot with XML and XSLT files, and hence I often need to grep files (i.e. search for text strings in files) for certain term. Windows Explorer does a great job for most file types, but unfortunately the Windows Explorer search function does not include any XSLT files when searching. Just try to create a dummy file called a.xslt with a dummy text string FINDME, and try to locate it using Windows Explorer search. You will not find it. Sad, but true. If someone knows of a simple tweak (e.g. registry change) to change this behavior, then please post it in the comment section of this entry. I'm not sure why XSLT files are excluded. Afterall, JavaScript (.js) and C# (.cs) files are included. Maybe XSLT just made the Microsoft black list since it is not a Microsoft standard.

Assuming for now that we cannot change the Windows Explorer search behavior, I wanted to show you how to search all files including XSLT files, using UNIX command tools. If you just need to search the current folder, you can of course you use
grep FINDME *.xslt

And if you have a fixed 3 level folder structure you may do something like this
grep FINDME *\*\*.xslt
but note that this will ONLY look for files that are exactly 3 directories deep (e.g. it will not search the file named a.xslt in the root folder).

So we need a much more flexible solution that can search files located at any directory level. But let me first introduce yet another UNIX command line tool: find. The command find, recursively lists all files and folders matching a given criterion (e.g. name, create date, etc) and may execute a command on each file found. Again, do a "man find" to get an overview of the options.

List all files recursively:
find . -type f

List all files or directories with the string xslt in them:
find . | grep -i xslt

Find any file that contains the text FINDME
find . -type f -exec grep FINDME "{}" ";"

Note that when using -exec you also need to add those special "{}" ";" characters at the end. Don't want to go into great detail here, but you need to always add it. Otherwise you can basically specify any command you want after exec, like we did with grep in the above case.

Nifty.

What's the difference?

If you work with plain text files, you often have 2 almost identical text files and you need to know what the difference between them is. This is quite easy to do with the UNIX command line tool diff. It will show you all the lines that are different and tell you in which file the additional text is. The diff command has a lot of cool options, so don’t forget to do a "man diff" on it first to see all the options. I personally use -w and -b the most as they ignore all white-spaces, and white-spaces at end of line, respectively.

Question:
What is the difference between 2 text files, ignoring all white-spaces?
Answer:
diff -w 1.txt 2.txt

However, sometimes you do not want to get all the differences, but merely know which text string occurs in one file that does not occur in the other.

Question:
You have 2 text files, and want to see which text strings are unique in column 1 in either file.
Answer:
cut -f1 1.txt | sort -u > 1u.txt
cut -f1 2.txt | sort -u > 2u.txt
cat 1u.txt 2u.txt | sort | uniq -u

Explained:
Cut column 1 from file 1 | remove duplicates | save in temporary file. Repeat for file 2. Then concatenate the 2 temp files with the unique entries | sort the content | show only unique entries as duplicates indicate that the line was present in both files.

If you wanted to show which lines were in both files instead, simply change the option for uniq to -d (to show only the duplicates). cat 1u.txt 2u.txt | sort | uniq -d

Note that this solution does not show you which file the unique string came from. Also, cat is like the DOS command type, though much cooler. It concatenates the content of all files given as argument as sends that to stdout (as most UNIX commands do).

Nifty.

Command line tools for dummies: Stringing them together

In the last blog entry we introduced a few basic UNIX command line tools. Now we'll start stringing them together to make them really useful.

We'll use the following 3-column tab delimited file containing:
line1 A Mike
line2 B Andy
line3 D Chang
line4 A Tom
line5 B Mike
line6 C Brad

All UNIX command line tool read from stdin (standard in) and output to stdout (standard out) by default. This allows us to string these commands together so the output of one command is used for input for the next command. On both UNIX and Windows, you use the pipe character (aka vertical bar, aka |) to string commands together.

So "grep Mike file.txt | wc –l" will first find lines with Mike in the file named file.txt and then send those lines (2 of them in our case) on to the wc command, which in turn will count them and output the result (2 in our case). Let’s try a few quick examples.

Question:
How many different grades (column 2) did we hand out?
Answer:
cut –f 2 | file.txt | sort -u | wc –l
4
Explained:
Cut field 2 | sort them uniquely (-u removes duplicates) | count how many. An alternative solution is of course: "cut –f 2 file.txt | sort | uniq | wc –l", but "sort -u" is shorter than "sort uniq".

Question:
What was the distribution of grades (column 2)
Answer:
cut –f 2 file.txt | sort | uniq -c
2 A
2 B
1 C
1 D
Explained:
Cut field 2 | sort them | unique them (remove duplicates) | add a count (note that uniq expects sorted input).

Question:
Same question as the above, but if you have a rather large file with many values, then your distribution will get very long. Often you may just want to see the top 2.
Answer:
cut –f 2 file.txt | sort | uniq -c | sort -rn | head -2
2 A
2 B
Explained:
Cut field 2 | sort them | unique them (remove duplicates) | sort results in reverse and treat as numeric (since we want to sort the added count) | show first 2.

Question:
What grades (column 2) did Mike get?
Answer:
cut –f 2,3 file.txt | grep Mike | cut –f 1
A
B
Explained:
Cut field 2 and 3 (otherwise we could potentially find Mike elsewhere) | find only lines containing Mike | cut out the grade (now in field 1 of the 2 fields we have left).

In the next blog, we'll look at handling multiple input files. But for now, practice with the above.

Nifty.

Command line tools for dummies

Let me start by saying that I receive no kickbacks or any other compensation from any of the products that I mention. I would certainly like to, but I don't :-) So the products mentioned are simply products that I have come to love and simply can't live without in my daily life.

Back in the days at the school of engineering at Santa Clara University, I took a great course in UNIX command line tools (grep, awk, sed, etc), and ever since I have not been able to live without them. I mean how do you quickly view the last few lines of a very large log file without a cool command line tool like tail, or how do you quickly find which XLST file that contains the XML tag named recordIdentifier without using a tool like grep.


So when my career took a turn from a UNIX environment to a Windows environment, I was scrambling to find some cool command line tools for Windows. I found my comfort in the MKS Toolkit, which are all the beloved UNIX command line tools compiled for Windows. It is not the cheapest tool ($479), but it is worth every penny if you for example do a lot of text file handling like me. It comes with hundreds of UNIX command line tools, shells, etc.

So the next few blog entries will be dedicated to learning a few nifty tricks on how to use UNIX command line tools. It is not going to be extremely advanced, but it will solve a lot of the everyday problems you encounter. Warning: Once you learn these little tricks, your co-workers with regard you as a complete nerd or even a hack :-) If you are working on a UNIX platform, the tools are of course readily available at the prompt. If you are on Windows, go purchase a set of command line tools (like the MKS Toolkit I mentioned). If you work on a Mac, you probably haven't heard the term command line before :-)

Anyway, let's start out with the very basics by first introducing a few simple tools. The power of command line tools is in stringing them together, but before we get there, we need to understand what each tool does. Please note that you can always access the online help for each tool by typing "man toolname". These online help pages (aka man pages) contain descriptions, available options, and examples, and they are a great source for exploring the tools even further.

For all the examples below, assume that you have a very simple tab delimited text file named file.txt with 3 columns. An easy way to create it is to create it in Excel and then paste it into your favorite text editor. The file content is shown below:
line1 A Mike
line2 C Chang
line3 A Tom
line4 B Mike

OK, here we go with the top 7 command line tools I use the most. Each tool has a large number of options, but the examples just shows one of them.

grep: Find lines containing or not containing text string
-i: option to ignore case (i.e. case insensitive)
C:\>grep -i mike file.txt
line1 A Mike
line4 B Mike

wc (word count): Show number of lines, word, etc
-l: option to get only the line count
C:\>wc -l file.txt
4 file.txt

tail: Show only last few lines.
head: Show only first few lines.
-10: you can specify how many lines you want
C:\>tail -2 file.txt
line3 A Tom
line4 B Mike
C:\>head -1 file.txt
line1 A Mike

cut: Cut file vertically by characters or fields
-f: option to specify fields to cut
C:\>cut -f1,3 file.txt
line1 Mike
line2 Chang
line3 Tom
line4 Mike

uniq: show unique or repeated lines
-c: option to count number of times a line occurs
C:\>uniq -c file.txt
1 line1 A Mike
1 line2 C Chang
1 line3 A Tom
1 line4 B Mike

sort: Sort input lines
-k: specify which fields to sort on
C:\>sort -k2 file.txt
line1 A Mike
line3 A Tom
line4 B Mike
line2 C Chang

Ok, that was the very very basic stuff. Not terribly useful when you only use one command. The real nifty stuff we'll look at in the next blog entry.

Nifty

Copy your AutoCorrect list

As you may have read in the latest blog, AutoCorrect is a very useful feature in most Microsoft Office applications.

Once you have created your own personal list of favorite words to expand you probably want to bring a copy of that AutoCorrect list with you to all the other computers that you use. This can easily be accomplished a few different ways. There are several comprehensive tools out there, including some Microsoft tools, but I mostly only need to copy the AutoCorrect list file, so I do it the old fashioned way. I copy the physical file.

The AutoCorrect list file has an extension of ACL (AutoCorrect List). The file is named MSOxxxx.acl where xxxx designates the language that the file is for. There is an ACL file for each language you have used. For example, MSO1033.acl is the ACL file for language of English (US). Lastly, the file is typically located in your application data folders, so you'll likely find it in "C:\Documents and Settings\NN.NN\Application Data\Microsoft\Office", where Windows NN.NN is your user name.

So simply copy the file (e.g. "C:\Documents and Settings\NN.NN\Application Data\Microsoft\Office\MSO1033.acl") to the same location on your other PC and you now have the same AutoCorrect list on both.

As you will now start growing your very own AutoCorrect list, you can even share it between home, work, client assignments, etc. I personally have a USB stick (aka Pen Drive) that I always carry with me, and in my tools folder on the USB drive, I always have the latest version of my ACL file.

Nifty.