Monday, February 18, 2008

Nifty Regular Expressions (RegExp)

I personally use TextPad as my default text editor. There are probably some much fancier ones out there, but TextPad has all the features that I demand, like great regExp support, vertical cut/paste, and a few other nifty features. It is free to download, and only something like $15 to buy. Well worth it. After all, that is not much more than that venti Caramel Macchiato that you normally order. And TextPad has great help on regular expression. I strongly encourage you to read their regExp help. And by the way, if you try TextPad, then I would suggest that you first change these 2 settings:
1) Under Preference/General, set Context Menu, so you can quickly send any file to TextPad
2
) Under Preference/Editor, set Microsoft compatible.
3) Under Preference/View, set line numbers.
4) Under Preference/Assoc Files, add any file type you wish to open with TextPad (e.g. .txt)

OK, back to our scheduled programming.
Below are a few tricks I've come to appreciate with regular expression. I do expect the reader to have basic familiarity with regexp (^ means beginning of line, $ means end of line, . means any character, etc)

Problem: Remove all lines containing a certain text string.
Find: ^.*FINDME.*\n
Replace:
Explained: Locate the string you want, and select the entire line (including the newline character). Simply replace with nothing. Note that if you chose $ instead of \n, then you would end up with a lot of blank lines instead, but assuming that you want to completely remove these lines, you need to use \n at the end.

Problem: Insert line numbers in front of all lines
Find: ^
Replace: \i(100,10)\t
Explained: Find beginning of line, replace with a numeric counter starting at 100 and incrementing by 10 followed by a tab. So now you'll have a first column with numbers 100,110,120, etc. in it. By the way, \i by default starts at 1 and increments by 1.

Problem: You have a text files with dates in a DDMMYYYY (day, month, year) format in column 1 and you would like to quickly convert them over to an YYYYMMDD format. This is quickly done using regular expression.
Find: ^\(..\)\(..\)\(....\)
Replace: \3\2\1
Explained: Create 3 match sets of 2, 2, and 4 characters respectively from the beginning of the line. Simply put the 3 match sets in the desired order. If you are unfamiliar with match sets, then \( and \) define each set and \1 refers to the first set, \2 to the second, etc. Please note that if you use POSIX style regexp then you do not need to escape the parentheses (i.e. use (..) instead of \(..\) ) to create the match sets.

Nifty

Saturday, February 2, 2008

Searching XSLT files

I work a lot with XML and XSLT files, and hence I often need to grep files (i.e. search for text strings in files) for certain term. Windows Explorer does a great job for most file types, but unfortunately the Windows Explorer search function does not include any XSLT files when searching. Just try to create a dummy file called a.xslt with a dummy text string FINDME, and try to locate it using Windows Explorer search. You will not find it. Sad, but true. If someone knows of a simple tweak (e.g. registry change) to change this behavior, then please post it in the comment section of this entry. I'm not sure why XSLT files are excluded. Afterall, JavaScript (.js) and C# (.cs) files are included. Maybe XSLT just made the Microsoft black list since it is not a Microsoft standard.

Assuming for now that we cannot change the Windows Explorer search behavior, I wanted to show you how to search all files including XSLT files, using UNIX command tools. If you just need to search the current folder, you can of course you use
grep FINDME *.xslt

And if you have a fixed 3 level folder structure you may do something like this
grep FINDME *\*\*.xslt
but note that this will ONLY look for files that are exactly 3 directories deep (e.g. it will not search the file named a.xslt in the root folder).

So we need a much more flexible solution that can search files located at any directory level. But let me first introduce yet another UNIX command line tool: find. The command find, recursively lists all files and folders matching a given criterion (e.g. name, create date, etc) and may execute a command on each file found. Again, do a "man find" to get an overview of the options.

List all files recursively:

find . -type f


List all files or directories with the string xslt in them:

find . | grep -i xslt


Find any file that contains the text FINDME
find . -type f -exec grep FINDME "{}" ";"

Note that when using -exec you also need to add those special "{}" ";" characters at the end. Don't want to go into great detail here, but you need to always add it. Otherwise you can basically specify any command you want after exec, like we did with grep in the above case.

Nifty.

What's the difference?

If you work with plain text files, you often have 2 almost identical text files and you need to know what the difference between them is. This is quite easy to do with the UNIX command line tool diff. It will show you all the lines that are different and tell you in which file the additional text is. The diff command has a lot of cool options, so don’t forget to do a "man diff" on it first to see all the options. I personally use -w and -b the most as they ignore all white-spaces, and white-spaces at end of line, respectively.

Question:
What is the difference between 2 text files, ignoring all white-spaces?
Answer:
diff -w 1.txt 2.txt

However, sometimes you do not want to get all the differences, but merely know which text string occurs in one file that does not occur in the other.

Question:
You have 2 text files, and want to see which text strings are unique in column 1 in either file.
Answer:
cut -f1 1.txt | sort -u > 1u.txt
cut -f1 2.txt | sort -u > 2u.txt
cat 1u.txt 2u.txt | sort | uniq -u

Explained:
Cut column 1 from file 1 | remove duplicates | save in temporary file. Repeat for file 2. Then concatenate the 2 temp files with the unique entries | sort the content | show only unique entries as duplicates indicate that the line was present in both files.

If you wanted to show which lines were in both files instead, simply change the option for uniq to -d (to show only the duplicates). cat 1u.txt 2u.txt | sort | uniq -d

Note that this solution does not show you which file the unique string came from. Also, cat is like the DOS command type, though much cooler. It concatenates the content of all files given as argument as sends that to stdout (as most UNIX commands do).

Nifty.

Thursday, January 17, 2008

Command line tools for dummies: Stringing them together

In the last blog entry we introduced a few basic UNIX command line tools. Now we'll start stringing them together to make them really useful.

We'll use the following 3-column tab delimited file containing:
line1 A Mike
line2 B Andy
line3 D Chang
line4 A Tom
line5 B Mike
line6 C Brad

All UNIX command line tool read from stdin (standard in) and output to stdout (standard out) by default. This allows us to string these commands together so the output of one command is used for input for the next command. On both UNIX and Windows, you use the pipe character (aka vertical bar, aka |) to string commands together.

So "grep Mike file.txt | wc –l" will first find lines with Mike in the file named file.txt and then send those lines (2 of them in our case) on to the wc command, which in turn will count them and output the result (2 in our case). Let’s try a few quick examples.

Question:
How many different grades (column 2) did we hand out?
Answer:
cut –f 2 | file.txt | sort -u | wc –l
4
Explained:
Cut field 2 | sort them uniquely (-u removes duplicates) | count how many. An alternative solution is of course: "
cut –f 2 file.txt | sort | uniq | wc –l", but "sort -u" is shorter than "sort uniq".

Question:
What was the distribution of grades (column 2)
Answer:
cut –f 2 file.txt | sort | uniq -c
2 A
2 B
1 C
1 D
Explained:
Cut field 2 | sort them | unique them (remove duplicates) | add a count (note that uniq expects sorted input).


Question:
Same question as the above, but if you have a rather large file with many values, then your distribution will get very long. Often you may just want to see the top 2.
Answer:
cut –f 2 file.txt | sort | uniq -c | sort -rn | head -2
2 A
2 B

Explained:
Cut field 2 | sort them | unique them (remove duplicates) | sort results in reverse and treat as numeric (since we want to sort the added count) | show first 2.

Question:
What grades (column 2) did Mike get?
Answer:
cut –f 2,3 file.txt | grep Mike | cut –f 1
A
B
Explained:
Cut field 2 and 3 (otherwise we could potentially find Mike elsewhere) | find only lines containing Mike | cut out the grade (now in field 1 of the 2 fields we have left).


In the next blog, we'll look at handling multiple input files. But for now, practice with the above.

Nifty.

Wednesday, January 16, 2008

Command line tools for dummies

Let me start by saying that I receive no kickbacks or any other compensation from any of the products that I mention. I would certainly like to, but I don't :-) So the products mentioned are simply products that I have come to love and simply can't live without in my daily life.

Back in the days at the school of engineering at Santa Clara University, I took a great course in UNIX command line tools (grep, awk, sed, etc), and ever since I have not been able to live without them. I mean how do you quickly view the last few lines of a very large log file without a cool command line tool like tail, or how do you quickly find which XLST file that contains the XML tag named recordIdentifier without using a tool like grep.



So when my career took a turn from a UNIX environment to a Windows environment, I was scrambling to find some cool command line tools for Windows. I found my comfort in the MKS Toolkit, which are all the beloved UNIX command line tools compiled for Windows. It is not the cheapest tool ($479), but it is worth every penny if you for example do a lot of text file handling like me. It comes with hundreds of UNIX command line tools, shells, etc.

So the next few blog entries will be dedicated to learning a few nifty tricks on how to use UNIX command line tools. It is not going to be extremely advanced, but it will solve a lot of the everyday problems you encounter. Warning: Once you learn these little tricks, your co-workers with regard you as a complete nerd or even a hack :-) If you are working on a UNIX platform, the tools are of course readily available at the prompt. If you are on Windows, go purchase a set of command line tools (like the MKS Toolkit I mentioned). If you work on a Mac, you probably haven't heard the term command line before :-)

Anyway, let's start out with the very basics by first introducing a few simple tools. The power of command line tools is in stringing them together, but before we get there, we need to understand what each tool does. Please note that you can always access the online help for each tool by typing "man toolname". These online help pages (aka man pages) contain descriptions, available options, and examples, and they are a great source for exploring the tools even further.

For all the examples below, assume that you have a very simple tab delimited text file named file.txt with 3 columns. An easy way to create it is to create it in Excel and then paste it into your favorite text editor. The file content is shown below:
line1 A Mike
line2 C Chang
line3 A Tom
line4 B Mike

OK, here we go with the top 7 command line tools I use the most. Each tool has a large number of options, but the examples just shows one of them.

grep: Find lines containing or not containing text string

-i: option to ignore case (i.e. case insensitive)
C:\>grep -i mike file.txt
line1 A Mike

line4 B Mike

wc (word count): Show number of lines, word, etc
-l: option to get only the line count
C:\>wc -l file.txt
4 file.txt

tail: Show only last few lines.
head: Show only first few lines.
-10: you can specify how many lines you want
C:\>tail -2 file.txt
line3 A Tom
line4 B Mike
C:\>head -1 file.txt
line1 A Mike

cut: Cut file vertically by characters or fields
-f: option to specify fields to cut
C:\>cut -f1,3 file.txt
line1 Mike
line2 Chang
line3 Tom
line4 Mike

uniq: show unique or repeated lines
-c: option to count number of times a line occurs
C:\>uniq -c file.txt
1 line1 A Mike
1 line2 C Chang
1 line3 A Tom
1 line4 B Mike

sort: Sort input lines
-k: specify which fields to sort on
C:\>sort -k2 file.txt
line1 A Mike
line3 A Tom
line4 B Mike
line2 C Chang

Ok, that was the very very basic stuff. Not terribly useful when you only use one command. The real nifty stuff we'll look at in the next blog entry.

Nifty

Wednesday, January 2, 2008

Copy your AutoCorrect list

As you may have read in the latest blog, AutoCorrect is a very useful feature in most Microsoft Office applications.

Once you have created your own personal list of favorite words to expand you probably want to bring a copy of that AutoCorrect list with you to all the other computers that you use. This can easily be accomplished a few different ways. There are several comprehensive tools out there, including some Microsoft tools, but I mostly only need to copy the AutoCorrect list file, so I do it the old fashioned way. I copy the physical file.

The AutoCorrect list file has an extension of ACL (AutoCorrect List). The file is named MSOxxxx.acl where xxxx designates the language that the file is for. There is an ACL file for each language you have used. For example, MSO1033.acl is the ACL file for language of English (US). Lastly, the file is typically located in your application data folders, so you'll likely find it in "C:\Documents and Settings\NN.NN\Application Data\Microsoft\Office", where Windows NN.NN is your user name.

So simply copy the file (e.g. "C:\Documents and Settings\NN.NN\Application Data\Microsoft\Office\MSO1033.acl") to the same location on your other PC and you now have the same AutoCorrect list on both.

As you will now start growing your very own AutoCorrect list, you can even share it between home, work, client assignments, etc. I personally have a USB stick (aka Pen Drive) that I always carry with me, and in my tools folder on the USB drive, I always have the latest version of my ACL file.

Nifty.

Wednesday, December 19, 2007

Use AutoCorrect to type really fast

Like most people, I also use Microsoft Word to write documentation, email, etc. and one of the features I use the most is AutoCorrect. Warning: Once you start using it, it can be hard to live without it.

AutoCorrect is simply a list of characters that will get replaced with other characters. It is the automatic "replace as you type" function, and it is commonly used to fix typical misspellings and create emoticons such as a smiley. For example, if you type in "your here" in Microsoft Word, it
will automatically get expanded to a "you're here". To see the complete list, go to Tools->AutoCorrect Options in Microsoft Word. Note that AutoCorrect is language sensitive, so you have a different list for each language you compose messages in. This of course makes sense since you do not want to autocorrect abotu to about for Danish, but only for English.

But instead of using it to correct misspelled words, I use it to write longer words that I use over and over again. Below is a small sample of my own personal list.

Commonly used words and sentences:
sw=software
dev=develop
devn=development
anyq=Any questions, please do not hesitate to contact me.

You can also use AutoCorrect to write the correct capitalization of a given word or sentence. If you enter a word in lowercase in autocorrect, then Microsoft Word will automatically expand to the correct case. E.g. if you have a lowercase entry of sw for software, then sw will expand to software, and SW (in uppercase) will expand SOFTWARE, and Software if in the beginning of a sentence. Works pretty well. But if you always want to have SOFTWARE in uppercase since it may be a trademark name, etc, then all you have to enter is sw for SOFTWARE.

There are some words that you may want to autocorrect some of the times, but not all the times. The easiest way to handle this is to press undo (ALT-BACKSPACE or CTRL-Z) immediately after the expanding. So if you need to enter the address "123 SW MAIN ST", you want to press undo right after SW otherwise you'll send the letter to "123 SOFTWARE MAIN ST".

Lastly, note that you could also have an entry of uppercase SW. However, this means that SW is only expanded if you enter the uppercase SW. If you entered sw it would not get expanded.

I suggest you play around with it yourself. It is kind of like reading a book on riding a bicycle. It may get you an idea, but you'll not get the feel for it, until you jump on the bike yourself.

By the way, this entry was writing using 8 different AutoCorrect entries.

Nifty.

Tuesday, December 18, 2007

There are stars and then there are stars

Microsoft Word and Microsoft Excel are both fantastic programs that most of us use everyday. Personally I vote for Microsoft Excel as the best and most useful program ever written, but that is a completely different story. But it is funny to see how differently the two programs function despite both being developed by the very same company (but clearly by different departments).

Take searching in Word versus Excel. In Word you can replace an asterisk (*) with a B by simply using Find=* and Replace=B. So A*C becomes ABC.
It (kinda) makes sense, but in Excel the asterisk (*) is suddenly used as a wildcard character that you can not easily turn off. So A*C now becomes B as the whole cell is matched. God forbid someone as Microsoft should discover standard regular expressions.

But how do you then replace an plain asterisk (*) inside text in your Excel file. Well, it turns out that you put a tilde (~) in front of the asterisk. Yeah, don't ask me how tilde suddenly became the escape character instead of backslash. So when you use Find=~* and Replace=B, then
A*C correctly becomes ABC. Again, God forbid someone as Microsoft should discover regular expressions and using \ as the escape character.

Microsoft, in case you are listening, please add an option to use real UNIX style regular expressions. I mean it is a standard after all. But at least we have a way to replace asterisks in Excel.

Nifty.

Saturday, December 15, 2007

DOS Prompt FOR command

The FOR command available in the DOS prompt is surprisingly useful. As you may have seen in the earlier blog the help on the FOR command (i.e. "HELP FOR") has a lot of useful information. I personally find that I use the FOR /L option more than any of the others. For example, if you need to create 100 directories named NewDir0 to NewDir99 you clearly do not want to do that from the Windows Explorer (AKA Windows Exploder). And you do not want to start typing mkdir manually from the command prompt either.

Instead, use the FOR command. But before you start creating a hundred directories, you should always use the echo command to test your FOR loop.

The FOR /L allows you to specify the start, step, and end of the loop, so the command:
C:\>FOR /L %i in (0,1,99) do @echo NewDir%i

Will echo:
NewDir0
NewDir1
...
NewDir99

Once it works the way you want it to, simply replace echo with your desired command (e.g. mkdir) and you whoopty you got yourself 100 directories.

Nifty

DOS prompt file information

Ever need to get the size of a file or get the short DOS 8.3 name from the command line or from a BAT script. Simply type in "HELP FOR" in the DOS prompt to get help on the FOR command. In here you'll also find the nifty tags to get that special information about files passed as arguments. For example, %~s1 will get the short DOS 8.3 name of the first argument, %~z1 will get the file size, %~p1 will get the path, etc.

You may for example create a small BAT file (e.g. named dos83.bat) that contains the following line. When called it will show the short DOS8.3 name of the argument passed in.
@echo %~s1

Output Example:
U:\My Documents\My Templates>dos83 .
U:\MYDOCU~1\MYTEMP~1

Nifty

DOS prompt command history

When working in a DOS prompt you often need to repeat commands, or at least execute very similar commands. There are a few nifty ways to repeat commands in a DOS prompt.

F3: Copy last command to command line.
F7: Shows your entire command history. Press enter to select a command and execute it. Press -> or <- (right or left arrow key) to copy the command to the prompt, so you can modify it.
F8: Command completion (like filename completion). Enter the first few letters of any previous command, press F8 to rotate through all previous commands starting with these letters.

Nifty

DOS prompt navigation (file name completion)

So now that you have the DOS prompt (see earlier blog entry), you need to make your DOS prompt navigation as painless as possible, so make sure that you have file name completion turned on. It is very easy to turn on (if not already on). It is a small registry tweak. Simply paste the following into a file with extension .reg and execute it.

Press TAB and SHIFT-TAB in any DOS prompt to get the file and directory names auto-filled in. You can even type the starting letters to limit to matches found. E.g. in C:\ type in P and press TAB and you'll see all the files and directories starting with P, e.g. "Program Files".

Nifty.

REGEDIT4
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor]
"CompletionChar" = dword:00000009

DOS prompt in any folder

Ever need to quickly get a DOS prompt in a certain folder? It is not easy to do in a standard windows configuration. I often need to get a DOS prompt to run various tools (e.g. Perl tools), and it frustrated me that I have to open a command prompt (i.e. cmd) and then navigate to the folder.

So here is the solution. It is a small registry tweak that allows you to right-click on any folder and get a prompt. Simply paste the following into a file with extension .reg and execute it. In an Explorer window you can also right-click on the folder icon in the very top left corner to get a DOS prompt.

Nifty.

REGEDIT4
[HKEY_CLASSES_ROOT\Folder\shell\Prompt\command]
""="cmd.exe /n \"%1\""

Wednesday, December 12, 2007

Capture screenshot of drop down menu

Once in a blue moon you need to capture a screenshot (aka screendump) and paste it into a document or a web page. As you probably know this is quite easy using the PrintScreen (aka PrntScrn) key which will capture the screenshot and put it on your clipboard. Typically there are two ways to do this.

PrintScreen: Screenshot of the entire screen (i.e. all windows)
ALT + PrintScreen: Screenshot of only current window (i.e. window in focus).

However, what if you need to capture a screenshot of the current window with a drop down menu showing? That is a bit more difficult as the drop down menu disappears as soon as you press the ALT key. So there are 2 ways to do this:

You could grab the entire screen using PrintScreen and bring the capture into Paint (or any other paint program) and cut out the part that you need.

However, there is a much simpler solution. Put focus on the window you want to capture. Close the drop down menu if open. Press ALT + (letter to open the drop down), keep holding ALT down, and press PrintScreen. You can even move the mouse around to highlight the item you want as long as you keep holding down ALT.

For the screen shot below, I pressed ALT + v (to open View), then I moved the mouse to open "Text Size", then I pressed ALT + PrintScreen to capture the window plus drop down. The important thing is to hold down the ALT key the whole time.





2017 update:  You can also use the new Windows built-in capture tool called Snipping Tool. Press Windows key (lower left, typically next to ALT button), and type "snip" and you should see it pop up in your menu. Press ENTER to run. Now drag to capture. Automatically copied to clip-board.  To capture menus, open Snipping Tool, and press ESC to temporarily disable to capture. Now open menus to capture, and press CTRL-PrintScreen to open Snipping Tool.

Happy screen capturing.

Welcome !!

Welcome to the first posting on the "Nifty PC tricks" blog. Here I'll post solutions to and discussions about some of the many PC issues or peculiarities that I run into during my busy life. And I just want to thank Microsoft for keeping the list long. Personally I fall in the category of advanced PC users, somewhere between expert IT dude and novice surfer, so the solution and discussions will be targeted to the advanced PC users too.

Anyway, enough talk. Let's talk nifty tricks.