JSON file validator locally

JSON files are great for many things, but they can also be the root of very strange issues. Hence, you should always validate your JSON files whenever you update them to ensure you don't spend hours debugging issues that a really caused by a simple error in a JSON file (yes, I talk from experience).

Validating a JSON file is easy. There are many good online ones, like http://jsonlint.com/. However, if you want to validate many files, it is cumbersome, or if you have sensitive data in your JSON file, you may not want to copy/paste that to a public site.

The solution is simple. Simply install a JSON validator / checker locally. And they also allow for “pretty printing” of your JSON too.  There are a few ways to skin this cat.

Steps (jsonlint):

1. Install NPM. Part of nodejs. Get it from http://nodejs.org/download/
2. Get jsonlint from https://www.npmjs.org/package/jsonlint
3. Reboot (if on Windows)

To run simply open a prompt and type “jsonlint FILE” or "jsonlint -h" to see options.
For multiple files, script it. E.g. on Windows use “for %i in (content\*.json) do jsonlint -q %i”

Steps (Cygwin):

1. Make sure that Python is installed.

To run simply open a Cygwin prompt and type “cat FILE | python -mjson.tool”.

For multiple files, script it. E.g. in BASH “for FILE in content/*.json \n do \n cat $FILE | python -mjson.tool \n done”

Nifty,
Thomas

Cygwin DOS path names issue on Windows 8

I finally upgraded to a Windows 8 machine and noticed that the Cygwin command prompt gave me issues with DOS style path names. This is very easily fixable by using an environment variable called CYGWIN.

However, I already had the environment variable named CYGWIN set, but it still didn't work. Turns out I was using the incorrect case. CYGWIN MUST BE UPPERCASE. Hopefully this quick post can solve your headache too.

Issue:
Every time you use a DOS style path (backwards slash) you get an error like this
cygwin warning:
  MS-DOS style path detected: C:\Intel
  Preferred POSIX equivalent is: /cygdrive/c/Intel
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames

Fix:
In a DOS command prompt (cmd) type
set cygwin

I got the response
cygwin=nodosfilewarning

This is wrong. Not sure why the cygwin word is lowercase. Should be UPPERCASE CYGWIN. To fix, press WINDOWS keys, type "environment" to open the system environment variables window. Under system variables, create a new one called CYGWIN and set to nodosfilewarning. And delete the older "cygwin" (lowercase) one.

Next time you open a DOS prompt and type "set cygwin" you should see
CYGWIN=nodosfilewarning

Now all the DOS errors should go away.
Nifty,
Thomas

Setting function keys back to normal on HP Envy Notebook

Just got a new HP ENVY 15 and so far I love it. Well, except for one little thing that drove my crazy.

Not sure what those pesky engineers at HP were thinking when they decided to change the default way the function (fn) keys work on the newer HP ENVY laptops. By default you do not need to press the (fn) key to get function key functionality.

Example: If you press F5 (even in a browser window) it turns the backlit keyboard on and off, rather than refreshing the browser. Really???

This "amazing" technology is called "Action Keys Mode", and the good news is that it is very easily turned off to get your laptop back to working the standard way of having to press the fn key.

Steps to fix:
- Shut Down your computer completely. Don't put in Sleep or Hibernate
- Press Power button to start
- Immediately press F10 key a few times
- This will start the BIOS
- Go to 3rd page (System Configuration). Press right arrow key to go to next page
- Select the "Action Keys Mode". Press ENTER. Select DISABLE
- Save and Exit

Nifty,
Thomas

Useful RegEx expression for text manipulation

When doing text manipulation, there are a number of regular expressions (RegEx) that I tend to use over and over again. Maybe you'll find them useful too.

Problem	Find	Replace	Example: Before	Example: After
Move text in brackets from end of line to beginning of line	^(.*) (\[.*\])$	\2: \1	Yada yada [BC4] Yada yada [POS123]	[BC4]: Yada yada [POS123]: Yada yada
Convert dates from MM-DD-YYYY to YYYYMMDD	^(..)-(..)-(....)	\3\1\2	11-30-1998 01-23-2004	19981130 20040123
Clean phone numbers (remove space, paren, dash, period)	[ \-\.\(\)]	N/A	999.888.7777 (555) 444-3333	9998887777 5554443333

Nifty,
/T

Deleting multiple empty Google contact groups

My beloved Google Contacts suddenly have ~300 empty "Starred in Android" groups. Not sure how or why they showed up, but many people who have the very same problem are pointing fingers at Nook, and yes, I was using a Nook at the time it occurred.

Anyway, there does not seem to be an easy way to delete all these empty contact groups except manually one by one, which I simply refuse to do. So I wrote a simple script for Google Apps Script to solve the problem. The script is listed below.

Disclaimer: I have tested the script many ways, but please use at your own risk !

To run simply:
- Go to https://script.google.com,
- Start a blank project, and copy/paste the script below
- Run once with ContactsApp.deleteContactGroup commented out (as is)
- Press Ctrl-Enter to view logs to make sure everything looks fine.
- Uncomment the ContactsApp.deleteContactGroup line, and run again to delete empty groups.
- If you want to remove all empty groups independent of name, uncomment the line referencing "Starred in Android"

Nifty,
Thomas

// ============================================================
// What:  Deletes all empty "Starred in Android' contact groups from your Google contacts
// How:   1) Run once to verify that correct groups will be deleted.
//           Review log file (press ctrl-enter)
//        2) Uncomment deleteContactGroup line below and rerun.
//
// Jan 2014, Thomas Gemal
// ============================================================
function DeleteAllEmptyGroups() {
  // Grab all contact groups
  var myContactGroups = ContactsApp.getContactGroups();

  // Use this to limit number of groups processed if performance is an issue
  var MAXCOUNT = 1000;
  var count = 0;
  
  // For each contact group
  for(var i = 0; i < myContactGroups.length; i++) { 

    // Only delete MAXCOUNT groups at a time (since execution can take long)
    if (count == MAXCOUNT) {
      break;
    }
    
    // Get name of a contact group
    var group = myContactGroups[i];
    var group_name = group.getName();

    // Keep this code to only deal with "Starred in Android". 
    // To delete all empty groups, simply comment this 'if' out
    if (group_name != "Starred in Android") {
      Logger.log(Utilities.formatString("%s: Skipped. Not SiA", group_name));
      continue;
    }

    // Get number of contacts for a group
    var num_contacts = ContactsApp.getContactsByGroup(group).length;

    // We are only looking for empty groups
    if (num_contacts == 0) {
      // Ignore special groups
      if (group_name.substring(0,13) != "System Group:") {
        // Log that we want to delete
        Logger.log(Utilities.formatString("%s (%d):  DELETE",group_name,num_contacts));
        count++;
      
        // !!!!! Uncomment line below to physically delete the groups  !!!!!
        // Make sure you have first verified that list to delete is OK
        // ContactsApp.deleteContactGroup(group);
      } else {
        Logger.log(Utilities.formatString("%s (%d):  Skipped. System group",group_name,num_contacts));
      }   
    } else {
      Logger.log(Utilities.formatString("%s (%d):  Skipped. Not empty",group_name,num_contacts));
    }   
  }
  Logger.log(Utilities.formatString("Empty groups processed: %d", count));
}

Fixing localhost email on CodeIgniter under Windows

Argh. After spending a LOT of agonizing hours trying to get localhost email to work under CodeIgniter (PHP) on my Windows 7 machine, I finally figured it out. And since this is such a non-intuitive fix, hopefully I can ease the pain for someone else by sharing this.

Setup:

• Using XAMPP + CodeIgniter
• I had my email.php configured as very standard for localhost
• $config['protocol']  = 'smtp';
• $config['smtp_host'] = 'localhost';
• $config['smtp_port'] = '25';
• $config['smtp_timeout']='10';

Issue:

• My PHP code kept hanging in "$this->email->send()" giving me a "waiting for localhost".
• In my php_error_log I had "Fatal error: Maximum execution time of 300 seconds exceeded in [stuff]\system\libraries\Email.php on line 1869

Resolution:

• Turns out that you need to specify the newlines on Windows
• Simply update your email.php to include:
• $config['newline'] = "\r\n";
• $config['crlf'] = "\r\n";
• Some additional details at the end of this thread http://ellislab.com/forums/viewthread/184596/

That's it.
Nifty

Fixing Carbonite UI skin issues

If you use the Carbonite online backup you may have noticed that Carbonite InfoCenter UI skin at times looks very odd. You may experience javascript errors, and see HTML code referencing Starbucks or hotels. See picture below

The good news is that this is just a caching issue, and that your backup is working as planned. However, it is really annoying to look at. It apparently can happen after you visit public WiFi networks that redirects you to a login page. Anyway, this is how to fix it.

1) Remove the folder with the cached files.
On my Windows 7 system, remove "C:\ProgramData\Carbonite\Carbonite Backup\dyncache"

2) Reload the UI
Go to the system tray
Press the left (not the right) SHIFT + CTRL key while left clicking on the green Carbonite icon. Go under "Skin" and "Reload UI"

Nifty,
Thomas

Missing leading zeroes in XLS

If you are not careful when importing data into Excel (XLS), you often see leading zeroes be removed as XLS tries to be smart about the data it imports. However, this causes issues in Zip codes, SSNs, account numbers, etc. E.g. east coast or Puerto Rico Zip codes start with one or two leading 0s, and suddenly ZIP code 00555 turns into 555. This of course creates all sorts of fun issues in comparison, or if data is used as input into other processes, etc.

To append leading zeroes to a field, simply add a new column with the following formula, and then copy/paste the new column as values to save it. The formula works on any number of leading zeroes missing. Note that “5” in the formula below is the number of digits you expect in the correct length of the field (e.g. 5 for ZIP)

 A       B
ZIP   Fixed_ZIP
555   =RIGHT(“00000” & A2,5)

To add extra protection in case there are blank ZIP codes, simply change the formula to:
=IF(TRIM(A2)="","",RIGHT("00000"&A2,5))

FYI, the simplest way to avoid the leading zeroes being removed is to import the field as text. This is done in the "Text Import Wizard" in XLS that launches automatically when you open a text file. In step 3, simply select set the ZIP fields as text.

Nifty,
Thomas

Grep for value in a column

When doing data analysis, you often need to find records with a given value in a given column, but you want to return the whole line. The grep command will be able to do some of that but has a number of drawbacks. See older post about that at
http://niftypctricks.blogspot.com/2008/08/greping-for-column-value.html

A much simpler (and cleaner) way is to use the awk or gawk command from the command line. Note that the awk/gawk command line may look slightly different depending on whether you are on Linux, Cygwin, MKS toolkit, DOS, etc.

Assuming we have a sample tab delimited text file named in.txt:
line1 joe smith ca 4085551212
line2 joe carlson az 3334445555
line3 carl smith ny 2049998888
line4 joe smith or 5035551234
line5 mike erwin ca 4159876543
line6 mike erwin CA 4159876543

To find all records when column 5 (phone number) contained the value 555, simply use
cat in.txt | gawk -v FS='\t' 'match($5,"555")' > out.txt

The awk / gawk command "match" returns true if pattern1 ($5, meaning column 5) contains pattern2 (555), and by default that prints the whole line, so no need to explicitly put "print $0" in there. So the code sample above will return line 1, 2, and 4.

Nifty #NerdTip,
/T

Pipe to tab

Converting a pipe delimited file to a tab delimited file should be extremely simple. And many times it is if you run under a pure UNIX environment. But if you are under Windows / DOS (using Cygwin) it can be a little tricky, but there is a simple solution.

To convert a pipe delimited file, simply type:
cat FILE | tr "|" '\t' 

The trick here is that the first argument to the tr command has to use double-quotes. The standard single quotes will not work under DOS if the character you are converting is a pipe.

Nifty,
/T

Text editor tab in browser

This is such a cool and simple trick. Creates a simple text editor / notepad in a browser tab. This is useful for storing various kinds of temporarily data, e.g. when copy / pasting multiple items, etc.

To instantiate, simply create a new tab (in an HTML 5 compatible browser), and type in the following in the URL address bar:
data:text/html, <title>Text Editor</title><body contenteditable>

Store as a bookmark so you have easy access in the future.

Or if you want a more permanent version (uses browser's LocalStorage), type:
a5.gg

Much more details, and many "interesting" incarnations at:
https://coderwall.com/p/lhsrcq

Nifty,
Thomas

DOS prompt output to clip board

This trick is so simple, yet so powerful if you dabble in the DOS Prompt a lot. You can pipe the output from any command to the clipboard simply by piping it to "clip". That;s it. No more copy paste from the line wrapped DOS prompt output.

Example:
dir | clip
will give you the output of the DIR command to the clipboard.

Or if you are running Cygwin or MKS toolkit, and want to get the counts of instances in column 3 of a file
cut -f3 myfile.txt | sort | uniq -c | clip

Nifty,

Mailto fix for Gmail

I love Google. Well, in a nerdy kind-of way. I love search, Chrome, Android, Gmail, Docs, Calendar, Goggles, flight search, Wallet, etc. If it says Google, there is a good chance that I will like the product.

So when on my Windows machine at work (no, we have not upgraded to chrome OS yet), it is handy to be able to click an email address link (i.e. a mailto: link) and have a browser with my Gmail pop up instead of my Outlook popping up. Notice that this will switch to a web browser Gmail for all mailto links whether they are on the web, or in an email inside Outlook.

Two ways to skin the cat: 1) You can either download the Gmail Notifier application and set the default there (look at this Gmail help) , or 2) (even better) you can simply edit one key in the registry yourself. Simply open your favorite text editor (e.g. TextPad), copy the following 3 lines (including quotes and backslashes), and save the file with a .reg extension, e.g. Gmail.reg. Then simply execute the file. You can also manual update the registry using RegEdit, but that is a tad more error prone. Please note that you need to replace USERNAME with your own Windows login name.

For Windows XP, use:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\mailto\shell\open\command]
@="\"C:\\Documents and Settings\\USERNAME\\Local Settings\\Application Data\\Google\\Chrome\\Application\\chrome.exe\" https://mail.google.com/mail?extsrc=mailto&url=%1"

For Windows 7, use:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\mailto\shell\open\command]
@="\"C:\\Users\\USERNAME\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe\" https://mail.google.com/mail?extsrc=mailto&url=%1"

Now, any email (mailto) link you click on will open up your gmail web.
Nifty.

Searching in Excel and Word

Searching and replacing text and special characters in Microsoft Excel and Word can get, uhmm, a tad complicated. Even for people very familiar with standard regular expressions (RegExp) since Microsoft unfortunately decided to build their own standard rather than using the normal regular expressions. And to complicate things, Microsoft Word and Excel don't use the same wildcards.

So here is a quick recap on how to search and replace in Microsoft Excel and Word. Please ignore all the double quotes as they are only to make the terms stand out.

For both Word and Excel:

• Use "?" (without quotes) to find a single character (a letter, a digit, etc). So to find the words "sat" or "sit", you can search for "s?t". Note that it will not find "shoot" as "?" can only be a single character. The "?" is the same as "." (period) in regular expressions.
• Use "*" (without quotes) to find a any number of characters (letters, digits, spaces, tabs, newlines, etc). So to find the words "shot" or "shoot", you can search for "sh*t". Note that it will also find the string "shot" in the word "gunshot", and it will even find the phrase "show up at" since it starts with "sh" and ends with "t". The is the same as ".*" in regular expressions.

The following only works in Word (not Excel)

• Use "<" and ">" (without quotes) to mark the beginning and ending of a word. So if we only wanted to find the word "show" and not "gunshow" then we can use "<show" to look for a starting word boundary, but of course it would also fine "showing". To only find the word "show" simply use "<show>"  Note that "<sh*t>" may find phrases too as * includes spaces, so it may find "shot" or even "show up at".
• Use "^t" (without quotes) to search for the tab character.  The is the same as "\t" in regular expressions.
• Use "^p" (without quotes) to search for a paragraph mark (i.e. newline).  The is the same as "\n" in regular expressions.
• Use "@" (without quotes) to search for one or more occurrences of the previous character.  So "f@" will find both "of" and "off". The is the same as "+" in regular expressions. 

To search for the wildcard characters themselves (e.g. wanting to find a *), please refer to earlier post.

So thanks to Microsoft for another straight forward implementation :-)

(Somewhat) Nifty

Picture date-time stamp editing

I use Google's Picasa to edit / track my pictures, but I was very disappointed to learn that it does not have a basic tool to edit the date/time stamp of a JPEG.

Simple scenario: Your not-so-technical friend sends you 200 pics taken with their brand new camera. But of course the camera date was set to Jan 01 2001. So now you need to update all the pictures with the correct date in the header.

Solution: Go download a small free command line tool named jhead.
To change the date simply go:
jhead -da NEWDATE-OLDDATE *.JPG

(Looks like their documentation incorrectly specifies the opposite). And try it on a single file first. Their full documentation is here. It calculates the difference between OLDDATE and NEWDATE and applies that difference to all the pictures. So simple and so powerful.

To have Picasa notice the updated JPEG header, you may have to change the date stamp of the file too (not only the JPEG header). To do that:
jhead -ft *.JPG

Nifty.