Looks like there is a PAM module for it, but I've never implemented anything like this before.  Doesn't look like it would be that hard.
http://code.google.com/p/google-authenticator/