Red, Green, Refactor

Last night, I was asked why one should refactor if the test is passing. The test-driven development mantra of “red, green, refactor,” is ingrained in many practitioners' head, but why exactly should you bother with this ‘frivolous’ third step? When you write perfect code, why bother?