The most common reason to refinance is savings of mortgage interest; but one has to consider all the factors involved in refinancing such as:

Penalties for early termination for existing loans.
Fees for new loan
Remaining tenure and loan amount.

The final decision should rest on whether the nett interest saving is worse the hasssle of refinancing.

I do agree that one must take into consideration the saving of the interest as well as the penalties.There are certain banks which do not impose any such penalties.Still the refinancing depends on one's needs and circumstances.

The basic criteria is simple.  Tabulate your NETT interest savings (more savings if no premature termination penalties from the bank) and see if it is worth the trouble of doing the extra paper work due to switching from one bank to another

We don't yet have a house but my intentions are not to EVER refinance for the simple fact that I think it would be less beneficial unless you get a very LARGE interest decrease.