There is no time limit, however, on civil fines for tax fraud. You are correct. The IRS can go back as long as they want.