The problem isn't one of absolute numbers, but of the ratio of retirees to workers -- specifically, of people drawing Social Security benefits to people paying into the Social Security fund. In 1960 there were about 5.1 workers per retiree in the U.S. economy, currently there are about 3.4 workers per retiree, and the number is projected to drop to about 2.1 workers per retiree in 2030.
Additional projections include the following:
(1) Annual Social Security tax receipts will exceed annual Social Security payouts until 2016, when expenditures will be greater than tax receipts.
(2) Annual Social Security Trust income (tax receipts plus interest) will exceed annual Social Security payouts until 2025, when expenditures will be greater than income and will initiate a drawdown of the Trust.
(3) The Social Security Trust will be depleted in 2038. Ongoing tax income will cover 73 percent of promised benefits, continuing down to 67 percent in 2075 and heading even lower from there.
Of course, we know all too well that government projections are fallible; but the general demographic trend seems firmly established, and the financial consequences follow from the demographic constraints.
A useful document on all of this is the Social Security Advisory Board's 2001 report, "Why Action Should Be Taken Soon" (650+ KB):