Their only similarity is that they reduce the number of shares on the market, and thereby drive up the share price because the company should have the same market cap, but now divided by the new lower number of outstanding shares, so each share has a higher value.
The BIG difference between a RS and a buyback is that in a RS the number of shares each stockholder owns is reduced by the divisor. So if you have 100 shares, and there's a 1:2 reverse split, you'll now have 50 shares, and each share will be double the price it was a day prior, but the $$ amount total is the same. A 1:10 will mean you'll have 10 shares, and each share is worth 10x as much as the day before, but again, the $$ amount is the same.
With a buyback, you have the same amount of shares as you originally owned. The company is buying shares from the open market and retiring those shares. So the share price goes up, and now your $$ increases too. So a RS is pretty terrible, and a buyback is pretty awesome.