Remember most of the plant is not the fruit and most of both is carbon, oxygen, and hydrogen which are as readily available as N2. Phosphorus and magnesium are highly utilized and aren't so readily available so they must be replaced. Same for sulfur, but maybe to a lesser extent. The only other things necessary are micronutrients (iron, Mo, etc.) but they are consumed in minuscule quantities. Most of the P, Mg, S, micronutrients , as well as a lot of C, H, O, and N, get recycled in the non-fruit parts (leaves, stems, roots).