I used to feel the exact same way about Norton until I tried their A/V program on a trial with the last computer I bought and the difference between their new version, and previous versions is beyond night and day. I do not notice any slow downs at all using Norton now.
Heres a screen shot of running processes, you can see that Norton (Symantec) isn't that big of a memory hog at all anymore. They really have come a long way towards streamlining their products.