For a average computer user 4GB is more then enough adding more RAM would not make a difference in speed not even when you use 64-bit version of Vista. This is because your PC only uses the RAM needed and knowing that most computer users wont use that much memory it is a total waste of money.
Its a different story for Advanced users or for Server.