I think four days would be about right in Siem Reap. It's a small place and it would be enough to allow a couple of days seeing temples and a couple more doing other things like shopping, visiting the cultural centre or looking around the market or the town in general.
We found seeing two temples was quite enough - very hot and dusty, also tiring.
If you cut back on Siem Reap, you could spend the extra 2 days in either Hanoi or Saigon (we were in Saigon at Easter and thought it was great)
We didn't get to do all we wanted in the time we were there but enjoyed it just the same - see my JBR.