Yes by all means visit Kampot and Kep on your way to Sihanoukville. Since it is only a 3hr drive and if you leave early, you can be there by noon. You have to go through Kampot to get to Kep (25kms - 1/2hr drive) and you could have lunch here and sample their famous fresh seafood. The road opposite the beach has many food vendors. Just sit yourself down at a picnic table and one will come right over. Most have english menues. Crab is the speciality. When you order make sure he knows how much food you want and what the price will be.
After lunch look around the small town and then on to Kampot for a short visit. Both places do not get a lot of tourists (this is changing as more people become aware of how nice the area iis) and the people are very friendly and especially the kids whom what to practice their limited english on you. Quite a change from the touristry overpriced Siem Reap and even PP. Gives you another aspect of Cambodia.