Can someone recommend a Northeast vacation destination?

I live in New York City and am looking to take a short summer trip of about 5 days, I would like to go somewhere in New England but on the coast. I am looking for a charming but quiet seaside place, sort of undiscovered without a lot of tourist traffic. A craggy New England coast with maybe a lighthouse, good seafood and a quiet vibe. Not looking for nightclubs, sports or children's attractions. I just a quiet place where I can enjoy ocean walks, cooler weather, seeing boats and historic homes, and have nice meals. Is there any place like that I can get to without flying? I don't mind a drive of maybe 5-6 hours.