While you don’t have to have a U.S. passport to vacation in the islands, or to come back to the mainland, you do need a passport if you’re going to do some sightseeing in the BVIs.
That mans if you’re planning on taking a day trip or charter to Foxy’s or the Baths on Virgin Gorda. bring your passport\. You’ll need it as proof of citizenship when you return from your BVI visit and clear through the U.S. Customs office in Cruz Bay.
Here's a link to the passport Office's regulations concerning the British Virgin islands.