All religions treat sexuality with suspicion. There are possible evolutionary reasons behind this. It is why more than one spiritual figure has called women the gateway to hell. It is why many temples have rules against allowing menstruating women in. It is why many orders of monks spend lifetimes without any sexual contact.
The hole in this approach of course is that if celibacy had been mandatory for spirituality, Krishna would be one of the least spiritual figures. So would Rama, and all the Pandavas.
Hinduism is not really about a single path. It is about a single goal which can be reached using many paths. For some, forsaking women is a way to achieve the kind of mental discipline they require for spiritual progress. Others take another path, one where women are not absent.