During a live television interview, he advised women to cover up to prevent temptation.
Mr Khan added "in any society where vulgarity is prevalent, there are consequences."
Hundreds of people have signed a statement demanding an apology.
Pakistan is ranked as one of the most dangerous countries in the world for women in terms of safety and equality.
Sexual abuse, so-called "honour" killings and forced marriage are common, and criminal complaints are rarely reported to police, or seriously investigated.
In a two-hour-long, question-and-answer interview with the public on live television, Imran Khan was asked what steps his government had taken to prevent sexual abuse.
After denouncing crimes against women and children, he said that sexual violence was a result of "increasing obscenity", adding it was a product of India, the West and Hollywood movies.
The prime minister said women in Pakistan should remove "temptation" because "not everyone has willpower". He said women should observe purdah - a term referring to women wearing modest clothes around men or segregation between sexes.