Mr Karroubi's son told the BBC he had heard his father had been moved, but did not know where he had been taken.
A website close to Mr Mousavi claims the men have been taken to Heshmatiyeh jail in Tehran. Iran denied the report, a semi-official news agency said.
The incident comes ahead of planned protests to be held on Tuesday.
The semi-official Fars news agency quoted an unnamed official in Iran's judiciary as denying that the men had been taken.
"The two are currently in their homes and there have only been restrictions on their contacts," the official said.
Both Mr Mousavi and Mr Karroubi have called for demonstrations in Iran in the light of recent uprisings in Tunisia and in Egypt.
Earlier this month the two men, along with their wives, were detained in their respective homes in Tehran as protests were staged on the streets of the capital.
On Monday one of Mr Karroubi's sons told BBC Persian service he had been told his father had been "taken by security forces to an unspecified location".
Mr Mousavi's Kaleme website reported that the men and their wives "have been arrested and were transferred to the Heshmatiyeh prison of Tehran".
"According to credible sources, the arrest and the transfer to jail is sure but the date when it occurred is still uncertain," it said.
A neighbour said that he had seen eight security vans outside the politician's home late on Thursday night. Shortly afterwards the vans then left the area along with one car, he said.
Thousands of supporters of Mr Mousavi and Mr Karroubi took to the streets of Tehran on 14 February, amid clashes with security forces.
Some 1,500 people have been detained, opposition groups said, but official figures put the number at 150.
Following the deadly clashes, Iranian MPs called for the two opposition leaders to be tried and executed.