The Australian Services Union, which represents workers in the female-dominated community services sector, organised the nationwide rally - Australia's biggest equal pay march since the 1970s.
Women earn 18 per cent less than men, which amounts to about $1 million over a lifetime, recent Australian Bureau of Statistics show.
But the union's NSW secretary Sally McManus said the pay gap could be as high as 35 per cent in the community sector.
"It is unacceptable that the pay gap is getting bigger and not smaller," she told a crowd of about 2000 people inside Sydney's Town Hall today.
Ms McManus said the rally was organised to support the union's test case with Fair Work Australia on pay equity for community sector workers.
She said the union, with the support of a range of other unions, was fighting for pay rises of about 25 per cent for community sector workers.
This would equate to an average $100 a week pay increase for the 200,000 people who work in women's refuges and help migrants, the homeless and those with drug and alcohol addictions.
After an hour of speeches in the Town Hall, the rowdy crowd poured into the city's streets, waving banners and placards and chanting "What do we want? Equal pay. When do we want it? Now".
They marched to the head office of Employers First, which Ms McManus said was the one employer body that was opposing the union's fight for better pay.
"There will be employers who don't want us to win this case," she told the crowd.
"There is one employer body opposing us now. This group is Employers First.
"They have a $1.5 million war chest dedicated to opposing us."
Today's rally comes 38 years after it was decided by the Arbitration Commission that women who were performing the same work as men should get the same award rate of pay.