Mr Brown, prime minister since 2007, said he wanted a successor to be in place by the time of the party's conference in September.
Mr Brown announced his intention to quit in a statement in Downing St in which he also said his party was to start formal talks with the Lib Dems.
The Conservatives won the most seats and most votes in the election and have been in talks with the Lib Dems.
Mr Brown said no party had won an overall majority in the UK general election and, as Labour leader, he had to accept his part in that.
He said he had no desire to stay in his position longer than was needed to form a stable government, and that he would ask the Labour Party to set in form the process of a leadership contest.
He said it could be in the interests of the country to form a "progressive" government - possibly in coalition with the Lib Dems - the UK's third largest party.
It emerged earlier that the Lib Dem negotiating team, who have held days of talks with the Conservatives, had also met senior Labour figures in private.
The BBC's political editor Nick Robinson said one of the stumbling blocks to any Lib Dem-Labour deal had been Mr Brown himself.
In spite of myself, I'm sad to see him go. He's handled himself with a lot of dignity over the last week or so. But I think he can go happily knowing that he's made Cameron's life one hell of a lot harder.