Now, in a clear break from the past, a new prime minister has gone where none of his predecessors dared go: He has ordered a panel of ministry officials and academics to investigate the secret agreements.
The findings, due out this month, are part of Prime Minister Yukio Hatoyama's wide-ranging campaign to wrest power from the bureaucracy and make government more open than under the conservatives, who ruled Japan for most of the past 50 years.
They also could intensify public debate about the future of Japan's long-standing security alliance with the U.S., which has bases here. Hatoyama, a liberal who took office in September, has called for making the relationship more balanced, starting with efforts to evict an unpopular U.S. base from the island of Okinawa.
That Japan agreed to let nuclear-armed ships enter its ports and waters ceased to be a secret some years ago with the declassification of American documents. Such ships had routinely docked in various Japanese ports since the 1960s, sometimes setting off protests.
But in a nation where memories of Hiroshima and Nagasaki drive a fierce aversion to nuclear weapons, a formal admission of the secret agreements would be a stunning reversal, and confirm that previous governments systematically lied to the public.
"The Foreign Ministry repeatedly denied their existence, even in statements before Parliament," lawmaker Muneo Suzuki said in an interview with The Associated Press.
Suzuki held top political posts at the Foreign Ministry, yet although he had heard about the secret documents, he said that even he could not pry them out of his officials.
"The Foreign Ministry should be held deeply accountable," said Suzuki, who has switched sides and is now a member of Hatoyama's coalition.
Historical accounts show that Prime Minister Masayoshi Ohira, who died in office in 1980, considered going public on the secret pacts, but was advised against it by his aides as politically too dangerous.
Only a few Foreign Ministry bureaucrats have spoken out in recent years.
One, Kazuhiko Togo, said he and other high-ranking officials kept quiet for fear that disclosure of the agreements would trigger riots and perhaps topple the prime minister.
"The political costs were too great," Togo told the AP.
Even after American officials acknowledged the pacts in the 1990s, leaders of the long-ruling Liberal Democratic Party persistently denied them, right up to Taro Aso, the last LDP prime minister before Hatoyama's Democrats took over.
"They did not exist," Aso said in a nationally televised response to a reporter's question last July.
"It all goes to show how far behind Japan is in administrative transparency," said Koichi Nakano, professor of political science at Sophia University in Tokyo.
Even the name of revered Eisaku Sato, the prime minister viewed as the architect of Japan's postwar pacifism and resistance to nuclear weapons, has been thrust into the debate.
Three weeks ago, Sato's son revealed a document he found in Sato's desk after his death in 1975 and which he kept hidden.
The 1969 document, signed by Sato and President Richard Nixon, showed they agreed that U.S.-occupied Okinawa would be returned to Japan, but the U.S. would retain the right to have nuclear weapons on the island if the necessity arose. The agreements on Okinawa were a key part of the secret pacts that also covered U.S. warships entering ports throughout Japan.
Back then, it was the height of the Cold War, and the U.S. felt it needed a free hand to confront nuclear-armed China and the Soviet Union.
But the deal with Nixon was a clear violation of Sato's pledge that Japan would not make, own or allow the entry of nuclear weapons. Sato won the 1974 Nobel Peace Prize in large part for pushing those principles. According to Japanese media accounts, the trade-off drove him to tears of remorse. But the principles became policy all the same.
The previously declassified U.S. documents include State Department papers on the 1960 U.S.-Japan security pact, accounts of meetings at which the entry of warships with nuclear weapons was discussed and a memorandum on the 1969 Nixon-Sato meeting, where the Okinawa deal was discussed.
And even in the 1990s, after U.S. warships stopped carrying battle-ready nukes and the issue became moot, it remained sensitive enough for governments to go on misleading the public.
Japanese today are more shocked by the cover-up than by the deed itself, but they remain attached to the non-nuclear principle.
A survey by the Mainichi newspaper, which interviewed more than 4,500 people, found 72 percent of the 2,600 respondents want to stick with the principles, and the number rose to about 80 percent among Japanese in their 20s and 30s. No margin of error was given.
Shoji Niihara, a scholar of U.S.-Japan relations, said Japanese are hoping their new reformist prime minister will redefine Japan's relationship with the U.S. and work with President Barack Obama in his call for a world free of nuclear weapons.
"There's a strong feeling that Japan was never truly treated as an independent country," he said.
Robert A. Wampler, a senior fellow at the National Security Archive, an American group that seeks to declassify historical documents, welcomed Hatoyama's investigation.
"The longer they denied this, the harder it was for them to come forward and say they weren't telling the truth. They backed themselves into a corner on this one," Wampler said in a telephone interview from Washington, D.C.
Bunroku Yoshino, a former Foreign Ministry official who oversaw relations with the U.S., did his part on Dec. 1.
Testifying in a lawsuit brought by a former newspaper reporter, 91-year-old Yoshino reversed his earlier denials and acknowledged signing some of the Okinawa agreements.
"It is a major historical truth," he said afterward.