Mr. Obama was asked about the remarks Wednesday while concluding an economic talk with residents in Ohio.
The president's comments on the mosque proposal have drawn fierce criticism from Republicans and others who say an Islamic center should not be built near the site where more than 2,600 people were killed by al-Qaida.
President Obama later clarified that though he supports the "right" to build a mosque near the site, he would not comment on the "wisdom" of doing so.
Speaker Nancy Pelosi said Wednesday that where the mosque is located is a "local decision." She echoed President Obama's comment that freedom of religion is a constitutional right. Pelosi also called for transparency about who is funding the effort to build the Islamic center and who is funding the attacks against its construction.
A statewide poll indicates nearly two-thirds of New York voters also believe the project is protected by the Constitution, even though a majority oppose the plan.
Sixty-three percent of voters surveyed in the Siena College poll are against the project, while 27 percent support it. The remaining 10 percent expressed no opinion.
An opinion poll released last week by CNN/Opinion Research found nearly 70 percent of all Americans do not agree with the mosque proposal. The mosque and cultural center would be erected less than one kilometer from the site known as "Ground Zero," where al-Qaida hijackers crashed two passenger planes into the twin towers of the World Trade Center
A group dedicated since the 2001 attacks to opposing what it describes as "radical mosques," jihad and "harsh sharia law" is planning a rally this Sunday to protest construction of the mosque.
The Coalition to Honor Ground Zero says firefighters, victims' families and construction workers vowing not to build the mosque will also take part in the event.
Tuesday, a group called the Muslim American Society Freedom warned of what it called a "growing pattern" of opposition to mosque construction across the country. Many estimates put the current number of mosques in the United States at about 2,000.
Supporters of the 'Ground Zero' mosque say it will help bridge divisions between the West and the Muslim world, and say the terrorists who carried out the attacks do not represent Islam.
The prayer room would be part of a $100 million Islamic center featuring a 500-seat auditorium, sports facilities, theater and restaurant, and would be open to all visitors.