Three bulldozers were working under police protection to demolish part of the former Hotel Shepherd, said an AFP photographer at the scene in the Sheikh Jarrah neighbourhood.
"The north half of the building is being demolished and the Israeli authorities want to build in its place 20 homes to benefit settlers who are forming a Jewish neighbourhood here," Hagit Ofran, an anti-settlement activist with Peace Now, told AFP.
Police spokesman Micky Rosenfeld confirmed that "police forces have been deployed to the area to maintain calm." Protesters were at the scene, but there were no initial reports of conflict.
Final approval for construction of the 20 new apartments came in March, as Israeli Prime Minister Benjamin Netanyahu held talks in Washington.
The project is financed by US millionaire Irving Moskowitz, who has reportedly provided the money for dozens of Jewish settlement projects in east Jerusalem.
The project envisages 20 luxury apartments, to be built around part of the hotel that will remain standing.
The now-dilapidated former hotel was once the residence of Jerusalem's mufti Haj Amin al-Husseini, who became infamous for his ties to Nazi leader Adolf Hitler.
Though occupied east Jerusalem is largely Palestinian, an increasing number of hardline Israeli settlers have moved into the area's neighbourhoods, sparking fights with Arab residents.
An estimated 2,000 Jewish settlers live in Palestinian neighbourhoods in east Jerusalem, although the exact number of properties owned by them is unclear.
The Palestinians regard east Jerusalem as the capital of their promised state and fiercely oppose any attempts to extend Israeli control over it.
Israel captured east Jerusalem in the 1967 Six-Day War and later annexed it in a move never recognised by the rest of the world. The Jewish state considers the whole of Jerusalem its "eternal and indivisible" capital.