The State Attorney's Office and police are trying to bring down Prime Minister Ehud Olmert because of his government's crusade against the legal establishment, sources close to Olmert charged over the weekend in reaction to the new investigation against him announced on Friday. The State Attorney's Office accused Olmert of asking several charities and institutions to pay for the same work-related trips abroad, including Yad Vashem, the Simon Wiesenthal Center, the Soldiers Welfare Association, and the Industry and Trade Ministry, and then using the excess funds to pay for private trips for himself and his family. Speaking to reporters before leaving for Paris on Saturday night, Olmert called the accusations that he stole thousands of dollars from the charities "distorted" and "detestable." The sources close to Olmert suggested that if it were not for Olmert's appointment of Justice Minister Daniel Friedmann and the legal reforms he had initiated, the prime minister would not have been targeted so aggressively. "The only way to remove Friedmann is to end the tenure of Olmert, and no one in the police or the prosecution would be upset if Friedmann left office," a source close to Olmert said. Police suspect the "considerable sums" that remained after the work-related flights were paid for "were transferred by Olmert to a special fund that [his] travel agency administered for him. These monies were used to finance private trips abroad by Olmert and his family," police and the Justice Ministry said in a joint statement. In response, Olmert said he had devoted much of his time and energy over the years to help the organizations named and to raise funds for them. "Using this matter to try to link the investigation to favors for my family is abominable," he said. He also blasted police leaks to the media, saying they "went against the acceptable norms of a democratic society." Speaking to reporters on his plane, Olmert warned that as a result of this type of conduct, the public was in danger of losing confidence in law-enforcement authorities. Olmert's press adviser, Amir Dan, downplayed the allegations. He said they were exaggerated and that the timing of the prosecution's announcement was not coincidental. "The earth did not shake and the sky did not fall," Dan said. "The police and the State Prosecutor's Office probably realized that there is nothing behind Talansky's testimony and therefore, even before it had begun, are trying to generate a [media] spin by creating drama from nothing. "This material had already been transferred to the police three months ago. The urgency of this matter is unclear. Why now, other than as an attempt to divert the public's attention from Talansky's testimony?" Olmert's associates said the prime minister was "surprised and hurt" by the new allegations. "He never took a penny for himself from the organizations," a source close to the prime minister said. "He raised money on their behalf and was paid for his expenses and that's all." Olmert's opponents in Kadima said the new investigation would not expedite their efforts to remove Olmert from office, because a Kadima primary had already been set for between September 14 and 18. They said that if Olmert were forced to quit, he would remain prime minister of a caretaker government for 90 days, while the Kadima primary was at most 67 days away. The new allegations are expected to make it even less likely than before that Olmert will decide to run in the primary. "We didn't need this latest development to realize that we must to replace Olmert," a source close to Foreign Minister Tzipi Livni said. "Kadima will elect a new chairman soon anyway." Labor MKs Shelly Yacimovich and Ophir Paz-Pines released a joint statement calling on Livni and the three other Kadima leadership candidates to take action to remove Olmert and to "stop doing nothing and prove that they really are committed to the rule of law." Yacimovich called on Olmert to resign immediately, saying "his decision to remain in office is a heavy burden on Israeli society, which deserves a prime minister who isn't suspected of criminal activities." Likud faction chairman Gideon Sa'ar mocked Labor for remaining in the government and allowing Olmert to remain prime minister. National Religious Party chairman Zevulun Orlev said the new allegations against Olmert "are new black clouds that are joining the other black clouds that are already floating above Olmert's head." "In any civilized country Olmert would have lost his position a long time ago," Orlev said. MK Zehava Gal-On (Meretz) said Attorney- General Menahem Mazuz "will be abusing his position if he doesn't remove Olmert from his position immediately." Former Supreme Court justice and current Israel Press Council president Dalia Dorner said Saturday that Israel's political culture was deficient and the public seemed indifferent to the corruption of its leadership. Dorner said the general public must eject leaders such as Olmert from office when severe allegations of corruption were made against them. "When there are clear facts, and in Olmert's case there are many facts that he himself does not dispute, public figures must reach the necessary conclusions," Dorner said. "I expect the public to apply heavy pressure, so that in a situation such as Olmert's, the prime minister will not be able to retain his position." In Paris on Sunday, Olmert plans to hold a bilateral meeting with Palestinian Authority President Mahmoud Abbas and a trilateral one with Abbas and French President Nicolas Sarkozy. Jonathan Beck and AP contributed to this report.