Prominent Jewish philanthropist Stanley Chais may have lost his $178 million charitable fund in the collapse of Bernard Madoff's Ponzi scheme, but a court trustee says Chais owes other swindled investors $1 billion in profits he made from the scam. According to a complaint filed Friday by Irving Picard, who is overseeing the bankruptcy proceedings against Madoff's money-management firm, Chais was paid out of money that came from new investors, not from legitimate market trades. Picard asked the court to force Chais to forfeit the money to help pay those thousands of investors back. Picard alleged that Chais's telephone number was at the top of the speed-dial list in Madoff's offices, and argued that indicated "unusually intimate access" to Madoff that may have allowed him special access to information about the way Madoff operated. Chais's accounts "received unrealistically high and consistent annual returns of between 20 and 24 percent," according to the complaint. Picard described those returns as "implausible." Chais, who funnelled capital to Madoff from his own money-management business in Los Angeles, is also being sued by his own investors on claims that he breached his fiduciary duty by handing their savings over to Madoff. Chais has insisted since Madoff's arrest in December that he knew nothing about the fraud. His lawyer said Friday that Chais also lost his personal fortune in the scam. "To the extent that the Trustee has alleged that Mr. Chais and his family received any kind of preferential or beneficial treatment from Madoff, it is important to understand that Mr. Chais and his family have suffered astounding and ruinous losses from the Madoff scheme," the attorney, Eugene Licker, said in a statement. Chais's family foundation gave about $12.5 million annually to support Jewish communities in Eastern Europe and the former Soviet Union, but was shuttered in December after its endowment evaporated, along with anywhere between $20 and $62 billion in money Madoff claimed to have held for unwitting investors. Madoff pleaded guilty in March to fraud and remains in a lower Manhattan jail awaiting sentencing. The 70-year-old faces up to 150 years in prison. A lawyer for Picard, David Sheehan, said the claim against Chais was the first that would be brought against middlemen who may have acted as insiders or who benefited from the scam to the detriment of other customers. AP contributed to this report.