Like so many others before him, Bernard L. Madoff arrived in the heart of the financial world to no acclaim. It took years, but he built a business. It was a family firm, he was proud to say, with his wife, brother, sons and nephews and a niece all working there. Family became a constant refrain in the life of Bernard Madoff, or Bernie, as everyone called him. But the family has also become an important part of the Madoff story amid the charge he carried out the biggest financial scam of all time - a $50 billion fraud that has deeply shaken Wall Street, the nation's financial regulatory system, and investors across the world. While there is no allegation at this point that family members did anything wrong, they have become central figures in the Madoff saga and his quiet, up-from-the-bootstraps journey from humble beginnings to financial powerhouse to global fraud. There are the two sons who turned him in and are reported to be cooperating with prosecutors. Madoff's wife, who once compiled a Kosher cookbook, has come under scrutiny during the investigation. And Madoff's brother, Peter, was a driving force in transforming Madoff's firm into a renowned investment house and remains a top official in the company. "They were two struggling kids from Queens. They worked hard," said Thomas Morling, who worked closely with Madoff and his brother Peter in the mid-1980s setting up and running computers that made their firm a leader in off-floor trading. "They studied hard, they got good grades and put the firm together with a lot of hard work and sweat," was how Morling heard it. And unlike others in that world, he said, "When Peter or Bernie said something that they were going to do, their word was their bond. Everybody trusted what they said." Those values of family and trust were among the most enduring elements in Madoff's life, but each image was shattered by his stunning fall. Co-workers and friends are left questioning what they really knew of the man whose life's work may end up being a complete lie. "I will say this to you: They were nice, nice people," said Sydelle Meyer, a member of the Palm Beach Country Club where the Madoffs were members and had a vacation home. "I knew them for 30 years. I never, ever would've believed it." Long before the country clubs and cocktail parties there was Laurelton, a low-rise, lower middle-class town with a vibrant Jewish community. Those who grew up there recall Raab's ice-cream parlor, the local movie theater, summers on the wide beaches at the nearby Rockaways. In the financial world, the 70-year-old Madoff's story was well-known - how Bernie Madoff left the city's borough of Queens for Wall Street in 1960 with a few thousand dollars saved from working as a lifeguard and installing sprinklers. As the financial business prospered, the two brothers moved to nice homes on Long Island - Bernie to the upper-middle class community of Roslyn where both sons went to high school. Sons Andrew and Mark Madoff repeated the history for a look at family firms published by a trade publication in 2000. Business was part of daily life at home, older son Mark Madoff said. "All of his family members grew up with this being our lives," he told the trade publication, Wall Street & Technology. After college, each of the sons came to work for their father, both rising to positions of prominence. So did Peter's daughter, Shana; a nephew, Charles Wiener, the son of the Madoffs' sister, Sondra; and another son of Peter's who worked with the firm before leaving for journalism, and then dying at 32 of lymphoma. Shana made headlines following the arrest after it was revealed that she married a former federal securities regulator in 2007. "What makes it fun for all of us is to walk into the office in the morning and see the rest of your family sitting there. That's a good feeling to have. To Bernie and Peter, that's what it's all about," Mark Madoff said in 2000. The sons were clearly on a path toward authority, but that didn't mean it came without dues. Morling, who was there when the sons joined the firm after college, remembered one of them getting roughly rebuffed for proposing a trade regarding IBM that didn't meet Peter's standards. "They were publicly chastised for not having the right stuff together on a trade," he said. "They were being treated just like any other trader in the firm. They were given plum assignments. But they had to perform." The perks of the job included the enormous wealth that their father accumulated: a $7 million apartment on the Upper East Side, a home in Palm Beach, another in the Hamptons. There are reports of a villa in the south of France. There is a 55-foot (16.7-meter) Rybovich fishing boat built in 1969, just a few years after Sports Illustrated called these models "the last word in snob appeal." Memberships in a half-dozen golf clubs scattered across New York and Florida. A share in a corporate jet. "He wasn't the kind of guy that walked into the room and filled the room with his presence," said Charles Gradante, co-founder of a hedge fund advising firm in New York who knew Madoff from charity balls and cocktail parties in Palm Beach. "If you had to pick one of the nicest guys on Wall Street - and there aren't too many - he would've been one of them. He was not a ruthless, tycoon type." Joyce Greenberg had both a family connection and an investment connection to Madoff. Her father began investing with him in 1971. Later, her husband invested with him. "The man was very low-key. He was not a salesman," the retired Houston stockbroker recalled from her one visit to his office in 1987, though she saw him again at a family affair a few years ago. "He didn't try to sell us anything." Madoff made the couple wait a week before he agreed to take their money. She wouldn't say how much she lost, but said her 95-year-old stepmother lost everything and now must rely solely on Social Security. Madoff's appeal to investors was simple: He brought in solid returns of 10 percent to 12% every year, sometimes a bit more. The exclusivity made it even more convincing. Then there's the secrecy. Investigators say Bernie Madoff kept his money management operations on a separate floor from the trading operations. Some potential investors said he refused to share details of his operation. Even less visible has been Bernie's wife, Ruth Madoff. She had helped with the business, at least at the start. She didn't always accompany her husband on the Palm Beach cocktail party circuit, Gradante said. She co-edited a cookbook, "The Great Chefs of America Cook Kosher." One story offers a glimpse of her and her life with Bernie. Five years ago, Michael Skakun, a writer with the Yiddish Forward, was called to the Madoff's Upper East Side home to discuss a surprise project - a professionally written scrapbook of Bernie's life for his 65th birthday. Skakun recalled the ample home, Napoleonic-style desk, Greek and Egyptian statues; a photo of Bernie as a gap-toothed boy with a baseball bat in hand. And then Ruth scrapped the idea, having second thoughts about surprising her husband with something so personal and revealing. In the days since his arrest, the Madoffs have been spending their days in seclusion, with Bernie under house arrest at his Manhattan apartment. Maybe Bernie Madoff summed it up best. He came clean about the fraud to his sons and later to two FBI agents, an FBI agent said in the criminal complaint. "There is no innocent explanation," he told the agents. And to his sons, he said: "It's all just one big lie."