A suburban New York town is threatening criminal action if work continues at Donald Trump's estate, where the Libyan government pitched a tent that could be used by Muammar Gaddafi, the town's attorney said Wednesday. Inspectors planned to visit the property Wednesday in Bedford, about 64 kilometers north of Manhattan. "If no activity has been undertaken, so far as either removing the tent or other equipment, or removing any individuals who may be residing in the tent, then we would then proceed to take one of two types of enforcement actions," said attorney Joel Sachs, who represents the town. He said a criminal summons could be served for violating town codes, or the town might "seek an injunction to have the tent taken down and individuals residing in the tent removed from the property." Sachs said it was not immediately clear what entity would be the target of any legal action. On Tuesday, officials found workers constructing the tent but could not communicate with them because they didn't speak English. They gave the stop work order to the property's caretaker. The Trump Organization said Gaddafi would not be coming to the property and insisted that Trump has not rented property to him. But it said part of the estate "was leased on a short-term basis to Middle Eastern partners, who may or may not have a relationship to Mr. Gaddafi." A call to Trump's office requesting further comment was not immediately returned on Wednesday. Sachs said Trump once proposed an 18-hole golf course on the site. Those plans fell through, and he's now seeking approval for "a high-end residential development" there. A State Department official, who spoke on condition of anonymity because of diplomatic sensitivity and protocol concerns, told the AP that the property was obtained for the duration of this week's United Nations General Assembly. The official said no one would be staying there overnight. Gaddafi's motorcade arrived at the Libyan Mission to the United Nations in Manhattan, just blocks from UN headquarters Tuesday evening. Dozens of police and Secret Service officers blocked off the venue. Several dozen pro-Gaddafi demonstrators rallied near the office building housing the mission, carrying his portraits and chanting pro-Libyan slogans. In Bedford, no-parking signs went up near Trump's Seven Springs estate. Area residents include Martha Stewart and Ralph Lauren. TV helicopters showed a tent on the Trump property. Police would not comment, and Secret Service spokesman Ed Donovan would say only that the agency does not discuss the schedules or logistics of people it's protecting. As word got out, local officials quickly objected to Gaddafi's anticipated presence. Gaddafi will likely face protests over Scotland's recent release of Libyan Abdel Baset al-Megrahi, who was convicted of the 1988 bombing of Pan Am Flight 103 which killed 270 people. Gaddafi had wanted to pitch a tent at Libya's five-acre (two-hectare) estate in Englewood, New Jersey, and live and entertain there during the UN assembly. But local opposition turned him away. Later, the Libyan government asked to use Manhattan's Central Park for a tent, but the request was denied. US Rep. Nita Lowey said Gaddafi had shown a lack of remorse for the bombing and was "unwelcome throughout the New York area."