Gary Walters has a “special feeling” about the White House East Wing. He met his future wife Barbara when she worked in the visitors’ office there. But asked to contemplate the wing’s destruction by Donald Trump, the former chief usher evidently still believes that discretion is the better part of valour.

“All the presidents and first ladies have made changes in one manner or another – some larger than others,” Walters, 78, says with the measured cadence of a man who has spent a lifetime guarding privacy. “One of the things that I have seen not commented on was back to when the West Wing was built.

“It used to be on the west side of the White House there were large glass conservatories and stables for a short period of time. Those were all torn down, which allowed the colonnades to the west side and then the building of the West Wing. Nobody’s talking about that, but that was just as large a project.”

It is a typically diplomatic response from a man who spent 37 years serving seven presidents and their families and trying not to upset the apple cart. His book White House Memories 1970-2007: Recollections of the Longest-Serving Chief Usher, published on Friday, strikes a rare bipartisan chord by praising Democratic and Republican presidents alike.

Walters began his career as an officer in the executive protective service with an assignment to help protect President Richard Nixon and then his successor, Gerald Ford. He joined the usher’s office as an assistant in 1976 and a decade later was promoted to chief usher, serving Ronald Reagan, George HW Bush, Bill Clinton and George W Bush. He retired in 2007 and lives in Great Falls, Virginia.

The chief usher is responsible for making the White House as comfortable a home as possible for the first family. It is comparable to a general manager, overseeing maintenance, construction and renovation projects as well as food service and administrative, financial and personnel functions. A staff of about 90 to 100 butlers, housekeepers, cooks, florists, electricians, engineers, plumbers and others report to the chief usher.

Walters speaks of Reagan and his wife Nancy with particular warmth. “There was so much controversy about the president being from Hollywood and he was an actor and he was on stage. President Reagan was a person where what you saw all the time on the television or wherever was what you got all the time. He was a tremendously cooperative and enjoyable person to be around.”

Walters recalls that Nancy Reagan initiated “trial dinners”, full dress rehearsals for state banquets where she and the president would sample the menu and service to ensure perfection for their guests. “Mrs Reagan wanted to make sure that her guests at the White House were treated extremely well,” he notes.

Among those guests were Margaret Thatcher and her husband, Denis. Walters continues: “When Mrs Thatcher went over to speak with the president, Denis Thatcher was left behind, and I ended up spending about 20 minutes with him.

“I showed him around the White House and I even took him outside and showed him what we saved from the burning of the White House in the war of 1812. When we repainted the White House, we didn’t paint a couple of areas: we left the scorch marks there as a remembrance. He wryly told me that I didn’t need to show him that.”

The usher’s office gave Walters a front-row seat to the end of the cold war. He recalls the visit of the Soviet leader Mikhail Gorbachev in 1987. “There was a rather sharp divide at that time,” he says, recalling the tense summit in Reykjavik, Iceland, but when Gorbachev arrived at the White House, the atmosphere shifted.

“They signed the Intermediate-Range Nuclear Forces Treaty in the East Room, and then after that they walked down the hall back to the State Dining Room. There was a fire going in the room. There were two podiums in front of the fireplace and an interpreters’ booth on one side of the room and the television cameras.

“I was one of the only people in the room as this was taking place. As the two presidents talked to the world, I literally could feel the thawing of the cold war and that was a very moving experience.”

Walters came to know the domestic habits of the most powerful people on Earth better than anyone else. When asked, his team would help look after presidential pets. “President George HW Bush at one point sent out a memo to his staff that they had to stop feeding the dog treats, that even Ranger was putting on a few too many pounds.”

Each presidential family arrives with their own particular needs. Clinton, who had pounded the open streets of Little Rock, Arkansas, wanted to continue jogging, which was a potential security nightmare.

Walters says: “When he came to the White House, there was a lot of concern about the president being out jogging around Washington. Before they actually moved in, Mrs Clinton asked me about the possibility of providing a jogging track.

“I came up, along with the National Park Service, with removing a portion of the south roadway and putting a synthetic turf jogging track all the way around the interior south roadway. That became the jogging track for the president to use.”

When in January 1998 the Drudge Report broke news of Clinton’s affair with the young intern Monica Lewinsky, questions were asked of almost everyone about what they knew. Walters examined the usher’s daily logs and found no mention of anyone named Lewinsky going to the private quarters in the executive residence.

In the September, Clinton hosted a breakfast for Black religious leaders. When he finished his opening remarks, Walters writes in the book, “some of the ministers started to crowd around the president, and I started to move to ask them to remain in their seats. But Mrs Clinton held me back, saying, ‘Leave them alone. The president needed this.’”

Next was Bush, who was such an avid baseball fan that Walters arranged a special satellite connection so the president could watch different games.

Then came 11 September 2001. With Bush in Sarasota, Florida, the White House was preparing for the annual congressional picnic, scheduled for that evening. A catering company from Texas had brought tractor trailers with massive cooking grills and had been cooking for two days already.

Walters says: “We had serving tents up and picnic tables out in stages. As soon as I learned that there had been another attack by a second plane up in New York, we went out and started dismantling as much as we could, because I knew in my heart that President Bush was going to come back to the White House.”

The south lawn was set up with more than 160 tables for the event, blocking the usual landing pad for Marine One, the presidential helicopter. Walters and his remaining staff began to clear the lawn by hand, carrying some of the heavy tables to the perimeter to make room for Marine One to touch down.

The Secret Service ordered an evacuation but Walters and his skeleton staff stayed behind. As the day unfolded, a new threat emerged: a report of another hijacked plane heading towards Washington.

“At one point I actually made a joke to the fellows that were there with me. I said, I’m standing here with my legs spread apart because, if I put my knees any closer together, they’d probably be banging together like a bass drum. We were all aware of what was going on and it wasn’t comfortable.”

Walters is convinced that United Airlines flight 93, which crashed in a field in Shanksville, Pennsylvania, was destined for the White House. “The third plane was downed by wonderful American patriots in Shanksville, Pennsylvania. In my heart, I still believe that the terrorist plane was headed for the White House, not for the Capitol, and I believe that those people saved my life.”

Walters was on hand when Bush returned to the White House to address the nation that evening. “I walked with him back from the Oval Office to the residence and took him up on the elevator and we had brief conversations about his travel during the day, and he was very sombre but also very concerned about what was going to happen from that point forward.”

Washington had instantly changed. “Kind of on a war footing. Didn’t know what the next shoe to drop was. Cautious but determined.”

Walters initiated a tradition of presenting the outgoing president with two flags: the one that flew over the White House on his first day in office and the one that flew on his last. As for all the ghost stories reported down the years by various alleged witnesses, Walters never saw an apparition himself.

But he does note: “There were people on the staff who thought they had interactions or had odd feelings when they were going in the vicinity of the Lincoln Bedroom.

“One young man on the staff swears that he went into the room one day to retrieve something for the curator’s office and he walked in the room and he heard a sound and he looked over and the rocking chair was moving. He swore that President Lincoln was in the room with him.”