Sometimes border guards take the job because they like going through people's underwear. Sometimes it's the only job in the area with decent pay and benefits. Sometimes they believe they're helping protect their nation.
I've been crossing the US-Canadian border for five decades because my family comes from northern Minnesota, and my parents and sister became Canadian citizens in the '80s. My experiences with Canadian guards were usually good—even when I was a young hippie who looked like the enemy of all they stood for. My experiences with American guards were usually bad—even when I became a middle-aged, conventionally dressed and coiffed fellow who drove cars that had been made within the last few years.
There were two good crossings into the US that I remember. Both were at small stations, which tend to be more pleasant than big ones. At the first, I declared a 100-pound bag of wild rice that had been given to me by my family, who were buying wild rice from Ojibwe people in northern Ontario. The guard asked, "Is that for personal use?" with a straight face. I said "Yes," and maybe I added something about the family, and we both kind of smiled, and he waved me through. He understood stocking up on the good stuff.
The other time, Emma and I mentioned we had been to a convention because we're science fiction writers. The guard had worked with Jack McDevitt, one of many people who demonstrate that you can be a customs guard and a damn fine person.
But it is always best to assume that people with authority are power-mad half-wits who will happily squash you like a bug given the least opportunity. Be very polite to them, and make no sudden moves, and you may be pleasantly surprised.
As for Peter Watts, his only mistake was being a Canadian who did not realize he had entered a third-world nation whose citizens are expected to cower before authority.
ETA: Just to make this clearer: I've known great cops, soldiers, and guards. I can't imagine a human society in which there would not be some form of cops, soldiers, or guards, simply because some people because of mental illness or greed (yes, that may be redundant) will want to make other people suffer. All societies need people who will protect the weak from the malicious.
But you're still safest with cops, soldiers, and guards if you assume they're looking for an excuse to hurt you.
ETA 2: The best way to deal with cops, soldiers, and guards is to smile and be as helpful as you possibly can. They didn't make the rules they're enforcing.