The saying "Take only pictures, Leave only footprints, Kill only time", which likely first appeared as a slogan from the National Parks Service in the 1950's as "Take only pictures, Leave only Tracks"1 comes to mind for this principle. The first part is what we are really talking about with the second part being #2 Travel and Camp on Durable Surfaces and the third part being #6 Respect Wildlife.
Think of the wonder you have hiking through the woods and coming across that set of shedded antlers. Shouldn't the next hiker have the joy of seeing them too? What about when you find that cool rock by the lake? Shouldn't the next person there be able to see it too? This all goes to the idea of us changing the environment by taking something home. You may have heard someone say, "If everyone did it, there would be nothing left". While this may seem like hyperbole, it is the truth.
There may also be legal considerations here too. Depending on the state or park, even man made stuff might be protected. It could be that something that was left there as little as 20 years ago is now considered an artifact, and illegal to remove. When you're talking remnants of the indigenous people, you may run into higher protections. Instead of trying to know all the laws for every area you are, its best to just take a picture and leave it be. In the event of artifacts relating to indigenous peoples, you should report it to the appropriate agency so that they are aware of it.2
In the last decade or more, the National Park Service and other conservation and ecological organizations again has been pushing to stop rock stacking. There are many reason for this both visually and ecologically. Stacking rocks can disturb habitat for animals, especially when you're talking about rocks in and along streams and rivers. It also can disturb the beauty of the area, even though it may be beautiful to the person that stacked the rocks. Like art, it's in the eye of the beholder. It is best to just leave the rocks where you found them. If you find rock stacks that are not for navigational purposes (see below), feel free to spread the rocks around, being careful to not do any extra harm in the process.
That all said, there are places where stacked or placed rocks are appropriate. This is when they are being used for navigation. Once you get above tree line, especially in places where you experience winter, cairns are used for navigation. These can be massive in size and take months to construct so that they withstand the harsh environment. There are also rocks that are used to line the edge of the trail, or pack into dirt to support it and prevent erosion. As hikers, unless directed by the land manager, we should not be touching or moving these rocks at all. That said, the anti-cairn movement has had negative consequences with 11 of 12 cairns that had stood for decades on an Adirondack high peak were destroyed by what has to be multiple people in one afternoon/evening.2
Another place you may find cairns that you should not remove are along herd paths. These are typically small, generally only a couple to dozens of rocks and rarely artistic but more piled. These are navigational cairns, but not anything official. These are usually erected by hikers to help point out where to cross streams or where to exit a slide. They should be treated like navigational cairns, and these are ok to add a rock or two if you happen to be inclined. Finally it's not uncommon to find cairns to mark the peak of a mountain or hold up a sign. These are landmark cairns and again should not be removed because they are a specialized type of navigational cairns.
In the end, when in doubt, take a picture and don't touch or change anything.