serendipitously, over the last year I switched to 165's on my MTBs after being on 175s for basically ever. It started with some curiosity and gas money not being used so diverted on a closeout crankarm set.
I've had them "all over" including Western NC riding, used them with 29er and 27.5 wheels, 30t and 32t ring (with same cassette).
Upshot is that I don't see any down sides, for me and my physiology, and stupidity level, anyway. There's enough punchy stuff in some of my NC trails that I'd stack against BCGB punch to think that there wouldn't necessarily be a problem there.
If someone is concerned about raw power loss you'll be withing small % difference by simply changing the chainring 2t smaller, when I initially switched I put a 30t on but have since gone back to 32t.
Do I think everyone should just down-size? Probably not.
Do I think people are mostly over-cranked? Probably yes.
The only tangible reason I can think someone should down size is if there's some kinestic reason (e.g. hugely better bike fit / pedalling dynamic - if that exists) or of they really REALLY need to cut down on pedal strikes.
And yes, I did have to raise my saddle 10mm to compensate for the change, I ended up having a knee issue before I did that.
Now I just need to find a cheap Campy 2x crankset to change out my road bike ... ha ha ha .. like that's going to happen.