Baldurs Gate 3 had a similar issue that they had to fix. Performance by Act 3 would be super bad and things would load very slowly or crash. Turns out it was due to all the NPCs judging every single crime you’ve committed over your entire playthrough.

I thought it was really neat.

I just got the game recently, and I guessed as much considering the dips are mainly where there are lots of NPCs, and noticing that their pathing and behavior are just … Inefficient. Every time I let a pawn take the lead somewhere they themselves say they know the way to, they take the longest possible route and even backtrack a ton of times.