The lag in Dire Dire Docks is because the collision for the submarine (which is either 200 or 700 triangles, i can't remember) is recalculated every single frame. O2 is like a bandage to the overarching issue.
The only real cause for lag in SM64 is that the RDP graphics chip has a fixed rate for how many pixels it can draw per frame, and that's quickly filled up by large triangles, transparency, any 2-cycle effects like fog, overdraw, etc. (and that's not even taking texture loading latency into account)
I read somewhere that SM64 collisions have no fancy optimization but are just brute force and a first glance at the source seemed to confirm this. Which shows that
1) you really don’t need to optimize prematurely
2) something is very wrong with engines used in modern (indie) games of similar complexity when they perform badly on modern hardware
I don't know how fancy these optimizations are, but it is optimized.
* Tris are sorted by whether they are static (level geometry) or dynamic (moving platforms). So only dynamic tris need to be reinitialized every frame.
* Tris are further sorted into which cells they touch. A cell is a spatial partition of a level. So collision detection only needs to check against tris in your current cell.
* Tris are further sorted into walls, floors, and ceilings (depending on their normal). The find_wall_collisions, find_floor_collisions, etc. functions are each optimized for that particular kind of tri.
* Tris extend their hitbox along a coordinate axis (the up axis for floors/ceilings, the one closest to their normal for walls), instead of along the normal itself. This is cheaper to test, since you can just project down along that axis and it becomes a point-in-tri test in 2D.
The only real cause for lag in SM64 is that the RDP graphics chip has a fixed rate for how many pixels it can draw per frame, and that's quickly filled up by large triangles, transparency, any 2-cycle effects like fog, overdraw, etc. (and that's not even taking texture loading latency into account)