XBox Series X - Event lobby games stuttering/framerate drop

Triplehoo

I've come across in some games where my framerate would drop significantly when playing in event lobbies. Was just playing in "Mother's Dwelling" and every time I was/went inside the main building (or it's surrounding area), my framerate would drop. And same was when around Killer Shack with other players.

Games in normal lobbies work just fine