How to seperate buttons so that auto find focus distinguishes between them

I am making a menu for my game, and it has many different “steps” for example and item menu and stats menu, etc.
the problem is when im inside one of those “sub menus” i can press a button in the direction of the “main” menu and it will exit me out, i only want that to happen if you press the back button, how do i do this?