CoinOPS is using a newly built feature I haven't fully tested and released yet: collectionUp/Down keys. It acts exactly like pressing back, next/previous collection, enter, with the possibility of scrolling through multiple collections before letting go of the key. It works quite well with SSDs, but entering collection takes a bit of time with an (external) HDD, so not everyone may like it.
The easiest way is probably to grab the default Aeon Nox layout.xml or a CoinOPS layout.xml file, and work from there. I presume you've read the manual in this regard?
What should be understood (and perhaps better explained in the manual) is that when you enter a layout file, the menuIndex is set to 0. For every collection you enter, menuIndex is increased, and for every collection you exit menuIndex is decreased. All menu definitions and animations use this menuIndex. If a menu doesn't have a menuIndex parameter, it simply takes the previous menuIndex, and increases it by 1. By using the menuIndex as part of the animation parameters, you can make each menu animate however you want depending on the menuIndex it's at. If for instance you want the main menu to remain visible as you scroll through the games, simply don't fade it out as you leave menuIndex 0.