You also need to specify which launcher RetroFE will use in the settings.conf of your collection.
So, in order, I think the steps are:
1 - Create a collection (see the included CREATING_COLLECTIONS.txt in the collections folder for instructions . You can also ignore it completely and simply copy one of the already existing collections, changing only the name of the folder. Also works.)
2 - Include your collection in the main RetroFE menu, by adding its name in the menu.txt of the Main collection folder.
3 - Create a launcher for your collection content. Just create a .txt file in the launchers directory with whatever name you like. My build is portable and entirely focused on RetroArch, so my launchers tend to look like this:
executable = %RETROFE_PATH%\..\Emulators\RetroArch\retroarch.exe
arguments = -f -L "%RETROFE_PATH%\..\Emulators\RetroArch\cores\nestopia_libretro.dll" "%ITEM_FILEPATH%"
It varies according to which emulator you're gonna use and where your stuff is located.
4 - Inside your collection 'settings.conf' file, search for the line:
launcher=mame
And change it to the name of the launcher you just created.
It may take some trial and error. If it doesn't work, check the log.txt in the RetroFE folder to see what went wrong.