I made some progress. RetroFE runs very well on wine. Everything works, except launching games. I can't figure out how to launch a native app and tell it to open a file. The closest I've gotten is opening RetroArch with wine cmd and:
cmd /c start Z:/Applications/RetroArch.app/Contents/MacOS/RetroArch --verbose
That's just from the terminal. I can't yet create a launcher that opens RetroArch.
I'd make a separate thread, but only mac users would care to use wine I think.
edit: got it working! Has some issues, but it launches. For anyone who wants it, try something like this as a launcher for RetroArch mGBA:
executable = c:\Windows\System32\cmd.exe
arguments = cmd /c start Z:/Applications/RetroArch.app/Contents/MacOS/RetroArch --verbose -L "/Applications/RetroArch.app/Contents/Resources/cores/mgba_libretro.dylib" "/Volumes/Games/Console/Game Boy Advance/%ITEM_NAME%/%ITEM_FILENAME%"