RetroFE will throw that error of launching the executable results in an error code by the program. The program not being executable or not present would result in this message, but if Project64.exe would close itself, and throw back an error code, it would also print this message. If however, as you say, the program should open up with a gui with an error message, then it does seem either the file path of the executable is wrong, or the executable itself.
Can you upload a screenshot of the path in the executable properties, and see if you can launch it from the command line?