RetroFE Meta Database

[dustind900]

I am wondering if the database gets recreated every launch or if the data is persistent? I would like to import data directly to the database instead of using hyperlist xml files. If this is possible I could easily write a program that will scrape data directly to the database. I already have the code to search and fuzzy match results from GameFaqs, MobyGames, and TheGamesDB just laying around.

[Pieter Hulshoff]

The database gets recreated every time the RetroFE executable or any of the xml files are newer than the meta.db file, at least with the most recent versions of RetroFE.

What language is that scraping code written in?

Sent from my SM-G920F using Tapatalk

[dustind900]

C#

Would you prefer a different language? Or can you import dlls into your code?

[Pieter Hulshoff]

Not in Linux; I'd like to take a look though to see if I can port it to C++.

Sent from my SM-G920F using Tapatalk

[Agent47]

Just a heads up, GameFAQs will block your IP for scraping so I'm not sure that's something you would want to include in your codebase.

[Pieter Hulshoff]

Just a heads up, GameFAQs will block your IP for scraping so I'm not sure that's something you would want to include in your codebase.

Thanks for the warning. 😀

Sent from my SM-G920F using Tapatalk

[dustind900]

Just a heads up, GameFAQs will block your IP for scraping so I'm not sure that's something you would want to include in your codebase.

No, that's not entirely true. GameFaqs blocks people because they are GREEDY and try to scrape entire collections at once. Unless a site offers an API you should not do a mass data scrape. If you want to scrape data from GameFaqs you have to do it just like you are surfing the site manually and copy/pasting the info. I have been scraping data from them for a long time now without issue.

I can probably manage to rewrite the code in C++. I'm not too familiar with the language, but it would take me less than a week to learn it. Or I can make an app for everyone to use. I thought about making a scraper that will import HyperLists add missing data using all sources listed above. But the user could set the order in which the data is retrieved. Search whatever site is first on the list and if some data is still missing fallback to the other sites to get the rest.

[Agent47]

I should've been more specific. I know people have been blocked, but not the specifics of how they were going about it. I assumed if the scraper was implemented to grab info upon adding new collections that it could trigger a block or have to be severely throttled to avoid it.

[Dunuin]

A scraper would be great. You could also use no-intro p/c DATs so the scraper knows the names of all clones. So it can lookup
 the clones if the ROMs name isn't found online. That would really help non-US collectors.

An example:
"Blackhawk (Europe) (En,Fr,De)" and "Blackthorne (USA)" are the same SNES game with probably the same genre, number of players, manufacturer etc (but release year and languages may differ). So if the German "Blackhawk" isn't found online on a English DB, why not offer an option to look for clones like "Blackthorne" also and crawl atleast the genre and other infos which normally doesn't change between regions.

And don't forget to add an option to remove the "(*)" from the ROM name before searching online.
And if a ROM name ending with ", The" isn't found online, also try to search with a "The " in front of the name.

[dustind900]

A scraper would be great. You could also use no-intro p/c DATs so the scraper knows the names of all clones. So it can lookup
 the clones if the ROMs name isn't found online. That would really help non-US collectors.

An example:
"Blackhawk (Europe) (En,Fr,De)" and "Blackthorne (USA)" are the same SNES game with probably the same genre, number of players, manufacturer etc (but release year and languages may differ). So if the German "Blackhawk" isn't found online on a English DB, why not offer an option to look for clones like "Blackthorne" also and crawl atleast the genre and other infos which normally doesn't change between regions.

And don't forget to add an option to remove the "(*)" from the ROM name before searching online.
And if a ROM name ending with ", The" isn't found online, also try to search with a "The " in front of the name.

I absolutely hate language tags. Grrrr. I have removed all revision and language tags from my roms.

As far as using a dat, that could be done rather easily.

Most of the games on these sites are easy to match using replacements instead of fuzzy matching.


If this is something people would use I'll start a new topic and we can go from there. The data would need to have a static format that way it would be universal. Rating for example we can't just have "E" for one game then "E - Everyone" for another. It has to be uniform.

Sent from my LG-H950 using Tapatalk