Even tough the target for this blog its not to rank sky high, some things must be done properly, so this is why decided to modify robots.txt file.
The good thing is that Nucleus CMS its super simple, so as you expect, you’ll find that file on the root of your site. Download it with your usual WYSIWYG editor, or FTP program, modify it, and upload it again. Kid’s stuff!!
By default looks like this:
But I propose something more refined:
Meaning that all the web crawlers ( User-agent: * ) would have to NOT ( Disallow: ) directories like /extra/ ( where some goodies are stored ), and /skins/ ( obvious ).
Action.php its a file containing actions that can be performed by visitors of the site, like adding comments, etc… so by default its ok to protect it.
Not clear to add more items to robots.txt, think it’s time for sitemap.xml.