Google's newest method for crawling your websites, is to use your RSS/Atom feeds to discover new page URLs. Google found the RSS/Atom feeds far surpass Google's traditional crawling method in speed and accuracy. Google traditionally follows links from one source to another in a spiderweb fashion discovering and indexing new URLs as it crawls through the pages.

Google recently adopted this new method to speed up the discovery process and they feel it has allowed them to discover and deliver new content faster. Rss/Atom feeds have been gaining popularity recently, and can deliver a lot of content very rapidly. Google will use your Google Reader, as well as directly crawling the feeds to access the new links.

They are also considering accessing publishing hubs like PubSubHubbub to identify newly updated content. If you want to benefit from this new method and have your links indexed more quickly, you want to make sure you allow Google's crawlers to access those feeds on your site. The best way to check that is to look in your robots.txt file. Google Webmaster tools has a robots.txt tester on their site click here to check it out robots.txt.

Get your seo articles written by an seo expert. For seo articles that bring your website to life, check out this page SEO Article Writing