Web

Is Twitter telling Google and Bing it wants to be indexed more?

Is Twitter telling Google and Bing it wants to be indexed more?

File this under intriguing: Twitter has made several changes to its instructions to search engines today which indicate that the social network is looking to become more visible in your search results.

Twitter’s robots.txt file – a small file that webmasters use to allow or prevent search engines from indexing parts of their sites – was changed today to allow search engines to index more of its site.

Specifically, Twitter is now allowing Google, Yahoo!, Bing, Yandex, and others, to index its own search results pages.  Until today Twitter had prevented its search pages from being indexed.

Twitter’s current robots.txt file tells search engines that they can now index twitter.com/search pages but are not allowed to index searches for users, or searches for images or videos (Disallow: /search/*/grid).

Twitter’s current Robots file

Allow: /search
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/with_friends

As recently as September 11 Twitter’s robots file prevented indexing of all search content. e.g;

Allow: /*?*_escaped_fragment_
Disallow: /search
Disallow: /*?
Disallow: /*/with_friends

Twitter's old Robots file

Twitter’s old Robots file

Twitter has never fully prevented search engines from indexing its content but the site has been less visible in search results over the past 18 months.  Previously, Twitter had an agreement with Google which allowed real time Twitter content to appear in Google search results – that agreement expired in July 2011 and has not been renewed.

Twitter has been undergoing a series of changes of late; some, such as the launch of redesigned profiles last week, have been received well while others, such as the increased limitations on developer access, have been controversial.

This change seems to be designed to give Twitter more visibility but we’ll have to wait and see how Google, Bing, and the others view the newly accessible content.

Web
@pdscott

Piers Dillon-Scott is co-editor of The Sociable and writes about stuff he finds. He likes technology, media, and using the Oxford comma (because it just makes sense).

More in Web

The Four Courts, Dublin

Chilling Effects indicates it may list ‘Right to be forgotten’ requests from EU citizens

Piers Dillon ScottJune 7, 2014
The new Google Maps – On Browser – Hello World – Google Maps

Google I/O gives us a sexy new Google Maps

Piers Dillon ScottMay 15, 2013
Big Ben, Saint Stephen's Towier in London

Why the Queen wants your IP address

Piers Dillon ScottMay 8, 2013
slow down

Google says the web is getting faster (unless you live in certain countries)

Piers Dillon ScottApril 16, 2013
Cloud storage provider Dropbox gets serious about business

Dropbox gets serious about business, introducing added security features

Darren McCarraApril 10, 2013
The Field, available to watch on RTÉ Player later this week

RTÉ Player to include movies for the first time

Darren McCarraApril 2, 2013
Page load times of the top ecommerce sites are increasing

Top ecommerce sites around the world are slowing down

Darren McCarraMarch 26, 2013
Internet Explorer on Windows 8

Is Microsoft going to piss off every web developer with the release of IE11?

Piers Dillon ScottMarch 25, 2013
Red phone boxes

Google reportedly developing a unified chat service, Babble, to rival all others

Darren McCarraMarch 19, 2013