" />
Web

Is Twitter telling Google and Bing it wants to be indexed more?

Is Twitter telling Google and Bing it wants to be indexed more?
piers.scott@sociable.co'

File this under intriguing: Twitter has made several changes to its instructions to search engines today which indicate that the social network is looking to become more visible in your search results.

Twitter’s robots.txt file – a small file that webmasters use to allow or prevent search engines from indexing parts of their sites – was changed today to allow search engines to index more of its site.

Specifically, Twitter is now allowing Google, Yahoo!, Bing, Yandex, and others, to index its own search results pages.  Until today Twitter had prevented its search pages from being indexed.

Twitter’s current robots.txt file tells search engines that they can now index twitter.com/search pages but are not allowed to index searches for users, or searches for images or videos (Disallow: /search/*/grid).

Twitter’s current Robots file

Allow: /search
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/with_friends

As recently as September 11 Twitter’s robots file prevented indexing of all search content. e.g;

Allow: /*?*_escaped_fragment_
Disallow: /search
Disallow: /*?
Disallow: /*/with_friends

Twitter's old Robots file

Twitter’s old Robots file

Twitter has never fully prevented search engines from indexing its content but the site has been less visible in search results over the past 18 months.  Previously, Twitter had an agreement with Google which allowed real time Twitter content to appear in Google search results – that agreement expired in July 2011 and has not been renewed.

Twitter has been undergoing a series of changes of late; some, such as the launch of redesigned profiles last week, have been received well while others, such as the increased limitations on developer access, have been controversial.

This change seems to be designed to give Twitter more visibility but we’ll have to wait and see how Google, Bing, and the others view the newly accessible content.

View Comments (12)

12 Comments

  1. miguel@silvar.net'

    Miguel SilvaR

    September 26, 2012 at 7:33 AM

    Good catch. Twitter should definitely open to search engines; it shouldn’t have dropped the Google agreement in the first place. The whole web might benefit from this, unlike with their recent API policy changes.

  2. colin.m.mcdermott@googlemail.com'

    Colin McDermott

    September 26, 2012 at 12:03 PM

    Allowing search results to be indexed is generally speaking not considered to be good practice, can create near infinite amounts of pages for Google to crawl…
     
     

    • miguel@silvar.net'

      Miguel SilvaR

      September 27, 2012 at 8:52 AM

       @Colin McDermott  Indeed it can be detrimental if there’s no Sitemap listing the pages that are more important to crawl, or if the searches create duplicate content. Since Twitter content is so fleeting there shouldn’t be duplicate content issues, but that also means that indexed pages will be outdated very quickly…

  3. Pingback: Is Twitter telling Google and Bing it wants to be indexed more? - The Sociable | Local Search Engine Marketing for Contractors | Scoop.it

  4. Pingback: SearchCap: The Day In Search, September 26, 2012

  5. Pingback: Twitter Opens Up To More Crawling, But Do Search Engines Want Its Search Results In Theirs?

  6. Pingback: Twitter Ad Revenue Play: Now Allowing More Search Engine Indexing

  7. Pingback: Twitter ‘People you may know’ Weekly Email; Certified Products Program Introduced | diTii.com - All About Technology

  8. dwividsharma3090@gmail.com'

    dwividsharma3090

    November 2, 2012 at 9:16 AM

    Hmmm…
    I think that if twitter wants to make its sites spam free then he should keep it continue.
    Bu this will decrease its number of members.
    Hireprofessionalseoexpert

  9. Pingback: Searching For Mobile Apps May Be In The Cards

  10. vedatyami@gmail.com'

    vedatsafakyami

    June 9, 2013 at 9:50 PM

    search engines do not index your twitter profile..

  11. Pingback: Twitter Ad Revenue Play: Now Allowing More Search Engine Indexing | Search Engine Watch

Leave a Reply

Your email address will not be published. Required fields are marked *

Web
piers.scott@sociable.co'
@pdscott

Piers Dillon-Scott is co-editor of The Sociable and writes about stuff he finds. He likes technology, media, and using the Oxford comma (because it just makes sense).

More in Web

24 senators introducing bill to kill FCC Internet privacy rules received a combined $2.29M from industry PACs

Tim HinchliffeMarch 21, 2017
g suite

Startup Workep gives G Suite the project management tool it’s been lacking

Tim HinchliffeMarch 21, 2017
wikileaks google

Is Google’s hesitance on WikiLeaks’ cybersecurity offer due to a technicality or actual CIA involvement?

Tim HinchliffeMarch 20, 2017
utilities

Moving home? My Utilities sifts through 75K competing providers so you don’t overpay on bills

Tim HinchliffeMarch 7, 2017
perspective google

Google’s new Perspective API can help you not sound like a jerk while commenting

Tim HinchliffeFebruary 23, 2017
microsoft ceo

AI represents the ‘3rd run time’ in tech behind OS, browser: Microsoft CEO

Tim HinchliffeFebruary 21, 2017
techstars

BoozeTech: 10 companies fill Techstars Connection Demo Day lineup to the frothy brim

Tim HinchliffeFebruary 7, 2017
online privacy

These Online Privacy Hacks Will Help Protect Your Identity

Thomas MilvaFebruary 4, 2017
cia journalist

Who was the late German journalist who claimed CIA controlled European media?

Tim HinchliffeJanuary 18, 2017