Categories: Web

Is Twitter telling Google and Bing it wants to be indexed more?

File this under intriguing: Twitter has made several changes to its instructions to search engines today which indicate that the social network is looking to become more visible in your search results.

Twitter’s robots.txt file – a small file that webmasters use to allow or prevent search engines from indexing parts of their sites – was changed today to allow search engines to index more of its site.

Specifically, Twitter is now allowing Google, Yahoo!, Bing, Yandex, and others, to index its own search results pages.  Until today Twitter had prevented its search pages from being indexed.

Twitter’s current robots.txt file tells search engines that they can now index twitter.com/search pages but are not allowed to index searches for users, or searches for images or videos (Disallow: /search/*/grid).

Twitter’s current Robots file

Allow: /search
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/with_friends

As recently as September 11 Twitter’s robots file prevented indexing of all search content. e.g;

Allow: /*?*_escaped_fragment_
Disallow: /search
Disallow: /*?
Disallow: /*/with_friends

Twitter’s old Robots file

Twitter has never fully prevented search engines from indexing its content but the site has been less visible in search results over the past 18 months.  Previously, Twitter had an agreement with Google which allowed real time Twitter content to appear in Google search results – that agreement expired in July 2011 and has not been renewed.

Twitter has been undergoing a series of changes of late; some, such as the launch of redesigned profiles last week, have been received well while others, such as the increased limitations on developer access, have been controversial.

This change seems to be designed to give Twitter more visibility but we’ll have to wait and see how Google, Bing, and the others view the newly accessible content.

Ajit Jain

Ajit Jain is marketing and sales head at Octal Info Solution, a leading iPhone app development company and offering platform to hire Android app developers for your own app development project. He is available to connect on Google Plus, Twitter, Facebook, and LinkedIn.

View Comments

  • Good catch. Twitter should definitely open to search engines; it shouldn't have dropped the Google agreement in the first place. The whole web might benefit from this, unlike with their recent API policy changes.

  • Allowing search results to be indexed is generally speaking not considered to be good practice, can create near infinite amounts of pages for Google to crawl...
     
     

    •  @Colin McDermott  Indeed it can be detrimental if there's no Sitemap listing the pages that are more important to crawl, or if the searches create duplicate content. Since Twitter content is so fleeting there shouldn't be duplicate content issues, but that also means that indexed pages will be outdated very quickly...

  • Hmmm...
    I think that if twitter wants to make its sites spam free then he should keep it continue.
    Bu this will decrease its number of members.
    Hireprofessionalseoexpert

Recent Posts

GAN, Tec de Monterrey partnership highlights cross-border startup ecosystem building in Latin America amid trade dispute

Despite recent tensions between the United States government and Latin American countries over migration and…

3 days ago

This founder started out with US $5K to his name. Now, he owns a multi-million-dollar global business

Meet Nitin Seth, the Co-Founder and CEO of Screen Magic (SMS Magic), a messaging leader…

3 days ago

Building smarter: AI, the ultimate tool transforming an old-age industry

In this Brains Byte Back, we sit down with Hari Vasudevan, founder and CEO of…

4 days ago

When AI Goes Rogue: 8 Lessons from Implementing LLMs in the Healthcare Industry that Could Save the Future

By Santosh Shevade, Principal Data Consultant at Gramener – A Straive company All pharmaceutical companies…

5 days ago

Digital Public Infrastructure will enable public, private entities to control your access to essential goods, services & mobility

Digital Public Infrastructure is a top-down agenda coming from unelected globalists, bureaucrats, and their partners…

2 weeks ago

Open Source Claims to Be a Meritocracy—So Why Are Companies Buying Their Way In?

Imagine that you are a maintainer of a widely used open source project relied upon…

2 weeks ago