Twitter (X) has recently updated its robots.txt file to block Bingbot, the web crawling bot used by Bing Search, from accessing and indexing content on X.com. This decision is interesting in the context of search engine indexing and how social media platforms manage their relationships with search engines.
X has explicitly disallowed Bingbot from crawling and indexing its content by adding a directive in its robots.txt file. Website owners use Robots.txt files to communicate with web crawlers about which parts of their site should or should not be crawled and indexed.
When you use a “site:” command on Bing, you only see 180,000 X URLs, whereas Google shows 422 million X URLs. This suggests that X’s content is far less visible on Bing than on Google, likely due to this recent blocking of Bingbot.
X had previously blocked Google from indexing its content, which resulted in a significant drop in X’s visibility on Google Search. However, they later reversed this decision. It’s unclear why they made this change, but it could have been related to negotiations or concerns about content control.
via SERoundTable
