Skip to content

mypapit gnu/linux

Free and Open Source blogger with an attitude

Tag: crawler

Posted on November 19, 2007November 19, 2007

robots.txt – Uncrawlable code.google.com ?

I found this thing by accident when I tried to search for my project hosted on Google Code Hosting service website.

http://code.google.com/robots.txt
[code]
User-agent: *
Disallow:
[/code]

Does Google intentionally block other web crawler from indexing Google Code site? or was it a glitch?

  • X
  • GitHub
  • YouTube
  • LinkedIn

Follow Me

Mypapit GNU/Linux Blog

Subscribe

Subscribe RSS Feed

Pages

  • About
  • Contact Me
  • Links
  • Privacy Policy

linux blog

734d5b34a0fe661550799aea702038486fe25365e7ca00f28b
Privacy Policy Proudly powered by WordPress